NVMe benchmarking

That’s strange.

For me, for a collection of 4 middle-of-the-road somewhat mature computers with at most NVMe PCIe Gen 3 SSDs (but all with SSDs of some sort), I get consistently somewhere in the range 94-118 MiB/s. For a much more recent computer with an NVMe PCIe Gen 4 SSD, it varies wildly between 170 and 670 MiB/s but then suddenly seems to settle for a while at around 20 MiB/s. So, yeah, something odd with the testing. And not just your computer / your environment.

  1. NVMe drives can get hot, it could be throttling due to temperature.
  2. The speed for most NVMe is very dependent on cache speeds and flattening that much is a symptom of cache exhaustion (small) combined with slower NAND storage.

I wondered about that so used smartctl to confirm that, no, the temperature is fine. It was way way way below the “warning” temperature (as displayed by that command) albeit that I can’t guarantee that it wouldn’t throttle below that but in any case a very mild temperature (below 30C).

On a different computer, where benchmark gives around 90 MiB/s, it says that the NVMe drive is at 46C (after the test) and warning is at 85C.

I think it’s very likely it would start to throttle below the warning temperature because 85C is typically the top of the range, both operating and non-operating. But I doubt that I would see thermal throttling for the temperatures that I am seeing.

I wondered about that too.

I note that the RAM cache on the NVMe drive in the computer mentioned in this post is (reportedly, I don’t know the command to check it) only 512 MB. So a 4GB test is going to fill the cache many times over and you would have to think that the throughput displayed by the test reflects the true throughput that the drive can sustain.

12 or 20 MiB/s is not really credible even, say, if the on-drive cache were disabled, is it?

There could be some Linux weirdness, since none of my computers is (yet) running anything like the latest kernel. But really I think it needs more investigation. (I observe that just last night, for the first time, Ubuntu is badgering me everywhere to do an in-place upgrade to 26.04, which you are probably getting too.)

It depends on how slow your NAND is on that drive. On some drives it can be pretty slow.

It’s not completely characteristic of a decrease during a single benchmark run, but it is important to note that fstrim (TRIM) is turned off on FDE systems in many distros due to security issues with TRIM. It’s a tradeoff, of course, but I would turn fstrim on (it’s three steps …).

And I guess encryption itself is one factor in the speed of writing to the disk. (I had kind of hoped that it was smart enough to bypass the disk encryption, even though that would be quite tricky to do.) I think it may be better just to test this on a simpler disk setup i.e. encryption off and TRIM on.