Why does my NVMe SSD disappear under load until I power-cycle?

1. 1.Read every nvme line the kernel logged, in order. The sequence tells you more than any one line, because the driver escalates: it aborts a timed-out command, shuts down and resets the controller if the same command times out again, and gives up if the reset fails.

   sudo dmesg -T | grep -i nvme

   Reading'I/O tag N (...) QID N timeout, aborting' means a command ran past nvme_core.io_timeout (30 seconds by default). 'timeout, reset controller', or a second timeout on the same tag, means the driver is resetting the controller. 'timeout, completion polled' means the driver found the command complete by polling; the kernel's comment there is 'Did we miss an interrupt?'. 'controller is down; will reset: CSTS=0xffffffff' means the controller status register read back as all ones: the host could not reach the device at all (power state, link, power or the drive). Any other CSTS value with bit 1 set means the controller itself reported Controller Fatal Status. On current kernels the all-ones case, and only that case, adds 'Does your device have a faulty power saving mode enabled?' with a suggested set of boot options. 'Device not ready; aborting reset' means the drive did not report ready during the reset. 'Disabling device after reset failure' (older kernels log 'Removing after probe failure status: -19') means the driver has given up. The -T timestamps drift after suspend and resume.
2. 2.Before you power-cycle, read the controller's state from sysfs. Use the controller name from the log in place of nvme0.

   cat /sys/class/nvme/nvme0/state

   Reading'live' is healthy. 'resetting' or 'connecting' means the driver is mid-recovery. 'dead' means the reset failed and the driver gave up. If /sys/class/nvme/nvme0 no longer exists, the controller was removed.
3. 3.After the power cycle, read the kernel log of the boot that lost the drive. -b -1 selects the previous boot.

   journalctl -k -b -1 | grep -i nvme

   ReadingThese are the same lines from the failed boot. Empty output, with no earlier boots in 'journalctl --list-boots', means the journal is volatile and the previous boot's log was not kept: set Storage=persistent in journald.conf (or create /var/log/journal) before waiting for the next drop-out. If the drive that dropped out holds /var, the last lines of the failure may not have reached disk. Note the time of the first timeout against what the machine was doing: idle for hours points at power states (p3), sustained writes or scrubs at heat or power (p2, p4).

1. 1.Read the drive's own temperature thresholds and capabilities first, because the thermal counters in the SMART log depend on them.

   sudo nvme id-ctrl -H /dev/nvme0

   Readingwctemp is the Composite Temperature at which the drive reports overheating while it keeps operating; cctemp marks a critical overheating condition. Both are in Kelvin: subtract 273 for °C. hctma, mntmt and mxtmt say whether, and within what range, the host can set thermal throttling. apsta reads 'Autonomous Power State Transitions Supported' or 'Not Supported'. npss and the 'ps N' descriptors give each power state's max power, entry and exit latency in microseconds, and whether it is non-operational. fr is the running firmware revision. Labels here are from nvme-cli 2.8 and can differ in other versions.
2. 2.Read the SMART log, repeat the workload that preceded a drop-out, then read it again. The thermal fields are lifetime running totals, so the change between the two readings is what matters.

   sudo nvme smart-log /dev/nvme0

   Readingtemperature is the current Composite Temperature in °C and K. 'Warning Temperature Time' counts minutes at or above wctemp and below cctemp; 'Critical Composite Temperature Time' counts minutes at or above cctemp. A count that rises across the workload means the drive reached its own thresholds under that load. 'Thermal Management T1/T2 Trans Count' and 'Total Time' (seconds) record host-controlled thermal management transitions. Both minute counters read 0 if the drive's threshold is 0, so a 0 proves nothing until you have checked wctemp and cctemp. critical_warning: 0x2 temperature, 0x4 reliability degraded, 0x8 media placed in read-only mode. media_errors above 0 means unrecovered data integrity errors: a drive problem, not a power-saving one.
3. 3.If id-ctrl reports Host Controlled Thermal Management as supported, read the thresholds the host has set.

   sudo nvme get-feature /dev/nvme0 -f 0x10 -H

   Reading'Thermal Management Temperature 1 (TMT1)' and '2 (TMT2)' print in K and °C; 0 means not set. Feature 0x10 is Host Controlled Thermal Management.
4. 4.Compare what you measured with the maker's rating for your model. Two vendor documents were read for this page and each applies only to its own model. Samsung rates the 990 PRO (M.2 2280) for 0°C to 70°C measured by its SMART temperature, with 'Proper airflow recommended', and lists 5.0-5.8 W average active power against 50-55 mW idle with APST on. Kioxia lists its CD8P-R drives (2.5-inch, 15 mm) with operating ranges of 0 to 72 °C, 0 to 73 °C and 0 to 76 °C on the SMART composite temperature, and 18-24 W typical active power by capacity. We have not found a vendor document stating how much power an M.2 slot, a U.2 cable or an OCuLink or SlimSAS adapter can deliver, so this page gives no figure for one.

1. 1.Check what the kernel lets the drive do on its own. Linux enables APST (Autonomous Power State Transition) by default and never lets the drive enter, by itself, a non-operational power state whose latency exceeds nvme_core.default_ps_max_latency_us.

   cat /sys/module/nvme_core/parameters/default_ps_max_latency_us

   Reading100000 is the kernel default: APST may use states up to 100 ms of latency. 0 means APST is off, so a boot option to that effect took hold. A drive-specific value such as 6000, used by testers in one Ubuntu bug, keeps APST but excludes deeper states whose latency exceeds it. The I/O timeout is alongside it in /sys/module/nvme_core/parameters/io_timeout (30 by default).
2. 2.Ask the drive whether APST is enabled. APST is feature 0x0c.

   sudo nvme get-feature /dev/nvme0 -f 0x0c -H

   Reading'Autonomous Power State Transition Enable (APSTE): Enabled' means the drive moves itself between power states under the table the kernel programmed; 'Disabled' means it does not (after default_ps_max_latency_us=0 or a quirk that turns APST off). Add -s 3 to read the supported capabilities.
3. 3.Note the drive's PCI vendor:device ID and its model (the model from nvme id-ctrl), and compare them with the kernel's quirk tables. A drive or drive-and-board pair the kernel already knows about gets its deepest power state, or APST entirely, switched off automatically on a kernel that carries the entry.

   lspci -nn

   ReadingAs read from the kernel source on 2026-10-14; the lists change between kernel versions, and the entries named here are examples, not the full tables. Entries that block the deepest state (NVME_QUIRK_NO_DEEPEST_PS) by PCI ID include Intel 600P/P3100 (8086:f1a5), Silicon Motion generic IDs (126f:1001, 126f:2262), SanDisk SN550 (15b7:5009), ADATA SX8200PNP 512GB (1cc1:8201), Kingston SKC2000 (2646:2262) and A2000 (2646:2263), ZHITAI TiPro5000 (1e49:0021) and TiPro7000 (1e49:0041), and Solidigm P44 Pro (025e:f1ac). Board-specific entries include: Samsung 144d:a802 loses its deepest state only on Dell XPS 15 9550 and Precision 5510; the Samsung 960 EVO (144d:a804) loses APST only on ASUS PRIME B350M-A and PRIME Z370-A; an SSD on the Lenovo N60z board loses APST because it 'drops off the PCIe bus after system idle for 10 hours'. Entries that turn APST off by model string include the Toshiba THNSF5256GPUK and the Kioxia CD6-V Series / HPE PE8030 (KCD6XVUL6T40), whose quirk comment says it 'times out and aborts I/O during any load', more easily with discards (fstrim), and that 'nvme set-feature' could not then disable APST but booting with nvme_core.default_ps_max_latency_us=0 worked. That Kioxia entry is the only one in the tables read whose comment describes failure under load; the sources read do not state its connector, and the entry matches that one model string, so other CD6-V capacities are not named. The board-specific entries show the fault can belong to a drive-and-platform pair rather than to the drive alone.
4. 4.Read ASPM on the link, for the drive and for the port above it (find the port with lspci -tv). Run it as root: some config space is root-only and the capability lines may not print otherwise.

   sudo lspci -vv -s <bus:dev.fn> | grep -E 'LnkCap|LnkCtl|LnkSta'

   ReadingLnkCap 'ASPM L1' or 'ASPM L0s L1' means supported; LnkCtl 'ASPM L1 Enabled' or 'ASPM L0s L1 Enabled' means in use, 'ASPM Disabled' means off. The kernel option pcie_aspm=off leaves the firmware's ASPM settings as they were, so confirm here that ASPM is actually off before ruling it out. LnkSta marked '(downgraded)' belongs to p4.
5. 5.Test one change at a time. First turn APST off alone: add this option to the kernel command line and reboot, then confirm with the first two steps that it took effect (0, and APSTE Disabled). It changes boot configuration only; remove it to restore power saving.

   nvme_core.default_ps_max_latency_us=0

   ReadingIf the drop-outs stop with APST off alone, the drive or the drive-and-board pair has a power-state fault. A drive already in the kernel's quirk tables is handled by a kernel that carries its entry.
6. 6.If they continue, add the other two options the kernel's own hint names to the kernel command line and reboot, and switch ASPM off in the BIOS as well: pcie_aspm=off only tells the kernel not to touch ASPM, it does not turn off ASPM the firmware enabled. pcie_port_pm=off disables power management of all PCIe ports. Check LnkCtl again after the reboot.

   nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

   ReadingDisputed: reported outcomes for these options conflict, so treat them as a test, not a fix. In one LKML report (SK Hynix drive, kernel 6.17.8) the three options together made the problems go away. In others the drop-outs continued: a linux-nvme reporter already ran with APST and ASPM off, a Manjaro poster found that switching off PCIe power saving in the BIOS changed nothing, and a Proxmox poster was fixed by a power-supply swap after the options had not helped. Keith Busch, a maintainer of the kernel's NVMe driver, wrote on the linux-nvme list that these are 'just the most readily available things we can tune at this level' and have helped on some platform and device combinations. If drop-outs continue with all three set and LnkCtl reading 'ASPM Disabled', go to p4.

1. 1.Keith Busch read an all-ones CSTS in a linux-nvme thread as the driver being unable to read the device's memory: 'You are showing that the driver can't read from the device's memory, and there's nothing the driver can do about that.' With power management ruled out, look below the driver: platform firmware, power delivery, heat, the link or the drive. In that thread he pointed the reporter at the platform BIOS.
2. 2.Check the drive firmware against the maker's current release.

   sudo nvme fw-log /dev/nvme0; cat /sys/class/nvme/nvme0/firmware_rev

   ReadingLists the firmware in each slot and the active revision. Compare it with the drive maker's current release and its release notes. We have not read a release note that ties a firmware version to this drop-out, so this page does not claim that an update fixes it.
3. 3.A firmware update writes to the drive: a wrong or interrupted image can leave the drive unusable, and you can lose the data on it. Back up first, and prefer the maker's own update tool and image. With nvme-cli the update is two commands; commit action 1 replaces the slot's image and activates it at the next reset.
   Destructive. This command destroys what is on the device and cannot be undone. Confirm the device path twice and have a backup of anything you still need before running it.

   sudo nvme fw-download /dev/nvme0 --fw=<image> && sudo nvme fw-commit /dev/nvme0 --slot=<n> --action=1

   ReadingThe new firmware runs after the next reset; read nvme fw-log again to confirm the active revision.
4. 4.Check whether the link trained below the drive's capability.

   sudo lspci -vv -s <bus:dev.fn> | grep LnkSta

   Reading'(downgraded)' after the speed or width means the link trained below what the device supports: a slot, riser, cable or adapter question. In the Proxmox thread (t1), with the drive on a bifurcation card, posters tried extra 12V power to the bifurcator, reseating, and moving the drive to the CPU-attached slot.
5. 5.Work through the platform. In the Proxmox thread (t1) the posts that report a fix credit different things: a BIOS update for a Samsung 980 Pro, a power-supply replacement for a Kingston KC3000 after the kernel options had not helped, lower-power drives for a ZFS NAS that failed under scrubs and rebuilds, and new thermal pads and faster fans for drives running in the low 60s °C under load. These are reports from individual systems, not established causes.
6. 6.If smart-log showed media_errors above 0, or the reliability (0x4) or read-only (0x8) bits of critical_warning, the drive itself is the problem rather than power management: back it up and plan its replacement.

The default I/O timeout is 30 seconds, which is why an I/O stall of about 30 seconds comes before each I/O timeout line; admin commands time out after 60 seconds. No source we read says raising the timeout fixes a drop-out, so this page does not recommend it.

nvme.quirks= on the kernel command line adds or removes quirks for a VendorID:ProductID pair, in the form VendorID:ProductID:quirk_names. The kernel document we read does not give the name for the deepest-power-state quirk, so this page prints none; nvme_core.default_ps_max_latency_us is the sourced way to limit or switch off APST.

The enterprise evidence we have is narrow: the kernel's quirk for the Kioxia CD6-V Series / HPE PE8030 (KCD6XVUL6T40), whose comment says it times out and aborts I/O during any load; the sources we read do not state that drive's connector. Separately, one ZFS NAS post in a Proxmox thread failed under scrubs and rebuilds until it moved to lower-power drives. We did not verify an enterprise-specific forum thread. Kioxia lists 18-24 W typical active power for its 2.5-inch CD8P-R drives against 5.0-5.8 W for Samsung's 990 PRO M.2, but we have not found a document stating what an M.2 slot, U.2 cable or OCuLink or SlimSAS adapter can supply.

No purchase is needed to diagnose this, and the first fixes are configuration: APST, ASPM and port power management, the board BIOS and the drive firmware. Hardware enters only if smart-log shows the drive reaching its thresholds under load (cooling), the link trains below capability (a slot, riser, cable or adapter), the drive logs media errors or critical warnings (replacement), or power delivery is left after elimination, where threads report a power-supply swap or lower-power drives. This page lists no product: no field we hold says which drives avoid a given drop-out, and the pages linked above list drives without knowing that either.