Is my Ceph cluster slow because of consumer SSDs, and how do I prove it?

1. 1.Print per-OSD latency. This is read-only and safe on a live cluster. Use it comparatively: OSDs on the same drive model and host class should read alike.

   ceph osd perf

   ReadingColumns commit_latency(ms) and apply_latency(ms). We found no Ceph document giving a threshold. A set of OSDs sitting tens of milliseconds above their peers points at those devices; one Proxmox forum cluster on Crucial MX500s showed some OSDs at 81-124 ms while others sat at 3-4 ms.
2. 2.Locate each slow OSD so you know which host and physical drive to look at.

   ceph osd find osd.<id>

   ReadingGives the OSD's location (its host). ceph osd tree shows the same OSDs grouped under their hosts.
3. 3.Show utilization per OSD under its host, and check whether the slow OSDs sit on one node. This view does not name the drive model: identify the model of the slow OSDs' drives on that host. Proxmox's benchmark paper says a cluster that mixes disk types drops to the performance of the slowest disk.

   ceph osd df tree

   ReadingOSD utilization per OSD, grouped by host and device class, not by drive model. Read it beside ceph osd perf: if the high-latency OSDs sit on one host, find which drives they are on that host and test that model in path p3.

1. 1.Check the health warnings. SLOW_OPS alone does not identify the SSD: Ceph lists extreme load, a slow storage device or a software bug as possible causes.

   ceph health detail

   ReadingSLOW_OPS means OSD or monitor requests are taking a long time. BLUESTORE_SLOW_OP_ALERT means BlueStore logged 'slow operation observed' latency lines, which Ceph says may indicate a failing drive that should be investigated. Start with the OSDs named.
2. 2.On the host of a named OSD, dump its recent slow operations to see where the time went. dump_ops_in_flight and dump_historic_ops_by_duration give the same view for current and longest ops.

   ceph daemon osd.<id> dump_historic_slow_ops

   ReadingEach op carries an event timeline. Long gaps at 'waiting for subops from [X,Y]' mean the op waited on replica OSDs X and Y: check those OSDs' devices and the network to them, because the time was not necessarily spent on this OSD's own disk.
3. 3.Check the kernel log on the OSD's host. Ceph lists a failing drive and kernel file-system bugs among the causes of slow requests and says to check dmesg for them.

   dmesg

   ReadingI/O errors, resets or timeouts against the OSD's device point to a failing drive rather than a merely slow one. A clean log rules out the two causes Ceph says to check in dmesg. It leaves a slow storage device (test it in path p3), an overloaded cluster (check system load and iostat), a ceph-osd bug, or, on HDD clusters with mClock, a suboptimal OSD shard configuration.

1. 1.Do not clear a drive on the OSD bench alone. Ceph warns that the OSD bench can show unrealistic or inflated results and recommends establishing IOPS capacity with fio. Its arguments write 12,288,000 bytes (about 12 MB) in 4 KiB writes, spread over up to 100 objects of 4 MiB, through the live OSD, so expect it to add some load.

   ceph tell osd.0 bench 12288000 4096 4194304 100

   ReadingArguments are TOTAL_BYTES, BYTES_PER_WRITE, OBJ_SIZE, NUM_OBJS, so this writes 4 KiB blocks. A good number here does not outweigh a poor fio sync-write result: on the MX500 cluster above, osd bench reported about 13,058 IOPS at 4KB while commit latency sat at 81-124 ms.
2. 2.Run Proxmox's published sync-write test, which gives figures you can compare directly. WARNING: Proxmox states 'This command will destroy any data on your disk.' Run it only on a drive that is not an OSD: before you create the OSD, or after the OSD is destroyed and the cluster has finished recovering. Replace /dev/sdx with that drive. The PDF prints the filename flag with a typographic dash; type two hyphens as shown here.
   Destructive. This command destroys what is on the device and cannot be undone. Confirm the device path twice and have a backup of anything you still need before running it.

   fio --ioengine=libaio --filename=/dev/sdx --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --name=fio --output-format=terse,json,normal --output=fio.log --bandwidth-log

   ReadingSingle job, queue depth 1, synchronous writes. We found no Ceph document giving a pass/fail threshold, so compare drives against each other and against Proxmox's table (January/February 2018, this command): Samsung SM863 240GB 17,485 IO/s at 0.06 ms; Intel DC S3510 120GB 12,518 IO/s; Samsung 850 EVO 1TB 339 IO/s at 2.94 ms; Crucial MX100 512GB 254 IO/s at 3.93 ms. Those are measurements of those exact models, not of any other drive.

1. 1.Read the SATA or SAS drive's volatile write cache state. These are read-only queries. Ceph's paragraph on disabling the cache is about enterprise SSDs and HDDs with power loss protection, and it tells you to benchmark with fio and keep whichever setting measures best.

   hdparm -W /dev/sda; sdparm --get WCE /dev/sda; smartctl -g wcache /dev/sda; cat /sys/class/scsi_disk/0\:0\:0\:0/cache_type

   Reading'write back' in cache_type means the volatile cache is on; 'write through' means it is off. Changing it alters a drive setting, not data, and some drives need it reapplied at every boot. On consumer drives the forum record runs the other way: it helped MX500s only temporarily, and on Samsung 990 Pros it pushed latency above 50 ms.
2. 2.For an NVMe drive, read feature 0x06 (Volatile Write Cache). Read-only.

   nvme get-feature /dev/nvme0 -f 0x06

   ReadingThe returned value shows whether the drive's volatile write cache is enabled. Setting it with nvme set-feature changes drive behaviour; measure with fio before and after if you try it.

Do not count on it. Ceph's documentation says disabling the volatile write cache may increase IOPS and cut commit latency on enterprise SSDs and HDDs with power loss protection, and tells you to benchmark with fio and keep the best setting. On consumer drives, Proxmox forum users found it made Samsung 990 Pro latency worse (above 50 ms) and helped Crucial MX500s only for a while.

Read the maker's own document for that exact model; form factor alone does not prove it. Samsung's 2016 white paper describes capacitors in its enterprise SSDs that keep the controller, DRAM and NAND powered long enough to flush the cache, and Solidigm's D7-P5510 brief names its scheme power-loss imminent (PLI). Those statements cover the models they name, not other drives.

Samsung's white paper says a UPS is not enough to prevent unexpected power loss, because a power supply glitch can cut power to a drive on its own, and client SSDs lose the pending write data held in DRAM when that happens.

Ceph's mClock reference warns that the OSD bench may show unrealistic or inflated results depending on the drive configuration, and recommends establishing an OSD's IOPS capacity with fio instead. The value it sets at OSD start (osd_mclock_max_capacity_iops_ssd) comes from the same bench.