Which SSDs should I buy for Ceph OSDs and BlueStore DB/WAL, and how big should they be?

1. 1.List the drives Ceph already uses so the order either matches them or replaces them on purpose. Read-only.

   ceph device ls

   ReadingLists the storage devices in use. 'ceph device ls-by-host <host>' narrows the list to one host, and 'ceph device get-health-metrics <devid>' returns the health metrics Ceph has stored for a device.
2. 2.See capacity and fill for every OSD, grouped by host. Proxmox says Ceph performs best when every node carries an evenly sized, evenly distributed set of disks, so the order should keep host totals close to one another.

   ceph osd df tree

   ReadingSizes and %USE per OSD under each host. We found no Ceph document defining an imbalance threshold: compare hosts against each other and plan the new drives so no host ends up much larger than its peers.
3. 3.Check whether an OSD is on rotational media and which min_alloc_size it was built with. That value is fixed at OSD creation, so existing OSDs keep theirs whatever drives you add.

   ceph osd metadata osd.<id> | egrep rotational\|alloc

   ReadingThis is Ceph's own example for reading those two attributes. Since Octopus for SSD and Pacific for HDD the default is 4KB; an OSD built with another value has to be rebuilt to change it. Drop the grep to print all of that OSD's metadata.

1. 1.Look for spillover first: it is the signal that the DB volume in front of an HDD OSD is too small for what that OSD holds.

   ceph health detail

   ReadingBLUEFS_SPILLOVER names OSDs whose separate DB device filled, so metadata spilled back onto the slow device. Ceph lists two remedies. One is to expand the DB logical volume, then stop the OSD daemon and inform BlueFS of the size change with 'ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-$ID'. The other is to reprovision the OSD, which Ceph notes involves data migration and recovery.
2. 2.Map each HDD OSD to the SSD that holds its DB. Read-only.

   ceph-volume lvm list

   ReadingFor a logical volume the 'devices' key names the physical drive behind it, and BlueStore volumes carry db_device and wal_device tags. Count how many HDD OSDs point at each DB drive: Ceph's Squid minimum-hardware table pairs 4 to 5 HDD OSDs with a SATA SSD and up to 10 with an NVMe SSD. The list page's printed example predates BlueStore, so field layout in your output may differ.
3. 3.Work out the per-OSD block.db share from the HDD size and the workload. The general figure differs between Ceph documentation versions, so name the version you follow.
   ReadingCeph's Reef BlueStore reference puts block.db between 1% and 4% of the data device; the latest reference says at least 2.5% when WAL and DB are offloaded. Both versions say at least 4% for RGW (object) workloads, which lean on block.db for omap metadata, and both say RBD usually needs no more than 1% to 2%. For one 16TB HDD OSD that is 160GB to 320GB at most at the RBD ceiling, 400GB at 2.5% and 640GB at 4%; multiply by the HDD OSDs the DB drive will serve.
4. 4.Preview the layout ceph-volume would build for a planned NVMe DB drive. With --report it prints the plan and exits, writing nothing.

   ceph-volume lvm batch --report /dev/sdb /dev/sdc /dev/sdd --db-devices /dev/nvme0n1

   ReadingIt shows the data devices and the block.db it would carve from the NVMe drive; with no explicit size it makes block.db as large as possible. With --block-db-size it tries to meet the size and deploys no OSDs if it cannot. Compare its per-OSD figure with the step above. Without --report, ceph-volume asks you to confirm (unless --yes is passed) and then creates OSDs on the listed devices.
5. 5.After the drives arrive, create each HDD OSD with its DB size set on purpose. This is a deployment step, not a check: it consumes /dev/sdX as an OSD. Proxmox says a previously used disk must first be wiped with 'ceph-volume lvm zap /dev/sdX --destroy', and warns that this will destroy all data on the disk.
   Destructive. This command destroys what is on the device and cannot be undone. Confirm the device path twice and have a backup of anything you still need before running it.

   pveceph osd create /dev/sdX -db_dev /dev/nvme0n1 -db_size <GiB>

   Reading-wal_dev and -wal_size work the same way for a separate WAL. If -db_size is left out and no Ceph configuration value applies, Proxmox falls back to 10% of the OSD size for DB and 1% for WAL, well above the 1% to 2% Ceph gives for RBD.

1. 1.Read the memory target a running OSD uses, including the default. Plain 'ceph config show' lists only non-default settings, so on an OSD left at the default it prints nothing for osd_memory_target; the show-with-defaults form also shows all defaults.

   ceph config show-with-defaults osd.<id> | grep osd_memory_target

   ReadingThe default is 4 GiB; when TCMalloc is available and cache autotuning is enabled, the OSD tries to keep that many bytes mapped in memory. Ceph's Reef page says below 2GB is not recommended, 2GB to 4GB may degrade performance, and more than 4GB can help, above all with fast NVMe OSDs.
2. 2.Multiply by the OSD count you plan for each node and compare the result with installed RAM.
   ReadingCeph's Squid page advises about 8GB per BlueStore OSD and Proxmox recommends at least 8 GiB per OSD, so a node with ten OSDs wants about 80GB for them. One ceph-users reply suggested roughly 10GB per NVMe OSD; that is one operator's practice, not a Ceph figure.

1. 1.Show the CRUSH hierarchy and the per-class hierarchies.

   ceph osd tree; ceph osd crush tree --show-shadow

   Readingceph osd tree gives a simple view of the CRUSH hierarchy, including weights. Each OSD sets its device class to hdd, ssd or nvme at startup from the device behind it, and --show-shadow prints the per-class hierarchies that a class-restricted rule draws from. Confirm which class the new drives will take, and which pools would start using them.
2. 2.To keep a pool on one class, create a replicated rule limited to that class and point the pool at it.

   ceph osd crush rule create-replicated <rule-name> <root> <failure-domain> <class>

   Reading'ceph osd pool set <pool-name> crush_rule <rule-name>' assigns the rule; with host as the failure domain each replica lands on a different host. For a custom tier Ceph provides 'ceph osd crush set-device-class <class> <osd-name>' and 'ceph osd crush rm-device-class <osd-name>'. A Level1Techs reply gave the same advice for a cluster mixing SAS and SATA SSDs: one class and rule per tier, separate pools.

1. 1.Start from the redundancy scheme. A default replicated pool keeps three copies, so raw capacity is three times usable. With host as the rule's failure domain, CRUSH stores each replica on a unique host.
   ReadingExample: three nodes with four 7.68TB OSDs each give 92.16TB raw, about 30.72TB usable at size 3, before whatever free-space margin you keep.
2. 2.For an erasure-coded pool, the space overhead is (k+m)/k, and most deployments need at least k+m failure domains, usually hosts or racks.
   ReadingA 4+2 profile stores 1.5 times the data and, with host as the failure domain, most deployments of it need at least six hosts. Ceph adds that EC usually needs more failure domains than replication and suits only some components and use cases.
3. 3.Spread the order: one drive per OSD daemon, a dedicated (ideally mirrored) OS drive, and the same disk count and sizes in every node. Proxmox's minimum for hyper-converged Ceph is three servers, preferably identical.
   ReadingWe found no Ceph or Proxmox document giving a recommended number of OSDs per host, so the per-node count falls out of this capacity sum, the RAM check in p3 and the hosts your failure domain needs. Ceph also says that hosting several OSDs on one SAS or SATA HDD is not a good idea.

1. 1.Set the endurance floor. Ceph's Squid hardware page says many OSD deployments need no more than 1 DWPD ('read-optimized'), suggests a model rated 1+ DWPD or the TBW equivalent, and calls 3 DWPD mixed-use drives often overkill at a significantly higher cost.
   ReadingCeph's Reef page adds that a 0.3 DWPD drive may be fine for OSDs holding sequentially written, read-mostly data but is not a good choice for Ceph Monitor duty, so keep Monitors off the lowest-rated drives.
2. 2.Convert a TBW-only rating before holding it against the 1 DWPD line: divide TBW by the capacity in TB, by 365 and by the warranty years the rating covers.
   ReadingExample: a 7.68TB drive rated 14,016 TBW over five years works out at 14,016 / (7.68 x 365 x 5) = 1.0 DWPD. For the rating you actually need, start from your measured daily writes; in a size-3 pool every client byte is stored on three OSDs.
3. 3.Buying more capacity than the data needs also buys endurance.
   ReadingCeph's Squid page says a larger drive than technically required gives more endurance for a given write workload, because it effectively has greater overprovisioning.
4. 4.Confirm power loss protection model by model in the maker's own document. A form factor or an enterprise label does not prove it.
   ReadingCeph's current hardware page states that production clusters use enterprise-class drives with power loss protection. Makers name it differently: Samsung's 2016 white paper describes it as capacitor-backed, and Solidigm's D7-P5510 brief calls its scheme power-loss imminent (PLI). How PLP works and how to check a drive you already own are on the Ceph Slow on Consumer SSDs page linked below.

Ceph's Reef hardware page says QLC SSDs are closing the cost gap with HDDs for object storage, and that a 0.3 DWPD drive may be fine for OSDs holding sequentially written, read-mostly data but not for Monitor duty. We found no Ceph document giving QLC endurance thresholds, so judge a QLC model by its stated rating against your measured writes.

Ceph recommends one drive for each OSD daemon and Proxmox one OSD per physical disk. ceph-volume does offer --osds-per-device (its batch page shows '--osds-per-device 2'), but we found no Ceph document recommending a number of OSDs per NVMe drive.

Ceph's Squid page says that with NVMe SSDs no HBA is needed. For SAS or SATA drives, Proxmox says to avoid RAID controllers and use a host bus adapter.

No. 4% is Ceph's minimum for RGW (object) workloads in both the Reef and the latest BlueStore references; for RBD both say block.db usually needs no more than 1% to 2%. The general figure is where they part: Reef says 1% to 4%, the latest reference at least 2.5%.