SAS expander or a 24-port HBA: which does a 24-bay chassis need?

1. 1.Ask the kernel. The mpt3sas driver used by LSI/Broadcom HBAs selects the SAS transport class, and that class registers every expander it finds under /sys/class/sas_expander.

   ls /sys/class/sas_expander/ && for e in /sys/class/sas_expander/expander-*; do echo "$e $(cat $e/vendor_id) $(cat $e/product_id) $(cat $e/product_rev)"; done

   ReadingEach expander-<host>:<n> line is one registered expander: vendor_id and product_id name the chip or backplane, product_rev is the revision string the device reports, which need not match a vendor's release numbering. An empty directory means no expander is registered, so the drives are direct-attached or the controller's driver does not use this class.
2. 2.List every SCSI device with its transport address and sg name. The lsscsi man page states that for SATA disks attached to a SAS expander, the expander provides the SAS address.

   lsscsi -t -g

   ReadingA SATA drive behind an expander carries a sas: address the expander manufactured; a SATA drive cabled straight to the HBA has what the man page calls an indeterminate SAS address. A device of peripheral type 0xd is an SES enclosure: note its /dev/sgN for sg_ses below. The kernel's libsas code names such a drive an 'STP/SATA domain device', which is indirect evidence of tunnelling only; we found no T10 or SNIA document on STP, so this page says no more than that the expander carries the SATA traffic using STP.
3. 3.Map drives to expander phys. smp_discover sends SMP DISCOVER requests to an SMP target, typically an expander, and --multiple loops over its phys. Put your expander's SMP device in place of <SMP_DEVICE>: the smp_discover man page identifies the target by device and SAS address and shows no example path, so we print none. If your SMP_DEVICE identifies the HBA rather than the expander, add --sa=<expander SAS address> (in hex with a 0x prefix), as the man page describes. The kernel's libsas documentation describes an smp_portal file in each expander's sysfs directory that 'gives you complete control of the expander', but that applies to libsas-based drivers, and mpt3sas does not select libsas.

   smp_discover --multiple --brief <SMP_DEVICE>

   ReadingOne line per attached phy: the phy number, the attached SAS address, the protocol in t(...) (SATA, SSP) or exp for another expander, and the negotiated rate in Gbps, as in the man page's example 'phy 11:T:attached:[500605b000000afb:00 t(SATA)] 1.5 Gbps'. --brief drops phys with nothing attached.
4. 4.If lsscsi showed an enclosure device, read it slot by slot.

   sg_ses --join /dev/sgN

   ReadingJoins the element descriptors, enclosure status and Additional Element Status pages, so each slot shows its status and the SAS address attached to it. This form only reads; sg_ses --set and --clear change enclosure state such as LEDs and have no place in diagnosis. An enclosure on its own is not proof of an expander: StorCLI documents enclosure detection on backplanes with no SES or expander, through SGPIO or an I2C SEP.
5. 5.On an LSI HBA with IT or IR firmware, ask the card for its device table. Use sas2ircu on SAS2 cards.

   sas3ircu list && sas3ircu 0 display

   ReadingThe 'Enclosure information' section gives Enclosure#, Logical ID, Numslots and StartSlot; each drive block gives Enclosure #, Slot #, SAS Address, State, Protocol (SAS or SATA) and Drive Type. Broadcom's own sample shows an enclosure with eight slots on a SAS3008, an eight-port chip, so an Enclosure entry does not settle the question: decide on the sysfs and smp_discover results.
6. 6.On a MegaRAID controller, list the enclosures it sees instead. StorCLI's exposeencldevice property decides whether the driver exposes enclosure devices, expanders included, to the operating system.

   storcli /c0/eall show all

   ReadingGeneral information, inquiry data and element counts for each enclosure. Drives behind an enclosure are addressed /c0/eE/sS; directly attached drives are /c0/sS.
7. 7.Read the backplane's part number and open its manual. Supermicro's BPN-SAS3-846EL1 has one expander that reaches all 24 bays from host connectors J49 and J50, and its 12V_LED2 is the expander heartbeat: blinking is normal, steady on or off is not. The direct-attach BPN-SAS3-826A instead has three Mini SAS HD connectors of four ports each, one per bay. On the dual-expander 846EL2, each dual-port SAS drive is reachable through both expanders; the SAS drive not-detected page linked below covers that case.

1. 1.Count the lanes between card and expander. Broadcom's 12Gb/s HBA guides give 1200 MB/s per lane in each direction, and one SFF-8643 connector is one x4 wide port: 4,800 MB/s each way, or 9,600 MB/s counting both directions in the 9300-16i guide's full-duplex row. An 8-port HBA with two x4 connectors can therefore run at most two uplinks.
2. 2.Read the rate every phy negotiated, on the HBA and on the expander.

   grep -H . /sys/class/sas_phy/*/negotiated_linkrate

   Readingphy-<host>:<n> entries belong to the HBA and phy-<host>:<n>:<m> entries to an expander. Values read 1.5, 3.0, 6.0 or 12.0 Gbit. A SATA drive cabled straight to the HBA cannot show more than 6.0 Gbit, because these HBAs run SATA links at 3 or 6Gb/s only. For a SATA drive behind an expander, the cap is the drive's own interface: Western Digital's HC550 data sheet, for example, gives SATA 6Gb/s, 600 MB/s.
3. 3.Add up what the drives can stream. Our arithmetic, from Western Digital's HC550 data sheet and Broadcom's guides: 24 Ultrastar DC HC550 18TB drives at their 269 MB/s maximum sustained rate total 6,456 MB/s, more than one x4 uplink (4,800 MB/s each way) and less than two (9,600 MB/s). WD's figure is a maximum 'based on internal testing', other drives differ, and no source we read says how often every drive in a NAS streams at once. So this is the all-drives-at-full-speed case: with two uplinks it fits inside the SAS links, with one it does not.
4. 4.Compare the total with the slot. Broadcom rates the PCIe 3.0 x8 host link of the 9300-16i, 9305-16i and 9305-24i at up to 64Gb/s in one direction, which is 8,000 MB/s by plain unit conversion; the guides we read give no usable-throughput figure after protocol overhead. With two uplinks cabled, a card with that rating hits its slot before its SAS links, and a card with more ports does not widen that limit. No 9300-8i user guide was read for this page, so we do not attribute the PCIe figure to that card.
5. 5.Redo the sum for SATA SSDs. At the 600 MB/s SATA interface maximum, 8 SATA SSDs fill one x4 uplink and 16 fill two (our arithmetic). With two uplinks cabled, a wider card's own PCIe 3.0 x8 rating in the step above (8,000 MB/s) is below the two uplinks' 9,600 MB/s, so a wider card helps an SSD array only when one uplink is all that can be cabled and the SSDs can bypass the expander, or when your 8-port card's host link is slower than that rating; we read no 8-port card's rating. 600 MB/s is an interface ceiling, not a measured SSD speed. The sum adds data rates: we found no Broadcom, Intel or Microchip document stating how a 12Gb/s expander uplink carries 6Gb/s or 3Gb/s SATA links, and none on mixing 3Gb/s and 6Gb/s SATA drives behind one expander, so this page makes no claim either way about how the expander matches those rates.

1. 1.Read every phy's error counters while the array is under load, a scrub for example, then read them again a few minutes later.

   grep -H . /sys/class/sas_phy/*/invalid_dword_count /sys/class/sas_phy/*/loss_of_dword_sync_count /sys/class/sas_phy/*/running_disparity_error_count /sys/class/sas_phy/*/phy_reset_problem_count

   ReadingThe kernel keeps these four counters for each phy, the expander's included. A count that keeps climbing on one phy while the rest hold still points at that link: its cable, its backplane slot or the drive on it. The kernel sets no threshold, so compare phys with each other rather than judging one number.
2. 2.On a MegaRAID controller, look at the controller's own phys.

   storcli /c0/pall show all

   ReadingPHY layer information for every controller phy: which ones have a link and at what rate.
3. 3.Separate the expander from everything else, using what forum posters tried as a guide rather than as a diagnosis. On ServeTheHome, Event ID 153 timeouts hit only the drives behind an Intel RES2SV240 and stopped once the IBM M1015 HBA itself was replaced. On Level1Techs, UDMA CRC errors across about 20 SATA disks behind an AEC-82885T stopped when they were cabled straight to the 9500-16i with a breakout; the same disks had also run clean on a 9300 with no expander. Another poster swapped a 9300-8i and RES2SV240 that randomly dropped drives for a 9300-16i. A wider card is not a sure cure: in another ServeTheHome thread, a poster with drops behind an Intel RES3TV360 ordered a 9305-24i to run without expanders and still saw drives drop under load, and no root cause was confirmed. These are reports, not documented causes. The common test is to move a few affected drives onto direct HBA lanes and load them again.
4. 4.Check firmware at both ends, and treat any change as a trial. The expander chip runs its own firmware, which StorCLI can update from a MegaRAID controller, and posters in these threads tried both expander and HBA firmware changes; in the 82885T thread the expander could not be updated because maxView did not see it on the 9500. We found no Broadcom, Intel or Microchip compatibility matrix between HBA firmware and expander firmware.
5. 5.On a MegaRAID controller you can force one phy's speed to test a suspect link. This writes the controller's configuration, and we have not found a Broadcom statement of what a forced change does to I/O in flight on that phy, so treat it as a step that can lose data: stop I/O to the drives on that phy and record its current setting before you run it. StorCLI is built for the MegaRAID line; it is not the tool for an IT-mode HBA.
   Destructive. This command destroys what is on the device and cannot be undone. Confirm the device path twice and have a backup of anything you still need before running it.

   storcli /c0/pN set linkspeed=6

   ReadingForces phy N to 6Gb/s (StorCLI accepts 0 for auto, 1.5, 3, 6 or 12). Use it only as a deliberate test of one link, then set the phy back to the value you recorded (0 is auto).

1. 1.A direct-attach backplane needs one HBA lane per bay. Supermicro's 12-bay BPN-SAS3-826A gives each bay its own port on three four-port connectors, so a 24-bay direct-attach chassis needs 24 lanes; that multiplication is ours, from the 12-bay manual, because the 24-bay direct-attach manual we tried was the wrong URL. Broadcom's 9305-24i provides 24 lanes on six SFF-8643 connectors.
2. 2.An expander backplane already does the fan-out. Supermicro lists cables to the BPN-SAS3-846EL from both SAS2 (SFF-8087) and SAS3 (SFF-8643) HBAs, so an 8-port HBA with two x4 connectors can serve all 24 bays. The 846EL manual we read does not state the lane width of J49 and J50; that each is x4 is our reading of its SFF-8643 cable list and Broadcom's 'internal x4' connector wording. Extra lanes on a card only help if there are host connectors to cable them to.
3. 3.If you do go wider, know which card is which. The 9300-16i is two 8-port SAS3008 controllers behind a PCIe switch, 26.9 W nominal; the 9305-16i and 9305-24i are single-controller cards at 16.2 W typical, and posters in three threads weighed exactly that difference. The 9305 guides contradict themselves: sections 1.1 to 1.3 name the SAS 3216 (16i) and SAS 3224 (24i) while section 1.8 names the SAS 3324; the features lists say sixteen and twenty-four x4 connectors while the 24i's installation step says six SFF-8643 connectors; and the 16i guide's text maps ports 0-7 to Core 1 and 8-15 to Core 0 while its Table 2 shows Core 0 and Core 2. We go by card names, not chip names.
4. 4.A standalone expander card (Intel RES2SV240 or RES3TV360, Adaptec AEC-82885T) is the other route people take; a TrueNAS forum regular advising a 24-drive build named a 9305-24i or, by preference, a 9207/9300-8i with an expander. We found no Intel or Microchip document for these expanders, so this page states no port split, power draw or rate for them. Forum posters say the 82885T uses its PCIe slot only for power and can run from Molex, and one owner of a 36-port Intel unit found only 24 of its ports usable for drives. We track no standalone expander listing.

The BPN-SAS3-846EL manual we read lists two host connectors for the expander (J49 and J50) and calls its ports bidirectional, but does not say whether one or both must be cabled. A Level1Techs reply describes expanders as working like network switches: one uplink reaches every downstream port and a second adds bandwidth and redundancy. That is a forum explanation, not a specification; the bandwidth sums on this page show when the second cable matters.

Only within one core. Broadcom's 9305-24i guide manages its 24 PHYs in three groups of eight (ports 0-7, 8-15 and 16-23) and states: 'A single wide port cannot combine individual ports or PHYs sourced by different SAS cores.'

We found no Broadcom, Intel or Microchip document that answers it, so we do not say either way. The nearest thing we read is Linux's libsas code, which logs and lowers a SATA device's link rate behind an expander when it exceeds the 'min pathway connection rate'; LSI/Broadcom HBAs use the mpt3sas driver, which uses the SAS transport class rather than libsas, so that behaviour is not theirs.

We found no Intel document for the RES3TV360, so we cannot state its port split. On Level1Techs, an owner who bought these expanders expecting up to 32 drives reported that Intel's guide reserves three x4 ports for HBAs, and a MegaRAID 9361-4i showed 28 slots of which four were the uplink, leaving 24 for drives. Why was not resolved in the thread.