Flashing the Supermicro variant of SAS9201-16e (SAS2116)

Published: 2021-12-05, Revised: 2022-05-11

schema_ipsec

TL;DR Flashing SAS Cards feels like a trip to the 90s, with all the peculiarities included. This particular LSI SAS9201-16e card left me puzzled, until I realized it is the Supermicro variant that needs special handling. Let's see if I can pull together the important pieces, so that those who come after me don't need to give up on their cards.

With ZFS, an increasing number of raid cards are converted from "IR" to "IT" mode, to serve as simple passthru of SAS/ SATA drives, without the raid functionality. The acronym for such cards is HBA.

There are endless variants of cards available, with different generations of SAS and different branding. This means that flashing newer IT firmware can be done in minutes, or take days. With this particular card, I encountered one of the more complicated variants.

Warning

Use at your own risk.

Therefore, before proceeding, check out these general introductions below. It is quite likely that your card is not the same as the one described here. Read carefully. These cards are very robust and quite hard to "brick" (as we will see later), but YMMV 1.

Flashing SAS Cards#

I found following guides to provide a quite good overview of the process.

I will not repeat the excellent knowledge that is provided through these contributions.

Prerequisites#

Preparing files:

First, get together all the necessary files. As a start, see this introductory post on serverbuilds.net 2 with the bare minimum of the UEFI Flashing Utility (sas2flash) and the DOS variants (sas2flsh, megarec).

Click for a list of recommendations
  • There are different versions available. It is good to have more available, but start with the newest version of sas2flash (P20).
  • There are different firmwares from different brands. You will want to start with the default variants from LSI/Broadcom. The flash utilities will usually warn you if a firmware or bios is not compatible with the card.
  • Getting your USB stick to boot into UEFI shell can be cumbersome.
    • Use rufus to format your stick as FAT32, with FreeDOS.
    • Do not overwrite any of these files with those found in the links above (LOCALE etc.)
    • Copy/rename the UEFI shell (shellx64.efi/bootx64.efi) to 6 different locations on the stick.2 Your motherboard will (hopefully) pick up one of these files.
    • Disable any "Secure Boot" settings in your Bios settings, otherwise you will not be able to boot into UEFI shell or the FreeDos shell.
    • Also, I had to use a USB 2.0 Port because USB 3.0 would not boot the stick.
  • All guides usually refer to sas2flash, sas2flsh, and megarec. The actual filenames may vary. For instance, sas2flash is the UEFI version and usually ends with .efi, whereas sas2flsh is the DOS variant and ends with .exe. The same applies to megarec. You can have multiple versions in one folder, e.g. LSI-P20-sas2flash.efi and LSI-P10-sas2flash.efi.
  • If you see a PAL error when loading the DOS variants, this commonly means that your UEFI motherboard is interfering and you need to use UEFI shell.3
Some notes on using the UEFI Shell
  • The UEFI Shell follows Linux syntax, e.g. lf to list files (etc.)
  • When you need to "scroll up" (to read previous output), press SHIFT+PAGE UP
  • The USB-drive seems to be mapped first, enter sf0: when asked

Sas2flash: Controller is not operational#

This was the message that I would be greeted with my card on on a regular boot. At this point, I did not know anything about this card, except that it was bought through an official IT distributor, so likely not a counterfeit product from China.

Unable to load LSI Corporation MPT BIOSMPT BIOS Fault 05h encountered at adapter PCI(07h,00h,00h)

After booting into UEFI shell, my controller was found but not operational.

sas2flash -listall

Adapter Selected is a LSI SAS: SAS20116_1(B1)

Controller is not operational. A firmware download is required.
Either firmware file name or quit to exit:

Using the default firmware from Broadcom here (9201-16e-P20.bin) did not work.

Chip is in Fault state. Attempting Host Boot...
ERROR: Firmware Host Boot Failed!

Firmware fault occurred. Fault code: 0A

Now, understanding this message is quite hard. A firmware download is required means that the card needs to "download" a firmware to actually be able to accept or process any commands (because the current firmware is not operational).

In other words, in order to process a command, (e.g.) to flash a specific firmware (sas2flash -o -f 9201-16e-P20.bin), a firmware needs to be provided to execute this command.

Finding the correct firmware#

Here you are at your own. I could find (1) at (2) least (3) three reports with a similar pattern, one of these discussions pointed me to the Supermicro's version of the 9201-16e firmware.6

Only the P19 version is currently available from Supermicro, but a friendly user provided the P20 version, from a Supermicro Support response, on servethehome.com.

This got me one step further.

There is a SMC2116T.bat, included in the Supermicro zip, which lists the following steps:

@echo off 
sas2flsh -o -e 7
cls
sas2flsh -f 2116T207.ROM
sas2flsh -b mptsas2.rom
sas2flsh -b x64sas2.rom
cls
sas2flsh -o -sasaddhi 5003048

Since I cannot use sas2flsh (DOS), due to PAL errors, I substituted sas2flash (P20) (UEFI).

sas2flash -o -e 7
# Controller is not operational. ... Enter firmware file name or quit to exit: 2116T207.ROM
First, note that -e 7 is the "hammer" method. Check the SAS2Flash_ReferenceGuide.pdf 7, which explains parameter 7 as Erase complete flash.

There are reports that this can brick your card. This was not my feeling - the card would disappear for a brief few seconds and then revert back to fault state, if anything went wrong.

Second, note that the Firmware file from Supermicro does not end with .bin, but .ROM.

Output 
Controller is not operational. A firmware Hostboot is required.
Enter firmware file name or quit to exit: > 2116T207.ROM

Firmware Version 20.00.00.00
Firmware Image compatible with Controller.

Valid NVDATA Image found.
NVDATA Version 14.00.00.00
Checking for a compatible NVData image...

NVDATA Device ID and Chip Revision match verified.
Valid Initialization Image verified.
Valid BootLoader Image verified.

Performing Host Boot...
Firmware Host Boot Successful !

Executing Operation: Erase Flash

Erasing Entire Flash Region (including MPB) ...

Erase Flash Operation Successful!

Resetting Adapter...
Reset Successful!

Finished Processing Commands Successfully.
Exiting SAS2Flash.

The only things I cared about here were:

Firmware Host Boot Successful ! ... Erase Flash Operation Successful!

Apparently, the card was not dead.

Next step:

sas2flash -listall

Returns: No LSI SAS adapters found!

No need for a cardiac arrest. Running the command a second time is back to:

Adapter Selected is a LSI SAS: SAS2116_1(B1)

Controller is not operational. A firmware download is required.
Enter firmware file name or quit to exit:

Let's see if I can flash the Supermicro .ROM from above.

sas2flash -o -f 2116T207.ROM
# Controller is not operational. ... Enter firmware file name or quit to exit: 2116T207.ROM
# ...Chip is in Fault state.

Somehow, I was stuck in a loop.

I later realized that after issuing sas2flash -o -e 7, you were not supposed to do any other command (e.g. do not try to list cards with sas2flash -listall), since this would revert the card from RESET state back to FAULT state).

Secondly, unlike in the Supermicro .bat, the bios and firmware needed to be flashed in a single command.

sas2flash -o -e 7
sas2flash -o -c 0 -f 2116T207.ROM -b mptsas2.rom
Output: Firmware Download Successful. ...Flash BIOS Image Successful. 
Controller is not operational. A firmware Hostboot is required.
Enter firmware file name or quit to exit: > 2116T207.ROM

Firmware Version 20.00.00.00
Firmware Image compatible with Controller.

Valid NVDATA Image found.
NVDATA Version 14.00.00.00
Checking for a compatible NVData image...

Chip is in FAULT state. Attempting Host Boot...
Firmware Host Boot Successful !

Mfg Page 2 Mismatch Detected.
Writing Current Mfg Page 2 Settings to NVRAM.
updating Mfg Page 2.

Resetting Adapter...
Adapter Successfully Reset.

Chip is in RESET state. Performing Host Boot...
Firmware Host Boot Successful.

Beginning Firmware Download...
Firmware Download Successful.

Resetting Adapter...
Adapter Successfully Reset.

Executing Operation: Flash BIOS Image

Validating BIOS Image...
BIOS Header Signature is Valid
BIOS Image has a Valid Checksum.
BIOS PCI Structure Signature Valid.
BIOS Image Compatible with the SAS Controller.
Attempting to Flash BIOS Image...
Verifying Download...
Flash BIOS Image Successful.
Updated BIOS Version in BIOS Page 3.

Finished Processing Commands Successfully.

Yes!

Now the card would also show up with sas2flash -listall

Num    Ctlr          FW Ver        NVDATA          x86-BIOS           PCI Addr
-------------------------------------------------------------------------------------------
0 SAS2116_1(B1)    20.00.07.00      14.01.40.00          07.37.01.00         00:07:00:00

Finished Processing Commands Successfully.

Afterwards, I followed the remaining two steps from the Supermicro .bat.

sas2flash -b x64sas2.rom
sas2flash -o -sasaddhi 5003048

There was a sticker with the remaining 9 digits of the SAS Address on the card, which needs to be provided in the last step above.

Check the detailed card info with sas2flash -c 0 -list. 
Adapter Selected is a LSI SAS: SAS2116_1(B1)

Controller Number: 0
Controller: SAS2116_1(B1)
PCI Address: 00:07:00:00
SAS Address: 5003048-x-xxxx-xxxx
NVDATA Version (Default) :14.01.40.00
NVDATA Version (Persistent): 14.01.40.00
Firmware Product ID: 0x2213 (IT)
Firmware Version: 20.00.07.00
NVDATA Vendor: LSI
NVDATA Product ID: LSI2116-IT
BIOS Version: 07.37.01.00
UEFI BSD Version: 07.27.01.00
FCODE Version: N/A
Board Name: LSI2116-IT
Board Assembly: N/A
Board Tracer Number: N/A

The Supermicro SAS9201-16e version (SAS2116_1(B1) chip) appears to be a native HBA, with no consideration of raid from the outset. This is perhaps also why megarec did not find this card, but sas2flash did. Megarec appears to be specifically tailored to raid branded cards.

For now, I was happy to be greeted with this old DOS revival on regular boot.

dns-cf

To test connected drives: 

sas2flash -o -testlsall

Output 
Advanced Mode Set

Adapter Selected is a LSI SAS: SAS2116_1(B1)

Executing Operation: Test Link State ALL

Phy 0: 6.0 Gbps
Phy 1: 6.0 Gbps
Phy 2: 6.0 Gbps
Phy 3: 6.0 Gbps
Phy 4: Link Down
Phy 5: Link Down
Phy 6: Link Down
Phy 7: Link Down
Phy 8: Link Down
Phy 9: Link Down
Phy 10: Link Down
Phy 11: Link Down
Phy 12: Link Down
Phy 13: Link Down
Phy 14: Link Down
Phy 15: Link Down
Phy 16: Link Down

Test Link State All PASSED!

Finished Processing Commands Successfully.
Exiting SAS2Flash.

Erase Boot Services#

One remaining step is to erase the "Boot Services" (the boot menu).

This is entirely optional, but since the boot menu is not needed for ZFS/HBA and only postpones server startup (and perhaps interferes with ZFS), I removed it. The HBA firmware allows passthru of drives flawlessly without it.

This is the command:

sas2flash -o -c 0 -e 5
Output 
Advanced Mode Set

Adapter Selected is a LSI SAS: SAS2116_1(B1)

Executing Operation: Erase Flash

Erasing Boot Services Region...

Erase Flash Operation Successful!

Finished Processing Commands Successfully.
Exiting SAS2Flash.

At any time it is possible to flash the Boot Services BIOS back with the above files.

Note

If you want to boot from a HDD that is connected through your HBA, you will likely need the Boot Services. Otherwise, necessary drivers for communication may not be loaded as early as is required for the Master Boot Record (MBR) check.

This is not common - the usual setup is to connect boot drives directly to the server SAS/SATA ports, avoiding the HBA entirely.

Conclusion(s)#

A first draft of this post appeared in a thread on Servethehome.8

  1. "Your Mileage May Vary" urbandictionary.com ↩

  2. Updating your LSI SAS Controller with a UEFI Motherboard serverbuilds.net ↩↩↩

  3. Unraid Docs: Crossflashing Controllers. wiki.unraid.net ↩↩

  4. Detailed newcomers' guide to crossflashing LSI HBA and variants truenas.com/community ↩

  5. LSI 9201-16e HBA Card and how the hell you get it all working together rkalla.me ↩

  6. SAS/LSI Supermicro Firmware download supermicro.com/wdl ↩

  7. SAS2Flash Utility Quick Reference Guide docs.broadcom.com ↩

  8. First draft of this post at forums.servethehome.com/ ↩