I recently got a good eBay deal on a DRAC 5 remote access card for my Dell PowerEdge 2900 server.
Well it seemed like a good deal until I installed the card and got this message on bootup:
Remote Access Controller detected
!!*** Error: Remote Access Controller initialization failure***
RAC virtual USB devices may not be available...
According to the seller, the card was a “working pull,” but the box looked like it had been wrapped around a basketball during shipment, so who knows what it endured underway.
I spent an hour or so on the phone with Dell tech support and after confirming that the card was genuine, having me separately remove its two connector cables, and a few other tests, the rep concluded that the board was faulty. Flashing the ROM might help, but he checked the docs and confirmed that there is no way to flash the ROM if the card will not initialize. I was not looking forward to the lengthy USPS insurance claim process.
Read the Readme
I happened to be working with another DRAC 5 this week and needed to upgrade its firmware to version 1.51 so it would work with Internet Explorer 8 (on Windows 7).
Hey, what’s this in the Readme about “CONFIGURING THE RAC CARD FOR FLASH RECOVER MODE”? Wow it sounds complicated—console port redirection, serial connection, command line junk. But what have I got to lose? It’s worth a shot.
And it worked! After following the instructions, the DRAC initialized correctly and was accessible from Ctrl-E during boot and, once configured, from a web browser.
Update 01/07/2011: There’s a chance you can find out what’s wrong without going through the entire flash process. Once you have a COM connection, you should see some status information as soon as you turn on the server. See the last section of this article, “After the Recovery.”
Readme with Pictures
Here now the annotated readme in
code font, with my comments in normal font:
1. Connect the RAC card to a TFTP server using a network cable to the system network or a cross-over cable to the host system.
I used a network cable connected to the network’s router.
2. Point the TFTP server root path to the folder containing the firmware update image, "firmimg.d5".
3. Using the BIOS setup screen, configure the system serial mux such that the RAC card is connected internally to the external DB-9 serial port.
Not as hard as it sounds. Just set the external serial port to point to the Remote Access Device (but see the Note below the picture):
Note: I could not access the Setup (BIOS) while the DRAC was installed. I pressed F2 to enter Setup, and the Setup screen would flash by for half a second, but then Windows started loading immediately. To make the change shown above, I had to temporarily disconnect the DRAC 5 cable that gets power from the motherboard (can’t recall which one that is, but when the LEDs on the DRAC 5 go out, you’ve got it).
4. Install a NULL modem cable between the system DB-9 serial port and a client machine.
This made me nervous because as far as I know, all my serial cables are “regular,” not null modem. So I just used a regular DB9 serial cable. (Yes, this required digging through a box of stuff that’s ten or more years old!)
5. On the client machine, open a terminal communications program like "HyperTerminal" or "Mini-Comm" using,  baudrate=115200,  bits=8,  parity=none, and  flow control=none
Another “gotcha” because my client machine, a Lenovo T60p laptop in its docking station, is running Window 7, which no longer includes HyperTerminal. Fortunately PuTTY SSH, also on hand to manage the Linux phone server, works great as a serial port terminal program. I did have to go back in to the laptop’s BIOS to enable the COM port, probably disabled by default since it is only available through the docking station. Note to self: be sure to keep at least one machine around that has a good old COM port!
6. Enter a carriage return and a "RAC Recover Mode" prompt should appear. If not, then you may not be in recover mode. Recheck the serial mux and terminal settings.
Wow, it worked! Actually it says
7. View "recover network settings"
RAC Recover> recover getniccfg
This showed me that the default IP is something like 192.168.0.120.
8. If needed, set static network settings (DHCP is NOT supported in recover)
RAC Recover> recover setniccfg
<IPaddress> <Subnetmask> <Gateway>
I used this command to set the IP to an unused IP address on my local network.
9. Verify the settings and connection by pinging the TFTP server.
RAC Recover> recover ping
<tftp server ip>
(If ping does not work, you may need to enter recover racreset first).
Worked again! Note that you have to type the word “recover” before “ping”.
10. Download and update the flash part.
RAC Recover> recover fwupdate -g -a
<tftp server ip>
This took a while, but it provided some interesting messages during the install (unlike the Windows install, which goes silent for so long you wonder if it is stuck):
11. RAC card will now update and reboot on completion.
Which means it looks like this when it finishes:
After the Recovery
After the firmware was installed, I rebooted the machine (no more “initialization failure” message!), pressed Ctrl-E when prompted, and set up an IP address on my network (it had reverted to the default again). After that, I was able to log in to the DRAC via a web browser as usual, using the user root and the default password calvin.
Once the DRAC was functioning again, it was no longer possible to get to the
RAC Recover> prompt. Pressing Return in the Putty session had no effect. Why not? Does the card somehow know when initialization is failing, and only allow recovery in that state? It was interesting, though, to see what happens as soon as you plug in the computer, before even turning it on. The DRAC initializes and displays these messages over the COM connection:
Now that I see that, it occurs to me that there may have been some other messages available before the recovery, perhaps indicating what was wrong, if I had only known to connect the COM port before. Next time…