Fixing DRAC 5 Error: Remote Access Controller initialization failure

Mark Berry March 4, 2010

I recently got a good eBay deal on a DRAC 5 remote access card for my Dell PowerEdge 2900 server.

Well it seemed like a good deal until I installed the card and got this message on bootup:

Remote Access Controller detected
!!*** Error: Remote Access Controller initialization failure***
RAC virtual USB devices may not be available...

According to the seller, the drive was a “working pull,” but the box looked like it had been wrapped around a basketball during shipment, so who knows what it endured underway.

I spent an hour or so on the phone with Dell tech support and after confirming that the card was genuine, having me separately remove its two connector cables, and a few other tests, the rep concluded that the board was faulty. Flashing the ROM might help, but he checked the docs and confirmed that there is no way to flash the ROM if the card will not initialize. I was not looking forward to the lengthy USPS insurance claim process.

Read the Readme

I happened to be working with another DRAC 5 this week and needed to upgrade its firmware to version 1.51 so it would work with Internet Explorer 8 (on Windows 7).

Hey, what’s this in the Readme about “CONFIGURING THE RAC CARD FOR FLASH RECOVER MODE”? Wow it sounds complicated—console port redirection, serial connection, command line junk. But what have I got to lose? It’s worth a shot.

And it worked! After following the instructions, the DRAC initialized correctly and was accessible from Ctrl-E during boot and, once configured, from a web browser.

Update 01/07/2011:  There’s a chance you can find out what’s wrong without going through the entire flash process. Once you have a COM connection, you should see some status information as soon as you turn on the server. See the last section of this article, “After the Recovery.”

Readme with Pictures

Here now the annotated readme in code font, with my comments in normal font:

1. Connect the RAC card to a TFTP server using a network cable to the system network or a cross-over cable to the host system.

I used a network cable connected to the network’s router.

2. Point the TFTP server root path to the folder containing the firmware update image, "firmimg.d5".

To get this file, download the “Hard Drive” install from Dell, f_drac5v151_A00.exe and extract it. I used the TFTP server on my Linux phone server. I blogged here about how to set up the TFTP server.

3. Using the BIOS setup screen, configure the system serial mux such that the RAC card is connected internally to the external DB-9 serial port.

Not as hard as it sounds. Just set the external serial port to point to the Remote Access Device (but see the Note below the picture):

Note:  I could not access the Setup (BIOS) while the DRAC was installed. I pressed F2 to enter Setup, and the Setup screen would flash by for half a second, but then Windows started loading immediately. To make the change shown above, I had to temporarily disconnect the DRAC 5 cable that gets power from the motherboard (can’t recall which one that is, but when the LEDs on the DRAC 5 go out, you’ve got it).

4. Install a NULL modem cable between the system DB-9 serial port and a client machine.

This made me nervous because as far as I know, all my serial cables are “regular,” not null modem. So I just used a regular DB9 serial cable. (Yes, this required digging through a box of stuff that’s ten or more years old!)

5. On the client machine, open a terminal communications program like "HyperTerminal" or "Mini-Comm" using, [1] baudrate=115200, [2] bits=8, [3] parity=none, and [4] flow control=none

Another “gotcha” because my client machine, a Lenovo T61p laptop in its docking station, is running Window 7, which no longer includes HyeprTerminal. Fortunately PuTTY SSH, also on hand to manage the Linux phone server, works great as a serial port terminal program. I did have to go back in to the laptop’s BIOS to enable the COM port, probably disabled by default since it is only available through the docking station. Note to self:  be sure to keep at least one machine around that has a good old COM port!

6. Enter a carriage return and a "RAC Recover Mode" prompt should appear. If not, then you may not be in recover mode. Recheck the serial mux and terminal settings.

Wow, it worked! Actually it says
RAC Recover>

7. View "recover network settings"
RAC Recover> recover getniccfg

This showed me that the default IP is something like 192.168.0.120.

8. If needed, set static network settings (DHCP is NOT supported in recover)
RAC Recover> recover setniccfg <IPaddress> <Subnetmask> <Gateway>

I used this command to set the IP to an unused IP address on my local network.

9. Verify the settings and connection by pinging the TFTP server.
RAC Recover> recover ping <tftp server ip>
(If ping does not work, you may need to enter recover racreset first).

Worked again! Note that you have to type the word “recover” before “ping”.

10. Download and update the flash part.
RAC Recover> recover fwupdate -g -a <tftp server ip>

This took a while, but it provided some interesting messages during the install (unlike the Windows install, which goes silent for so long you wonder if it is stuck):

11. RAC card will now update and reboot on completion.

Which means it looks like this when it finishes:

After the Recovery

After the firmware was installed, I rebooted the machine (no more “initialization failure” message!), pressed Ctrl-E when prompted, and set up an IP address on my network (it had reverted to the default again). After that, I was able to log in to the DRAC via a web browser as usual, using the user root and the default password calvin.

Once the DRAC was functioning again, it was no longer possible to get to the RAC Recover> prompt. Pressing Return in the Putty session had no effect. Why not? Does the card somehow know when initialization is failing, and only allow recovery in that state? It was interesting, though, to see what happens as soon as you plug in the computer, before even turning it on. The DRAC initializes and displays these messages over the COM connection:

Now that I see that, it occurs to me that there may have been some other messages available before the recovery, perhaps indicating what was wrong, if I had only known to connect the COM port before. Next time…

7 Comments

1. Michael   |  January 07, 2011 at 7:40 am

   Hi Mark,

   Thanks for your post – I had the exact same problem and followed your steps exactly and got it working again. I was very lucky a colleague managed to find a null modem cable as I’m halfway across the world from the datacentre.

   I can’t really add anything to your post except that I’ve got ESXi installed with the Dell OpenManage so instead of changing the IP address from BIOS I was able to change it from the OpenManage Server Administrator page (which saved me having to ask my colleage to change the address).

   The IP address automatically assigned to the DRAC was 192.168.0.120 – I wonder if an address from this range is always assigned. It could help someone else who needs to change their IP address (configure a server on the 192.168.0.x range and scan for IP addresses)?

   I also didn’t see the original recovery messages. Only read your last paragraph after starting the update.

   Thanks again, Michael

   
2. Mark Berry   |  January 07, 2011 at 9:14 am

   Glad this helped someone. I’ve added a note in the article to check for status messages before starting the flash process.

   
3. Adolf von Fidjestövel   |  February 02, 2011 at 3:34 am

   I had the same error message, but got it fixed by updating the BMC. It also fixed the fans on 100% problem that I had.

   
4. Pablo Castro   |  August 09, 2011 at 5:32 am

   Thanks Mark for this post, it worked perfect for me in a PoweEdge 1950.

   There is a newer fw version in DELL’s web site.

   Pablo

   
5. Florin   |  September 19, 2011 at 10:53 am

   Man, you just saved my day.

   those morons from dELL, they keept me on the phone for ~40 minutes and give me NO SOLUTION !!!
   just… to replece the DRAC 5

   
6. Chendra DW   |  December 22, 2011 at 11:34 am

   First of all, thanks for posting this solution. Unfortunately, I don’t have a unix system. I tried using Solarwind Free TFTP server on windows but every time I tried the fwupdate it gives me the following

   ERROR:Cannot load update file, Please Check:
   1) Is TFTP server running
   2) Did you enter the correct -a
   3) Is the TFTP server path pointing to the update image file.

   Do you have a tftp server that can be accessed from public?

   Thank you.

   
7. Chendra DW   |  December 22, 2011 at 11:39 am

   Oops, nevermind. I installed the tftp server on a server. I guess that server blocked some ports. I installed on a windows 7 and it’s transferring now. Thanks Mark. You’re awesome.