Today, I had an “accident” configuring my new firewalls IPMI. Okay, I’ll admit. It was a PEBKAC situation: I wanted to configure the firewalls IPMI to use a dedicated network interface, but the option was disabled in the web UI. So I used Chromes debug mode to force-enable the drop-down menu, selected ‘Dedicated’, and saved. After a few seconds, there was an error dialog box on the web interface stating it failed to save the settings. And bam, the remote KVM console was disconnected, and the web interface went dead. A bit of investigation revealed that the IPMI had dropped off the network.
I went over to the firewall, checked that the network ports were connected and had a link. Everything seemed fine there. Then I went to the console and entered BIOS to look at the IPMI settings. The IPMI section said the IPMI was functioning normally. Okay, strange. IP settings & MAC address were still correcet; stranger still. Since I had a static IP address for it, I decided to switch to DHCP, save and cycle power. Nope, still no go. And the DHCP server didn’t get any queries from it, either.
I pulled the power cord, waited 30 seconds, and plugged it back in. Still no change. Reset BIOS to failsafe defaults; still didn’t work. I dismounted the firewall from the rack, opened the case, took a few pictures of the insides which I’ll post in another entry, and took out the BIOS battery for 30 seconds. I assembled and re-mounted the firewall, and… IPMI still didn’t work. Apparently, resetting BIOS doesn’t affect the IPMI configuration at all. And with hindsight being 20/20, this makes a lot of sense.
At this point, I conclude that forcing it to use ‘dedicated’ mode for its network interfaces made it ignore them entirely instead. It was time to find a way of resetting the IPMI. There were no options for it in the BIOS, so I figured I should get FreeBSD up and running so I could use “ipmitool”. As I grabbed the memstick, I realized it had FreeBSD 10.0 AMD64, but I needed i386 for the firewall. And I had no such thing. Time to go make a new memstick. A few minutes later, I booted from the memstick, and installed FreeBSD.
Once FreeBSD was installed and booted, I installed ipmitools using pkgng, and poked around a bit. There were no documented option to reset all IPMI configuration. Sigh. I tried manipulating the network settings again, just in case, but to no avail. Went back to the desktop, googled a little, and found this FAQ entry at SuperMicros website. It stated that executing “ipmitool raw 0x3c 0x40” would reset the IPMI to its factory settings. I executed that command, and… IPMI was back up and running. Two hours after I caused the problem in the first place.
I’ve learned that when form elements on a management UI are disabled, it’s probably done for a good reason. I do however think it’s scary, and a little bit amazing, that it accepted such erroneous input in the first place.
In conclusion: If you force your IPMI to do things it doesn’t wanna do, bad things are gonna happen. Luckily, it made a full recovery… this time.
Savagedlight's Road 