Headache provisioning baremetal servers? Basic tips to clean, reset, and reboot BMCs

New to out-of-band management (OOB)? BMC, IPMI, IDRAC, ILO…

No matter how confusing the different terms may sound, what I can guarantee you is that continously re-deploying Kubernetes clusters on the same baremetal servers will eventually mess up their BMC. So here’s my simple cheatsheet to go back in time!

Reset iLO for fully cleaning up server #

When servers boot from past ignition files, reset iLO.

Cleaning up Boot order options #

First attempt to simply move the Storage Slots to the top of the list, as shown below:

And if that does not work, other option is to clean up all boot options so they get generated again.

To do this, first set One Time boot to System Utilities or press F9 when powering on

Then, navigate in the following order until UEFI Boot Settings

Select Delete Boot Options

CLI commands to boot from iso #

If enabled, we an calso SSH into the server and manage its BMC from its terminal.

ssh Administrator@192.168.72.138
User:Administrator logged-in to ILOMXQ15205KM.wei.com(192.168.72.138 / FE80::5EBA:2CFF:FE39:66AC)
Integrated Lights-Out 5
iLO Advanced 2.63 at  Jan 20 2022
Server Name: 
Server Power: On

</>hpiLO->

We can set a specific image to be booted directly.

cd /map1/oemHPE_vm1/cddr1
set oemHPE_image=https://assisted-image-service-multicluster-engine.apps.acm.cert2.bos2.lab/images/5150e8e3-d3ea-4e3c-a7ea-ccbf[…]86_64&type=minimal-iso&version=4.10
set oemHPE_boot=connect
set oemHPE_boot=once
reset /system1

The host gets automatically rebooted and launched from iso file.

IPMITOOL for remote management #

ipmitool allows the user to control IPMI enabled systems through command line from a remote device. An example of how I power on, reset, or just read params in the server from my local laptop:

 ipmitool -H $i -U admin -P  P@ssw0rd -p 623 chassis power off
 ipmitool -H $i -U admin -P  P@ssw0rd -p 623 mc reset cold
 ipmitool -H $i -U admin -P  P@ssw0rd -p 623 chassis bootparam get 5
 ipmitool -H $i -U admin -P  P@ssw0rd -p 623 chassis bootdev pxe
 ipmitool -H $i -U admin -P  P@ssw0rd -p 623 chassis power status

Wrap up #

If you are used to work with baremetal servers there is a very remote chance this post helped you, I know it just covers how to manage daily tasks through BMC.

However, in my case (and by contrast), this is the post I most take a look at for that very reason. I tried to simply list the most common day to day magic tricks with baremetal servers to clean them and make them work again. The new “have you tried switching it off” obvious tip but for those who continuously forget how to browse the iLO menus!