Death by Secure Boot

Yes, I bricked my machine with a flag literally called "yes-this-might-brick-my-machine" and I'm still surprised I wasn't safe.

The Setup: Hypervisors, VMs, And Graphics Cards

I'm working on an idea I've had for some time now where I set up my powerful desktop machine as a hypervisor with a small amount of dedicated RAM, then I create these "appliance" operating systems for specific purposes. I can then swap the appliances when I swap major tasks such as development work, LLM/AI Explorations, Linux gaming, Windows gaming, and so on.

The idea is that I want to be able to very quickly switch the bulk of the power of that machine, including its reasonably powerful graphics card, to super-clean and automated environments. I want this control remotely (I've chosen over SSH), and I want the machine to be usable by others in the family without denying me access to my current projects and code.

More specifically, I planned to provision atelier, an LXC container for coding and CPU work, and KVM/QEMU VM called legion for GPU work. Most of the games my family plays run well in Linux, and I want to keep supporting Linux gaming. Unfortunately, Minecraft Bedrock Edition, which my daughter and the rest of my family plays, is not available in Linux without android emulation.

During my research, it seemed that binding and unbinding of a graphics card from an OS is hard when you do it multiple times across multiple VMs and the host. To avoid this, I acquired a second used low-power card (RX 560), then configured the hypervisor to ignore the high-powered card (my RTX 2070) so it could be passed through. This avoids the hypervisor kernel needing to bind and unbind the card.

I scripted the provisioning of both the hypervisor and legion for Linux gaming using my demo auto-typer so that I could potentially make another video showcasing this end-to-end.

Two nights ago, I got this working. I successfully ran No Man's Sky in the Linux Gaming VM with the RTX 2070 passed through. I spent 5 minutes looking around before I went to bed. It was great.

Iterations

To create the demos, I essentially make the changes to the machine, and duplicate it in the scripts. After I get it working, I rerun the scripts which reprovisions everything from scratch.

The Setback

With the manual install working, and a set of untested provisioning scripts present, I did a cleanup pass. During my cleanup pass, I noticed that sb enroll-keys was used, however it failed due to Option ROMs. I had seen this in my VMs and there is a flag called --yes-this-might-brick-my-machine which I used to get past it then.

Great, I looked up the flag to double check and I understood the Option ROMs were used to validation during secure boot. I concluded that in the worst case scenario, my OS would not boot, and I could just go to the BIOS and shut off Secure Boot, then move on.

I was wrong. Very wrong. After my provisioning finished and I attempted to restart the machine, no video loads from any of the cards. No Video, no ability to manipulate the BIOS. I used the CMOS jumper pins and even tried removing the CMOS battery for an hour but without any success. Seems Secure Boot's platform key might be in some form of non-volatile RAM. I then tried removing all of the PCI cards and the RAM to see if something else was causing this. No luck.

That's unfortunate, now it's time to theorize about what went wrong. The two major changes during this run were the flag (sbctl enroll-keys --yes-this-might-brick-my-machine), and the fact that I went into the MSI BIOS, went to Secure Boot, and cleared the variables to force it into UEFI "Setup Mode" so new Platform Keys could be enrolled. My working theory is that sbctl wrote to the flash, but something silently failed during the write and corrupted the platform key.

It seems that Secure Boot can, and will (at least in my case) stop PCI cards from loading during POST if their Option ROMs don't validate, which they definitely won't if platform keys are corrupted. As a result, no video.

With no ability to see, and no ability to clear BIOS settings or restore the default platform key, it sounds like I might need to flash my BIOS. I understand this resets platform keys too.

My motherboard is an MS-7C75v2.1 and does NOT have MSI's flashback, so reflashing the BIOS via a button press is out. Unfortunately this motherboard also doesn't seem to have the AMI Restore working, the one where holding CTRL+HOME tells the AMI BIOS to flash itself from a single file on a fat32 partitioned USB 2.0 flash drive. I tried a host of filenames on two flash drives and had no success.

And so, there lies my expensive and prized desktop.

Murdered by security.

Killed by cryptography.

Death by Secure Boot.

Letter to Motherboard and UEFI designers

Dear Motherboard Designers and UEFI Spec Writers,

In a world where most people don't set a BIOS password, and Secure Boot settings are modifiable unauthenticated, a sane default recovery method to help consumers would be to have a secure boot recovery jumper-pin paired with a firmware variable in the bios which says "allow secure boot recovery". A simple AND operation between allows normal consumers to save themselves in cases such as these, and high-security environments to lock it down at the same time.

When sane default recovery methods like this exist, adoption is a lot faster. Us geeks can more safely explore, which turns into more open source adoption overall.

Still appreciate you more than you know while I holler into the void, Christopher Prevoe

Recovery Plan

My research indicates that my only real hope now is to use an SPI programmer to flash the BIOS chip directly on the motherboard. I purchased a programmer online, and it comes tomorrow.

For now, we hope.