grub is missing secure boot support for compressed kernels (our arm64 kernels)

Bug #1954683 reported by Julian Andres Klode
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
grub2 (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned
Impish
Won't Fix
Undecided
Unassigned
Jammy
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
arm64 systems currently cannot have their grub updated when booted in secure boot mode, despite booting fine, as the postinst runs grub-check-signatures and grub-check-signatures does not handle the compressed kernels used on arm64.

It seems the cloud images only ship unsigned grub at the moment, but there may be other images shipping signed boot stack that customers would expect to work (and then fail during a grub update) and we can also consider this hardware enablement of arm64 secure boot.

[Attack plan]

1. Modify grub-check-signatures to optionally decompress kernels before passing them to sbverify
2. Modify grub to either
   a) verify after decompress
   b) disable shim_lock verifier on arm64, and only use the rhboot

We do not know if this is a long-term solution, we really should migrate back to kernels that are proper EFI executables themselves such that we can use standard EFI functions to run them as well.

[Test plan]
= focal =
1. Grab an arm64 cloud image
2. Boot in qemu, WITHOUT SECURE BOOT (AAVMF_VARS.fd): Install grub-efi-arm64-signed shim-signed if not already installed
2b) On bionic, install linux-virtual-hwe-18.04 and reboot (the LTS kernel in bionic is not signed on arm64)
3. Reboot in secure mode (using AAVMF_VARS.ms.fd)
4. See that the system boots
5. apt install --reinstall grub-efi-arm64-signed, observe failure message about unsigned kernels
6. apt install ?installed?source-package((^grub2$)/<series>-proposed (aka grub2-common grub-common)
7. observe success when reinstalling grub-efi-arm64-signed.
8. Install another signed kernel, observe success (*)

shim/grub may need to be launched manually in qemu as MS variables place the EFI shell first in the boot order, select the boot entry for the disk from the interrupt menu, or do FS0:, followed by \EFI\BOOT\BOOTAA64.EFI in the EFI shell to launch it.

(*) Surprisingly maybe, kernel upgrades are not affected as they do not call grub-check-signatures, so installing an unsigned kernel still succeeds, however, this is not a regression from this patch.

[Where problems could occur]
We only modify the grub-check-signatures script in the SRU to add decompression. This could change the behavior of the script, and introduce new bugs that cause false positives or false negatives.

[Other info: Impact; 2.06-only, not relevant for SRU]
In 2.06, the verifiers framework runs before any decompression, causing the kernels to fail verification, as it tries to verify the compressed data. In grub 2.04, we manually verified the file after we had opened it (hence after all filters).

Related branches

no longer affects: grub2 (Ubuntu)
tags: added: regression-proposed
tags: added: rls-jj-incoming
tags: added: fr-1931
tags: removed: rls-jj-incoming
summary: - grub is missing secure boot support for compressed kernels
+ grub is missing secure boot support for compressed kernels (our arm64
+ kernels)
description: updated
Changed in grub2-unsigned (Ubuntu Jammy):
status: New → Fix Committed
affects: grub2-unsigned (Ubuntu Focal) → grub2 (Ubuntu Focal)
Revision history for this message
Julian Andres Klode (juliank) wrote (last edit ):

Uploaded focal.

Changed in grub2 (Ubuntu Focal):
status: New → In Progress
Changed in grub2 (Ubuntu Impish):
status: New → Won't Fix
Changed in grub2 (Ubuntu Impish):
status: Won't Fix → Triaged
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.06-2ubuntu4

---------------
grub2 (2.06-2ubuntu4) jammy; urgency=medium

  * UBUNTU: Move verifiers after decompressors (LP: #1954683)
  * grub-check-signatures: Support gzip compressed kernels (LP: #1954683)

 -- Julian Andres Klode <email address hidden> Mon, 10 Jan 2022 14:52:04 +0100

Changed in grub2 (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Robie Basak (racb) wrote :

> Compressed kernels as we have on arm64 cause grub to fail in two ways...

OK, but you haven't specified how this impacts the user. Please justify the SRU on the basis of what impact this issue has *on the user*. Otherwise I can't review it for appropriateness under our SRU policy. For example: it sounds like what you are describing had never worked on arm64, so are you enabling the entire architecture with grub2? Or something else? What's the failing user story here?

Revision history for this message
Robie Basak (racb) wrote :

Or is it that you're looking to enable secure boot on this architecture? In that case, please make that clear, and provide a Test Plan that includes testing that complete user story.

Revision history for this message
Julian Andres Klode (juliank) wrote :

@Robie The impact is clearly specified, it fails to upgrade grub. The user impact is failing to upgrade grub. Apart from this, existing arm64 images work fine under secure boot; we have always tested secure boot, but never tested *upgrading* grub.

description: updated
description: updated
Revision history for this message
Robie Basak (racb) wrote :

I've discussed this further with Julian and he will re-reproduce the issue locally to answer some of my questions, and then we'll work on the User Impact and Test Plan from there.

Changed in grub2 (Ubuntu Focal):
status: In Progress → Incomplete
Changed in grub2 (Ubuntu Impish):
status: Triaged → Won't Fix
Revision history for this message
Julian Andres Klode (juliank) wrote :

I have updated the Impact and Test Plan to provide an end-to-end testing of secure boot on arm64.

description: updated
Changed in grub2 (Ubuntu Focal):
status: Incomplete → In Progress
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Julian, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in grub2 (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hey Julian! This looks good now so I'll accept it to focal-proposed. That being said, why is the impish task set to 'Won't Fix'? Is this something that does not happen on impish?

Revision history for this message
Julian Andres Klode (juliank) wrote :

I just don't see anyone launching arm64 secure boot products based on impish in its few remaining months, especially with jammy just a month away, so no need to spend resources on SRUs there and force updates to everyone for this.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Instructions updated as I accidentally wrote to upgrade grub-efi-arm64-signed, but duh, it's grub{2,}-common you want to upgrade from proposed.

description: updated
description: updated
Revision history for this message
Julian Andres Klode (juliank) wrote :

Verification:

4. observed that system booted in secure mode after installing shim-signed, checked mokutil --sb-state to verify
5. observed failure in reinstalling grub-efi-arm64-signed w/ old src:grub2
6. upgraded grub{2,}-common to proposed ubuntu26.15 from ubuntu26.13
7. observed success in reinstalling grub-efi-arm64-signed
8. observed success installing linux-image-5.4.0-106-generic (*)
9. system still boots (*)

I ran out of space but managed to fully install the kernel, just not the modules, so rebooted into the normal kernel, but it installed, so I'm happy.

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Robie Basak (racb) wrote :

Does grub2-signed need rebuilding? I don't think I fully understand the interaction between the two packages, so I'm being extra cautious before releasing for fear of breaking the world.

Revision history for this message
Julian Andres Klode (juliank) wrote :

It does not show up on excuses as broken, so I'd say no. grub2-signed is built from grub2-unsigned in all releases, not grub2.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.04-1ubuntu26.15

---------------
grub2 (2.04-1ubuntu26.15) focal; urgency=medium

  [ Mauricio Faria de Oliveira ]
  * Call hwmatch only on the grub-pc platform (LP: #1840560)

grub2 (2.04-1ubuntu26.14) focal; urgency=medium

  * grub-check-signatures: Support gzip compressed kernels (LP: #1954683)

 -- Julian Andres Klode <email address hidden> Tue, 11 Jan 2022 16:09:48 +0100

Changed in grub2 (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in grub2 (Ubuntu Bionic):
status: New → Confirmed
Revision history for this message
Julian Andres Klode (juliank) wrote :

This also seems to be an issue in bionic, see bug 2004437 (duplicate).

Changed in grub2 (Ubuntu Bionic):
status: Confirmed → Triaged
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Julian, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.26 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in grub2 (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done
dann frazier (dannf)
description: updated
Revision history for this message
dann frazier (dannf) wrote :

bionic verified w/ 2.02-2ubuntu8.26

description: updated
tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.02-2ubuntu8.26

---------------
grub2 (2.02-2ubuntu8.26) bionic; urgency=medium

  * Have grub-common depend on efibootmgr on amd64, arm64, i386 (LP: #1936857)
  * grub-check-signatures: Support gzip compressed kernels (LP: #1954683)

 -- Julian Andres Klode <email address hidden> Wed, 01 Feb 2023 18:49:01 +0100

Changed in grub2 (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.