Can not boot impish in Cavium ThunderX

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1942633

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Im not able to run: apport-collect

I can reproduce the issue on phanpy with 5.13.0-14-generic:

...
[ 169.882563] random: fast init done
[ 169.926830] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... Failed to send exit request: Connection refused
done.
[ 171.269768] systemd[1]: Inserted module 'autofs4'
[ 171.421523] systemd[1]: systemd 248.3-1ubuntu3 running in system mode. (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=hybrid)
[ 171.463836] systemd[1]: Detected architecture arm64.

Welcome to Ubuntu Impish Indri (development branch)!

[ 171.523576] systemd[1]: Hostname set to <phanpy>.
[ 173.864657] random: crng init done
...

I've noticed some stack traces during boot, i'll try to extract them.

With 5.13.0-16.16, the issue is way more evident.

5.14.0-9.9 exhibits kernel stack corruption on boot.

I can attempt a bisect - let me know if someone else is already doing that :)

Feel free to bisect, i'm checking our SAUCE patches/configs.

BTW:

Firmware Version: 2017-10-12 12:34:31

it's the last FW version available, isn't it? I'm wondering about the device tree.

For the CRB1S systems, the following is the firmware string that matters, and it shows that phanpy is running the latest (and very likely final) version:

BIOS Date: 06/14/2018 14:42:48 Ver: 0ACGA022

Note that this is also reproducible on our Gigabyte R120 systems which use acpi=force by default, which suggests this is not a device tree problem.

I can reproduce with upstream v5.13, and I've a bisect between that and v5.11 in progress.

First bad commit:

commit 9ec37efb87832b578d7972fc80b04d94f5d2bbe3 (HEAD, refs/bisect/bad)
Author: Marc Zyngier <email address hidden>
Date: Tue Mar 30 16:11:42 2021 +0100

    PCI/MSI: Make pci_host_common_probe() declare its reliance on MSI domains

    The generic PCI host driver relies on MSI domains for MSIs to
    be provided to its end-points. Make this dependency explicit.

    This cures the warnings occuring on arm/arm64 VMs when booted
    with PCI virtio devices and no MSI controller (no GICv3 ITS,
    for example).

    It is likely that other drivers will need to express the same
    dependency.

    Link: https://<email address hidden>
    Signed-off-by: Marc Zyngier <email address hidden>
    Signed-off-by: Lorenzo Pieralisi <email address hidden>
    Acked-by: Bjorn Helgaas <email address hidden>

I think it's a red herring: even after reversing this commit on top of 5.13-rc1 (the first tag showing the issue) or Impish/master-next HEAD, the kernel still Oops and hang there.

hm.. you're right. I was tracking down a different issue which had the following backtrace:

[ 10.701967] Call trace:
[ 10.704415] ata_host_activate+0x160/0x170
[ 10.708518] ahci_host_activate+0x170/0x1e0
[ 10.712711] ahci_init_one+0x898/0xd74 [ahci]
[ 10.717116] local_pci_probe+0x4c/0xc0
[ 10.720883] work_for_cpu_fn+0x28/0x40
[ 10.724635] process_one_work+0x20c/0x4d0
[ 10.728647] worker_thread+0x250/0x564
[ 10.732393] kthread+0x134/0x140
[ 10.735616] ret_from_fork+0x10/0x18

Reverting the bisect-id'd commit from upstream 5.13-rc1 does fix this issue for me, but perhaps it was already fixed in v5.13 final.

I *can* reproduce the backtrace here w/ the archive 5.13.0-16, but I can not with a locally built 5.13-rc1 (w/ the above patch reverted). The fact that you can reproduce with 5.13-rc1 makes me wonder if the issue maybe toolchain related. I'm building in a hirsute environment. I'll try building 5.13.0-16 in a hirsute environment to test that theory.

Update: The issue does not follow toolchain, which is good. Rather I did find that upstream v5.13-rc1 is stable - with the patch in comment #15 reverted - while upstream v5.13 is not. I bisected v5.13-rc1..v5.13, reverting the comment #15 patch at each test. A "bad" kernel wouldn't always fail the same way, but would always fail before completing boot. The bisect hit this commit:

commit 0c6c2d3615efb7c292573f2e6c886929a2b2da6c (HEAD, refs/bisect/bad)
Author: Mark Brown <email address hidden>
Date: Wed Apr 28 13:12:31 2021 +0100

    arm64: Generate cpucaps.h

While this looks innocuous, it is messing with the code that chooses which "features" a CPU has, which includes erratum that may need kernel workarounds. So I went back and compared the CPU features messages between a "good" kernel and a "bad" one. Noticeably missing from a "bad" one was this message:

[ 0.000000] CPU features: kernel page table isolation forced OFF by ARM64_WORKAROUND_CAVIUM_27456

I went back and tested Ubuntu's 5.13.0-16 w/ kpti=off, and it booted fine. As does upstream v5.15-rc2, where previously I was also seeing stack overflows/corruption. So, seems like the above change is likely the problem, next step is to figure out why.

Root caused and reported upstream:
 https://www.spinics.net/lists/arm-kernel/msg921821.html

Patch submitted:
https://www.spinics.net/lists/arm-kernel/msg922152.html

A patch has been merged into the arm64 tree and is tagged for stable 5.13+

Thanks a lot dann.

I dont know the release process of the kernel, and when our public images are generated by CPC.

I have to test it when is uploaded here: https://images.maas.io/ephemeral-v3/candidate/impish/arm64/

Do you have any tip about how to be aware when this should happen?

The patch was merged into 5.13.0-17.17, which is currently in impish-proposed. At some point it should promulgate to the release pocket. You can view the status here:
 https://launchpad.net/ubuntu/+source/linux

I don't know how long after that it will before it appears in images. As a workaround, you can test with kpti=off on the kernel command line.

This bug is awaiting verification that the linux-hwe-5.13/5.13.0-17.17~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Verified:

ubuntu@doerfel:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.13.0-18-generic root=UUID=36113164-ebf4-4fed-9d98-4b0b859bf98e ro acpi=force
ubuntu@doerfel:~$ cat /proc/version
Linux version 5.13.0-18-generic (buildd@bos02-arm64-027) (gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #18-Ubuntu SMP Mon Oct 4 14:52:32 UTC 2021

Also good in focal:
ubuntu@doerfel:~$ cat /proc/version
Linux version 5.13.0-17-generic (buildd@bos02-arm64-034) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #17~20.04.1-Ubuntu SMP Tue Sep 28 14:05:10 UTC 2021

This bug was fixed in the package linux - 5.13.0-19.19

---------------
linux (5.13.0-19.19) impish; urgency=medium

  * impish/linux: 5.13.0-19.19 -proposed tracker (LP: #1946337)

  * impish:linux-aws 5.13 panic during systemd autotest (LP: #1946001)
    - [Config] disable KFENCE

 -- Andrea Righi <email address hidden> Thu, 07 Oct 2021 11:09:51 +0200