Kernel panic - not syncing: IO-APIC + timer doesn't work!

From a successful boot:

[17179570.588000] Total of 2 processors activated (6485.23 BogoMIPS).
[17179570.588000] ENABLING IO-APIC IRQs
[17179570.588000] ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
[17179570.720000] checking TSC synchronization across 2 CPUs: passed.
[17179570.724000] Brought up 2 CPUs

Note that 'bug 8254' is not detected on this bootup.

Ok. The problem goes something like this:

This function is called to decide if the timer IRQ is "working".

static int __init timer_irq_works(void)
{
        unsigned long t1 = jiffies;

        local_irq_enable();
        /* Let ten ticks pass... */
        mdelay((10 * 1000) / HZ);

        /*
         * Expect a few ticks at least, to be sure some possible
         * glue logic does not lock up after one or two first
         * ticks in a non-ExtINT mode. Also the local APIC
         * might have cached one ExtINT interrupt. Finally, at
         * least one tick may be lost due to delays.
         */
        if (jiffies - t1 > 4 && jiffies - t1 < 16)
                return 1;

        return 0;
}

Note the use of mdelay(). Since we're testing the advancement of jiffies we can't use a jiffy-based timer. mdelay delays with a busy loop that runs a certain number of iterations based on the BogoMIPS rating of your machine.

The problem is that BogoMIPs are detected somewhat randomly on my laptop.

The "calibrating delay..." message always gets a value of about 3994.7 give or take a couple tenths. The message that comes later gets wildly different values on different boots.

[17179570.588000] Total of 2 processors activated (6485.23 BogoMIPS).

Depending on the BogoMIPS detected, the delay loop runs in much more or much less time and therefore too few or too many IRQs are detected while attempting to determine if the timer IRQ is connected, causing the test to fail.

Ok. The BogoMIPS output I am quoting is the sum of the separate readings of the various CPUs.

CPU1 is always fine, at 3994.7ish CPU0 varies wildly and this is why:

[17179569.184000] hpet0: at MMIO 0xfed00000 (virtual 0xf8800000), IRQs 2, 8, 0
[17179569.184000] hpet0: 3 64-bit timers, 14318180 Hz
[17179569.184000] Using HPET for base-timer
[17179569.184000] Using HPET for gettimeofday
[17179569.184000] Detected 1997.379 MHz processor.
[17179569.184000] Using hpet for high-res timesource
[17179569.272000] calibrate_delay_direct() failed to get a good estimate for loo
ps_per_jiffy.
[17179569.272000] Probably due to long platform interrupts. Consider using "lpj=
" boot option.
[17179569.272000] Calibrating delay loop... 2097.15 BogoMIPS (lpj=4194304)

Since 3994.70 seems to be the average BogoMIPS count when detected properly I just forced the kernel to that by giving lpj=7989400 on the commandline.

Fixes the problem -- but a real fix would be nice.

Googling, I found this http://www.ubuntuforums.org/showthread.php?t=198453&page=5 . So you're not alone ...

The problem is caused by SMIs being thrown while the kernel is trying to calibrate the delay loop.

This is ICH7. If I save the value of the SMI register then disable SMI_EN before running the calibration then it succeeds.

You have to restore the value of the SMI register after you're done calibrating. There are a few bits in that register like "Enables legacy USB circuit to cause SMI#." that cause USB problems later on if not reenabled (really really slow device detection and keyboard is laggy and each letter you type appears twice).

From page 436 of the ICH7 spec (January 2006)

bit 3 of SMI_EN:

LEGACY_USB_EN — R/W.
0 = Disable.
1 = Enables legacy USB circuit to cause SMI#.

temporarily disabling this single bit allows the delay loop calibration to work.

Do you have a patch to make this work better?

i have a patch but I'm currently hashing it out with matt over what the best way to handle this is. will keep you informed.

Confirming it's still a problem with Ubuntu 6.10 (Edgy) and its kernel 2.6.17-10-generic!

Appending lpj=8000000 at boot fixes this 100% of the time for my 2.0GHz MacBook. The number for the 1.83 should be lower. Thanks to Ryan Lortie for the information.

I'm using a 2.6.18.1 vanilla kernel with all mactel patches .
This problem seems to be fixed, never seen it gain.

I try create ubuntu kernel from vanilla kernel + mactel patch + rt (mingo) patch. And there is problem when I want to have 1000 hz timer frequency - and only with this - on every startup I get this message: Kernel panic - not syncing: IO-APIC + timer doesn't work!

Whisky

Any movement on this?

It'd be nice to submit a fix for this to the mainline kernel within the three week window after 2.6.20 is released (RSN).

thanks for the poke. i'll hack something.

ok. here's the fix in a cleaned-up form that detects that you're on a macbook before enabling itself (just incase there are ill effects of doing this on non-macbooks).

please test that this fixes the problem for you. you'll need to rebuild your kernel.

i just "backported" the patch to edgy's kernel (by doing nothing at all) and it seems to work nicely there too.

further: i've tested the kernel on my PC and the quirk isn't enabled (giving some indication that my DMI detection stuff is legit).

perhaps this patch is also a good candidate for the edgy-updates kernel (if you have one planned)

Is seem, I have similar problem on my PC with Herd5 kernel:

Total of 1 processors activated (4424.05 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=0 apic2=-1 pin2=-1
..MP-BIOS bug: 8254 timer not connected to IO-APIC
...trying to set up timer (IRQ0) through the 8259A ... failed.
...trying to set up timer as Virtual Wire IRQ... failed.
...trying to set up timer as ExtINT IRQ... failed :(.
Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot w
ith apic=debug and send report. Then try booting with 'noapic' option

It is single processor box with AMD Athlon 64 3500+

i'm still seeing this on my laptop with feisty beta.

is there any reason that this problem isn't being fixed?

This problem is still around in feisty with 2.6.20-15-generic.

Ryan,

Have you tested your patch with feisty's kernel? Should this patch be submitted upstream?

Thanks.

Issue still around as of Gutsy 2.6.22-8

Patch applied. Should be in next kernel release. Please test and confirm then.

will it be in Tribe 3's kernel?

On 7/17/07, Amit Kucheria <email address hidden> wrote:
>
> Patch applied. Should be in next kernel release. Please test and confirm
> then.
>
> ** Changed in: linux-source-2.6.22 (Ubuntu)
> Assignee: (unassigned) => Amit Kucheria
>
> --
> Kernel panic - not syncing: IO-APIC + timer doesn't work!
> https://bugs.launchpad.net/bugs/54621
> You received this bug notification because you are a direct subscriber
> of the bug.
>

This problem is so old that it certainly does not hurt to move it to Tribe-4. Pushing a new kernel into Tribe-3 takes way too much time and is too risky.

Patch seems to fix the problem with my custom kernel...

On 7/17/07, Martin Pitt <email address hidden> wrote:
>
> This problem is so old that it certainly does not hurt to move it to
> Tribe-4. Pushing a new kernel into Tribe-3 takes way too much time and
> is too risky.
>
> ** Changed in: linux-source-2.6.22 (Ubuntu)
> Target: tribe-3 => tribe-4
>
> --
> Kernel panic - not syncing: IO-APIC + timer doesn't work!
> https://bugs.launchpad.net/bugs/54621
> You received this bug notification because you are a direct subscriber
> of the bug.
>

linux-source-2.6.22 (2.6.22-9.19) gutsy; urgency=low

  [Amit Kucheria]

  * Fix for FTBFS bug 123178
  * Fix for FTBFS bug 123178
  * Add devices to USB quirks to prevent USB autosuspend
  * More devices added to USB quirks
    - LP: #85488
  * Support for ENE CB-712/4 SD card reader
  * Reorder quirk list based on Vendor/Product ID

  [Ben Collins]

  * ubuntu: Enable HOTPLUG_CPU in sparc64-smp config.
  * ubuntu: Add xen to amd64 custom builds
  * ubuntu: Update real-time kernel to -rt4
  * rt: Patch from Alessio Igor Bogani <email address hidden> for RT-8

  [Chuck Short]

  * IDE: add MHV2080BH to NCQ blacklist
  * XEN: update to 2.6.22 final and amd64 support.
  * NET: Add more pci-ids to zd1211rw
  * IDE: add new PCI ID
  * USB: fix oops in ftdi_sio

  [Eric Piel]

  * ACPI: Allow custom DSDT tables to be loaded from initramfs

  [Ryan Lortie]

  * Macbook calibration loop fix
    - LP: #54621

  [Upstream Kernel Changes]

  * NETFILTER: {ip, nf}_conntrack_sctp: fix remotely triggerable NULL ptr
    dereference (CVE-2007-2876)
  * Linux 2.6.22.1
  * [SPARC64]: Use KERN_ERR in sun4v IRQ printk()'s.
  * [SPARC64]: Add LDOM virtual channel driver and VIO device layer.
  * [SPARC64]: Add Sun LDOM virtual network driver.
  * [SPARC64]: Add Sun LDOM virtual disk driver.
  * [SPARC64]: Create proper obppath sysfs files for VIO bus devices.
  * [SPARC64] LDC: Do limited polled retry on setting RX queue head.
  * [SUNVNET]: Validate RX descriptor size field.
  * [SPARC64]: Add missing symbol exports for LDOM infrastructure.
  * [SPARC64]: Temporary workaround for LDC INO double-delivery.
  * [SPARC64]: Create 'devspec' nodes for vio devices.
  * [SPARC64]: vdev->type can be NULL, handle this in devspec_show().
  * [SPARC64]: Assorted LDC bug cures.
  * [SPARC64]: Add domain-services nodes to VIO device tree.
  * [SPARC64]: Export powerd facilities for external entities.
  * [SPARC64]: Initial domain-services driver.
  * [SPARC64]: Use more mearningful names for IRQ registry.
  * [SPARC64]: Abstract out mdesc accesses for better MD update handling.
  * [SPARC64]: Fix MD property lifetime bugs.
  * [SPARC64]: Fix setting of variables in LDOM guest.
  * [SPARC64]: Initial LDOM cpu hotplug support.
  * [SPARC64]: Unconditionally register vio_bus_type.
  * [SPARC64]: Fix build regressions added by dr-cpu changes.
  * [SPARC64]: mdesc.c needs linux/mm.h
  * [SPARC64]: SMP build fixes.
  * [SPARC64]: More sensible udelay implementation.
  * [SPARC64]: Process dr-cpu events in a kthread instead of workqueue.
  * [SPARC64]: Add ->set_affinity IRQ handlers.
  * [SPARC64]: Fix leak when DR added cpu does not bootup.
  * [SPARC64]: Clear cpu_{core,sibling}_map[] in
    smp_fill_in_sib_core_maps()
  * [SPARC64]: Give more accurate errors in dr_cpu_configure().
  * [SERIAL]: Fix console write locking in sparc drivers.
  * [TIMER]: Fix clockevent notifications on 64-bit.
  * [SPARC64]: dr-cpu unconfigure support.
  * [SPARC64]: Fix UP build.
  * [SPARC64]: SMP build fix.
  * [SPARC64]: Fix race between MD update and dr-cpu add.
  * [SERIAL] SUNHV: Fix jerky console on LDOM guests.
  * [SPARC64]: Kill explicit %gl register reference.
  * [SPARC64]: Add basic inf...

Reverting this patch since it causes bug #129226 (https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/129226)

@Ryan: Could you spend some time of figuring out the reason for the oops during calibration?

This bug is still in kubuntu 7.10 tribe 4 on Asus M2n4-sli, athlon 5000+x2,2 gb ram, and Nvidia GF8600
With the noapic option I can boot but then I loos mouse and keyboard.

this bug have been in Debian Lenny but is now gonde.Maybe look at those solution???

WIll it be fixed before 7.10 stable?

Hi,

I also have macbook. And kernel in gutsy works well for me. But only generic. Realtime (-rt) kernel crash on kernel panic every time! What's wrong with this?

Whisky

I still got:
 Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option

in tribe 5, and same resolt with noapic as before. The only version of kubuntu there does not have the bug is 6.06 LTS

beer: are you on a macbook or another machine? it's possible that you don't have the same bug, but a related one.

I am sitting on a x86 mashine (x2 5000+ , m2n4-sli) But if it is a different bug why do I get the same error msg?

Waiting for correct fix from developer

this has regressed in -12

I can confirm that in the ubuntu kernel 2.6.22-12 realtime version is still same problem!

@Ryan - I reverted the patch a while ago since it causes oops on dual core systems (bug #129226). Please read comments above.

@Whisky - -rt kernels are unsupported. Please contact the developer: Alessio Igor Bogani. That said, if Ryan sends a fixed patch that is applied to Gutsy, then -rt will get it automatically.

I can confirm the bug using an asus m2n4 card and gutsy

I can also confirm this bug using a Gigabyte GA-M55S-S3 (rev 1.1) - nforce 550 chipset

Ooops - forgot to add that this was with Gutsy RC...

Fyi: I received what appears to be the similar if not exactly the same symptoms when I attempted to boot a ubuntu-7.10-alternate-i386.iso based cdrom on a Powerspec 7121 ( http://www.powerspec.com/systems/system_specs.phtml?selection=7121 ) box. The boot-from-cdrom process freezes/hangs at the "send a report" error message.

Is there anything I can do to get 7.10 to work on my Powerspec 7121 system?

Please pardon me if my post is out of context.

Further details:

My error message on the text-based display:

"
[ 45.781270] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[ 46.024508] Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option
"

I should also note:

I've been dual-booting WinXP and Unbuntu Desktop 6.06.1 on this same Powerspec 7121 machine for months now. I don't see any hardware problems when I reboot back to these OSes after attempting the Gutsy/7.10 alternate install.

Note that I try the altnernate installs now because the 7.04 Desktop-Live install on this machine doesn't work because it appears to hang on the video output (I have a nvidia dual-monitor card which might cause the problem).

I got this bug while trying to upgrade my motherboard to an ASUS M5N, with a bottom-of-the-line AMD X2. Updating my BIOS fixed the issue.

This, or something with identical symptoms, was discussed on the Linux kernel mailing list about a year ago: <http://www.ussg.iu.edu/hypermail/linux/kernel/0612.2/0101.html> There it was concluded that it was caused by commit b0268726, and patches were proposed. I'm not sure if anything ever came of it.

Same error with M2N8-VMX

> Same error with M2N8-VMX (Asus)
now I workaround by disabling APIC in BIOS

I am assigning this bug to the 'ubuntu-kernel-team' per their bug policy. For future reference you can learn more about their bug policy at https://wiki.ubuntu.com/KernelTeamBugPolicies .

Since 2.6.22.14.21 linux kernel, this bug affect Dell C521 computers. Since this bug cause a kernel panic on core ubuntu components and with essential hardware ( according to https://wiki.ubuntu.com/Bugs/Importance ), I suggest to raise the importance to high. Should-I change the importance?

Saïvann, I would say yes.

Bug #91279 may be related to this bug.

Marked Bug #91279 as a duplicate, set Importance to High and Status to Triaged since we have a lot of relevant informations and debug files from this bug and bug #91279

This seems to be fixed in the latest kernel builds (i.e., Hardy, specifically 2.6.24). I'm using an Asus M5N motherboard with an AMD X2 5000+. Steps to fix for me:

1) Disable ACPI in BIOS so I can actually boot.

2) Edit /etc/apt/sources.list and add at the bottom:
deb http://archive.ubuntu.com/ubuntu/ hardy main restricted

3) sudo apt-get update

4) sudo apt-get install linux-source-2.6.24-4-generic

5) Reboot, reenabling ACPI in BIOS and then selecting the new kernel to boot to

6) Comment out the sources.list line so it doesn't try to update *everything* to Hardy. (I don't know what will happen when it sees that I have a package installed that's not in any repo, though.)

Er, that should be linux-image-2.6.24-4-generic, of course. I got the instructions from bug #154969 comment #2, by Leann Ogasawara.

I just want to add a comment here. Hardy Alpha2 and subsequent releases contain an updated version of the kernel. You can download and try the new Hardy Heron Alpha release from http://cdimage.ubuntu.com/releases/hardy/ . Note that Alpha 4 wil be out in a few days so you may want to wait to grab the newer release. You should be able to then test the new kernel via the LiveCD. This avoids having to manually install the new kernel. Manually installation was only necessary during the Hardy Alpha1 release as it went out with an older 2.6.22 version of the kernel. If you can, please verify if this bug still exists or not and report back your results. General information regarding the release can also be found here: http://www.ubuntu.com/testing/ . Thanks!

closing invalid Baltix task

I got this error also in Hardy Alpha 4 (32bit). System can boot only with 'noapic'.

Processor: AMD Athlon 64 3200+, Motherboard: GIGABYTE M55S-S3 nF550

Ryan, since you are the original bug reporter, care to comment if this is still an issue for you? Based on the comments posted this appears to be fixed for some and not for others. Thanks.

Please note this report will remain open against the actively developed kernel but is being closed against 2.6.22. Thanks!

I´m a new user but perhaps my experience will help. I´m using Ubuntu 7.10 with an AMD 4400 64 X2; Asus A8N SLI Premium. I get the error, but the system will boot and seems to operate great.

Tested here on a computer which was not able to boot with Gutsy, booted 7 times without problem with Hardy, so setting to fix released.

I'm now seeing this again. It works with 2.6.24-4 (an early Hardy alpha kernel) but not the final release kernel, 2.6.24-16. In 2.6.24-4, I'm now getting the following message about where I used to get the error message:

[ 17.289012] pnpacpi: exceeded the max number of IO resources: 24
[ 17.289043] pnpacpi: exceeded the max number of IO resources: 24

Probably doesn't help, but just saying. It's not fixed, at any rate, at least not for everyone. Since this seems to be some kind of timing bug or race condition or something, I guess making even very slight changes to the way the kernel starts might cause issues.

Hardy final 8.04 generic works fine. -rt version panics.

This issue still exists in hardy heron amd64. I've got an Asus M3N78-EH as motherboard. But the only way I get this message is when I try to boot the breezy Live CD. In hardy with the options splash and quiet removed it stops after it detected the SATA II harddisk. If I disable APIC in the BIOS there is one more line, which tells me that there is no floppy and it stops too. After it times out it tries again and again until I get into BusyBox.
I had almost the same issue with my old motherboard, and Asus M2NPV-VM. If you can read Dutch, I've posted the story about that here: <http://www.qense.nl/posts/nieuwe-computernieuwe-computer/>.

It's actually <http://www.qense.nl/posts/nieuwe-computer/>, my excuses.

I'm sorry, but I forgot again something. The first story happened after I replaced my old motherboard, without reinstalling the system. The second case happened much earlier with the motherboard I replaced. I had just to install the system in with the alternate CD and remove powernowd before/very soon after the graphical environment was loaded. That would stop it from freezing and APIC could be re-enabled. After I updated the BIOS of the M2NPV-VM I could enable APIC again and install powernowd without problems. I filed bug 85370 to report this.

While I was making my previous comments the computer that's having the problems was booting the LiveCD. When I returned to it to do some testing I expected it to be in the BusyBox, but instead it had booted! I'm attaching kern.log, messages, acpid and syslog. You can see the SATA 3gb/s 5 s timeout part, that's what I was talking about in the first comment. But this time it got somehow past it!

Another small bit of information, I'm really sorry for all the mail noise, is that when I try to boot the installed install, instead of the LiveCD with the noapic and nolapci options(like I used with the LiveCD) is that I still get booted into BusyBox.The error I get is that the hardisk (.dev/sda1 or a long UUID) can't be found. Maybe it's a SATA problem.

I'm sorry, but this is of course not this bug. It wasn't fixed in breezy, but it is in hardy. The real problem is this: bug #231159 . If you think you can help with that please do. I'm sorry for the useless mail noise.

Will somebody repair that -rt problem? Cause I have macbook 1,1 and with -rt
hardy kernel (2.6.24-18 but also with older) I have something like 75%
probablity that I will not boot and give my kernel panic message. And this
problem is here from gutsy and therefore I am asking who is responsible for
this?

2008/5/18 Sense Hofstede <email address hidden>:

> I'm sorry, but this is of course not this bug. It wasn't fixed in
> breezy, but it is in hardy. The real problem is this: bug #231159 . If
> you think you can help with that please do. I'm sorry for the useless
> mail noise.
>
> --
> Kernel panic - not syncing: IO-APIC + timer doesn't work!
> https://bugs.launchpad.net/bugs/54621
> You received this bug notification because you are a direct subscriber
> of the bug.
>

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.