[ASRock P55M Pro] nointremap needed - Blocked an interrupt request due to source-id verificiation failure

Bug #605686 reported by Dave Gilbert
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Expired
Low
Unassigned

Bug Description

Machine works fine on latest 2.6.32 in lucid, fails to boot with black screen on todays Maverick 2.6.35-7-generic #12-Ubuntu. From grub if I remove the load_video and set gfxpayload=keep I see the content shown in the attached jpg which is the tail end of a crash, from within check_timer and I see a:
INTR-REMAP:[fault reason 38] Blocked an interrupt request due to source-id verification failure
Kernel panic - not syncing: timer doesn't work through Interrupt remapped IO-APIC

Removing the quiet and splash by themselves didn't help. nomodeset didn't help. Removing the netconsole line I've got didn't help either - adding nointremap gets me booting. The Lucid 2.6.32 still boots. My current booting command line is:
BOOT_IMAGE=/boot/vmlinuz-2.6.35-7-generic root=UUID=d8314979-8ace-401c-88df-728ad6b89cb5 ro ignore_loglevel netconsole=4444@192.168.66.33/eth0,514@192.168.66.25/00:a0:d1:63:68:a6 nointremap

I've also extracted the acpidump/dmar :
acpidump --table DMAR -b > dmar.out

that gives Wrong checksum for generic table, but gives the output; and I then used iasl to pull that apart - output also attached.

WORKAROUND: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/605686/+attachment/1456378/+files/intremap-hack.patch

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: linux-image-2.6.35-7-generic 2.6.35-7.12
Regression: Yes
Reproducible: Yes
ProcVersionSignature: Ubuntu 2.6.35-7.12-generic 2.6.35-rc4
Uname: Linux 2.6.35-7-generic x86_64
AcpiTables: Error: [Errno 13] Permission denied
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.23.
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: VT1708S Analog [VT1708S Analog]
   Subdevices: 2/2
   Subdevice #0: subdevice #0
   Subdevice #1: subdevice #1
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: dg 1861 F.... pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xfbcf8000 irq 52'
   Mixer name : 'VIA VT1708S'
   Components : 'HDA:11060397,18490397,00100000'
   Controls : 37
   Simple ctrls : 21
Card1.Amixer.info:
 Card hw:1 'HDMI'/'HDA ATI HDMI at 0xfbffc000 irq 53'
   Mixer name : 'ATI R6xx HDMI'
   Components : 'HDA:1002aa01,00aa0100,00100100'
   Controls : 4
   Simple ctrls : 1
Card1.Amixer.values:
 Simple mixer control 'IEC958',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
CheckboxSubmission: f2d10bd9f943a85b486a282e7840a570
CheckboxSystem: 0531969bcfd4f03af7405c98dc94a948
Date: Thu Jul 15 01:59:50 2010
HibernationDevice: RESUME=UUID=6f692bcf-9da3-442b-854d-1d1c129a65fd
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.

 virbr0 no wireless extensions.
MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M.
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.35-7-generic root=UUID=d8314979-8ace-401c-88df-728ad6b89cb5 ro ignore_loglevel netconsole=4444@192.168.66.33/eth0,514@192.168.66.25/00:a0:d1:63:68:a6 nointremap
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_GB.utf8
 SHELL=/bin/bash
RelatedPackageVersions: linux-firmware 1.37
RfKill:

SourcePackage: linux
dmi.bios.date: 09/10/2009
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: P1.50
dmi.board.name: P55M Pro
dmi.board.vendor: ASRock
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrP1.50:bd09/10/2009:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnASRock:rnP55MPro:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To Be Filled By O.E.M.
dmi.product.version: To Be Filled By O.E.M.
dmi.sys.vendor: To Be Filled By O.E.M.

Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :
Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :
summary: - [maverick] noioremap needed - Blocked an interrupt request due to
+ [maverick] nointremap needed - Blocked an interrupt request due to
source-id verificiation failure
Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote : Re: [maverick] nointremap needed - Blocked an interrupt request due to source-id verificiation failure
Download full text (4.9 KiB)

I've got a horrible patch that seems to work for me - I just turn off interrupt remapping in check-timer if it fails;
I'm sure there are a bunch of other things you are supposed to do; but hey what can I say - it's booted, running X, sound, ether, disks.

dmesg now has:

[ 0.041763] DMAR: Host address width 36
[ 0.041809] DMAR: DRHD base: 0x000000fed91000 flags: 0x0
[ 0.041863] IOMMU 0: reg_base_addr fed91000 ver 1:0 cap c9008010e60262 ecap f0206a
[ 0.041927] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.041979] IOMMU 1: reg_base_addr fed90000 ver 1:0 cap c90780106f0462 ecap f020ea
[ 0.042043] DMAR: RMRR base: 0x000000000e4000 end: 0x000000000e8bff
[ 0.042093] DMAR: RMRR base: 0x000000c77eb400 end: 0x000000c77fffff
[ 0.042209] IOAPIC id 8 under DRHD base 0xfed90000 IOMMU 1
[ 0.042258] HPET id 0 under DRHD base 0xfed90000
[ 0.042305] HPET id 0 under DRHD base 0xfed90000
[ 0.042352] HPET id 0 under DRHD base 0xfed90000
[ 0.042399] HPET id 0 under DRHD base 0xfed90000
[ 0.042446] HPET id 0 under DRHD base 0xfed90000
[ 0.042493] HPET id 0 under DRHD base 0xfed90000
[ 0.042540] HPET id 0 under DRHD base 0xfed90000
[ 0.042587] HPET id 0 under DRHD base 0xfed90000
[ 0.042637] DRHD: handling fault status reg 202
[ 0.042686] INTR-REMAP: Request device [[00:1f.0] fault index 1
[ 0.042687] INTR-REMAP:[fault reason 38] Blocked an interrupt request due to source-id verification failure
[ 0.043047] Enabled Interrupt-remapping
[ 0.043095] Setting APIC routing to flat
[ 0.043141] alloc irq_desc for 40 on node 0
[ 0.043187] alloc kstat_irqs on node 0
[ 0.043240] alloc irq_desc for 41 on node 0
[ 0.043285] alloc kstat_irqs on node 0
[ 0.043490] alloc irq_2_iommu on node 0
[ 0.043540] alloc irq_2_iommu on node 0
[ 0.043589] alloc irq_2_iommu on node 0
[ 0.043637] alloc irq_2_iommu on node 0
[ 0.043686] alloc irq_2_iommu on node 0
[ 0.043735] alloc irq_2_iommu on node 0
[ 0.043783] alloc irq_2_iommu on node 0
[ 0.043832] alloc irq_2_iommu on node 0
[ 0.043881] alloc irq_2_iommu on node 0
[ 0.043930] alloc irq_2_iommu on node 0
[ 0.043979] alloc irq_2_iommu on node 0
[ 0.044028] alloc irq_2_iommu on node 0
[ 0.044076] alloc irq_2_iommu on node 0
[ 0.044125] alloc irq_2_iommu on node 0
[ 0.044174] alloc irq_2_iommu on node 0
[ 0.044363] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.048368] DRHD: handling fault status reg 2
[ 0.048417] INTR-REMAP: Request device [[00:1f.0] fault index 1
[ 0.048418] INTR-REMAP:[fault reason 38] Blocked an interrupt request due to source-id verification failure
[ 0.058349] DRHD: handling fault status reg 102
[ 0.058398] INTR-REMAP: Request device [[00:1f.0] fault index 1
[ 0.058399] INTR-REMAP:[fault reason 38] Blocked an interrupt request due to source-id verification failure
[ 0.068330] DRHD: handling fault status reg 202
[ 0.068378] INTR-REMAP: Request device [[00:1f.0] fault index 1
[ 0.068379] INTR-REMAP:[fault reason 38] Blocked an interrupt request due to source-id verification failure
[ 0.078310] DRHD: handling fault status reg 302
[ 0.078358] INTR-R...

Read more...

tags: added: patch
Revision history for this message
Youquan Song (youquan-song) wrote :

diff --git a/drivers/pci/intr_remapping.c b/drivers/pci/intr_remapping.c
index 1315ac6..0ccb571 100644
--- a/drivers/pci/intr_remapping.c
+++ b/drivers/pci/intr_remapping.c
@@ -453,7 +453,8 @@ int free_irte(int irq)
 static void set_irte_sid(struct irte *irte, unsigned int svt,
                         unsigned int sq, unsigned int sid)
 {
- irte->svt = svt;
+ printk("svt=0x%x,sq=0x%x,sid=0x%x\n",svt,sq,sid);
+ irte->svt = 0;
        irte->sq = sq;
        irte->sid = sid;
 }

please try this patch, and show me the printk information.

Can you upload the BIOS dump information by acpidump tool in pmtools?

Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

Hi Youquan,
  Thanks - it seems to boot with that; here is the dmesg output and acpidump

Dave

Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

Note acpidump does give 'Wrong checksum for generic table!'

Revision history for this message
papukaija (papukaija) wrote :

Looking at the attachments in this bug report, I noticed that an attachment was not flagged as a patch. A patch contains changes to an Ubuntu package that will resolve a bug and this attachment is one! Subsequently, I've checked the patch flag for it. In the future when submitting patches please use the patch checkbox as there are some Launchpad searches that use this feature. You can learn more about the patch workflow at https://wiki.ubuntu.com/Bugs/Patches.

Revision history for this message
papukaija (papukaija) wrote :

Sorry, wrong bug. Please ignore comment 7.

summary: - [maverick] nointremap needed - Blocked an interrupt request due to
- source-id verificiation failure
+ nointremap needed - Blocked an interrupt request due to source-id
+ verificiation failure
Revision history for this message
Tim Gardner (timg-tpi) wrote : Re: nointremap needed - Blocked an interrupt request due to source-id verificiation failure

Youquan Song - Is your patch a real solution, or just a debug patch? If its real, are you pushing it upstream?

Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

It would be nice to make sure we had some fix for this in the release; I can't believe I'm going to be the only one to hit it.

Dave

Revision history for this message
Youquan Song (youquan-song) wrote :

It is the platform issue of not support source id check since my drafting/testing patch works.
Currently, there is an upstream patch which will allow the interrupt remppaing to ignore source id checking, so it will be possible to work round this issue by adding kernel option "intremap=nosid"

Thanks
-Youquan

commit d1423d5679875ebbbc2fc63b33d465baceee0430
Author: Chris Wright <email address hidden>
Date: Tue Jul 20 11:06:49 2010 -0700

    intr-remap: allow disabling source id checking

    Allow disabling the source id checking while programming the interrupt
    remap table entry. Useful for debugging or working around the broken
    source id checks on some platforms.

    Signed-off-by: Chris Wright <email address hidden>
    Acked-by: Suresh Siddha <email address hidden>
    Acked-by: Weidong Han <email address hidden>
    Signed-off-by: David Woodhouse <email address hidden>

Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

Hi Youquan,
  Thanks for the answer.
OK but I can already work around this at boot time by passing nointremap.

Is there no safe way to make this automatically fall back if intremap or source-id isn't working?

Remember, that the behaviour as seen by the user is a black screen/failure to boot - it's not very
helpful. Is my patch actually safe? If it's safe then I suggest putting something similar to it in so that
if its failing the kernel falls back to something safe.

Dave

Revision history for this message
Youquan Song (youquan-song) wrote :

No. your work round is very risk by merely jumping off the panic check. It will push the whole system into unstable situation if the timer interrupt is not guaranteed to route through the interrupt remapping when interrupt remapping is actually enabled.

The source id checking in interrupt remapping enable the incoming interrupt resource validation, but some platform or chipset have issue for this feature, so we need the kernel adding a option to ensure user to disable it on the issued platform.
source id only a small feature in interrupt remapping, so it is no good choice to disable whole interrupt remapping.

So I suggestion to backport the "nosid" kernel option, tell end user or by dmidecode to backlist this feature on the target platform.

 Thanks
-Youquan

Revision history for this message
Riccardo (unmak1) wrote :

If can help I attach my dmidecode output because I have the same problem.

As you can see I have an older BIOS version but the CPU and motherboard is the same.

Brad Figg (brad-figg)
Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Alin Posorovaschi (alinposho) wrote :

Today I tried to install Ubuntu 11.04 amd64 on my PC(Intel Core i7 860, ASRock P55 Deluxe motherboard) and immediately after booting from the CD I get:
"kernel panic - not syncing: timer doesn't work through Interrupt-remapped IO-APIC" (this is just the first line), and the installation stopped, of course.

tags: removed: regression-potential
Revision history for this message
Brad Figg (brad-figg) wrote :

The attached "patch" would never be considered for inclusion in a stable kernel. Therefor, I marked it as not a patch.

Revision history for this message
Brad Figg (brad-figg) wrote :

The relevant patch is in the Natty kernel. Is this still an issue for anyone? If it is, which release are you running?

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Dave Gilbert (ubuntu-treblig) wrote :

Still does it in PP-alpha-1

Linux major 3.2.0-2-generic #6-Ubuntu SMP Thu Dec 1 19:31:52 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

if I remove the nointremap I get (among the rest of stuff scrolling past):

INTR-REMAP: Request device [[00:1f.0] fault index 1
INTR-REMAP:[fault reason 38] Blocked an interrupt request due to source-id verification failure
DRHD: handling fault status reg 102

Dave

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
klhsieh@kunlung.com (klhsieh) wrote :

Hi Youquan Song,

Your comment #13 is helpful to me.

I also meet source-id verification failure in other distribution.
You said "some platform or chipset have issue for this feature". Now I have a PCIe SSD card (a product of my compnay), I want to clarify the issue is platform issue or my product defect.

My product can be flashed to a AHCI SSD or a SCSI-like SSD.
When it is AHCI SSD, inbox AHCI driver is loaded, everything is OK.
When it is SCSI-like SSD, it needs to installed our driver. I saw source-id verification failure when loading driver.
The interesting thing is, if my driver doesn't use 'Message Signaled Interrupts', the problem doesn't occur.
('Message Signaled Interrupts' default is on)

My OS is RHEL 6.1 (kernel is 2.6.32-131.0.15.el6.x86_64). (Sorry here is Ubuntu forum :P)
My platform is HP ProLiant DL388 G7, two Xeon E5620, chipset is 5520 from lspci.

Revision history for this message
Youquan Song (youquan-song) wrote :

Hi klhsieh,

Does the "intremap=nosid" kernel options work for you?

Thanks
-Youquan

Revision history for this message
klhsieh@kunlung.com (klhsieh) wrote :

Hi Youquan,

Yes, it works.

But I want to know the root cause: 5520 chipset has issue? The hardware of my product has issue? or driver of my product has issue?

My plan is: try to print IRTE; if IRTE content is not what I expect, try to find the time the kernel created the entry and check why the kernel filled with an unexpected value.

Revision history for this message
Youquan Song (youquan-song) wrote :

Hi klhsieh,

I did not encouter the issue of source-id verification failure when interrupt remapping opend at platform with 5520 chipset.
What's the result of "print IRTE"? Adding "apic=verbose" kernel option, it also can printf out of setting IRTE.

Thanks
-Youquan

penalvch (penalvch)
tags: added: bios-outdated-2.30
tags: added: precise
Revision history for this message
penalvch (penalvch) wrote :

Dave Gilbert, this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? If so, could you please test for this with the latest development release of Ubuntu? ISO images are available from http://cdimage.ubuntu.com/daily-live/current/ .

If it remains an issue, could you please run the following command in the development release from a Terminal (Applications->Accessories->Terminal), as it will automatically gather and attach updated debug information to this report:

apport-collect -p linux <replace-with-bug-number>

Also, could you please test the latest upstream kernel available (not the daily folder) following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested. For example:
kernel-fixed-upstream-v3.13-rc3

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description. As well, please remove the tag:
needs-upstream-testing

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

As well, please remove the tag:
needs-upstream-testing

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

description: updated
summary: - nointremap needed - Blocked an interrupt request due to source-id
- verificiation failure
+ [ASRock P55M Pro] nointremap needed - Blocked an interrupt request due
+ to source-id verificiation failure
Changed in linux (Ubuntu):
importance: Undecided → Low
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

Changed in linux (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.