OOPS: "Unable to handle kernel NULL pointer dereference"

I've now seen several more instances of this crash, both during start-up and after the system has been running for some time.

When it occurs in a user session it has always killed all user input via the (PS/2-style) keyboard (not a Bluetooth keyboard). All user-space keyboard active fails with only the mouse (ironically, a Bluetooth mouse!) operating.

In addition, the GUI will fail to respond to many actions such as displaying the taskbar (I have it set to auto-hide), the shutdown/log-off/suspend/hibernate/lock menu buttons do not cause their actions to fire, and slowly the system will become unresponsive.

Even Sys-Rq key sequences fail to cause their expected actions although, if at a tty screen, the kernel sys-log will show the key-pressed were received by the kernel.

Here's the /var/log/kern.log for a mid-session crash:

kernel: [78323.255743] Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP:
kernel: [78323.255752] [klist_del+0x18/0x80] klist_del+0x18/0x80
kernel: [78323.255764] PGD 56ed8067 PUD 63565067 PMD 0
kernel: [78323.255770] Oops: 0000 [2] SMP
kernel: [78323.255775] CPU 0
kernel: [78323.255778] Modules linked in: snd_usb_audio snd_usb_lib ov51x_jpeg uvcvideo isofs udf hidp hid r5u870 usbcam videodev v4l1_compat compat_ioctl32 v4l2_common videobuf_dma_sg videobuf_core tun binfmt_misc af_packet bnep rfcomm l2cap nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp bridge kvm_intel kvm ppdev ipv6 acpi_cpufreq cpufreq_stats cpufreq_conservative cpufreq_userspace cpufreq_powersave cpufreq_ondemand freq_table sbs sbshc dock container iptable_filter ip_tables x_tables sbp2 parport_pc lp parport arc4 ecb joydev pcmcia hci_usb snd_hda_intel bluetooth snd_pcm_oss snd_mixer_oss nvidia(P) iwl3945 snd_pcm iwlwifi_mac80211 snd_page_alloc snd_hwdep snd_seq_dummy pcspkr cfg80211 snd_seq_oss evdev i2c_core sony_laptop iTCO_wdt snd_seq_midi tifm_7xx1 iTCO_vendor_support snd_rawmidi snd_seq_midi_event serio_raw video output snd_seq psmouse tifm_core button snd_timer snd_seq_device yenta_
kernel: ocket rsrc_nonstatic pcmcia_core battery ac snd shpchp pci_hotplug intel_agp soundcore dm_multipath ext3 jbd mbcache sha256_generic aes_x86_64 cbc blkcipher nls_iso8859_1 nls_cp437 vfat fat usb_storage libusual sg sr_mod sd_mod cdrom ata_piix pata_acpi dm_crypt ata_generic ohci1394 libata scsi_mod e100 mii ieee1394 ehci_hcd uhci_hcd usbcore dm_mirror dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse
kernel: [78323.255947] Pid: 9, comm: events/0 Tainted: P D 2.6.24-21-generic #1
kernel: [78323.255951] RIP: 0010:[klist_del+0x18/0x80] [klist_del+0x18/0x80] klist_del+0x18/0x80
kernel: [78323.255959] RSP: 0018:ffff81007bde5e30 EFLAGS: 00010286
kernel: [78323.255963] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffffffffd8
kernel: [78323.255967] RDX: 0000000000000000 RSI: ffff81007bde5e30 RDI: ffff8100652aa5c0
kernel: [78323.255971] RBP: ffff8100652aa598 R08: ffff81007bde4000 R09: 0000000000000000
kernel: [78323.255974] R10: ffff810001013fe0 R11: 0000000000000001 R12: ffff8100652aa5c0
kernel: [7...

This appears to be caused by:
----
commit c7202b637779f7e26decd6525a2f4463db918aaf
Author: Michael Buesch <email address hidden>
Date: Sat Apr 19 16:53:00 2008 +0200

    b43: Add more btcoexist workarounds
    Bug: #257020

    This adds more workarounds for devices with broken BT bits.

    Backported from 9fc38458355525f801cd2ab403ac89850489a05e.
 ----

Bluetooth Device details (verbose "lsusb -s 3:2 -v" log attached):

Bus 003 Device 002: ID 044e:300d Alps Electric Co., Ltd
Device Descriptor:
  bLength 18
  bDescriptorType 1
  bcdUSB 2.00
  bDeviceClass 224 Wireless
  bDeviceSubClass 1 Radio Frequency
  bDeviceProtocol 1 Bluetooth
  bMaxPacketSize0 64
  idVendor 0x044e Alps Electric Co., Ltd
  idProduct 0x300d
  bcdDevice 19.15
  iManufacturer 1 ALPS
  iProduct 2 UGX
  iSerial 3 01f4233a
  bNumConfigurations 1

I should add that it has been very noticable that since the 2.6.24-21.40 update, the time it takes for the Bluetooth mouse to achieve a connection has become ridiculous. Before the update the mouse would be connected and active by the time the GDM log-in screen appeared.

After the update it can sometimes take 10 minutes and several power on/off cycles of the mouse, as well as moving it vigorously, before a connection is made.

Also, during a user session the mouse will sometimes lose the connection and take a lot of persuasion to reconnect.

I noted Tim Gardner's email on kernel-team@ to Larry Finger asking about the WiFi flag B43_BFL_BTCOEXIST.

This is the WiFi adapter in use on this Sony Vaio VGN-FE41Z:

$ sudo lspci -s 6:0 -vvvnn
06:00.0 Network controller [0280]: Intel Corporation PRO/Wireless 3945ABG Network Connection [8086:4222] (rev 02)
 Subsystem: Intel Corporation Unknown device [8086:1051]
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
 Latency: 0, Cache Line Size: 64 bytes
 Interrupt: pin A routed to IRQ 506
 Region 0: Memory at cc000000 (32-bit, non-prefetchable) [size=4K]
 Capabilities: [c8] Power Management version 2
  Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
  Address: 00000000fee0300c Data: 41a9
 Capabilities: [e0] Express Legacy Endpoint IRQ 0
  Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
  Device: Latency L0s <512ns, L1 unlimited
  Device: AtnBtn- AtnInd- PwrInd-
  Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
  Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
  Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
  Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s L1, Port 0
  Link: Latency L0s <128ns, L1 <64us
  Link: ASPM L1 Enabled RCB 64 bytes CommClk+ ExtSynch-
  Link: Speed 2.5Gb/s, Width x1

> This appears to be caused by:
> ----
> commit c7202b637779f7e26decd6525a2f4463db918aaf
> Author: Michael Buesch <email address hidden>
> Date: Sat Apr 19 16:53:00 2008 +0200
>
> b43: Add more btcoexist workarounds

The b43 driver isn't even loaded. So this patch to the b43 driver cannot cause any regression for you.
How did you find out that it was "caused by" this patch? By guessing?

Boot both kernels (2.6.24-20-generic and 2.6.24-21-generic from hardy-proposed). 2.6.24-20-generic never (and still doesn't) exhibit the issue.

As this report states, first time 2.6.24-21-generic booted this OOPS occurred. Since then it has repeated about a dozen times, sometimes on boot but also after a user session has been running some time.

The git-log shows the only difference between the two kernels is the b43 commit and the ABI bump.

git-log --pretty=format:"%h %ci %s" Ubuntu-2.6.24-20.39..Ubuntu-2.6.24-21.40
c1fc5db 2008-08-11 17:32:35 -0600 UBUNTU: Bump ABI to -21
1f53031 2008-08-11 13:17:43 -0600 UBUNTU: Ubuntu-2.6.24-20.40
c7202b6 2008-08-11 14:53:14 -0400 b43: Add more btcoexist workarounds
d7990b4 2008-08-11 14:52:49 -0400 UBUNTU: Ubuntu-2.6.24-20.39

Not really any point doing a git-bisect on that, is there?

Thank you for taking the time to report this bug and helping to make Ubuntu better. Unfortunately we can't fix it without more information.

Please include the following additional information, if you have not already done so (pay attention to lspci's additional options), as required by the Ubuntu Kernel Team:
1. Please include the output of the command "uname -a" in your next response. It should be one, long line of text which includes the exact kernel version you're running, as well as the CPU architecture.
2. Please run the command "dmesg > dmesg.log" after a fresh boot and attach the resulting file "dmesg.log" to this bug report.
3. Please run the command "sudo lspci -vvnn > lspci-vvnn.log" and attach the resulting file "lspci-vvnn.log" to this bug report.
4. Please attach your /var/log/kern.log and /var/log/kern.log.0 files to this bug report.
5. Please run Memory test as described at https://help.ubuntu.com/community/MemoryTest.

Also, your two call traces look like two separate issues. If your memory test comes back ok, it would probably be best to open a separate bug report for the second oops.

For your reference, the full description of procedures for kernel-related bug reports is available at https://wiki.ubuntu.com/KernelTeamBugPolicies Thanks in advance!

> Not really any point doing a git-bisect on that, is there?

Please revert this patch manually and recompile the kernel. You will see that this won't fix the issue.

I want to repeat, that the b43 driver is _not_ loaded when your oops occurs.
There doesn't show up anything b43 related in the logs and the oops message does tell us b43 module is not loaded.

It is not loaded, because you don't even seem to have a broadcom wireless card in the machine (from your comments it seems you have intel wifi).
So without b43 being loaded, it is pretty hard to believe how it could possibly cause any regression.

Michael:

Apologies if it looks like I just 'guessed' at the cause, but I assure you I did the basic 'diff' checks I do for all kernel issues. The git-log seemed pretty clear - the only change was the 'b43' patch and when I saw it was a bluetooth/WiFi interaction patch I didn't think I needed to dig much further.
At the time it happened, knowing I was being a guinea-pig for hardy-proposed, I was more interested in simply reporting the experience to prevent it getting promoted to hardy-updates if there was a problem.

I'm in the Kernel ACPI Team and I've spent most of my time recently building a semi-automated testing system (using DKMS instead of SystemTap) for easier (remote) investigation of this kind of issue (ACPI is plagued by them) and experience there is look to the basics (all recent changes are prime suspects unless proved otherwise!).

Yes, b43 apparently not even being used is perplexing me. I was fully aware the module wasn't in use but unless the ABI bump is somehow causing an issue with something in user-space I couldn't see how it could be anything other than some weird side effect.

I've now scheduled some time to do a git-bisect, just to try without the ABI bumps since they are now suspect.

The PC checks out in every other way (memory passes tests, no other kernel module differences or configuration between the two kernels, (I checked the initrd image just in case), etc.).

Testing is slightly hampered by the fact the OOPS is relatively rare, and random, and when it happens the only solution is a reboot. SysReq+S will sometimes sync the disks so the log should be saved, fortunately.

I've had the PC running with v2.6.24-20-generic for a while now and no sign of the OOPS or any other instability.

The only thing I've done relating in any way to the on-board radios was (about 1 week before the hardy-proposed v2.6.24-21-generic update) install Alexander Sack's NetworkManager v0.7 PPA builds to test the 3G and OpenVPN connections functionality. There is/was no problem with that with kernel 2.6.24-20-generic.

The common thing in all the OOPS back-traces is the Bluetooth module trying to add or del[ete] a connection (for the mouse). It seems to be trying to double-add or double-free too, which would explain the "sysfs: duplicate filename" or "NULL pointer deference" if it was destroyed on the first, for the second attempts.

This functionality was introduced with

commit 47b66fe95afa8400cefaea06263ab8948d8465ba
Author: Dave Young <email address hidden>
Date: Fri Feb 15 01:34:03 2008 -0800

    BLUETOOTH: Add conn add/del workqueues to avoid connection fail.

$ git-describe --contains 47b66fe
v2.6.24.3~17

I'm dropping some debug code in a build of v2.6.24-21 for the calls in hci_conn_add_conn() and hci_conn_del_conn() in net/bluetooth/hci_sysfs.c, and drivers/base/core.c device_add() and device_del(). These functions schedule and execute the workqueue activity that is seen in the back-traces.

I'm also going to monitor the user-space hal responses to the uvents.

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

This bug report is being closed because we received no response to the previous inquiry for information. Please reopen if this is still an issue in the current Ubuntu release, Jaunty Jackalope 9.04 - http://www.ubuntu.com/getubuntu/download. If the issue remains in Jaunty, please test the latest upstream kernel build - https://wiki.ubuntu.com/KernelMainlineBuilds . To reopen the bug, click on the current status under the Status column and change the status back to "New". Thanks.