Memory corruption when inserting usb 3 uas device

Bug #757242 reported by Khashayar Naderehvandi
36
This bug affects 6 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Herton R. Krzesinski

Bug Description

The kernel panics, consistently, upon inserting a usb 3 hard drive. This issue wasn't there in previous ubuntu releases (at least not in maveric). After inserting a usb 3-device, the only thing to do is a hard reboot.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: linux-image-2.6.38-8-generic 2.6.38-8.41
Regression: Yes
Reproducible: Yes
ProcVersionSignature: Ubuntu 2.6.38-8.41-generic 2.6.38.2
Uname: Linux 2.6.38-8-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.23.
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: khashayar 2035 F.... pulseaudio
 /dev/snd/pcmC0D0p: khashayar 2035 F...m pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xfbff8000 irq 55'
   Mixer name : 'Realtek ALC889'
   Components : 'HDA:10ec0889,1458a022,00100004'
   Controls : 38
   Simple ctrls : 21
Card1.Amixer.info:
 Card hw:1 'HDMI'/'HDA ATI HDMI at 0xfbafc000 irq 56'
   Mixer name : 'ATI R6xx HDMI'
   Components : 'HDA:1002aa01,00aa0100,00100100'
   Controls : 4
   Simple ctrls : 1
Card1.Amixer.values:
 Simple mixer control 'IEC958',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
Card2.Amixer.info:
 Card hw:2 'Generic'/'HD-Audio Generic at 0xfb8fc000 irq 57'
   Mixer name : 'ATI R6xx HDMI'
   Components : 'HDA:1002aa01,00aa0100,00100200'
   Controls : 4
   Simple ctrls : 1
Card2.Amixer.values:
 Simple mixer control 'IEC958',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
Date: Mon Apr 11 10:09:41 2011
HibernationDevice: RESUME=UUID=83d1e0cf-a843-4045-817c-693889e1945e
InstallationMedia: Kubuntu 11.04 "Natty Narwhal" - Alpha amd64 (20110303)
MachineType: Gigabyte Technology Co., Ltd. P55A-UD4
ProcEnviron:
 LANGUAGE=
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.38-8-generic root=UUID=4342d9cc-40e1-4cc7-9cb3-1ec4e89787eb ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-2.6.38-8-generic N/A
 linux-backports-modules-2.6.38-8-generic N/A
 linux-firmware 1.50
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/23/2010
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F12
dmi.board.name: P55A-UD4
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrF12:bd06/23/2010:svnGigabyteTechnologyCo.,Ltd.:pnP55A-UD4:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnP55A-UD4:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: P55A-UD4
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

Revision history for this message
Khashayar Naderehvandi (khashayar) wrote :
Revision history for this message
Khashayar Naderehvandi (khashayar) wrote :

I think I managed to save an output of dmesg after the panic (through ssh), I'll attach it here.

summary: - Kernel pacnic upon inserting usb 3 device
+ Kernel panic upon inserting usb 3 device
Changed in linux (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Herton R. Krzesinski (herton) wrote : Re: Kernel panic upon inserting usb 3 device

Khashayar, can you boot the kernel with slub_debug kernel boot parameter? If you don't know how to add slub_debug, follow the guide at https://wiki.ubuntu.com/Kernel/KernelBootParameters , feel free to ask here if you have any doubt. After you boot with slub_debug, do the same test and attach again the dmesg/panic you get after plugging the usb3 device. This may help tracking down further the real issue (not sure it's a memory corruption case, but can help).

Changed in linux (Ubuntu):
status: Triaged → Incomplete
Revision history for this message
Khashayar Naderehvandi (khashayar) wrote :

Herton,

I'll get around to do this as soon as I can. It might take a little while though, our router here recently broke down so I can't ssh in from another computer.

Revision history for this message
Jeremy Foshee (jeremyfoshee) wrote :

natty has no proposed candidates ergo, this cannot be a regression-proposed bug. Removing the tag.

~JFo

tags: removed: regression-proposed
Revision history for this message
Khashayar Naderehvandi (khashayar) wrote :

Finally, here's the dmesg output after the same procedure, except I booted with slub_debug.

Jeremy, sorry for mistagging. However, this bug is a regression, since inserting a device in a usb 3.0 port worked just fine in maverick meerkat.

On a sidenote: It was a long time ago I was involved in any ubuntu project, but surely, a hard crash like this which is consistently and easily triggered, should warrant an importance level higher than medium?

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in linux (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Herton R. Krzesinski (herton) wrote :

Please install the kernel I uploaded at http://people.canonical.com/~herton/lp757242/

Just download linux-image-2.6.38-9-generic_2.6.38-9.43~lp757242r1_amd64.deb and install it with sudo dpkg -i linux-image-2.6.38-9-generic_2.6.38-9.43~lp757242r1_amd64.deb

Keep using the slub_debug with this new kernel, boot into it and do the same test, then please attach the dmesg output of it. I enabled usb and xhci-hcd debugging to try to get more information about the crash.

Changed in linux (Ubuntu):
assignee: nobody → Herton R. Krzesinski (herton)
status: Triaged → Incomplete
Revision history for this message
Khashayar Naderehvandi (khashayar) wrote :

Here's the new dmesg output, running the new kernel with slub_debug.
Hope it helps.

Changed in linux (Ubuntu):
status: Incomplete → Triaged
Revision history for this message
Herton R. Krzesinski (herton) wrote :

Thanks for the output, unfortunately the log is truncated because of the much more extra debugging info. Can you add also log_buf_len=1M to the kernel boot parameters and redo the test, attaching the resulting dmesg?

Changed in linux (Ubuntu):
status: Triaged → Incomplete
Revision history for this message
Reinout (snotgurg) wrote :

I noticed similar behaviour on my system. With the standard ' linux-image-2. 2.6.38-8.42' i get consistent kernel panics with either the mouse or the USB3 disk inserted on the USB3 host port. So i installed the kernel provided here and set the debugging options as requested.
Unfortunately with that kernel i get no response at all: no panic and also no working USB device. The only response i get is this message in syslog:
Apr 28 19:53:45 gribus kernel: [ 91.531064] do_IRQ: 0.112 No irq handler for vector (irq -1)

If there still is a way i can help please let me know.

Revision history for this message
Herton R. Krzesinski (herton) wrote :

Reinout, may be your issue is different, can you provide dmesg with the oops? And full dmesg with the testing kernel.

Revision history for this message
Reinout (snotgurg) wrote :

Herton, Let me try to provide you with some information to see if you are right.
First here's the dmesg output of the testing kernel

Revision history for this message
Reinout (snotgurg) wrote :

Since the machine dies upon insertion of the USB disk i cannot issue a dmesg command. I had to fiddle with netcat to get some form of output. It does not even appear in the kernel log.
Anyway: here's a copy of the console output immediately after inserting the USB disk with the standard kernel (2.6.38-8-generic #42)
I'm no kernel developer but it looks like a lot of framebuffer/graphics related messages to me. I hope it's of any use.

Revision history for this message
Herton R. Krzesinski (herton) wrote :

@Reinout -- not sure the exact cause of the crash/panic with the 2.6.38-8-generic #42 kernel, unfortunately the traces don't provide much useful info, but probably the issue is solved with the usb fixes that went after 2.6.38.2 (the testing kernel I posted is rebased to 2.6.38.4). Anyway, you still have a problem, your usb 3 host doesn't work with msi it seems, there is a similar report at http://www.spinics.net/lists/linux-usb/msg44951.html

But this irq/msi problem is different from the original issue reported here, can you open a new bug report, following instructions for launchpad report at https://wiki.ubuntu.com/Kernel/Bugs ?

@Khashayar Naderehvandi -- any news about adding log_buf_len=1M and provide new dmesg?

summary: - Kernel panic upon inserting usb 3 device
+ Memory corruption when inserting usb 3 device
summary: - Memory corruption when inserting usb 3 device
+ Memory corruption when inserting usb 3 uas device
Revision history for this message
Khashayar Naderehvandi (khashayar) wrote : Re: [Bug 757242] Re: Kernel panic upon inserting usb 3 device

Not sure if I can post here simply through replying to the email address,
but it's worth a shot.

Unfortunately, I had to leave town for a couple of days. I'll try to get the
full log once I'm back.

Sent from phone.

Regards,
K.
Den 29 apr 2011 16:36 skrev "Herton R. Krzesinski" <
<email address hidden>>:

Revision history for this message
Reinout (snotgurg) wrote :

@Herton,
Thanks for your help. At least i am beginning to understand the nature of my issues now. I seem to have several problems with this machine (A brand new laptop with Sandy Bridge, Hybrid Graphics w/Optimus that cannot be disabled from bios) It will take some time and effort to disentangle the symptoms and connect them to their individual causes. I feel like i am one step in that direction, so thanks again.

Revision history for this message
Peter Maloney (peter-maloney) wrote :

This still happens in kubuntu 11.10, which I just dist upgraded today. (and kubuntu 10.something that I had before today)

It does not happen when I plug my USB 2.0 mouse in the USB 3.0 port. It does not happen when I plug in a small port-powered USB 3.0 hard disk into the USB 3.0 port (which I then tested at 70 MB/s). But it happens when I add a big external "Iomega Prestige Desktop Hard Drive" in. I get lots of general protection faults on various processes (first is usually "ata_id" process). And when switching tty consoles, it starts rapidly spewing messages like "[ 376.000000 ] sr 7:0:0:1: sense urb submission failure". And after enough processes crash, it is not possible to switch consoles, because the system panicked.

$ uname -a
Linux peter 3.0.0-17-generic #30-Ubuntu SMP Thu Mar 8 20:45:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.10
DISTRIB_CODENAME=oneiric
DISTRIB_DESCRIPTION="Ubuntu 11.10"

I plan to test it on a SuSE or Debian live CD to compare.

Revision history for this message
Peter Maloney (peter-maloney) wrote :

Update: It works 100% in openSUSE 12.1, but spews these messages constantly:

Apr 5 14:51:32 linux-z66y kernel: [ 5399.002383] xhci_hcd 0000:0a:00.0: WARN: Stalled endpoint
Apr 5 14:51:34 linux-z66y kernel: [ 5400.537743] xhci_hcd 0000:0a:00.0: WARN: Stalled endpoint
Apr 5 14:51:34 linux-z66y kernel: [ 5401.049646] xhci_hcd 0000:0a:00.0: WARN: Stalled endpoint

The thing that motivated me to try it was an extreme problem with Ubuntu, where on these specific Iomega USB disks, using rsync to copy from a server to the disks would hang the system completely. With 6 disks at once, it hangs within 10 seconds. Using ionice and nice might make it take a bit longer, but not much longer. With 3 disks (with ionice and nice), it took about 3 hours to hang. So my desktop is now openSUSE.

A week ago, I copied a few disks to the server without problems. And 3 weeks ago, I copied 9 2TB disks (not Iomega) simultaneously to the server. Also yesterday I had 9 Iomega disks hooked up, but 4 of the USB ports won't recognize the disks.

This also reminds me of a year ago, where I had about 4 of these same Iomega disks hooked up to a Dell Precision (different machine), and tried to copy from disk 1 to 2, and from 3 to 4 at the same time, and then after a while, the USB bus stopped working. So I moved the USB cables to different ports, and it worked again, and failed again eventually. After I ran out of ports and not even the USB mouse worked, I rebooted. Initially I blamed the hardware... but it does the same thing on server hardware with Ubuntu. And now the previously described funny behavior (similar but not same as old behavior) with my new machine, but only with these Iomega disks.

And correction: it was natty a few days ago (11.04), but I said it was Ubuntu 10 in my last post.

Revision history for this message
Herton R. Krzesinski (herton) wrote :

I belive this was related to the uas driver, which is known to cause issues and because of that was marked as broken and disabled on all stable kernels recently (3.0.57 and onwards, commit fb37ef98015f864d22be223a0e0d93547cd1d4ef). Natty doesn't got the fix as it is already EOL, but Oneiric and later have now the uas driver disabled, so I'm marking this fixed. If you still experience the problem, please set the status back from fix released.

Changed in linux (Ubuntu):
status: Incomplete → Fix Released
Revision history for this message
Subhendu Malakar (subhendu) wrote :

Somehow this looks like the same problem i'm having now.
I have a 1TB external hard disk drive which has usb3.0 port. many a times, it doesn't detect my hard disk and after plugging it out, the computer freezes due to high cpu usage by rsyslogd and kworker. surfing the net found out that this is caused by the uas module. checked via usb-devices and found that's true. after plugging out the device, rsyslogd writes this in infinite loop:

cmd urb submission failed

how to disable uas module permanently?

To post a comment you must log in.