[intrepid] 2.6.27 e1000e driver places Intel ICH8 and ICH9 gigE chipsets at risk

Description of problem:
I am unable to use my Ethernet controller: Intel Corporation 82566DC Gigabit Network Connection (rev 03). System does not see it. Pleae find dmesg output.

e1000e: Intel(R) PRO/1000 Network Driver - 0.2.0
e1000e: Copyright (c) 1999-2007 Intel Corporation.
ACPI: PCI Interrupt 0000:00:19.0[A] -> GSI 22 (level, low) -> IRQ 22
PCI: Setting latency timer of device 0000:00:19.0 to 64
iTCO_vendor_support: vendor-support=0
0000:00:19.0: The NVM Checksum Is Not Valid
ACPI: PCI interrupt for device 0000:00:19.0 disabled
e1000e: probe of 0000:00:19.0 failed with error -5

Version-Release number of selected component (if applicable):
Driver version 0.2.0

How reproducible:
Happens everytime

Steps to Reproduce:
1.Boot computer

What kernel version is this? Has this adapter ever worked under Fedora. If yes when did it stop?

I am sorry, i totally forgot about these details.
Kernels which i have:
2.6.25.11-97.fc9.x86_64
2.6.25.14-108.fc9.x86_64

I guess it stopped shortly after i upgraded to F9. It must have been one of first kernel updates. I am not sure if that ever worked in F9.

Strange thing, on ubuntu i can not use it too. I do not have dmesg output yet. I will try and see if this matches.

Can you post the output of 'lspci -nn -s 0000:00:19.0'?

Output you have requested:

00:19.0 Ethernet controller [0200]: Intel Corporation 82566DC Gigabit Network Connection [8086:104b] (rev 03)

Updated today my work station after two or three weeks to current Factory kernel and since then the onboard network card doesn't show up anymore:

 e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k2
 e1000e: Copyright (c) 1999-2008 Intel Corporation.
 e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
 e1000e 0000:00:19.0: setting latency timer to 64
 input: PC Speaker as /devices/platform/pcspkr/input/input3
 0000:00:19.0: 0000:00:19.0: The NVM Checksum Is Not Valid
 e1000e 0000:00:19.0: PCI INT A disabled
 e1000e: probe of 0000:00:19.0 failed with error -5

Booted an openSUSE 11.0 installation and same issue there now too. Some BIOS/checksum got broken?

Created attachment 239061
dmesg of Factory installation

Created attachment 239062
dmesg of 11.0 installation

The driver you have supports your hardware, but is erroring out on load.
The "NVM checksum is not valid" means that something corrupted your system BIOS flash.

Can you please give us details about the hardware in your system, attach the output of
# lspci -vvv > lspci.txt

# dmidecode > dmiout.txt

we have some reports that Lenovo systems (a lot of them) are starting to have this issue.

Please DO NOT run ibautil as some sites on the web suggest to try to fix this issue. It will likely cause you to have to replace your motherboard to get LAN functionality back.

Created attachment 316491
dmiout.txt

Created attachment 316492
lspci.txt

I have messed around a little with my card. Just wanted to check some suggestions point out here http://www.thinkwiki.org/wiki/Problem_with_e1000:_EEPROM_Checksum_Is_Not_Valid#Solutions

Little orange led on my ethernet is constantly flashing, when i tried with unloading e1000e module it did not changed anything. When i plugged in cable it stopped and green led showed up, meaning that connection is ok though driver still failed to load.
If you need any other info i will gladly help.

okay, so you have an HP machine with an ICH8 chipset. I don't know what the little orange LED flashing means, I will have to check on that.

can you get into the iAMT setup just after BIOS completes by pressing CTRL-p?
not sure if that might help you or not.

If I attach a debug driver here would you be willing to compile and run it?

I am not able to open iAMT setup. I believe that i do not have that option as i have found that to enable that i need to go to my BIOS settings and turn it on in Power section. Well, i do not have it there.

Yes, please attach driver.

*** Bug 428180 has been marked as a duplicate of this bug. ***

Intel did cleanup e1000 and e1000e to have no duplicate PCI IDs in both drivers. Maybe they removed this on the wrong driver.
Can you please try to unload e1000e and load e1000 manually if the card is not detected, then please add the ids to the driver on runtime:

echo "vendor device subvendor subdevice class class_mask driver_data" > \
/sys/bus/pci/drivers/e1000/new_id

All fields are passed in as hexadecimal values (no leading 0x).
The vendor and device fields are mandatory, the others are optional.

See bug 428180 - I can load the e1000 just fine but it does not work at all.

after: echo "8086 104b" > /sys/bus/pci/drivers/e1000/new_id

I see the following in dmesg:
e1000 0000:00:19.0: setting latency timer to 64
e1000: 0000:00:19.0: e1000_probe: The EEPROM Checksum Is Not Valid
/*********************/
Current EEPROM Checksum : 0xffff
Calculated : 0xbaf9
Offset Values
======== ======
00000000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
Include this output when contacting your support provider.
This is not a software error! Something bad happened to your hardware or
EEPROM image. Ignoring this problem could result in further problems,
possibly loss of data, corruption or system hangs!
The MAC Address will be reset to 00:00:00:00:00:00, which is invalid
and requires you to set the proper MAC address manually before continuing
to enable this network device.
Please inspect the EEPROM dump and report the issue to your hardware vendor
or Intel Customer Support.
/*********************/
e1000: 0000:00:19.0: e1000_probe: Invalid MAC Address
e1000: 0000:00:19.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 00:00:00:00:00:00
e1000: 0000:00:19.0: e1000_probe: This device (id 8086:104b) will no longer be supported by this driver i
n the future.
e1000: 0000:00:19.0: e1000_probe: please use the "e1000e" driver instead.
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection

I fear that the EEPROM was deleted. This may be the reason, the fix is e1000 related, but it seems that e1000e has the same reason.
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=78566fecbb12a7616ae9a88b2ffbc8062c4a89e3

I hope that Intel can help here and has a way to reprogram the EEPROM.

And how was the EEPROM deleted? This is very hard to do without having things wrong with the system itself.

First is the kernel 2.6.27-rcx? If so did the system experience a panic of some sort (not NIC related)? Was some other tool run on the system once it got into this state? Also, is does this system have some sort of manageability enabled on it? If so, disable it and try things again. Let me know about the other questions.

Yes it is based on 2.6.27-rc6 and we have no idea how the system get in this state, but we got multiple reports now :-(, all the same, installing Beta1 with an e1000e card (I will collect PCI ids of all reports later) and during the first driver load you see:

e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k2
e1000e: Copyright (c) 1999-2008 Intel Corporation.
e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
e1000e 0000:00:19.0: setting latency timer to 64
input: PC Speaker as /devices/platform/pcspkr/input/input3
0000:00:19.0: 0000:00:19.0: The NVM Checksum Is Not Valid
e1000e 0000:00:19.0: PCI INT A disabled
e1000e: probe of 0000:00:19.0 failed with error -5

in the dmesg output (for more details see attachment #1 for the full log)
If you then try to boot into a other OS version (which was working before) the network card does not work anymore with the same error, which let me think that
the eeprom was overwritten or deleted and later I found the commit in later kernels for e1000 (comment #7) which sounds somehow related for me.

Our e1000e driver differs from mainline in 3 additional patches requested by
Kent Liu (in CC now)
1. http://tinyurl.com/6253yl
2. http://tinyurl.com/5bd8v2
3. http://tinyurl.com/6rj8j7

Stephan reading your first comment again, you did only install a new kernel, you did not install Beta1, correct ?

The it is unlikely some program or configuration probing causing this, it is the e1000e driver itself.
Please also give us the PCI IDs of your e1000e card.

*** Bug 428322 has been marked as a duplicate of this bug. ***

> you did only install a new kernel, you did not install Beta1, correct ?

Correct, just "zypper dup" + reboot. openSUSE 11.1 Beta 1 didn't exist yet when I reported that bug. :-)

29: PCI 19.0: 0200 Ethernet controller
  [Created at pci.318]
  UDI: /org/freedesktop/Hal/devices/pci_8086_294c
  Unique ID: kpGf.nWnnnRlG_JE
  SysFS ID: /devices/pci0000:00/0000:00:19.0
  SysFS BusID: 0000:00:19.0
  Hardware Class: network
  Model: "Intel 82566DC-2 Gigabit Network Connection"
  Vendor: pci 0x8086 "Intel Corporation"
  Device: pci 0x294c "82566DC-2 Gigabit Network Connection"
  SubVendor: pci 0x8086 "Intel Corporation"
  SubDevice: pci 0x0000
  Revision: 0x02
  Memory Range: 0x92200000-0x9221ffff (rw,non-prefetchable)
  Memory Range: 0x92224000-0x92224fff (rw,non-prefetchable)
  I/O Ports: 0x3400-0x341f (rw)
  IRQ: 216 (no events)
  Module Alias: "pci:v00008086d0000294Csv00008086sd00000000bc02sc00i00"
  Driver Info #0:
    Driver Status: e1000e is active
    Driver Activation Cmd: "modprobe e1000e"
  Config Status: cfg=no, avail=yes, need=no, active=unknown

I have an 11.0 system with just this new kernel. I even went into Windows (which could not get a dhcp address) and installed a new BIOS version. the BIOS shows me that my macid has only FFs.

Karsten, is the fix mentioned in #7 part of our kernel CVS?

Yes with the update to rc7. But note, this is not a fix to this issue.

Some more info about affected PCI IDs:
We got 2 reports about (this and bug #428322)
  Vendor: pci 0x8086 "Intel Corporation"
  Device: pci 0x294c "82566DC-2 Gigabit Network Connection"

And one report with (bug #428180)
Vendor: pci 0x8086 "Intel Corporation"
Device: pci 0x104b "82566DC Gigabit Network Connection"

And I got one report about a working installation of Beta1 with e1000e driver

Vendor: pci 0x8086 "Intel Corporation"
Device: pci 0x109a "82573L Gigabit Ethernet Controller"

maybe that helps.

John I did add you to the duplicate bugs as well.

In just looking at this quickly, I think something is getting
balled up in one of the e1000_nvm_operations structures. This would misdirect
the driver to the improper NVM operation and potentially cause the erasure of
the EEPROM.

Created attachment 317425
driver with csum check bypass

here is a driver that just prints the message but doesn't error out if the checksum validation fails.

This should allow you to run ethtool -e ethX after loading the driver.

the difference in the driver I just attached is:
diff -rup e1000e-0.4.1.7.orig/src/netdev.c e1000e-0.4.1.7/src/netdev.c
--- e1000e-0.4.1.7.orig/src/netdev.c 2008-06-23 09:27:33.000000000 -0700
+++ e1000e-0.4.1.7/src/netdev.c 2008-09-22 16:06:59.000000000 -0700
@@ -56,7 +56,7 @@

 #define DRV_DEBUG

-#define DRV_VERSION "0.4.1.7" DRV_NAPI DRV_DEBUG
+#define DRV_VERSION "0.4.1.7_nocsum" DRV_NAPI DRV_DEBUG
 char e1000e_driver_name[] = "e1000e";
 const char e1000e_driver_version[] = DRV_VERSION;

@@ -5309,8 +5309,10 @@ static int __devinit e1000_probe(struct
                        break;
                if (i == 2) {
                        e_err("The NVM Checksum Is Not Valid\n");
+ /* JJJ skip around error path
                        err = -EIO;
                        goto err_eeprom;
+ JJJ end */
                }
        }

bob can you elaborate?

we have reports (linked at debian bugzilla) of a user that had a graphics panic and then ran into the issue. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/263555

Upstream bug references to this bug:

        http://lkml.org/lkml/2008/8/8/123
        http://lkml.org/lkml/2008/9/22/23
        http://bugzilla.kernel.org/show_bug.cgi?id=11382

also, whole piles of reports now starting to converge, many of them linked here:

http://bugzilla.kernel.org/show_bug.cgi?id=11382

I'm trying to work a plan to help address this soonest.

In my case: I saw the error message with a previous 2.6.27 kernel first but did not report it :-( My log files show the first occurence on the 10th of September which would mean that this was 2.6.27-rc6 or even 2.6.27-rc5 with SUSE patches.

I had some crashes during boot where my graphics display was somehow screwed up (did not succeed in debugging with serial console, so no report for that yet).

So, yes it could be that another error broke this but since I mainly use wireless and not the ethernet port, I only noticed this problem recently and cannot say for sure when and how it happened.

All,

If you read the entire thread pointed to in #7 you would have read Intel response to this. The thread has to do with VMWare and not Linux. VMWare is based on a very old kernel that has some poor kernel locking issues. We pointed this out to them and asked the question is any of the people responding had actually seen this problem on Linux. This is where thing went awry a bit. All (_all_) of the reports that we have so far have had this gfx panic just before this problem comes up. The current belief is that the gfx panic is scribbling all over our NVM space somehow. We are not sure how this is happening. Since we can't repro the problem without this panic happing first this is very hard for us to look at. If the gfx panic does not happen there have been no problems reported. I do not know the status of a fix for the gfx panic. We are working on repro cases but because the problem relies on a panic to another driver this is very hard for us to work on. Work will continue.

okay, so two votes for a graphics panic possibly related to the issue.
Andreas, would you be willing to comment out the code just after the
nvm.validate (validate_eeprom) in netdev.c and then try to dump your eeprom
using ethtool -e ethX

if it returns all 0xff 0xff ... then can you please download ethregs utility
from sourceforge.net/projects/e1000 and build/run it and attach or send me the
output?

we are still trying to reproduce, I'm raising the issue in priority and we will continue to update as we make progress.

Michal, have you ever booted a Fedora 10 Alpha or rawhide disk on that system?

Jesse, the question I'm left with is "what TYPE of eeprom interface does the broken NIC's implement?" I.e. what specific chip and what interface does it provide to the eeprom. Part of that you get from the customer, and part you get from Intel. Considering the Intel 7256x and 7257x parts, you have 4 different interface - that I know of.

For SPI / uW its hard to "accidentally" overwrite the prom as you have to sent it the write enable opcode first and then shift in the data. That's typically to complicated to happen via random garbage from a crash. If these are parts with SPI / uW addressed proms, then the overwrite is most likely the result of e1000e code being called. If that's the case, the debug builds of the driver would dump out the "I'm writing to the prom" messages. ( Hint: that's not the case. )

For the parallel eeprom, which are memory mapped, they can be overwritten by writing junk over top of thir address space. I think the 72573 has this type. The 72566 part also has a similar, but different memory mapped eeprom.

Point being, if the SPI based parts are having issues, this points towards programatic overwrite - as its unlikely to happen by accidental overwrite of I/O or memory space. Damage to the memory mapped parts, on the other hand, could happen because accidental overwrites are poorly defended.

Perhaps John can give us a clue as to the Intel parts which use SPI vs uW vs shadow RAM vs memory mapped. I don't see much in the code that latches the memory mapped eeproms from accidental overwrite. For the shadow ram method, it wants the write through flag to be set to true, well FF's are "true" so if this is the means to protect the underlying prom data, its a pretty feeble one. If the gfx crash writes all FF's over a swatch of memory, there you have it.

Ad comment #24: I'm not using vmware at all on this machine.

Ad comment #25: Karsten, could you compile a kernel and the tool for me? I'll then test it (note I'm the whole day in meetings and have only little time to do anything extra myself). Stephan, will you test as well?

Hi Bob, thanks for the informative reply.

all the broken ones seem to use the FLAG_HAS_FLASH, at this point all the
reports I've heard have BAR1 mapped to an area that has registers used for
read/writing the NVM. The people I've heard reporting this have a laptop based
on the ICH8 or ICH9 chipsets, aka the lan part is the 82566 or 82567.

This NVM is usually part of a larger flash which contains the BIOS and possibly
the PXE boot code as well as the LAN Non-volatile-memory (NVM)

none of our "configuration" areas use direct memory mapped mode, and unless you
call the "write_eeprom" function using ethtool, nothing should be calling
e1000e_write_flash_data_ich8lan.

we have some patches ready for setting the registers to read-only that is
mapped at the flash BAR1 area. They are not tested, but we will test a little
and post them to the mailing list tomorrow.

If you have any references to any users that *don't* have an 82566 or 567, then
please point it out.

Ad comment #23:
Do you still have this kernel around with the exact version (best would be the rpm or kernel source) ?
I added the 3 additional patches on Wed Sep 10 17:18:30 CEST 2008 . So this may related.
Ad comment #27:
I will prepare a kernel based on 11.0 and the tools.

Yes, i have rawhide on my system.
Last two kernels i have
2.6.27-0.226.rc1.git5.fc10.i686
2.6.27-0.244.rc2.git1.fc10.i686

I do not know which one killed my port. If you want me to run it or something i am unable to have any internet connection on that kernels, wifi does not work, eth you know.

I got an other report about a working system (82566MM Gigabit Network Connection [8086:1049] (rev 03)) but with a pre Beta1 kernel which is mostely identically but _without_ the 3 additional patches.

As per comment #19, I also have 8086:109a here and beta1 (kernel-pae-2.6.27-7.2) works and has not broken the hardware yet. I installed beta1 at the weekend not knowing about this bug, but am loath to boot it again if it will cook my ethernet.

I had a machine (hp 2510p) which had a lockup with garbled X display and refused to boot at all afterwards (got the mainboard replaced on warranty) and we have another 2510p which, after updating to an early 2.6.27rc, suddenly "forgot" it's 1280x800 video mode. They both have not shown e1000e trouble (yet ;), but it hints into the "graphics problem overwrites system flash" direction.

The 2510p has

00:02.0 VGA compatible controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c)
00:19.0 Ethernet controller: Intel Corporation 82566MM Gigabit Network Connection (rev 03)

00:02.0 0300: 8086:2a02 (rev 0c)
00:19.0 0200: 8086:1049 (rev 03)

Sorry to interfere, our ESX expert told me that under ESX 3.5 it wasn't necessary to have processes _writing_ to the EEPROM for this problem to occur. Rather, it happened if two processes were just _reading_ the EEPROM at the same time, due to a broken locking bit in the HW of some NICS (82546, that's what I was told).

OK I did built the ethregs package and a 11.0 kernel with the error paths commented out. The kernel is only available internally in mbuild diabelli-kkeil-179 (kernel-default).
ethregs is available in the buildservice search for it on http://software.opensuse.org/search

Please stop talking about VMWare. That OS has a problem that was fixed by a work-around in the driver. What the VMWare guys saw has never been seen in Linux. It has _nothing_ to do with this bug.

As Jesse said in his mail we are working on some NVM protection patches for our driver. This won't fix the root cause of writing over our NVM area but it might help to find what/who is writing all over it.

For comment #30 and #31, if the system did not see one of the gfx driver panics our NVM remains fine. So if that panic does not happen our NIC NVM remains fine as well.

As soon as the patches are out of test we'll be pushing them upstream. We'll post links to this BZ once this happens today.

If the graphics crash uses DMA to override the MMIO area (assuming it's really
the graphics crash) then it would require VT-d to protect it. But write protecting it to on the CPU level is probably a good start, just it might not be not enough.

Does this mean Fedora 9 is not to blame for killing e1000e?

Slashdot reported that Fedora 9 and 10 are affected, but it sounds like only rawhide has the problem.

Agreed Andi. This is all we can do however. Let's see what happens with the patches with the people that are actually seeing the problem.

Those would first need to fix their mac addresses to try again right?
Also is there other vital information in that EEPROM?

Some of those systems should have VT-d. Presumably it would give an message
on a stray DMA? Of course the CPU protection would be needed too.

(In reply to comment #38 from Andi N Kleen)
> Those would first need to fix their mac addresses to try again right?
> Also is there other vital information in that EEPROM?

By the way the current driver doesn't get even bound to the card that has wrong EEPROM CRC, right? So it's even not possible to easily fix its contents up using ethtool from within default installation.

Karsten already patched and built the kernel so that it binds the driver to the card even in cases of broken EEPROM checksum.

FWIW, I've heard of similar problems with recent -RT kernels.

The modified driver is in mbuild diabelli-kkeil-184 (diabelli-kkeil-179 was wrong and crash).
With this driver I see the card ethtool -e shows all FF and the MAC is also read as FF, but I set set the old MAC address with ifconfig and it seems that the card is working, but only at 10 Mbit. ethregs output follows.

Created attachment 241194
ethregs eth1 output

Ad comment #29: The kernel is not available anymore.

I strongly recommend if you are going to test for this bug or haven't seen it
yet on your ich8/9 system, that you RIGHT NOW, do ethtool -e ethX >
savemyeep.txt

Having a saved copy of your eeprom means we can help you write it back to your
system.

Jesse, What happens with those that have already a broken eeprom?

Could this be the same as bug #57976?

At the time I did not notice any connection with a graphics corruption - but would not rule it out either.
In any case the intel graphics driver was a totally different one, the one we used
at the time (i810) is no longer shipped with 11.1.

Can someone elaborate a little on the 'graphics panic'? Was this a total lockup or just a screen corruption?

(In reply to comment #44 from Andreas Jaeger)
> Jesse, What happens with those that have already a broken eeprom?
>

Point is: if it's related to #57976 writing back the eeprom doesn't work.
At the time I attributed this to preproduction hardware.

(In reply to comment #46 from Egbert Eich)
> Point is: if it's related to #57976 writing back the eeprom doesn't work.
> At the time I attributed this to preproduction hardware.

I don't have access to that one... what was the issue? Can you elaborate?

In comment #40, just setting the MAC address in the NVM is most likely not going to restore everything to working. The HW reads a lot of things out it when it's coming up. You will probably have to do a BIOS update to get everything in the system back. You will also have to put the MAC address back in as well. The BIOS update might do this for you so make sure it's right before trying to update.

We would like to know that this works so if you could try it on this system it would help a lot.

I could write the MAC address with ethtool but now the driver do not load completely insmod hangs for about a minute and then it disable the IRQ.
After this here is no eth1 this are the dmesg:
e1000e: Intel(R) PRO/1000 Network Driver - 0.2.0
e1000e: Copyright (c) 1999-2007 Intel Corporation.
ACPI: PCI Interrupt 0000:00:19.0[A] -> GSI 20 (level, low) -> IRQ 20
PCI: Setting latency timer of device 0000:00:19.0 to 64
ACPI: PCI interrupt for device 0000:00:19.0 disabled
e1000e: Intel(R) PRO/1000 Network Driver - 0.2.0
e1000e: Copyright (c) 1999-2007 Intel Corporation.
ACPI: PCI Interrupt 0000:00:19.0[A] -> GSI 20 (level, low) -> IRQ 20
PCI: Setting latency timer of device 0000:00:19.0 to 64
ACPI: PCI interrupt for device 0000:00:19.0 disabled

I think, now with a valid checksum it interprets some of the still 0xFF values as valid and set some registers wrong.

(In reply to comment #47 from Bob Mahar)
> (In reply to comment #46 from Egbert Eich)
> > Point is: if it's related to #57976 writing back the eeprom doesn't work.
> > At the time I attributed this to preproduction hardware.
>
> I don't have access to that one... what was the issue? Can you elaborate?
>

In bug #57976 an SPI type eeprom which seemed to still hold valid content (at least not 0xff) but a bogus checksum could not be restored as no matter to which byte offset a value was written it always ended up at offset 0 or 1.

However looking at comment #49 it doesn't seem to be related as in this case the checksum could be fixed.

Since we do not have a similar machine, we cannot dump the EEProm. The board is a Intel DQ35JO. Maybe someone from Intel has access to this board and can provide a ethtool -e dump ?

I suggest this is severity urgent now.

Intel has just posted patches to lkml [1] [2] [3] that mark the memory mapped EEPROM region as read-only. Therefore if the EEPROM is garbled by any bug in kernel code, after these patches are applied, the EEPROM would no longer be overwritten, and stack trace would be dumped instead, which will hopefully point to the code that is corrupting the memory.

If, however, userspace is corrupting the memory region (most probably X.Org), then this protection is rendered useless, but it still is worth trying so that we can potentially rule out either userspace or kernelspace code completely.

I have built the kernel with these three patches applied, for those who are willing (and able) to test. The kernel RPM can be obtained from

        http://labs.suse.cz/jikos/download/bug-425480/

Any testing would be highly appreciated.

[1] http://lkml.org/lkml/2008/9/23/427
[2] http://lkml.org/lkml/2008/9/23/431
[3] http://lkml.org/lkml/2008/9/23/432

(In reply to comment #52 from Jiri Kosina)
> If, however, userspace is corrupting the memory region (most probably X.Org),
> then this protection is rendered useless, but it still is worth trying so that
> we can potentially rule out either userspace or kernelspace code completely.

In fact, testing whether booting the system only in text-mode (so that xorg won't be started at all) also triggers the bug or not would also be a valuable test.

Patches to the e1000e driver to protect the NVM were posted to netdev a few ours ago. They need to be tried on this problem. Either it will fix the problem or it should point to what is causing the problem. The patches are obviously for the 2.6.27-rc kernels.

Thanks Jiri. Yes these patches need tried. We tested them but we have not been seeing the problem.

Andreas, could you please give us the model and serial number form the system that had the NVM corrupted? We have some people here at Intel that think that they can track down the EEPROM image. They want to see if things are correct with it.

Thanks.

John, my system is a Lenovo thinkpad X61s:
  System Info: #1
    Manufacturer: "LENOVO"
    Product: "766929G"
    Version: "ThinkPad X61s"
    Serial: "L3A2878"
    UUID: undefined, but settable
    Wake-up: 0x06 (Power Switch)
  Board Info: #2
    Manufacturer: "LENOVO"
    Product: "766929G"
    Version: "Not Available"
    Serial: "1ZDMN77215Z"

The e1000 is:
26: PCI 19.0: 0200 Ethernet controller
  [Created at pci.310]
  UDI: /org/freedesktop/Hal/devices/pci_8086_104b
  Unique ID: kpGf.mInfNyjoCrB
  SysFS ID: /devices/pci0000:00/0000:00:19.0
  SysFS BusID: 0000:00:19.0
  Hardware Class: network
  Model: "Intel 82566DC Gigabit Network Connection"
  Vendor: pci 0x8086 "Intel Corporation"
  Device: pci 0x104b "82566DC Gigabit Network Connection"
  SubVendor: pci 0x8086 "Intel Corporation"
  SubDevice: pci 0x0000
  Revision: 0x03

The motherboard from this bug is:

Handle 0x0007, DMI type 2, 20 bytes
Base Board Information
        Manufacturer: Intel Corporation
        Product Name: DQ35JO
        Version: AAD82085-801
        Serial Number: BQJO749006WD

The onboard NIC:
33: PCI 19.0: 0200 Ethernet controller
  [Created at pci.310]
  UDI: /org/freedesktop/Hal/devices/pci_8086_294c
  Unique ID: kpGf.CUCsNZz8jz8
  SysFS ID: /devices/pci0000:00/0000:00:19.0
  SysFS BusID: 0000:00:19.0
  Hardware Class: network
  Model: "Intel 82566DC-2 Gigabit Network Connection"
  Vendor: pci 0x8086 "Intel Corporation"
  Device: pci 0x294c "82566DC-2 Gigabit Network Connection"
  SubVendor: pci 0x8086 "Intel Corporation"
  SubDevice: pci 0x0000
  Revision: 0xfd
  Memory Range: 0x92200000-0x9221ffff (rw,non-prefetchable)
  Memory Range: 0x92224000-0x92224fff (rw,non-prefetchable)
  I/O Ports: 0x3400-0x341f (rw)
  IRQ: 20 (no events)
  Module Alias: "pci:v00008086d0000294Csv00008086sd00000000bc02sc00i00"

The original MAC: 00:1c:c0:2b:74:3a

(In reply to comment #53 from Jiri Kosina)
> In fact, testing whether booting the system only in text-mode (so that xorg
> won't be started at all) also triggers the bug or not would also be a valuable
> test.

It is, unfortunately, not that easy. I have rebooted my machine (hp 2510p) with e1000e 17 times since Sep 15 with 2.6.27-rc5-git9+ Kernels (always pretty recent STABLE) and I did not encounter any problems.
So it is pretty hard to prove the absence of this bug.

(In reply to comment #57 from Stefan Seyfried)
> It is, unfortunately, not that easy. I have rebooted my machine (hp 2510p) with
> e1000e 17 times since Sep 15 with 2.6.27-rc5-git9+ Kernels (always pretty
> recent STABLE) and I did not encounter any problems.
> So it is pretty hard to prove the absence of this bug.

And did this machine expose the problem at least once previously? Apparently not all systems having e1000e hardware are being hit by the issue, either only specific product IDs are affected, or it might be chipset-dependent, etc.

Also, please do not forget to back up contents of your EEPROM before you start playing with this :)

(In reply to comment #49 from Karsten Keil)
> I could write the MAC address with ethtool but now the driver do not load
> completely insmod hangs for about a minute and then it disable the IRQ.
> After this here is no eth1 this are the dmesg:
> e1000e: Intel(R) PRO/1000 Network Driver - 0.2.0
> e1000e: Copyright (c) 1999-2007 Intel Corporation.
> ACPI: PCI Interrupt 0000:00:19.0[A] -> GSI 20 (level, low) -> IRQ 20
> PCI: Setting latency timer of device 0000:00:19.0 to 64

at this point (without trying to activate the device) does ethtool -e still
work? I would assume not.

> ACPI: PCI interrupt for device 0000:00:19.0 disabled

I looked at your ethregs dump (thank you!!!) and in the EECD register, bit 8 is
not set, indicating the valid bits in the eeprom are not set.
bit 9 is set indicating the hardware tried to read the eeprom.
bit 22 is only valid if bit 8 and 9 is set, but it would indicate which of the
two eeprom banks had a valid signature.

I'm curious if the other bank on the eeprom might still be okay. I'll have to
figure out tomorrow if we can switch to the other bank. I may be able to get
you some internal tools since this is an intel board, I'll have to see what is
available.

BTW this is the first desktop machine I've heard of that reported the problem.

(In reply to comment #58 from Jiri Kosina)
> And did this machine expose the problem at least once previously? Apparently
> not all systems having e1000e hardware are being hit by the issue, either only
> specific product IDs are affected, or it might be chipset-dependent, etc.

I am not sure. See comment #32. It might of course also just have been a broken joint on the mainboard.

> Also, please do not forget to back up contents of your EEPROM before you start
> playing with this :)

If it hits me the same as last time, this won't help :) (and yes, i backed it up)

(In reply to comment #59 from Jesse Brandeburg)
> BTW this is the first desktop machine I've heard of that reported the problem.
Regarding the desktop: it also has intel integrated graphics, and it had recurring problems with the graphics driver (lockups) before the ethernet broke. Maybe that's one common factor.

(In reply to comment #52 from Jiri Kosina)
> Intel has just posted patches to lkml [1] [2] [3] that mark the memory mapped
> EEPROM region as read-only. Therefore if the EEPROM is garbled by any bug in
> kernel code, after these patches are applied, the EEPROM would no longer be
> overwritten, and stack trace would be dumped instead, which will hopefully
> point to the code that is corrupting the memory.
>

This is indeed a valuable test.

> If, however, userspace is corrupting the memory region (most probably X.Org),
> then this protection is rendered useless, but it still is worth trying so that
> we can potentially rule out either userspace or kernelspace code completely.

Not necessarily. If X overwrites this memory from user space, yes.
However if it is overwritten from kernel space (by DRM - either
from the Xserver or from a DRM client) we will be able to catch it.

For now I would rule out user space. From user space the Xserver
cannot access memory unless it is explicitly mapped. You can find
out the mapped memory ranges from /proc/<pid>/maps. It would be
instructive to know which ranges show up there on the affected
machines and compare them to an lspci -v output.
I may have missed this, but I have not seen any analysis which
access method is used on the affected systems to write to the
EEPROM.

(In reply to comment #59 from Jesse Brandeburg)
Yes ethtool does not work any more.
What I saw while I was programming the MAC was, that a other word in th EEProm
changed as well (I assume the checksum) and one other byte also get some other
value (maybe BF, but I'm not sure) it was in an other line of the ethtool -e dump (2. or 3. line) between MAC (first 6 bytes) and the checksum.

(In reply to comment #63 from Egbert Eich)
> For now I would rule out user space. From user space the Xserver
> cannot access memory unless it is explicitly mapped. You can find
> out the mapped memory ranges from /proc/<pid>/maps. It would be
> instructive to know which ranges show up there on the affected
> machines and compare them to an lspci -v output.

It could be some temporary mapping that goes away after a while, so that it doesn't show in /proc/<pid>/maps permanently, but yes, this of course can be tried.

Could please someone, who has access to affected hardware, provide output of

       cat /proc/`pidof Xorg`/maps
       lspci -v

commands, so that we can see if there is possibly some lethal overlap?

Also, the locking in e1000e seems indeed to be dodgy, Thomas Gleixner reported this to be spotted by lockdep on his system with 2.6.27-rc7:

e1000e: Intel(R) PRO/1000 Network Driver - 0.3.3.3-k2
e1000e: Copyright (c) 1999-2008 Intel Corporation.
e1000e 0000:00:19.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
e1000e 0000:00:19.0: setting latency timer to 64
0000:00:19.0: eth0: (PCI Express:2.5GB/s:Width x1) 00:15:58:84:9f:94
0000:00:19.0: eth0: Intel(R) PRO/1000 Network Connection
0000:00:19.0: eth0: MAC: 4, PHY: 6, PBA No: ffffff-0ff
------------[ cut here ]------------
WARNING: at /home/tglx/work/kernel/git/linux-2.6/kernel/mutex.c:135 mutex_lock_nested+0x5c/0x26d()
Modules linked in: e1000e i915 drm ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp bnep rfcomm l2cap bluetooth autofs4 coretemp fuse sunrpc ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi scsi_transport_iscsi e1000 cpufreq_ondemand acpi_cpufreq ext2 dm_mirror dm_log dm_multipath dm_mod ipv6 kvm_intel kvm snd_hda_intel snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq arc4 snd_seq_device snd_pcm_oss ecb snd_mixer_oss snd_pcm video crypto_blkcipher snd_timer snd_page_alloc iwlagn i2c_i801 i2c_core firewire_ohci iwlcore mac80211 snd_hwdep firewire_core crc_itu_t iTCO_wdt iTCO_vendor_support rtc_cmos snd soundcore output ac battery pcspkr cfg80211 sr_mod thinkpad_acpi rfkill cdrom sg hwmon button joydev ata_piix ahci libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
Pid: 3484, comm: ip Not tainted 2.6.27-rc7-00006-gcec5eb7-dirty #89

Call Trace:
 <IRQ> [<ffffffff8103654d>] warn_on_slowpath+0x51/0x77
 [<ffffffff810572e9>] __lock_acquire+0x6ad/0x716
 [<ffffffff8129c285>] mutex_lock_nested+0x5c/0x26d
 [<ffffffffa04f85c2>] e1000_acquire_swflag_ich8lan+0x59/0x74 [e1000e]
 [<ffffffffa04fd753>] e1000e_read_kmrn_reg+0x18/0x62 [e1000e]
 [<ffffffffa04f8606>] e1000e_gig_downshift_workaround_ich8lan+0x29/0x71 [e1000e]
 [<ffffffffa0503e07>] e1000_intr_msi+0x46/0xec [e1000e]
 [<ffffffff81076fa5>] handle_IRQ_event+0x1e/0x51
 [<ffffffff81078295>] handle_edge_irq+0xe8/0x12b
 [<ffffffffa04fb312>] e1000e_update_mc_addr_list_generic+0x0/0x18e [e1000e]
 [<ffffffff8100ea88>] do_IRQ+0x6c/0xd4
 [<ffffffff8100c556>] ret_from_intr+0x0/0xf
 <EOI> [<ffffffffa04fb312>] e1000e_update_mc_addr_list_generic+0x0/0x18e [e1000e]
 [<ffffffffa04fb38f>] e1000e_update_mc_addr_list_generic+0x7d/0x18e [e1000e]
 [<ffffffffa04fb359>] e1000e_update_mc_addr_list_generic+0x47/0x18e [e1000e]
 [<ffffffffa0500ace>] e1000_set_multi+0xe2/0x11b [e1000e]
 [<ffffffff8121a1e8>] dev_set_rx_mode+0x21/0x2d
 [<ffffffff8121d1a6>] dev_open+0x85/0x9e
 [<ffffffff8121b172>] dev_change_flags+0xa6/0x15d
 [<ffffffff81262588>] devinet_ioctl+0x242/0x58a
 [<ffffffff812109a5>] sock_ioctl+0x1d8/0x1ff
 [<ffffffff810b80a9>] vfs_ioctl+0x21/0x6b
 [<ffffffff810b834c>] do_vfs_ioctl+0x259/0x272
 [<ffffffff81055a14>] trace_hardirqs_on_caller+0xf2/0x115
 [<ffffffff810b83b6>] sys_ioctl+0x51/0x73
 [<ffffffff8100bf4b>] system_call_fastpath+0x16/0x1b

Haven't looked into the code yet to see if this cou...

(In reply to comment #65 from Jiri Kosina)
> It could be some temporary mapping that goes away after a while, so that it
> doesn't show in /proc/<pid>/maps permanently, but yes, this of course can be
> tried.

The Xserver itself doesn't have any temporary mappings. It could be buffers
requested from DRM which (depending on the implementation) could be requested and discarded during runtime of the server.

Created attachment 241376
ethtool -e eth0 > e1000e.txt

Requested by Karsten Keil (comment #51).

Hardware: Laptop Lenovo Thinkpad T61

$ uname -a
Linux mandachuva 2.6.26.2 #1 SMP Sat Aug 16 19:08:09 BRT 2008 i686 GNU/Linux

$lspci -vvv

00:19.0 Ethernet controller: Intel Corporation 82566MM Gigabit Network Connection (rev 03)
        Subsystem: Lenovo Device 20b9
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 216
        Region 0: Memory at fe200000 (32-bit, non-prefetchable) [size=128K]
        Region 1: Memory at fe225000 (32-bit, non-prefetchable) [size=4K]
        Region 2: I/O ports at 1840 [size=32]
        Capabilities: <access denied>
        Kernel driver in use: e1000e
        Kernel modules: e1000e

Someone try this patchs from Jeff Kirsher (Intel)?
http://lkml.org/lkml/2008/9/23/427
http://lkml.org/lkml/2008/9/23/431
http://lkml.org/lkml/2008/9/23/432

And I think that is a good idea change priority and severity to higher, because this bug can DAMAGED a hardware.

Best regards,
Renato

Vladimir Botka just posted this to an internal mailing list:
--------------------------------
So, I am the one who tried (without an intention). The installation of
SLED11 Beta1 on TP T61 crashed in the moment installer was probing the
X configuration. Instead of the well known notification "Dont panic ..."
an error message appeared for a few seconds telling something about
"yast, proposal, etc ...". Then the machine restarted. The network card
stop responding.

Here is the card info:

00:19.0 Ethernet controller: Intel Corporation 82566DC Gigabit Network
Connection (rev 03) Subsystem: Intel Corporation Device 0000
        Flags: fast devsel, IRQ 20
        Memory at fe000000 (32-bit, non-prefetchable) [size=128K]
        Memory at fe025000 (32-bit, non-prefetchable) [size=4K]
        I/O ports at 1840 [size=32]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable- Kernel modules: e1000e
---------------------------------

Given that this is increasingly looking like it's closely related to video,
can people with affected machines please post the lspci output for their gfx
chip?

Olaf, I really don't fell confortable to test Kernel 2.6.27-rc if it can damaged my ethernet device, so I don't know if my hardware is affected, but my video card is:

01:00.0 VGA compatible controller: nVidia Corporation Quadro NVS 140M (rev a1)
        Subsystem: Lenovo Device 20d8
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at d6000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at d4000000 (64-bit, non-prefetchable) [size=32M]
        Region 5: I/O ports at 2000 [size=128]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nvidia, nvidiafb

And my ethernet device is listed in Comment #68

Best regards,
Renato

kernel-2.6.27-0.352.rc7.git1.fc10 (http://koji.fedoraproject.org/koji/buildinfo?buildID=64060) includes a fix for e1000 and (temporarily) disables e1000e.

This is probably sufficient for F10Beta (pending some regression testing)

I guess that will work, but you've now killed the wired network on quite a few hardware platforms. Pulling the patches from comment #20 would probably be better for F10Beta.

The Ethernet on this Thinkpad R400 is already toasted, after an installation attempt some time ago (Micha, do you still know when you tried it?)

00:19.0 Ethernet controller: Intel Corporation 82566DC-2 Gigabit Network Connection (rev 03)
        Subsystem: Intel Corporation Device 0000
        Flags: fast devsel, IRQ 218
        Memory at fc000000 (32-bit, non-prefetchable) [size=128K]
        Memory at fc024000 (32-bit, non-prefetchable) [size=4K]
        I/O ports at 1820 [size=32]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+ Count=1/1 Enable+
        Capabilities: [e0] PCIe advanced features <?>
        Kernel modules: e1000e

00:02.0 VGA compatible controller: Intel Corporation Mobile 4 Series Chipset Integrated Graphics Controller (prog-if 00 [VGA controller])
        Subsystem: Lenovo Device 20e4
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at f4400000 (64-bit, non-prefetchable) [size=4M]
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 1800 [size=8]
        Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Count=1/1 Enable-
        Capabilities: [d0] Power Management version 3

Is all hardware which currently know to be affected by the bug driven by i915 DRM driver? ("dmesg | grep -i drm" should be enough to learn that).

(In reply to comment #71 from Renato Yamane)
> Olaf, I really don't fell confortable to test Kernel 2.6.27-rc if it can
> damaged my ethernet device, so I don't know if my hardware is affected, but my
> video card is:
>
> 01:00.0 VGA compatible controller: nVidia Corporation Quadro NVS 140M (rev a1)
> Subsystem: Lenovo Device 20d8
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-

Definitely not.
And if this system does show to be affected it would point away from the gfx driver.

But all other have i915.

I would also be really interested to know whether the bug triggers when booting with 'nopat' kernel option, as that's other piece that might go wrong on the way between Xorg, graphics card, ethernet card and MMIO.

And so far I understand comment #71, Renatos machine is not affected yet, only on the list of potential victims of NICs with writable FLASH.

So yes upto now all affected machines are used the 915 DRM driver.

My laptop uses the 915 DRM:
dmesg | grep -i drm
[drm] Initialized drm 1.1.0 20060810
[drm] Initialized i915 1.6.0 20060119 on minor 0

# hwinfo --gfxcard
27: PCI 02.1: 0380 Display controller
  [Created at pci.310]
  UDI: /org/freedesktop/Hal/devices/pci_8086_2a03
  Unique ID: ruGf.a6pkzICrUB2
  SysFS ID: /devices/pci0000:00/0000:00:02.1
  SysFS BusID: 0000:00:02.1
  Hardware Class: graphics card
  Model: "Intel Mobile GM965/GL960 Integrated Graphics Controller"
  Vendor: pci 0x8086 "Intel Corporation"
  Device: pci 0x2a03 "Mobile GM965/GL960 Integrated Graphics Controller"
  SubVendor: pci 0x17aa "Lenovo"
  SubDevice: pci 0x20b5
  Revision: 0x0c
  Memory Range: 0xf8200000-0xf82fffff (rw,non-prefetchable)
  Module Alias: "pci:v00008086d00002A03sv000017AAsd000020B5bc03sc80i00"
  Config Status: cfg=no, avail=yes, need=no, active=unknown

28: PCI 02.0: 0300 VGA compatible controller (VGA)
  [Created at pci.310]
  UDI: /org/freedesktop/Hal/devices/pci_8086_2a02
  Unique ID: _Znp.3gR64TvADaC
  SysFS ID: /devices/pci0000:00/0000:00:02.0
  SysFS BusID: 0000:00:02.0
  Hardware Class: graphics card
  Model: "Intel 965 GM"
  Vendor: pci 0x8086 "Intel Corporation"
  Device: pci 0x2a02 "965 GM"
  SubVendor: pci 0x17aa "Lenovo"
  SubDevice: pci 0x20b5
  Revision: 0x0c
  Memory Range: 0xf8100000-0xf81fffff (rw,non-prefetchable)
  Memory Range: 0xe0000000-0xefffffff (rw,prefetchable)
  I/O Ports: 0x1800-0x1807 (rw)
  IRQ: 16 (2124 events)
  I/O Ports: 0x3c0-0x3df (rw)
  Module Alias: "pci:v00008086d00002A02sv000017AAsd000020B5bc03sc00i00"
  Driver Info #0:
    XFree86 v4 Server Module: intel
  Driver Info #1:
    XFree86 v4 Server Module: intel
    3D Support: yes
    Extensions: dri
  Config Status: cfg=no, avail=yes, need=no, active=unknown

Primary display adapter: #28

I had the problem on my T60p. It does not appear to use the 915 DRM.

hwinfo --gfxcard
11: PCI 100.0: 0300 VGA compatible controller (VGA)
  [Created at pci.318]
  UDI: /org/freedesktop/Hal/devices/pci_1002_71c4
  Unique ID: VCu0.s+GMGFf+eZ0
  Parent ID: vSkL.rxAOeWuq8i6
  SysFS ID: /devices/pci0000:00/0000:00:01.0/0000:01:00.0
  SysFS BusID: 0000:01:00.0
  Hardware Class: graphics card
  Model: "Lenovo ThinkPad T60p"
  Vendor: pci 0x1002 "ATI Technologies Inc"
  Device: pci 0x71c4 "Mobility FireGL V5200"
  SubVendor: pci 0x17aa "Lenovo"
  SubDevice: pci 0x2007 "ThinkPad T60p"
  Memory Range: 0xd0000000-0xdfffffff (rw,prefetchable)
  I/O Ports: 0x2000-0x2fff (rw)
  Memory Range: 0xee100000-0xee10ffff (rw,non-prefetchable)
  Memory Range: 0xee120000-0xee13ffff (ro,prefetchable,disabled)
  IRQ: 11 (no events)
  I/O Ports: 0x3c0-0x3df (rw)
  Module Alias: "pci:v00001002d000071C4sv000017AAsd00002007bc03sc00i00"
  Driver Info #0:
    XFree86 v4 Server Module: radeonhd
  Config Status: cfg=no, avail=yes, need=no, active=unknown
  Attached to: #27 (PCI bridge)

Primary display adapter: #11

I guess I really don't know for sure if I have the same problem. I did not inspect the contents of the EEPROM using ethtool, so I really can't verify. I have subsequently flashed my BIOS to recover.

What I experienced was a blank screen upon rebooting my system after installing SLED 11 Beta - build47. I was unable to install anything after that (SLED 10, XP, etc.). I continued to get a blank screen during installation. After updating my BIOS using Lenovo's bootable BIOS CD I was able to install build48 (without loading the e1000e module).

That would be the first case when this would be reported to happen on non-i915 hardware.

On the other hand, this mail that came to LKML just a while ago is quite interesting too:

           http://lkml.org/lkml/2008/9/24/133

apparently some realtek ethernet device stopped working, because it has a lots of 0xff somewhere in its configuration space ....

I think the problem with T60p is different (it has a different lan chip 82573 with a real eeprom (not NVM based) that should not be able to be corrupted in the same manner as 82566/82567.

let's leave Jared's problem off to the side as a (possibly) new issue, maybe a new bug if it is reproducible?

(In reply to comment #83 from Jesse Brandeburg)
> I think the problem with T60p is different (it has a different lan chip 82573
> with a real eeprom (not NVM based) that should not be able to be corrupted in
> the same manner as 82566/82567.

Oh, if it were only that simple. The T60p has the 82573L c.f. the Intel docs...

http://download.intel.com/design/network/products/LAN/manuals/316080.pdf

See section 2.3... "The 82573E/82573V/82573L supports both FLASH memory and EEPROM; however, only one device can be connected at a time (not both).

So while the 8257x's for the most part SEEPROMs, the -E,V,L suffixed part could go either way - oh joy!

Forgive me this dumb question - if this is due to an accidental overwrite with random data (DMA), why would the EEPROM contain only FFs afterwards? Can't we infer something from that?

(In reply to comment #85 from Martin Wilck)
> Forgive me this dumb question - if this is due to an accidental overwrite with
> random data (DMA),

It's not DMA but MMIO. The data are not that random, it is really all 0xff.

> why would the EEPROM contain only FFs afterwards? Can't we infer something from > that?

Well, we weren't able to use this to identify source of the corruption so far. We have patches that could help to point to the guilty, but first we need reliable way to restore the EEPROM contents, otherwise the debugging is almost impossible.

Looking at the lspci output from the system mentioned on LKML (comment #82)
that machine seems to have an i945G graphics controller, which AFAIK is
also driven by the i915 driver. The chipset is ICH7.

Has anyone tried to resurrect AJ's laptop via a BIOS update?

Is there any way Intel can help resuscitate these e1000e NVMs - this is
really preventing us from doing further debugging.

Olaf, I did a BIOS update to the latest Lenovo BIOS. It did not help at all.

Jesse, I'm setting this bug as NEEDINFO to you.
The biggest roadblock right now is our inability to bring those dead
NICs back to life. Without this, we cannot proceed with testing, and
we are somewhat reluctant to try this ourselves, as it seems someone at
RedHat has bricked a laptop this way.

We tried a BIOS update on one of the affected laptops, but this didn't
help. And since we weren't aware of the problem in advance, we don't
have an ethregs dump of these.

So can you please get someone from Intel to help us with restoring the
NVM to a working condition? Thanks a lot!

I realize that my issue might be something different (see comment #83). However, updating my BIOS seemed to bring my bricked T60p back to life.

One more question to Intel.

There's a question whether the NVM we're talking about here is actually larger,
and is used by components other than the e1000e. If for instance the video BIOS
maps all of the NVM and, due to some bug, scribbles over parts of it that
include the e1000e's config space - is there a way to verify this?

Update:
Since numerous people have seen this problem during installation when the Xserver is probed I've looked into what's happening at this stage:
The Xserver is started with a standard config - only the line containing the bus id and driver name is special. The installation program then connects to the xserver to obtain the randr version and information about the available outputs. Nothing is ever drawn (except for the standard X background.
For now I would also rule out drm as it is initialized but never used (2d operations don't do use drm on this driver).
The probing scenario during installation can be reproduced on any running system with the command:
sysp -s xstuff
I'm currently condensing down the part that involves the X connection of this for better reproduction.
My goal is to narrow down where to look. The X driver is still a considerable chunk of code so it would be beneficial to reduce the possible sources of the problem.

(In reply to comment #91 from Olaf Kirch)
> One more question to Intel.
>
> There's a question whether the NVM we're talking about here is actually larger,
> and is used by components other than the e1000e. If for instance the video BIOS
> maps all of the NVM and, due to some bug, scribbles over parts of it that
> include the e1000e's config space - is there a way to verify this?
>

I don't think this is the case: the driver only maps the POSTed copy of the VBIOS. This is copied into RAM at POST time (to the 0xC-segment). This copy is then made read only. This copy is (should be) entirely independent of the EEPROM containing the PCI ROM.

Another question came up: does this happen on both 64 and 32 bit installations?

(In reply to comment #89 from Olaf Kirch)
> Jesse, I'm setting this bug as NEEDINFO to you.
> The biggest roadblock right now is our inability to bring those dead
> NICs back to life. Without this, we cannot proceed with testing, and
> we are somewhat reluctant to try this ourselves, as it seems someone at
> RedHat has bricked a laptop this way.

We'll get you a utility today to help with this, and at the same time we're working on a quick hack to the driver to take in an ethtool eeprom dump and push it back to the NVM. We hope to have that done and working today.

> We tried a BIOS update on one of the affected laptops, but this didn't
> help. And since we weren't aware of the problem in advance, we don't
> have an ethregs dump of these.

so it depends on whether the BIOS version has the LAN part included. Some bios versions do, and some do not. I know that in particular there were a couple versions of the bioses for the X60/T60 line that had LAN NVM updates.

> So can you please get someone from Intel to help us with restoring the
> NVM to a working condition? Thanks a lot!

We are working on it, a couple of different avenues.

(In reply to comment #91 from Olaf Kirch)
> There's a question whether the NVM we're talking about here is actually larger,
> and is used by components other than the e1000e. If for instance the video BIOS
> maps all of the NVM and, due to some bug, scribbles over parts of it that
> include the e1000e's config space - is there a way to verify this?

the NVM in question is a single part that the entire machine (VGA, BIOS, LAN, Manageability, AHCI, etc) all use.

I couldn't tell you how to verify if something else is mapping over the top of the LAN area of the NVM. The only reports I've heard are that the LAN NVM is corrupted. If you managed to corrupt the BIOS area, the machine wouldn't boot.

(In reply to comment #94 from Egbert Eich)
> Another question came up: does this happen on both 64 and 32 bit installations?

At this point we don't know. At least one reported I worked with was running 32 bit.

(In reply to comment #96 from Jesse Brandeburg)

> The only reports I've heard are that the LAN NVM is
> corrupted. If you managed to corrupt the BIOS area, the machine wouldn't boot.

Helmut Schaa has an HP 2510p that lost some of its display modes after a hard X crash on an early 2.6.27-rc kernel (it now no longer knows that it has a 1280x800 panel but thinks that it only has 1024x768, the BIOS screen is in the upper left corner instead of centered on the screen). Even though we don't know that this is the same problem, it shows that sh*t happens.

Do we have any dumps of the gfx related crashes? Comment #98 seems to indicate that the video ROM may have also become corrupted (either that or the EEPROM containing the EDID), but I don't currently have any theories about how the gfx driver could cause that...

> And I think that is a good idea change priority and severity to higher,
> because this bug can DAMAGED a hardware.

Nobody is changing priority and severity because those fields are meaningless. We should really remove those fields from the interface.

please see my message on lkml titled "e1000e NVM corruption issue status"