host protected area prevents detection of isw (intel matrix storage manager) raid arrays

lspci details

fdisk output:
$ sudo fdisk -u -l /dev/sda

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x00000000

(note that the signature is exactly 2115 sectors back from the end of the disk).

It output
/dev/sdb: isw, "isw_cahfcbgdcf", GROUP, ok, 1953523053 sectors, data@ 0
/dev/sda: isw, "isw_cahfcbgdcf", GROUP, ok, 1953523053 sectors, data@ 0

to the console, and the attached subdirectory.

Just for clarity - with my patch I've successfully installed Ubuntu, though it wants me to manually dmraid -ay on boot; I'll track that down later.

What ICH version are you using?
(couldnt read it from your lspci)

00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller

If I read that correctly its a ICH10R. If there is somewhere else I should look, let me know and I'll go do so.

Yep looks that way.
I'm only on ICH8R, ICH10 is the latest Intel ICH controller from Q4 last year. (according to their datasheets)

Its odd that they would have moved the metadata, I'm fairly sure the RAID sets are compatible across versions of intel matrix raid. I can't find any reference to the metadata locations in the datasheet.

In the location where the metadata should be is there any reference to the area where yours is located?
(could try DD dumping it, might make for a better detection routine)

My guess is they have added more data/larger journal space - something like that, and that older formats will be read by the new controller but not vice verca. I'll poke around there tomorrow.

I've added some instrumentation and made my printf a log entry. dmraid -rd now shows:

DEBUG: not isw at 1000204884992
DEBUG: isw metadata found at 1000203803136 from probe at 1000203803136

DEBUG: not isw at 1000204884992
DEBUG: isw metadata found at 1000203803136 from probe at 1000203803136

Is the output.

Concretely, isw formats appear to be meant to have a signature 1K back (2 sectors) from end of drive. This can have an extended data block which can point to other areas etc etc.

However, for my drive the signature is simply not found (DEBUG: not isw at 1000204884992), which means it doesn't move onto doing extended block processing or whatever.

I've also updated the patch to be (somewhat) less hardcoded - it now jumps 2115 sectors back so may work for other people with 'no RAID Drives found'.

My changes add no compile warnings.

This is the dd of the last 1K of disk (which is where the pre-existing isw search code would look). As you can see its mainly NULL's but there is a suspicious ICH10 late in the piece.

hexdump too:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 24 47 41 46 52 10 41 08 00 00 00 00 00 00 00 00 |$GAFR.A.........|
00000210 00 00 00 00 00 00 00 00 00 00 6f 65 70 74 00 00 |..........oept..|
00000220 00 00 00 08 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000300 22 31 32 2f 33 30 2f 32 30 30 38 2d 58 35 38 2d |"12/30/2008-X58-|
00000310 49 43 48 31 30 2d 37 41 38 39 51 47 30 41 43 2d |ICH10-7A89QG0AC-|
00000320 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

2113 in hex is 841, so the 0x4108 at bytes 0x206-0x207 (=518-519) could be a dereference (-2113 sectors from here) except that it should be 842 as those bytes are in the -1 sector, not the -2.

If so, then this is definitely a new ISW metadata format; I'd have to guess that the $GAFR is a new signature for it Note too that the first sector is totally blank, it appears to only need the last sector now in terms of bootstrapping.

I can whip up a generic probe based on this, but whats mostly likely to be accepted into the package.

You could do a probe for the ICH10 tag for now.
That would probably be safer than guessing a dereference, then we need some more ICH10R based people to test which will take time.

You could try building another RAID set with a couple of other disks and see if the meta is in the same place on those disks also.

On Sat, 2009-05-09 at 10:57 +0000, danwood76 wrote:
> You could do a probe for the ICH10 tag for now.
> That would probably be safer than guessing a dereference, then we need some more ICH10R based people to test which will take time.
>
> You could try building another RAID set with a couple of other disks and
> see if the meta is in the same place on those disks also.

I think the ICH10 tag is actually a serial or some such; its at an odd
spot there.

What do you think of treating GAFR as a magic bytes, then using the
sector offset following that?

As long as the final location is checked for still being > 0 we won't
try to read negative offsets, and the followup is_isw call will fail on
non-isw environments.

It seems pretty safe to me to assume that:
find GAFR
follow a word sector offset
find a 10 byte isw signature

is only going to occur when someone actually has a ICH environment.

re testing: I don't have any disks I can wipe I'm sorry.

-Rob

Robert Collins wrote:
> On Sat, 2009-05-09 at 10:57 +0000, danwood76 wrote:
>
>> You could do a probe for the ICH10 tag for now.
>> That would probably be safer than guessing a dereference, then we need some more ICH10R based people to test which will take time.
>>
>> You could try building another RAID set with a couple of other disks and
>> see if the meta is in the same place on those disks also.
>>
>
> I think the ICH10 tag is actually a serial or some such; its at an odd
> spot there.
>
> What do you think of treating GAFR as a magic bytes, then using the
> sector offset following that?
>
> As long as the final location is checked for still being > 0 we won't
> try to read negative offsets, and the followup is_isw call will fail on
> non-isw environments.
>
> It seems pretty safe to me to assume that:
> find GAFR
> follow a word sector offset
> find a 10 byte isw signature
>
> is only going to occur when someone actually has a ICH environment.
>
> re testing: I don't have any disks I can wipe I'm sorry.
>
> -Rob
>
>
Sounds good, if someone else with ICH10 can test it would be great.
Write a patch based on your idea and I will test it to verify it still
works with ICH8 also but I obviously cannot test the ICH10.

regards,
Dnany

On Sun, 2009-05-10 at 13:13 +0000, danwood76 wrote:

> Sounds good, if someone else with ICH10 can test it would be great.
> Write a patch based on your idea and I will test it to verify it still
> works with ICH8 also but I obviously cannot test the ICH10.

Thanks, shall do.

-Rob

This updated patch is ready for testing; it uses the gafr string as a signature, no build warnings added, finds my metadata using that on both drives.

Of course, it could be coincidence, but then finding some users with the same issue will be needed. I'm going to post to ubuntu-devel once I build this in my ppa.

Hmm, I still need to bounds-cap the read call. I'll do that tomorrow, I think its unlikely to cause issues in practice but safe >> sorry.

I have the same setup on one of the drives in my ich10 raid. Looking into it, it seems to be a bios recovery feature for gigabyte motherboards, which apparently uses the ata HPA, which suggested bug#219393, which says to add "libata.ignore_hpa=0" to the kernel command line, which fixes the problem for me.

(for reference, my motherboard is a ga-ex58-ud3r, and last 1k of the affected drive looks like this (offsets from beginning of intel raid signature):
00108400: 2447 4146 5210 4108 0000 0000 0000 0000 $GAFR.A.........
00108410: 0000 0000 0000 0000 0000 3f5b 381d 0000 ..........?[8...
00108420: 0000 0008 0000 0000 0000 0000 0000 0000 ................
.....
00108500: 2231 322f 3330 2f32 3030 382d 5835 382d "12/30/2008-X58-
00108510: 4943 4831 302d 3741 3839 5147 3041 432d ICH10-7A89QG0AC-
00108520: 001f d000 00af a95f 4445 4347 5f03 0010 ......._DECG_...
00108530: 0000 0000 0000 0000 0000 0000 0000 0000 ................

where 3f 5b 38 1d = 0x1d385b3f is the sector 2 sectors after the isw raid metadata)

This isnt the HPA bug.
This is actually the metadata being moved to a different location so dmraid cant find it.

> This is actually the metadata being moved to a different location so dmraid cant find it.

Right, the metadata is being moved by the HPA, so when the kernel disables HPA, dmraid can't find the metadata any more.

With default settings (Ubuntu 9.04, 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03 UTC 2009 x86_64 GNU/Linux since I didn't specify before), my drives both look like this:
---
$> sudo fdisk -l -u /dev/sda

Disk /dev/sda: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders, total 490234752 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0xb782b782

   Device Boot Start End Blocks Id System
/dev/sda1 * 63 490207409 245103673+ 7 HPFS/NTFS
----

and dmraid can only find the metadata on /dev/sdb

With 'libata.ignore_hpa=0' in the kernel command line, I get

---
$ sudo fdisk -l -u /dev/sda

Disk /dev/sda: 250.9 GB, 250999111168 bytes
255 heads, 63 sectors/track, 30515 cylinders, total 490232639 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0xb782b782

   Device Boot Start End Blocks Id System
/dev/sda1 * 63 490207409 245103673+ 7 HPFS/NTFS

$ sudo fdisk -l -u /dev/sdb

Disk /dev/sdb: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders, total 490234752 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0xb782b782

   Device Boot Start End Blocks Id System
/dev/sdb1 * 63 490207409 245103673+ 7 HPFS/NTFS
---

with 2113 fewer total blocks on /dev/sda, and dmraid finds the metadata on both disks.

> with 2113 fewer total blocks on /dev/sda, and dmraid finds the metadata on both disks.

sorry, 2113 fewer total sectors, not blocks

On Mon, 2009-05-11 at 20:25 +0000, 3b wrote:
> > This is actually the metadata being moved to a different location so
> dmraid cant find it.

> with 2113 fewer total blocks on /dev/sda, and dmraid finds the metadata
> on both disks.

So, I may have been writing an hpa interpreter.

Could you try with my patch, with and without 'libata.ignore_hpa=0' ?

Thanks,
Rob

Sure, if i can figure out how to build it :)

The version in https://bugs.launchpad.net/ubuntu/+source/dmraid/+bug/372170/comments/20 seems to be missing changes to isw.h though: ISW10_* and struct isw10 aren't defined that I could see.

On Mon, 2009-05-11 at 23:10 +0000, 3b wrote:
> Sure, if i can figure out how to build it :)

apt-get source dmraid
apt-get build-dep dmraid
wget
http://launchpadlibrarian.net/26558415/17_fix_isw_raid_detection_1TB.patch
cd dmraid-1.0.0.rc15 (I think, you may have a slightly different name)
export QUILT_PATCHES=$(pwd)/debian/patches
quilt import -p0 ../17_fix_isw_raid_detection_1TB.patch (I think, you
may need -p1)
dpkg-buildpackage -us -uc -b

now you should have a couple of ../libdmraid*.deb files. sudo dpkg -i
the one without -dev in its name.

then dmraid -r -d.

Cheers,
Rob

OK, looks like the patch is incomplete:

...

Applying patch 17_fix_isw_raid_detection_1TB.patch
patching file 1.0.0.rc15/lib/format/ataraid/isw.c
Hunk #1 succeeded at 300 (offset -7 lines).
Hunk #2 succeeded at 497 (offset -7 lines).
Hunk #3 succeeded at 516 (offset -7 lines).
Hunk #4 succeeded at 535 (offset -7 lines).
Hunk #5 succeeded at 548 (offset -7 lines).

...

        gcc -c -I. -I../include -I../lib -g -O2 -O2 -DDMRAID_NATIVE_LOG -DHAVE_G
ETOPTLONG -fPIC -Wall -Wundef -Wcast-align -Wwrite-strings -Winline -DDMRAID_TES
T -O2 -D_LARGEFILE64_SOURCE -O2 -DDMRAID_NATIVE_LOG -DHAVE_GETOPTLONG -fPIC -Wal
l -Wundef -Wcast-align -Wwrite-strings -Winline -DDMRAID_TEST -O2 -D_LARGEFILE64
_SOURCE format/ataraid/isw.c -o format/ataraid/isw.o
format/ataraid/isw.c: In function ‘isw_try_10’:
format/ataraid/isw.c:569: warning: implicit declaration of function ‘ISW_10_CONFIGOFFSET’
format/ataraid/isw.c:571: error: ‘ISW10_SIGNATURE_SIZE’ undeclared (first use in this function)
format/ataraid/isw.c:571: error: (Each undeclared identifier is reported only once
format/ataraid/isw.c:571: error: for each function it appears in.)
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: ‘ISW10_SIGNATURE’ undeclared (first use in this function)
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:571: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:575: error: dereferencing pointer to incomplete type
format/ataraid/isw.c:578: warning: implicit declaration of function ‘ISW_SECTOR_TO_OFFSET’
make[2]: *** [format/ataraid/isw.o] Error 1

quilt fail :(. This has the isw.h changes needed.

Tested dmraid from karmic with Robert's patch included on my ICH9 controller, and everything works fine.

My server has ICH10, since it was built last year, and likely has HPA stuff I could enable, however I am not about to take it offline and plug a pair of spare disks in to test unless there is an absolute need to do so.

seems to work:
---
unpatched dmraid, without ignore_hpa=0:
$ sudo dmraid -r -d
/dev/sdb: isw, "isw_ddiggddhgi", GROUP, ok, 490234750 sectors, data@ 0

---

unpatched, with ignore_hpa=0:
$ sudo dmraid -r -d
/dev/sdb: isw, "isw_ddiggddhgi", GROUP, ok, 490234750 sectors, data@ 0
/dev/sda: isw, "isw_ddiggddhgi", GROUP, ok, 490232637 sectors, data@ 0

---

patched dmraid, without ignore_hpa=0:
$ sudo dmraid -r -d
DEBUG: not isw at 203928108032
DEBUG: Found isw 10 gafr signature.
DEBUG: isw 10 sector offset calculated at 2115.
DEBUG: not isw at 203927026176
DEBUG: isw trying hard coded -2115 offset.
DEBUG: not isw at 203927026176
DEBUG: isw metadata found at 251000192000 from probe at 251000192000

DEBUG: not isw at 251000192000
DEBUG: Found isw 10 gafr signature.
DEBUG: isw 10 sector offset calculated at 2115.
DEBUG: isw metadata found at 250999110144 from probe at 250999110144

/dev/sdb: isw, "isw_ddiggddhgi", GROUP, ok, 490234750 sectors, data@ 0
/dev/sda: isw, "isw_ddiggddhgi", GROUP, ok, 490232637 sectors, data@ 0

---

patched dmraid, ignore_hpa=0
$ sudo dmraid -r -d
DEBUG: not isw at 203928108032
DEBUG: Found isw 10 gafr signature.
DEBUG: isw 10 sector offset calculated at 2115.
DEBUG: not isw at 203927026176
DEBUG: isw trying hard coded -2115 offset.
DEBUG: not isw at 203927026176
DEBUG: isw metadata found at 251000192000 from probe at 251000192000

DEBUG: isw metadata found at 250999110144 from probe at 250999110144

/dev/sdb: isw, "isw_ddiggddhgi", GROUP, ok, 490234750 sectors, data@ 0
/dev/sda: isw, "isw_ddiggddhgi", GROUP, ok, 490232637 sectors, data@ 0

---

patched dmraid, ignore_hpa=0, after cold boot:
DEBUG: not isw at 203927026176
DEBUG: isw trying hard coded -2115 offset.
DEBUG: not isw at 203925944320
DEBUG: isw metadata found at 251000192000 from probe at 251000192000

DEBUG: isw metadata found at 250999110144 from probe at 250999110144

/dev/sdb: isw, "isw_ddiggddhgi", GROUP, ok, 490234750 sectors, data@ 0
/dev/sda: isw, "isw_ddiggddhgi", GROUP, ok, 490232637 sectors, data@ 0

---

The values around 203928108032 are from /dev/sdc, which has the
$GAFR signature but isn't part of the raid, and is on a different
controller which apparently doesn't reset HPA on a warm boot.
The second controller is
07:00.0 SATA controller: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 02)
07:00.1 IDE interface: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 02)

On Tue, 2009-05-12 at 00:41 +0000, Luke Yelavich wrote:
> Tested dmraid from karmic with Robert's patch included on my ICH9
> controller, and everything works fine.
>
> My server has ICH10, since it was built last year, and likely has HPA
> stuff I could enable, however I am not about to take it offline and plug
> a pair of spare disks in to test unless there is an absolute need to do
> so.

Well, we now know I haven't utterly borked everything.

I'd just upload it :).

-Rob

On Tue, 2009-05-12 at 00:58 +0000, 3b wrote:
> seems to work:

...

Cool, thanks.

I think this validates the hpa theory, and we should rename the patch
accordingly. Better still would be to lookup the specs on hpa and
create||use a libhpa to adjust di->sectors. But that can wait.

-Rob

Hi Robert,

Robert Collins ha scritto:
> So, I may have been writing an hpa interpreter.
>
> Could you try with my patch, with and without 'libata.ignore_hpa=0' ?

For the HPA issue, Tejun Heo wrote[1] a patch but upstream never commented it.
He works on his NV raid, but I think make isw use the new alt_size he
implemented, shouldn't be too complicated. Perhaps you could take a look...

[1]http://lwn.net/Articles/317492/
   http://patchwork.kernel.org/patch/4950/
   http://patchwork.kernel.org/patch/4952/
   http://patchwork.kernel.org/patch/4951/

Cheers,
Giuseppe.

On Tue, 2009-05-12 at 07:59 +0000, Giuseppe Iuculano wrote:
> Hi Robert,
>
> Robert Collins ha scritto:
> > So, I may have been writing an hpa interpreter.
> >
> > Could you try with my patch, with and without 'libata.ignore_hpa=0' ?
>
>
> For the HPA issue, Tejun Heo wrote[1] a patch but upstream never commented it.
> He works on his NV raid, but I think make isw use the new alt_size he
> implemented, shouldn't be too complicated. Perhaps you could take a look...

Certainly looks like the right approach. That will depend on a kernel
bump at least, which makes it more complex.

-Rob

This tweak to the patch bounds-checks what we got, so we don't try to read before the start of the disk.

I am a newcomer to Linux. I'm trying to get a dual-boot Windows XP x64 working with Ubuntu 9.04. The machine has an ICH10R motherboard with a new Windows OS load on a two-drive RAID 1 array using the Intel Matrix Manager RAID management. After initially loading XP, I started Ubuntu from a Live CD and using the included dmraid had errors when trying to set up the RAID using the instructions at:

https://help.ubuntu.com/community/FakeRaidHowto

When entering
$ sudo dmraid -ay

I had the following errors:

Error: isw device for volume "Volume0" broken on /dev/sda in RAID set "isw_df . . .
Error: wrong # of devices in RAID set "isw_df . . ." [1/2] on /dev/sda

These errors were repeated for /dev/sdb

I did not try the "$ sudo dmraid -r -d" entry. After searching I found Rob's request for testing the dmraid patch for ICH10R. I updated with his new version of dmraid, and tried the sudo dmraid -r -d entry. Here are the results:

ubuntu@ubuntu:~$ sudo dmraid -r -d
DEBUG: isw metadata found at 640135027712 from probe at 640135027712

DEBUG: isw metadata found at 640135027712 from probe at 640135027712

/dev/sdb: isw, "isw_ifbbajief", GROUP, ok, 1250263726 sectors, data@ 0
/dev/sda: isw, "isw_dfdjaigcaf", GROUP, ok, 1250263726 sectors, data@ 0

However, when trying to return to my previous instructions, I got the same errors again when executing $ sudo dmraid -ay

Hope this helps. If I can provide any more information I will be happy to, but I will need fairly explicit directions.

David

Hi David, your drive setup isn't suffering from the HPA offset issue. Please file a new bug so that your problem can be tracked - the HPA bug will be closed soon and I wouldn't want to prevent your issue getting examined.

Thanks for the reply, Robert.

David

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Robert Collins
Sent: Saturday, May 30, 2009 10:32 PM
To: <email address hidden>
Subject: [Bug 372170] Re: host protected area prevents detection of isw (intel matrix storage manager) raid arrays

Hi David, your drive setup isn't suffering from the HPA offset issue.
Please file a new bug so that your problem can be tracked - the HPA bug
will be closed soon and I wouldn't want to prevent your issue getting
examined.

--
host protected area prevents detection of isw (intel matrix storage manager) raid arrays
https://bugs.launchpad.net/bugs/372170
You received this bug notification because you are a direct subscriber
of the bug.

Status in “dmraid” source package in Ubuntu: In Progress
Status in dmraid in Ubuntu Jaunty: New

Bug description:
For some reason, my isw based system has the raid metadata 2115 sectors back. I've hard coded the offset in which got me working; I'm not sure what algorithm should be getting used - perhaps its a misdetect between BIOS and linux, or some other enjoyable weirdness.

I can confirm DMRAID does not work correctly on ICH10R controller.

I have 2 velociraptors 150Gb in RAID0 with 2 subsets, 1 - 100Gb 2- 178Gb (gaming - main)

The first array is correctly identified and activated, the second array stays in "array inactive"

I have no messages of #wrong number of devices, HPA is not interfering (I have it on ignore=0) and the second subset is not activating no matter what..

Im using Karmic Koala daily build (19 july 2009)

Any workaround this ?

I manually wipe clean both disks with /dev/zero, I created both arrays on BIOS, same thing, there is no way the second array becomes active, it always stays in "inactive"

When succesfully installed dmraid and after creating second subset, linux kernel will not boot correctly staying "sdc size too short" and it keeps saying that over and over...

Any ideas?

EagleDM your issue is not to do with this bug so please file a new one and include as much detail as possible.

I already did, thank you :)

Fixed in karmic

Closing jaunty task as it is past EOL.