Lucid Default live-cd install fails with 4K sector / Advanced Format drives

util-linux-ng (Ubuntu's "util-linux" package) could also be updated to version 2.17.1 (lucid currently uses 2.17) to better support 4k sectors disks.

ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/v2.17.1-ReleaseNotes

It's not critical to update util-linux, since the installer doesn't use fdisk itself, but it would be good to stop people shooting themselves in the foot.

So, for the most part this is all obviously sensible. The installer can and should create partitions with optimal alignment (and I've made that change, pending testing), and it makes complete sense for mkfs.ext4 to complain when you do something that will result in poor performance.

The bit that is a little bit awkward is this: imagine that you take an existing partition which was created before we fixed the partitioner to deal with optimal alignment, and try to overwrite it with a new ext4 filesystem (which may seem to be a perfectly reasonable action to a user unfamiliar with the minutiae of logical sector sizes). mkfs.ext4 will complain in the manner you describe, and there seems to be no straightforward way to force it to proceed anyway (short of 'yes | ...', which I don't think is acceptable!). We should probably implement the same check in the partitioner when we're about to create a new filesystem, but if the user says that they don't want to move the start of the filesystem, what are we to do? Ideally we'd be able to run mkfs.ext4 under the covers with a "yes, I know, just get on with it" option.

Ted, would you be willing to make the alignment warning here overridable using a command-line option?

On Mar 11, 2010, at 8:30 AM, Colin Watson wrote:
> So, for the most part this is all obviously sensible. The installer can
> and should create partitions with optimal alignment (and I've made that
> change, pending testing), and it makes complete sense for mkfs.ext4 to
> complain when you do something that will result in poor performance.
>
> The bit that is a little bit awkward is this: imagine that you take an
> existing partition which was created before we fixed the partitioner to
> deal with optimal alignment, and try to overwrite it with a new ext4
> filesystem (which may seem to be a perfectly reasonable action to a user
> unfamiliar with the minutiae of logical sector sizes). mkfs.ext4 will
> complain in the manner you describe, and there seems to be no
> straightforward way to force it to proceed anyway (short of 'yes | ...',
> which I don't think is acceptable!). We should probably implement the
> same check in the partitioner when we're about to create a new
> filesystem, but if the user says that they don't want to move the start
> of the filesystem, what are we to do? Ideally we'd be able to run
> mkfs.ext4 under the covers with a "yes, I know, just get on with it"
> option.
>
> Ted, would you be willing to make the alignment warning here overridable
> using a command-line option?

Eric Sandeen has already sent me a patch to do this via the -F option. It will be in e2fsprogs 1.41.11, which will be out Real Soon Now (e2fsprogs 1.41.10 has this unfortunate problem where e2fsck -D corrupts file systems...)

-- Ted

This bug was fixed in the package partman-base - 138ubuntu3

---------------
partman-base (138ubuntu3) lucid; urgency=low

  * Apply optimal alignment constraints to new partitions, or when
    maximising an extended partition (LP: #530071).
  * Add an ALIGNMENT_OFFSET command which can be used to detect whether a
    partition is misaligned.
 -- Colin Watson <email address hidden> Thu, 11 Mar 2010 20:59:25 +0000

Ah, thank you!

Hi,
I've tried the daily build of "Alternate Install" from 12 March on a 4K drive (WD10-EARS) and although the partitions now align on good 4K boundaries, I think the algorithm for creating partitions is working in cylinder units rather than sectors. So using the default install I get partition 1 starting at cylinder 2048 which as LBA 32901120 is a 4KB boundary but it is 16.8GB in to the disk. I think we should be looking at LBA 2048 as the start of the partition, not cylinder.

root@ubuntu1003a3120310ears4k:~# parted
GNU Parted 2.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit cyl
(parted) p
Model: ATA WDC WD10EARS-00Y (scsi)
Disk /dev/sda: 121601cyl
Sector size (logical/physical): 512B/4096B
BIOS cylinder,head,sector geometry: 121601,255,63. Each cylinder is 8225kB.
Partition Table: msdos

Number Start End Size Type File system Flags
 1 2048cyl 120831cyl 118784cyl primary ext4 boot
 2 120832cyl 121600cyl 769cyl extended
 5 120832cyl 121600cyl 768cyl logical linux-swap(v1)

(parted) unit s
(parted) p
Model: ATA WDC WD10EARS-00Y (scsi)
Disk /dev/sda: 1953525168s
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number Start End Size Type File system Flags
 1 32901120s 1941166079s 1908264960s primary ext4 boot
 2 1941166080s 1953520064s 12353985s extended
 5 1941168128s 1953519615s 12351488s logical linux-swap(v1)

(parted) unit gb

(parted) p
Model: ATA WDC WD10EARS-00Y (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos

Number Start End Size Type File system Flags
 1 16.8GB 994GB 977GB primary ext4 boot
 2 994GB 1000GB 6.33GB extended
 5 994GB 1000GB 6.32GB logical linux-swap(v1)

Hi,

I think it would be useful to consider defaulting to 1MB boundaries in any case if drive appears to be "traditional" 512/512. I have seen the case where external USB adapters do not pass on the drive's alignment / offset data due to the cut-down SCSI command set used by USB to SATA adapters. Identify Drive works as a SATA pass through command but the kernel works at the SCSI layer and does not get the data required to set up sysfs alignment parameters.

Regards

Steve

Right, thanks - we needed to set another flag on the PedDisk, and I'm tracking this as bug 539456. It's release-critical for beta-1 and I'm in the process of pushing up a fix. It should be possible to get a more accurate assessment of the state of the world after that fix, as right now it's all a bit skewed by the error.

I've tested on the USB-adapted drive you sent me (thanks again!), and confirmed that the beta-1 installer defaults to 1MiB boundaries on that drive. This should apply to all USB drives, since the Linux kernel suppresses the SPC-3 command set unilaterally for USB mass-storage devices regardless of what the adapter does.

Hi Colin,

Looks good - defaulting to 1MB boundaries seems to be a good safe option for
all concerned. I hope the beta becomes visible soon so I can get a copy!

That USB enclosure only reports SPC-2 and so it would not be expected to get
READ CAPACITY(16) from the host to report sector size/alignment data. In
fact with the current firmware, it will support READ CAPACITY(16) but does
not fill in the sector/alignment bytes.

If we added SPC-3 reporting and fixed the appropriate commands, are you
saying that as a USB device, we will still never get the chance to return
our parameters from READ CAPACITY(16) as the kernel will force the USB
entity to be SPC-2? We are looking at how we can accommodate Advanced Format
drives in USB in the US for our external drives.

Cheers

Steve

Steve Perkins
Field Technical Consultant
Western Digital (UK) Ltd.
Prime House,Challenge Court,
Barnett Wood Lane, Leatherhead
Surrey KT22 7DE
+44 1372 366034
<email address hidden>
Mob: +44 7780 990809

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
> Colin Watson
> Sent: Thursday, March 18, 2010 22:11
> To: Steve Perkins
> Subject: [Bug 530071] Re: Lucid Default live-cd install fails with 4K
> sector /Advanced Format drives
>
> I've tested on the USB-adapted drive you sent me (thanks again!), and
> confirmed that the beta-1 installer defaults to 1MiB boundaries on that
> drive. This should apply to all USB drives, since the Linux kernel
> suppresses the SPC-3 command set unilaterally for USB mass-storage
> devices regardless of what the adapter does.
>
> --
> Lucid Default live-cd install fails with 4K sector / Advanced Format
> drives
> https://bugs.launchpad.net/bugs/530071
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in "e2fsprogs" package in Ubuntu: New
> Status in "partman-base" package in Ubuntu: Fix Released
> Status in "partman-ext3" package in Ubuntu: In Progress
> Status in "ubiquity" package in Ubuntu: Triaged
> Status in "util-linux" package in Ubuntu: New
> Status in "e2fsprogs" source package in Lucid: New
> Status in "partman-base" source package in Lucid: Fix Released
> Status in "partman-ext3" source package in Lucid: In Progress
> Status in "ubiquity" source package in Lucid: Triaged
> Status in "util-linux" source package in Lucid: New
>
> Bug description:
> Binary package hint: ubiquity
>
> Use default partition layout for Lucid Allpha 3 on new hard disk fails to
> account for 4K sector / Advanced Format drives. For example WD10-EARS
> which reports sector alignment in Identify Drive as 4KB internal sectors,
> 512 external.
> Ubiquity hangs during "Partition Formatting" at 5% with "Creating ext4
> file system for / in partition #1 of SCSI1 (0,0,0) (sda)...".
>
> Open terminal and fdisk -l shows partitions are created correctly but
> don't "start on optimal I/O boundary". Shut down ubiquity and attempt a
> manual mkfs.ext4 and I get a warning:
> /dev/sda1 alignment is offset by 512 bytes.
> This may result in very poor performance, (re)-partitioning suggested.
> Proceed anyway? (y,n)
>
> So the tools here (fdisk, mkfs) are correctl...

Exactly. Here's the relevant bit of code in drivers/usb/storage/scsiglue.c:

                /* Some devices report a SCSI revision level above 2 but are
                 * unable to handle the REPORT LUNS command (for which
                 * support is mandatory at level 3). Since we already have
                 * a Get-Max-LUN request, we won't lose much by setting the
                 * revision level down to 2. The only devices that would be
                 * affected are those with sparse LUNs. */
                if (sdev->scsi_level > SCSI_2)
                        sdev->sdev_target->scsi_level =
                                        sdev->scsi_level = SCSI_2;

I suspect this predates widespread use of large physical sectors, and needs to be revised.

Have uploaded 2.17.2 to lucid

This bug was fixed in the package e2fsprogs - 1.41.11-1ubuntu1

---------------
e2fsprogs (1.41.11-1ubuntu1) lucid; urgency=low

  * Merge from Debian unstable, remaining changes:
    - Do not build-depend on dietlibc-dev, which is universe.
    - Do now allow pkg-create-dbgsym to operate on this package.
    - Always use external libblkid and libuuid from util-linux, rather than
      building our own.
    - Includes debian/control in the source package to force the above.
    - Build with -O2 on powerpc to avoid a suspected toolchain bug
      (LP: #450214).
    - Do not include /etc/e2fsck.conf and remove on upgrade.
    (Fixes LP: #521648, #537483, #530071)

e2fsprogs (1.41.11-1) unstable; urgency=medium

  * New upstream release
  * Add Heimdal function com_right_r() to libcom_err (Closes: #558910)
  * Allow e2fsck to run even if the physical device has more than 2**32 blocks
  * Debugfs's "logdump -b <blk>" now properly shows the allocation status
    of the block <blk>. (Closes: #564084)
  * Make e2fsck's "the filesystem is mounted" message is now more scary
    to hopefully dissuade users from thinking, "surely that message
    doesn't apply to *me*" :-(
  * e2fsck -n will now always open the file system read-only. We now
    disallow certain combination of options which previously were manual
    exceptions; this is bad because it causes users to think they are
    smarter than they really are. So "-n -c", "-n -l", "-n -L", and
    "-n -D" are no longer supported.
  * If the partition is badly aligned, have mke2fs just print a warning
    message and continue. Previously mke2fs would ask to confirm, and
    this broke distro installation scripts.
  * Fix a bug in libext2fs caused the creation of very large journals
    for ext4 to be _very_ slow.
  * E2fsck now understands the EOFBLOCKS_FL flag which will be used in
    2.6.34 kernels to make e2fsck not complain about blocks deliberately
    fallocated() beyond an inode's i_size.
  * Fix a bug in e2fsck which could cause e2fsck -D to corrupt
    non-indexed directories. (Closes: #572453)
  * debian/rules: can be compiled statically with stack protector now.
    (Closes: #573923)
  * Update debian policy compliance to 3.8.4
 -- Scott James Remnant <email address hidden> Mon, 22 Mar 2010 17:48:20 +0000

This bug was fixed in the package partman-ext3 - 58ubuntu2

---------------
partman-ext3 (58ubuntu2) lucid; urgency=low

  * Warn when reformatting an existing misaligned partition for use as
    ext3/ext4, since it will result in poor performance and this is about
    the only sensible opportunity to fix it (LP: #530071).
 -- Colin Watson <email address hidden> Wed, 31 Mar 2010 16:29:29 +0100

Older boards might not like the new partitioning, see bug 551965

There's nothing specific to ubiquity in this, so invalidating that task.

Sorry to crap on your bug thread, but it seems like you guys can answer this. I was going to buy a wd6400bevt 2.5" drive and put it in a portable USB case, but wanna wait until we can partition these things properly. Somebody please let me know when I won't take a performance and or size hit. I'm not planning on using the ubuntu installer, rather plan on using gparted, since I'm not installing OS to the disk.

Hi, ubiquity no more hang when installing ubuntu on a WD10EARS drive but the partitions aren't correctly aligned.
Could someone check this bug: https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/583790 ?