coreutils: du(1) doesn't summarize correctly

Confirmed in Ubuntu 9.10 32-bit (apt-get upgraded as of 21 August 2009)

1. Windows 7
Folder Size: 3.09 GB (3,322,187,937 bytes)

2. Ubuntu 9.10 using du ( does not look right at all )
rrosso@u910:/media/mediazone$ du -sh 2009.TriNations/
397M 2009.TriNations/

3. Ubuntu 9.10 using nautilus
7 items, totalling 3.1 GB

4. rrosso@u910:/media/mediazone$ sudo aptitude show coreutils | grep Version
Version: 7.4-2

*** Note this test was done with a SMB mounted folder

Thank you for reporting this bug and helping make Ubuntu better.

I cannot reproduce this issue. I wonder if symbolic links may be part of the different values shown.

Can you please run 'du -shL'?

Thank you

This is certainly weird. Without any changes to the folder in question, 'du -hs' now reports a size of 75M. The result for 'du -hsL' is the same. The folder or any content within hasn't been touched since I reported the bug. The only thing that occurred in between were system restarts and a kernel update (latest Ubuntu updates).

Any idea?

I checked what Nautilus says about the size of the folder and it still says by its original output:

"2,966 items, totalling 54.0 MB"

So even if du now shows more than just 4MB, more than 70MB are still off the mark.

Something is very wrong here.

Right now, no idea. Time to dig in.

1. what type of filesystem is this?
2. is it local or remote?
3. does it happen on other directories/FSs?
4. can you copy it to another place, and test again?

1. what type of filesystem is this?

Information from 'mount -v':
...
/dev/sda12 on /home type ext3 (rw,relatime)
...
/home/username/.Private on /home/username type ecryptfs (ecryptfs_sig=60cc62a1039711h8,ecryptfs_fnek_sig=3257a12b81aff999,ecryptfs_cipher=aes,ecryptfs_key_bytes=16)

So, it's an encrypted home directory on an ext3 partition.

It might be ecryptfs related maybe?

2. is it local or remote?

local

3. does it happen on other directories/FSs?

Yes, most certainly! I pulled three projects from a Perforce server, all seem to report the wrong size with 'du -hsL'. In some cases, the difference is factor 2!

4. can you copy it to another place, and test again?

I guess you assume to copy it to a non-ecryptfs filesystem. Sure thing. Result:

Original folder now shows 59M copied to /home (local ext3), Nautilus shows 54M on ecryptfs and on ext3. So it's closer, but not identical.

I have tested on same system with all local files and do not see an issue. I will circle back to a similar test where I used a SMB mount(see my comment on 2009-08-21) and test again.

root@U910-64:~# uname -a
Linux U910-64 2.6.31-10-generic #35-Ubuntu SMP Tue Sep 22 17:33:14 UTC 2009 x86_64 GNU/Linux

root@U910-64:/# grep DATA /etc/fstab
UUID=AAF6E3F4F6E3BF27 /DATA ntfs defaults,umask=007,gid=46 0 1
UUID=d4af6bf2-3584-45a2-9bcb-cb686a453cbe /DATABANK ext4 defaults 0 1

du -sh of /DATA
rrosso@U910-64:/$ du -sh DATA
62G DATA

Nautilus /DATA on ntfs filesystem
36,356 items, totalling 61.5 GB

Copied to ext4 system
root@U910-64:/# cd / ; tar cpf - DATA | (cd /media/DATABANK/ ; tar xpf -)

Nautilus /media/DATABANK/DATA on ext4 filesystem
36,356 items, totalling 61.5 GB

root@U910-64:~# du -sh /media/DATABANK/DATA/
62G /media/DATABANK/DATA/

root@U910-64:~# aptitude show coreutils | grep Version
Version: 7.4-2

Update. This bug exists in du. I should add this is a virtualbox mounted folder shared from the local host. However Nautilus and ls has the sizes correct so I don't think it matters that it is a virtualbox mount.

rrosso@u910:/media/mediazone$ uname -a
Linux u910 2.6.31-11-generic #36-Ubuntu SMP Fri Sep 25 06:37:23 UTC 2009 x86_64 GNU/Linux

Du reports wrong size
rrosso@u910:/media/mediazone$ du -sh 2009.TriNations/
767M 2009.TriNations/

Nautilus /media/mediazone
10 items, totalling 6.0 GB

Windows 7 D:\Mediazone\2009.TriNations
5.98 GB (6,425,764,846 bytes)

My own C code I wrote to tally:
rrosso@u910:~/Desktop$ ./myDu /media/mediazone/2009.TriNations/
6.0G /media/mediazone/2009.TriNations/

rrosso@u910:/media/mediazone$ sudo aptitude show coreutils | grep Version
Version: 7.4-2

Filesizes wrong with du:
rrosso@u910:~/Desktop$ du -shL /media/mediazone/2009.TriNations/*
130M /media/mediazone/2009.TriNations/090912_NZvSA.wmv
117M /media/mediazone/2009.TriNations/2009.Tri.Nations.Game.2.South.Africa.vs.New.Zealand.Bloemfontein
110M /media/mediazone/2009.TriNations/e-090919_NZvAUS.wmv
131M /media/mediazone/2009.TriNations/Rugby-(Tri Nations 2009_ South Africa vs. Australia)-2009-08-08-0.mkv
149M /media/mediazone/2009.TriNations/Rugby-(Tri Nations 2009_ South Africa vs. New Zealand)-2009-08-01-0.mkv
132M /media/mediazone/2009.TriNations/Rugby.Union.Tri.Nations.2009.Australia.v.New.Zealand.20090822.Sky.NZ.mp4

Filesizes correct with ls:
rrosso@u910:~/Desktop$ ls -lh /media/mediazone/2009.TriNations/
total 650M
-rwxrwxrwx 1 root root 1.1G 2009-09-12 20:26 090912_NZvSA.wmv
drwxrwxrwx 1 root root 4.0K 2009-07-27 18:19 2009.Tri.Nations.Game.2.South.Africa.vs.New.Zealand.Bloemfontein
-rwxrwxrwx 1 root root 876M 2009-09-20 09:17 e-090919_NZvAUS.wmv
-rwxrwxrwx 1 root root 1.1G 2009-08-08 20:06 Rugby-(Tri Nations 2009_ South Africa vs. Australia)-2009-08-08-0.mkv
-rwxrwxrwx 1 root root 1.2G 2009-08-02 13:44 Rugby-(Tri Nations 2009_ South Africa vs. New Zealand)-2009-08-01-0.mkv
-rwxrwxrwx 1 root root 1.1G 2009-08-25 13:44 Rugby.Union.Tri.Nations.2009.Australia.v.New.Zealand.20090822.Sky.NZ.mp4

In my particular case the problem is that coreutils probably picks none or the wrong value for ST_NBLOCKS for the VBOXSF file system. I looked at the du source in the coreutils-7.4 source and it appears like ST_NBLOCKS are not getting set. Problem is most likely in system.h where ST_NBLOCKS should be set.

In du.c where file size is not computed correctly.
duinfo_set (&dui,(apparent_size? sb->st_size
     : (uintmax_t) ST_NBLOCKS (*sb) * ST_NBLOCKSIZE), <------ ST_NBLOCKS is incorrect ----------->
    (time_type == time_mtime ? get_stat_mtime (sb): time_type == time_atime ? get_stat_atime (sb):get_stat_ctime (sb)));

If "apparent size" is used to compute the file size du works correctly. So using "du -bh" or "du -sh --apparent-size" works correctly. Let me know if there is anything else you want me to test in my particular case.

I noticed this weird behavior a few days ago, I also use an encrypted home (created my the ubuntu installer), so it might be the cause of the issue.
My details:
Jaunty 9.04 amd64
$ uname -a
Linux nexon 2.6.30-9-generic #10-Ubuntu SMP Wed Jun 10 12:45:25 UTC 2009 x86_64 GNU/Linux (have to run this because of the intel vga...)
$ du --version
du (GNU coreutils) 6.10 (...)
# aptitude show coreutils | grep Version
Version: 6.10-6ubuntu1

tests (using the same directory):
$ du -sh
92K .
$ du -shL
92K .
$ du -sh --apparent-size
131M . (this is correct!)

after I copied this directory to the root ext3 fs (which is not encrypted):
$ du -sh
108M . (not correct)
du -sh --apparent-size
131M . (this is correct again)

can reproduce this issue on a current karmic:

c@n:~/Pictures$ du -shL *
16K Picasa Exports
c@n:~/Pictures$ du -shL --apparent-size *
10M Picasa Exports
c@n:~/Pictures$ uname -a
Linux n 2.6.31-17-generic #54-Ubuntu SMP Thu Dec 10 17:01:44 UTC 2009 x86_64 GNU/Linux

du (GNU coreutils) 7.4

Have the same on 9.10, also use an encrypted home

This appears to be resolved in U10.04. For vboxsf file systems at least. Not sure about encryptfs.

rrosso@u10:/media/Mediazone$ du -sh s14r11
5.8G s14r11 < --- same in NAutilus as well as on the Windows 7 host

rrosso@u10:/media/Mediazone$ df -h .
Filesystem Size Used Avail Use% Mounted on
Mediazone 849G 408G 442G 48% /media/Mediazone/

rrosso@u10:/media/Mediazone$ mount | grep Mediazone
Mediazone on /media/Mediazone/ type vboxsf (rw)

rrosso@u10:/media$ uname -a
Linux u10.04 2.6.32-21-generic #32-Ubuntu SMP Fri Apr 16 08:09:38 UTC 2010 x86_64 GNU/Linux

rrosso@u10:/media/Mediazone$ sudo aptitude show coreutils | grep Version
Version: 7.4-2ubuntu2