Excessive memory usage on live CD

List of modules in /cow

It would also be sensible to unmount the l-r-m tmpfs if memory is short

The reason why debconf writes a second file is so that it can rename it into place, avoiding breakage in the event that it runs out of disk space. However, it *does* have a check to avoid rewriting the file needlessly when it isn't dirty, so exactly why we end up with a templates.dat-old is a good question. I'll investigate.

I've been making some effort today to reduce the size of the templates file by stripping out stuff that's automatically built into ubiquity.templates but that isn't actually needed (e.g. base-installer/*). That may help ...

Hey, is it so hard to add one command - 'rm /var/cache/debconf/templates.dat-old' in casper's scripts (after this file is changed during a casper session) ?

This saves more than 7 MB RAM and maybe let Ubuntu 7.10 (Gutsy) LiveCD to start on systems with 256 RAM (currently Ubuntu Gutsy LiveCD doesn't start on such systems, because of newly added deskbar-applet and increased size of restricted-modules and /var/cache/debconf/templates.dat* files).

On Tue, Sep 25, 2007 at 09:59:54PM -0000, Mantas Kriaučiūnas wrote:
> Hey, is it so hard to add one command - 'rm /var/cache/debconf/templates
> .dat-old' in casper's scripts (after this file is changed during a
> casper session) ?

No, it isn't hard. I would happily review and sponsor a casper update with
this functionality.

--
 - mdz

I'll do this, and it might help with live CD startup memory use. I'm not convinced it will make any difference to ubiquity, since bits of ubiquity also use db_register at run-time which will cause the templates database to be rewritten.

I've traced the /lib/modules entries to a bug in unionfs: modprobe opens the kernel modules O_RDWR in order that it can take out a write lock (but it does not actually copy them), and when it close()s them unionfs copies-up an entry to the copy-on-write filesystem, even though that isn't necessary. I'm reporting this to the unionfs mailing list now.

casper (1.106) gutsy; urgency=low

  * Support preseed/early_command; code supplied here will be run using 'sh
    -c' from casper-bottom, and can chroot to /root if needed (LP: #144845).
  * Make sure we don't accidentally continue to boot if casper-md5check
    fails (LP: #82856).
  * Clear out debconf database backup files after preseeding to save memory
    (LP: #43706).
  * Mount everything as noatime to avoid unnecessary metadata writes.

 -- Colin Watson <email address hidden> Fri, 28 Sep 2007 15:27:52 +0100

If it isn't practical to fix the O_RDWR unionfs issue, what could we do to work around it?

- Delete the copied-up versions of the .ko files after hardware detection is complete?
- Add a --no-lock to modprobe? Why does it care?

Matt Zimmerman wrote:
> what could we do to work around it?
>- Delete the copied-up versions of the .ko files after hardware detection is complete?

I think there is the very simple solution for Gutsy - nvidia*.ko (nvidia_legacy.ko, nvidia_new.ko, nvidia.ko) and fcdsl*.ko take biggest part of space (more, than 25 MB, ATI/fglrx takes only about 1,5 MB), so, it's very simple to detect if computer has nvidia/ATI hardware and do not build 3 nvidia modules if there are no nvidia hardware on the system.
Of course, the best solution would be to build only these .ko files, which can be useful on running system, but if this solution is too complicated (requires too much work), then we can just not to build biggest modules)

Btw, why this bug isn't assigned to linux-restricted-modules-common ?

On Thu, Oct 04, 2007 at 10:52:28PM -0000, Mantas Kriaučiūnas wrote:
> Matt Zimmerman wrote:
> > what could we do to work around it?
> >- Delete the copied-up versions of the .ko files after hardware detection is complete?
>
> I think there is the very simple solution for Gutsy - nvidia*.ko (nvidia_legacy.ko, nvidia_new.ko, nvidia.ko) and fcdsl*.ko take biggest part of space (more, than 25 MB, ATI/fglrx takes only about 1,5 MB), so, it's very simple to detect if computer has nvidia/ATI hardware and do not build 3 nvidia modules if there are no nvidia hardware on the system.

We disable restricted-manager in the live session anyway, so it should be
fine to simply disable lrm-manager entirely.

> Btw, why this bug isn't assigned to linux-restricted-modules-common ?

Because there are two slightly different issues here:

1. casper wants to reduce the memory needed for the live session, and can do
so by disabling the video modules unconditionally (since they're difficult
to enable in that environment anyway)

2. It would be nice if linux-restricted-modules-common would dynamically
generate only the appropriate modules

--
 - mdz

The linux-restricted-modules .ko files aren't the ones that are copied up by unionfs, so aren't what I was talking about. You're right, though, I think it would be a good idea not to generate them.

Matt: deleting the copied-up versions in /cow is probably the simplest answer, but (a) requires showmounts or else can only be done in the initramfs (b) is a code path in unionfs we don't otherwise exercise so I don't want to prod it now. I'm not sure I feel like chasing down all the possible places where modprobe might be called.

On Fri, Oct 05, 2007 at 07:57:55PM -0000, Colin Watson wrote:
> Matt: deleting the copied-up versions in /cow is probably the simplest
> answer, but (a) requires showmounts or else can only be done in the
> initramfs (b) is a code path in unionfs we don't otherwise exercise so I
> don't want to prod it now. I'm not sure I feel like chasing down all the
> possible places where modprobe might be called.

Agreed.

--
 - mdz

For the record:

casper (1.110) gutsy; urgency=low

  * Don't generate fglrx or the nvidia modules in the live session at boot;
    we disable restricted-manager in the live session anyway so they aren't
    straightforward to use (see #43706). Saves about 19 MB of memory.

 -- Colin Watson <email address hidden> Fri, 05 Oct 2007 21:28:29 +0100

The 18 month support period for Gutsy Gibbon 7.10 has reached its end of life -
http://www.ubuntu.com/news/ubuntu-7.10-eol . As a result, we are closing the
linux-ubuntu-modules-2.6.22 task. It would be helpful if you could test the
new Jaunty Jackalope 9.04 release and confirm if this issue remains -
http://www.ubuntu.com/getubuntu/releasenotes/904overview. If the issue still exists with the Jaunty
release, please update this report by changing the Status of the "linux (Ubuntu)"
task from "Incomplete" to "New". Thanks in advance.

The kernel part of this issue had to do with unionfs, per
https://bugs.edge.launchpad.net/ubuntu/+source/casper/+bug/43706/comments/7

I have no idea whether the same issue affects aufs, though it's possible.
This needs to be verified on 9.04.

--
 - mdz

mdz

I've run a system with the recommended minimum ram of 384 for live cd. Using mem=384M as recommended by cjwatson.

The system is slow but usable.

Fire fox is a killer

pre firefox
ubuntu@ubuntu:~$ free
             total used free shared buffers cached
Mem: 379840 374980 4860 0 38220 179908
-/+ buffers/cache: 156852 222988
Swap: 3074972 564 3074408

Post firefox

ubuntu@ubuntu:~$ free
             total used free shared buffers cached
Mem: 379840 361140 18700 0 14972 169088
-/+ buffers/cache: 177080 202760
Swap: 3074972 92140 2982832

the system nearly died of shock opening OO.o

ubuntu@ubuntu:~$ free
             total used free shared buffers cached
Mem: 379840 365420 14420 0 28924 182760
-/+ buffers/cache: 153736 226104
Swap: 3074972 150584 2924388

Matt what would help me is to know how you got you results so I can get a real metric with yours. I will chase this up with you tomorrow though and some fresh tests.

On Thu, Sep 17, 2009 at 09:49:37PM -0000, Dave Morley wrote:
> I've run a system with the recommended minimum ram of 384 for live cd.
> Using mem=384M as recommended by cjwatson.
>
> The system is slow but usable.

Did you check whether the problems described in this bug report are still
present, e.g. spurious copies of modules in the copy-on-write filesystem?

--
 - mdz

I advised Dave on IRC to use the 'showmounts' kernel parameter and poke
around in /cow.

there is no /cow/lib

ubuntu@ubuntu:/cow$ ls -al
total 0
drwxr-xr-x 13 root root 280 2009-09-18 07:45 .
drwxr-xr-x 31 root root 280 2009-09-18 07:45 ..
drwxr-xr-x 3 root root 60 2009-09-18 07:45 casper
drwxr-xr-x 2 root root 40 2009-09-18 07:45 cdrom
drwxr-xr-x 2 root root 40 2009-09-18 07:45 cow
drwxr-xr-x 14 root root 620 2009-09-18 07:47 etc
drwxr-xr-x 3 root root 60 2009-09-18 07:45 home
drwxr-xr-x 2 root root 40 2009-09-18 07:45 rofs
drwxrwxrwt 2 root root 40 2009-09-18 07:46 tmp
drwxr-xr-x 5 root root 100 2009-09-17 01:40 usr
drwxr-xr-x 6 root root 120 2009-09-17 01:47 var
-r--r--r-- 9 root root 0 2009-09-18 07:45 .wh..wh.aufs
drwx------ 2 root root 40 2009-09-18 07:45 .wh..wh.orph
drwx------ 2 root root 260 2009-09-18 07:46 .wh..wh.plnk

I'm having a root around to be sure it isn't showing up else where

There doesn't appear to be a template old either

ubuntu@ubuntu:/cow/var/cache$ ls -al debconf/
total 9216
drwxr-xr-x 2 root root 100 2009-09-18 07:46 .
drwxr-xr-x 7 root root 140 2009-09-17 01:46 ..
-rw-r--r-- 1 root root 103097 2009-09-18 07:46 config.dat
-rw------- 1 root root 1102 2009-09-18 07:46 passwords.dat
-rw-r--r-- 1 root root 9300455 2009-09-18 07:46 templates.dat

Current usage for /cow/vat/cache/debconf is

ubuntu@ubuntu:/cow/var/cache$ du -h /cow/var/cache/debconf/
9.0M /cow/var/cache/debconf/

Usage for Cow as a whole is 43M break down below

ubuntu@ubuntu:/cow$ du -h /cow
44K /cow/usr/sbin
0 /cow/usr/lib/update-notifier
4.0K /cow/usr/lib/locale/en_US.utf8/LC_MESSAGES
1.3M /cow/usr/lib/locale/en_US.utf8
1.3M /cow/usr/lib/locale
1.3M /cow/usr/lib
0 /cow/usr/share/ppd/custom
0 /cow/usr/share/ppd
0 /cow/usr/share/gdm
0 /cow/usr/share/xsessions
0 /cow/usr/share/doc
12K /cow/usr/share/applications
12K /cow/usr/share
1.3M /cow/usr
0 /cow/tmp
0 /cow/home/ubuntu/.mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}
0 /cow/home/ubuntu/.mozilla/extensions
3.4M /cow/home/ubuntu/.mozilla/firefox/fwi6syee.default/Cache
0 /cow/home/ubuntu/.mozilla/firefox/fwi6syee.default/extensions
8.0K /cow/home/ubuntu/.mozilla/firefox/fwi6syee.default/chrome
17M /cow/home/ubuntu/.mozilla/firefox/fwi6syee.default
17M /cow/home/ubuntu/.mozilla/firefox
17M /cow/home/ubuntu/.mozilla
280K /cow/home/ubuntu/.gstreamer-0.10
0 /cow/home/ubuntu/.nautilus
0 /cow/home/ubuntu/.update-notifier
312K /cow/home/ubuntu/.cache/compizconfig
328K /cow/home/ubuntu/.cache
0 /cow/home/ubuntu/.gnome2_private
0 /cow/home/ubuntu/.gnome2/keyrings
0 /cow/home/ubuntu/.gnome2/nautilus-scripts
0 /cow/home/ubuntu/.gnome2/panel2.d/default/launchers
0 /cow/home/ubuntu/.gnome2/panel2.d/default
0 /cow/home/ubuntu/.gnome2/panel2.d
12K /cow/home/ubuntu/.gnome2/accels
12K /cow/home/ubuntu/.gnome2
0 /cow/home/ubuntu/.ssh
16K /cow/home/ubuntu/.gnupg
0 /cow/home/ubuntu/.gvfs
20K /cow/home/ubuntu/.fontconfig
84K /cow/home/ubuntu/.gconfd
152K /cow/home/ubuntu/.pulse
4.0K /cow/home/ubuntu/.dbus/session-bus
4.0K /cow/home/ubuntu/.dbus
108K /cow/home/ubuntu/.local/share/gvfs-metadata
108K /cow/home/ubuntu/.local/share
108K /cow/home/ubuntu/.local
0 /cow/home/ubuntu/Videos
0 /cow/home/ubuntu/Pictures
0 /cow/home/ubuntu/Music
0 /cow/home/ubuntu/Documents
0 /...

Why en_US.UTF-8 locale is generated every time during liveCD/liveUSB startup? See these lines in less /var/log/casper.log :

Begin: Setting up locales... ... Generating locales...
  en_US.UTF-8... done
Generation complete.
done.

Locale generation uses a lot of memory and slowdowns startup.
I think en_US.UTF-8 locale should be already generated in filesystem.squashfs and other locales should be generated during liveCD/liveUSB startup only if user chooses another language in syslinux.

It seems Ubuntu 12.04 and newer releases already contains en_US and other en_* locales in filesystem.squashfs (they are generated during installation of language-pack-en-base package), but script casper-bottom/14locales doesn't check if locale isn't already generated - I think command "localedef --list-archive" should be used for checking if needed locale isn't already generated.

I've reported separate bug #1218429 (casper-bottom/14locales should check if locale isn't already generated)
It seems command /usr/sbin/locale-gen , used in 14locales script, should detect that en_US .UTF-8 is already generated, but doesn't detect during startup of live system and generates already generated locale :(

This version has expired