Add compcache modules, allowing ubiquity installs on 256MB machines.

Bug #200765 reported by John McCabe-Dansted
28
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Baltix
Invalid
Undecided
Unassigned
linux (Ubuntu)
Fix Released
Medium
Andy Whitcroft

Bug Description

The attached debdiff adds the compcache module. This module creates a swap device that stores swapped out pages in memory compressed using lzo. This greatly reduces the amount of memory required by the LiveCD before the physical swap file is created.

 I have tested ubiquity+gnome installs with 180MB and 120MB of ram on 7.10 [k]ubuntu and xubuntu respectively. By comparison 7.10 Ubuntu boots but is unusable with 192MB of ram.

 It appears that with compcache the ubiquity-only install in hardy will still function with only 256 MB of ram.

Overview of testing: I have tested this with
 - [kx]ubuntu-7.10 and ubuntu-hardy alpha 4 and Mar 08 daily build.
 - VirtualBox, Qemu [smp,nosmp], native DualCore, native P4
 - 120MB through 1GB of ram.
 - token testing with ia64 (but have not enabled ia64 in this patch)
 - have tested that Ubuntu does not try to hibernate onto compcache partition (by default, you could probably edit text files to do this if you really wanted to shoot yourself in the foot.).

The only negative I found with compcache-0.2 is that the mouse appears more jerky under heavy load than when physical swap is used. I have provided demo patchs against gusty liveCDs at:
  http://www.ucc.asn.au/~mccabedj/ccache/

I have build the arch using
  fakeroot debian/rules binary-arch arch=i386 flavours="generic"
as suggested in debian/rules. It generates the deb fine. (also, "pbuilder build" completed successfully)
I installed the deb without any problems, and am using it now. No packages currently depend on compcache, so they cannot be broken.

Two problems that are easily worked around
1) that ubiquity tries to install onto the compcache device (on the demo cd I rename the device), and
2) compcache uses kernel memory, which is limited to 1GB on i386. To ensure we don't run out, I limit the compcache size to under 200MB so less than 100MB of kernel memory is used. (The compression ratio for the liveCD seems to be robustly 2:1).

See also
  bug 193552 "Support 256MB machines in Ubiquity"
  bug 200418 "Memory usage increase in LiveCD"
https://lists.ubuntu.com/archives/sounder/2008-February/011089.html
https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2008-March/003493.html
https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2008-March/003516.html

These currently allow.

This replaces

 debian/changelog | 9
 debian/config/i386 | 3
 ubuntu/Makefile | 2
 ubuntu/misc/compat.h | 35 ++
 ubuntu/misc/compcache.c | 428 ++++++++++++++++++++++++++++++
 ubuntu/misc/compcache.h | 148 ++++++++++
 ubuntu/misc/config | 2
 ubuntu/misc/tlsf.c | 669 ++++++++++++++++++++++++++++++++++++++++++++++++
 ubuntu/misc/tlsf.h | 93 ++++++
 ubuntu/misc/tlsf_int.h | 179 ++++++++++++
 10 files changed, 1568 insertions(+)

ProblemType: Bug
Architecture: i386
Date: Mon Mar 10 16:49:01 2008
DistroRelease: Ubuntu 8.04
Package: linux-ubuntu-modules-2.6.24-12-generic 2.6.24-12.16
PackageArchitecture: i386
SourcePackage: linux-ubuntu-modules-2.6.24
Uname: Linux 2.6.24-12-generic i686

Revision history for this message
John McCabe-Dansted (gmatht) wrote :
Revision history for this message
John McCabe-Dansted (gmatht) wrote :
Revision history for this message
John McCabe-Dansted (gmatht) wrote :
Revision history for this message
John McCabe-Dansted (gmatht) wrote :

BTW, compcache is not loaded by default, so the initscript that loads it is responsible for the workarounds above. see:
   http://www.ucc.asn.au/~mccabedj/ccache/patch_livecd.tar.bz2
   (newroot/filestoadd/etc/init.d/compcache)
for an example initscript. With this proper deb, the insmod-ing nastyness of the initscript is unnecessary. A simply "modprobe compcache" will do.

description: updated
Changed in linux-ubuntu-modules-2.6.24:
assignee: nobody → ubuntu-kernel-team
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
John McCabe-Dansted (gmatht) wrote :

I have now also made a freeze exception request on bug 193552, with modifications that cause ubiquity to load compcache modules and use compcache when no physical swap is available.

Revision history for this message
Johan Kiviniemi (ion) wrote :

I’ve done some initial work on packaging compcache:
http://gitweb.heh.fi/?p=ion/ubuntu/compcache.git

It builds a compcache-source package, which can be used with ‘module-assistant auto-install compcache’ until compcache makes it into linux-ubuntu-modules, as well as a compcache-utils package, which contains the following init script and setting file:
http://gitweb.heh.fi/?p=ion/ubuntu/compcache.git;a=blob;f=debian/compcache-utils.init;hb=ubuntu
http://gitweb.heh.fi/?p=ion/ubuntu/compcache.git;a=blob;f=debian/compcache-utils.default;hb=ubuntu

Revision history for this message
Steve Langasek (vorlon) wrote :

Sorry, this sort of installer change needs to happen earlier in the release cycle. As nice as it would be to have the live installer working on lower-memory computers, we shouldn't be introducing new kernel modules into the installer the week of the beta.

Revision history for this message
John McCabe-Dansted (gmatht) wrote :

I presume the best way forward is for me to wait until development on Ibex starts, then resubmit these debdiffs againsts an early snapshot of Ibex?

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 200765] Re: Add compcache modules, allowing ubiquity installs on 256MB machines.

On Tue, Mar 18, 2008 at 01:05:24PM -0000, John McCabe-Dansted wrote:
> I presume the best way forward is for me to wait until development on
> Ibex starts, then resubmit these debdiffs againsts an early snapshot of
> Ibex?

Yes, that's correct.

Changed in linux-ubuntu-modules-2.6.24:
milestone: none → later
Revision history for this message
Johan Kiviniemi (ion) wrote :

I changed my packaging (linked above) to use debconf for the cache size setting instead of a config file in /etc/default.

I reckon the installer would just need to preseed compcache-utils/cache-size to a value such as ‘100 %’ to use something else than the compcache module’s default (25 % of the amount of RAM).

http://gitweb.heh.fi/?p=ion/ubuntu/compcache.git;a=blob;f=debian/compcache-utils.templates;hb=ubuntu

Revision history for this message
John McCabe-Dansted (gmatht) wrote :

OK, I have updated this debdiff with compcache-0.3.

Unfortunately I am having difficulty creating a working intrepid base.tgz, so I do not know if it actually builds correctly on Intrepid.

It builds with debuild on hardy.

FYI, this is the error.
$ sudo pbuilder create --distribution intrepid --othermirror "deb http://archive.ubuntu.com/ubuntu intrepid universe multiverse"
...
Building dependency tree... Done
You might want to run `apt-get -f install' to correct these.
The following packages have unmet dependencies:
  aptitude: Depends: libapt-pkg-libc6.7-6-4.6
  libept0: Depends: libapt-pkg-libc6.7-6-4.6
E: Unmet dependencies. Try using -f.

Revision history for this message
Johan Kiviniemi (ion) wrote :

I uploaded the compcache-utils package to REVU: http://revu.tauware.de/details.py?package=compcache

The package contains an init script to start compcache early and a debconf script to configure the size.

Of course, that’s just the userspace portion and this bug is still pending on getting the compcache kernel patch to Ubuntu.

Revision history for this message
John McCabe-Dansted (gmatht) wrote :

Note there is now a improved version of Compcache available, "better than compcache-0.3 in terms of space efficiency, speed and also has lot of cleanups". However this new version of compcache is a kernel patch rather than a module.

See: http://code.google.com/p/compcache/

Revision history for this message
Johan Kiviniemi (ion) wrote :

Good point. I removed the kernel module source altogether from the package, now it only contains the system scripts.

http://revu.tauware.de/details.py?package=compcache-debian

Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

Migrating this forward to the actively developed "linux" kernel package since linux-ubuntu-modules was merged with the "linux" kernel package for Intrepid.

Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

Additionally, this looks like it may be resolved with Intrepid since I'm seeing the following. Care to test and confirm. Thanks.

ogasawara@emiko:~$ modinfo compcache
filename: /lib/modules/2.6.27-7-generic/kernel/ubuntu/compcache/compcache.ko
description: Compressed RAM Based Swap Device
author: Nitin Gupta <email address hidden>
license: GPL
srcversion: EA8EA5496ED1E0C3A19D7B8
depends: lzo_decompress,tlsf,lzo_compress
vermagic: 2.6.27-7-generic SMP mod_unload modversions 586
parm: compcache_size_kbytes:compcache device size (in KB) (ulong)

Changed in linux:
status: Triaged → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote : Kernel team bugs

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.

Revision history for this message
Shirish Agarwal (shirishag75) wrote :

Got the same thing in jaunty .

$ modinfo compcache
filename: /lib/modules/2.6.28-4-generic/kernel/ubuntu/compcache/compcache.ko
description: Compressed RAM Based Swap Device
author: Nitin Gupta <email address hidden>
license: GPL
srcversion: EA8EA5496ED1E0C3A19D7B8
depends: lzo_decompress,tlsf,lzo_compress
vermagic: 2.6.28-4-generic SMP mod_unload modversions 586
parm: compcache_size_kbytes:compcache device size (in KB) (ulong)

Revision history for this message
juancarlospaco (juancarlospaco) wrote :

juancarlospaco@ibex:~$ lsb_release --release --codename ; echo " " ; cat /etc/initramfs-tools/initramfs.conf | grep --ignore-case Compcache ; sudo apt-cache search compcache ; dpkg -l | grep --ignore-case compcache ; dpkg -l | grep --ignore-case comp-cache
Release: 8.10
Codename: intrepid

# COMPCACHE_SIZE: [ "x K" | "x M" | "x G" | "x %" ]
# An empty value - compcache isn't used, or added to the initramfs at all.
# You can optionally install the compcache package to configure this setting
# via debconf and have userspace scripts to load and unload compcache.
COMPCACHE_SIZE=""
juancarlospaco@ibex:~$

so..., Compcache is missing from the Repos!!!!!!!

Revision history for this message
Andy Whitcroft (apw) wrote :

Since Intrepid at least the compcache module have been included in the Kernel packages. The kernel side of this issue is therefore closed. The bug discription only mentions the kernel so presumably the entire bug is closed for the development releases. Moving Fix Released.

Changed in linux (Ubuntu):
assignee: nobody → Andy Whitcroft (apw)
status: Incomplete → Fix Released
Revision history for this message
AhronZombi (ahron) wrote :

I need to add compcache to a desktop install, not a live cd. its ubuntu 9.10 . anyone know where i can get a repo with debs for that?

Revision history for this message
John McCabe-Dansted (gmatht) wrote :

@Ahron

I think this question would be best asked somewhere like
    https://answers.launchpad.net/

However I'll note that I don't think that compcache-0.6 is packaged as a deb anywhere, although it is trivial to install from the .tar.gz.

I'd suggest submitting a bug requesting that compcache be updated for Lucid.

In the meanwhile, compcache-0.5 appears to be in 9.10 by default (no need to apt-get anything), although without the wrapper scripts. You can enable it by doing something like:

$ sudo modprobe -q --ignore-install ramzswap disksize_kb="50"
$ sudo swapon /dev/ramzswap0

You can then disable it by doing:

$ sudo swapoff /dev/ramzswap0
$ sudo rmmod ramzswap

Revision history for this message
albertoj.diaz (albertoj-diaz) wrote :

Please apply the notify patch, which allows freeing the unused memory.

Revision history for this message
John McCabe-Dansted (gmatht) wrote :

@albertoj.diaz

The notify patch will be in Maverick (Because Maverick will use 2.6.35 and the 2.6.35 upstream will have the notify patch). Someone will need to update the Ubuntu userspace to use the new userspace API though.

Revision history for this message
Eugene San (eugenesan) wrote :

Due to changes upstream, previous solution must be reworked for Maverick.
Should we reopen this bug or create new?

Refer to:
https://bugs.edge.launchpad.net/ubuntu/+source/linux/+bug/381059
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=drivers/staging/ramzswap/ramzswap.txt

Also (for near future, in case patches will land in 2.6.35):
http://kerneltrap.org/mailarchive/linux-kernel/2010/6/1/4577590

tags: added: initramfs-tools ubiquity
Revision history for this message
Jeremy Foshee (jeremyfoshee) wrote :

Eugene,
    I'd prefer a new bug that way we have distinct threads from both times it was reworked. I defer to Andy on his personal preference, but I suspect he will say much the same. As an aside, my main reason for saying this is the fact that it will be reworked based on a different kernel than this one. :)

Thanks!

~JFo

Revision history for this message
Oibaf (oibaf) wrote :

This was replaced by zram, see zram-config:
https://launchpad.net/ubuntu/+source/zram-config

affects: linux-ubuntu-modules-2.6.24 (Baltix) → baltix
Changed in baltix:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.