[SRU] memory leaks in apache2 when running mod_ssl

Hi,

I've got the same bug. I've tried disabling mod_ssl without any success, the memory leak is still happening... Do you have an idea ? Maybe it's not/not only SSL after all ?

BTW, here's some debug information from /var/log/kern.log :

May 3 19:51:22 yoda kernel: [10632.509889] Out of memory: kill process 9047 (apache2) score 19040 or a child
May 3 19:51:22 yoda kernel: [10632.510045] Killed process 9047 (apache2)
May 3 19:51:22 yoda kernel: [10634.310259] named invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
May 3 19:51:22 yoda kernel: [10634.310291] Pid: 4539, comm: named Not tainted 2.6.24-16-server #1
May 3 19:51:22 yoda kernel: [10634.310370] [oom_kill_process+0x10a/0x120] oom_kill_process+0x10a/0x120
May 3 19:51:22 yoda kernel: [10634.310460] [out_of_memory+0x167/0x1a0] out_of_memory+0x167/0x1a0
May 3 19:51:22 yoda kernel: [10634.310510] [agpgart:__alloc_pages+0x34c/0x380] __alloc_pages+0x34c/0x380
May 3 19:51:22 yoda kernel: [10634.310573] [filemap_fault+0x10a/0x420] filemap_fault+0x10a/0x420
May 3 19:51:22 yoda kernel: [10634.310638] [__do_fault+0x83/0x4c0] __do_fault+0x83/0x4c0
May 3 19:51:22 yoda kernel: [10634.310747] [handle_mm_fault+0x21b/0xb80] handle_mm_fault+0x21b/0xb80
May 3 19:51:22 yoda kernel: [10634.310778] [<c012b0c0>] default_wake_function+0x0/0x10
May 3 19:51:22 yoda kernel: [10634.310875] [sched_clock+0x13/0x30] sched_clock+0x13/0x30
May 3 19:51:22 yoda kernel: [10634.310914] [snd_pcm:getnstimeofday+0x34/0x9790] getnstimeofday+0x34/0xf0
May 3 19:51:22 yoda kernel: [10634.310966] [snd_pcm:ktime_get_ts+0x1e/0x4f0] ktime_get_ts+0x1e/0x60
May 3 19:51:22 yoda kernel: [10634.310996] [do_page_fault+0x143/0x900] do_page_fault+0x143/0x900
May 3 19:51:22 yoda kernel: [10634.311025] [sys_futex+0x98/0x120] sys_futex+0x98/0x120
May 3 19:51:22 yoda kernel: [10634.311054] [snd_seq:do_gettimeofday+0x34/0xa450] do_gettimeofday+0x34/0xe0
May 3 19:51:22 yoda kernel: [10634.311090] [sys_gettimeofday+0x28/0x80] sys_gettimeofday+0x28/0x80
May 3 19:51:22 yoda kernel: [10634.311118] [do_page_fault+0x0/0x900] do_page_fault+0x0/0x900
May 3 19:51:22 yoda kernel: [10634.311134] [error_code+0x72/0x78] error_code+0x72/0x78
May 3 19:51:22 yoda kernel: [10634.311173] [unix_create1+0x50/0x120] unix_create1+0x50/0x120
May 3 19:51:22 yoda kernel: [10634.311229] =======================
May 3 19:51:22 yoda kernel: [10634.311236] Mem-info:
May 3 19:51:22 yoda kernel: [10634.311243] DMA per-cpu:
May 3 19:51:22 yoda kernel: [10634.311253] CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
May 3 19:51:22 yoda kernel: [10634.311264] Normal per-cpu:
May 3 19:51:22 yoda kernel: [10634.311273] CPU 0: Hot: hi: 186, btch: 31 usd: 87 Cold: hi: 62, btch: 15 usd: 43
May 3 19:51:22 yoda kernel: [10634.311291] Active:87819 inactive:295 dirty:0 writeback:0 unstable:0
May 3 19:51:22 yoda kernel: [10634.311297] free:976 slab:3176 mapped:2 pagetables:1570 bounce:0
May 3 19:51:22 yoda kernel: [10634.311314] DMA free:1552kB min:104kB low:128kB high:156kB active:10208kB inactive:16kB present:16256kB pages_scanned:127731 all_unreclaimable? yes
May 3 19:51:22 yoda kernel: [10634.311327] lowmem_reserve[]: 0 364 364 364
May 3 19:51:22 yoda kernel: [10634.311347] Normal free:2352kB min:2384kB low:2980kB high:3576kB active:341068kB inactive:1164kB p...

Hello,

Can you provide your ssl configuration so I can reproduce this?
Please include the output of dpkg -l | grep apache.

Thanks
chuck

I've reproduced this in a Hardy VM with 128MB of memory.

Beyond the default server install, I installed:
apache2-mpm-worker ssl-cert

I enabled ssl with a2enmod.

I created a very simple ssl config for apache and enabled it with a2ensite.

Then, from the VM host, I ran:
time ab -n 100000 -c 20 -f tls1 https://192.168.122.74:443/

It locked up the VM. I could see apache2 consuming all virtual memory, and the load pegged at 20.87.

Interestingly, I couldn't shutdown or ctrl-alt-delete the VM to gracefully reset it to a sane state.

:-Dustin

Chuck-

SSL configuration:
NameVirtualHost *:443
<VirtualHost *:443>
 SSLEngine On
 SSLCertificateFile /etc/ssl/certs/ssl.pem
 DocumentRoot /var/www
 <Directory />
  Options Indexes
 </Directory>
</VirtualHost>

:-Dustin

The only way to reboot the machine safely is to use the Magic SysRq keys. For instance, I used : AltGr + SysRq + s, AltGr + SysRq + u and AltGr + SysRq + b which syncs the mounted file systems, remounts all the mounted file systems in Read Only mode and finally reboots the computer.

But I still can't prevent Apache from crashing the system even with SSL disabled.

Léobaillard-

I cannot reproduce this with SSL disabled.

Can you give very detailed steps for reproducing this problem without SSL?

:-Dustin

FWIW, I have also reproduced this problem in Gutsy as well, which uses openssl-0.9.8e-5ubuntu3.

However, Dapper-Feisty do not exhibit this problem. Feisty uses openssl-0.9.8c-4ubuntu0.2.

It seems that the regression was introduced sometime between openssl-0.9.8c and 0.9.8e.

:-Dustin

Upstream bug filed with Apache:
https://issues.apache.org/bugzilla/show_bug.cgi?id=44975

They suggested applying this patch:
http://svn.apache.org/viewvc?view=rev&revision=654119

Test packages available in my PPA.

This patch fixes half of the problem. It solves the out-of-memory errors, however, SSL connections are simply dropped when the server is flooded. Note that Feisty's apache2+mod_ssl is able to handle the flood on both the client and server without errors.

:-Dustin

Please disregard my last comment. Those packages did not actually have the patch applied due to some nuances with dpatch in the build. As a result those tests are invalid.

I have since cleanly applied the patch, built test packages, and tested.

These test have succeeded, with 100,000 SSL request being issued and completed successfully, without loss of SSL connection, and without out-of-memory errors.

I'm submitting the attached debdiff as the fix for this issue, and Chuck will be bringing for the SRU. The patch is the one suggested by Apache ( http://svn.apache.org/viewvc?view=rev&revision=654119 )

:-Dustin

The bug is in Apache, not OpenSSL.

Patch submitted.

Depending on the system load apache with apache-mpm-worker and mod_ssl enabled will cause ssl to run out of memory and crash. The following patch resolves this issue. It will be needed to be ported to intrepid since it is also vulnerable to this condition.

Steps to reproduce: (TEST CASE)

1. Install apache-mpm-worker and ssl-cert
2. Confgure the SSL cert according to https://help.ubuntu.com/8.04/serverguide/C/certificates-and-security.html.
3. Use the following config in your /etc/apache2/sites-enabled/default.

NameVirtualHost *:443
<VirtualHost *:443>
 SSLEngine On
 SSLCertificateFile /etc/ssl/certs/ssl.pem
 DocumentRoot /var/www
 <Directory />
  Options Indexes
 </Directory>
</VirtualHost>

4. Run the following command:

ab -n 100000 -c 20 -f tls1 https://<ip address>:443/

You should get OOM errors in a couple of minutes of running the test.

If you have any questions let me know.

Regards
chuck

2.2.8-4 with the patch has just been uploaded to Debian

Accepted into -proposed, please test and give feedback here

Since Intrepid is a sync now, I take the intrepid task.

Verification step 1: I set up apache2 in hardy final and successfully reproduced the OOM condition. I will test the updated package with that setup, too.

I upgraded to the hardy-proposed version and ran the hammering benchmark again. Memory was constantly acquired and released now and never got exhausted.

I did both tests on a 256 MB VM.

I synced the fixed Debian version to intrepid.

Tests done, unassigning.

Copied to hardy-updates.