We are experiencing the same issue. Fresh install of Ubuntu 10.10LTS with default apache2, mysql, php. I can say for certain that it's not a hardware issue. This is a virtual server running on an ESX host. Let me know if there are any other details I can pull to help with this issue. This is a production server and it crashes with this OOM about once a week.
We are experiencing the same issue. Fresh install of Ubuntu 10.10LTS with default apache2, mysql, php. I can say for certain that it's not a hardware issue. This is a virtual server running on an ESX host. Let me know if there are any other details I can pull to help with this issue. This is a production server and it crashes with this OOM about once a week.
Apache2 Loaded Modules:
core mod_log_config mod_logio prefork http_core mod_so mod_alias mod_auth_basic mod_authn_file mod_authz_default mod_authz_groupfile mod_authz_host mod_authz_user mod_autoindex mod_cgi mod_deflate mod_dir mod_env mod_mime mod_negotiation mod_php5 mod_reqtimeout mod_rewrite mod_setenvif mod_status
PHP Version: 5.3.3-1ubuntu9.5
The default also came with Suhosin Patch 0.9.9.1.
root@public- lamp:/var/ log# mysql --version
mysql Ver 14.14 Distrib 5.1.49, for debian-linux-gnu (x86_64) using readline 6.1
2.6.35-30-server #59-Ubuntu SMP 64-bit
Sep 21 12:31:32 public-lamp kernel: [1707807.719708] apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 331>] dump_header+ 0x81/0xc0 3f1>] oom_kill_ process+ 0x81/0x180 928>] __out_of_ memory+ 0x58/0xd0 a26>] out_of_ memory+ 0x86/0x1c0 48e>] __alloc_ pages_slowpath+ 0x58e/0x5a0 60c>] __alloc_ pages_nodemask+ 0x16c/0x1d0 9ca>] alloc_pages_ current+ 0x9a/0x100 a37>] __page_ cache_alloc+ 0x87/0x90 c09>] __do_page_ cache_readahead +0xc9/0x210 d71>] ra_submit+0x21/0x30 125>] ondemand_ readahead+ 0x115/0x240 219>] ? __wake_ up_common+ 0x59/0x90 351>] page_cache_ sync_readahead+ 0x31/0x50 259>] filemap_ fault+0x419/ 0x450 304>] __do_fault+ 0x54/0x560 b69>] handle_ mm_fault+ 0x1b9/0x440 d20>] ? mntput_ no_expire+ 0x30/0x110 a75>] do_page_ fault+0x125/ 0x350 5f5>] page_fault+ 0x25/0x30 anon:236889 isolated_anon:0 e:2483 slab_unreclaima ble:5104 e:0kB slab_unreclaima ble:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no anon:2399240kB inactive_ anon:599996kB active_file:212kB inactive_file:688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:168kB mapped:156kB shmem:4kB slab_reclaimabl e:652kB slab_unreclaima ble:5820kB kernel_stack:1240kB pagetables:25256kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no anon:1389800kB inactive_ anon:347560kB active_file:1124kB inactive_ file:1708kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3102720kB mlocked:0kB dirty:0kB writeback:92kB mapped:1132kB shmem:36kB slab_reclaimabl e:9280kB slab_unreclaima ble:14580kB kernel_stack:1224kB pagetables:24352kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
Sep 21 12:31:32 public-lamp kernel: [1707807.797392] apache2 cpuset=/ mems_allowed=0
Sep 21 12:31:32 public-lamp kernel: [1707807.824987] Pid: 16345, comm: apache2 Not tainted 2.6.35-30-server #56-Ubuntu
Sep 21 12:31:32 public-lamp kernel: [1707807.824991] Call Trace:
Sep 21 12:31:32 public-lamp kernel: [1707807.877827] [<ffffffff81105
Sep 21 12:31:32 public-lamp kernel: [1707807.995049] [<ffffffff81105
Sep 21 12:31:32 public-lamp kernel: [1707807.995053] [<ffffffff81105
Sep 21 12:31:32 public-lamp kernel: [1707807.995057] [<ffffffff81105
Sep 21 12:31:32 public-lamp kernel: [1707807.995061] [<ffffffff81109
Sep 21 12:31:32 public-lamp kernel: [1707807.995065] [<ffffffff81109
Sep 21 12:31:32 public-lamp kernel: [1707807.995075] [<ffffffff8113b
Sep 21 12:31:32 public-lamp kernel: [1707807.995079] [<ffffffff81102
Sep 21 12:31:32 public-lamp kernel: [1707807.995090] [<ffffffff8110c
Sep 21 12:31:32 public-lamp kernel: [1707807.995095] [<ffffffff8110c
Sep 21 12:31:32 public-lamp kernel: [1707807.995098] [<ffffffff8110d
Sep 21 12:31:32 public-lamp kernel: [1707808.004712] [<ffffffff81049
Sep 21 12:31:32 public-lamp kernel: [1707808.005747] [<ffffffff8110d
Sep 21 12:31:32 public-lamp kernel: [1707808.005751] [<ffffffff81104
Sep 21 12:31:32 public-lamp kernel: [1707808.005756] [<ffffffff8111f
Sep 21 12:31:32 public-lamp kernel: [1707808.005760] [<ffffffff81122
Sep 21 12:31:32 public-lamp kernel: [1707808.012723] [<ffffffff81170
Sep 21 12:31:32 public-lamp kernel: [1707808.022743] [<ffffffff815a6
Sep 21 12:31:32 public-lamp kernel: [1707808.033163] [<ffffffff815a3
Sep 21 12:31:32 public-lamp kernel: [1707808.033168] Mem-Info:
Sep 21 12:31:32 public-lamp kernel: [1707808.033171] Node 0 DMA per-cpu:
Sep 21 12:31:32 public-lamp kernel: [1707808.033174] CPU 0: hi: 0, btch: 1 usd: 0
Sep 21 12:31:32 public-lamp kernel: [1707808.033177] CPU 1: hi: 0, btch: 1 usd: 0
Sep 21 12:31:32 public-lamp kernel: [1707808.033179] Node 0 DMA32 per-cpu:
Sep 21 12:31:32 public-lamp kernel: [1707808.033182] CPU 0: hi: 186, btch: 31 usd: 32
Sep 21 12:31:32 public-lamp kernel: [1707808.033185] CPU 1: hi: 186, btch: 31 usd: 0
Sep 21 12:31:32 public-lamp kernel: [1707808.033187] Node 0 Normal per-cpu:
Sep 21 12:31:32 public-lamp kernel: [1707808.033189] CPU 0: hi: 186, btch: 31 usd: 18
Sep 21 12:31:32 public-lamp kernel: [1707808.033192] CPU 1: hi: 186, btch: 31 usd: 6
Sep 21 12:31:32 public-lamp kernel: [1707808.033246] active_anon:947260 inactive_
Sep 21 12:31:32 public-lamp kernel: [1707808.033248] active_file:334 inactive_file:562 isolated_file:0
Sep 21 12:31:33 public-lamp kernel: [1707808.033249] unevictable:0 dirty:0 writeback:65 unstable:0
Sep 21 12:31:33 public-lamp kernel: [1707808.033250] free:9539 slab_reclaimabl
Sep 21 12:31:33 public-lamp kernel: [1707808.033251] mapped:322 shmem:10 pagetables:12402 bounce:0
Sep 21 12:31:33 public-lamp kernel: [1707808.033254] Node 0 DMA free:15836kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15708kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimabl
Sep 21 12:31:33 public-lamp kernel: [1707808.040340] lowmem_reserve[]: 0 3000 6030 6030
Sep 21 12:31:33 public-lamp kernel: [1707808.040346] Node 0 DMA32 free:17168kB min:4936kB low:6168kB high:7404kB active_
Sep 21 12:31:33 public-lamp kernel: [1707808.040358] lowmem_reserve[]: 0 0 3030 3030
Sep 21 12:31:33 public-lamp kernel: [1707808.040983] Node 0 Normal free:5152kB min:4984kB low:6228kB high:7476kB active_
Sep 21 12:31:33 public-lamp kernel: [1707808.040996] lowmem_reserve[]: 0 0 0 0
Sep 21 12:31:33 public-lamp kernel: [1707808.041001] Node 0 DMA: 3*4kB 0*8kB 1*16kB 2*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15836kB
Sep 21 12:31:33 public-lamp kernel: [1707808.041021] Node 0 DMA32: 28*4kB 6*8kB 8*16kB 10*32kB 5*64kB 1*128kB 13*256kB 15*512kB 1*1024kB 0*2048kB 1*4096kB = 17184kB
Sep 21 12:31:33 public-lamp kernel: [1707808.041044] Node 0 Normal: 1236*4kB 14*8kB 6*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5152kB
Sep 21 12:31:33 public-lamp kernel: [1707808.041055] 40476 total pagecache pages
Sep 21 12:31:33 public-lamp kernel: [1707808.041057] 39519 pages in swap cache
Sep 21 12:31:33 public-lamp kernel: [1707808.041063] Swap cache stats: add 465967, delete 426448, find 30615/43793
Sep 21 12:31:33 public-lamp kernel: [1707808.041066] Free swap = 40kB
Sep 21 12:31:33 public-lamp kernel: [1707808.041067] Total swap = 1331196kB
Sep 21 12:31:33 public-lamp kernel: [1707813.575934] 1572848 pages RAM
Sep 21 12:31:33 public-lamp kernel: [1707813.575938] 41073 pages reserved
Sep 21 12:31:33 public-lamp kernel: [1707813.575940] 79247 pages shared
Sep 21 12:31:33 public-lamp kernel: [1707813.575942] 1520465 pages non-shared