And now (still 2.6.34-999) when the raids are syncing the initial behavier occur (Lost IRQ, dropping drives) ... [ 2085.811299] CE: hpet increased min_delta_ns to 7500 nsec [ 8050.011293] ata1: lost interrupt (Status 0x50) [ 8050.012395] sd 0:0:0:0: [sda] Unhandled error code [ 8050.013751] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8050.015121] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 5e 87 1b 00 00 08 00 [ 8050.016592] end_request: I/O error, dev sda, sector 6194971 [ 8050.017704] Buffer I/O error on device dm-1, logical block 750218 [ 8050.018609] lost page write due to I/O error on dm-1 [ 8080.040103] ata1: lost interrupt (Status 0x50) [ 8080.040995] sd 0:0:0:0: [sda] Unhandled error code [ 8080.041858] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8080.042828] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 5e 87 c3 00 00 08 00 [ 8080.043738] end_request: I/O error, dev sda, sector 6195139 [ 8080.044813] Buffer I/O error on device dm-1, logical block 750239 [ 8080.046276] lost page write due to I/O error on dm-1 [ 8111.043832] ata1: lost interrupt (Status 0x50) [ 8111.044933] sd 0:0:0:0: [sda] Unhandled error code [ 8111.046353] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8111.048085] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 00 5c 0f 13 00 00 08 00 [ 8111.049970] end_request: I/O error, dev sda, sector 6033171 [ 8142.010949] ata1: lost interrupt (Status 0x50) [ 8142.012058] sd 0:0:0:0: [sda] Unhandled error code [ 8142.013166] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8142.014373] sd 0:0:0:0: [sda] CDB: Read(10): 28 00 00 5c 0f 13 00 00 08 00 [ 8142.015971] end_request: I/O error, dev sda, sector 6033171 [ 8161.060308] INFO: task jbd2/dm-1-8:615 blocked for more than 120 seconds. [ 8161.061450] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8161.062595] jbd2/dm-1-8 D ffff880001e95780 0 615 2 0x00000000 [ 8161.063675] ffff880221d93d10 0000000000000046 0000000000000000 ffff880221d93fd8 [ 8161.064863] ffff880223f11700 0000000000015780 0000000000015780 ffff880221d93fd8 [ 8161.066282] 0000000000015780 ffff880221d93fd8 0000000000015780 ffff880223f11700 [ 8161.067496] Call Trace: [ 8161.068891] [] jbd2_journal_commit_transaction+0x1e7/0x11d0 [ 8161.070346] [] ? __switch_to+0xbb/0x2e0 [ 8161.071493] [] ? put_prev_entity+0x2e/0x80 [ 8161.072598] [] ? _raw_spin_lock_irqsave+0x34/0x50 [ 8161.073878] [] ? lock_timer_base+0x3b/0x70 [ 8161.073882] [] ? autoremove_wake_function+0x0/0x40 [ 8161.073886] [] kjournald2+0xc3/0x230 [ 8161.073889] [] ? autoremove_wake_function+0x0/0x40 [ 8161.073891] [] ? kjournald2+0x0/0x230 [ 8161.073894] [] kthread+0x96/0xa0 [ 8161.073897] [] kernel_thread_helper+0x4/0x10 [ 8161.073900] [] ? kthread+0x0/0xa0 [ 8161.073902] [] ? kernel_thread_helper+0x0/0x10 [ 8161.073918] INFO: task kjournald:1695 blocked for more than 120 seconds. [ 8161.073920] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8161.073921] kjournald D ffff880001e15780 0 1695 2 0x00000000 [ 8161.073924] ffff880225ef7d40 0000000000000046 ffff880225ef7cf0 ffff880225ef7fd8 [ 8161.073927] ffff8802252e5c00 0000000000015780 0000000000015780 ffff880225ef7fd8 [ 8161.073929] 0000000000015780 ffff880225ef7fd8 0000000000015780 ffff8802252e5c00 [ 8161.073931] Call Trace: [ 8161.073935] [] journal_commit_transaction+0x17a/0xdf0 [ 8161.073938] [] ? __switch_to+0xbb/0x2e0 [ 8161.073941] [] ? put_prev_entity+0x2e/0x80 [ 8161.073948] [] ? default_spin_lock_flags+0x9/0x10 [ 8161.073950] [] ? _raw_spin_lock_irqsave+0x34/0x50 [ 8161.073953] [] ? autoremove_wake_function+0x0/0x40 [ 8161.073956] [] ? try_to_del_timer_sync+0x51/0xe0 [ 8161.073958] [] ? _raw_spin_lock_irqsave+0x34/0x50 [ 8161.073961] [] kjournald+0xf3/0x260 [ 8161.073964] [] ? autoremove_wake_function+0x0/0x40 [ 8161.073966] [] ? kjournald+0x0/0x260 [ 8161.073969] [] kthread+0x96/0xa0 [ 8161.073971] [] kernel_thread_helper+0x4/0x10 [ 8161.073974] [] ? kthread+0x0/0xa0 [ 8161.073976] [] ? kernel_thread_helper+0x0/0x10 [ 8161.073980] INFO: task rsyslogd:1884 blocked for more than 120 seconds. [ 8161.073982] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8161.073983] rsyslogd D ffff880001f95780 0 1884 1 0x00000000 [ 8161.073985] ffff88021fe117a8 0000000000000086 ffff880215d27400 ffff88021fe11fd8 [ 8161.073988] ffff880225c2c500 0000000000015780 0000000000015780 ffff88021fe11fd8 [ 8161.073990] 0000000000015780 ffff88021fe11fd8 0000000000015780 ffff880225c2c500 [ 8161.073992] Call Trace: [ 8161.073995] [] io_schedule+0x47/0x70 [ 8161.073999] [] sync_buffer+0x45/0x50 [ 8161.074001] [] __wait_on_bit_lock+0x5a/0xb0 [ 8161.074003] [] ? sync_buffer+0x0/0x50 [ 8161.074006] [] ? sync_buffer+0x0/0x50 [ 8161.074008] [] out_of_line_wait_on_bit_lock+0x7d/0x90 [ 8161.074011] [] ? wake_bit_function+0x0/0x40 [ 8161.074016] [] __lock_buffer+0x2c/0x30 [ 8161.074018] [] do_get_write_access+0x48a/0x4b0 [ 8161.074021] [] journal_get_write_access+0x31/0x50 [ 8161.074027] [] __ext3_journal_get_write_access+0x34/0x70 [ 8161.074030] [] ext3_free_data+0x3a/0x130 [ 8161.074033] [] ? ext3_get_branch+0xb2/0x110 [ 8161.074035] [] ext3_free_branches+0x1c7/0x1d0 [ 8161.074038] [] ext3_truncate+0x34f/0x540 [ 8161.074041] [] ? ext3_orphan_add+0x1c4/0x1d0 [ 8161.074043] [] ext3_truncate_failed_write+0x29/0x30 [ 8161.074046] [] ext3_write_begin+0x1c5/0x210 [ 8161.074048] [] ? ext3_get_block+0x0/0x120 [ 8161.074052] [] generic_perform_write+0xc1/0x1a0 [ 8161.074055] [] ? __mark_inode_dirty+0x3a/0x190 [ 8161.074058] [] generic_file_buffered_write+0x65/0xa0 [ 8161.074060] [] __generic_file_aio_write+0x251/0x3f0 [ 8161.074063] [] ? ext3_mark_iloc_dirty+0x1d/0x30 [ 8161.074066] [] ? __ext3_journal_stop+0x34/0x60 [ 8161.074068] [] generic_file_aio_write+0x5c/0xb0 [ 8161.074073] [] do_sync_write+0xd5/0x120 [ 8161.074079] [] ? vfs_ioctl+0x3a/0xc0 [ 8161.074084] [] ? security_file_permission+0x16/0x20 [ 8161.074086] [] vfs_write+0xcc/0x1a0 [ 8161.074089] [] sys_write+0x55/0x90 [ 8161.074091] [] system_call_fastpath+0x16/0x1b [ 8161.074098] INFO: task flush-252:1:2196 blocked for more than 120 seconds. [ 8161.074099] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8161.074100] flush-252:1 D ffff880001e95780 0 2196 2 0x00000000 [ 8161.074103] ffff8802247c16b0 0000000000000046 ffff88021fef4d20 ffff8802247c1fd8 [ 8161.074105] ffff88022462ae00 0000000000015780 0000000000015780 ffff8802247c1fd8 [ 8161.074107] 0000000000015780 ffff8802247c1fd8 0000000000015780 ffff88022462ae00 [ 8161.074110] Call Trace: [ 8161.074112] [] io_schedule+0x47/0x70 [ 8161.074117] [] get_request_wait+0xd0/0x170 [ 8161.074121] [] ? cfq_find_rq_fmerge+0x65/0x80 [ 8161.074124] [] ? autoremove_wake_function+0x0/0x40 [ 8161.074127] [] ? elv_merge+0xf3/0x110 [ 8161.074130] [] __make_request+0x8f/0x480 [ 8161.074132] [] ? down_read+0x16/0x30 [ 8161.074135] [] generic_make_request+0x1dc/0x3f0 [ 8161.074137] [] ? mempool_alloc+0x6d/0x140 [ 8161.074140] [] ? block_write_full_page_endio+0x116/0x120 [ 8161.074144] [] ? noalloc_get_block_write+0x0/0x60 [ 8161.074146] [] submit_bio+0x7a/0x100 [ 8161.074149] [] submit_bh+0x101/0x150 [ 8161.074152] [] __block_write_full_page+0x1c5/0x340 [ 8161.074154] [] ? end_buffer_async_write+0x0/0x160 [ 8161.074156] [] ? noalloc_get_block_write+0x0/0x60 [ 8161.074159] [] ? end_buffer_async_write+0x0/0x160 [ 8161.074161] [] block_write_full_page_endio+0x116/0x120 [ 8161.074164] [] ? noalloc_get_block_write+0x0/0x60 [ 8161.074166] [] block_write_full_page+0x15/0x20 [ 8161.074169] [] ext4_writepage+0x1bd/0x290 [ 8161.074171] [] mpage_da_submit_io+0x13b/0x1a0 [ 8161.074174] [] ext4_da_writepages+0x3e5/0x590 [ 8161.074177] [] ? __writepage+0x0/0x40 [ 8161.074180] [] ? bit_waitqueue+0x17/0xd0 [ 8161.074183] [] do_writepages+0x21/0x40 [ 8161.074185] [] writeback_single_inode+0xdb/0x250 [ 8161.074188] [] writeback_sb_inodes+0xdd/0x160 [ 8161.074191] [] writeback_inodes_wb+0xd2/0x110 [ 8161.074193] [] wb_writeback+0x155/0x210 [ 8161.074195] [] ? _raw_spin_lock_irqsave+0x34/0x50 [ 8161.074198] [] wb_check_old_data_flush+0xa5/0xb0 [ 8161.074201] [] ? process_timeout+0x0/0x10 [ 8161.074203] [] wb_do_writeback+0xaa/0xc0 [ 8161.074206] [] bdi_writeback_task+0x43/0xc0 [ 8161.074209] [] ? bdi_start_fn+0x0/0x100 [ 8161.074211] [] bdi_start_fn+0x8c/0x100 [ 8161.074214] [] ? bdi_start_fn+0x0/0x100 [ 8161.074216] [] kthread+0x96/0xa0 [ 8161.074219] [] kernel_thread_helper+0x4/0x10 [ 8161.074221] [] ? kthread+0x0/0xa0 [ 8161.074224] [] ? kernel_thread_helper+0x0/0x10 [ 8161.074226] INFO: task flush-252:3:2197 blocked for more than 120 seconds. [ 8161.074227] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8161.074228] flush-252:3 D ffff880001e95780 0 2197 2 0x00000000 [ 8161.074231] ffff8802259af790 0000000000000046 ffff88021fef4d20 ffff8802259affd8 [ 8161.074233] ffff880224629700 0000000000015780 0000000000015780 ffff8802259affd8 [ 8161.074235] 0000000000015780 ffff8802259affd8 0000000000015780 ffff880224629700 [ 8161.074237] Call Trace: [ 8161.074239] [] io_schedule+0x47/0x70 [ 8161.074242] [] get_request_wait+0xd0/0x170 [ 8161.074245] [] ? cfq_find_rq_fmerge+0x65/0x80 [ 8161.074248] [] ? autoremove_wake_function+0x0/0x40 [ 8161.074250] [] ? elv_merge+0xf3/0x110 [ 8161.074252] [] __make_request+0x8f/0x480 [ 8161.074255] [] ? down_read+0x16/0x30 [ 8161.074257] [] generic_make_request+0x1dc/0x3f0 [ 8161.074260] [] ? mempool_alloc+0x6d/0x140 [ 8161.074262] [] submit_bio+0x7a/0x100 [ 8161.074265] [] submit_bh+0x101/0x150 [ 8161.074267] [] __block_write_full_page+0x1c5/0x340 [ 8161.074270] [] ? end_buffer_async_write+0x0/0x160 [ 8161.074273] [] ? blkdev_get_block+0x0/0x70 [ 8161.074275] [] ? end_buffer_async_write+0x0/0x160 [ 8161.074278] [] block_write_full_page_endio+0x116/0x120 [ 8161.074280] [] ? blkdev_get_block+0x0/0x70 [ 8161.074283] [] block_write_full_page+0x15/0x20 [ 8161.074285] [] blkdev_writepage+0x18/0x20 [ 8161.074288] [] __writepage+0x17/0x40 [ 8161.074290] [] write_cache_pages+0x267/0x3b0 [ 8161.074293] [] ? __writepage+0x0/0x40 [ 8161.074296] [] generic_writepages+0x24/0x30 [ 8161.074298] [] do_writepages+0x21/0x40 [ 8161.074301] [] writeback_single_inode+0xdb/0x250 [ 8161.074303] [] writeback_sb_inodes+0xdd/0x160 [ 8161.074306] [] writeback_inodes_wb+0xd2/0x110 [ 8161.074308] [] wb_writeback+0x155/0x210 [ 8161.074311] [] ? _raw_spin_lock_irqsave+0x34/0x50 [ 8161.074313] [] wb_check_old_data_flush+0xa5/0xb0 [ 8161.074316] [] ? process_timeout+0x0/0x10 [ 8161.074318] [] wb_do_writeback+0xaa/0xc0 [ 8161.074321] [] bdi_writeback_task+0x43/0xc0 [ 8161.074323] [] ? bdi_start_fn+0x0/0x100 [ 8161.074325] [] bdi_start_fn+0x8c/0x100 [ 8161.074328] [] ? bdi_start_fn+0x0/0x100 [ 8161.074330] [] kthread+0x96/0xa0 [ 8161.074333] [] kernel_thread_helper+0x4/0x10 [ 8161.074335] [] ? kthread+0x0/0xa0 [ 8161.074338] [] ? kernel_thread_helper+0x0/0x10 [ 8161.074360] INFO: task nagios3:4031 blocked for more than 120 seconds. [ 8161.074361] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8161.074363] nagios3 D ffff880001f95780 0 4031 1 0x00000000 [ 8161.074365] ffff880224b6dd58 0000000000000082 0000000224b6de38 ffff880224b6dfd8 [ 8161.074367] ffff88021ff1dc00 0000000000015780 0000000000015780 ffff880224b6dfd8 [ 8161.074370] 0000000000015780 ffff880224b6dfd8 0000000000015780 ffff88021ff1dc00 [ 8161.074372] Call Trace: [ 8161.074375] [] start_this_handle+0x1cc/0x450 [ 8161.074378] [] ? autoremove_wake_function+0x0/0x40 [ 8161.074380] [] ? filldir+0x0/0xf0 [ 8161.074383] [] ? filldir+0x0/0xf0 [ 8161.074385] [] jbd2_journal_start+0x74/0xb0 [ 8161.074389] [] ext4_journal_start_sb+0x57/0x70 [ 8161.074392] [] ext4_dirty_inode+0x2a/0x60 [ 8161.074394] [] __mark_inode_dirty+0x3a/0x190 [ 8161.074397] [] touch_atime+0xe8/0x100 [ 8161.074400] [] vfs_readdir+0xcc/0xd0 [ 8161.074402] [] sys_getdents+0x8a/0xe0 [ 8161.074405] [] system_call_fastpath+0x16/0x1b [ 8172.040132] ata1: lost interrupt (Status 0x50) [ 8172.040958] sd 0:0:0:0: [sda] Unhandled error code [ 8172.041805] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8172.043032] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 23 73 bb 00 00 08 00 [ 8172.044503] end_request: I/O error, dev sda, sector 2323387 [ 8172.045685] Buffer I/O error on device dm-1, logical block 266270 [ 8172.047060] lost page write due to I/O error on dm-1 [ 8203.040073] ata1: lost interrupt (Status 0x50) [ 8203.040966] sd 0:0:0:0: [sda] Unhandled error code [ 8203.042013] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8203.043382] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 23 73 eb 00 00 08 00 [ 8203.044962] end_request: I/O error, dev sda, sector 2323435 [ 8203.046451] Buffer I/O error on device dm-1, logical block 266276 [ 8203.047575] lost page write due to I/O error on dm-1 [ 8234.040061] ata1: lost interrupt (Status 0x50) [ 8234.041117] sd 0:0:0:0: [sda] Unhandled error code [ 8234.042489] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8234.044019] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 23 74 13 00 00 08 00 [ 8234.045248] end_request: I/O error, dev sda, sector 2323475 [ 8234.046376] Buffer I/O error on device dm-1, logical block 266281 [ 8234.047464] lost page write due to I/O error on dm-1 [ 8265.040451] ata1: lost interrupt (Status 0x50) [ 8265.042112] sd 0:0:0:0: [sda] Unhandled error code [ 8265.043694] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8265.045454] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 25 27 1b 00 00 10 00 [ 8265.046746] end_request: I/O error, dev sda, sector 2434843 [ 8265.048040] Buffer I/O error on device dm-1, logical block 280202 [ 8265.049336] lost page write due to I/O error on dm-1 [ 8265.050692] Buffer I/O error on device dm-1, logical block 280203 [ 8265.052139] lost page write due to I/O error on dm-1 [ 8281.071251] INFO: task jbd2/dm-1-8:615 blocked for more than 120 seconds. [ 8281.072269] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8281.073736] jbd2/dm-1-8 D ffff880001e95780 0 615 2 0x00000000 [ 8281.075071] ffff880221d93d10 0000000000000046 0000000000000000 ffff880221d93fd8 [ 8281.076577] ffff880223f11700 0000000000015780 0000000000015780 ffff880221d93fd8 [ 8281.077674] 0000000000015780 ffff880221d93fd8 0000000000015780 ffff880223f11700 [ 8281.078771] Call Trace: [ 8281.079815] [] jbd2_journal_commit_transaction+0x1e7/0x11d0 [ 8281.080903] [] ? __switch_to+0xbb/0x2e0 [ 8281.081940] [] ? put_prev_entity+0x2e/0x80 [ 8281.083012] [] ? _raw_spin_lock_irqsave+0x34/0x50 [ 8281.084074] [] ? lock_timer_base+0x3b/0x70 [ 8281.085120] [] ? autoremove_wake_function+0x0/0x40 [ 8281.086169] [] kjournald2+0xc3/0x230 [ 8281.087213] [] ? autoremove_wake_function+0x0/0x40 [ 8281.088273] [] ? kjournald2+0x0/0x230 [ 8281.089333] [] kthread+0x96/0xa0 [ 8281.090401] [] kernel_thread_helper+0x4/0x10 [ 8281.091485] [] ? kthread+0x0/0xa0 [ 8281.091488] [] ? kernel_thread_helper+0x0/0x10 [ 8281.091495] INFO: task kjournald:1695 blocked for more than 120 seconds. [ 8281.091496] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8281.091498] kjournald D ffff880001e15780 0 1695 2 0x00000000 [ 8281.091500] ffff880225ef7d40 0000000000000046 ffff880225ef7cf0 ffff880225ef7fd8 [ 8281.091503] ffff8802252e5c00 0000000000015780 0000000000015780 ffff880225ef7fd8 [ 8281.091505] 0000000000015780 ffff880225ef7fd8 0000000000015780 ffff8802252e5c00 [ 8281.091507] Call Trace: [ 8281.091510] [] journal_commit_transaction+0x17a/0xdf0 [ 8281.091513] [] ? __switch_to+0xbb/0x2e0 [ 8281.091515] [] ? put_prev_entity+0x2e/0x80 [ 8281.091519] [] ? default_spin_lock_flags+0x9/0x10 [ 8281.091522] [] ? _raw_spin_lock_irqsave+0x34/0x50 [ 8281.091525] [] ? autoremove_wake_function+0x0/0x40 [ 8281.091527] [] ? try_to_del_timer_sync+0x51/0xe0 [ 8281.091530] [] ? _raw_spin_lock_irqsave+0x34/0x50 [ 8281.091532] [] kjournald+0xf3/0x260 [ 8281.091535] [] ? autoremove_wake_function+0x0/0x40 [ 8281.091538] [] ? kjournald+0x0/0x260 [ 8281.091540] [] kthread+0x96/0xa0 [ 8281.091543] [] kernel_thread_helper+0x4/0x10 [ 8281.091545] [] ? kthread+0x0/0xa0 [ 8281.091547] [] ? kernel_thread_helper+0x0/0x10 [ 8281.091551] INFO: task rsyslogd:1884 blocked for more than 120 seconds. [ 8281.091552] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8281.091553] rsyslogd D ffff880001f95780 0 1884 1 0x00000000 [ 8281.091556] ffff88021fe117a8 0000000000000086 ffff880215d27400 ffff88021fe11fd8 [ 8281.091558] ffff880225c2c500 0000000000015780 0000000000015780 ffff88021fe11fd8 [ 8281.091560] 0000000000015780 ffff88021fe11fd8 0000000000015780 ffff880225c2c500 [ 8281.091562] Call Trace: [ 8281.091565] [] io_schedule+0x47/0x70 [ 8281.091569] [] sync_buffer+0x45/0x50 [ 8281.091571] [] __wait_on_bit_lock+0x5a/0xb0 [ 8281.091573] [] ? sync_buffer+0x0/0x50 [ 8281.091576] [] ? sync_buffer+0x0/0x50 [ 8281.091578] [] out_of_line_wait_on_bit_lock+0x7d/0x90 [ 8281.091581] [] ? wake_bit_function+0x0/0x40 [ 8281.091584] [] __lock_buffer+0x2c/0x30 [ 8281.091587] [] do_get_write_access+0x48a/0x4b0 [ 8281.091589] [] journal_get_write_access+0x31/0x50 [ 8281.091593] [] __ext3_journal_get_write_access+0x34/0x70 [ 8281.091596] [] ext3_free_data+0x3a/0x130 [ 8281.091598] [] ? ext3_get_branch+0xb2/0x110 [ 8281.091601] [] ext3_free_branches+0x1c7/0x1d0 [ 8281.091604] [] ext3_truncate+0x34f/0x540 [ 8281.091606] [] ? ext3_orphan_add+0x1c4/0x1d0 [ 8281.091609] [] ext3_truncate_failed_write+0x29/0x30 [ 8281.091611] [] ext3_write_begin+0x1c5/0x210 [ 8281.091614] [] ? ext3_get_block+0x0/0x120 [ 8281.091617] [] generic_perform_write+0xc1/0x1a0 [ 8281.091621] [] ? __mark_inode_dirty+0x3a/0x190 [ 8281.091623] [] generic_file_buffered_write+0x65/0xa0 [ 8281.091626] [] __generic_file_aio_write+0x251/0x3f0 [ 8281.091628] [] ? ext3_mark_iloc_dirty+0x1d/0x30 [ 8281.091631] [] ? __ext3_journal_stop+0x34/0x60 [ 8281.091634] [] generic_file_aio_write+0x5c/0xb0 [ 8281.091637] [] do_sync_write+0xd5/0x120 [ 8281.091640] [] ? vfs_ioctl+0x3a/0xc0 [ 8281.091643] [] ? security_file_permission+0x16/0x20 [ 8281.091646] [] vfs_write+0xcc/0x1a0 [ 8281.091648] [] sys_write+0x55/0x90 [ 8281.091651] [] system_call_fastpath+0x16/0x1b [ 8281.091653] INFO: task rsyslogd:1887 blocked for more than 120 seconds. [ 8281.091654] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8281.091656] rsyslogd D ffff880001f95780 0 1887 1 0x00000000 [ 8281.091658] ffff88021fe39c98 0000000000000086 ffff880225ce9700 ffff88021fe39fd8 [ 8281.091660] ffff880225ce9700 0000000000015780 0000000000015780 ffff88021fe39fd8 [ 8281.091662] 0000000000015780 ffff88021fe39fd8 0000000000015780 ffff880225ce9700 [ 8281.091664] Call Trace: [ 8281.091667] [] start_this_handle+0x1b6/0x3e0 [ 8281.091670] [] ? autoremove_wake_function+0x0/0x40 [ 8281.091672] [] journal_start+0x74/0xb0 [ 8281.091675] [] ext3_journal_start_sb+0x4f/0x60 [ 8281.091678] [] ext3_dirty_inode+0x3d/0xa0 [ 8281.091680] [] __mark_inode_dirty+0x3a/0x190 [ 8281.091683] [] inode_setattr+0x7c/0x170 [ 8281.091686] [] ext3_setattr+0x112/0x240 [ 8281.091688] [] notify_change+0x179/0x310 [ 8281.091691] [] chown_common+0x98/0xc0 [ 8281.091693] [] sys_fchown+0x98/0xc0 [ 8281.091696] [] system_call_fastpath+0x16/0x1b [ 8296.040039] ata1: lost interrupt (Status 0x50) [ 8296.040954] sd 0:0:0:0: [sda] Unhandled error code [ 8296.041811] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8296.042724] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 25 27 bb 00 00 08 00 [ 8296.044216] end_request: I/O error, dev sda, sector 2435003 [ 8296.045715] Buffer I/O error on device dm-1, logical block 280222 [ 8296.047430] lost page write due to I/O error on dm-1 [ 8327.040037] ata1: lost interrupt (Status 0x50) [ 8327.041230] sd 0:0:0:0: [sda] Unhandled error code [ 8327.042897] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8327.044609] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 25 27 fb 00 00 08 00 [ 8327.046003] end_request: I/O error, dev sda, sector 2435067 [ 8327.047078] Buffer I/O error on device dm-1, logical block 280230 [ 8327.048147] lost page write due to I/O error on dm-1 [ 8358.040057] ata1: lost interrupt (Status 0x50) [ 8358.041244] sd 0:0:0:0: [sda] Unhandled error code [ 8358.042272] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8358.043339] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 25 28 23 00 00 08 00 [ 8358.044398] end_request: I/O error, dev sda, sector 2435107 [ 8358.045399] Buffer I/O error on device dm-1, logical block 280235 [ 8358.046568] lost page write due to I/O error on dm-1 [ 8389.040043] ata1: lost interrupt (Status 0x50) [ 8389.041064] sd 0:0:0:0: [sda] Unhandled error code [ 8389.042040] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8389.043055] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 26 da d3 00 00 08 00 [ 8389.044079] end_request: I/O error, dev sda, sector 2546387 [ 8389.045146] Buffer I/O error on device dm-1, logical block 294145 [ 8389.046705] lost page write due to I/O error on dm-1 [ 8420.040043] ata1: lost interrupt (Status 0x50) [ 8420.041484] sd 0:0:0:0: [sda] Unhandled error code [ 8420.043109] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8420.044625] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 26 da f3 00 00 08 00 [ 8420.045746] end_request: I/O error, dev sda, sector 2546419 [ 8420.046781] Buffer I/O error on device dm-1, logical block 294149 [ 8420.047868] lost page write due to I/O error on dm-1 [ 8451.040050] ata1: lost interrupt (Status 0x50) [ 8451.041288] sd 0:0:0:0: [sda] Unhandled error code [ 8451.042557] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8451.043795] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 26 db 03 00 00 08 00 [ 8451.045299] end_request: I/O error, dev sda, sector 2546435 [ 8451.046669] Buffer I/O error on device dm-1, logical block 294151 [ 8451.047880] lost page write due to I/O error on dm-1 [ 8482.040048] ata1: lost interrupt (Status 0x50) [ 8482.041454] sd 0:0:0:0: [sda] Unhandled error code [ 8482.043283] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8482.045019] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 26 db 23 00 00 08 00 [ 8482.046417] end_request: I/O error, dev sda, sector 2546467 [ 8482.047557] Buffer I/O error on device dm-1, logical block 294155 [ 8482.048852] lost page write due to I/O error on dm-1 [ 8513.011293] ata1: lost interrupt (Status 0x50) [ 8513.012616] sd 0:0:0:0: [sda] Unhandled error code [ 8513.013995] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8513.015583] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 26 db bb 00 00 08 00 [ 8513.017064] end_request: I/O error, dev sda, sector 2546619 [ 8513.018275] Buffer I/O error on device dm-1, logical block 294174 [ 8513.019508] lost page write due to I/O error on dm-1 [ 8543.041376] ata1: lost interrupt (Status 0x50) [ 8543.042643] sd 0:0:0:0: [sda] Unhandled error code [ 8543.043859] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8543.045648] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 26 db db 00 00 08 00 [ 8543.047630] end_request: I/O error, dev sda, sector 2546651 [ 8543.049434] Buffer I/O error on device dm-1, logical block 294178 [ 8543.050928] lost page write due to I/O error on dm-1 [ 8574.040180] ata1: lost interrupt (Status 0x50) [ 8574.041671] sd 0:0:0:0: [sda] Unhandled error code [ 8574.043357] sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [ 8574.045391] sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 00 48 91 cb 00 00 b0 00 [ 8574.046957] end_request: I/O error, dev sda, sector 4755915 [ 8574.048199] Aborting journal on device dm-1-8. [ 8574.049756] EXT4-fs error (device dm-1) in ext4_orphan_add: Journal has aborted [ 8574.050325] EXT4-fs error (device dm-1): ext4_journal_start_sb: Detected aborted journal [ 8574.050327] EXT4-fs (dm-1): Remounting filesystem read-only [ 8574.053432] EXT4-fs error (device dm-1) in add_dirent_to_buf: Journal has aborted [ 8574.055376] EXT4-fs error (device dm-1) in ext4_delete_inode: Journal has aborted