I rebooted the system in the meantime. It now had Kernel 4.2.0-30.36 from previous updates:
cat /proc/version_signature Ubuntu 4.2.0-30.36~14.04.1-generic 4.2.8-ckt3 During the reboot, the system performed some disk I/O for several hours. Before services like sshd became available. I guess this disk operation happened before the file system was mounted in read-write mode. What I can say is that the system tried to continue the failed balance operation during the boot (from syslog) without intervention from my side: BTRFS info (device dm-0): continuing balance BTRFS info (device dm-0): relocating block group 9553211031552 flags 36 found 106150 extents The btrfs balance operation did not succeed again, but in a manner which I was used to before writing this bug report. This is MUCH better than going into an unpredictable state (ro mode with Input/Output errors). I guess the OS totally got out of sync. BTRFS info (device dm-0): 2 enospc errors during balance The btrfs file system was mounted in READ-WRITE-mode this time (not in read-only mode as before!). And this time, I can read the files and navigate through directories without any Input/Output errors (before the reboot, I had lots of them!). Finally, everything looks fine for a btrfs file system with no space left on the disk. Now that the fs is online and accessible, I can try to add another disk to the fs to have some empty space again. After that, I hopefully can delete some data and hopefully redo the btrfs balance operation. /var/log/syslog gives me: Mar 22 12:42:37 server kernel: [ 3481.725273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 22 12:42:37 server kernel: [ 3481.726122] mount D ffff88043e296640 0 587 455 0x00000000 Mar 22 12:42:37 server kernel: [ 3481.726125] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 Mar 22 12:42:37 server kernel: [ 3481.726127] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff Mar 22 12:42:37 server kernel: [ 3481.726129] ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 Mar 22 12:42:37 server kernel: [ 3481.726130] Call Trace: Mar 22 12:42:37 server kernel: [ 3481.726136] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 3481.726138] [<ffffffff817b8ac7>] schedule+0x37/0x80 Mar 22 12:42:37 server kernel: [ 3481.726140] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 Mar 22 12:42:37 server kernel: [ 3481.726143] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 Mar 22 12:42:37 server kernel: [ 3481.726146] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 Mar 22 12:42:37 server kernel: [ 3481.726147] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 3481.726149] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 Mar 22 12:42:37 server kernel: [ 3481.726151] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 Mar 22 12:42:37 server kernel: [ 3481.726152] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 Mar 22 12:42:37 server kernel: [ 3481.726155] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 Mar 22 12:42:37 server kernel: [ 3481.726157] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 Mar 22 12:42:37 server kernel: [ 3481.726159] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 Mar 22 12:42:37 server kernel: [ 3481.726160] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 Mar 22 12:42:37 server kernel: [ 3481.726175] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726184] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726190] [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726195] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726202] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726210] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726215] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726222] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726227] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726233] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726239] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726247] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726253] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726258] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726260] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 Mar 22 12:42:37 server kernel: [ 3481.726262] [<ffffffff8120c413>] do_mount+0x843/0xcf0 Mar 22 12:42:37 server kernel: [ 3481.726263] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 Mar 22 12:42:37 server kernel: [ 3481.726265] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 Mar 22 12:42:37 server kernel: [ 3481.726266] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 Mar 22 12:42:37 server kernel: [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds. Mar 22 12:42:37 server kernel: [ 4562.226559] Tainted: G U 4.2.0-30-generic #36~14.04.1-Ubuntu Mar 22 12:42:37 server kernel: [ 4562.227415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [...] Mar 22 12:42:37 server kernel: [ 3481.726122] mount D ffff88043e296640 0 587 455 0x00000000 Mar 22 12:42:37 server kernel: [ 3481.726125] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 Mar 22 12:42:37 server kernel: [ 3481.726127] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff Mar 22 12:42:37 server kernel: [ 3481.726129] ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 Mar 22 12:42:37 server kernel: [ 3481.726130] Call Trace: Mar 22 12:42:37 server kernel: [ 3481.726136] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 3481.726138] [<ffffffff817b8ac7>] schedule+0x37/0x80 Mar 22 12:42:37 server kernel: [ 3481.726140] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 Mar 22 12:42:37 server kernel: [ 3481.726143] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 Mar 22 12:42:37 server kernel: [ 3481.726146] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 Mar 22 12:42:37 server kernel: [ 3481.726147] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 3481.726149] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 Mar 22 12:42:37 server kernel: [ 3481.726151] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 Mar 22 12:42:37 server kernel: [ 3481.726152] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 Mar 22 12:42:37 server kernel: [ 3481.726155] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 Mar 22 12:42:37 server kernel: [ 3481.726157] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 Mar 22 12:42:37 server kernel: [ 3481.726159] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 Mar 22 12:42:37 server kernel: [ 3481.726160] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 Mar 22 12:42:37 server kernel: [ 3481.726175] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726184] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726190] [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726195] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726202] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726210] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726215] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726222] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726227] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726233] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726239] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726247] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726253] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726258] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] Mar 22 12:42:37 server kernel: [ 3481.726260] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 Mar 22 12:42:37 server kernel: [ 3481.726262] [<ffffffff8120c413>] do_mount+0x843/0xcf0 Mar 22 12:42:37 server kernel: [ 3481.726263] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 Mar 22 12:42:37 server kernel: [ 3481.726265] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 Mar 22 12:42:37 server kernel: [ 3481.726266] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 Mar 22 12:42:37 server kernel: [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds. [...] Mar 22 12:42:37 server kernel: [ 4562.228280] mount D ffff88043e296640 0 587 455 0x00000000 Mar 22 12:42:37 server kernel: [ 4562.228283] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 Mar 22 12:42:37 server kernel: [ 4562.228285] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff Mar 22 12:42:37 server kernel: [ 4562.228286] ffff88043e5c05e8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 Mar 22 12:42:37 server kernel: [ 4562.228288] Call Trace: Mar 22 12:42:37 server kernel: [ 4562.228294] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 4562.228296] [<ffffffff817b8ac7>] schedule+0x37/0x80 Mar 22 12:42:37 server kernel: [ 4562.228297] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 Mar 22 12:42:37 server kernel: [ 4562.228301] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 Mar 22 12:42:37 server kernel: [ 4562.228304] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 Mar 22 12:42:37 server kernel: [ 4562.228305] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 Mar 22 12:42:37 server kernel: [ 4562.228307] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 Mar 22 12:42:37 server kernel: [ 4562.228309] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 Mar 22 12:42:37 server kernel: [ 4562.228310] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 Mar 22 12:42:37 server kernel: [ 4562.228313] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 Mar 22 12:42:37 server kernel: [ 4562.228315] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 Mar 22 12:42:37 server kernel: [ 4562.228317] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 Mar 22 12:42:37 server kernel: [ 4562.228318] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 Mar 22 12:42:37 server kernel: [ 4562.228332] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228334] [<ffffffff81180df1>] ? set_page_dirty+0x41/0x70 Mar 22 12:42:37 server kernel: [ 4562.228343] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228348] [<ffffffffc0498071>] ? btrfs_release_path+0x21/0xa0 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228354] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228361] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228369] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228374] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228381] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228386] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228392] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228398] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228405] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228412] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228417] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] Mar 22 12:42:37 server kernel: [ 4562.228419] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 Mar 22 12:42:37 server kernel: [ 4562.228421] [<ffffffff8120c413>] do_mount+0x843/0xcf0 Mar 22 12:42:37 server kernel: [ 4562.228422] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 Mar 22 12:42:37 server kernel: [ 4562.228424] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 Mar 22 12:42:37 server kernel: [ 4562.228425] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 Mar 22 12:42:37 server kernel: [ 7975.560908] BTRFS info (device dm-0): continuing balance [...] Mar 22 12:44:03 server kernel: [ 8073.441009] BTRFS info (device dm-0): relocating block group 9553211031552 flags 36 [...] Mar 22 12:51:33 server kernel: [ 8524.324198] btrfs-transacti D ffff88043e3d6640 0 409 2 0x00000000 Mar 22 12:51:33 server kernel: [ 8524.324204] ffff880424983d68 0000000000000046 ffff88042b9b8000 ffff880425be5280 Mar 22 12:51:33 server kernel: [ 8524.324208] 0000000000000246 ffff880424984000 ffff8803ff9884e0 ffff8804265801b0 Mar 22 12:51:33 server kernel: [ 8524.324211] ffff88042bbe5000 0000000000000000 ffff880424983d88 ffffffff817b8ac7 Mar 22 12:51:33 server kernel: [ 8524.324215] Call Trace: Mar 22 12:51:33 server kernel: [ 8524.324225] [<ffffffff817b8ac7>] schedule+0x37/0x80 Mar 22 12:51:33 server kernel: [ 8524.324247] [<ffffffffc04c06d0>] wait_for_commit.isra.17+0x40/0x70 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324252] [<ffffffff810b7670>] ? prepare_to_wait_event+0xf0/0xf0 Mar 22 12:51:33 server kernel: [ 8524.324267] [<ffffffffc04c149c>] btrfs_commit_transaction+0x15c/0xb10 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324281] [<ffffffffc04c1ee3>] ? start_transaction+0x93/0x580 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324295] [<ffffffffc04bc99a>] transaction_kthread+0x1ba/0x240 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324307] [<ffffffffc04bc7e0>] ? btrfs_cleanup_transaction+0x540/0x540 [btrfs] Mar 22 12:51:33 server kernel: [ 8524.324311] [<ffffffff81095499>] kthread+0xc9/0xe0 Mar 22 12:51:33 server kernel: [ 8524.324315] [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0 Mar 22 12:51:33 server kernel: [ 8524.324318] [<ffffffff817bc75f>] ret_from_fork+0x3f/0x70 Mar 22 12:51:33 server kernel: [ 8524.324322] [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0 Mar 22 12:52:23 server kernel: [ 8574.196463] init: ureadahead main process (449) killed by KILL signal Mar 22 12:59:29 server kernel: [ 9000.470433] BTRFS info (device dm-0): found 106150 extents Mar 22 12:59:54 server kernel: [ 9025.568882] BTRFS info (device dm-0): 2 enospc errors during balance dmesg gives me [ 3481.723605] INFO: task mount:587 blocked for more than 120 seconds. [ 3481.724438] Tainted: G U 4.2.0-30-generic #36~14.04.1-Ubuntu [ 3481.725273] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 3481.726122] mount D ffff88043e296640 0 587 455 0x00000000 [ 3481.726125] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 [ 3481.726127] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff [ 3481.726129] ffff88043e5aeee8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 [ 3481.726130] Call Trace: [ 3481.726136] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 [ 3481.726138] [<ffffffff817b8ac7>] schedule+0x37/0x80 [ 3481.726140] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 [ 3481.726143] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 [ 3481.726146] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 [ 3481.726147] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 [ 3481.726149] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 [ 3481.726151] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 [ 3481.726152] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 [ 3481.726155] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 [ 3481.726157] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 [ 3481.726159] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 [ 3481.726160] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 [ 3481.726175] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] [ 3481.726184] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] [ 3481.726190] [<ffffffffc04acdbf>] ? btrfs_run_delayed_refs.part.72+0x1bf/0x280 [btrfs] [ 3481.726195] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] [ 3481.726202] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] [ 3481.726210] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] [ 3481.726215] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] [ 3481.726222] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] [ 3481.726227] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] [ 3481.726233] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] [ 3481.726239] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] [ 3481.726247] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] [ 3481.726253] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] [ 3481.726258] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] [ 3481.726260] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 [ 3481.726262] [<ffffffff8120c413>] do_mount+0x843/0xcf0 [ 3481.726263] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 [ 3481.726265] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 [ 3481.726266] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 [ 4562.225662] INFO: task mount:587 blocked for more than 120 seconds. [ 4562.226559] Tainted: G U 4.2.0-30-generic #36~14.04.1-Ubuntu [ 4562.227415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4562.228280] mount D ffff88043e296640 0 587 455 0x00000000 [ 4562.228283] ffff88042798f6a8 0000000000000086 ffff88042b98b700 ffff880427946e00 [ 4562.228285] ffff88042798f788 ffff880427990000 0000000000000000 7fffffffffffffff [ 4562.228286] ffff88043e5c05e8 ffffffff817b9240 ffff88042798f6c8 ffffffff817b8ac7 [ 4562.228288] Call Trace: [ 4562.228294] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 [ 4562.228296] [<ffffffff817b8ac7>] schedule+0x37/0x80 [ 4562.228297] [<ffffffff817bb4a1>] schedule_timeout+0x201/0x2a0 [ 4562.228301] [<ffffffff8101d749>] ? read_tsc+0x9/0x10 [ 4562.228304] [<ffffffff810e6e5e>] ? ktime_get+0x3e/0xa0 [ 4562.228305] [<ffffffff817b9240>] ? bit_wait+0x60/0x60 [ 4562.228307] [<ffffffff817b8136>] io_schedule_timeout+0xa6/0x110 [ 4562.228309] [<ffffffff817b925f>] bit_wait_io+0x1f/0x70 [ 4562.228310] [<ffffffff817b8e90>] __wait_on_bit+0x60/0x90 [ 4562.228313] [<ffffffff81176708>] ? find_get_pages_tag+0xc8/0x170 [ 4562.228315] [<ffffffff81175980>] wait_on_page_bit+0xc0/0xd0 [ 4562.228317] [<ffffffff810b76b0>] ? autoremove_wake_function+0x40/0x40 [ 4562.228318] [<ffffffff81175a80>] filemap_fdatawait_range+0xf0/0x180 [ 4562.228332] [<ffffffffc04dc283>] btrfs_wait_ordered_range+0x73/0x110 [btrfs] [ 4562.228334] [<ffffffff81180df1>] ? set_page_dirty+0x41/0x70 [ 4562.228343] [<ffffffffc05075a3>] btrfs_wait_cache_io+0x63/0x1d0 [btrfs] [ 4562.228348] [<ffffffffc0498071>] ? btrfs_release_path+0x21/0xa0 [btrfs] [ 4562.228354] [<ffffffffc04b069b>] btrfs_start_dirty_block_groups+0x18b/0x400 [btrfs] [ 4562.228361] [<ffffffffc04c1501>] btrfs_commit_transaction+0x1c1/0xb10 [btrfs] [ 4562.228369] [<ffffffffc04e35af>] ? free_extent_buffer+0x4f/0x90 [btrfs] [ 4562.228374] [<ffffffffc04a7f9b>] ? btrfs_block_rsv_check+0x2b/0x70 [btrfs] [ 4562.228381] [<ffffffffc04c287b>] __btrfs_end_transaction+0x39b/0x3d0 [btrfs] [ 4562.228386] [<ffffffffc04b2c48>] ? btrfs_update_root+0x198/0x320 [btrfs] [ 4562.228392] [<ffffffffc04c2c03>] btrfs_end_transaction_throttle+0x13/0x20 [btrfs] [ 4562.228398] [<ffffffffc04ab4c2>] btrfs_drop_snapshot+0x492/0x830 [btrfs] [ 4562.228405] [<ffffffffc0514c72>] merge_reloc_roots+0xd2/0x230 [btrfs] [ 4562.228412] [<ffffffffc05159f3>] btrfs_recover_relocation+0x353/0x3b0 [btrfs] [ 4562.228417] [<ffffffffc04947c7>] btrfs_remount+0x407/0x560 [btrfs] [ 4562.228419] [<ffffffff811ee66a>] do_remount_sb+0x6a/0x1c0 [ 4562.228421] [<ffffffff8120c413>] do_mount+0x843/0xcf0 [ 4562.228422] [<ffffffff8120ba9a>] ? copy_mount_options+0x3a/0x150 [ 4562.228424] [<ffffffff8120cbab>] SyS_mount+0x8b/0xe0 [ 4562.228425] [<ffffffff817bc332>] entry_SYSCALL_64_fastpath+0x16/0x75 [ 7975.560908] BTRFS info (device dm-0): continuing balance [ 7977.590090] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) [ 7978.881574] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null) [ 7982.053560] init: failsafe main process (1047) killed by TERM signal [...] [ 8073.441009] BTRFS info (device dm-0): relocating block group 9553211031552 flags 36 [ 8081.239604] audit: type=1400 audit(1458647050.858:22): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=1299 comm="apparmor_parser" [ 8171.758863] init: mysql post-start process (1322) terminated with status 1 [ 8190.153034] audit: type=1400 audit(1458647159.668:23): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/var/tmp/mysql/" pid=1321 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=106 ouid=0 [ 8216.355398] init: plymouth-upstart-bridge main process ended, respawning [ 8524.324045] INFO: task btrfs-transacti:409 blocked for more than 120 seconds. [ 8524.324102] Tainted: G U 4.2.0-30-generic #36~14.04.1-Ubuntu [ 8524.324149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8524.324198] btrfs-transacti D ffff88043e3d6640 0 409 2 0x00000000 [ 8524.324204] ffff880424983d68 0000000000000046 ffff88042b9b8000 ffff880425be5280 [ 8524.324208] 0000000000000246 ffff880424984000 ffff8803ff9884e0 ffff8804265801b0 [ 8524.324211] ffff88042bbe5000 0000000000000000 ffff880424983d88 ffffffff817b8ac7 [ 8524.324215] Call Trace: [ 8524.324225] [<ffffffff817b8ac7>] schedule+0x37/0x80 [ 8524.324247] [<ffffffffc04c06d0>] wait_for_commit.isra.17+0x40/0x70 [btrfs] [ 8524.324252] [<ffffffff810b7670>] ? prepare_to_wait_event+0xf0/0xf0 [ 8524.324267] [<ffffffffc04c149c>] btrfs_commit_transaction+0x15c/0xb10 [btrfs] [ 8524.324281] [<ffffffffc04c1ee3>] ? start_transaction+0x93/0x580 [btrfs] [ 8524.324295] [<ffffffffc04bc99a>] transaction_kthread+0x1ba/0x240 [btrfs] [ 8524.324307] [<ffffffffc04bc7e0>] ? btrfs_cleanup_transaction+0x540/0x540 [btrfs] [ 8524.324311] [<ffffffff81095499>] kthread+0xc9/0xe0 [ 8524.324315] [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0 [ 8524.324318] [<ffffffff817bc75f>] ret_from_fork+0x3f/0x70 [ 8524.324322] [<ffffffff810953d0>] ? kthread_create_on_node+0x1c0/0x1c0 [ 8574.196463] init: ureadahead main process (449) killed by KILL signal [ 9000.470433] BTRFS info (device dm-0): found 106150 extents [ 9025.568882] BTRFS info (device dm-0): 2 enospc errors during balance -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1559723 Title: soft lockup during btrfs balance Status in linux package in Ubuntu: Incomplete Bug description: I performed a btrfs balance for a disk which ran out of space. When I did this some months ago, I got an error message, but this time, I got a soft lockup and the file system got mounted read-only: # cat /proc/version_signature Ubuntu 4.2.0-27.32~14.04.1-generic 4.2.8-ckt1 I see multiple errors in dmesg output, like [3099814.088753] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [btrfs:27516] [3099814.089403] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c intel_rapl w83627ehf x86_pkg_temp_thermal intel_powerclamp hwmon_vid coretemp kvm_intel kvm snd_hda_codec_realtek snd_hda _codec_generic lp joydev input_leds ppdev gpio_ich 8250_fintek parport_pc parport nuvoton_cir rc_core mac_hid snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer snd lpc_ich mei_me soundcore mei s hpchp serio_raw btrfs drbg ansi_cprng dm_crypt raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid0 multipath linear raid1 crct10dif_pclmul crc32_pclmul hid_generic i915 aes ni_intel ahci aes_x86_64 lrw usbhid video gf128mul i2c_algo_bit glue_helper e1000e ablk_helper drm_kms_helper libahci ptp cryptd hid psmouse pps_core drm pata_acpi [3099814.089436] CPU: 3 PID: 27516 Comm: btrfs Tainted: G U W L 4.2.0-27-generic #32~14.04.1-Ubuntu [3099814.089437] Hardware name: /DQ67OW, BIOS SWQ6710H.86A.0067.2014.0313.1347 03/13/2014 [3099814.089438] task: ffff8800aca0a940 ti: ffff880110c5c000 task.ti: ffff880110c5c000 [3099814.089438] RIP: 0010:[<ffffffffc031f9d1>] [<ffffffffc031f9d1>] __del_reloc_root+0xd1/0x100 [btrfs] [3099814.089451] RSP: 0018:ffff880110c5fa60 EFLAGS: 00000246 [3099814.089452] RAX: ffff880142cd36f8 RBX: ffffffffc02a912a RCX: 00000000ffffffe2 [3099814.089453] RDX: 000007ba9c902000 RSI: ffffffffc034e280 RDI: ffff88002a9e8d70 [3099814.089454] RBP: ffff880110c5fa78 R08: 0000000000000000 R09: 00000000ffffffe2 [3099814.089454] R10: ffffffffc02a912a R11: ffffea00029bc9c0 R12: 00000000ffffffe2 [3099814.089455] R13: ffff880110c5fa78 R14: 000000002b4bf801 R15: ffff88042b4bf800 [3099814.089456] FS: 00007f93651f0880(0000) GS:ffff88043e2c0000(0000) knlGS:0000000000000000 [3099814.089457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [3099814.089458] CR2: 0000559c1ffff7d0 CR3: 00000002fca04000 CR4: 00000000000406e0 [3099814.089459] Stack: [3099814.089460] ffff88002a9e8800 ffff88042b52b000 ffff880110c5fab8 ffff880110c5fa98 [3099814.089461] ffffffffc031fa25 ffff88042b4bfbc0 ffff88042b4bfbc0 ffff880110c5faf8 [3099814.089462] ffffffffc0325f2a ffff88002a9e8d78 000000002a9e8d78 ffff88042b4bdbc0 [3099814.089464] Call Trace: [3099814.089471] [<ffffffffc031fa25>] free_reloc_roots+0x25/0x40 [btrfs] [3099814.089477] [<ffffffffc0325f2a>] merge_reloc_roots+0x16a/0x230 [btrfs] [3099814.089483] [<ffffffffc0326246>] relocate_block_group+0x256/0x600 [btrfs] [3099814.089489] [<ffffffffc03267b3>] btrfs_relocate_block_group+0x1c3/0x2d0 [btrfs] [3099814.089496] [<ffffffffc02f9dee>] btrfs_relocate_chunk.isra.39+0x3e/0xc0 [btrfs] [3099814.089502] [<ffffffffc02fb26f>] __btrfs_balance+0x48f/0x8c0 [btrfs] [3099814.089509] [<ffffffffc02fba1d>] btrfs_balance+0x37d/0x650 [btrfs] [3099814.089516] [<ffffffffc03085e2>] ? btrfs_ioctl_balance+0x282/0x530 [btrfs] [3099814.089522] [<ffffffffc0308756>] btrfs_ioctl_balance+0x3f6/0x530 [btrfs] [3099814.089528] [<ffffffffc030a17f>] btrfs_ioctl+0x56f/0x2470 [btrfs] [3099814.089532] [<ffffffff8118559b>] ? lru_cache_add_active_or_unevictable+0x2b/0xa0 [3099814.089534] [<ffffffff811a5e55>] ? handle_mm_fault+0xba5/0x1840 [3099814.089536] [<ffffffff811fdfbd>] do_vfs_ioctl+0x2cd/0x4b0 [3099814.089539] [<ffffffff810642a6>] ? __do_page_fault+0x1b6/0x430 [3099814.089541] [<ffffffff811fe219>] SyS_ioctl+0x79/0x90 [3099814.089543] [<ffffffff817bc3b2>] entry_SYSCALL_64_fastpath+0x16/0x75 [3099814.089543] Code: f0 01 00 00 49 89 84 24 c0 03 00 00 49 89 84 24 c8 03 00 00 48 81 c7 38 09 00 00 c6 07 00 66 66 66 90 48 89 df e8 b0 d3 ea c0 5b <41> 5c 41 5d 41 5e 5d c3 76 09 48 8b 5b 08 e9 5a ff ff ff 49 8d I also can't access some of the directories now (even in read-only mode): cd somedir -su: cd: somedir: Eingabe-/Ausgabefehler # lsb_release -rd Description: Ubuntu 14.04.4 LTS Release: 14.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1559723/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
