Verification with bionic-proposed of the I/O Error path.
All good, working as expected (see comments #11 to #16).
# uname -rv
4.15.0-56-generic #62-Ubuntu SMP Wed Jul 24 20:18:55 UTC 2019
test 1
------
# ./setup.sh >/dev/null 2>&1
[ 369.375820] bcache: register_bdev() registered backing device dm-0
[ 369.395195] bcache: run_cache_set() invalidating existing data
[ 369.410278] bcache: register_cache() registered cache device dm-1
[ 371.393391] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set
c1126837-e029-4d08-bad3-38ff8bc08054
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
loop1 7:1 0 1G 0 loop
└─fake-loop1 253:1 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
On another shell:
# fio --name=write --rw=randwrite --filename=/dev/bcache0 --bs=4k
--iodepth=8 --ioengine=libaio --runtime=300s --continue_on_error=all
# [ 425.656209] bcache: bch_count_io_errors() dm-1: IO error on writing btree,
recovering
[ 425.684837] bcache: error on c1126837-e029-4d08-bad3-38ff8bc08054:
[ 425.684840] journal io error
[ 425.686537] , disabling caching
[ 425.688849] Buffer I/O error on dev bcache0, logical block 2807, lost async
page write
[ 425.691541] Buffer I/O error on dev bcache0, logical block 2808, lost async
page write
[ 425.694131] bcache: conditional_stop_bcache_device()
stop_when_cache_set_failed of bcache0 is "auto" and cache is clean, keep it
alive.
[ 425.698343] Buffer I/O error on dev bcache0, logical block 2810, lost async
page write
[ 425.702522] Buffer I/O error on dev bcache0, logical block 2812, lost async
page write
[ 425.705326] Buffer I/O error on dev bcache0, logical block 2813, lost async
page write
[ 425.707896] Buffer I/O error on dev bcache0, logical block 2814, lost async
page write
[ 425.710692] Buffer I/O error on dev bcache0, logical block 2816, lost async
page write
[ 425.713524] Buffer I/O error on dev bcache0, logical block 2817, lost async
page write
[ 425.716512] Buffer I/O error on dev bcache0, logical block 2818, lost async
page write
[ 425.719156] Buffer I/O error on dev bcache0, logical block 2819, lost async
page write
[ 425.742817] bcache: cached_dev_detach_finish() Caching disabled for dm-0
[ 425.746933] bcache: bch_count_io_errors() dm-1: IO error on writing btree,
recovering
[ 425.750502] bcache: cache_set_free() Cache set
c1126837-e029-4d08-bad3-38ff8bc08054 unregistered
fio finished:
Run status group 0 (all jobs):
WRITE: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=1024MiB
(1074MB), run=4830-4830msec
bcache not on top of caching device:
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
loop1 7:1 0 1G 0 loop
fake-loop1 253:1 0 1G 0 dm
test 2
------
# ./setup.sh >/dev/null 2>&1
[ 23.946411] bcache: register_bdev() registered backing device dm-0
[ 23.952262] bcache: run_cache_set() invalidating existing data
[ 23.966564] bcache: register_cache() registered cache device dm-1
[ 25.949934] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set
d7a3c644-e21e-49bb-bcee-e14709a65745
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
loop1 7:1 0 1G 0 loop
└─fake-loop1 253:1 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
# echo writeback > /sys/block/bcache0/bcache/cache_mode
# dd if=/dev/zero of=/dev/bcache0 bs=4k
dd: error writing '/dev/bcache0': No space left on device
262142+0 records in
262141+0 records out
1073729536 bytes (1.1 GB, 1.0 GiB) copied, 4.2152 s, 255 MB/s
# ./dm_fake_dev.sh /dev/loop1 bad
[ 55.515809] Buffer I/O error on dev dm-1, logical block 262128, async page
read
[ 55.522181] Buffer I/O error on dev dm-1, logical block 262128, async page
read
[ 55.528425] bcache: register_bcache() error /dev/dm-1: device already
registered
# [ 74.716322] bcache: bch_count_io_errors() dm-1: IO error on reading dirty
data from cache, recovering
[ 74.719766] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 74.808148] bcache: bch_count_io_errors() dm-1: IO error on reading dirty
data from cache, recovering
[ 74.813328] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 74.964134] bcache: bch_count_io_errors() dm-1: IO error on reading dirty
data from cache, recovering
...
[ 85.184111] bcache: bch_count_io_errors() dm-1: IO error on reading dirty
data from cache, recovering
[ 85.189769] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 85.364144] bcache: bch_count_io_errors() dm-1: IO error on reading dirty
data from cache, recovering
[ 85.369008] bcache: bch_cached_dev_error() stop bcache0: too many IO errors
on backing device dm-0
[ 85.369008]
[ 85.374890] bcache: bch_count_io_errors() dm-1: IO error on reading dirty
data from cache, recovering
[ 85.379004] bcache: bcache_device_free() bcache0 stopped
bcache device is removed
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
loop1 7:1 0 1G 0 loop
fake-loop1 253:1 0 1G 0 dm
test 3
------
# ./setup.sh >/dev/null 2>&1
[ 31.671004] bcache: register_bdev() registered backing device dm-0
[ 31.676981] bcache: run_cache_set() invalidating existing data
[ 31.686989] bcache: register_cache() registered cache device dm-1
[ 33.678658] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set
12ed8698-10f0-4a6d-bee1-4f11d5d6bf20
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
loop1 7:1 0 1G 0 loop
└─fake-loop1 253:1 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
# echo 1 > /sys/block/bcache0/bcache/detach
# [ 43.394627] bcache: cached_dev_detach_finish() Caching disabled for dm-0
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
loop1 7:1 0 1G 0 loop
└─fake-loop1 253:1 0 1024M 0 dm
# dd if=/dev/zero of=/dev/bcache0 bs=4k
dd: error writing '/dev/bcache0': No space left on device
262142+0 records in
262141+0 records out
1073729536 bytes (1.1 GB, 1.0 GiB) copied, 3.69032 s, 291 MB/s
# sync
# iostat -xy 1 10 /dev/bcache0 | grep bcache0
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
bcache0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
#
test 4
------
# ./setup.sh >/dev/null 2>&1
[ 19.867266] bcache: register_bdev() registered backing device dm-0
[ 20.032234] bcache: run_cache_set() invalidating existing data
[ 20.310143] bcache: register_cache() registered cache device dm-1
[ 22.034294] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set
5304f479-8f93-487e-a5cb-eb940e21751a
# echo writeback > /sys/block/bcache0/bcache/cache_mode
# cat /sys/block/bcache0/bcache/cache_mode
writethrough [writeback] writearound none
# ./dm_fake_dev.sh /dev/loop0 bad
[ 39.618330] Buffer I/O error on dev dm-0, logical block 262128, async page
read
[ 39.623830] Buffer I/O error on dev dm-0, logical block 262128, async page
read
[ 39.631524] bcache: register_bcache() error /dev/dm-0: device already
registered (emitting change event)
[ 39.633892] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 39.639076] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 39.642480] Buffer I/O error on dev bcache0, logical block 262112, async
page read
# [ 39.648190] bcache: bch_count_backing_io_errors() dm-0: IO error on
backing device, unrecoverable
[ 39.651837] Buffer I/O error on dev bcache0, logical block 262112, async
page read
[ 39.654309] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 39.657289] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 39.660116] Buffer I/O error on dev bcache0, logical block 1, async page read
# dd if=/dev/zero of=/dev/bcache0 bs=4k
[ 60.268137] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 60.271183] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 60.273467] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 60.275693] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 60.277945] Buffer I/O error on dev bcache0, logical block 0, lost async
page write
[ 60.279908] Buffer I/O error on dev bcache0, logical block 1, lost async
page write
[ 60.280160] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 60.281866] Buffer I/O error on dev bcache0, logical block 2, lost async
page write
[ 60.286340] Buffer I/O error on dev bcache0, logical block 4, lost async
page write
...
[ 60.456411] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 60.459153] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 60.461861] bcache: bch_cached_dev_error() stop bcache0: too many IO errors
on backing device dm-0
[ 60.461861]
dd: error writing '/dev/bcache0': No space left on device
262142+0 records in
262141+0 records out
1073729536 bytes (1.1 GB, 1.0 GiB) copied, 2.12236 s, 506 MB/s
[ 60.931271] bcache: bcache_device_free() bcache0 stopped
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
loop1 7:1 0 1G 0 loop
└─fake-loop1 253:1 0 1024M 0 dm
fake-loop0 253:0 0 1G 0 dm
# ls /dev/bcache0
ls: cannot access '/dev/bcache0': No such file or directory
test 5
------
# modprobe scsi_debug dev_size_mb=1024
# dmesg | tail
[ 72.008739] scsi host2: scsi_debug: version 1.86 [20160430]
dev_size_mb=1024, opts=0x0, submit_queues=1, statistics=0
[ 72.009273] scsi 2:0:0:0: Direct-Access Linux scsi_debug 0186
PQ: 0 ANSI: 7
[ 72.010159] sd 2:0:0:0: Power-on or device reset occurred
[ 72.012001] sd 2:0:0:0: Attached scsi generic sg0 type 0
[ 72.018462] sd 2:0:0:0: [sda] 2097152 512-byte logical blocks: (1.07 GB/1.00
GiB)
[ 72.022566] sd 2:0:0:0: [sda] Write Protect is off
[ 72.022571] sd 2:0:0:0: [sda] Mode Sense: 73 00 10 08
[ 72.030790] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled,
supports DPO and FUA
[ 72.121972] sd 2:0:0:0: [sda] Attached SCSI disk
# ./setup-sda.sh >/dev/null 2>&1
[ 83.685946] bcache: run_cache_set() invalidating existing data
[ 83.697155] bcache: register_cache() registered cache device dm-0
[ 83.795953] bcache: register_bdev() registered backing device sda
[ 84.638672] bcache: bch_cached_dev_attach() Caching sda as bcache0 on set
de4439a0-7a3e-42dc-9415-c10525eb1e6d
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
sda 8:0 0 1G 0 disk
└─bcache0 251:0 0 1024M 0 disk
# echo writeback > /sys/block/sda/bcache/cache_mode
# echo 1 > /sys/block/sda/device/delete
[ 99.172506] sd 2:0:0:0: [sda] Synchronizing SCSI cache
# sleep 10
[ 104.284209] bcache: cached_dev_status_update() sda: device offline for 5
seconds
[ 104.287916] bcache: cached_dev_status_update() bdev0: disable I/O request
due to backing device offline
[ 104.292588] bcache: bcache_device_free() bcache0 stopped
#
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
# ls /dev/bcache0
ls: cannot access '/dev/bcache0': No such file or directory
test 6
------
# ./setup-two-bcache-one-cache.sh >/dev/null 2>&1
[ 23.100810] bcache: register_bdev() registered backing device dm-0
[ 23.114149] bcache: run_cache_set() invalidating existing data
[ 23.123286] bcache: register_cache() registered cache device dm-2
[ 23.123552] bcache: register_bdev() registered backing device dm-1
[ 24.130116] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set
866e8bce-38c8-491c-a845-097cc5373ff6
[ 24.136297] bcache: bch_cached_dev_attach() Caching dm-1 as bcache1 on set
866e8bce-38c8-491c-a845-097cc5373ff6
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
loop1 7:1 0 1G 0 loop
└─fake-loop1 253:1 0 1024M 0 dm
└─bcache1 251:128 0 1024M 0 disk
loop2 7:2 0 1G 0 loop
└─fake-loop2 253:2 0 1024M 0 dm
├─bcache0 251:0 0 1024M 0 disk
└─bcache1 251:128 0 1024M 0 disk e
writebackic-bcache:~# echo writeback | tee
/sys/block/bcache*/bcache/cache_mode
e_set_failedbcache:~# echo always | tee
/sys/block/bcache*/bcache/stop_when_cache
always
# ./dm_fake_dev.sh /dev/loop0 bad
[ 54.190905] Buffer I/O error on dev dm-0, logical block 262128, async page
read
[ 54.196046] Buffer I/O error on dev dm-0, logical block 262128, async page
read
[ 54.202079] bcache: register_bcache() error /dev/dm-0: device already
registered (emitting change event)
[ 54.204484] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 54.209396] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 54.212415] Buffer I/O error on dev bcache0, logical block 262112, async
page read
# [ 54.217664] bcache: bch_count_backing_io_errors() dm-0: IO error on
backing device, unrecoverable
[ 54.221146] Buffer I/O error on dev bcache0, logical block 262112, async
page read
[ 54.223735] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 54.226221] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 54.228466] Buffer I/O error on dev bcache0, logical block 1, async page read
root@guest-bcache:~# dd if=/dev/zero of=/dev/bcache1 bs=4k & dd
if=/dev/zero of=/dev/bcache0 bs=4k &
[1] 1385
[2] 1386
# [ 66.991499] bcache: bch_count_backing_io_errors() dm-0: IO error on
backing device, unrecoverable
[ 67.003045] Buffer I/O error on dev bcache0, logical block 0, lost async
page write
[ 67.005222] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.008246] Buffer I/O error on dev bcache0, logical block 1, lost async
page write
[ 67.010836] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.013786] Buffer I/O error on dev bcache0, logical block 2, lost async
page write
[ 67.016412] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.019387] Buffer I/O error on dev bcache0, logical block 3, lost async
page write
[ 67.021986] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.025013] Buffer I/O error on dev bcache0, logical block 4, lost async
page write
[ 67.027636] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.030614] Buffer I/O error on dev bcache0, logical block 5, lost async
page write
[ 67.033269] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.036262] Buffer I/O error on dev bcache0, logical block 6, lost async
page write
[ 67.038893] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.041868] Buffer I/O error on dev bcache0, logical block 7, lost async
page write
[ 67.044524] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.047529] Buffer I/O error on dev bcache0, logical block 8, lost async
page write
[ 67.050115] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.053136] Buffer I/O error on dev bcache0, logical block 9, lost async
page write
[ 67.056327] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.059360] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
...
[ 67.206725] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.210109] bcache: bch_count_backing_io_errors() dm-0: IO error on backing
device, unrecoverable
[ 67.212578] bcache: bch_cached_dev_error() stop bcache0: too many IO errors
on backing device dm-0
[ 67.212578]
dd: error writing '/dev/bcache0': No space left on device
262142+0 records[ 68.393613] bcache: bcache_device_free() bcache0 stopped
in
262141+0 records out
1073729536 bytes (1.1 GB, 1.0 GiB) copied, 2.32385 s, 462 MB/s
dd: error writing '/dev/bcache1': No space left on device
262142+0 records in
262141+0 records out
1073729536 bytes (1.1 GB, 1.0 GiB) copied, 4.5021 s, 238 MB/s
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
loop1 7:1 0 1G 0 loop
└─fake-loop1 253:1 0 1024M 0 dm
└─bcache1 251:128 0 1024M 0 disk
loop2 7:2 0 1G 0 loop
└─fake-loop2 253:2 0 1024M 0 dm
└─bcache1 251:128 0 1024M 0 disk
fake-loop0 253:0 0 1G 0 dm
only bcache0 was stopped. bcache1 remains working.
# reboot
# ./setup-two-bcache-one-cache.reboot.sh >/dev/null 2>&1
[ 17.697259] bcache: register_bdev() registered backing device dm-0
[ 17.770242] bcache: register_bdev() registered backing device dm-1
[ 17.902012] bcache: bch_journal_replay() journal replay done, 57093 keys in
34 entries, seq 161
[ 17.908344] bcache: bch_cached_dev_attach() Caching dm-1 as bcache1 on set
866e8bce-38c8-491c-a845-097cc5373ff6
[ 17.914819] bcache: bch_cached_dev_attach() Caching dm-0 as bcache0 on set
866e8bce-38c8-491c-a845-097cc5373ff6
[ 17.918484] bcache: register_cache() registered cache device dm-2
# lsblk -e 252
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 1G 0 loop
└─fake-loop0 253:0 0 1024M 0 dm
└─bcache0 251:0 0 1024M 0 disk
loop1 7:1 0 1G 0 loop
└─fake-loop1 253:1 0 1024M 0 dm
└─bcache1 251:128 0 1024M 0 disk
loop2 7:2 0 1G 0 loop
└─fake-loop2 253:2 0 1024M 0 dm
├─bcache0 251:0 0 1024M 0 disk
└─bcache1 251:128 0 1024M 0 disk
both bcache devices reattached after reboot.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1829563
Title:
bcache: risk of data loss on I/O errors in backing or caching devices
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1829563/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
