* Changlong Xie (xiecl.f...@cn.fujitsu.com) wrote:
> On 02/01/2016 09:18 AM, Wen Congyang wrote:
> >On 01/29/2016 06:47 PM, Dr. David Alan Gilbert wrote:
> >>* Wen Congyang (we...@cn.fujitsu.com) wrote:
> >>>On 01/29/2016 06:07 PM, Dr. David Alan Gilbert wrote:
> >>>>* Wen Congyang (we...@cn.fujitsu.com) wrote:
> >>>>>On 01/27/2016 07:03 PM, Dr. David Alan Gilbert wrote:
> >>>>>>Hi,
> >>>>>> I've got a block error if I kill the secondary.
> >>>>>>
> >>>>>>Start both primary & secondary
> >>>>>>kill -9 secondary qemu
> >>>>>>x_colo_lost_heartbeat on primary
> >>>>>>
> >>>>>>The guest sees a block error and the ext4 root switches to read-only.
> >>>>>>
> >>>>>>I gdb'd the primary with a breakpoint on quorum_report_bad; see
> >>>>>>backtrace below.
> >>>>>>(This is based on colo-v2.4-periodic-mode of the framework
> >>>>>>code with the block and network proxy merged in; so it could be my
> >>>>>>merging but I don't think so ?)
> >>>>>>
> >>>>>>
> >>>>>>(gdb) where
> >>>>>>#0 quorum_report_bad (node_name=0x7f2946a0892c "node0", ret=-5,
> >>>>>>acb=0x7f2946cb3910, acb=0x7f2946cb3910)
> >>>>>> at /root/colo/jan-2016/qemu/block/quorum.c:222
> >>>>>>#1 0x00007f2943b23058 in quorum_aio_cb (opaque=<optimized out>,
> >>>>>>ret=<optimized out>)
> >>>>>> at /root/colo/jan-2016/qemu/block/quorum.c:315
> >>>>>>#2 0x00007f2943b311be in bdrv_co_complete (acb=0x7f2946cb3f60) at
> >>>>>>/root/colo/jan-2016/qemu/block/io.c:2122
> >>>>>>#3 0x00007f2943ae777d in aio_bh_call (bh=<optimized out>) at
> >>>>>>/root/colo/jan-2016/qemu/async.c:64
> >>>>>>#4 aio_bh_poll (ctx=ctx@entry=0x7f2945b771d0) at
> >>>>>>/root/colo/jan-2016/qemu/async.c:92
> >>>>>>#5 0x00007f2943af5090 in aio_dispatch (ctx=0x7f2945b771d0) at
> >>>>>>/root/colo/jan-2016/qemu/aio-posix.c:305
> >>>>>>#6 0x00007f2943ae756e in aio_ctx_dispatch (source=<optimized out>,
> >>>>>>callback=<optimized out>,
> >>>>>> user_data=<optimized out>) at /root/colo/jan-2016/qemu/async.c:231
> >>>>>>#7 0x00007f293b84a79a in g_main_context_dispatch () from
> >>>>>>/lib64/libglib-2.0.so.0
> >>>>>>#8 0x00007f2943af3a00 in glib_pollfds_poll () at
> >>>>>>/root/colo/jan-2016/qemu/main-loop.c:211
> >>>>>>#9 os_host_main_loop_wait (timeout=<optimized out>) at
> >>>>>>/root/colo/jan-2016/qemu/main-loop.c:256
> >>>>>>#10 main_loop_wait (nonblocking=<optimized out>) at
> >>>>>>/root/colo/jan-2016/qemu/main-loop.c:504
> >>>>>>#11 0x00007f29438529ee in main_loop () at
> >>>>>>/root/colo/jan-2016/qemu/vl.c:1945
> >>>>>>#12 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> >>>>>>out>) at /root/colo/jan-2016/qemu/vl.c:4707
> >>>>>>
> >>>>>>(gdb) p s->num_children
> >>>>>>$1 = 2
> >>>>>>(gdb) p acb->success_count
> >>>>>>$2 = 0
> >>>>>>(gdb) p acb->is_read
> >>>>>>$5 = false
> >>>>>
> >>>>>Sorry for the late reply.
> >>>>
> >>>>No problem.
> >>>>
> >>>>>What it the value of acb->count?
> >>>>
> >>>>(gdb) p acb->count
> >>>>$1 = 1
> >>>
> >>>Note, the count is 1, not 2. Writing to children.0 is in flight. If
> >>>writing to children.0 successes,
> >>>the guest doesn't know this error.
> >>>>>If secondary host is down, you should remove quorum's children.1.
> >>>>>Otherwise, you will get
> >>>>>I/O error event.
> >>>>
> >>>>Is that safe? If the secondary fails, do you always have time to issue
> >>>>the command to
> >>>>remove the children.1 before the guest sees the error?
> >>>
> >>>We will write to two children, and expect that writing to children.0 will
> >>>success. If so,
> >>>the guest doesn't know this error. You just get the I/O error event.
> >>
> >>I think children.0 is the disk, and that should be OK - so only the
> >>children.1/replication should
> >>be failing - so in that case why do I see the error?
> >
> >I don't know, and I will check the codes.
> >
> >>The 'node0' in the backtrace above is the name of the replication, so it
> >>does look like the error
> >>is coming from the replication.
> >
> >No, the backtrace is just report an I/O error events to the management
> >application.
> >
> >>
> >>>>Anyway, I tried removing children.1 but it segfaults now, I guess the
> >>>>replication is unhappy:
> >>>>
> >>>>(qemu) x_block_change colo-disk0 -d children.1
> >>>>(qemu) x_colo_lost_heartbeat
> >>>
> >>>Hmm, you should not remove the child before failover. I will check it how
> >>>to avoid it in the codes.
> >>
> >> But you said 'If secondary host is down, you should remove quorum's
> >> children.1' - is that not
> >>what you meant?
> >
> >Yes, you should excute 'x_colo_lost_heartbeat' fist, and then excute
> >'x_block_change ... -d ...'.
> >
> Hi david
Hi Xie,
Thanks for the response.
> It seems we missed 'drive_del' command, and will document it in next
> version. Here is the right commands order:
>
> { "execute": "x-colo-lost-heartbeat" }
> { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk',
> 'child': 'children.1'}}
> { 'execute': 'human-monitor-command', 'arguments': {'command-line':
> 'drive_del xxxxx'}}
OK, however, you should fix the seg fault if you don't issue the drive_del;
qemu should never crash.
(Also I still get the IO error in the guest if I do the x-colo-lost-heartbeat).
Dave
> Thanks
> -Xie
> >>
> >>>>12973 Segmentation fault (core dumped)
> >>>>./try/x86_64-softmmu/qemu-system-x86_64 -enable-kvm $console_param -S
> >>>>-boot c -m 4080 -smp 4 -machine pc-i440fx-2.5,accel=kvm -name
> >>>>debug-threads=on -trace events=trace-file -device virtio-rng-pci
> >>>>$block_param $net_param
> >>>>
> >>>>#0 0x00007f0a398a864c in bdrv_stop_replication (bs=0x7f0a3b0a8430,
> >>>>failover=true, errp=0x7fff6a5c3420)
> >>>> at /root/colo/jan-2016/qemu/block.c:4426
> >>>>
> >>>>(gdb) p drv
> >>>>$1 = (BlockDriver *) 0x5d2a
> >>>>
> >>>> it looks like the whole of bs is bogus.
> >>>>
> >>>>#1 0x00007f0a398d87f6 in quorum_stop_replication (bs=<optimized out>,
> >>>>failover=<optimized out>,
> >>>> errp=<optimized out>) at /root/colo/jan-2016/qemu/block/quorum.c:1213
> >>>>
> >>>>(gdb) p s->replication_index
> >>>>$3 = 1
> >>>>
> >>>>I guess quorum_del_child needs to stop replication before it removes the
> >>>>child?
> >>>
> >>>Yes, but in the newest version, quorum doesn't know the block replication,
> >>>and I think
> >>>we shoud add an reference to the bs when starting block replication.
> >>
> >>Do you have a new version ready to test? I'm interested to try it (and
> >>also interested
> >>to try the latest version of the colo-proxy)
> >
> >I think we can post the newest version this week.
> >
> >Thanks
> >Wen Congyang
> >
> >>
> >>Dave
> >>
> >>>Thanks
> >>>Wen Congyang
> >>>
> >>>>(although it would have to be careful not to block on the dead nbd).
> >>>>
> >>>>#2 0x00007f0a398a8901 in bdrv_stop_replication_all
> >>>>(failover=failover@entry=true, errp=errp@entry=0x7fff6a5c3478)
> >>>> at /root/colo/jan-2016/qemu/block.c:4504
> >>>>#3 0x00007f0a3984b0af in primary_vm_do_failover () at
> >>>>/root/colo/jan-2016/qemu/migration/colo.c:144
> >>>>#4 colo_do_failover (s=<optimized out>) at
> >>>>/root/colo/jan-2016/qemu/migration/colo.c:162
> >>>>#5 0x00007f0a3989d7fd in aio_bh_call (bh=<optimized out>) at
> >>>>/root/colo/jan-2016/qemu/async.c:64
> >>>>#6 aio_bh_poll (ctx=ctx@entry=0x7f0a3a6c21d0) at
> >>>>/root/colo/jan-2016/qemu/async.c:92
> >>>>#7 0x00007f0a398ab110 in aio_dispatch (ctx=0x7f0a3a6c21d0) at
> >>>>/root/colo/jan-2016/qemu/aio-posix.c:305
> >>>>#8 0x00007f0a3989d5ee in aio_ctx_dispatch (source=<optimized out>,
> >>>>callback=<optimized out>,
> >>>> user_data=<optimized out>) at /root/colo/jan-2016/qemu/async.c:231
> >>>>#9 0x00007f0a3160079a in g_main_context_dispatch () from
> >>>>/lib64/libglib-2.0.so.0
> >>>>#10 0x00007f0a398a9a80 in glib_pollfds_poll () at
> >>>>/root/colo/jan-2016/qemu/main-loop.c:211
> >>>>#11 os_host_main_loop_wait (timeout=<optimized out>) at
> >>>>/root/colo/jan-2016/qemu/main-loop.c:256
> >>>>#12 main_loop_wait (nonblocking=<optimized out>) at
> >>>>/root/colo/jan-2016/qemu/main-loop.c:504
> >>>>#13 0x00007f0a396089ee in main_loop () at
> >>>>/root/colo/jan-2016/qemu/vl.c:1945
> >>>>#14 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized
> >>>>out>) at /root/colo/jan-2016/qemu/vl.c:4707
> >>>>
> >>>>Dave
> >>>>
> >>>>>Thanks
> >>>>>Wen Congyang
> >>>>>
> >>>>>>
> >>>>>>(qemu) info block
> >>>>>>colo-disk0 (#block080): json:{"children": [{"driver": "raw", "file":
> >>>>>>{"driver": "file", "filename": "/root/colo/bugzilla.raw"}}, {"driver":
> >>>>>>"replication", "mode": "primary", "file": {"port": "8889", "host":
> >>>>>>"ibpair", "driver": "nbd", "export": "colo-disk0"}}], "driver":
> >>>>>>"quorum", "blkverify": false, "rewrite-corrupted": false,
> >>>>>>"vote-threshold": 1} (quorum)
> >>>>>> Cache mode: writeback, direct
> >>>>>>
> >>>>>>Dave
> >>>>>>
> >>>>>>* Changlong Xie (xiecl.f...@cn.fujitsu.com) wrote:
> >>>>>>>Block replication is a very important feature which is used for
> >>>>>>>continuous checkpoints(for example: COLO).
> >>>>>>>
> >>>>>>>You can get the detailed information about block replication from here:
> >>>>>>>http://wiki.qemu.org/Features/BlockReplication
> >>>>>>>
> >>>>>>>Usage:
> >>>>>>>Please refer to docs/block-replication.txt
> >>>>>>>
> >>>>>>>This patch series is based on the following patch series:
> >>>>>>>1.
> >>>>>>>http://lists.nongnu.org/archive/html/qemu-devel/2015-12/msg04570.html
> >>>>>>>
> >>>>>>>You can get the patch here:
> >>>>>>>https://github.com/Pating/qemu/tree/changlox/block-replication-v13
> >>>>>>>
> >>>>>>>You can get the patch with framework here:
> >>>>>>>https://github.com/Pating/qemu/tree/changlox/colo_framework_v12
> >>>>>>>
> >>>>>>>TODO:
> >>>>>>>1. Continuous block replication. It will be started after basic
> >>>>>>>functions
> >>>>>>> are accepted.
> >>>>>>>
> >>>>>>>Changs Log:
> >>>>>>>V13:
> >>>>>>>1. Rebase to the newest codes
> >>>>>>>2. Remove redundant marcos and semicolon in replication.c
> >>>>>>>3. Fix typos in block-replication.txt
> >>>>>>>V12:
> >>>>>>>1. Rebase to the newest codes
> >>>>>>>2. Use backing reference to replcace 'allow-write-backing-file'
> >>>>>>>V11:
> >>>>>>>1. Reopen the backing file when starting blcok replication if it is not
> >>>>>>> opened in R/W mode
> >>>>>>>2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
> >>>>>>> when opening backing file
> >>>>>>>3. Block the top BDS so there is only one block job for the top BDS and
> >>>>>>> its backing chain.
> >>>>>>>V10:
> >>>>>>>1. Use blockdev-remove-medium and blockdev-insert-medium to replace
> >>>>>>>backing
> >>>>>>> reference.
> >>>>>>>2. Address the comments from Eric Blake
> >>>>>>>V9:
> >>>>>>>1. Update the error messages
> >>>>>>>2. Rebase to the newest qemu
> >>>>>>>3. Split child add/delete support. These patches are sent in another
> >>>>>>>patchset.
> >>>>>>>V8:
> >>>>>>>1. Address Alberto Garcia's comments
> >>>>>>>V7:
> >>>>>>>1. Implement adding/removing quorum child. Remove the option
> >>>>>>>non-connect.
> >>>>>>>2. Simplify the backing refrence option according to Stefan Hajnoczi's
> >>>>>>>suggestion
> >>>>>>>V6:
> >>>>>>>1. Rebase to the newest qemu.
> >>>>>>>V5:
> >>>>>>>1. Address the comments from Gong Lei
> >>>>>>>2. Speed the failover up. The secondary vm can take over very quickly
> >>>>>>>even
> >>>>>>> if there are too many I/O requests.
> >>>>>>>V4:
> >>>>>>>1. Introduce a new driver replication to avoid touch nbd and qcow2.
> >>>>>>>V3:
> >>>>>>>1: use error_setg() instead of error_set()
> >>>>>>>2. Add a new block job API
> >>>>>>>3. Active disk, hidden disk and nbd target uses the same AioContext
> >>>>>>>4. Add a testcase to test new hbitmap API
> >>>>>>>V2:
> >>>>>>>1. Redesign the secondary qemu(use image-fleecing)
> >>>>>>>2. Use Error objects to return error message
> >>>>>>>3. Address the comments from Max Reitz and Eric Blake
> >>>>>>>
> >>>>>>>Wen Congyang (10):
> >>>>>>> unblock backup operations in backing file
> >>>>>>> Store parent BDS in BdrvChild
> >>>>>>> Backup: clear all bitmap when doing block checkpoint
> >>>>>>> Allow creating backup jobs when opening BDS
> >>>>>>> docs: block replication's description
> >>>>>>> Add new block driver interfaces to control block replication
> >>>>>>> quorum: implement block driver interfaces for block replication
> >>>>>>> Implement new driver for block replication
> >>>>>>> support replication driver in blockdev-add
> >>>>>>> Add a new API to start/stop replication, do checkpoint to all BDSes
> >>>>>>>
> >>>>>>> block.c | 145 ++++++++++++
> >>>>>>> block/Makefile.objs | 3 +-
> >>>>>>> block/backup.c | 14 ++
> >>>>>>> block/quorum.c | 78 +++++++
> >>>>>>> block/replication.c | 545
> >>>>>>> +++++++++++++++++++++++++++++++++++++++++++++
> >>>>>>> blockjob.c | 11 +
> >>>>>>> docs/block-replication.txt | 227 +++++++++++++++++++
> >>>>>>> include/block/block.h | 9 +
> >>>>>>> include/block/block_int.h | 15 ++
> >>>>>>> include/block/blockjob.h | 12 +
> >>>>>>> qapi/block-core.json | 33 ++-
> >>>>>>> 11 files changed, 1089 insertions(+), 3 deletions(-)
> >>>>>>> create mode 100644 block/replication.c
> >>>>>>> create mode 100644 docs/block-replication.txt
> >>>>>>>
> >>>>>>>--
> >>>>>>>1.9.3
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>--
> >>>>>>Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
> >>>>>>
> >>>>>>
> >>>>>>.
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>--
> >>>>Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
> >>>>
> >>>>
> >>>>.
> >>>>
> >>>
> >>>
> >>>
> >>--
> >>Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
> >>
> >>
> >>.
> >>
> >
> >.
> >
>
>
--
Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
