Hi,
Extra fresh data about the same issue, hope this helps.
Remote server (Ubuntu Server 14.04 LTS):
libc6:amd64 2.19-0ubuntu6.9
openssh-server 1:6.6p1-2ubuntu2.7
rsync 3.1.0-2ubuntu0.2
zlib1g:amd64 1:1.2.8.dfsg-1ubuntu1
Local server (Debian GNU/Linux Sid):
libc6:amd64 2.23-4
openssh-client 1:7.2p2-8
rsync 3.1.1-3
zlib1g:amd64 1:1.2.8.dfsg-2+b1
Command:
rsync --progress --size-only --numeric-ids --delete-before -avHz -e ssh
--exclude=/proc --exclude=/dev --exclude=/sys
--exclude=/data/*/*/*/releases/*/tmp --exclude=/data/*/*/*/shared/cached-copy
--exclude=/var/cache/apt/archives
--exclude=/var/lib/lxc/*/rootfs/var/cache/apt/archives
--exclude=/root/.cache/duplicity u...@xx.xx.xx.xx:/ local_dir
### With -z
data/prod/backup/linuxfr-daily.dump.gz
2,141,938 0% 583.96kB/s 0:30:47
inflate returned -3 (431 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(557)
[receiver=3.1.1]
rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820) [generator=3.1.1]
### Without -z
OK, no error.
### With --info=all4 --debug=all4
31,918,700 0% 19.49kB/s 1236:53:32
data recv 16378 at 31911801
data recv 16374 at 31928179
data recv 7941 at 31944553
chunk[972] of size 32872 at 31951584 offset=31952494
inflate returned -3 (0 bytes)
[receiver] _exit_cleanup(code=12, file=token.c, line=557): entered
rsync: [generator] write error: Broken pipe (32)
rsync error: error in rsync protocol data stream (code 12) at token.c(557)
[receiver=3.1.1]
[receiver] _exit_cleanup(code=12, file=token.c, line=557): about to call
exit(12)
[generator] _exit_cleanup(code=10, file=io.c, line=820): entered
rsync error: error in socket IO (code 10) at io.c(820) [generator=3.1.1]
[generator] _exit_cleanup(code=10, file=io.c, line=820): about to call exit(10)
### With on local side: valgrind --trace-children=yes --leak-check=full
==21134== Memcheck, a memory error detector
==21134== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21134== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==21134== Command: rsync --progress --size-only --numeric-ids --delete-before
-avHz -e ssh --exclude=/proc --exclude=/dev --exclude=/sys
--exclude=/data/*/*/*/releases/*/tmp --exclude=/data/*/*/*/shared/cached-copy
--exclude=/var/cache/apt/archives
--exclude=/var/lib/lxc/*/rootfs/var/cache/apt/archives
--exclude=/root/.cache/duplicity u...@xx.xx.xx.xx:/ local_dir
==21134==
==21135== Memcheck, a memory error detector
==21135== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21135== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==21135== Command: /usr/bin/ssh -l user xx.xx.xx.xx rsync --server --sender
-vlHogDtprze.Lsfx --numeric-ids . /
==21135==
==21135== Warning: invalid file descriptor 1036 in syscall close()
==21135== Warning: invalid file descriptor 1037 in syscall close()
==21135== Warning: invalid file descriptor 1038 in syscall close()
==21135== Warning: invalid file descriptor 1039 in syscall close()
==21135== Use --log-fd=<number> to select an alternative log fd.
==21135== Warning: invalid file descriptor 1040 in syscall close()
==21135== Warning: invalid file descriptor 1041 in syscall close()
==21135== Warning: invalid file descriptor 1042 in syscall close()
receiving file list ...
1273655 files to consider
(...)
data/prod/backup/linuxfr-daily.dump.gz
31,813,341 2% 787.78kB/s 0:22:12
inflate returned -3 (0 bytes)
==21135==
==21135== Process terminating with default action of signal 10 (SIGUSR1)
rsync error: error in rsync protocol data stream (code 12) at token.c(557)
[receiver=3.1.1]
==21135== at 0x5E22F53: __select_nocancel (syscall-template.S:84)
==21135== by 0x11E4DC: ??? (in /usr/bin/ssh)
==21135== by 0x1133DD: ??? (in /usr/bin/ssh)
==21135== by 0x5D6372F: (below main) (libc-start.c:291)
==21166==
==21166== HEAP SUMMARY:
==21166== in use at exit: 103,527,990 bytes in 169,367 blocks
==21166== total heap usage: 169,396 allocs, 29 frees, 130,237,027 bytes
allocated
==21166==
==21135==
==21135== HEAP SUMMARY:
==21135== in use at exit: 2,377,983 bytes in 2,789 blocks
==21135== total heap usage: 10,979 allocs, 8,190 frees, 177,507,635 bytes
allocated
==21135==
==21135== 15 bytes in 1 blocks are definitely lost in loss record 23 of 694
==21135== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21135== by 0x14CAED: ??? (in /usr/bin/ssh)
==21135== by 0x14CBD7: ??? (in /usr/bin/ssh)
==21135== by 0x111DDC: ??? (in /usr/bin/ssh)
==21135== by 0x5D6372F: (below main) (libc-start.c:291)
==21135==
==21135== 56 bytes in 1 blocks are definitely lost in loss record 586 of 694
==21135== at 0x4C2DBE5: calloc (vg_replace_malloc.c:711)
==21135== by 0x14CB4B: ??? (in /usr/bin/ssh)
==21135== by 0x122EAC: ??? (in /usr/bin/ssh)
==21135== by 0x1265A3: ??? (in /usr/bin/ssh)
==21135== by 0x1229A3: ??? (in /usr/bin/ssh)
==21135== by 0x11304D: ??? (in /usr/bin/ssh)
==21135== by 0x5D6372F: (below main) (libc-start.c:291)
==21135==
==21135== 197 bytes in 1 blocks are definitely lost in loss record 619 of 694
==21135== at 0x4C2DBE5: calloc (vg_replace_malloc.c:711)
==21135== by 0x15DAFF: ??? (in /usr/bin/ssh)
==21135== by 0x125F35: ??? (in /usr/bin/ssh)
==21135== by 0x122992: ??? (in /usr/bin/ssh)
==21135== by 0x11304D: ??? (in /usr/bin/ssh)
==21135== by 0x5D6372F: (below main) (libc-start.c:291)
==21135==
==21135== 291 bytes in 1 blocks are definitely lost in loss record 638 of 694
==21135== at 0x4C2DDEF: realloc (vg_replace_malloc.c:785)
==21135== by 0x5E3A07C: __vasprintf_chk (vasprintf_chk.c:88)
==21135== by 0x14CCA2: ??? (in /usr/bin/ssh)
==21135== by 0x1263F0: ??? (in /usr/bin/ssh)
==21135== by 0x122992: ??? (in /usr/bin/ssh)
==21135== by 0x11304D: ??? (in /usr/bin/ssh)
==21135== by 0x5D6372F: (below main) (libc-start.c:291)
==21135==
==21135== 767 (176 direct, 591 indirect) bytes in 1 blocks are definitely lost
in loss record 659 of 694
==21135== at 0x4C2DBE5: calloc (vg_replace_malloc.c:711)
==21135== by 0x170155: ??? (in /usr/bin/ssh)
==21135== by 0x1113D4: ??? (in /usr/bin/ssh)
==21135== by 0x5D6372F: (below main) (libc-start.c:291)
==21135==
==21135== LEAK SUMMARY:
==21135== definitely lost: 735 bytes in 5 blocks
==21135== indirectly lost: 591 bytes in 21 blocks
==21135== possibly lost: 0 bytes in 0 blocks
==21135== still reachable: 2,376,657 bytes in 2,763 blocks
==21135== suppressed: 0 bytes in 0 blocks
==21135== Reachable blocks (those to which a pointer was found) are not shown.
==21135== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==21135==
==21135== For counts of detected and suppressed errors, rerun with: -v
==21135== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 0 from 0)
==21166== 22 bytes in 1 blocks are possibly lost in loss record 5 of 32
==21166== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21166== by 0x54D0CB9: strdup (strdup.c:42)
==21166== by 0x11084C: ??? (in /usr/bin/rsync)
==21166== by 0x547372F: (below main) (libc-start.c:291)
==21166==
==21166== 37 bytes in 2 blocks are definitely lost in loss record 6 of 32
==21166== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21166== by 0x524A8E9: poptGetNextOpt (in
/lib/x86_64-linux-gnu/libpopt.so.0.0.0)
==21166== by 0x136F14: ??? (in /usr/bin/rsync)
==21166== by 0x1106D6: ??? (in /usr/bin/rsync)
==21166== by 0x547372F: (below main) (libc-start.c:291)
==21166==
==21166== 166 bytes in 8 blocks are definitely lost in loss record 13 of 32
==21166== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21166== by 0x52492EC: ??? (in /lib/x86_64-linux-gnu/libpopt.so.0.0.0)
==21166== by 0x524A5FE: poptGetNextOpt (in
/lib/x86_64-linux-gnu/libpopt.so.0.0.0)
==21166== by 0x136F14: ??? (in /usr/bin/rsync)
==21166== by 0x1106D6: ??? (in /usr/bin/rsync)
==21166== by 0x547372F: (below main) (libc-start.c:291)
==21166==
==21166== LEAK SUMMARY:
==21166== definitely lost: 203 bytes in 10 blocks
==21166== indirectly lost: 0 bytes in 0 blocks
==21166== possibly lost: 22 bytes in 1 blocks
==21166== still reachable: 103,527,765 bytes in 169,356 blocks
==21166== suppressed: 0 bytes in 0 blocks
==21166== Reachable blocks (those to which a pointer was found) are not shown.
==21166== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==21166==
==21166== For counts of detected and suppressed errors, rerun with: -v
==21166== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)
rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820) [generator=3.1.1]
==21134==
==21134== HEAP SUMMARY:
==21134== in use at exit: 112,731,465 bytes in 338,357 blocks
==21134== total heap usage: 1,310,620 allocs, 972,263 frees, 72,542,843,606
bytes allocated
==21134==
==21134== 22 bytes in 1 blocks are possibly lost in loss record 5 of 30
==21134== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21134== by 0x54D0CB9: strdup (strdup.c:42)
==21134== by 0x11084C: ??? (in /usr/bin/rsync)
==21134== by 0x547372F: (below main) (libc-start.c:291)
==21134==
==21134== 37 bytes in 2 blocks are definitely lost in loss record 6 of 30
==21134== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21134== by 0x524A8E9: poptGetNextOpt (in
/lib/x86_64-linux-gnu/libpopt.so.0.0.0)
==21134== by 0x136F14: ??? (in /usr/bin/rsync)
==21134== by 0x1106D6: ??? (in /usr/bin/rsync)
==21134== by 0x547372F: (below main) (libc-start.c:291)
==21134==
==21134== 166 bytes in 8 blocks are definitely lost in loss record 14 of 30
==21134== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21134== by 0x52492EC: ??? (in /lib/x86_64-linux-gnu/libpopt.so.0.0.0)
==21134== by 0x524A5FE: poptGetNextOpt (in
/lib/x86_64-linux-gnu/libpopt.so.0.0.0)
==21134== by 0x136F14: ??? (in /usr/bin/rsync)
==21134== by 0x1106D6: ??? (in /usr/bin/rsync)
==21134== by 0x547372F: (below main) (libc-start.c:291)
==21134==
==21134== 14,994 bytes in 232 blocks are possibly lost in loss record 21 of 30
==21134== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21134== by 0x113D4E: ??? (in /usr/bin/rsync)
==21134== by 0x1153BD: ??? (in /usr/bin/rsync)
==21134== by 0x116DCA: ??? (in /usr/bin/rsync)
==21134== by 0x119B16: ??? (in /usr/bin/rsync)
==21134== by 0x11BA63: ??? (in /usr/bin/rsync)
==21134== by 0x11BFF4: ??? (in /usr/bin/rsync)
==21134== by 0x121918: ??? (in /usr/bin/rsync)
==21134== by 0x12D8A7: ??? (in /usr/bin/rsync)
==21134== by 0x12E217: ??? (in /usr/bin/rsync)
==21134== by 0x110E60: ??? (in /usr/bin/rsync)
==21134== by 0x547372F: (below main) (libc-start.c:291)
==21134==
==21134== 9,343,062 bytes in 168,759 blocks are definitely lost in loss record
27 of 30
==21134== at 0x4C2BBCF: malloc (vg_replace_malloc.c:299)
==21134== by 0x113D4E: ??? (in /usr/bin/rsync)
==21134== by 0x1153BD: ??? (in /usr/bin/rsync)
==21134== by 0x116DCA: ??? (in /usr/bin/rsync)
==21134== by 0x119B16: ??? (in /usr/bin/rsync)
==21134== by 0x11BA63: ??? (in /usr/bin/rsync)
==21134== by 0x11BFF4: ??? (in /usr/bin/rsync)
==21134== by 0x121918: ??? (in /usr/bin/rsync)
==21134== by 0x12D8A7: ??? (in /usr/bin/rsync)
==21134== by 0x12E217: ??? (in /usr/bin/rsync)
==21134== by 0x110E60: ??? (in /usr/bin/rsync)
==21134== by 0x547372F: (below main) (libc-start.c:291)
==21134==
==21134== LEAK SUMMARY:
==21134== definitely lost: 9,343,265 bytes in 168,769 blocks
==21134== indirectly lost: 0 bytes in 0 blocks
==21134== possibly lost: 15,016 bytes in 233 blocks
==21134== still reachable: 103,373,184 bytes in 169,355 blocks
==21134== suppressed: 0 bytes in 0 blocks
==21134== Reachable blocks (those to which a pointer was found) are not shown.
==21134== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==21134==
==21134== For counts of detected and suppressed errors, rerun with: -v
==21134== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 0 from 0)
### With remote valgrind ( --rsync-path=/root/myrsync , file containing
valgrind --trace-children=yes --leak-check=full --log-file="/tmp/myrsync.%p"
/usr/bin/rsync $@ )
receiving file list ...
1274360 files to consider
data/prod/backup/
data/prod/backup/linuxfr-daily.dump.gz
31,862,527 2% 641.03kB/s 0:27:17
inflate returned -3 (0 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(557)
[receiver=3.1.1]
rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820) [generator=3.1.1]
and on remote side:
==26229== Memcheck, a memory error detector
==26229== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==26229== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==26229== Command: /usr/bin/rsync --server --sender -vlHogDtprze.Lsfx
--numeric-ids . /
==26229== Parent PID: 26228
==26229==
==26229==
==26229== HEAP SUMMARY:
==26229== in use at exit: 108,102,459 bytes in 169,384 blocks
==26229== total heap usage: 349,613 allocs, 180,229 frees, 6,042,974,555
bytes allocated
==26229==
==26229== LEAK SUMMARY:
==26229== definitely lost: 0 bytes in 0 blocks
==26229== indirectly lost: 0 bytes in 0 blocks
==26229== possibly lost: 0 bytes in 0 blocks
==26229== still reachable: 108,102,459 bytes in 169,384 blocks
==26229== suppressed: 0 bytes in 0 blocks
==26229== Reachable blocks (those to which a pointer was found) are not shown.
==26229== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==26229==
==26229== For counts of detected and suppressed errors, rerun with: -v
==26229== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
--
Benoît Sibaud
