Den lör 17 juli 2021 kl 15:02 skrev Attila <atiw...@gmx.net>: > On 14 Jul 2021, at 21:42, Nathan Hartman <hartman.nat...@gmail.com> wrote: > > > On Tue, Jul 13, 2021 at 5:49 PM Attila <atiw...@gmx.net> wrote: > > > Hi > > I have a problem getting the svn log in a branch after sync-merging a > commit from trunk. > This commit in trunk is a merge of an old and complex branch with many > commits. > > The client accessing the repository over svn:// url. > (paths and text is redacted) > The head revision is: 10801 > > When I run the following command on the client (in the working copy), it > prints a long partial xml-log output, then hangs. > /usr/bin/svn log --xml -g -v -r 10701:HEAD /path/to/branch-wc > > When observing in "top", the command uses no visible CPU resources on > hang. (I waited ca. 2 minutes) > The hanging command does mot exits on CTRL-c, it does not exits on "kill > -TERM pid", I have to send "kill -KILL pid" to terminate it. > > When I run the command with strace it hangs at read(4, > ...SNIP... > read(4, " ( ) ( 4:file true false ) ) ( 1"..., 16384) = 16384 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d83000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d81000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d7f000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d7d000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d7b000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d79000 > read(4, "***/****-***_redacted_*****_****"..., 16384) = 14773 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d77000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d75000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d73000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d71000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d6f000 > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = > 0x7f63f1d6d000 > read(4, > > When I observe the server, there is a CPU activity at the begin, but when > the client hangs, the server seems to be in idle. > Just a corresponding svnserve process is there with no visible cpu usage. > In svnserve.log is nothing relevant to see. > > The svnserve command is: > svnserve -d -r /path/to/repositories \ > --log-file=/var/log/svnserve.log \ > --memory-cache-size 1024 \ > --cache-txdeltas yes \ > --cache-fulltexts yes > > When I try to get the xml-log on the server with the corresponding file:// > repository URL: > /usr/bin/svn log --xml -g -v -r 10701:HEAD > file://path/to/local/repositories/project/branch > The command finishes in ca 5-10 seconds and I get the xml output, but the > output has a way too many </logentry> lines. > > There are 1217 occurrences of the string “<logentry" and 33191 occurrences > of the string "</logentry>" in the output xml. > There are several thousand lines of </logentry> in a row in many places in > repeated blocks. > > Details: > Client and Server OS: > Linux 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27) x86_64 > GNU/Linux > > The repository is ca. 4 GB. > Running "svnadmin verify" on the server founds no errors. > I have no other problems with the server, checkout and commit works normal. > > svn --version > svn, version 1.10.4 (r1850624) > compiled Feb 10 2021, 20:15:45 on x86_64-pc-linux-gnu > > Copyright (C) 2019 The Apache Software Foundation. > This software consists of contributions made by many people; > see the NOTICE file for more information. > Subversion is open source software, see http://subversion.apache.org/ > > The following repository access (RA) modules are available: > > * ra_svn : Module for accessing a repository using the svn network > protocol. > - with Cyrus SASL authentication > - handles 'svn' scheme > * ra_local : Module for accessing a repository on local disk. > - handles 'file' scheme > * ra_serf : Module for accessing a repository via WebDAV protocol using > serf. > - using serf 1.3.9 (compiled with 1.3.9) > - handles 'http' scheme > - handles 'https' scheme > > The following authentication credential caches are available: > > * Plaintext cache in /home/username/.subversion > * Gnome Keyring > * GPG-Agent > * KWallet (KDE) > > > > Hello Attila, > > Thanks for this detailed explanation. There are quite a few important > clues here. > > To help narrow down the search for the culprit: Are you able to run > the same 'svn log' command against the same working copy, but without > the '--xml', and get a correct output in a reasonable amount of time? > > > Hi Nathan > > thanks for the suggestions. > > The command without the --xml parameter hangs on the client and works on > the server. > The text output on the server seems be ok. > This suggests that the hang and the closing xml tags are two unrelated > problems. >
Hi Attila, Thanks for the detailed explaination! I agree these are two different problems. More below > If log without '--xml' works and log with '--xml' gets stuck, is it > feasible to try to bisect the revision range, e.g., run log on the > range r10701:10751, and in a separate invocation on r10751:10801, and > > > I tried that, it seems that the problem was not the last sync-merge commit. > The command succeeds when there is no sync-merge commit in the range. > There are two sync-merges in the range the first one is a small merge of > 3-4 changed files. These 3-4 files are unmodified in the branch. > Whenever i include one of these sync-merges in the range, the command > hangs. > What if you run the command on the whole range but without '-g'? I'm expecting that should work, both with and without '--xml'. try to see if there's a revision in particular where SVN is getting > confused? If so, then perhaps knowing a little more about that > revision could help narrow it down. > > By the way, is this (1.10.4) the SVN version on the server, the > client, or both? > > > both. > The server and client is Debian Buster and I have updated both to the > latest version hoping it will help. > The installed subversion package is: > subversion/stable,stable,now 1.10.4-1+deb10u2 amd64 [installed] > > The client is a private vagrant vm running on macOS in VMware Fusion pro, > the server is an ESXi vm. > > Googling around gives me these two somewhat related hits: > https://issues.apache.org/jira/browse/SVN-4856 > https://issues.apache.org/jira/browse/SVN-4711 > > But I do'nt use the --search parameter. > > > It could nevertheless be related. The spurious XML closing tags seems > to ring a bell. > > I have been chasing svn log --xml for the last two days. As far as I understand, both issues are caused by the -g (or --use-merge-history) argument. This forces the code to recursively consider the log of the merge history. In #4711 it is the --search filtering that causes the client to emit a </logentry> tag even when it didn't emit the corresponding <logentry> tag. I believe I have found a solution., details in http://mail-archives.apache.org/mod_mbox/subversion-dev/202108.mbox/%3C8215e3c9-2b81-c703-eaf9-4199ada8e485%40gmail.com%3E. When this is committed, I'm planning to go after 4856. In #4856 it seems that the library is issuing the callback for the closing tag (</logentry>) even when it shouldn't. The same issue would affects the log in plain text format, but since there is no closing tag, it is not visible in the output. It is fairly easy to produce the issue of merging in multiple steps. Is this a bug or are there any suggestions how to solve this problem? > > > It sounds like a bug. My suggestions above. > > > Here is a preliminary backtrace of the hanging client (without debug > symbols for now) > I need some time to spin up a copy of the server and the client and > re-compile svn with debug symbols. > I will come back with more details as soon as i have the time to do more > tests. > > # gdb /usr/bin/svn 1399 > GNU gdb (Debian 8.2.1-2+b3) 8.2.1 > Copyright (C) 2018 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later < > http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "x86_64-linux-gnu". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>. > Find the GDB manual and other documentation resources online at: > <http://www.gnu.org/software/gdb/documentation/>. > > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /usr/bin/svn...(no debugging symbols found)...done. > Attaching to program: /usr/bin/svn, process 1399 > Reading symbols from /lib/x86_64-linux-gnu/libsvn_client-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_wc-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_ra-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_delta-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_diff-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_subr-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libaprutil-1.so.0...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libapr-1.so.0...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading > symbols from > /usr/lib/debug/.build-id/e9/1114987a0147bd050addbd591eb8994b29f4b3.debug...done. > done. > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols > from > /usr/lib/debug/.build-id/18/b9a9a8c523e5cfe5b5d946d605d09242f09798.debug...done. > done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_ra_local-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_repos-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_fs-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_ra_serf-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libexpat.so.1...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsqlite3.so.0...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/liblz4.so.1...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libutf8proc.so.2...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libuuid.so.1...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols > from > /usr/lib/debug/.build-id/5d/cf98ad684962be494af28a1051793fd39e4ebc.debug...done. > done. > Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols > from > /usr/lib/debug/.build-id/c6/966ba263bacba7f67c23643ba373aa51a162e9.debug...done. > done. > Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols > from > /usr/lib/debug/.build-id/d3/583c742dd47aaa860c5ae0c0c5bdbcd2d54f61.debug...done. > done. > Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from > /usr/lib/debug/.build-id/f2/5dfd7b95be4ba386fd71080accae8c0732b711.debug...done. > done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_fs_fs-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_fs_x-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_fs_base-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsvn_fs_util-1.so.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libserf-1.so.1...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols > from > /usr/lib/debug/.build-id/88/5dda4b4a5cea600e7b5b98c1ad86996c8d2299.debug...done. > done. > Reading symbols from /lib/x86_64-linux-gnu/libdb-5.3.so...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading > symbols from > /usr/lib/debug/.build-id/02/6c3ba167f64f631eb8781fca2269fbc2ee7ca5.debug...done. > done. > Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.1...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.1...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libk5crypto.so.3...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging > symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libkrb5support.so.0...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no > debugging symbols found)...done. > Reading symbols from /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so...(no > debugging symbols found)...done. > Reading symbols from /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so...(no > debugging symbols found)...done. > Reading symbols from /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so...(no > debugging symbols found)...done. > Reading symbols from /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so...(no > debugging symbols found)...done. > Reading symbols from /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so...(no > debugging symbols found)...done. > Reading symbols from /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so...(no > debugging symbols found)...done. > Reading symbols from /usr/lib/x86_64-linux-gnu/sasl2/libplain.so...(no > debugging symbols found)...done. > Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading > symbols from > /usr/lib/debug/.build-id/4b/ff8b782e1602c596e856bdef06e642e50e7fa7.debug...done. > done. > Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2...Reading > symbols from > /usr/lib/debug/.build-id/bb/8b756d3859e8fe20d7aa6d8255f27942589f47.debug...done. > done. > 0x00007f276a1fc50e in __libc_read (fd=4, buf=0x7f2768e19000, nbytes=16384) > at ../sysdeps/unix/sysv/linux/read.c:26 > 26 ../sysdeps/unix/sysv/linux/read.c: No such file or directory. > (gdb) bt > #0 0x00007f276a1fc50e in __libc_read (fd=4, buf=0x7f2768e19000, > nbytes=16384) at ../sysdeps/unix/sysv/linux/read.c:26 > #1 0x00007f276a22f37f in apr_socket_recv () from > /lib/x86_64-linux-gnu/libapr-1.so.0 > #2 0x00007f2769fc1353 in ?? () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #3 0x00007f2769fc158c in svn_ra_svn.stream_read () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #4 0x00007f2769fbb242 in ?? () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #5 0x00007f2769fbb61a in ?? () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #6 0x00007f2769fbb68b in ?? () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #7 0x00007f2769fbbc7f in ?? () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #8 0x00007f2769fbbcbe in ?? () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #9 0x00007f2769fbde67 in svn_ra_svn.read_item () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #10 0x00007f2769fb583a in ?? () from > /lib/x86_64-linux-gnu/libsvn_ra_svn-1.so.1 > #11 0x00007f276a46033a in svn_client_log5 () from > /lib/x86_64-linux-gnu/libsvn_client-1.so.1 > #12 0x000055f9507b6779 in ?? () > #13 0x000055f9507c2541 in ?? () > #14 0x000055f9507a7513 in ?? () > #15 0x00007f276a04e09b in __libc_start_main (main=0x55f9507a7490, argc=7, > argv=0x7ffee0fc23c8, init=<optimized out>, fini=<optimized out>, > rtld_fini=<optimized out>, stack_end=0x7ffee0fc23b8) at > ../csu/libc-start.c:308 > #16 0x000055f9507a759a in ?? () > (gdb) > Any chance to do the same on the server? Can you attach to the repository in any other way (http/https or file) to verify if the error is in svnserve or in the repository code? Kind regards Daniel Sahlberg
