Re: Searching entire repository for a file (fastsvncrawler)

[Stefan Sperling]

On Wed, Jul 22, 2020 at 03:36:36PM -0700, Kenneth Porter wrote:
> I need to locate a file in a client's large repository. I found
> fastsvncrawler which uses svn_ra_do_status2 to rapidly dump the entire
> repository as a directory listing. Has anyone built a Windows binary? Or
> perhaps it's made it into the distribution? (I access the repo over a Cisco
> VPN from Windows, or I'd just build it on Linux. I fear I'll have to learn
> how to build Subversion on Windows, which looks daunting.)
> 
> 
> 
> How it works:
> 
> 
> 

Are you aware of the built-in svn list --search feature, which has
been available since SVN 1.10.0?

For example:

$ svn list --depth=infinity --search svn.c ^/subversion/trunk
subversion/svn/svn.c
$

Also as of SVN 1.10 the server supports a special-purpose 'list' request
to speed this up.

In any case, the fastest way to search will likely be with a file:// URL,
assuming you can get direct access to the repository for this purpose.

Regards,
Stefan

Re: Searching entire repository for a file (fastsvncrawler)

[Stefan Sperling]

On Thu, Jul 23, 2020 at 11:54:52AM +0200, Stefan Sperling wrote:
> On Wed, Jul 22, 2020 at 03:36:36PM -0700, Kenneth Porter wrote:
> > I need to locate a file in a client's large repository. I found
> > fastsvncrawler which uses svn_ra_do_status2 to rapidly dump the entire
> > repository as a directory listing. Has anyone built a Windows binary? Or
> > perhaps it's made it into the distribution? (I access the repo over a Cisco
> > VPN from Windows, or I'd just build it on Linux. I fear I'll have to learn
> > how to build Subversion on Windows, which looks daunting.)
> > 
> > 
> > 
> > How it works:
> > 
> > 
> > 
> 
> Are you aware of the built-in svn list --search feature, which has
> been available since SVN 1.10.0?
> 
> For example:
> 
> $ svn list --depth=infinity --search svn.c ^/subversion/trunk
> subversion/svn/svn.c
> $

I forgot to mention that that this feature supports pattern matching,
and that the pattern argument may need quoting. From 'svn help list':

  --search ARG : use ARG as search pattern (glob syntax, case-
 and accent-insensitive, may require quotation marks
 to prevent shell expansion)

> Also as of SVN 1.10 the server supports a special-purpose 'list' request
> to speed this up.
> 
> In any case, the fastest way to search will likely be with a file:// URL,
> assuming you can get direct access to the repository for this purpose.
> 
> Regards,
> Stefan

Re: svnmucc --revision 0 no longer works when creating a file

[sebb]

On Thu, 23 Jul 2020 at 01:17, Daniel Shahaf  wrote:
>
> sebb wrote on Wed, 22 Jul 2020 22:34 +0100:
> > On Wed, 22 Jul 2020 at 17:46, Nathan Hartman  
> > wrote:
> > >
> > > On Wed, Jul 22, 2020 at 12:10 PM sebb  wrote:
> > > > > Use the machine-parseable E42 error codes.  That's exactly what
> > > > > they're for.  (which-error.py and svn_error_symbolic_name() can be 
> > > > > used
> > > > > to convert numbers to symbolic names.)
> > > >
> > > > Where are these error codes defined?
> > > > I could not find any reference to them in the documentation.
> > >
> > > If you mean where in the source code:
> > >
> > > subversion/include/svn_error_codes.h
> > > which is included by subversion/include/svn_error.h, which is further
> > > included by subversion/svnmucc/svnmucc.c.
> > >
> > > Hope that helps,
> >
> > Thanks, but not really.
> >
> > If the error codes are intended to be machine-parseable then the
> > programmer needs to have documentation of the values and their
> > meanings.
> > There needs to be at least a mention in the documentation that such
> > codes exist and where to find the values. If there is such a mention,
> > I could not find it.
>
> Doxygen docs of svn_error_t::apr_err.

But how does one find that information?

> > Also, I had a look at the file and it does not show the numeric
> > values,
>
> I refer you again to which-error.py.

Again, how is one supposed to know about this script?
Also, I assume this requires Python.

> > and whilst there is a text string associated with each one it
> > does not detail when it might be used.
> > Nor indeed do the strings agree with the actual messages generated by
> > SVN as far as I can tell.
>
> No, they don't.  The strings in that file are defaults, only used when
> the line of code that raised an error didn't provide a more specific
> string.
>
> > e.g. svn list reports the following:
> >
> > svn: warning: W160013: Path '/x' not found
> > svn: E29: Could not list all targets because some targets don't exist
>
> Great.  Run which-error.py on this and you'll find that the former code
> is SVN_ERR_FS_NOT_FOUND.  That's basically the svn API equivalent of
> ENOENT, i.e., "There's no node by that name in the versioned
> filesystem".  Therefore, you'll want to write your script to handle
> E160013 errors by ignoring them and continuing.

I would need to know whether E29 can only be produced when a
target does not exist.
Otherwise it might look like the file does not exist when it does.

> The complication here is that `svn list` supports multiple target
> arguments, and has to handle the case that some but not all of them are
> invalid.  It does so by reporting the errors from individual targets as
> W* codes rather than E* codes and adding a generic E29 error at the
> end.  You can either parse stderr despite this complication, or use the
> API directly, in which case you'll sidestep this complication entirely
> (you'll get just one integer, rather than two).

What API are you referring to here?

> Daniel

Re: svnmucc: new commands: add and modify/update

[sebb]

On Thu, 23 Jul 2020 at 00:59, Daniel Shahaf  wrote:
>
> sebb wrote on Wed, 22 Jul 2020 22:44 +0100:
> > The SVN put command can add a new file or update an existing one.
> >
> > As part of a batch update it may be necessary to ensure that a
> > particular file will be created and not updated - or vice versa.
> > That is currently not at all easy to do, which is a shame as svnmucc
> > is otherwise very useful for writing atomic updates that are not
> > possible with the svn client.
>
> What can svnmucc(1) do that svn(1) can't?

Atomic updates.

AIUI an svnmucc script only succeeds if all the individual commands succeed.

To do that with single commands would require reversion and/or retries.

If the reason for an svn client failure is a network or server issue,
it's going to be very difficult to tidy up.

> > Would there be any support for extending svnmucc to add these operations?
> > Either as options to put, or as separate commands.
>
> The standard solutions for your situation are:
>
> 1.
> - Get HEAD's value as an integer
> - Check file existence/inexistence in that revision
> - Run 'svnmucc -r'
>
> 2.
> svn co --depth=empty $URL wc
> svn up --set-depth=infinity wc/foo
> ⋮
> # (as posted in 
> https://mail-archives.apache.org/mod_mbox/subversion-users/202007.mbox/%3C20200712142604.128f80eb%40tarpaulin.shahaf.local2%3E)
>
> In what ways do they fall short?
>

Re: Searching entire repository for a file (fastsvncrawler)

[Kenneth Porter]

--On Thursday, July 23, 2020 12:54 PM +0200 Stefan Sperling  
wrote:



Are you aware of the built-in svn list --search feature, which has
been available since SVN 1.10.0?

For example:

$ svn list --depth=infinity --search svn.c ^/subversion/trunk
subversion/svn/svn.c
$

Also as of SVN 1.10 the server supports a special-purpose 'list' request
to speed this up.

In any case, the fastest way to search will likely be with a file:// URL,
assuming you can get direct access to the repository for this purpose.


I was not. Very nice, particularly the pattern matching. Can I determine 
the server version remotely to see if this is supported? (Alas, I don't 
have direct access.) Is the globbing done on the server with the 1.10 
support? That would make it as fast as direct access.

Re: Searching entire repository for a file (fastsvncrawler)

[Stefan Sperling]

On Thu, Jul 23, 2020 at 08:05:12AM -0700, Kenneth Porter wrote:
> --On Thursday, July 23, 2020 12:54 PM +0200 Stefan Sperling 
> wrote:
> 
> > Are you aware of the built-in svn list --search feature, which has
> > been available since SVN 1.10.0?
> > 
> > For example:
> > 
> > $ svn list --depth=infinity --search svn.c ^/subversion/trunk
> > subversion/svn/svn.c
> > $
> > 
> > Also as of SVN 1.10 the server supports a special-purpose 'list' request
> > to speed this up.
> > 
> > In any case, the fastest way to search will likely be with a file:// URL,
> > assuming you can get direct access to the repository for this purpose.
> 
> I was not. Very nice, particularly the pattern matching. Can I determine the
> server version remotely to see if this is supported?

Some servers will advertise the SVN version on pages which can be visited
with a web browser. But this depends on the server's configuration. You may
have to ask the administrator to be sure about the server's exact version.

If the server supports the feature it will send an "svn/list" DAV header.
The full capability string to look for in a trace of an SVN HTTP session
would be: "http://subversion.tigris.org/xmlns/dav/svn/list";

I suppose the easiest way forward might be to simply try it out and see
how long it takes.

> (Alas, I don't have
> direct access.) Is the globbing done on the server with the 1.10 support?

Yes, globbing would be done on the server side.

Re: Searching entire repository for a file (fastsvncrawler)

[Kenneth Porter]

--On Thursday, July 23, 2020 6:22 PM +0200 Stefan Sperling  
wrote:



Some servers will advertise the SVN version on pages which can be visited
with a web browser. But this depends on the server's configuration. You
may have to ask the administrator to be sure about the server's exact
version.


Drat. My own server is on CentOS 7 and is still back at 1.7. (The "joy" of 
using a conservative operating system.) So I need to look for someone 
packaging the latest Subversion release for RHEL for when I start using 
that with my own work. The server I need to search is a Windows server so 
it may be quite a bit newer.

Re: svnmucc: new commands: add and modify/update

[Daniel Shahaf]

sebb wrote on Thu, 23 Jul 2020 11:30 +0100:
> On Thu, 23 Jul 2020 at 00:59, Daniel Shahaf  wrote:
> >
> > sebb wrote on Wed, 22 Jul 2020 22:44 +0100:  
> > > The SVN put command can add a new file or update an existing one.
> > >
> > > As part of a batch update it may be necessary to ensure that a
> > > particular file will be created and not updated - or vice versa.
> > > That is currently not at all easy to do, which is a shame as svnmucc
> > > is otherwise very useful for writing atomic updates that are not
> > > possible with the svn client.  
> >
> > What can svnmucc(1) do that svn(1) can't?  
> 
> Atomic updates.
> 
> AIUI an svnmucc script only succeeds if all the individual commands succeed.
> 
> To do that with single commands would require reversion and/or retries.
> 
> If the reason for an svn client failure is a network or server issue,
> it's going to be very difficult to tidy up.

You're describing CVS, not Subversion.

Re: svnmucc --revision 0 no longer works when creating a file

[Daniel Shahaf]

sebb wrote on Thu, 23 Jul 2020 11:18 +0100:
> On Thu, 23 Jul 2020 at 01:17, Daniel Shahaf  wrote:
> > You can either parse stderr despite this complication, or use the
> > API directly, in which case you'll sidestep this complication entirely
> > (you'll get just one integer, rather than two).  
> 
> What API are you referring to here?

The C API or the various language bindings, bypassing the cmdline client.

Re: Searching entire repository for a file (fastsvncrawler)

[Daniel Shahaf]

Stefan Sperling wrote on Thu, 23 Jul 2020 17:22 +0200:
> Some servers will advertise the SVN version on pages which can be visited
> with a web browser. But this depends on the server's configuration. You may
> have to ask the administrator to be sure about the server's exact version.
> 
> If the server supports the feature it will send an "svn/list" DAV header.
> The full capability string to look for in a trace of an SVN HTTP session
> would be: "http://subversion.tigris.org/xmlns/dav/svn/list";
> 

I'm guessing the equivalent svnserve capability is called just "list".
svnserve capabilities are printed as soon as a TCP channel is
established, in the server's greeting.

> I suppose the easiest way forward might be to simply try it out and see
> how long it takes.

Re: svnmucc: new commands: add and modify/update

[sebb]

On Thu, 23 Jul 2020 at 23:44, Daniel Shahaf  wrote:
>
> sebb wrote on Thu, 23 Jul 2020 11:30 +0100:
> > On Thu, 23 Jul 2020 at 00:59, Daniel Shahaf  wrote:
> > >
> > > sebb wrote on Wed, 22 Jul 2020 22:44 +0100:
> > > > The SVN put command can add a new file or update an existing one.
> > > >
> > > > As part of a batch update it may be necessary to ensure that a
> > > > particular file will be created and not updated - or vice versa.
> > > > That is currently not at all easy to do, which is a shame as svnmucc
> > > > is otherwise very useful for writing atomic updates that are not
> > > > possible with the svn client.
> > >
> > > What can svnmucc(1) do that svn(1) can't?
> >
> > Atomic updates.
> >
> > AIUI an svnmucc script only succeeds if all the individual commands succeed.
> >
> > To do that with single commands would require reversion and/or retries.
> >
> > If the reason for an svn client failure is a network or server issue,
> > it's going to be very difficult to tidy up.
>
> You're describing CVS, not Subversion.

No: I'm not referring to tidying up the repository itself, but the
application level inconsistency that can result if a related sequence
of svn commits does not complete.

The issue I am trying to solve is how to ensure related changes to a
subversion repository either all occur or none do.

For example, adding a new file to a directory, and updating a listing
with details of that file.
The file must not already exist. The listing is in a different part of
the repository.

Whilst it would probably be possible to create a sparse checkout of
the different parts of the repository, this is not trivial.

It seems like an ideal job for svnmucc.

However svnmucc does not directly support svn 'add' functionality.

Whilst this can also be worked round, it is quite tedious and easy to get wrong.

I am suggesting that 'add' functionality could be added to svnmucc itself.
This would make it more versatile, especially for use in shell scripts.

(*) I am referring to application-level consistency here, not
consistency of the repository itself.

Re: svnmucc: new commands: add and modify/update

[Daniel Sahlberg]

Den fre 24 juli 2020 01:46sebb  skrev:

> I am suggesting that 'add' functionality could be added to svnmucc itself.
> This would make it more versatile, especially for use in shell scripts.
>

Unless I'm mistaken, add is a purely local action within a working copy.
And the point with svnmucc is that you work with a repository without
having a wc. How should add work with svnmucc?

Kind regards
Daniel Sahlberg

>

Re: svnmucc: new commands: add and modify/update

[Daniel Shahaf]

Daniel Sahlberg wrote on Fri, 24 Jul 2020 05:53 +00:00:
> Den fre 24 juli 2020 01:46sebb  skrev:
> > I am suggesting that 'add' functionality could be added to svnmucc itself.
> > This would make it more versatile, especially for use in shell scripts.
> 
> Unless I'm mistaken, add is a purely local action within a working 
> copy. And the point with svnmucc is that you work with a repository 
> without having a wc. How should add work with svnmucc?

It would work exactly like «svnmucc put $dirent $URL», except that it
would fail if $URL already exists, similar to open(O_EXCL) in C.