Re: Dealing with very old repo format (version 1)

[Nico Kadel-Garcia]

On Wed, Apr 29, 2015 at 12:03 AM, Branko Čibej  wrote:
> On 29.04.2015 05:09, Nico Kadel-Garcia wrote:
>> On Tue, Apr 28, 2015 at 5:03 PM, Andrew Reedick  wrote:
>>> Does anyone have any tips on how to upgrade a very old repo?  The db/format 
>>> lists "1".  A 1.8 svn client cannot hotcopy, dump or "svnadmin upgrade" 
>>> such an old repo, all of which fail with "svnadmin: E720002: Can't open 
>>> file 'devel\db\current': The system cannot find the file specified."
>>>
>>> Do I need find a really old svn client (1.3?) and upgrade?  Do I need to 
>>> manually create the db/current file?
>>>
>>>
>>> Supposedly , a format of "1" is from pre-svn 1.0.  
>>> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_repos/repos.h
>>>  -> "Formats 0, 1 and 2 were pre-1.0."
>> Why can't you do a fresh working copy, and copy all files except those
>> in '.svn' subdirectories?
>
> Without db/current, he can't perform a checkout, and if he could and
> just copied the file, he'd be throwing away all history.

What? "hotcopy", "dump", and "svnadmin upgrade" are all operations on
the repository itself. The database *of the repository* is irrelevant
to fresh checkouts, as long as they can speak any relevant network
protocols associated with the old service. And heck, if you need to
pull logic this way, use 'git-svn' to pull it into a git repository
and push it back to a new, clean, Subversion repository.

Unless, Andrew? Do you not have any network based service associated
with the old repository?

Re: Dealing with very old repo format (version 1)

[Branko Čibej]

On 29.04.2015 07:14, Nico Kadel-Garcia wrote:
> On Wed, Apr 29, 2015 at 12:03 AM, Branko Čibej  wrote:
>> On 29.04.2015 05:09, Nico Kadel-Garcia wrote:
>>> On Tue, Apr 28, 2015 at 5:03 PM, Andrew Reedick  wrote:
 Does anyone have any tips on how to upgrade a very old repo?  The 
 db/format lists "1".  A 1.8 svn client cannot hotcopy, dump or "svnadmin 
 upgrade" such an old repo, all of which fail with "svnadmin: E720002: 
 Can't open file 'devel\db\current': The system cannot find the file 
 specified."

 Do I need find a really old svn client (1.3?) and upgrade?  Do I need to 
 manually create the db/current file?


 Supposedly , a format of "1" is from pre-svn 1.0.  
 https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_repos/repos.h
  -> "Formats 0, 1 and 2 were pre-1.0."
>>> Why can't you do a fresh working copy, and copy all files except those
>>> in '.svn' subdirectories?
>> Without db/current, he can't perform a checkout, and if he could and
>> just copied the file, he'd be throwing away all history.
> What? "hotcopy", "dump", and "svnadmin upgrade" are all operations on
> the repository itself. The database *of the repository* is irrelevant
> to fresh checkouts, as long as they can speak any relevant network
> protocols associated with the old service.


You proposed copying "all files except those in .svn directories" from a
fresh checkout. This implies actually doing a checkout, which you can't
do if your repository is corrupt, and it's corrupt if it's a FSFS repo
without a db/current file. It's irrelevant whether you're doing a
checkout from a local repository or over the network or via git-svn or
whatever.

-- Brane

Re: Dealing with very old repo format (version 1)

[Philip Martin]

Branko Čibej  writes:

> In any case, 1.8 /should/ be able to dump an FSFSv1 repository, and the
> /db/current file should exists; it's been around since FSFSv1.
> You can try recreating it; the format is described here:
>
> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure

Running 'svnadmin recover' should recreate the missing file.

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

Re: Dealing with very old repo format (version 1)

[Nico Kadel-Garcia]

On Wed, Apr 29, 2015 at 1:32 AM, Branko Čibej  wrote:
> On 29.04.2015 07:14, Nico Kadel-Garcia wrote:
>> On Wed, Apr 29, 2015 at 12:03 AM, Branko Čibej  wrote:
>>> On 29.04.2015 05:09, Nico Kadel-Garcia wrote:
 On Tue, Apr 28, 2015 at 5:03 PM, Andrew Reedick  
 wrote:
> Does anyone have any tips on how to upgrade a very old repo?  The 
> db/format lists "1".  A 1.8 svn client cannot hotcopy, dump or "svnadmin 
> upgrade" such an old repo, all of which fail with "svnadmin: E720002: 
> Can't open file 'devel\db\current': The system cannot find the file 
> specified."
>
> Do I need find a really old svn client (1.3?) and upgrade?  Do I need to 
> manually create the db/current file?
>
>
> Supposedly , a format of "1" is from pre-svn 1.0.  
> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_repos/repos.h
>  -> "Formats 0, 1 and 2 were pre-1.0."
 Why can't you do a fresh working copy, and copy all files except those
 in '.svn' subdirectories?
>>> Without db/current, he can't perform a checkout, and if he could and
>>> just copied the file, he'd be throwing away all history.
>>
>> What? "hotcopy", "dump", and "svnadmin upgrade" are all operations on
>> the repository itself. The database *of the repository* is irrelevant
>> to fresh checkouts, as long as they can speak any relevant network
>> protocols associated with the old service.
>
> You proposed copying "all files except those in .svn directories" from a
> fresh checkout. This implies actually doing a checkout, which you can't
> do if your repository is corrupt, and it's corrupt if it's a FSFS repo
> without a db/current file. It's irrelevant whether you're doing a
> checkout from a local repository or over the network or via git-svn or
> whatever.

I was assuming his core problem was the age of the repository and
finding compatible server side tools to help with the upgrade. I also
admit, myself, that when migrating old repositories, I could generally
not care less about the history, which has often become quite
cluttered with old mistakes and accidental binary commits.

If it's genuinely corrupt, that's a distinct issue from "all my
current tools are unable to read such an old repository", which is
what it sounded like. I'll be curious to see if Philip Martin's
suggestion to use 'svnadmin recover' helps out with that.

I'd also make *absolutely sure* to use "rsync -a" or "cp -a" on Linux
systems to copy the repository somewhere else, and only touch the
*copy*, until this is straightened out.

RE: Dealing with very old repo format (version 1)

[Andrew Reedick]

> -Original Message-
> From: Branko Čibej [mailto:br...@wandisco.com] 

> Are we talking about the repository format or the FSFS format here? If 
> /db/fs-type says "fsfs" then the repository format
> (/format) is probably 3 and you're talking about /db/format, 
> yes? The distinction is important.

Yes, I'm referring to db/* files.
$ more format  fs-type
::
format
::
1
::
fs-type
::
fsfs

>
> In any case, 1.8 /should/ be able to dump an FSFSv1 repository, and the 
> /db/current file should exists; it's been around since FSFSv1.
> You can try recreating it; the format is described here:
>
> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_fs_fs/structure
>
> To find the youngest revision, look for the highest-numered file in 
> /db/revs. If you're just going to dump the repository, it should be 
> safe to set the next-node-id and next-copy-id to some large number, say 
> 99; but I wouldn't recommend trying to commit to the repository.
>
> Please report if the above works or I'm just talking through my hat. :)
>
> -- Brane

Good News:  Recreating the db/current file worked in that it allowed 'svnadmin 
dump' to run.  

Bad News:  However, it seems that I have bigger issues:
* Dumped revision 109662.
svnadmin: E720002: Can't open file 'devel_copy\db\revprops\109663': The 
system cannot find the file specified.

When I sort the files in db/revs numerically, I see gaps in the revs:
109661
109662
109668
109734
...
109735
157939
157940
157941
159433 
159607 
160600 
160601
162409
And 'ls | wc -l' in dev/revs shows that there are 141,768 files, but the 
highest rev in db/revs is 162409...

*sigh*  I guess I can try piecemeal dumps.

Thanks for the help everyone, but I'm thinking my missing db/current file is a 
symptom of the repo being mangled (probably due to inadequate backup procedures 
or a bad restore from tape.   Rev 1 is from 2006, and the repo was just around 
for reference so no real worries.) 

Re: Dealing with very old repo format (version 1)

[Philip Martin]

Andrew Reedick  writes:

> Bad News:  However, it seems that I have bigger issues:
>   * Dumped revision 109662.
>   svnadmin: E720002: Can't open file
> 'devel_copy\db\revprops\109663': The system cannot find the file
> specified.
>
> When I sort the files in db/revs numerically, I see gaps in the revs:
>   109661
>   109662
>   109668
>   109734
>   ...
>   109735
>   157939
>   157940
>   157941
>   159433 
>   159607 
>   160600 
>   160601
>   162409
> And 'ls | wc -l' in dev/revs shows that there are 141,768 files, but
> the highest rev in db/revs is 162409...

You can replace a missing revprop file with a file containing a single
line:

END

There is not much you can do replace missing revision files so this may
not help you.

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

Re: Cannot checkout or clean up using 1.9 dev build

[Bert Huijben]

I’m not sure if the \\?\ format is supported in the new rename api(as this is 
an option in certain publi apis and not a low level supported format), but it 
should certainly be used in the fallback codepath where the normal rename is 
tried, like hiw it is in apr.


But if the new api fails with the limit, it is quite likely that it supports 
the \\?\ format. Some more recent new in Windows 8+ apis just dropped the path 
length limit and therefore don’t support the \\?\ format… but that doesn’t 
apply here as this api was new in Vista.




I hope somebody looks at this earlier, but I’ll will look into this early next 
week.


Bert




Sent from Windows Mail





From: Ivan Zhakov
Sent: ‎Tuesday‎, ‎April‎ ‎28‎, ‎2015 ‎9‎:‎07‎ ‎AM
To: Benjamin Fritz, Bert Huijben
Cc: 'subversion'





On 27 April 2015 at 21:35, Benjamin Fritz  wrote:
> Apparently I'm not subscribed to get list emails; please CC me on future
> responses. I had to get this message from the archive :-)
>
> Branko Čibej wrote:
>> On 27.04.2015 18:06, Benjamin Fritz wrote:
>> > I get the following error when I try to check out a working copy from
>> > my repository at work (exact path names obfuscated but length
>> > unchanged, in case it matters
>>
>> It probably does matter ... FWIW, I believe you did change the length of
>> the file name, the workqueue record says 234 chars but the name in the
>> record is 240; still, that shouldn't be an issue.
>>
>
> Oops. You're right. My actual directory has only 3 characters where I
> replaced with "Prog" in the path string. Still, those 6 fewer
> characters are probably irrelevant.
>
>> > ), leaving a partially downloaded working copy:
>> >
>> > svn: E155009: Failed to run the WC DB work queue associated with
>> > 'C:\Project_Files\Proj_Dev\Prog_svn_1.9_cmdline', work item 2160
>> > (file-install 234
>> >
>> > Logical_Architecture/APP/App_PL_Verification/XXX/XXX_XXXs/XXX_7_APPDB_PROJ_PROG--0021_BE/APPDB_PROJ_PROG--0021.SUP/APPDB_PROJ_PROG--0021/APPDB_PROJ_PROG--0021/APPDB_PROJ_PROG--0021.SUP/XXX_APPDB_PROJ_PROG--002101.CRC
>> > 1 0 1 1)
>> > svn: E720123: The filename, directory name, or volume label syntax is
>> > incorrect.
>>
>> The full path of the file would be 281 characters long, which is more
>> than Windows' limit of 260. But Subversion goes to a lot of trouble to
>> avoid the path limit ... I wonder what's going on here.

It seems Windows specific implementation of
svn_stream__install_stream() doesn't prepend path with "\\?\" for long
names. Replacing svn_utf__win32_utf8_to_utf16() call with
svn_io__utf8_to_unicode_longpath() should fix the problem, but I
didn't try yet.

Bert, is it intentional behavior or just small typo?


-- 
Ivan Zhakov

Re: Cannot checkout or clean up using 1.9 dev build

[Branko Čibej]

On 29.04.2015 21:28, Bert Huijben wrote:
> I’m not sure if the \\?\  format is supported in
> the new rename api(as this is an option in certain publi apis and not
> a low level supported format), but it should certainly be used in the
> fallback codepath where the normal rename is tried, like hiw it is in apr.
>
> But if the new api fails with the limit, it is quite likely that it
> supports the \\?\  format. Some more recent new in
> Windows 8+ apis just dropped the path length limit and therefore don’t
> support the \\?\  format… but that doesn’t apply
> here as this api was new in Vista.
>
> I hope somebody looks at this earlier, but I’ll will look into this
> early next week.

FWIW, Ivan already changed the code and added a test case, and there's a
pending backport proposal in 1.9.x STATUS; the Windows buildbots seem
happy with the change.

-- Brane

> *From:* Ivan Zhakov 
> *Sent:* ‎Tuesday‎, ‎April‎ ‎28‎, ‎2015 ‎9‎:‎07‎ ‎AM
> *To:* Benjamin Fritz , Bert Huijben
> 
> *Cc:* 'subversion' 
>
> On 27 April 2015 at 21:35, Benjamin Fritz  wrote:
> > Apparently I'm not subscribed to get list emails; please CC me on future
> > responses. I had to get this message from the archive :-)
> >
> > Branko Čibej wrote:
> >> On 27.04.2015 18:06, Benjamin Fritz wrote:
> >> > I get the following error when I try to check out a working copy from
> >> > my repository at work (exact path names obfuscated but length
> >> > unchanged, in case it matters
> >>
> >> It probably does matter ... FWIW, I believe you did change the
> length of
> >> the file name, the workqueue record says 234 chars but the name in the
> >> record is 240; still, that shouldn't be an issue.
> >>
> >
> > Oops. You're right. My actual directory has only 3 characters where I
> > replaced with "Prog" in the path string. Still, those 6 fewer
> > characters are probably irrelevant.
> >
> >> > ), leaving a partially downloaded working copy:
> >> >
> >> > svn: E155009: Failed to run the WC DB work queue associated with
> >> > 'C:\Project_Files\Proj_Dev\Prog_svn_1.9_cmdline', work item 2160
> >> > (file-install 234
> >> >
> >> >
> Logical_Architecture/APP/App_PL_Verification/XXX/XXX_XXXs/XXX_7_APPDB_PROJ_PROG--0021_BE/APPDB_PROJ_PROG--0021.SUP/APPDB_PROJ_PROG--0021/APPDB_PROJ_PROG--0021/APPDB_PROJ_PROG--0021.SUP/XXX_APPDB_PROJ_PROG--002101.CRC
> >> > 1 0 1 1)
> >> > svn: E720123: The filename, directory name, or volume label syntax is
> >> > incorrect.
> >>
> >> The full path of the file would be 281 characters long, which is more
> >> than Windows' limit of 260. But Subversion goes to a lot of trouble to
> >> avoid the path limit ... I wonder what's going on here.
>
> It seems Windows specific implementation of
> svn_stream__install_stream() doesn't prepend path with "\\?\" for long
> names. Replacing svn_utf__win32_utf8_to_utf16() call with
> svn_io__utf8_to_unicode_longpath() should fix the problem, but I
> didn't try yet.
>
> Bert, is it intentional behavior or just small typo?
>
>
> -- 
> Ivan Zhakov

Svn rename doesn't copy custom properties

[Dan Ellis]

Hi,

We use some custom properties for tracking (e.g. my_prop:trace_to
req_12345) and have noticed that svn renames do not copy these across the
delete/add operation.  I've tried searching the archives to see if I could
find a rationale as to why, but haven't come across any.

I'm guessing this could be legitimately viewed as a feature or a problem,
depending on your point of view.

Is there a viable option to try and get svn renames to copy these over?

Thanks,
Dan

Re: Svn rename doesn't copy custom properties

[Daniel Shahaf]

Dan Ellis wrote on Wed, Apr 29, 2015 at 15:43:00 -0700:
> Hi,
> 
> We use some custom properties for tracking (e.g. my_prop:trace_to
> req_12345) and have noticed that svn renames do not copy these across the
> delete/add operation.  I've tried searching the archives to see if I could
> find a rationale as to why, but haven't come across any.
> 

'svn cp' and 'svn mv' do copy properties across.  How exactly are you
invoking those operations?  Are you perhaps using a wrapper that does
'svn rm && svn add'?  Can you show us a transcript?

On my system I see it working:

[[[
% svn pl -v baz
Properties on 'baz':
  k
v
% svn mv baz foo 
A foo
D baz
% svn ci -q -mm 
% svn pl -v foo
Properties on 'foo':
  k
v
]]]

> I'm guessing this could be legitimately viewed as a feature or a problem,
> depending on your point of view.
> 
> Is there a viable option to try and get svn renames to copy these over?
> 

Including properties in mv/cp is the default behaviour and cannot be
turned off (except by doing 'svn propdel' after the cp/mv and before
committing).

What version of the client are you using?

> Thanks,
> Dan

Re: Dealing with very old repo format (version 1)

[Daniel Shahaf]

[snipping the part about db/revs, this is only about db/revprops]

Philip Martin wrote on Wed, Apr 29, 2015 at 18:53:04 +0100:
> Andrew Reedick  writes:
> 
> > Bad News:  However, it seems that I have bigger issues:
> > * Dumped revision 109662.
> > svnadmin: E720002: Can't open file
> > 'devel_copy\db\revprops\109663': The system cannot find the file
> > specified.
> 
> You can replace a missing revprop file with a file containing a single
> line:
> 
> END
> 

The file must be exactly four bytes long and must end with an LF
character (0x0A).  In particular, just using 'echo END > foo' will work
*on unix* but not on windows.

Daniel

Re: Dealing with very old repo format (version 1)

[Daniel Shahaf]

For future reference: if svnadmin is version 1.9 or newer, it should
have the 'info' subcommand, which will display both the repository
format and the FS format.

http://subversion.apache.org/docs/release-notes/1.9#svnadmin-info

Daniel
(1.9 hasn't been released yet, as of this writing)

Andrew Reedick wrote on Tue, Apr 28, 2015 at 21:03:12 +:
> Does anyone have any tips on how to upgrade a very old repo?  The db/format 
> lists "1".  A 1.8 svn client cannot hotcopy, dump or "svnadmin upgrade" such 
> an old repo, all of which fail with "svnadmin: E720002: Can't open file 
> 'devel\db\current': The system cannot find the file specified."
> 
> Do I need find a really old svn client (1.3?) and upgrade?  Do I need to 
> manually create the db/current file?
> 
> 
> Supposedly , a format of "1" is from pre-svn 1.0.  
> https://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_repos/repos.h
>  -> "Formats 0, 1 and 2 were pre-1.0."  
> 
> 
> 

Re: Svn rename doesn't copy custom properties

[Dan Ellis]

On Wed, Apr 29, 2015 at 4:01 PM, Daniel Shahaf 
wrote:

> Dan Ellis wrote on Wed, Apr 29, 2015 at 15:43:00 -0700:
> > Hi,
> >
> > We use some custom properties for tracking (e.g. my_prop:trace_to
> > req_12345) and have noticed that svn renames do not copy these across the
> > delete/add operation.  I've tried searching the archives to see if I
> could
> > find a rationale as to why, but haven't come across any.
> >
>
> 'svn cp' and 'svn mv' do copy properties across.  How exactly are you
> invoking those operations?  Are you perhaps using a wrapper that does
> 'svn rm && svn add'?  Can you show us a transcript?
>
> On my system I see it working:
>
> [[[
> % svn pl -v baz
> Properties on 'baz':
>   k
> v
> % svn mv baz foo
> A foo
> D baz
> % svn ci -q -mm
> % svn pl -v foo
> Properties on 'foo':
>   k
> v
> ]]]
>
> > I'm guessing this could be legitimately viewed as a feature or a problem,
> > depending on your point of view.
> >
> > Is there a viable option to try and get svn renames to copy these over?
> >
>
> Including properties in mv/cp is the default behaviour and cannot be
> turned off (except by doing 'svn propdel' after the cp/mv and before
> committing).
>
> What version of the client are you using?
>
> > Thanks,
> > Dan
>


This is specific to the rename operation (svn 1.8.9, win7):

c:\Project_files\sandbox>svn pl -v .txt
Properties on '.txt':
  myprop:trace_to
req12345

c:\Project_files\sandbox>svn rename .txt .txt
A .txt
D .txt

c:\Project_files\sandbox>svn pl -v .txt

c:\Project_files\sandbox>

Re: Svn rename doesn't copy custom properties

[jblist]

> On Apr 29, 2015, at 4:13 PM, Dan Ellis  wrote:
> 
> This is specific to the rename operation (svn 1.8.9, win7):
> 
> c:\Project_files\sandbox>svn pl -v .txt
> Properties on '.txt':
>   myprop:trace_to
> req12345
> 
> c:\Project_files\sandbox>svn rename .txt .txt
> A .txt
> D .txt
> 
> c:\Project_files\sandbox>svn pl -v .txt
> 
> c:\Project_files\sandbox>


On OS X, svn 1.7.19, I see this after the rename:

$ svn pl file1 
Properties on 'file1':
  myprop
$ svn rename file1 file2
A file2
D file1
$ svn status
D   file1
A  +file2
$ svn pl file2
Properties on 'file2':
  myprop

Re: Svn rename doesn't copy custom properties

[Dan Ellis]

OK, so it gets stranger...

I admit I changed the property names a bit to simplify them.  When I ran
the simplified names, it does work.

Here's the exact example that does not work:

c:\Project_files\sandbox_v2>svn pl -v A.txt
Properties on 'A.txt':
  pebls:plcm
Test@4575
  pebls:sha1
8cd8818d6b4f5edcb8b6e25cdf471af62bca403c

c:\Project_files\sandbox_v2>svn rename A.txt AA.txt
A AA.txt
D A.txt

c:\Project_files\sandbox_v2>svn pl -v AA.txt

c:\Project_files\sandbox_v2>svn pl -v REN.txt

On Wed, Apr 29, 2015 at 4:13 PM, Dan Ellis  wrote:

>
> On Wed, Apr 29, 2015 at 4:01 PM, Daniel Shahaf 
> wrote:
>
>> Dan Ellis wrote on Wed, Apr 29, 2015 at 15:43:00 -0700:
>> > Hi,
>> >
>> > We use some custom properties for tracking (e.g. my_prop:trace_to
>> > req_12345) and have noticed that svn renames do not copy these across
>> the
>> > delete/add operation.  I've tried searching the archives to see if I
>> could
>> > find a rationale as to why, but haven't come across any.
>> >
>>
>> 'svn cp' and 'svn mv' do copy properties across.  How exactly are you
>> invoking those operations?  Are you perhaps using a wrapper that does
>> 'svn rm && svn add'?  Can you show us a transcript?
>>
>> On my system I see it working:
>>
>> [[[
>> % svn pl -v baz
>> Properties on 'baz':
>>   k
>> v
>> % svn mv baz foo
>> A foo
>> D baz
>> % svn ci -q -mm
>> % svn pl -v foo
>> Properties on 'foo':
>>   k
>> v
>> ]]]
>>
>> > I'm guessing this could be legitimately viewed as a feature or a
>> problem,
>> > depending on your point of view.
>> >
>> > Is there a viable option to try and get svn renames to copy these over?
>> >
>>
>> Including properties in mv/cp is the default behaviour and cannot be
>> turned off (except by doing 'svn propdel' after the cp/mv and before
>> committing).
>>
>> What version of the client are you using?
>>
>> > Thanks,
>> > Dan
>>
>
>
> This is specific to the rename operation (svn 1.8.9, win7):
>
> c:\Project_files\sandbox>svn pl -v .txt
> Properties on '.txt':
>   myprop:trace_to
> req12345
>
> c:\Project_files\sandbox>svn rename .txt .txt
> A .txt
> D .txt
>
> c:\Project_files\sandbox>svn pl -v .txt
>
> c:\Project_files\sandbox>
>