Re: A subersion build environment on CentOS 7.2

[Stefan Sperling]

On Sun, Aug 21, 2016 at 06:02:31PM -0500, horst.schl...@gmx.de wrote:
> Yes, this is why I was asking about the errors the makefile is giving. In
> the building part of the guide it says
> 
> To start development from scratch, simply download Makefile.svn, create a
> symlink called `GNUmakefile` that points to the file, and invoke `make`.
> http://subversion.apache.org/docs/community-guide/building.html

Ah, now I understand how you ended up in this situation.

This section was added to building.html 6 years ago by a Gsoc student who
also used my Makefile, and where the Makefile worked without problems.
http://svn.apache.org/viewvc?view=revision&revision=1040482
I approved this change at the time. But today I've changed my mind.

I believe it's better if new contributors start out with the plain 'configure;
make; make install;' procedure and system dependencies. Because this should
generally "just work" and make problems with dependencies less liekly.
And more people are able to provide help and guidance in case of problems.

I'm sorry that our website mislead you. I'll remove the misleading section ASAP.

Re: A subersion build environment on CentOS 7.2

[Nico Kadel-Garcia]

On Sun, Aug 21, 2016 at 3:24 PM, Adam Jensen  wrote:
> On 08/21/2016 07:00 AM, Stefan Sperling wrote:
> [snip]
>> If your goal is setting up a production environment, I would recommend
>> using binary packages system from http://subversion.apache.org/packages.html
>
> My CentOS (el7.x86_64) machine only has subversion 1.7.14 available:
>
> % yum info subversion
>
> Installed Packages
> Name: subversion
> Arch: x86_64
> Version : 1.7.14
> Release : 10.el7
> Size: 4.6 M
> Repo: installed
> From repo   : anaconda
>
> There doesn't seem to be an option for subversion 1.9.4. A brief scan of
> the offerings at:
>
> http://subversion.apache.org/packages.html#redhat

You might start from my old work at
https://github.com/nkadel/subversion-1.8.x-srpm

I used to publish updates for Repoforge, but that's been pretty moribund.

> isn't encouraging. CollabNet and WANdisco both require registration,
> SummerSoft is either out-of-date or incomprehensible, and:
>
> https://wiki.centos.org/AdditionalResources/Repositories/RPMForge
>
> says: "RPMForge/RepoForge is a dead project. It is not maintained. DO
> NOT USE."
>
> Given the situation, building from source might be a reasonable
> approach. I downloaded the 1.9.4 source a few days ago, to have a look
> at it, and I will probably need to build it on CentOS-7; so, any
> pointers Horst or others might post on this process will be appreciated
> (by me, at least).

Re: "Offset too large" error when packing repository in FSFS 7 format

[Radek Krotil]

Thanks Stefan and Daniel for your effort in analyzing this. Unfortunately,
I missed your replies as I expected that they will address my mailbox as
well. So let me restart the thread now..



Recently I hit this problem on another production repository from one of my
customers. My test machine where I work with the repositories has 512 GB
SSD drive, so I need to keep the repositories as small as possible, so I
try to migrate them to the latest format with deltification enabled and
pack them. Therefore I usually take the following steps:
1) Unpack the zipped repository the customer sends me under customer name
folder

2) Rename it to repo-x, where x is the format of the repository
3) Dump the repository

4) Create new repository named repo-1.9

5) Load the dump to the new repository

6) Pack the repo-1.9 repository

7) Configure Polarion ALM to use the repository

8) Start Polarion – only at this point is the repository first used by
Apache and svnserve



There is a svnserve process serving all repositories under
/opt/repositories/, where the customer folders are stored. However, the
repositories are not accessed until I’m fully done with the migration. The
test server is dedicated for my use only, so I’m confident there are no
other users reading/writing the repositories.



On this particular repository, I ran the dump/load cycle twice and in both
cases it resulted in the svnadmin pack command failure. I’ll re-try to do
dump/load on the other repository  as well later. Svnadmin verify confirmed
that the current repository is not corrupted. The repository contains
334243 revisions total. As suggested by Stefan, I did the grep on the
problematic repo file. The rev 203908 is about 231 MB big. This confirms my
suspicion that the problem is related to big revision data.



[root@babybear svn]# svnadmin pack repo-1.9/

Packing revisions in shard 203...svnadmin: E160056: Offset 232966338 too
large in revision 203908

[root@babybear 203]# grep -oba L2P-INDEX 203908

231917762:L2P-INDEX



I tried to restart Apache, svnserve, even the whole box. The problem still
persists. Unlike the other occurrences reported by other users, e.g. in
https://issues.apache.org/jira/browse/SVN-4588, this does not seem to be
related to invalid server cache, because I’m only using svnadmin command to
work with the repository.



Looking forward to further suggestions.



Best regards,

Radek Krotil



On 2016-06-04 18:57 (+0200), Daniel Shahaf  wrote:

> Stefan Fuhrmann wrote on Sat, Jun 04, 2016 at 08:04:42 -:

> > On 2016-06-03 09:36 (+0200), Radek Krotil 
wrote:

> > > Hello.

> > >

> > > Today, I encountered a problem when trying to pack a repository after

> > > migrating it to the FSFS 7 format by performing full dump / load
sequence.

> >

> > I assume you ran 'svnadmin load' onto a repository

> > that was not accessible to the server at that time,

> > so no remote user could accicentally write to it?

>

> Why would that matter?  What could happen if somebody makes a commit or

> a propedit in parallel to an 'svnadmin load'?  A concurrent commit will

> cause mergeinfo in later revisions to have to have off-by-one errors,

> but shouldn't cause FS corruption.

>

> > > Shortly, I get the following error

> > > “Packing revisions in shard 5...svnadmin: E160056: Offset 391658998
too

> > > large in revision 5102”

> >

> > This is basically an "invalid access" error message.

> > Typical causes include repository corruption and

> > admins tinkering with the repository without informing

> > the server process. A maybe similar issue:

> >

> > https://issues.apache.org/jira/browse/SVN-4588

> >

> > In your case, however, the corruption is probably in

> > the repository itself. Please run 'svnadmin verify' on it.

> >

> > > I was not able to understand from the documentation, what settings in

> > > fsfs.conf should be modified to workaround this problem. Neither
search in

> > > the Internet brought any light into this. Is it even possible?

> >

> > This is most definitely not a configuration issue like

> > "your data is too large". Maybe, we should prefix the

> > error message with "invalid access" to prevent

> > confusion.

>

> How about being even more specific:

>

> svnadmin: E1600NN: failed to locate representation of %s at revision
%ld, offset %lld

>

> where %s identifies the origin of the offset value or the object that

> was expected to be located at that offset.

>

> ?

>

> Cheers,

>

> Daniel

>

Migrating old format repositories (format 4), running into bad line endings

[Eric Johnson]

So I wrote myself a tool to go through all commit comments, and verify that
no "CRs" appear in the svn:log entry. If I find them, I rewrite the log
entry. That way, I can clean up existing repositories.

However, I ran into a surprise. CR characters have also snuck into the
svn:ignore property, which is *not* a revprop, and cannot be fixed by any
of the tools available.

I've seen that other people recommend editing the dump file directly, but
that makes me quite nervous, especially on an 8MB dump file. Hoping there's
a better way.

Eric.

Re: Migrating old format repositories (format 4), running into bad line endings

[Daniel Shahaf]

Eric Johnson wrote on Mon, Aug 22, 2016 at 11:11:17 -0700:
> I've seen that other people recommend editing the dump file directly, but
> that makes me quite nervous, especially on an 8MB dump file. Hoping there's
> a better way.

You could just pass --bypass-prop-validation to 'svnadmin load'.  That
will permit the load to go through, *keeping* the CRs in place.

If you care to patch the source, I think this is what you want:

[[[
Index: subversion/libsvn_repos/load-fs-vtable.c
===
--- subversion/libsvn_repos/load-fs-vtable.c(revision 1757264)
+++ subversion/libsvn_repos/load-fs-vtable.c(working copy)
@@ -871,6 +871,15 @@ set_node_property(void *baton,
 }
 }
 
+  if (svn_prop_needs_translation(name))
+if (strchr(value->data, '\r'))
+  {
+apr_array_header_t *parts = svn_cstring_split(value->data, "\r", FALSE,
+  nb->pool);
+value = svn_string_create(svn_cstring_join(parts, "", nb->pool),
+  nb->pool);
+  }
+
   return change_node_prop(rb->txn_root, nb->path, name, value,
   pb->validate_props, nb->pool);
 }
]]]

I have not tested this patch, so while I *think* it's correct, it might
be buggy in some way.  If you use it, then you should verify manually
that it effected the correct transformation and didn't have unexpected
side effects.

The patch is against 1.9.x.

I haven't used the svn_*translate APIs pointed to by
svn_prop_needs_translation()'s docstring because those transcode, which
wouldn't be appropriate in this case (we can't assume the dumpfile data
is in the system encoding of the system the 'load' is done on).

And again: the patch is completely untested; it is *probably* correct,
but don't trust it to the same degree you trust our packaged stable
releases.  Until feedback on list blesses it as stable, validate its
effect manually.

Sorry for the caveats,

HTH,

Daniel

P.S. For anyone running into this in the archives: this patch only
affects node properties; it doesn't touch revision properties.

Re: Migrating old format repositories (format 4), running into bad line endings

[Mark Phippard]

On Mon, Aug 22, 2016 at 2:11 PM, Eric Johnson  wrote:

> So I wrote myself a tool to go through all commit comments, and verify
> that no "CRs" appear in the svn:log entry. If I find them, I rewrite the
> log entry. That way, I can clean up existing repositories.
>
> However, I ran into a surprise. CR characters have also snuck into the
> svn:ignore property, which is *not* a revprop, and cannot be fixed by any
> of the tools available.
>
> I've seen that other people recommend editing the dump file directly, but
> that makes me quite nervous, especially on an 8MB dump file. Hoping there's
> a better way.
>

Have you tried creating a new repository with the format you want and then
use svnsync to sync the data to it?  I seem to recall that svnsync will fix
the line endings as it syncs the log entries.  I could be wrong though
because I see that svnsync has a --source-prop-encoding option that we
added for fixing non UTF8 log messages.  So it is possible I am confusing
with that option.  I would try it though since it is a a simple method to
use.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: Migrating old format repositories (format 4), running into bad line endings

[Daniel Shahaf]

Mark Phippard wrote on Mon, Aug 22, 2016 at 17:06:05 -0400:
> On Mon, Aug 22, 2016 at 2:11 PM, Eric Johnson  wrote:
> 
> > So I wrote myself a tool to go through all commit comments, and verify
> > that no "CRs" appear in the svn:log entry. If I find them, I rewrite the
> > log entry. That way, I can clean up existing repositories.
> >
> > However, I ran into a surprise. CR characters have also snuck into the
> > svn:ignore property, which is *not* a revprop, and cannot be fixed by any
> > of the tools available.
> >
> > I've seen that other people recommend editing the dump file directly, but
> > that makes me quite nervous, especially on an 8MB dump file. Hoping there's
> > a better way.
> >
> 
> Have you tried creating a new repository with the format you want and then
> use svnsync to sync the data to it?  I seem to recall that svnsync will fix
> the line endings as it syncs the log entries.  I could be wrong though
> because I see that svnsync has a --source-prop-encoding option that we
> added for fixing non UTF8 log messages.  So it is possible I am confusing
> with that option.

svnsync normalizes line endings to LF unconditionally.
(sync.c:change_dir_prop() calls normalize_string())

Re: Migrating old format repositories (format 4), running into bad line endings

[Eric Johnson]

Hi Mark,

On 8/22/16 2:06 PM, Mark Phippard wrote:
On Mon, Aug 22, 2016 at 2:11 PM, Eric Johnson > wrote:


So I wrote myself a tool to go through all commit comments, and
verify that no "CRs" appear in the svn:log entry. If I find them,
I rewrite the log entry. That way, I can clean up existing
repositories.

However, I ran into a surprise. CR characters have also snuck into
the svn:ignore property, which is /not/ a revprop, and cannot be
fixed by any of the tools available.

I've seen that other people recommend editing the dump file
directly, but that makes me quite nervous, especially on an 8MB
dump file. Hoping there's a better way.


Have you tried creating a new repository with the format you want and 
then use svnsync to sync the data to it?  I seem to recall that 
svnsync will fix the line endings as it syncs the log entries.  I 
could be wrong though because I see that svnsync has 
a --source-prop-encoding option that we added for fixing non UTF8 log 
messages.  So it is possible I am confusing with that option.  I would 
try it though since it is a a simple method to use.


I did just try this - svnsync does fix up the line ending problem. Even 
reports a nice summary at the end of what it did. So that's promising!


So it seems like a sync-based equivalent to dump / load, in order to 
migrate from one repo format to another looks something like this:


 * svnadmin create ... // create target repo
 * // enable revprop changes
 * svnsync init ...
 * svnsync sync
 * svnadmin pack
 * svnadmin verify
 * // remove revprop change script
 * // remove all the props on revision 0
 * // copy over all the hooks from the original repo
 * svnadmin setuuid ... // set the uuid of the new repository to the
   same as the old.
 * // move original repo out of the way, move new copy into position.

Is there some utility somewhere that does all of the above? I can go 
change my script to use the above approach instead of dump/load, but it 
is annoying


Is there any particular reason that the "dump/load" process doesn't fix 
the line-endings? Shouldn't it? Why have the --bypass-prop-validation 
option? Why not just fix the line endings? As it is now, the dump-load 
process is effectively broken, because I have one of two seemingly poor 
choices: I can either "bypass validation" (that sounds bad), or I simply 
keep the old format.


Eric.


--
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: Migrating old format repositories (format 4), running into bad line endings

[Mark Phippard]

On Mon, Aug 22, 2016 at 5:29 PM, Eric Johnson  wrote:

> Hi Mark,
>
> On 8/22/16 2:06 PM, Mark Phippard wrote:
>
>> On Mon, Aug 22, 2016 at 2:11 PM, Eric Johnson > e...@tibco.com>> wrote:
>>
>> So I wrote myself a tool to go through all commit comments, and
>> verify that no "CRs" appear in the svn:log entry. If I find them,
>> I rewrite the log entry. That way, I can clean up existing
>> repositories.
>>
>> However, I ran into a surprise. CR characters have also snuck into
>> the svn:ignore property, which is /not/ a revprop, and cannot be
>> fixed by any of the tools available.
>>
>> I've seen that other people recommend editing the dump file
>> directly, but that makes me quite nervous, especially on an 8MB
>> dump file. Hoping there's a better way.
>>
>>
>> Have you tried creating a new repository with the format you want and
>> then use svnsync to sync the data to it?  I seem to recall that svnsync
>> will fix the line endings as it syncs the log entries.  I could be wrong
>> though because I see that svnsync has a --source-prop-encoding option that
>> we added for fixing non UTF8 log messages.  So it is possible I am
>> confusing with that option.  I would try it though since it is a a simple
>> method to use.
>>
>
> I did just try this - svnsync does fix up the line ending problem. Even
> reports a nice summary at the end of what it did. So that's promising!
>
> So it seems like a sync-based equivalent to dump / load, in order to
> migrate from one repo format to another looks something like this:
>
>  * svnadmin create ... // create target repo
>  * // enable revprop changes
>  * svnsync init ...
>  * svnsync sync
>  * svnadmin pack
>  * svnadmin verify
>  * // remove revprop change script
>  * // remove all the props on revision 0
>  * // copy over all the hooks from the original repo
>  * svnadmin setuuid ... // set the uuid of the new repository to the
>same as the old.
>  * // move original repo out of the way, move new copy into position.
>
> Is there some utility somewhere that does all of the above? I can go
> change my script to use the above approach instead of dump/load, but it is
> annoying
>
> Is there any particular reason that the "dump/load" process doesn't fix
> the line-endings? Shouldn't it? Why have the --bypass-prop-validation
> option? Why not just fix the line endings? As it is now, the dump-load
> process is effectively broken, because I have one of two seemingly poor
> choices: I can either "bypass validation" (that sounds bad), or I simply
> keep the old format.
>


You can always just script it, but note that most of these steps apply when
using dump/load too.

 * svnadmin create ... // create target repo
 * svnadmin dump oldrepos > dumpfile
 * svnadmin load newrepos < dumpfile
 * svnadmin pack newrepos
 * svnadmin verify newrepos
 * // copy over all the hooks from the original repo
 * // move original repo out of the way, move new copy into position.


-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: Migrating old format repositories (format 4), running into bad line endings

[Daniel Shahaf]

Eric Johnson wrote on Mon, Aug 22, 2016 at 14:29:47 -0700:
>  * // move original repo out of the way, move new copy into position.

At the comma you should restart the server to flush caches:


http://mail-archives.apache.org/mod_mbox/subversion-users/201606.mbox/%3c20160609125706.GA4020@tarsus.local2%3e

> Is there any particular reason that the "dump/load" process doesn't fix the
> line-endings? Shouldn't it?

'load' could transform CRs, yes, but that might need to be optional to
keep the "verbatim round-trip" possibility.

> Why have the --bypass-prop-validation option?
> Why not just fix the line endings? As it is now, the dump-load process is
> effectively broken, because I have one of two seemingly poor choices: I can
> either "bypass validation" (that sounds bad), or I simply keep the old
> format.

"Bypass" just means the existing values in the dumpfile will be
used verbatim.  Using this option makes the loaded repository be in the
same situation of the original 1.4 repository: a high-level invariant
("svn:* uses LF") does not hold, but the lower-level data format
invariants do hold.  In short, if you use this option you're postponing
the problem but not making your situation any worse.

Cheers,

Daniel

P.S. Feel free to file an issue about making 'load' transform CR in
svn:* nodeprops and revprops pointing to this thread.

Re: Migrating old format repositories (format 4), running into bad line endings

[Eric Johnson]

Thanks for the reply.

On 8/22/16 5:25 PM, Daniel Shahaf wrote:

Eric Johnson wrote on Mon, Aug 22, 2016 at 14:29:47 -0700:

  * // move original repo out of the way, move new copy into position.

At the comma you should restart the server to flush caches:

 
http://mail-archives.apache.org/mod_mbox/subversion-users/201606.mbox/%3c20160609125706.GA4020@tarsus.local2%3e


Yes, I was simplifying a little in my pseudo-code, because after I stop 
the server, I want to do another sync, in case additional changes come 
in during the verify and pack steps.



Is there any particular reason that the "dump/load" process doesn't fix the
line-endings? Shouldn't it?

'load' could transform CRs, yes, but that might need to be optional to
keep the "verbatim round-trip" possibility.


Why have the --bypass-prop-validation option?
Why not just fix the line endings? As it is now, the dump-load process is
effectively broken, because I have one of two seemingly poor choices: I can
either "bypass validation" (that sounds bad), or I simply keep the old
format.

"Bypass" just means the existing values in the dumpfile will be
used verbatim.  Using this option makes the loaded repository be in the
same situation of the original 1.4 repository: a high-level invariant
("svn:* uses LF") does not hold, but the lower-level data format
invariants do hold.  In short, if you use this option you're postponing
the problem but not making your situation any worse.

Cheers,

Daniel

P.S. Feel free to file an issue about making 'load' transform CR in
svn:* nodeprops and revprops pointing to this thread.

I've filed https://issues.apache.org/jira/browse/SVN-4650

Thanks so much.
Eric.