Re: EOL problems in cvs2svn converted repository
On Thu, 11 Jan 2018 09:51:54 -0600, Ryan Schmidt wrote: >> I found a section in the conf file named [auto-props] but it seems >> only to be applicable for commit and import, not for checkout AFAICT. >> And all of that section is commented out by default. > >If you want Subversion to perform EOL translation on a file, you must set >the svn:eol-style property on that file to the desired value. >There is no other way. Really? I imagine that a lot of people have been bit by this and someone must have come up with a way to fix the problem without having to manually go over all of the files in a repository matching the file type (by extension) and set the property. Is there no script or such available? > >If you want all files with a particular name or extension to get this >property when they are added to a working copy, set that up in the >auto-props section of your Subversion client's file. >This will apply to that client only and would need to be repeated for >all clients / all users. I have put a Readme.txt file into the repo that describes the way to set up the svn client side conf file. But it will help only for new files... > >If using Subversion client 1.8 or later, use a repository-dictated >auto-props configuration: > >https://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config > >Auto-props only cause the property to be added to the file when the file >is added to the working copy. A user can subvert auto-props, and a user >can also delete the property after it was added, before or after the file >was committed. > >If you want to enforce that this property is used on files that need it, >and/or prevent users from removing it, write a pre-commit hook that does that. But this only affects files added to the repo, right? I need something that can apply to *all* existing files of the affected types. >If you want all of the files you're importing from CVS that match the criteria >to have the property, adjust your cvs-to-svn conversion process to arrange for >that to be the case. > This will be a hurdle since the conversion is already done and the repository is in use... But *how* would one do this if a new conversion has to be done? I am not fluent in Python, so any such coding is beyond me... There is one repository holding a number of projects prepared for use by an external contractor, which has so far not been used for development. So in theory I can repeat the conversion cvs2svn for this with some cvs2svn settings tweak that will set the svn:eol-style conversion correctly. But I need to know how this can be done in that case. Is it possible to use dump/load? Is it possible to somehow edit an svn dump file (with a script) to look for all the target files and add the eol-style property setting for each file so it will be effected in the svn load? When I look at a dumnp file load it looks like what it does is automatically performing all the commit operations that was done in the source repository Or maybe simpler: - Create dump file of the repo. Create a new repo and set its automatic property settings as discussed above. Then load the dump file. Will this make the new repository contain the automatically set eol properties? I.e. will the load be affected by the global repository automatic property settings? Note that our repos are each containing a number of "projects" in the top level of the repo and all of these have the structure of a svn repo with trunk, branches and tags subdirectories. We use VisualSVN with svn version 1.9.7 on the server -- Bo Berglund Developer in Sweden
Re: EOL problems in cvs2svn converted repository
On Fri, Feb 9, 2018 at 11:44 AM, Bo Berglund wrote: > On Thu, 11 Jan 2018 09:51:54 -0600, Ryan Schmidt > wrote: > >>> I found a section in the conf file named [auto-props] but it seems >>> only to be applicable for commit and import, not for checkout AFAICT. >>> And all of that section is commented out by default. >> >>If you want Subversion to perform EOL translation on a file, you must set >>the svn:eol-style property on that file to the desired value. >>There is no other way. > > Really? I imagine that a lot of people have been bit by this and > someone must have come up with a way to fix the problem without having > to manually go over all of the files in a repository matching the file > type (by extension) and set the property. > Is there no script or such available? > >> >>If you want all files with a particular name or extension to get this >>property when they are added to a working copy, set that up in the >>auto-props section of your Subversion client's file. >>This will apply to that client only and would need to be repeated for >>all clients / all users. > > I have put a Readme.txt file into the repo that describes the way to > set up the svn client side conf file. > But it will help only for new files... > >> >>If using Subversion client 1.8 or later, use a repository-dictated >>auto-props configuration: >> >>https://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config >> >>Auto-props only cause the property to be added to the file when the file >>is added to the working copy. A user can subvert auto-props, and a user >>can also delete the property after it was added, before or after the file >>was committed. >> >>If you want to enforce that this property is used on files that need it, >>and/or prevent users from removing it, write a pre-commit hook that does that. > > But this only affects files added to the repo, right? > I need something that can apply to *all* existing files of the > affected types. > >>If you want all of the files you're importing from CVS that match the criteria >>to have the property, adjust your cvs-to-svn conversion process to arrange for >>that to be the case. >> > > This will be a hurdle since the conversion is already done and the > repository is in use... > But *how* would one do this if a new conversion has to be done? > I am not fluent in Python, so any such coding is beyond me... > > There is one repository holding a number of projects prepared for use > by an external contractor, which has so far not been used for > development. So in theory I can repeat the conversion cvs2svn for this > with some cvs2svn settings tweak that will set the svn:eol-style > conversion correctly. > But I need to know how this can be done in that case. > > Is it possible to use dump/load? > > Is it possible to somehow edit an svn dump file (with a script) to > look for all the target files and add the eol-style property setting > for each file so it will be effected in the svn load? > When I look at a dumnp file load it looks like what it does is > automatically performing all the commit operations that was done in > the source repository > > Or maybe simpler: > - > Create dump file of the repo. > Create a new repo and set its automatic property settings as discussed > above. > Then load the dump file. > Will this make the new repository contain the automatically set eol > properties? I.e. will the load be affected by the global repository > automatic property settings? > > Note that our repos are each containing a number of "projects" in the > top level of the repo and all of these have the structure of a svn > repo with trunk, branches and tags subdirectories. > > We use VisualSVN with svn version 1.9.7 on the server Standard dump/load cannot do this. However, you might be able to do it using this tool: https://github.com/jwiegley/svndumptool (it can manipulate dump files by setting properties and stuff like that, and performing various eol fixes) I'm wondering how robust this is though ... make sure to doublecheck the result (for instance by checking out / exporting some of these files from different points in history). The reason why I'm wondering is that, internally (in the repository) a text file is stored in "normalized form" (which is LF-terminated) if you have the svn:eol-style=native property set. But if it's not set, svn doesn't alter the incoming file in any way (it could be binary for all svn knows), so if that file was committed from Windows, it will probably be in CRLF format, so that's what will be stored in the repository. So changing this in historic revisions (in a dump file) is not just a question of adding the property, it also involves normalizing the corresponding file contents in the dump (and the dump format contains checksums and offsets, so manipulating actual contents in not easy). -- Johan
Re: EOL problems in cvs2svn converted repository
Johan Corveleyn wrote on Fri, 09 Feb 2018 13:40 +0100: > So changing this in historic revisions (in a dump file) is not just a > question of adding the property, it also involves normalizing the > corresponding file contents in the dump (and the dump format contains > checksums and offsets, so manipulating actual contents in not easy). Why wouldn't it be easy? If the dumpstream was generated without --deltas, the records are not dependent on each other, so writing some sort of stdin-to-stdout filter that renormalizes newlines of some file records would be straightforward. It wouldn't be doable with sed, for the reasons you give, but it'd be doable using the API. If the dumpstream was generated with --deltas, it would be easier to load the dumpstream into a repository and then to use svnsync to renormalize the newlines. (Conceptually, 'svnsync sync' should allow the source to be a dumpstream, reusing the dumpstream-to-editor backend of 'svnrdump load', but that hasn't been implemented.) Cheers, Daniel
Re: EOL problems in cvs2svn converted repository
On Fri, 9 Feb 2018 13:40:43 +0100, Johan Corveleyn wrote: >> We use VisualSVN with svn version 1.9.7 on the server > >Standard dump/load cannot do this. > >However, you might be able to do it using this tool: >https://github.com/jwiegley/svndumptool > >(it can manipulate dump files by setting properties and stuff like >that, and performing various eol fixes) > >I'm wondering how robust this is though ... make sure to doublecheck >the result (for instance by checking out / exporting some of these >files from different points in history). > >The reason why I'm wondering is that, internally (in the repository) a >text file is stored in "normalized form" (which is LF-terminated) if >you have the svn:eol-style=native property set. But if it's not set, >svn doesn't alter the incoming file in any way (it could be binary for >all svn knows), so if that file was committed from Windows, it will >probably be in CRLF format, so that's what will be stored in the >repository. So changing this in historic revisions (in a dump file) is >not just a question of adding the property, it also involves >normalizing the corresponding file contents in the dump (and the dump >format contains checksums and offsets, so manipulating actual contents >in not easy). OK, I think I will be OK with only setting the trunk version files correctly, if possible. After all this is what the consultants will use. And the development IDE:s seem to be able to cope fine with UNIX style source files (at least those I have tested with). Someone suggested that this would do it, but I am not sure: svn propset svn:eol-style 'native' -R * And he did not say how to do it, for example if I have to do that inside a working copy or if it is possible directly against the repository. In any case it seems like a dangerous proposition using * as file spec, since there are binary files also in the repo (icons, images, Windows executables, dll's etc). So maybe a script like this (executed at the top of a working copy): svn propset svn:eol-style 'native' -R *.txt svn propset svn:eol-style 'native' -R *.c svn propset svn:eol-style 'native' -R *.h svn propset svn:eol-style 'native' -R *.pas svn propset svn:eol-style 'native' -R *.dpr svn propset svn:eol-style 'native' -R *.ini svn propset svn:eol-style 'native' -R *.conf svn propset svn:eol-style 'native' -R *.py svn propset svn:eol-style 'native' -R *.cpp svn propset svn:eol-style 'native' -R *.lpr svn propset svn:eol-style 'native' -R *.lpi svn propset svn:eol-style 'native' -R *.bat svn propset svn:eol-style 'native' -R *.iss svn propset svn:eol-style 'native' -R *.xml svn propset svn:eol-style 'native' -R *.gld svn propset svn:eol-style 'native' -R *.inc svn propset svn:eol-style 'native' -R *.dsp svn propset svn:eol-style 'native' -R *.dsw And then a svn commit Would this work? What would happen if a file already has the requested property? -- Bo Berglund Developer in Sweden
Re: EOL problems in cvs2svn converted repository
On Feb 9, 2018, at 04:44, Bo Berglund wrote: > On Thu, 11 Jan 2018 09:51:54 -0600, Ryan Schmidt wrote: > >>> I found a section in the conf file named [auto-props] but it seems >>> only to be applicable for commit and import, not for checkout AFAICT. >>> And all of that section is commented out by default. >> >> If you want Subversion to perform EOL translation on a file, you must set >> the svn:eol-style property on that file to the desired value. >> There is no other way. > > Really? I imagine that a lot of people have been bit by this and > someone must have come up with a way to fix the problem without having > to manually go over all of the files in a repository matching the file > type (by extension) and set the property. > Is there no script or such available? What a lot of people were bit by was the way CVS handles eol translation. When applied to files that it should not be applied to, such as binary files, eol translation is destructive and irreversible. Subversion aims to be a better CVS, so Subversion improves on CVS by requiring you to be explicit about which files you want it to perform eol translation on, so that it never performs an eol translation you didn't request. >> If you want all files with a particular name or extension to get this >> property when they are added to a working copy, set that up in the >> auto-props section of your Subversion client's file. >> This will apply to that client only and would need to be repeated for >> all clients / all users. > > I have put a Readme.txt file into the repo that describes the way to > set up the svn client side conf file. > But it will help only for new files... Yes. >> If using Subversion client 1.8 or later, use a repository-dictated >> auto-props configuration: >> >> https://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config >> >> Auto-props only cause the property to be added to the file when the file >> is added to the working copy. A user can subvert auto-props, and a user >> can also delete the property after it was added, before or after the file >> was committed. >> >> If you want to enforce that this property is used on files that need it, >> and/or prevent users from removing it, write a pre-commit hook that does >> that. > > But this only affects files added to the repo, right? Yes. > I need something that can apply to *all* existing files of the > affected types. You need both. >> If you want all of the files you're importing from CVS that match the >> criteria >> to have the property, adjust your cvs-to-svn conversion process to arrange >> for >> that to be the case. >> > > This will be a hurdle since the conversion is already done and the > repository is in use... > But *how* would one do this if a new conversion has to be done? > I am not fluent in Python, so any such coding is beyond me... > > There is one repository holding a number of projects prepared for use > by an external contractor, which has so far not been used for > development. So in theory I can repeat the conversion cvs2svn for this > with some cvs2svn settings tweak that will set the svn:eol-style > conversion correctly. > But I need to know how this can be done in that case. > > Is it possible to use dump/load? > > Is it possible to somehow edit an svn dump file (with a script) to > look for all the target files and add the eol-style property setting > for each file so it will be effected in the svn load? > When I look at a dumnp file load it looks like what it does is > automatically performing all the commit operations that was done in > the source repository > > Or maybe simpler: > - > Create dump file of the repo. > Create a new repo and set its automatic property settings as discussed > above. > Then load the dump file. > Will this make the new repository contain the automatically set eol > properties? I.e. will the load be affected by the global repository > automatic property settings? > > Note that our repos are each containing a number of "projects" in the > top level of the repo and all of these have the structure of a svn > repo with trunk, branches and tags subdirectories. If you want to set the svn:eol-style property during the conversion, you'll have to read the documentation of the conversion tool or scripts you're using; I assume they would provide for this but I don't know. If you want to do it after the conversion and fix it in all revisions by editing repository history and forcing all users to throw away their old working copies and check out new ones, then you can "svnadmin dump" the repo, then use the third-party svndumptool program to set svn:eol-style where you want it, then "svnadmin load" the modified dumpfile. If you want to do it after the conversion by just setting the property starting now, without altering history, then check out a working copy, set the properties where you want them, then check
Re: EOL problems in cvs2svn converted repository
On Fri, Feb 9, 2018 at 8:04 PM, Bo Berglund wrote:
> On Fri, 9 Feb 2018 13:40:43 +0100, Johan Corveleyn
> wrote:
>
>>> We use VisualSVN with svn version 1.9.7 on the server
>>
>>Standard dump/load cannot do this.
>>
>>However, you might be able to do it using this tool:
>>https://github.com/jwiegley/svndumptool
>>
>>(it can manipulate dump files by setting properties and stuff like
>>that, and performing various eol fixes)
>>
>>I'm wondering how robust this is though ... make sure to doublecheck
>>the result (for instance by checking out / exporting some of these
>>files from different points in history).
>>
>>The reason why I'm wondering is that, internally (in the repository) a
>>text file is stored in "normalized form" (which is LF-terminated) if
>>you have the svn:eol-style=native property set. But if it's not set,
>>svn doesn't alter the incoming file in any way (it could be binary for
>>all svn knows), so if that file was committed from Windows, it will
>>probably be in CRLF format, so that's what will be stored in the
>>repository. So changing this in historic revisions (in a dump file) is
>>not just a question of adding the property, it also involves
>>normalizing the corresponding file contents in the dump (and the dump
>>format contains checksums and offsets, so manipulating actual contents
>>in not easy).
>
> OK, I think I will be OK with only setting the trunk version files
> correctly, if possible. After all this is what the consultants will
> use.
> And the development IDE:s seem to be able to cope fine with UNIX style
> source files (at least those I have tested with).
>
> Someone suggested that this would do it, but I am not sure:
>
> svn propset svn:eol-style 'native' -R *
>
> And he did not say how to do it, for example if I have to do that
> inside a working copy or if it is possible directly against the
> repository.
You'll need a working copy for that.
> In any case it seems like a dangerous proposition using * as file
> spec, since there are binary files also in the repo (icons, images,
> Windows executables, dll's etc).
>
> So maybe a script like this (executed at the top of a working copy):
>
> svn propset svn:eol-style 'native' -R *.txt
> svn propset svn:eol-style 'native' -R *.c
> svn propset svn:eol-style 'native' -R *.h
> svn propset svn:eol-style 'native' -R *.pas
> svn propset svn:eol-style 'native' -R *.dpr
> svn propset svn:eol-style 'native' -R *.ini
> svn propset svn:eol-style 'native' -R *.conf
> svn propset svn:eol-style 'native' -R *.py
> svn propset svn:eol-style 'native' -R *.cpp
> svn propset svn:eol-style 'native' -R *.lpr
> svn propset svn:eol-style 'native' -R *.lpi
> svn propset svn:eol-style 'native' -R *.bat
> svn propset svn:eol-style 'native' -R *.iss
> svn propset svn:eol-style 'native' -R *.xml
> svn propset svn:eol-style 'native' -R *.gld
> svn propset svn:eol-style 'native' -R *.inc
> svn propset svn:eol-style 'native' -R *.dsp
> svn propset svn:eol-style 'native' -R *.dsw
I don't think that will work. The *.txt will only be interpreted by
your shell, and it will not be passed to the recursive propset command
... so it'll only apply the propset to *.txt files in the current
directory. Perhaps you can whip something up in combination with the
find command:
find . -name '*.txt' -exec svn propset svn:eol-style native {} \;
or something like that (note: you don't need to put quotes around the
value, otherwise the value will actually become 'native', instead of
native).
> And then a svn commit
>
> Would this work?
> What would happen if a file already has the requested property?
I think that will work. Not sure what happens if the file already has
the property ... but it's easy to try it out, so just give it a try
:-).
ISTR there used to be a helper script to do this (to apply a certain
autoprops configuration to your working copy ...
svn-apply-autoprops.py or .pl or something), but I can't find it.
Maybe it was in the contrib folder which has been removed since (you
might be able to find a version of it googling).
--
Johan
Re: EOL problems in cvs2svn converted repository
On Feb 9, 2018, at 13:04, Bo Berglund wrote: > OK, I think I will be OK with only setting the trunk version files > correctly, if possible. After all this is what the consultants will > use. > And the development IDE:s seem to be able to cope fine with UNIX style > source files (at least those I have tested with). > > Someone suggested that this would do it, but I am not sure: > > svn propset svn:eol-style 'native' -R * > > And he did not say how to do it, for example if I have to do that > inside a working copy or if it is possible directly against the > repository. Like most nontrivial changes, you stage the changes in a working copy, examine the result and make sure everything looks ok, then you commit. > In any case it seems like a dangerous proposition using * as file > spec, since there are binary files also in the repo (icons, images, > Windows executables, dll's etc). Right, if you have binary files in the working copy, you don't want Subversion to corrupt them by performing eol translation on them, so you should not use "*"; you should be more selective about what types of files you apply the property to. > So maybe a script like this (executed at the top of a working copy): > > svn propset svn:eol-style 'native' -R *.txt > svn propset svn:eol-style 'native' -R *.c > svn propset svn:eol-style 'native' -R *.h > svn propset svn:eol-style 'native' -R *.pas > svn propset svn:eol-style 'native' -R *.dpr > svn propset svn:eol-style 'native' -R *.ini > svn propset svn:eol-style 'native' -R *.conf > svn propset svn:eol-style 'native' -R *.py > svn propset svn:eol-style 'native' -R *.cpp > svn propset svn:eol-style 'native' -R *.lpr > svn propset svn:eol-style 'native' -R *.lpi > svn propset svn:eol-style 'native' -R *.bat > svn propset svn:eol-style 'native' -R *.iss > svn propset svn:eol-style 'native' -R *.xml > svn propset svn:eol-style 'native' -R *.gld > svn propset svn:eol-style 'native' -R *.inc > svn propset svn:eol-style 'native' -R *.dsp > svn propset svn:eol-style 'native' -R *.dsw > > And then a svn commit > > Would this work? > What would happen if a file already has the requested property? Whether or not the file had the "svn:eol-style" property to begin with, running "svn propset svn:eol-style native" on it will set the "svn:eol-style" property to the value "native". Your above proposal won't work, unless you don't have subdirectories. This is because your shell, not Subversion, expands globs like "*.txt", and only in the current directory. For example, if your directory contains: a.txt b.txt c/ c/d.txt Then "svn propset svn:eol-style 'native' -R *.txt" will be expanded by the shell to "svn propset svn:eol-style 'native' -R a.txt b.txt" before Subversion ever sees the command. Then Subversion will set the property of a.txt and b.txt. The "-R" flag means "recursively process directories" but you didn't specify any directories, so no recursive directory processing occurs. What you may want to do instead is use the "find" command to recursively locate the files you want to modify. (I'm presuming you're running these commands on a UNIX-like operating system; I don't know what the equivalent Windows commands might be.) For example, in your working copy: find -E . -iregex '^.*\.(bat|c|conf|cpp|dpr|dsp|dsw|gld|h|inc|ini|iss|lpi|lpr|pas|py|txt|xml)$' -print0 | xargs -0 svn propset svn:eol-style native Now you can use "svn status" and "svn diff" to inspect the staged changes in the working copy, and either commit them if satisfied, or "svn revert -R ." if you want to discard the changes. This presumes that you can identify all of the files by extension. If your repository users use Windows, this might be the case. But on other operating systems, it's not unusual to find files with names like "README", with no ".txt" extension, which are nevertheless text files that you might want to apply the svn:eol-style property to. In that case, you may have to use a criterion other than extension to identify your files. For example, you could inspect the MIME type, and modify all files whose MIME type begins with "text/": find . -type f | file --mime-type --no-pad --separator :: --files-from - | sed -n 's,:: text/.*$,,p' | tr '\n' '\0' | xargs -0 svn propset svn:eol-style native
Re: EOL problems in cvs2svn converted repository
On Feb 9, 2018, at 14:48, Johan Corveleyn wrote: > ISTR there used to be a helper script to do this (to apply a certain > autoprops configuration to your working copy ... > svn-apply-autoprops.py or .pl or something), but I can't find it. > Maybe it was in the contrib folder which has been removed since (you > might be able to find a version of it googling). https://svn.apache.org/repos/asf/subversion/trunk/contrib/client-side/svn_apply_autoprops.py
