Re: svn merge operation extremely slow
[ Again: please don't top-post on this list. I'm moving your reply to
the bottom. More below ... ]
On Mon, Oct 3, 2011 at 2:24 AM, Kyle Leber wrote:
> On Sun, Oct 2, 2011 at 8:10 PM, Daniel Shahaf
> wrote:
>>
>> Kyle Leber wrote on Sun, Oct 02, 2011 at 20:05:19 -0400:
>> > Johan,
>> >
>> > I did a little more digging. There were a few different places where
>> > svn
>> > seems to get hung up so I ran the gprof report on just the first one
>> > (the
>> > merge takes hours otherwise). In this particular case, svn prints out
>> > that
>> > it is merging from a small text file while it is hanging for more than a
>> > minute @ 100% CPU. When I examine "lsof", however, it see it actually
>> > has a
>> > different file open. This one is a large (15 MB) "binary" file. It
>> > turns
>> > out this binary file did not have a property in the trunk (which I think
>> > means it's treated as text, right?). But in the branch it was marked as
>> > octet stream. So perhaps svn is doing a text-based diff on this binary
>> > file because it used to be incorrectly marked as text?
>> >
>>
>> If either side is marked as binary then svn will defer to the "Use
>> merge-right if merge-left == base, else conflict" algorithm.
>>
>> Could you share the value of 'svn proplist --verbose' on both files?
>>
> Yup, trunk version has empty properties
> branch version has:
>
> svn:mime-type
> application/octet-stream
>
What is the merge target? Is it a trunk working copy (the one without
mime-type), or a branch working copy (with
svn:mime-type=application/octet-stream)?
I think it's the mime-type of the merge target that determines if
merge will take the "binary" route, or the "text" route. See this
snippet from libsvn_wc/merge.c [1] (in the function
svn_wc__internal_merge):
[[[
/* Decide if the merge target is a text or binary file. */
if ((mimeprop = get_prop(&mt, SVN_PROP_MIME_TYPE))
&& mimeprop->value)
is_binary = svn_mime_type_is_binary(mimeprop->value->data);
else
{
const char *value = svn_prop_get_value(mt.actual_props,
SVN_PROP_MIME_TYPE);
is_binary = value && svn_mime_type_is_binary(value);
}
]]]
(mt is the merge target)
I'm not terribly familiar with this part of the codebase. But on first
sight, this seems to say:
(1) Look at the mime-type of the "base version" of the merge target.
If that's binary, then we'll go binary.
(2) If the "base" of the merge target doesn't have a mime-type, look
if it has one on the "actual" node (the uncommitted local
modifications). If that's binary, then we'll go binary.
(3) Else: text merge
So I'm guessing that you're merging to trunk, the target without
mime-type property, which makes svn take the "text" route for merging.
Is that correct?
If that's the case, maybe you can simply set the mime-type on that
binary file in your merge target, as a local modification (I don't
think you need to even commit it). Can you try that?
--
Johan
[1]
http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_wc/merge.c
Renamed/moved files diffing
We are using the CodeCollaborator system from SmartBear for code reviews. This system uses the command: svn -r "firstversion:lastversion" svnpath to generate a diff list used indicate which lines were modified for the code revision in question. Problem is with files that have been moved/renamed in the repository: The file under the old name is reported as deleted, under the new name as all new lines. CodeCollaborator cannot report which lines are changed/unchanged. It turns out that this is due to SVN, which In fact creates a diff file reporting exactly what CodeCollaborator displays. Except that there is a (cumbersome!) workaround: If the diff command explicitly names one single file down to the last character of the file extension, then SVN will generate a proper diff file. When you move a large tree, the number of files that must be handled one by one is large, and the manual work is significant. The reviewer will see the files as completely spearate resources, not as an element in a directory structure. Does anyone know of a way to force svn diff to create a proper diff for moved files even when diffing an entire directory, similar to what it does when diffing a single file? Or, as an alternative: Make CodeCollaborator automatically do the diffing in a file by file manner and present them in a directory structure (Obviously, I could ask th latter question in a CodeCollaborator forum, but I assume that SVN + CodeCollaborator users who read mailing lists are here as well... And the real problem is with SVN, not CodeCollaborator!) According to our IT systems guy, our SVN server runs version 1.6.6, r40053.
Re: svn merge operation extremely slow
On Mon, Oct 3, 2011 at 4:10 AM, Johan Corveleyn wrote:
> [ Again: please don't top-post on this list. I'm moving your reply to
> the bottom. More below ... ]
>
> On Mon, Oct 3, 2011 at 2:24 AM, Kyle Leber wrote:
> > On Sun, Oct 2, 2011 at 8:10 PM, Daniel Shahaf
> > wrote:
> >>
> >> Kyle Leber wrote on Sun, Oct 02, 2011 at 20:05:19 -0400:
> >> > Johan,
> >> >
> >> > I did a little more digging. There were a few different places where
> >> > svn
> >> > seems to get hung up so I ran the gprof report on just the first one
> >> > (the
> >> > merge takes hours otherwise). In this particular case, svn prints out
> >> > that
> >> > it is merging from a small text file while it is hanging for more than
> a
> >> > minute @ 100% CPU. When I examine "lsof", however, it see it actually
> >> > has a
> >> > different file open. This one is a large (15 MB) "binary" file. It
> >> > turns
> >> > out this binary file did not have a property in the trunk (which I
> think
> >> > means it's treated as text, right?). But in the branch it was marked
> as
> >> > octet stream. So perhaps svn is doing a text-based diff on this
> binary
> >> > file because it used to be incorrectly marked as text?
> >> >
> >>
> >> If either side is marked as binary then svn will defer to the "Use
> >> merge-right if merge-left == base, else conflict" algorithm.
> >>
> >> Could you share the value of 'svn proplist --verbose' on both files?
> >>
> > Yup, trunk version has empty properties
> > branch version has:
> >
> > svn:mime-type
> > application/octet-stream
> >
>
> What is the merge target? Is it a trunk working copy (the one without
> mime-type), or a branch working copy (with
> svn:mime-type=application/octet-stream)?
>
> I think it's the mime-type of the merge target that determines if
> merge will take the "binary" route, or the "text" route. See this
> snippet from libsvn_wc/merge.c [1] (in the function
> svn_wc__internal_merge):
>
> [[[
> /* Decide if the merge target is a text or binary file. */
> if ((mimeprop = get_prop(&mt, SVN_PROP_MIME_TYPE))
> && mimeprop->value)
>is_binary = svn_mime_type_is_binary(mimeprop->value->data);
> else
>{
> const char *value = svn_prop_get_value(mt.actual_props,
> SVN_PROP_MIME_TYPE);
>
> is_binary = value && svn_mime_type_is_binary(value);
>}
> ]]]
>
> (mt is the merge target)
>
> I'm not terribly familiar with this part of the codebase. But on first
> sight, this seems to say:
>
> (1) Look at the mime-type of the "base version" of the merge target.
> If that's binary, then we'll go binary.
>
> (2) If the "base" of the merge target doesn't have a mime-type, look
> if it has one on the "actual" node (the uncommitted local
> modifications). If that's binary, then we'll go binary.
>
> (3) Else: text merge
>
> So I'm guessing that you're merging to trunk, the target without
> mime-type property, which makes svn take the "text" route for merging.
> Is that correct?
>
> If that's the case, maybe you can simply set the mime-type on that
> binary file in your merge target, as a local modification (I don't
> think you need to even commit it). Can you try that?
>
> --
> Johan
>
> [1]
> http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_wc/merge.c
>
Johan,
Sorry for the top-post. Hopefully this is better :)
I set the mime-type to "application/octet-stream" in the working copy prior
to merge and this fixed the problem. No more heavy CPU usage or excessive
time spent on the file.
Kyle
Re: svn merge operation extremely slow
Kyle Leber wrote on Mon, Oct 03, 2011 at 08:16:53 -0400: > On Mon, Oct 3, 2011 at 4:10 AM, Johan Corveleyn wrote: > > (2) If the "base" of the merge target doesn't have a mime-type, look > > if it has one on the "actual" node (the uncommitted local > > modifications). If that's binary, then we'll go binary. > > > > (3) Else: text merge I stand corrected.
Re: svn merge operation extremely slow
On Mon, Oct 3, 2011 at 2:16 PM, Kyle Leber wrote:
>
>
> On Mon, Oct 3, 2011 at 4:10 AM, Johan Corveleyn wrote:
>>
>> [ Again: please don't top-post on this list. I'm moving your reply to
>> the bottom. More below ... ]
>>
>> On Mon, Oct 3, 2011 at 2:24 AM, Kyle Leber wrote:
>> > On Sun, Oct 2, 2011 at 8:10 PM, Daniel Shahaf
>> > wrote:
>> >>
>> >> Kyle Leber wrote on Sun, Oct 02, 2011 at 20:05:19 -0400:
>> >> > Johan,
>> >> >
>> >> > I did a little more digging. There were a few different places where
>> >> > svn
>> >> > seems to get hung up so I ran the gprof report on just the first one
>> >> > (the
>> >> > merge takes hours otherwise). In this particular case, svn prints
>> >> > out
>> >> > that
>> >> > it is merging from a small text file while it is hanging for more
>> >> > than a
>> >> > minute @ 100% CPU. When I examine "lsof", however, it see it
>> >> > actually
>> >> > has a
>> >> > different file open. This one is a large (15 MB) "binary" file. It
>> >> > turns
>> >> > out this binary file did not have a property in the trunk (which I
>> >> > think
>> >> > means it's treated as text, right?). But in the branch it was marked
>> >> > as
>> >> > octet stream. So perhaps svn is doing a text-based diff on this
>> >> > binary
>> >> > file because it used to be incorrectly marked as text?
>> >> >
>> >>
>> >> If either side is marked as binary then svn will defer to the "Use
>> >> merge-right if merge-left == base, else conflict" algorithm.
>> >>
>> >> Could you share the value of 'svn proplist --verbose' on both files?
>> >>
>> > Yup, trunk version has empty properties
>> > branch version has:
>> >
>> > svn:mime-type
>> > application/octet-stream
>> >
>>
>> What is the merge target? Is it a trunk working copy (the one without
>> mime-type), or a branch working copy (with
>> svn:mime-type=application/octet-stream)?
>>
>> I think it's the mime-type of the merge target that determines if
>> merge will take the "binary" route, or the "text" route. See this
>> snippet from libsvn_wc/merge.c [1] (in the function
>> svn_wc__internal_merge):
>>
>> [[[
>> /* Decide if the merge target is a text or binary file. */
>> if ((mimeprop = get_prop(&mt, SVN_PROP_MIME_TYPE))
>> && mimeprop->value)
>> is_binary = svn_mime_type_is_binary(mimeprop->value->data);
>> else
>> {
>> const char *value = svn_prop_get_value(mt.actual_props,
>> SVN_PROP_MIME_TYPE);
>>
>> is_binary = value && svn_mime_type_is_binary(value);
>> }
>> ]]]
>>
>> (mt is the merge target)
>>
>> I'm not terribly familiar with this part of the codebase. But on first
>> sight, this seems to say:
>>
>> (1) Look at the mime-type of the "base version" of the merge target.
>> If that's binary, then we'll go binary.
>>
>> (2) If the "base" of the merge target doesn't have a mime-type, look
>> if it has one on the "actual" node (the uncommitted local
>> modifications). If that's binary, then we'll go binary.
>>
>> (3) Else: text merge
>>
>> So I'm guessing that you're merging to trunk, the target without
>> mime-type property, which makes svn take the "text" route for merging.
>> Is that correct?
>>
>> If that's the case, maybe you can simply set the mime-type on that
>> binary file in your merge target, as a local modification (I don't
>> think you need to even commit it). Can you try that?
>>
>> --
>> Johan
>>
>> [1]
>> http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_wc/merge.c
>
> Johan,
>
> Sorry for the top-post. Hopefully this is better :)
Much better, thank you :).
> I set the mime-type to "application/octet-stream" in the working copy prior
> to merge and this fixed the problem. No more heavy CPU usage or excessive
> time spent on the file.
I'm glad it helped. Apart from the performance, it's important that
svn does this merge the "binary way", because as you said line-based
merges are not correct for this file.
--
Johan
Re: svn merge operation extremely slow
On Mon, Oct 3, 2011 at 2:35 PM, Johan Corveleyn wrote:
> On Mon, Oct 3, 2011 at 2:16 PM, Kyle Leber wrote:
>>
>>
>> On Mon, Oct 3, 2011 at 4:10 AM, Johan Corveleyn wrote:
>>>
>>> [ Again: please don't top-post on this list. I'm moving your reply to
>>> the bottom. More below ... ]
>>>
>>> On Mon, Oct 3, 2011 at 2:24 AM, Kyle Leber wrote:
>>> > On Sun, Oct 2, 2011 at 8:10 PM, Daniel Shahaf
>>> > wrote:
>>> >>
>>> >> Kyle Leber wrote on Sun, Oct 02, 2011 at 20:05:19 -0400:
>>> >> > Johan,
>>> >> >
>>> >> > I did a little more digging. There were a few different places where
>>> >> > svn
>>> >> > seems to get hung up so I ran the gprof report on just the first one
>>> >> > (the
>>> >> > merge takes hours otherwise). In this particular case, svn prints
>>> >> > out
>>> >> > that
>>> >> > it is merging from a small text file while it is hanging for more
>>> >> > than a
>>> >> > minute @ 100% CPU. When I examine "lsof", however, it see it
>>> >> > actually
>>> >> > has a
>>> >> > different file open. This one is a large (15 MB) "binary" file. It
>>> >> > turns
>>> >> > out this binary file did not have a property in the trunk (which I
>>> >> > think
>>> >> > means it's treated as text, right?). But in the branch it was marked
>>> >> > as
>>> >> > octet stream. So perhaps svn is doing a text-based diff on this
>>> >> > binary
>>> >> > file because it used to be incorrectly marked as text?
>>> >> >
>>> >>
>>> >> If either side is marked as binary then svn will defer to the "Use
>>> >> merge-right if merge-left == base, else conflict" algorithm.
>>> >>
>>> >> Could you share the value of 'svn proplist --verbose' on both files?
>>> >>
>>> > Yup, trunk version has empty properties
>>> > branch version has:
>>> >
>>> > svn:mime-type
>>> > application/octet-stream
>>> >
>>>
>>> What is the merge target? Is it a trunk working copy (the one without
>>> mime-type), or a branch working copy (with
>>> svn:mime-type=application/octet-stream)?
>>>
>>> I think it's the mime-type of the merge target that determines if
>>> merge will take the "binary" route, or the "text" route. See this
>>> snippet from libsvn_wc/merge.c [1] (in the function
>>> svn_wc__internal_merge):
>>>
>>> [[[
>>> /* Decide if the merge target is a text or binary file. */
>>> if ((mimeprop = get_prop(&mt, SVN_PROP_MIME_TYPE))
>>> && mimeprop->value)
>>> is_binary = svn_mime_type_is_binary(mimeprop->value->data);
>>> else
>>> {
>>> const char *value = svn_prop_get_value(mt.actual_props,
>>> SVN_PROP_MIME_TYPE);
>>>
>>> is_binary = value && svn_mime_type_is_binary(value);
>>> }
>>> ]]]
>>>
>>> (mt is the merge target)
>>>
>>> I'm not terribly familiar with this part of the codebase. But on first
>>> sight, this seems to say:
>>>
>>> (1) Look at the mime-type of the "base version" of the merge target.
>>> If that's binary, then we'll go binary.
>>>
>>> (2) If the "base" of the merge target doesn't have a mime-type, look
>>> if it has one on the "actual" node (the uncommitted local
>>> modifications). If that's binary, then we'll go binary.
>>>
>>> (3) Else: text merge
>>>
>>> So I'm guessing that you're merging to trunk, the target without
>>> mime-type property, which makes svn take the "text" route for merging.
>>> Is that correct?
>>>
>>> If that's the case, maybe you can simply set the mime-type on that
>>> binary file in your merge target, as a local modification (I don't
>>> think you need to even commit it). Can you try that?
>>>
>>> --
>>> Johan
>>>
>>> [1]
>>> http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_wc/merge.c
>>
>> Johan,
>>
>> Sorry for the top-post. Hopefully this is better :)
>
> Much better, thank you :).
>
>> I set the mime-type to "application/octet-stream" in the working copy prior
>> to merge and this fixed the problem. No more heavy CPU usage or excessive
>> time spent on the file.
>
> I'm glad it helped. Apart from the performance, it's important that
> svn does this merge the "binary way", because as you said line-based
> merges are not correct for this file.
It may also interest you (and other readers of this thread) that there
is an open enhancement request for making text-merges take the same
shortcut as binary-merges (if mine == merge-left then set merged :=
merge-right), to avoid expensive diffing [1]. But that hasn't been
addressed yet.
[1] http://subversion.tigris.org/issues/show_bug.cgi?id=4009 : Big
trivial text files merged MUCH slower than binary - pls optimize.
--
Johan
Re: svn merge operation extremely slow
Johan Corveleyn wrote on Mon, Oct 03, 2011 at 14:59:25 +0200: > It may also interest you (and other readers of this thread) that there > is an open enhancement request for making text-merges take the same > shortcut as binary-merges (if mine == merge-left then set merged := > merge-right), to avoid expensive diffing [1]. But that hasn't been > addressed yet. > > > [1] http://subversion.tigris.org/issues/show_bug.cgi?id=4009 : Big > trivial text files merged MUCH slower than binary - pls optimize. > Isn't "Set the svn:mime-type property locally, and revert it before commit" a workaround for that? > -- > Johan
Re: svn merge operation extremely slow
On Mon, Oct 03, 2011 at 02:59:25PM +0200, Johan Corveleyn wrote: > On Mon, Oct 3, 2011 at 2:35 PM, Johan Corveleyn wrote: > > On Mon, Oct 3, 2011 at 2:16 PM, Kyle Leber wrote: > >> I set the mime-type to "application/octet-stream" in the working copy prior > >> to merge and this fixed the problem. No more heavy CPU usage or excessive > >> time spent on the file. > > > > I'm glad it helped. Apart from the performance, it's important that > > svn does this merge the "binary way", because as you said line-based > > merges are not correct for this file. > > It may also interest you (and other readers of this thread) that there > is an open enhancement request for making text-merges take the same > shortcut as binary-merges (if mine == merge-left then set merged := > merge-right), to avoid expensive diffing [1]. But that hasn't been > addressed yet. > > > [1] http://subversion.tigris.org/issues/show_bug.cgi?id=4009 : Big > trivial text files merged MUCH slower than binary - pls optimize. > I think we should also file an issue about the problem discussed in this thread. svn should take properties on the left/right side of the merge into account when determining whether to treat a file as binary. I guess it should run the binary merge if any of left, right, or the target are marked as binary.
How to revert a directory to a previous revision
Hi, I want to revert 2 directories in my repos to a previous revision. In the repos view, when I right-click on the folder I want to revert I see an option "revert to previous version" or the like. Right-clicking the local directory to replace it with a previous version does not give me the option to revert to a specific version - just "Base revision", "Latest from repository" and "Base revision" What is the best way to do this? I have Subversion 1.6.17 I use the Subclipse (v1.6.18) Subversion plugin in Eclipse. I use Tortoise 1.6.16 (b 21511) Windows 7
Re: svn merge operation extremely slow
On Mon, Oct 3, 2011 at 3:02 PM, Daniel Shahaf wrote: > Johan Corveleyn wrote on Mon, Oct 03, 2011 at 14:59:25 +0200: >> It may also interest you (and other readers of this thread) that there >> is an open enhancement request for making text-merges take the same >> shortcut as binary-merges (if mine == merge-left then set merged := >> merge-right), to avoid expensive diffing [1]. But that hasn't been >> addressed yet. >> >> >> [1] http://subversion.tigris.org/issues/show_bug.cgi?id=4009 : Big >> trivial text files merged MUCH slower than binary - pls optimize. >> > > Isn't "Set the svn:mime-type property locally, and revert it before > commit" a workaround for that? Yes, it would seem so. Though it may not be very helpful in lots of situations (because people only discover the problem after they out-waited a merge of several hours). Still, it's useful information to work around it (maybe people can detect the problem in some wrapper scripts, ahead of merging), so maybe you should add it to the issue tracker. -- Johan
Re: svn merge operation extremely slow
On Mon, Oct 3, 2011 at 3:04 PM, Stefan Sperling wrote: > On Mon, Oct 03, 2011 at 02:59:25PM +0200, Johan Corveleyn wrote: >> On Mon, Oct 3, 2011 at 2:35 PM, Johan Corveleyn wrote: >> > On Mon, Oct 3, 2011 at 2:16 PM, Kyle Leber wrote: >> >> I set the mime-type to "application/octet-stream" in the working copy >> >> prior >> >> to merge and this fixed the problem. No more heavy CPU usage or excessive >> >> time spent on the file. >> > >> > I'm glad it helped. Apart from the performance, it's important that >> > svn does this merge the "binary way", because as you said line-based >> > merges are not correct for this file. >> >> It may also interest you (and other readers of this thread) that there >> is an open enhancement request for making text-merges take the same >> shortcut as binary-merges (if mine == merge-left then set merged := >> merge-right), to avoid expensive diffing [1]. But that hasn't been >> addressed yet. >> >> >> [1] http://subversion.tigris.org/issues/show_bug.cgi?id=4009 : Big >> trivial text files merged MUCH slower than binary - pls optimize. >> > > I think we should also file an issue about the problem discussed > in this thread. svn should take properties on the left/right side of the > merge into account when determining whether to treat a file as binary. > I guess it should run the binary merge if any of left, right, or the > target are marked as binary. Yes, maybe you're right. I don't know the specifics / historics of this behavior (maybe there is a reason for this?). But on the surface it looks like it should indeed do a binary merge if either one of left, right or target is marked as binary. Even if #4009 would be addressed, it would still make a difference in the situation where the shortcut-condition (mine == merge-left) doesn't hold. In that case, I think the "binary-merge" would always flag a conflict (because it can't do a line-based merge). Is that also the behavior we want f.i. if only merge-left (or only merge-right) were marked as binary, and all the other "players" are marked as text? I guess it's the safest thing to do ... -- Johan
Compatibility Check Windows 7 Movares TIGRIS
> Dear Sir/ Madam > > To understand the compatibility of > > SUBVERSION > > TORTOISESVN > > 1.4 > 1.5 > > onthe Microsoft Windows 7 platform, I would like to receive some > informationregarding the following questions: > > 1. Is the application compatible with Windows 7 Enterprise? > 2. Is the application compatible with Windows 7 Enterprise SP1? > 3. Is the application compatible with Windows 7 Enterprise (SP1) > 64-bit installation? > 4. Is the application compatible with Remote Desktop Services (RDS) on > Windows Server 2008 R2? > 5. Is there a 64-bit version of the application available? > 6. if so, what version? > 7. If not, it is planned and possibly when? > 8. If the application is compatible, then you recommend this > application to > 32 bit or 64 bit install (in case of any available 32-bit plugins, > add-ons, drivers, etc.)? > > Thank you in advance for your help. > > With kind regards, > > Jurgen Derksen -- C. Michael Pilato CollabNet <> www.collab.net <> Distributed Development On Demand Deze e-mail en de inhoud daarvan is vertrouwelijk. Indien dit bericht niet voor u bestemd is, verzoeken wij u deze e-mail direct aan ons te retourneren en daarna te vernietigen. In dit geval is het ook niet toegestaan deze e-mail en de inhoud daarvan te gebruiken, kopieren of openbaar te maken aan derden. Onze onderneming sluit elke aansprakelijkheid uit in verband met het niet juist, onvolledig of niet tijdig overkomen van de informatie in deze e-mail. Movares Nederland B.V. / Utrecht / Kamer van Koophandel 30124367. This e-mail and its contents are confidential and may be legally privileged. If this e-mail is not intended for you, please contact us immediately by reply e-mail and destroy the e-mail. In this case, please do not use, copy or disclose the e-mail and its contents to anyone. Our company is liable neither for the proper and complete transmission of the information in this e-mail nor for any delay in its receipt. Movares Nederland B.V. / Utrecht / Chamber of Commerce 30124367.
