Random files being "reverted" on one repository
Hi List, We administer subversion (v 1.4.2, r22196 on CentOS 5.5) for a development company, and have over 150 active repositories, but we are not subversion experts. We are experiencing an issue with just one particular repository. When programmers run an update against this one repository (using either TortoiseSVN 1.6.15 or Slik Subversion 1.6.16 on Windows 7 Pro SP1), they observe that they sometimes get---to coin their term---"frankenstein" versions of arbitrary files. As an example scenario: 1) - Developer A: adds, edits and commits a file X, 2) - Developer A: later, again edits and commits file X, 3) - Developer A: still later, again edits and commits file X, 4) - Developer N: who has never before seen file X, runs an update. She gets a weird version of file X which contains only *some* lines of the set of changes made by Developer A in each of the edit/commit sessions (1), (2), and (3). We cannot replicate the problem on demand, but it recurs with (seemingly) random files at random times. The worst thing is that when an update silently "reverts" some unknown file (to a "frankenstein" version), it is subsequently committed as a new version by the unsuspecting developer. We've tried exporting and re-importing the code to a new repository, but the issue has persisted. "Svnadmin verify" finds no errors in any revisions. Our latest move was to disable the Windows Search service, but if that's really the problem, our other developers should be seeing this with other repositories. Any advice on how to duplicate/troubleshoot the cause of this problem will be greatly appreciated. Thank you, Dave
Re: Random files being "reverted" on one repository
Campbell Allan wrote: On Wednesday 11 May 2011, Dave Tingling wrote: Hi List, We administer subversion (v 1.4.2, r22196 on CentOS 5.5) for a development company, and have over 150 active repositories, but we are not subversion experts. We are experiencing an issue with just one particular repository. When programmers run an update against this one repository (using either TortoiseSVN 1.6.15 or Slik Subversion 1.6.16 on Windows 7 Pro SP1), they observe that they sometimes get---to coin their term---"frankenstein" versions of arbitrary files. As an example scenario: 1) - Developer A: adds, edits and commits a file X, 2) - Developer A: later, again edits and commits file X, 3) - Developer A: still later, again edits and commits file X, 4) - Developer N: who has never before seen file X, runs an update. She gets a weird version of file X which contains only *some* lines of the set of changes made by Developer A in each of the edit/commit sessions (1), (2), and (3). We cannot replicate the problem on demand, but it recurs with (seemingly) random files at random times. The worst thing is that when an update silently "reverts" some unknown file (to a "frankenstein" version), it is subsequently committed as a new version by the unsuspecting developer. We've tried exporting and re-importing the code to a new repository, but the issue has persisted. "Svnadmin verify" finds no errors in any revisions. Our latest move was to disable the Windows Search service, but if that's really the problem, our other developers should be seeing this with other repositories. Any advice on how to duplicate/troubleshoot the cause of this problem will be greatly appreciated. Thank you, Dave Hi, I've seen something like this before when some of the users were remote nd were accessing the repository via a SVK mirror. The mirror was reverting files prior to committing to what I presume it thought the state should be. This was some years ago, but if I recall correctly the commits for the reverted files did not have any commit message. I don't know if it was an artifact of the setup we had (I never did the setup) but commits coming via the SVK mirror would be from a single user but the real user would be part of the commit message. Could something like this be happening for you? Campbell Thanks Campbell. No remote users, no SVK mirror in place, nor anything intermediate between the server and clients. -Dave
Re: Random files being "reverted" on one repository
Daniel Shahaf wrote: fsfs or bdb? Tried 1.5.7 / 1.6.16 server? Dave Tingling wrote on Wed, May 11, 2011 at 11:50:58 -0400: Hi List, We administer subversion (v 1.4.2, r22196 on CentOS 5.5) for a development company, and have over 150 active repositories, but we are not subversion experts. We are experiencing an issue with just one particular repository. When programmers run an update against this one repository (using either TortoiseSVN 1.6.15 or Slik Subversion 1.6.16 on Windows 7 Pro SP1), they observe that they sometimes get---to coin their term---"frankenstein" versions of arbitrary files. As an example scenario: 1) - Developer A: adds, edits and commits a file X, 2) - Developer A: later, again edits and commits file X, 3) - Developer A: still later, again edits and commits file X, 4) - Developer N: who has never before seen file X, runs an update. She gets a weird version of file X which contains only *some* lines of the set of changes made by Developer A in each of the edit/commit sessions (1), (2), and (3). We cannot replicate the problem on demand, but it recurs with (seemingly) random files at random times. The worst thing is that when an update silently "reverts" some unknown file (to a "frankenstein" version), it is subsequently committed as a new version by the unsuspecting developer. We've tried exporting and re-importing the code to a new repository, but the issue has persisted. "Svnadmin verify" finds no errors in any revisions. Our latest move was to disable the Windows Search service, but if that's really the problem, our other developers should be seeing this with other repositories. Any advice on how to duplicate/troubleshoot the cause of this problem will be greatly appreciated. Thank you, Dave FSFS. My understanding is that the 1.4.2 (reported by running "svn --version" on the server) is what ships with CentOS 5.5, and that it is "backported" so that it includes recent updates, bugfixes, etc. I'm not sure how to check what repo format is on the server, though. We have not yet tried 1.5.7 or 1.6.16 server. Thanks, -Dave
Re: Random files being "reverted" on one repository
Thorsten Schöning wrote: Guten Tag Dave Tingling, am Mittwoch, 11. Mai 2011 um 17:50 schrieben Sie: 1) - Developer A: adds, edits and commits a file X, 2) - Developer A: later, again edits and commits file X, 3) - Developer A: still later, again edits and commits file X, 4) - Developer N: who has never before seen file X, runs an update. She gets a weird version of file X which contains only *some* lines of the set of changes made by Developer A in each of the edit/commit sessions (1), (2), and (3). And this behaviour is reproducible, meaning that every time the file x is deleted from the local working copy of dev N on each and every update the newly created file is freaky? Is it freaky on clean checkouts, too? We cannot replicate the problem on demand, but it recurs with (seemingly) random files at random times. The worst thing is that when an update silently "reverts" some unknown file (to a "frankenstein" version), it is subsequently committed as a new version by the unsuspecting developer. How do you know that the update is the problem? If the file in the repository looks fine, I don't think the problem comes with the update, but afterwards, on any build actions or stuff like that. As you say: The freaky file is committed by dev N and that is the problem, between update and commit it is somehow changed. The update process just make changes to the local file, if already present, that are on the server and it's pretty likely that this will work as expected. The interesting part is between update and commit of dev N and in your case i would look at the diffs to maybe get an idea of the tool or process or whatever is responsible for the changes in the file. Maybe there's something common in every changes? Mit freundlichen Grüßen, Thorsten Schöning Thanks for the quick reply and your guidance Thorsten. I'm not 100% sure. I'll work with the developers to clarify, and then will report back. Thank you, -Dave
Re: Random files being "reverted" on one repository
Thanks Daniel. Results from in the repo dir on the server:: [dt@s..]$ cat format 5 [dt@s..]$ cat db/fs-type fsfs [dt@s..]$ cat db/format 2 Daniel Shahaf wrote: Dave Tingling wrote on Wed, May 11, 2011 at 12:38:08 -0400: I'm not sure how to check what repo format is on the server, though. cat $REPOS/format cat $REPOS/db/fs-type cat $REPOS/db/format (it'd be nice to document this in the FAQ along with the format number -> minor version number mappings) -- Dave Tingling Business Process Analyst Info Tech, Inc. 352-381-4512 (direct) 352-381- (fax)
Re: Random files being "reverted" on one repository
Thorsten Schöning wrote: Guten Tag Dave Tingling, am Mittwoch, 11. Mai 2011 um 17:50 schrieben Sie: And this behaviour is reproducible, meaning that every time the file x is deleted from the local working copy of dev N on each and every update the newly created file is freaky? Is it freaky on clean checkouts, too? Thorsten, It is not reproducible as you propose above: deleting the local file and updating again has (so far) resulted in a good copy. I received these further comments from one of the developers: In this case, there are no build actions to speak of. These are pretty much just perl files, and we don't use visual studio or any project management type apps. A handful of us use Komodo for project management, but the problem seems to impact people regardless of editor or SVN client. The problem does in fact happen during the update phase, because other people can do the same update and get the correct version of the file. When someone experiences the error, Their local copy will show as modified immediately after the update, and attempting to commit the working copy directory will include these changes. No conflict will be detected. To clarify, I confirmed that they do not use Komodo to talk directly to the server; they only point it to working-copies. Thanks, -Dave
Re: Random files being "reverted" on one repository
Daniel Shahaf wrote: Which tells me your repository was created by Subversion 1.4. Nothing unexpected... I've learned that clients update their working-copy layout to whatever that client is built for. Considering http://stackoverflow.com/questions/1364618/how-do-i-determine-svn-working-copy-layout-version, I realize that our 1.6.x clients indeed have '10' on the first line of .svn/entries. What do these $REPOS/format = 5 and $REPOS/db/format = 2 each mean, please? (I'll be happy to document in the FAQ if I can :-) Is it possible that we're seeing this problem because of using 1.6 clients against this server? Thanks, -Dave
Re: Random files being "reverted" on one repository
Thanks for those tips Konstantin. There are about 16 developers. Is there anything I can look at on the server side to determine whether such bad directory copy/moves have been done, and perhaps by whom? On the server: what protocol you are using to access the repository and commit the files? HTTPS...I'm not sure what mechanisms happen "within" or "lower than" that. Is this a helpful answer? If you need more info, please let me know how I can find it out. I personally am a facilitator for researching this issue, I'm not an admin. I hope there is no proxy in between you and client, that performs some sort of url rewriting? No proxy, no rewriting. We administer subversion (v 1.4.2, r22196 on CentOS 5.5) Anyway, 1.4.2 is very old, and misses a number of fixes in 1.4.x line, not to mention the later versions. In some other thread recently when "CentOS 5.5" was mentioned one of the answers was to upgrade to 5.6 ASAP. Upgrading is non-trivial for us, for internal policy reasons (recall 150+ active repos). We realize that "upgrade now" would be a very strong encouragement in this forum, and understand that rationale. With due respect to that idea, I'm hoping we can identify why only one repository manifests this issue, and maybe why, after an export / import to a new repository, the problem appears in the new repo. Again, thank you very much. -Dave
Re: Random files being "reverted" on one repository
Thorsten Schöning wrote: I would try to test a bit: Take a dev where the file never was freaky and let him produce small changes which are committable, e.g. add a newline at the end of the file and remove it or stuff like that. But it should be always the same changes. We will. Let me make sure I understand you: you're suggesting testing multiple commits, but each consecutive commit will be an "undoing" of the previous change. If that's true, after many many commits, I have only ever really committed two different variations of the file, in alternation. Is that what you intend? Assuming I understood you correctly above, would there be any value if I commit a completely new change to the same file each time, perhaps (for agument's sake) adding a new, predictable, unique line to the file? I would add the numeral '1' as line one, then next time add the numeral '2' as line 2, etc etc. always appending. What do you think? After each change let your problematic developer N update the file and see if the error happens and if it always results in the same wrong changes. Understood. Just to clarify our situation, there no single one problematic developer (nor file, for that matter). This could happen to anyone of the team---when some random person updates, a file is "wrong". It could be any one developer, any file. I can't believe the error is in the subversion code, sounds more like a client problem to me. Client side scripts, programs locking the file and all that kind of stuff. We are inclined to agree. Will proceed to test. Thanks very much. -Dave
Re: Random files being "reverted" on one repository
Hi Bob, Does the same team us more than one repository? Or is each repo used by different people and teams? Some members of this team do work with other repositories. But this small team (7-10 active devs) is the only one that touches this problematic repository. Is the update funky when there is no file on the disk, or only when merging into an existing file? Both are true...if the file never existed locally before, it arrives funky. And if it did exist before, it ends up funky. What encoding are the files using? I have found that svn doesn't really work well with Unicode files. I had thought that different encoding on different workstations/files might be an issue, but don't know of a good way to investigate (I found a Unix tool, but nothing concrete for Win7). Did you mean the files on the various workstations, or are you thinking of some setting on the server-side? Please suggest how to best check the encoding(s?) to answer your question. I should mention that at least one member of this team is using a 32-bit PC/OS and 32-bit current version of Tortoise, the others are using 64-bit versions of the OS/Tortoise combo. Is it always the same file(s) or different ones each time? Different files each time the incident is observed. This incident is not observed after every update---only occasionally. Thanks, -Dave
