Hi, I've created the script. Not sure how to share it with the list so I've created a gist - https://gist.github.com/plamentotev/b835dd62c74a3a5becd4c317b97403a4
It will migrate the tags for all repositories in plugins/maven-* Also it checks for the 'git-svn' string in the commit message so it will not migrate tags created after the git migration. And I have commented the `git tag` command so by default it will be run in "dry mode" that will print all errors found without changing the repositories. Speaking of errors - there are about 30 tags that could not be automatically migrated. They are actually two group of errors: * the "prepare release" commit does not have the same files as the tag - meaning that most likely the tag in SVN is not just a copy of the "prepare release" commit and contains some changes made after that * there is no "prepare release" commit reachable from master. That could mean that the tag is not created by the release plugin and sometimes that is the case, but it looks like the majority of the cases are because the release is created from branch and not from trunk. And while we're on the topic - looks like the branches are not migrated into the maven-plugins repository. Also as the script modifies the tag I fell obligated to mention the usual warning when changing tags - https://git-scm.com/docs/git-tag#_on_backdating_tags Regards, Plamen Totev On Thu, Dec 28, 2017 at 10:43 AM, Hervé BOUTEMY <[email protected]> wrote: > Hi Plamen, > > Thank you for your help > > Yes, I looked more in depth to maven-plugins svn2git mirror and the split > result (got from the split script): the issue with tags is already there > > I like the idea of recreating tags from "[maven-release-plugin] prepare > release <name of the tag>" commit, ignoring the "[maven-release-plugin] copy > for tag <name of the tag>" commit that don't change anything in code > > If you can prepare a script to create the tags with this convention, it would > really be useful (don't forget that sometimes there are multiple tries at one > release, then the useful tag is the last one in time) > > Regards, > > Hervé > > Le lundi 25 décembre 2017, 09:58:36 CET Plamen Totev a écrit : >> Hi, >> >> What I can see is that the tags actually introduce new branches that >> contain commits with the same content as the one in the master (trunk) >> branch. But this is not caused by the script that splits of the >> repository - the problem exists in the maven-plugins Git repository as >> well. Maybe it's not quite visible because there are a lot of commits, >> but it's there. >> >> How can it be fixed? I think it would be easier to fix the individual >> plugin repositories (after the split) as they are smaller and easier >> to work with. The plan is simple - as the branches created by the tags >> does not contain new commits(with a single exception but I'll explain >> that in a minute) and all of them are present in the master branch we >> could just change the tag to point to the same commits but in the >> master branch. That will cause all extra commits and branches to be >> garbage collected as they are accessible only from the tags. For our >> purposes commits are equal if they point to the same root tree (the >> same files with the same content). But unfortunately there is a catch. >> What I said about the branches created by the tags is not strictly >> true - they do have some extra commits. When you release in SVN you >> have the following sequence of commits: >> >> * [maven-release-plugin] copy for tag maven-war-plugin-3.2.0 (this is >> the tagged commit) >> * [maven-release-plugin] prepare release maven-war-plugin-3.2.0 >> >> After the migration to Git the master (trunk) branch contain only the >> 'prepare release' commit but not the 'copy for tag' commit. So if we >> apply my plan then the 'copy for tag' commit will be lost. It's just a >> copy (no changes, it contains the same files as the previous commit) >> so I don't think it's a problem but would be nice if somebody >> confirms. >> >> So the proposed plan have the following caveats: >> >> * the 'copy for tag' commits will be lost. No changes will be lost as >> those are only copy commits >> * the tags will point to the previous 'prepare release' commits >> * the tags SHA will be changed because they point to a different commit >> >> If that is OK I can write a script that will apply those changes. >> >> What causes the branches and all those duplicating commits? Well the >> master branch tracks >> https://svn.apache.org/repos/asf/maven/components/trunk/maven-plugins >> but the tags for the plugins are actually tracking the plugin >> directory (for example: >> https://svn.apache.org/repos/asf/maven/components/trunk/maven-plugins/maven-> >> war-plugin). So for maven-war-plugins you have two commits in the >> maven-plugins Git repository - one for >> https://svn.apache.org/repos/asf/maven/components/trunk/maven-plugins/maven-> >> war-plugin and one for >> https://svn.apache.org/repos/asf/maven/components/trunk/ no matter that >> actually those commits are the same - same date, >> author, message, files, etc. >> >> Regards, >> Plamen Totev >> >> On Sun, Dec 24, 2017 at 11:54 AM, <[email protected]> wrote: >> > I'd suggest to try the process to a personal personal repo on GitHub to >> > see if you're able to get a better result before involving manual work >> > from INFRA (on more than 60 repos...) >> > >> > (it's sad to see nobody try to explain what's happenning or improve the >> > documented commands, just get to a conclusion: re-do everything and pray) >> > >> > Regards, >> > >> > Hervé >> > >> > ----- Mail original ----- >> > De: "Karl Heinz Marbaise" <[email protected]> >> > À: "Maven Developers List" <[email protected]>, "Robert Scholte" >> > <[email protected]> Envoyé: Dimanche 24 Décembre 2017 10:47:43 >> > Objet: Re: [IMPORTANT] Re: Git migration next steps >> > >> > Hi, >> > >> > On 24/12/17 10:40, Robert Scholte wrote: >> >> How about a hard reset or dropping the repo and doing it all over again? >> > >> > If I correctly seen that ..there had no commit yet on the new git repos.. >> > >> > So I think it would be the easiest way to do as Robert suggest ...to >> > redo migration for those repos.. >> > >> > Kind regards >> > Karl Heinz >> > >> >> On Wed, 20 Dec 2017 10:42:36 +0100, Hervé BOUTEMY >> >> >> >> <[email protected]> wrote: >> >>> INFRA-15679 fixed by infra team >> >>> then I re-run migrate-plugins.sh script to split the svn2git mirror to >> >>> per- >> >>> plugin git repo >> >>> and I pushed "master2" branches for m-javadoc-p, m-site-p and m-pdf-p, >> >>> which >> >>> were the 3 plugins which suffered from missing commits >> >>> >> >>> on m-site-p and m-pdf-p, I'll cherry pick the unique commit that was >> >>> missed: >> >>> not a big deal >> >>> >> >>> on m-javadoc-p, the situation is more coplex, since there was a release >> >>> >> >>> I also noticed that I forgot to push tags when importing: I started to >> >>> do "git >> >>> push --tags", but the result does not look as I expected: it creates a >> >>> lot of >> >>> parallel branches >> >>> >> >>> I'll need help from git experts: what is happening? >> >>> >> >>> I stopped the tags push half the way, we'll need to decide what to do... >> >>> (I knew I was not a git expert and there was a risk for something >> >>> weird like >> >>> what's currently happening...) >> >>> >> >>> Any hint? >> >>> >> >>> Regards, >> >>> >> >>> Hervé >> >>> >> >>> Le samedi 16 décembre 2017, 16:28:48 CET Hervé BOUTEMY a écrit : >> >>>> ok, I was confused by the different takes at m-javadoc-p 3.0.0 >> >>>> >> >>>> yes, svn2git mirror is stuck [1] at r1815675 >> >>>> >> >>>> I just opened an INFRA Jira issue >> >>>> https://issues.apache.org/jira/browse/INFRA-15679 >> >>>> >> >>>> once the svn2git mirror will be updated, we'll have to re-run the split >> >>>> scripts and cherry pick the missing commits >> >>>> >> >>>> Regards, >> >>>> >> >>>> Hervé >> >>>> >> >>>> [1] https://github.com/apache/maven-plugins/commits/trunk >> >>>> >> >>>> Le samedi 16 décembre 2017, 13:01:05 CET Robert Scholte a écrit : >> >>>> > I was triggered by some failing unit tests, which should have been >> >>>> >> >>>> solved >> >>>> >> >>>> > in maven-javadoc-plugin-3.0.0 >> >>>> > >> >>>> > My last commit according to GIT was november 18th >> >>>> > My last commit according to SVN was december 3rd >> >>>> > >> >>>> > comparing svnlog with gitlog most of these commits are lost: >> >>>> > >> >>>> > moved to git >> >>>> > ---- >> >>>> > [maven-release-plugin] prepare for next development iteration >> >>>> > ---- >> >>>> > [maven-release-plugin] prepare release maven-javadoc-plugin-3.0.0 >> >>>> > ---- >> >>>> > [MJAVADOC-498] "module not found" when Java 9 module-info present >> >>>> > Support aggrated javadoc >> >>>> > ---- >> >>>> > Skip several unittests for Java9 >> >>>> > ---- >> >>>> > JDK-8032205 was closed as not an issue, so not solved in Java9. >> >>>> > Need to review the conclusion >> >>>> > ---- >> >>>> > Upgrade mockito to remove warning about illegal reflective access >> >>>> > ---- >> >>>> > Improve TestJavadocReportTest#testTestJavadoc >> >>>> > J8 warns and continues with missing dependency, J9 fails. >> >>>> > In fact test was wrong: dependency should have been on classpath >> >>>> > ---- >> >>>> > unittest should prefer JAVA_HOME when executing from cmdline >> >>>> > When running with Java9+ no need to switch from jre to jdk directory >> >>>> > (jep220) >> >>>> > ---- >> >>>> > MJAVADOC-502 Update DEFAULT_JAVA_API_LINKS >> >>>> > ---- >> >>>> > session is required parameter, so cannot be null. Fix related >> >>>> >> >>>> unittests >> >>>> >> >>>> > ---- >> >>>> > Add project/artifact key to set of sourcePaths to recognize reactor >> >>>> > projects versus dependencies >> >>>> > ---- >> >>>> > Group sets of sourcepaths per project, in prepare of usage of >> >>>> > module-source-path. >> >>>> > ---- >> >>>> > Switch from List to Collection to make it easier to use Sets when >> >>>> > preferred >> >>>> > ---- >> >>>> > [maven-release-plugin] prepare for next development iteration >> >>>> > ---- >> >>>> > [maven-release-plugin] prepare release maven-javadoc-plugin-3.0.0 >> >>>> > ---- >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > On Sat, 16 Dec 2017 12:53:23 +0100, Hervé BOUTEMY >> >>>> >> >>>> <[email protected]> >> >>>> >> >>>> > wrote: >> >>>> > > looking at commits@ content https://lists.apache.org/list.html? >> >>>> > > [email protected] with subject containing >> >>>> >> >>>> "maven/plugins/trunk" >> >>>> >> >>>> > > and plugins svn2git mirror >> >>>> > > https://github.com/apache/maven-plugins/commits/ >> >>>> > > trunk >> >>>> > > >> >>>> > > only 1 commit is missing: my latest commit on maven-site-plugin >> >>>> > > (the last commit for Git migration is not useful) >> >>>> > > >> >>>> > > >> >>>> > > Same on shared showed no missing commit. >> >>>> > > >> >>>> > > >> >>>> > > what latest commit of maven-javadoc-plugin are you looking for? >> >>>> > > >> >>>> > > Regards, >> >>>> > > >> >>>> > > Hervé >> >>>> > > >> >>>> > > Le samedi 16 décembre 2017, 11:56:31 CET Robert Scholte a écrit : >> >>>> > >> For everybody just a warning I faced today: >> >>>> > >> If you switch to the git repos, please make sure all commits are >> >>>> > >> migrated. >> >>>> > >> I noticed the latest commits of the maven-javadoc-plugin got lost. >> >>>> > >> >> >>>> > >> thanks, >> >>>> > >> Robert >> >>>> > >> >> >>>> > >> On Sat, 09 Dec 2017 17:06:09 +0100, Stephen Connolly >> >>>> > >> >> >>>> > >> <[email protected]> wrote: >> >>>> > >> > I see we have a large number of repos now on gitbox ;-) >> >>>> > >> > >> >>>> > >> > On Thu 7 Dec 2017 at 07:00, Hervé BOUTEMY >> >>>> > >> > <[email protected]> >> >>>> > >> >> >>>> > >> wrote: >> >>>> > >> >> ok, I didn't update my repo clone: now the run-its profile is >> >>>> > >> >> >>>> > >> activated >> >>>> > >> >> >>>> > >> >> then the plan should just confirm "it works!" :) >> >>>> > >> >> >> >>>> > >> >> and find which jobs are special, like maven-dist-tool (which >> >>>> >> >>>> has to >> >>>> >> >>>> > >> be >> >>>> > >> >> >>>> > >> >> scheduled daily instead of code change, and one platform only) >> >>>> > >> >> >> >>>> > >> >> Regards, >> >>>> > >> >> >> >>>> > >> >> Hervé >> >>>> > >> >> >> >>>> > >> >> Le mercredi 6 décembre 2017, 23:58:45 CET Stephen Connolly a >> >>>> >> >>>> écrit : >> >>>> > >> >> > On Wed 6 Dec 2017 at 22:38, Hervé BOUTEMY >> >>>> >> >>>> <[email protected]> >> >>>> >> >>>> > >> >> wrote: >> >>>> > >> >> > > Now that we have 2 ASF Organization Jenkins jobs (one for >> >>>> >> >>>> gitbox >> >>>> >> >>>> > >> [1] >> >>>> > >> >> >>>> > >> >> and >> >>>> > >> >> >> >>>> > >> >> > > one >> >>>> > >> >> > > for git-wip: thank you Stephen) and that it looks quite >> >>>> > >> >> >>>> > >> successful, >> >>>> > >> >> >>>> > >> >> let's >> >>>> > >> >> >> >>>> > >> >> > > plan >> >>>> > >> >> > > the next steps. >> >>>> > >> >> > > >> >>>> > >> >> > > Here is what I see: >> >>>> > >> >> > > 1. removal of hand-defined Jenkins jobs that are now >> >>>> >> >>>> duplicates >> >>>> >> >>>> > >> >> > > 2. preparation of the 60 new empty git repos for shared & >> >>>> > >> >> > > plugins >> >>>> > >> >> > > >> >>>> > >> >> > > 3. migration of the 1 shared component and 1 plugin using >> >>>> > >> >> >> >>>> > >> >> migrate-*.sh >> >>>> > >> >> >> >>>> > >> >> > > scripts >> >>>> > >> >> > > [3] to test and eventually rework the Jenkinsfile (I >> >>>> >> >>>> suppose it >> >>>> >> >>>> > >> will >> >>>> > >> >> >>>> > >> >> > > require >> >>>> > >> >> > > some little change, to run add "run-its" profile) >> >>>> > >> >> > >> >>>> > >> >> > As far as I recall, I added -P+run-its already >> >>>> > >> >> > >> >>>> > >> >> > For the plugin, I'd like to do the job for maven-site-plugin, >> >>>> > >> >> >>>> > >> since we >> >>>> > >> >> >>>> > >> >> > > expect >> >>>> > >> >> > > to release it soon. >> >>>> > >> >> > > For the shared component, I don't know if there is a best >> >>>> > >> >> >>>> > >> candidate >> >>>> > >> >> >>>> > >> >> > > 4. once previous step is ok, do the full migration: if >> >>>> >> >>>> there are >> >>>> >> >>>> > >> >> > > volunteers >> >>>> > >> >> > > for helping, that would be great, since populating 60 git >> >>>> >> >>>> repos >> >>>> >> >>>> > >> >> won't >> >>>> > >> >> be >> >>>> > >> >> >> >>>> > >> >> > > really fun... >> >>>> > >> >> > > >> >>>> > >> >> > > And as part of 60 empty git repos creation, I propose to >> >>>> >> >>>> migrate >> >>>> >> >>>> > >> the >> >>>> > >> >> >>>> > >> >> > > "Google >> >>>> > >> >> > > repo manifest" maven-aggregator [4] to ASF: my personal >> >>>> >> >>>> use has >> >>>> >> >>>> > >> been >> >>>> > >> >> >>>> > >> >> quite >> >>>> > >> >> >> >>>> > >> >> > > successful, I hope it's the same for others. Perhaps >> >>>> >> >>>> there are >> >>>> >> >>>> > >> >> better >> >>>> > >> >> >> >>>> > >> >> > > ideas >> >>>> > >> >> > > for its name: maven-aggregator >> >>>> > >> >> > > >> >>>> > >> >> > > Any other idea? any objection? >> >>>> > >> >> > > >> >>>> > >> >> > > Regards, >> >>>> > >> >> > > >> >>>> > >> >> > > Hervé >> >>>> > >> >> > > >> >>>> > >> >> > > [1] >> >>>> >> >>>> https://builds.apache.org/view/M-R/view/Maven/job/maven-box/ >> >>>> >> >>>> > >> >> > > [2] >> >>>> >> >>>> https://builds.apache.org/view/M-R/view/Maven/job/maven-wip/ >> >>>> >> >>>> > >> >> > > [3] >> >>>> > >> >> >>>> > >> https://svn.apache.org/viewvc/maven/sandbox/trunk/scripts/git/ >> >>>> > >> >> >>>> > >> >> > > [4] https://github.com/hboutemy/maven-aggregator >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
