thank you Plamen: this script is really awesome! I reworked it a little bit and just added it to our svn git migration scripts area [1]
And I just pushed the result on maven-acr-plugin: you can see the result live. As you can see, the tags on GitBox [2] are updated but not the tags on GitHub [3] even if I tried to force to GitHub (and it looked ok) I don't know if it's a major issue The rework of tags is ok for 420 tags on plugins, and fails for 31: - 16 tags don't point to a release plugin commit [4] - 15 tags have an issue I don't really understand yet [5] I'll still investigate on these 31 issues to see if some can be fixed easily Then we'll have to decide if we push these new tags on plugins (with rewrite for some). And same will have to be done on shared, and on skins (I checked: same branches issues that were not seen before...) Once again, thank you Plamen for your help: without it, we would have been in a hard situation Regards, Hervé [1] http://svn.apache.org/r1819632 [2] https://gitbox.apache.org/repos/asf?p=maven-acr-plugin.git;a=tags [3] https://github.com/apache/maven-acr-plugin/tree/maven-acr-plugin-3.0.0 [4] maven-assembly-plugin-2.2-beta-4 maven-assembly-plugin-2.6 maven-checkstyle-plugin-2.11 maven-compiler-plugin-2.0.1 maven-dependency-plugin-2.0-alpha-1-RC2 maven-dependency-plugin-2.0-alpha-2 maven-help-plugin-2.0.1 maven-project-info-reports-plugin-mvn%20release%3Aprepare maven-shade-plugin-1.0-alpha-13 maven-shade-plugin-1.0-alpha-14 maven-shade-plugin-1.0-alpha-15 maven-site-plugin-2.4 maven-site-plugin-3.0-beta-1 maven-site-plugin-3.0-beta-2 maven-site-plugin-3.0-beta-3 maven-site-plugin-pre-compat-with-doxia-1.0-alpha-7 [5] maven-antrun-plugin-1.5 - the commit created by the release plug-in 651052412a4b31b045974677d2f9030d9bd076ba does not have the same files as the tag 6dcdb8f1a8c53dfb8dfeaf12280c0995c6022502. Skipping. maven-assembly-plugin-2.1 - the commit created by the release plug-in f2dec7adec8255cb6f5919bcc098a31d1e07a46b does not have the same files as the tag fcbde17e04c600b6ad7b397b6bd2994650f0abf5. Skipping. maven-assembly-plugin-2.2-beta-1 - the commit created by the release plug-in a2622c8e159b7cac653c19ee372c396777da8773 does not have the same files as the tag 2e96765a965fce5e600f598171dcfefd8770946a. Skipping. maven-clean-plugin-2.3 - the commit created by the release plug-in a2397c4cb79bce5854571053e5e6bb57f811fe5b does not have the same files as the tag 83b55fa0efd197eda3ccf9d41f0e658427de16bf. Skipping. maven-dependency-plugin-2.0 - the commit created by the release plug-in 43a01f5f9d0a989560349271d9081fba402ed212 does not have the same files as the tag 435ab7df24f52e1a2be6b0718dbb6a6c98452deb. Skipping. maven-dependency-plugin-2.0-alpha-1 - the commit created by the release plug- in a08d65e68df737f88f532a4c828715249c30b57e does not have the same files as the tag e2a15d679e8e0c46ac0a4525ab0127d58dd89e16. Skipping. maven-dependency-plugin-2.1 - the commit created by the release plug-in b9cbabc0be2d44fa05c4cff2cdb10300cfe1c3b1 does not have the same files as the tag 6c71a9718c6ce879d5b64b1b5d08f5419e24e615. Skipping. maven-help-plugin-2.0 - the commit created by the release plug-in 277dbda0f10a298c6d72939a23fd6d36c04c837b does not have the same files as the tag 6f360e3bd621a9acd49d284006440df563eb56d7. Skipping. maven-invoker-plugin-1.8.1 - the commit created by the release plug-in 33bae0bd6a5ba92215b7b85fd82d0c68fb00bc8e does not have the same files as the tag b75b274d03c64dc9e9c164c118a2a25a0446b100. Skipping. maven-javadoc-plugin-2.2 - the commit created by the release plug-in 8a4d608571795131f7779cae1ec9dc14311b36b9 does not have the same files as the tag e2947944cd89055e90fb57b80a84697b83d55c5b. Skipping. maven-rar-plugin-2.1 - the commit created by the release plug-in ab2d136deee61dc947f6e9bbf5d1d43737604db6 does not have the same files as the tag b246cb558a94d3ee85da17a728e60590a7776e0e. Skipping. maven-remote-resources-plugin-1.0-alpha-6 - the commit created by the release plug-in ca0546b61541067983bd819350a13bad25592d72 does not have the same files as the tag 81943b766888c1902108dec11ace7c421360e3f9. Skipping. maven-shade-plugin-1.0 - the commit created by the release plug-in 0bbbcf18e5428c63dd0590a1a2435b62175e19cd does not have the same files as the tag b2d808a94f1bee635de1b32da2605d3a0b943c03. Skipping. maven-shade-plugin-2.4.3 - the commit created by the release plug-in a4a4bc95a61aee9e70e7d44ae140203440de64de does not have the same files as the tag e11dd3982249a452b1c1ec1ad3ccc2a2428f860f. Skipping. maven-source-plugin-2.0.2 - the commit created by the release plug-in 6dfc44b1facce54cdccbea44fea26db0967e6f37 does not have the same files as the tag 5029522716b5a886155967d9d4ca0e9aec363876. Skipping. Le vendredi 29 décembre 2017, 10:26:17 CET Plamen Totev a écrit : > Hi, > > I've created the script. Not sure how to share it with the list so > I've created a gist - > https://gist.github.com/plamentotev/b835dd62c74a3a5becd4c317b97403a4 > > It will migrate the tags for all repositories in plugins/maven-* Also > it checks for the 'git-svn' string in the commit message so it will > not migrate tags created after the git migration. > And I have commented the `git tag` command so by default it will be > run in "dry mode" that will print all errors found without changing > the repositories. > > Speaking of errors - there are about 30 tags that could not be > automatically migrated. They are actually two group of errors: > > * the "prepare release" commit does not have the same files as the tag > - meaning that most likely the tag in SVN is not just a copy of the > "prepare release" commit and contains some changes made after that > * there is no "prepare release" commit reachable from master. That > could mean that the tag is not created by the release plugin and > sometimes that is the case, but it looks like the majority of the > cases are because the release is created from branch and not from > trunk. And while we're on the topic - looks like the branches are not > migrated into the maven-plugins repository. > > Also as the script modifies the tag I fell obligated to mention the > usual warning when changing tags - > https://git-scm.com/docs/git-tag#_on_backdating_tags > > Regards, > Plamen Totev > > On Thu, Dec 28, 2017 at 10:43 AM, Hervé BOUTEMY <[email protected]> wrote: > > Hi Plamen, > > > > Thank you for your help > > > > Yes, I looked more in depth to maven-plugins svn2git mirror and the split > > result (got from the split script): the issue with tags is already there > > > > I like the idea of recreating tags from "[maven-release-plugin] prepare > > release <name of the tag>" commit, ignoring the "[maven-release-plugin] > > copy for tag <name of the tag>" commit that don't change anything in code > > > > If you can prepare a script to create the tags with this convention, it > > would really be useful (don't forget that sometimes there are multiple > > tries at one release, then the useful tag is the last one in time) > > > > Regards, > > > > Hervé > > > > Le lundi 25 décembre 2017, 09:58:36 CET Plamen Totev a écrit : > >> Hi, > >> > >> What I can see is that the tags actually introduce new branches that > >> contain commits with the same content as the one in the master (trunk) > >> branch. But this is not caused by the script that splits of the > >> repository - the problem exists in the maven-plugins Git repository as > >> well. Maybe it's not quite visible because there are a lot of commits, > >> but it's there. > >> > >> How can it be fixed? I think it would be easier to fix the individual > >> plugin repositories (after the split) as they are smaller and easier > >> to work with. The plan is simple - as the branches created by the tags > >> does not contain new commits(with a single exception but I'll explain > >> that in a minute) and all of them are present in the master branch we > >> could just change the tag to point to the same commits but in the > >> master branch. That will cause all extra commits and branches to be > >> garbage collected as they are accessible only from the tags. For our > >> purposes commits are equal if they point to the same root tree (the > >> same files with the same content). But unfortunately there is a catch. > >> What I said about the branches created by the tags is not strictly > >> true - they do have some extra commits. When you release in SVN you > >> have the following sequence of commits: > >> > >> * [maven-release-plugin] copy for tag maven-war-plugin-3.2.0 (this is > >> the tagged commit) > >> * [maven-release-plugin] prepare release maven-war-plugin-3.2.0 > >> > >> After the migration to Git the master (trunk) branch contain only the > >> 'prepare release' commit but not the 'copy for tag' commit. So if we > >> apply my plan then the 'copy for tag' commit will be lost. It's just a > >> copy (no changes, it contains the same files as the previous commit) > >> so I don't think it's a problem but would be nice if somebody > >> confirms. > >> > >> So the proposed plan have the following caveats: > >> > >> * the 'copy for tag' commits will be lost. No changes will be lost as > >> those are only copy commits > >> * the tags will point to the previous 'prepare release' commits > >> * the tags SHA will be changed because they point to a different commit > >> > >> If that is OK I can write a script that will apply those changes. > >> > >> What causes the branches and all those duplicating commits? Well the > >> master branch tracks > >> https://svn.apache.org/repos/asf/maven/components/trunk/maven-plugins > >> but the tags for the plugins are actually tracking the plugin > >> directory (for example: > >> https://svn.apache.org/repos/asf/maven/components/trunk/maven-plugins/mav > >> en-> war-plugin). So for maven-war-plugins you have two commits in the > >> maven-plugins Git repository - one for > >> https://svn.apache.org/repos/asf/maven/components/trunk/maven-plugins/mav > >> en-> war-plugin and one for > >> https://svn.apache.org/repos/asf/maven/components/trunk/ no matter that > >> actually those commits are the same - same date, > >> author, message, files, etc. > >> > >> Regards, > >> Plamen Totev > >> > >> On Sun, Dec 24, 2017 at 11:54 AM, <[email protected]> wrote: > >> > I'd suggest to try the process to a personal personal repo on GitHub to > >> > see if you're able to get a better result before involving manual work > >> > from INFRA (on more than 60 repos...) > >> > > >> > (it's sad to see nobody try to explain what's happenning or improve the > >> > documented commands, just get to a conclusion: re-do everything and > >> > pray) > >> > > >> > Regards, > >> > > >> > Hervé > >> > > >> > ----- Mail original ----- > >> > De: "Karl Heinz Marbaise" <[email protected]> > >> > À: "Maven Developers List" <[email protected]>, "Robert Scholte" > >> > <[email protected]> Envoyé: Dimanche 24 Décembre 2017 10:47:43 > >> > Objet: Re: [IMPORTANT] Re: Git migration next steps > >> > > >> > Hi, > >> > > >> > On 24/12/17 10:40, Robert Scholte wrote: > >> >> How about a hard reset or dropping the repo and doing it all over > >> >> again? > >> > > >> > If I correctly seen that ..there had no commit yet on the new git > >> > repos.. > >> > > >> > So I think it would be the easiest way to do as Robert suggest ...to > >> > redo migration for those repos.. > >> > > >> > Kind regards > >> > Karl Heinz > >> > > >> >> On Wed, 20 Dec 2017 10:42:36 +0100, Hervé BOUTEMY > >> >> > >> >> <[email protected]> wrote: > >> >>> INFRA-15679 fixed by infra team > >> >>> then I re-run migrate-plugins.sh script to split the svn2git mirror > >> >>> to > >> >>> per- > >> >>> plugin git repo > >> >>> and I pushed "master2" branches for m-javadoc-p, m-site-p and > >> >>> m-pdf-p, > >> >>> which > >> >>> were the 3 plugins which suffered from missing commits > >> >>> > >> >>> on m-site-p and m-pdf-p, I'll cherry pick the unique commit that was > >> >>> missed: > >> >>> not a big deal > >> >>> > >> >>> on m-javadoc-p, the situation is more coplex, since there was a > >> >>> release > >> >>> > >> >>> I also noticed that I forgot to push tags when importing: I started > >> >>> to > >> >>> do "git > >> >>> push --tags", but the result does not look as I expected: it creates > >> >>> a > >> >>> lot of > >> >>> parallel branches > >> >>> > >> >>> I'll need help from git experts: what is happening? > >> >>> > >> >>> I stopped the tags push half the way, we'll need to decide what to > >> >>> do... > >> >>> (I knew I was not a git expert and there was a risk for something > >> >>> weird like > >> >>> what's currently happening...) > >> >>> > >> >>> Any hint? > >> >>> > >> >>> Regards, > >> >>> > >> >>> Hervé > >> >>> > >> >>> Le samedi 16 décembre 2017, 16:28:48 CET Hervé BOUTEMY a écrit : > >> >>>> ok, I was confused by the different takes at m-javadoc-p 3.0.0 > >> >>>> > >> >>>> yes, svn2git mirror is stuck [1] at r1815675 > >> >>>> > >> >>>> I just opened an INFRA Jira issue > >> >>>> https://issues.apache.org/jira/browse/INFRA-15679 > >> >>>> > >> >>>> once the svn2git mirror will be updated, we'll have to re-run the > >> >>>> split > >> >>>> scripts and cherry pick the missing commits > >> >>>> > >> >>>> Regards, > >> >>>> > >> >>>> Hervé > >> >>>> > >> >>>> [1] https://github.com/apache/maven-plugins/commits/trunk > >> >>>> > >> >>>> Le samedi 16 décembre 2017, 13:01:05 CET Robert Scholte a écrit : > >> >>>> > I was triggered by some failing unit tests, which should have been > >> >>>> > >> >>>> solved > >> >>>> > >> >>>> > in maven-javadoc-plugin-3.0.0 > >> >>>> > > >> >>>> > My last commit according to GIT was november 18th > >> >>>> > My last commit according to SVN was december 3rd > >> >>>> > > >> >>>> > comparing svnlog with gitlog most of these commits are lost: > >> >>>> > > >> >>>> > moved to git > >> >>>> > ---- > >> >>>> > [maven-release-plugin] prepare for next development iteration > >> >>>> > ---- > >> >>>> > [maven-release-plugin] prepare release maven-javadoc-plugin-3.0.0 > >> >>>> > ---- > >> >>>> > [MJAVADOC-498] "module not found" when Java 9 module-info present > >> >>>> > Support aggrated javadoc > >> >>>> > ---- > >> >>>> > Skip several unittests for Java9 > >> >>>> > ---- > >> >>>> > JDK-8032205 was closed as not an issue, so not solved in Java9. > >> >>>> > Need to review the conclusion > >> >>>> > ---- > >> >>>> > Upgrade mockito to remove warning about illegal reflective access > >> >>>> > ---- > >> >>>> > Improve TestJavadocReportTest#testTestJavadoc > >> >>>> > J8 warns and continues with missing dependency, J9 fails. > >> >>>> > In fact test was wrong: dependency should have been on classpath > >> >>>> > ---- > >> >>>> > unittest should prefer JAVA_HOME when executing from cmdline > >> >>>> > When running with Java9+ no need to switch from jre to jdk > >> >>>> > directory > >> >>>> > (jep220) > >> >>>> > ---- > >> >>>> > MJAVADOC-502 Update DEFAULT_JAVA_API_LINKS > >> >>>> > ---- > >> >>>> > session is required parameter, so cannot be null. Fix related > >> >>>> > >> >>>> unittests > >> >>>> > >> >>>> > ---- > >> >>>> > Add project/artifact key to set of sourcePaths to recognize > >> >>>> > reactor > >> >>>> > projects versus dependencies > >> >>>> > ---- > >> >>>> > Group sets of sourcepaths per project, in prepare of usage of > >> >>>> > module-source-path. > >> >>>> > ---- > >> >>>> > Switch from List to Collection to make it easier to use Sets when > >> >>>> > preferred > >> >>>> > ---- > >> >>>> > [maven-release-plugin] prepare for next development iteration > >> >>>> > ---- > >> >>>> > [maven-release-plugin] prepare release maven-javadoc-plugin-3.0.0 > >> >>>> > ---- > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > On Sat, 16 Dec 2017 12:53:23 +0100, Hervé BOUTEMY > >> >>>> > >> >>>> <[email protected]> > >> >>>> > >> >>>> > wrote: > >> >>>> > > looking at commits@ content https://lists.apache.org/list.html? > >> >>>> > > [email protected] with subject containing > >> >>>> > >> >>>> "maven/plugins/trunk" > >> >>>> > >> >>>> > > and plugins svn2git mirror > >> >>>> > > https://github.com/apache/maven-plugins/commits/ > >> >>>> > > trunk > >> >>>> > > > >> >>>> > > only 1 commit is missing: my latest commit on maven-site-plugin > >> >>>> > > (the last commit for Git migration is not useful) > >> >>>> > > > >> >>>> > > > >> >>>> > > Same on shared showed no missing commit. > >> >>>> > > > >> >>>> > > > >> >>>> > > what latest commit of maven-javadoc-plugin are you looking for? > >> >>>> > > > >> >>>> > > Regards, > >> >>>> > > > >> >>>> > > Hervé > >> >>>> > > > >> >>>> > > Le samedi 16 décembre 2017, 11:56:31 CET Robert Scholte a écrit : > >> >>>> > >> For everybody just a warning I faced today: > >> >>>> > >> If you switch to the git repos, please make sure all commits > >> >>>> > >> are > >> >>>> > >> migrated. > >> >>>> > >> I noticed the latest commits of the maven-javadoc-plugin got > >> >>>> > >> lost. > >> >>>> > >> > >> >>>> > >> thanks, > >> >>>> > >> Robert > >> >>>> > >> > >> >>>> > >> On Sat, 09 Dec 2017 17:06:09 +0100, Stephen Connolly > >> >>>> > >> > >> >>>> > >> <[email protected]> wrote: > >> >>>> > >> > I see we have a large number of repos now on gitbox ;-) > >> >>>> > >> > > >> >>>> > >> > On Thu 7 Dec 2017 at 07:00, Hervé BOUTEMY > >> >>>> > >> > <[email protected]> > >> >>>> > >> > >> >>>> > >> wrote: > >> >>>> > >> >> ok, I didn't update my repo clone: now the run-its profile > >> >>>> > >> >> is > >> >>>> > >> > >> >>>> > >> activated > >> >>>> > >> > >> >>>> > >> >> then the plan should just confirm "it works!" :) > >> >>>> > >> >> > >> >>>> > >> >> and find which jobs are special, like maven-dist-tool (which > >> >>>> > >> >>>> has to > >> >>>> > >> >>>> > >> be > >> >>>> > >> > >> >>>> > >> >> scheduled daily instead of code change, and one platform > >> >>>> > >> >> only) > >> >>>> > >> >> > >> >>>> > >> >> Regards, > >> >>>> > >> >> > >> >>>> > >> >> Hervé > >> >>>> > >> >> > >> >>>> > >> >> Le mercredi 6 décembre 2017, 23:58:45 CET Stephen Connolly a > >> >>>> > >> >>>> écrit : > >> >>>> > >> >> > On Wed 6 Dec 2017 at 22:38, Hervé BOUTEMY > >> >>>> > >> >>>> <[email protected]> > >> >>>> > >> >>>> > >> >> wrote: > >> >>>> > >> >> > > Now that we have 2 ASF Organization Jenkins jobs (one > >> >>>> > >> >> > > for > >> >>>> > >> >>>> gitbox > >> >>>> > >> >>>> > >> [1] > >> >>>> > >> > >> >>>> > >> >> and > >> >>>> > >> >> > >> >>>> > >> >> > > one > >> >>>> > >> >> > > for git-wip: thank you Stephen) and that it looks quite > >> >>>> > >> > >> >>>> > >> successful, > >> >>>> > >> > >> >>>> > >> >> let's > >> >>>> > >> >> > >> >>>> > >> >> > > plan > >> >>>> > >> >> > > the next steps. > >> >>>> > >> >> > > > >> >>>> > >> >> > > Here is what I see: > >> >>>> > >> >> > > 1. removal of hand-defined Jenkins jobs that are now > >> >>>> > >> >>>> duplicates > >> >>>> > >> >>>> > >> >> > > 2. preparation of the 60 new empty git repos for shared > >> >>>> > >> >> > > & > >> >>>> > >> >> > > plugins > >> >>>> > >> >> > > > >> >>>> > >> >> > > 3. migration of the 1 shared component and 1 plugin > >> >>>> > >> >> > > using > >> >>>> > >> >> > >> >>>> > >> >> migrate-*.sh > >> >>>> > >> >> > >> >>>> > >> >> > > scripts > >> >>>> > >> >> > > [3] to test and eventually rework the Jenkinsfile (I > >> >>>> > >> >>>> suppose it > >> >>>> > >> >>>> > >> will > >> >>>> > >> > >> >>>> > >> >> > > require > >> >>>> > >> >> > > some little change, to run add "run-its" profile) > >> >>>> > >> >> > > >> >>>> > >> >> > As far as I recall, I added -P+run-its already > >> >>>> > >> >> > > >> >>>> > >> >> > For the plugin, I'd like to do the job for > >> >>>> > >> >> > maven-site-plugin, > >> >>>> > >> > >> >>>> > >> since we > >> >>>> > >> > >> >>>> > >> >> > > expect > >> >>>> > >> >> > > to release it soon. > >> >>>> > >> >> > > For the shared component, I don't know if there is a > >> >>>> > >> >> > > best > >> >>>> > >> > >> >>>> > >> candidate > >> >>>> > >> > >> >>>> > >> >> > > 4. once previous step is ok, do the full migration: if > >> >>>> > >> >>>> there are > >> >>>> > >> >>>> > >> >> > > volunteers > >> >>>> > >> >> > > for helping, that would be great, since populating 60 > >> >>>> > >> >> > > git > >> >>>> > >> >>>> repos > >> >>>> > >> >>>> > >> >> won't > >> >>>> > >> >> be > >> >>>> > >> >> > >> >>>> > >> >> > > really fun... > >> >>>> > >> >> > > > >> >>>> > >> >> > > And as part of 60 empty git repos creation, I propose to > >> >>>> > >> >>>> migrate > >> >>>> > >> >>>> > >> the > >> >>>> > >> > >> >>>> > >> >> > > "Google > >> >>>> > >> >> > > repo manifest" maven-aggregator [4] to ASF: my personal > >> >>>> > >> >>>> use has > >> >>>> > >> >>>> > >> been > >> >>>> > >> > >> >>>> > >> >> quite > >> >>>> > >> >> > >> >>>> > >> >> > > successful, I hope it's the same for others. Perhaps > >> >>>> > >> >>>> there are > >> >>>> > >> >>>> > >> >> better > >> >>>> > >> >> > >> >>>> > >> >> > > ideas > >> >>>> > >> >> > > for its name: maven-aggregator > >> >>>> > >> >> > > > >> >>>> > >> >> > > Any other idea? any objection? > >> >>>> > >> >> > > > >> >>>> > >> >> > > Regards, > >> >>>> > >> >> > > > >> >>>> > >> >> > > Hervé > >> >>>> > >> >> > > > >> >>>> > >> >> > > [1] > >> >>>> > >> >>>> https://builds.apache.org/view/M-R/view/Maven/job/maven-box/ > >> >>>> > >> >>>> > >> >> > > [2] > >> >>>> > >> >>>> https://builds.apache.org/view/M-R/view/Maven/job/maven-wip/ > >> >>>> > >> >>>> > >> >> > > [3] > >> >>>> > >> > >> >>>> > >> https://svn.apache.org/viewvc/maven/sandbox/trunk/scripts/git/ > >> >>>> > >> > >> >>>> > >> >> > > [4] https://github.com/hboutemy/maven-aggregator > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: [email protected] > >> > For additional commands, e-mail: [email protected] > >> > > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: [email protected] > >> > For additional commands, e-mail: [email protected] > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
