I like the idea of going back to a single repo for core db features. I would like Fauxton to still be in its own repo. As someone who wrote some very basic erlang code for CouchDB recently. I found the multiple repos quite tricky to manage and I couldn't see how it made anything easier.
On Wed, Apr 13, 2016 at 8:35 AM, Alexander Shorin <[email protected]> wrote: > Hi Robert, > > Point about flattening to a single repository is valid: in the end, we > have our apps repos in broken state all the time as they are not > declare their decencies. So noone can pick fabric@master and run it - > he'll spend quite a lot of time to figure the deps of the right > versions. But the idea to solve the problem by reducing set of > repositories we have to test is good. > > > Hi Iliya, > > I have alternative solution for you: > > - Turn-off Travis CI everywhere where we cannot be sure about testing > without depended PRs (all except third-party modules, fauxton, docs, > and few more independent projects like couch-epi); > - Require everyone to submit additional PR to apache/couchdb repo with > commit hashes update; > - On this apache/couchdb PR we'll run CI testing; > - If you rebase/update any of your subcomponent PRs you must update > commit hash on apache/couchdb one; > > Pros: > - We won't forget to update rebar.config when new changes lands; > - We will always run complete integration testing with all the right > deps states; > - We won't have to invent any complicated integration solutions to > deal with sub-repos testing; > - No new new steps/files/work introduced, so there is no need to care > about learning curve; > > Cons: > - Need to be a bit tricky on Travis builder to realize on which remote > (fork) new rebar.config hashes are to correctly checkout them, though > that is not a rocket science since we have access to git information > there. > > The Jenkins CI role here is to ensure that we have master build right > and releases build right, on the various OSes. > > Sounds simpler and better for me, how it does for you? > > -- > ,,,^..^,,, > > > On Wed, Apr 13, 2016 at 12:37 AM, Robert Samuel Newson > <[email protected]> wrote: > > I'd like us to instead consider flattening to a single repository. I've > found no value and only pain from the multiple repositories approach (43 in > total!). > > > > The contention is that multiple repositories enforces application > boundaries (low coupling / high cohesion) but I've not felt that in > reality. We don't, and couldn't meaningfully, release any of our components > separately, and, as Ilya makes clear, many enhancements require changes to > multiple repositories, and we break this into multiple commits, losing the > ability to look at an enhancement in toto. > > > > If what Ilya is proposing is the solution, I think it's the solution to > a problem we should not have. > > > > B. > > > >> On 12 Apr 2016, at 16:22, Ilya Khlopotov <[email protected]> wrote: > >> > >> > >> > >> Dear community, > >> > >> > >> There is a problem with contributors workflow which renders our CI > system > >> useless. As you might know couchdb project consists of multiple > >> repositories. Most of the time changes cross the repositories > boundaries. > >> When this happens the push to any of the repositories causes CI > failures. > >> CI fails since it uses the old version of dependencies from main > repository > >> of the project. Here is what we can do about it. > >> > >> # Proposal > >> > >> Let's use multiple files for dependency management. > >> > >> - deps.json - serves the same purpose as dependencies list from current > >> rebar.config.script > >> - proposed.deps.json - here we specify list of PRs we want to commit > >> atomically > >> - override.deps.json - local file outside of version control which we > >> consult in order to include development tools specific to contributor > (code > >> reloader, debugger, tracer, profiler, binpp, ...) > >> > >> Bellow is the example of a content of these files: > >> > >> ## deps.json > >> { > >> "src/b64url": [ > >> "https://github.com/apache/couchdb-b64url";, > >> "6895652d80f95cdf04efb14625abed868998f174" > >> ], > >> "src/cassim": [ > >> "https://github.com/apache/couchdb-cassim";, > >> "9bbfe82125284fa7cb3317079e8bc1dc876a07bf" > >> ], > >> "src/chttpd": [ > >> "https://github.com/apache/couchdb-chttpd";, > >> "54e8f6147486d9afc5245e0143d15a4dd1185654" > >> ], > >> "src/meck": [ > >> "https://github.com/apache/couchdb-meck";, > >> "tree/0.8.2" > >> ], > >> .... > >> } > >> > >> ## proposed.deps.json > >> { > >> "src/couch": "https://github.com/apache/couchdb-couch/pull/124";, > >> "src/chttpd": "https://github.com/apache/couchdb-chttpd/pull/108"; > >> "src/couch_tests": [ > >> "https://github.com/apache/couchdb-erlang-tests";, > >> "tree/branch" > >> ], > >> } > >> > >> # Interface > >> > >> I propose to write a simple CLI tool to work with this structure. > Bellow is > >> a list of commands which we need to support (for minimal version) > >> > >> ## Adding new dependency > >> > >> git propose add https://github.com/apache/couchdb-foo > >> a2d5ad2eedc960248b806f61df0a1009462bdb46 > >> git propose add https://github.com/apache/couchdb-bar tree/branch_name > >> > >> ## Adding new PR to the change set > >> > >> git propose add https://github.com/apache/couchdb-config/pull/4 > >> > >> ## Checking out right dependencies > >> > >> git propose checkout > >> > >> ## Checking out release > >> > >> git propose checkout --release # this would ignore proposed.deps.json > if it > >> exists > >> > >> ## Merge the change > >> > >> This command would do the following: > >> - Parse proposed.deps.json > >> - Retrieve merge commit sha for every PR (exit if dependency is not > merged > >> yet) > >> - Update dependencies in deps.json with correct merge commit sha > >> - remove proposed.deps.json > >> > >> # Workflow > >> > >> export GIT_EXEC_PATH=`pwd`/bin # or use tools like `direnv` > >> git checkout -b feature-ZZZ > >> cd src/X && hack dependency X > >> cd ../.. > >> cd src/Y && hack dependency Y > >> issue PRs for X and Y > >> cd ../.. > >> git propose add https://github.com/apache/couchdb-X/pull/4 > >> git propose add https://github.com/apache/couchdb-Y/pull/49 > >> git add proposed.deps.json > >> git commit -m "Commit feature {something} which does {a thing} and can > be > >> tested as {procedure}" > >> git push origin feature-ZZZ > >> ^ this would trigger our CI > >> CI would do > >> git propose checkout && ./configure && make check > >> > >> # Pros and Cons > >> > >> ## Pros > >> > >> - Changes are merged atomically > >> - CI runs against expected versions of deps > >> - Enables git bisect > >> - Reduce tasks that needs to be done by ASF committer (no need to update > >> rebar.config.script manually) > >> - Simplifies testing of PRs by reviewers > >> - Simplifies rebar.config since rebar is not used for managing deps > >> > >> ## Cons > >> > >> - some github.com specifics (concept of PRs and access to github API > to get > >> info about PR) > >> - we need to have github.com as one of the remotes > >> - we trigger CI only on push to main repository > >> > >> # Implementation > >> > >> We write a git-propose script in python and place it in ./bin. We add > ./bin > >> into either GIT_EXEC_PATH or PATH. You always can call the script > directly > >> (as ./bin/git-propose) if you don't like amending > >> your environment. > >> > >> # Later improvements > >> > >> - We can issue PRs from the tool itself > >> - We can merge from the tool itself > >> - We can implement support for multiple remotes (asf, github, private) > >> - We can implement support for multiple git transports (for the first > >> version all repositories in *.deps.json files would use https://) > >> > >> Sincerely, > >> ILYA KHLOPOTOV > > >
