Remko, I am all for speeding up the build. I have been complaining about this for well over a year and it keeps getting worse with every release. The compile of log4j-core takes less than 20 seconds on the machine provided by my employer. On the same machine the tests take 11 minutes. On this machine the whole build takes about 16 minutes. This is a bit frustrating as I perform a complete build at least once before I commit 95% of my changes.
My personal machine, which is where I run the release builds, is over 6 years old now and is considerably slower. On that machine a “normal” build currently takes . In addition, the release build does perform more work and runs through the build twice (although I am not sure it runs the tests twice anymore) so it is considerably longer. Also with every release we add new unit tests. It wasn’t too long ago that I was marveling that we had over 900 unit tests. We now have over 2,400. In addition, each release also requires building the web site, which can take a long time as well. While I suspect about 1/3 of the time of a “normal” build could be cut by executing some stuff in parallel that “fix” will only last for a short while as we keep adding components related to third party projects (Mongo, Cassandra, DBCP2, etc) that most likely don’t change much once they are done. I can’t remember the tag library module required changes. Many of them only change when the third party project changes (such as Mongo 2 vs Mongo 3). On that note I really don’t want to see Mongo 2 and Mongo 3 projects as part of the Log4j main build. In addition to speeding up the build I would prefer to have the main log4j releases follow the 80/20 rule - they should contain only the stuff 80% of our users will use. I would venture a guess that at least 80% only use the File and Console appenders. While we may not have as firm of a plugin API as Maven does, almost everything a user should need to create an Appender, Layout, Lookup or Filter should be as backward compatible as the Log4j API. If we can’t guarantee backward compatibility for those components we have a big problem as users are creating these kinds of components all the time. We cannot be breaking them. So from that standpoint we should have no problem moving those kinds of plugins out of the main build. Ralph > On Jan 21, 2018, at 5:51 PM, Remko Popma <remko.po...@gmail.com> wrote: > > Log4j2 is not like Maven. Maven (I assume) has a plugin API. Log4j2 modules > depend on the internals of log4j-core. > > I agree with Gary that not being able to verify that a core change doesn’t > break any modules when they live outside the main project is a valid concern. > > Why are we talking about modules and repos when the issue is that it takes > too long to do a release? Maven reports most modules as taking a handful of > seconds to build. > > I keep thinking: > > 1. We’re running the tests twice during a release. Let’s change this. > Logically, once should be enough. What prevents us from doing this? > > 2. Core tests take unnecessarily long. By running some tests in parallel and > some other tests without forking I have hope we can run all core tests in ~15 > minutes. > > Won’t those two changes be much more impactful in reducing the release time > than any amount of module or repo reshuffling? > > Remko > > (Shameless plug) Every java main() method deserves http://picocli.info > >> On Jan 22, 2018, at 8:14, Ralph Goers <ralph.go...@dslextreme.com> wrote: >> >> I am very much against the idea of a single repo. Yes, you can have multiple >> projects in the repo but I am not sure how that can be sanely released. I >> much prefer the model that Maven has taken. They are now using gitbox [1] >> which seems to allow GitHub to be the primary repo. Every Maven plugin is >> individually released. Scroll down the link below to the Maven section and >> you can see all the plugin repos. >> >> The upside to this is that it are: >> 1. It is far easier to perform releases of the individual components. >> 2. It is much easier to accept plugin contributions. >> The downsides are: >> 1. A page like https://maven.apache.org/plugins/ >> <https://maven.apache.org/plugins/> is needed to keep track of the plugin >> versions. >> 2. It could make sense to have log4j-parent and log4j-bom projects. The >> first to help keep the builds similar and the second to help customers pick >> up the latest versions. >> >> Ralph >> >> [1] https://gitbox.apache.org/repos/asf <https://gitbox.apache.org/repos/asf> >> >>> On Jan 21, 2018, at 8:55 AM, Gary Gregory <garydgreg...@gmail.com> wrote: >>> >>> Hi All, >>> >>> I am writing this to in an effort to understand how we can manage and grow >>> Log4j. I use the term 'grow' not in the 'bigger is better' sense but more >>> in the maturing sense. >>> >>> I am prompted to write this by Ralph's comment that log4j-core and the main >>> repo is too big and that releases take too long make, partially prompted by >>> my addition of a new module called log4j-jdbc-dbcp2 which currently >>> contains one small class with a dependency on Apache Commons DBCP 2. >>> >>> I would like to express my gratitude at Ralph's efforts to revitalized >>> Log4j and performing most release management duties. Thank you Ralph! >>> >>> I can see two main orthogonal issues: >>> - The size of the git repo. >>> - The size of the log4j-core module. >>> >>> Ralph has proposed that new modules like log4j-jdbc-dbcp2 be moved to >>> another 'plugin' repo. >>> >>> I really dislike this idea: >>> - The plugin repo has never been released. Not that one releases a repo but >>> you get the point. >>> - How do you keep things in sync between repos and code when we have no >>> official 'core' SPI. >>> >>> For my money, we should keep _everything_ in one repo. Good enough for >>> Google, so good enough for me. What we release out of that repo is a >>> different story and what I would like to discuss next. >>> >>> This is not the same as Maven plugins IMO but the case could be made for it >>> I suppose where a lot/most plugins live in their own repos. It is not the >>> same as with Maven IMO because our plugins rely on log4j-core and it's >>> guts, for which we only make loose compatibility guarantees -- as opposed >>> to log4j-core where we are strict(er). Maven OTOH, has a API for plugin >>> auteurs. >>> >>> For example, log4j-jdbc-dbcp2 replies on the guts of log4j-core and we have >>> no 'official' core SPI, so splitting it off into a separate repo would >>> greatly increase the risk of it falling out of sync. It is just so much >>> more easier to maintain when it is all in one repo. >>> >>> My proposal is to: >>> >>> - Put everything back into one repo (Chainsaw too?) >>> - Define a core SPI for plugin writers where we make some statement about >>> BC, more than the casual 'we try not break stuff.' >>> - Defining what Log4j project 'components' we have and release based on >>> that. For example, today, all of the main Log4j repo is one component with >>> many modules. Chainsaw would be another component of the Log4j project. >>> Maybe we need to redefine components: API, File, JDBC and so on. A >>> component can have one of more module >>> - Got for there. >>> >>> Thoughts? Help me flush this out? >>> >>> Thank you for reading! >>> Gary >> >
