Daniel Fagerstrom wrote:
IMO our huge community problem is what is most important. No, I'm certainly not talking about our excelent existing community, I'm talking about the fact that we only elected two new committers during 2004, compare that to around ten during 2003, and probably similar figures the years before that. Are our requirements on new committers to high? Maybe, maybe not, I think the standards have been high all the time.
Daniel is right in identifying a transition in our development model and I think I know why this is and has very little to do with block (even if that's important too).
Real story: a few days ago I was in a meeting, we were having a problem and we needed a solution. Fast. People suggested to hack a fake static page for a demo. We had 48 hours to show that. I said I could have done the whole thing for real. People laughted at me. [at this points, those who know me would have smiled]. I went home and I implemented the whole thing in cocoon in 4 hours, totalling 200 lines of XML. Next morning is up and running, jaws drop. (it's a transparent RDF-izing proxy for existing dspace installations... piece of cake with the TidyGenerator + XSLT)
But then I hit a wall: the TidyGenerator is not really a proxy, and the ProxyGenerator does not tidy. I ask around but nobody has something like this in place. All right: need to write the code.
I merged the two together generators together, an HTMLProxyGenerator. I wrote the whole thing without even trying it. Took me half an hour.
Eclipse says it compiles fine (that's the easy part now), but then you have to see it if really works.
Obviously, it doesn't.... the stacktrace is a million lines long... most of this code I don't recognize anymore, it's like looking at somebody else's project now and it's big and massive. And the logs were all fragmented (I fixed that :-)
So, all right, fix the code in eclipse, change to the terminal, run ant, hit cocoon-reload (twice! that stupid ehcache thing in the way), say "thanks sylvain" ten times (instead of a thousand, that's why you say that ;-), another stacktrace... go back to square one. One modification per minute... and I'm used to one modification every 5/10 seconds in debug mode.
Scripting made me hate java. It feels like driving a Ferrari stuck in first gear!!
I started in BASIC and that was interpretation, not scripting! (there was no library for anything!). Then I moved into compiled languages ever since, sometimes the compiler would compile it for me, sometimes the assembler, sometimes I would edit the machine code by hand!
Then the metal hit me hard (when the 3D cards killed my pure-assembly 3D engine dream and taught me a great lesson about how the moore law deals with early optimization) and I moved all the way to java, because it still felt like solid code.
My first experience with scripting was, in fact, JSPs.... I remember the feeling of hitting 'reload' on the browser and avoiding running the compiler... felt awesome: you could debug a servlet in seconds rather than minutes (at this point, though, the servlet engines didn't have wars and they were able to reload ther servlets)
That's what triggered me to write cocoon in the first place: I wanted to have the same thing for XSLT, instead of running the command line transformer. Cocoon 1.0 was born out of my lazyness of typing "lotusXSL" on the prompt!
Then the Cocoon 1.x reactor.
Then the Cocoon 2.x sitemaps.
Then XSP (for writing custom generators, mind you! not as a template language!)
[Never liked actions because you couldn't reload them on the fly, they always felt like a step backwards to me.... ]
Then sitemap + XSP + flowscript. [even without continuations, it would have been a major win for just the fast try/fail cycle!]
Then magic trio (sitemap+flowscript+cforms)
Then DHTML + magic trio.
Then DHTML + maci trio + python (cron jobs)
And now you ask me to write java code and take 60 seconds to see if anything I changed made a difference?
You gotta be kidding.
Scripting taught me (spoiled me?) with incredible high productivity but I think there is a problem when people are not able to go back and fix the underlying object model that the scripting is a glue for... because it forces people to hack things in their glue layer, which reduces reuse and *prevents* contributions to get back in the system!!!
For DHTML there is XHTML/XUL+CSS+Ecmascript+XPCOM and believe me, they work great, they are a wonderful platform to work with and our latest experiments prove that very well (see http://simile.mit.edu/piggy-bank). Linotype had a kick-ass, best-of-breed WYSIWYG DHTML editor, but I never was able to figure out how to modularize it enough for people to reuse (or include into cforms, which is probably the best solution).
For the sitemap, our object model is not super clean (actions should go, matchers and selectors should be unified, views should be unified with virtual pipeline components) but it's solid and well understood. I can live with the little nuances. The number of existing components is stunning and incredibly solid, the need to write custom pipeline components is very small... yet is that need emerges, people just prefer to hack around it in flow... but knowing how hacky their solutions are, they are never contributed back!
[sound of many people sighing around the virtual room]
The biggest and most continous patchflow over the last year came from the portal system and cform: those are the systems were the underlying infrastructure has to be able to support the load of the systems on top *and* the people working on it (Sylvain and Carsten) are the only few people left that have no problems in touching the core.
Everybody else, myself included, is either scared to touch anything anymore in the core, or too spoiled by the scripting hacks around it to do anything about it.
Scripting spoiled me so much (and my life habits changed too with my new job) that my attention span in coding is very much reduced: spending hours without seeing anything solidifying will very likely force me to leave it unfinished and uncommitted... then the next day something else comes along... and when I go back to it I forgot where I was (or the codebase would have already changed!) and I have to start over again!!!
I can't count how many times that happened!
When I think about implementing real blocks, I think about the *weeks* of not having anything visible and I never even start!! even if it might honestly just take a few days!
When Chris Oliver introduced the compiling classloader, I saw the light: that's what triggered me to work with Pier on the design of the new kernel with hot-swapping capabilities... too bad it turns out java is really not up to the task and we decided to remove that.
A private conversation about this triggered Sylvain to implement lazy loading to reduce that reload cycle (that gave him the Mr. Incredible look and his name on the "my hero" plate ;-) It helps, but it's not snappy enough, it feels like the good old compiled sitemap: clunky.
Another private conversation tried to convince Torsten of moving the compiling classloader into the pipeline system so that we could use pipeline components directly from java source code, effectively 'scripting' the pipelines... but I think Torsten is having too much fun with the continuation side of things to be concerned about this.
Another option was Ugo's implementation of the sitemap in Groovy (even if I think Groovy committed suicide by entering the JCP), but there was strong pushback on the idea of a unification of sitemap and flowscript (even if flowscript is being abused as a sitemap router more and more every day!) so I let that go.
I'm in *desperate* need to reduce that try/fail cycle in cocoon or I simply won't be able to touch its code anymore. Which is very frustrating, given the amount of things I could get done if I only could go back to it.
Moving the blocks out of the way and implementing a compile-time block deploying tool would do wonders in reducing the size of the codebase and making people think that cocoon is a more tractable problem... it might well increase the birth of an ecosystem of block creation (I really hope so!) but it won't solve the issue that cocoon *itself* is exhibiting: how scripting spoiled us.
Java is an incredibly verbose language, but Eclipse fixes that during writing and verbosity helps in reading other people's code, so it's not a bad language after all (and 1.5 will reduce a lot of that verbosity which is ever better)
Avalon is a very intrusive framework and a lot of the code feels like it's unnecessarely complex but its death allowed us to feel free about modifying our own container.. which is leading the innovations that Sylvain and Carsten introduced. This feels refreshing and has very nice promises for the evolution of our internal architecture (toward a simpler one, learning from what the other CoP containers did)
What we need is to help people write java code again and the only way to achieve this is by reducing the try/fail cycle.
If we don't do this, I fear that our layers and layers of scripting glue won't hold the pressure and collapse, leading to the need for smaller and more maintainable engines, eventually a code fork of the core. This would be very energy consuming, both technically and socially.
This is a wake up call people: our foundations are becoming old and are in danger of losing the knowledge that allows us to maintain it. (what happens if Sylvain and Carsten leave! think about it!)
An old foundation, no matter how solid at one point, becomes fragile and deteriorates if not actively maintained. We cannot allow to lose or deteriorate the collaborative knowledge on that, or we are doomed to fragmentation.
This is why I think we should make an effort to think about how we can solve this time/cognitive-energy discrepancy between the scripting and the java try/fail cycle.
And I suspect a lot can be done by just documenting how existing cocoon committers do their development today, not *on-top-of* cocoon, but *within* cocoon.
-- Stefano.
