[Note: This message was provided to Parrot Developer's Summit participants near the start of Saturday's summit meeting; I'm reposting it now to parrot-dev so we'll have a more permanent archive of the message than the gist nopaste that was used. Comments and feedback on this article are of course welcome. --Pm]
As a result of the last PDS in January, I sent out an email identifying what Rakudo needs from Parrot in 2011 [1]. This is an interim update to give my views on where Parrot has improved or not improved for Rakudo since that posting. The posting identified five things that Rakudo needed from Parrot: 0) anything that makes Parrot/Rakudo faster overall 1) improved garbage collection 2) fixes to profiling tools and documentation 3) serialization 4) create .pbc directly from a Parrot program Since that posting, excellent progress has been made on garbage collection (#1 above) -- commendations to all involved, especially bacek++. However, recent tests and benchmarks have shown that the primary motivation for the better GC -- that is, increased speed (#0) -- hasn't materialized on Parrot yet. I discovered this when creating the 2011.04 Rakudo Star release -- based on messages to the mailing lists I was expecting around a 20% performance improvement in Rakudo and Parrot; instead I found a slight degradation in performance on nearly all of the machines I tested. Since Rakudo has been relatively stable over this period of time, most (but certainly not all) of the variation in performance would seem to be attributable to changes in underlying Parrot performance. As a result of the above I've created a Rakudo benchmarking platform [2] that makes it easier to evaluate different Rakudo/Parrot configurations over time. The latest results (from this morning) can be seen at https://github.com/pmichaud/rakbench/blob/master/results/kiwi-x86_64-201105140653.txt https://github.com/pmichaud/rakbench/blob/master/results/plum-x86_64-201105140651.txt https://github.com/pmichaud/rakbench/blob/master/results/orange-x86_64-201105140653.txt What these results seem to show is that the new GC only helps in those places where GC was particularly problematic to begin with, such as the regex tests. In most other cases, released Parrots since January have performed worse (and sometimes *much* worse) than the version released in January of this year. Indeed, it was these bencmarks that pointed out some problems that ultimately led to the fixes that show the performance gains for rakudo-master and rakudo-bleed on some platforms. Bottom line: In spite of the new GC, overall performance has not really improved since January. The new rakbench suite underscores just how important it is that Parrot get some reliable profiling tools in place (#2). The ones that exist now aren't at all reliable, as detailed in my January message. Yesterday I inquired on #parrot about the state of profiling, and was told that since there were no GSOC takers for this task that Parrot will have to formulate a plan to work on them. From Rakudo's perspective, this continues to be an immediate and dire need, so I hope Parrot will do so. Yesterday evening I hacked together some very Rakudo-specific quick-and-dirty PIR-level-sub call count capabilities, and was able to almost instantly identify a 10% overall performance improvement for Rakudo by replacing a frequently-called 1-line method with a much faster 3-line version. Even better improvements should be forthcoming with more investigation. Given that there seems to be little movement on profiling on Parrot, we'll continue to develop the Rakudo-specific version and also likely instrumenting the new NQP with similar capabilities. But Parrot ought to be able to do a far richer form of sub-level profiling than what we can do, and when it's available we'll immediately be able to take advantage of it. Creating .pbc directly from a Parrot program (#4) is being worked on satisfactorily, and although I haven't seen any direct effort towards serialization (#3), I tend to view that a longer-term goal rather than something that is likely to happen immediately. Indeed, it should be restated that all of the items in my January message were being presented with "we'll be glad to see them whenever they can be made available" and no real expectations of delivery dates. That continues to the be the case-- anything Parrot can do to improve the items listed above will be help Rakudo immensely. It also continues to be the case that overall speed is critically important for us, and that thus far Parrot releases have trended in the wrong direction on that front. [Post-PDS note: The items described below were discussed extensively during PDS, and have led to new policies to establish 'relationship manager' roles to manage them. See [3] for the followup.] Lastly, there continues to be a great deal of friction between the Parrot and Rakudo teams regarding stability and deprecation. I don't have any direct answers. I do know that I personally (and several of the other Rakudo team members) are weary of all of the revisiting (I call it "re-litigating") of design, implementation, and process decisions that were supposedly "settled" by discussions long ago and even not-so-long ago. I can provide a list of detailed and recent examples of such "revisits" if we wish to analyze this further. As I've repeatedly stated in the past -- Rakudo developers know that breakages will occur, and that's overall okay with us. What hurts is that when we report the breakages, either after they occur or because we note that a proposed change to Parrot will cause us a serious issue, we end up spending hours and hours of collective discussion time trying to _justify_ to a few Parrot developers why the breakage is painful and unwarranted before we actually get someone in Parrot with sufficient authority/skill to relieve our pain. We don't mind the existence of breakages, we mind the absolutely herculean efforts we have to go through with various Parrot developers to get them to stop denying there's an issue (or that they've already addressed it) so that a mutual solution can be found. Anything that can be done to alleviate this ongoing problem would be a big help. [Post-PDS note: See [3] to see what was decided on this issue.] Pm [1] http://lists.parrot.org/pipermail/parrot-dev/2011-January/005410.html [2] http://github.com/pmichaud/rakbench [3] http://lists.parrot.org/pipermail/parrot-dev/2011-May/005891.html _______________________________________________ http://lists.parrot.org/mailman/listinfo/parrot-dev
