First of all I need to state that I'm just an undergraduate student and don't speak for FreeType.
Running continuous integration on Savannah isn't really an option. > The regression server can poll Savannah every 10 minutes. IMHO it is good enough for a project that makes one commit a day on average. According to their docs, Savannah can send notifications about commits. The user can also start regression manually. Some commits don't need regression testing. > However it is not really practical for me to set up servers for your organization. But a detailed installation instruction would be nice to have. > "Improving the UI" is really vague It is convenient to have a dashboard that allows to compare any pair of revisions, watch the job queue, cancel and reorder jobs in the queue, and manage disk space. I'd also like an option to test a patch before it is committed. Doing it locally is not practical because few developers have access to all supported platforms and architectures. For instance, I don't have a big endian box or a box that enforces aligned access to memory. > ... switching out the comparison tool used in the scripts shouldn't be difficult. Of course it is not, but bmpcmp clips out unchanged part of the image and generates several layers for mouse-over action. So some changes are required. I'd also like to see more error checking in the scripts because every external program can fail for a variety of reasons. Direct diff of ftbench output does not make sense because small variations in the run time will create many false positives. It needs a custom processor. Most likely FreeType already has a collection of fonts for testing, but I cannot find any references. A lot of fonts, including some broken ones, can be extracted from PDF files. An extraction tool would be quite handy to have. Besides raster image comparison, there's Valgrind, Memendo, and Coverity that will surely find something to do for developers.
