Greetings everyone! I participated in the Google Summer of Code 2025 program with Debian and want to share a recap of what I did this summer. The program officially spanned June to August, and I have been actively contributing to Salsa CI since April. Throughout the summer, I dedicated 450+ hours to development, testing, and collaboration. My main achievement was creating the Salsa Status page for visualizing Salsa CI statistics, which is a result of 160+ commits and 9,500+ lines of code. I also contributed to Salsa CI in various areas, enhancing pipeline speed (caching, auto-cancel) and fixing several issues, and submitted 17 MRs. Overall, the summer was very productive, and I’m proud of all that I accomplished.
## Salsa CI statistics (salsa-status.debian.net) ------------------------------------------------------------- The status page was my most important goal and took 3 weeks of planning and prototyping, and most of my time went toward it throughout the summer. It provides both Salsa CI users and the Salsa CI developer team with visibility into CI performance, helping identify wasteful practices, broken configurations, and optimization opportunities across the entire Debian package ecosystem. In the last 15 days alone, we have recorded 8500+ pipelines across 3500+ projects that have consumed 250,000+ CI minutes. We also spotted 55+ projects that haven’t been updated in the last 2 or more years, but still run daily/weekly scheduled pipelines that result in large amounts of traffic and unnecessary wastage of resources. We will be reaching out to them soon to resolve this. **Features**: The status page aggregates and presents pipeline and job statistics, success rates, performance trends, and detailed CI history with insights across all projects that use Salsa CI. It tracks CI duration trends and resource consumption patterns to optimize performance and identify bottlenecks. The aggregated stats are presented in an organized way and visualized using various charts. Automated notifications for performance degradation are sent on the Matrix room to stay informed of regression and dropped success rate of jobs in real-time. **Benefits to Salsa/Debian**: Among several benefits that can’t be listed here, the status page helps in early detection of failures, controlled feature rollouts, and post-merge regressions, all while identifying resource wastage of CI resources (multiple long runs, high failure rates). It helps with focused probing of issues using filters and allows developers to compare their project trends over time. The automated Matrix alerts help bring regressions to light in time. **The backend**: Located at /api/v1, it provides an API and is responsible for fetching, storing, and serving the data to the frontend. The pipelines using Salsa CI make an API call with pipeline identifiers when the `SALSA_CI_ENABLE_STATS` flag is on. The backend registers the pipeline and inserts their stats and final status into the database with their jobs, average time, result, project details, maintainer, etc. I have also implemented cron jobs to aggregate and insert hourly and daily stats for the pipelines and jobs in the database. This helps serve them quickly and saves the database from repeated strenuous calculations. Apart from this, we have cron jobs for MRs to keep track of whether any merge or feature rollout on Salsa CI pipeline resulted in regression. The backend sends Matrix alerts about regressions to https://matrix.to/#/#salsa-stats:matrix.org **The frontend**: Built using React, and the design is inspired from Dribbble. It was a challenge to make the design look good yet keep the implementation simple and easy to understand, but I feel I did a good job. The status page contains mainly 4 pages: Home Page - https://salsa-status.debian.net - Header shows global stats for all-time pipelines, jobs, projects, and CI time. - Interactive line chart with customizable durations (24h, 7d, 30d, 6m, 1y) visualizing total, passed, and failed pipelines with switchable success/failure rate mode. - MR avatar markers on chart lines (hover for MR details). - Job Types table with descriptions of jobs and 30-day success rate sparklines. Pipelines Page - https://salsa-status.debian.net/pipelines - Trend line chart and real-time pipeline grid. - Pipeline cards show ID, Gitlab-style status, timestamps, project details, and Salsa link. - Sortable by status/time/duration and filterable by status, duration ranges, and date ranges. Job Types Page - https://salsa-status.debian.net/job-types - Treemap showing job type quantities and success rates by stages (provisioning, build, test). - Grid view with descriptions, sparklines, and 30-day stats (runs, CI minutes, success rate, average duration). - Clicking job types shows a detailed page with line chart and jobs grid. Projects Page - https://salsa-status.debian.net/projects - Grid view of all registered projects using Salsa CI. - Cards show version (from pushed tags), last activity, maintainer info (from d/control), and last 30 days stats. - WIP: Project's CI config page. The dashboard was initially built to be hosted on Gitlab pages and backend on Render (render.com), but thanks to Otto, both frontend and backend are now at salsa-status.debian.net. The name was kept generic intentionally so that in the future, we may add more stats about Salsa in general and not just be limited to Salsa CI. The MRs on Salsa CI for enabling statistics: !622: Enable optional pipeline statistics reporting to Salsa CI Dashboard for public projects https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/622 !638: Update dashboard statistics API URL to salsa-status.debian.net https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/638 Perfecting it took time and is a result of 160+ commits and 9,500+ lines of code, all of which you can see at: Frontend: https://salsa.debian.org/44yu5h/ci-dashboard Backend: https://salsa.debian.org/44yu5h/salsa-ci-dashboard-backend (These will be moved to the Salsa CI team namespace soon.) Many of you may remember my earlier mail introducing the Salsa Status dashboard. I’d be grateful for feedback on which statistics developers find most useful (or still missing). You can share feedback and find the most up-to-date information at https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/413 ## Improve the pipeline’s speed ------------------------------------------ I implemented buildah caching to accelerate image builds, which would greatly improve contributors’ experience by reducing build times to ~65%. I also introduced auto-cancellation of redundant pipelines to prevent unnecessary resource consumption in the internal pipelines. These optimizations reduced execution time and improved efficiency. MR 634: Implement buildah caching for faster image builds Link to MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/634 MR 641: Avoid running apt-get upgrades in the jobs Link to MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/641 MR 610: Replace the old timeout implementation with new RUNNER_SCRIPT_TIMEOUT variable Link to MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/610 ## Bugfixes ---------------- In addition to my main projects, I was also an active member on the Salsa CI Team, where I fixed various issues, helped others with debugging issues, and provided code reviews and help in testing. Job Failure due to failure to pull docker images on parallel CI pipeline runs Issue: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/289 Fixed in MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/615 Support piuparts and blhc arguments containing spaces or quotes Issues:https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/127 and https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/325 Fixed in MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/636 Use filter-output for blhc job and return artifacts for piuparts Issue: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/225 and https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/299 Fixed in MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/612 **Other bugfixes/contributions**: !593 (#427): Add zstd installation to missing-breaks job !611 (#438): Fix autopkgtest images cleanup: use STABLE instead of RELEASE !594 (#365): Clean Images job: Add missing arm64v8 images !595 (#433): Clean Images job: Update Ubuntu release from oracular to plucky !625 (#455): Revert "Workaround reprotest bug #1108550 by disabling some variations" !601: Docs: Add instructions for uploading public GPG key to keyserver **Resolved old inconclusive issues**: Issue: Autopkgtest pipeline doesn't save autopkgtest artifacts to the salsa-ci artifacts Link: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/417 Issue: Reprotest fails chown on disorderfs Link: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/26 **Other side-work on pipeline**: I worked on ways to skip parts of or the whole pipeline for faster determination of outcomes. These are still some way from finding their way into the main repo. To skip extract-sources and the build jobs, for debugging tests when the tests don’t necessarily depend on the build job (like autopkgtest, piuparts, etc.); this can help greatly reduce iteration cycle time. Link to it: https://salsa.debian.org/44yu5h/salsa-ci/-/commit/db76bce4806c4a41a02a5951a22bd7b1d13fcb52 To skip the pipeline on changes to some particular files. Link to it: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/437 #### Reflections ----------------------- The initial few weeks were the most challenging for me. Getting familiar with a large and established codebase, learning the community's best practices, and connecting with new people all took considerable time and effort. At that stage, even tasks that seem very simple in hindsight took longer to complete, since I was still piecing together how everything fit. With time and support from my mentor and the Salsa CI Team, I was able to work more independently and gradually pick up speed. When I started the program, I had little experience with CI pipelines. I learned a lot by working hands-on, experimenting, and learning from mistakes. While not all the issues I worked on made it into my key achievements, I learned something from each one. While building the stats dashboard I learned how to design a database that can grow with more data, deal with some of the quirks of Gitlab’s API, and tune the backend so it stays efficient while keeping results up to date. On the frontend, I got much more comfortable with React and data visualization. I also realized how critical it is to keep performance in mind early, since even small inefficiencies multiply quickly at scale. Working remotely, I wasn’t able to attend DebConf in person, but I’ve been actively involved online with the Salsa CI team and the local India community! #### Acknowledgments -------------------------------- I want to take a moment to thank my mentor, Otto Kekäläinen, for all the guidance and patience (especially when I managed to break things :). I’d also like to thank Santiago Ruano Rincón, Ahmed Siam, and Aquila Macedo for their support and collaboration during the summer. Santiago and Ahmed offered valuable reviews and feedback within Salsa CI, while Aquila, a fellow GSoC contributor, was both a great collaborator and a generous reviewer of my MRs. Looking back on this summer, I’m proud of what I have accomplished. The Salsa Status website now provides valuable insights for the Debian community. I’m grateful for the support and collaboration throughout the summer from Otto and the Salsa CI team, and I look forward to seeing these contributions continue to benefit Salsa CI and make life a little easier for Debian developers. I truly appreciated the connections I built within the community. Collaborating on shared challenges, exchanging ideas, and thoughtfully debating different approaches not only helped us move things forward but also helped foster meaningful bonds along the way. #### What’s next? ------------------------ The summer has ended, but the adventure hasn’t. Hopefully you will see me again around somewhere bashing my head on some random bug :p I intend to stick with Debian and continue my journey to become a Debian Maintainer. Thank you so much! Aayush Raj My Salsa profile: https://salsa.debian.org/44yu5h MRs in Salsa CI (12 of 17 merged): https://salsa.debian.org/salsa-ci-team/pipeline/merge_requests?state=all&author_username=44yu5h
