How to Grow Healthy Open Source Project Infrastructures

Elizabeth K. JosephIn 2013 I joined the OpenStack Infrastructure team. In the four years I spent with the team, I learned a considerable amount about the value of hosting an infrastructure for an open source project in the open itself.

In 2014 I gave a talk at All Things Open and was interviewed by Jason Baker about how we'd done our systems administration in the open. My involvement on this team led me to advocate for systems administrators to use revision control and learn about tools for working with a distributed team. At the OpenStack Summit in Austin in 2016, our team did a talk on navigating the open source OpenStack Infrastructure.

The leadership of the OpenStack project in the space of open source infrastructures inevitably led us to encounter other open source projects that were similarly open sourcing their entire, or portions of, the infrastructure used in their project. In February of 2016, I launched OpenSourceInfra.org to begin tracking these infrastructures. The source for this site is hosted on GitLab, and we've seen a steady increase in merge proposals to add new projects over the past several months.

With this interest growing, I recently worked with Spencer Krum of IBM to finally host our first event, Open Source Infrastructure Day at SCaLE15x. We're incredibly grateful that Ilan Rabinovitch and the SCaLE15x crew were able to support us with space, with no certainty about how popular this first-time event would be. A lineup of six speakers from OpenStack to the Apache Foundation and Travis CI took attendees through a series of infrastructures and their wildly varied tooling for an informative morning of talks. In the afternoon, we held an unconference to swap tips and have discussions on topics that impact all of us. The full schedule for the event was published, and slides from each talk are now linked.

The day began with a talk from Monty Taylor titled From Zero to 2500: Managing Development for OpenStack completely in the Open. In this talk, he took us through the history of the OpenStack Infrastructure. The team had a number of missteps in their path to building a solid open source infrastructure, including organization and deployment of Puppet modules. Throughout the talk, he highlighted the challenges and solutions encountered that resulted in a very large continuous integration system and infrastructure with core contributors spanning several companies in the OpenStack community.

E. Dunham spoke on The Servo Browser Engine's Open Infrastructure where the Servo browser engine has been developed openly based on best practices from the Rust community. Before our break, we then heard from Daniel Pono Takamori who briefly introduced the Apache Software Foundation's open infrastructure before diving into The Macro and Micro of Donating to Infrastructure. In an effort to help communities learn how to support open infrastructures, he stressed the need for providing engineer time to open infrastructure projects and how to donate resources like hardware and cloud resources effectively.

The morning continued with a talk from Magnus Hagander of the PostgreSQL community who walked us through the Fully Automated ACME/Letsencrypt Integration now used by the project. All open, the tooling is shared with the community to help other projects manage SSL certificates across an infrastructure automatically. Carmen Andoh from Travis CI then joined us to give a peek under the hood of their infrastructure with Build Infrastructure Engineering for Open Source at Travis CI. While a deviation from fully open source infrastructures, their willingness to share tooling, engineering, and rationale behind the setup of their system brings great value to the hundreds of open source communities using their CI system for their projects. To complete our series of talks, R. Tyler Croy talked about Going Full Cloud for the Jenkins Project where he outlined the migration of the Jenkins project infrastructure to the Microsoft Azure cloud and heavy use of containers.

The unconference kicked off with a conversation that the whole room took part in, The Moral Obligation to Run FOSS for FOSS Projects. Proposed by Daniel Pono Takamori of the Apache Foundation, the question explored whether open source projects should be using proprietary services (most notably, GitHub, Slack) for their communities, or whether there is an obligation as open source projects to use open source tooling. There were a variety of opinions in the room, ranging from team resources, expediency, and long-term gains and losses. Some of the topics that arose were similar to those touched upon in Thierry Carrez's recent article, Using proprietary services to develop open source software.

A conclusion to this discussion wasn't expected, but learning thoughts from all sides was valuable. A mailing list to discuss this and more was also proposed to move the conversation forward online.

The rest of the afternoon included unconference sessions on the following:

• Handling donations fundraising
• Renting Macs for continuous integration testing and more
• Containers
• How your project does continuous integration testing

The event concluded with wrap-ups on each of these from R. Tyler Croy, Clint Byrum, E Dunham, and Karsten Wade.

All told, we had 35 people registered for this event, but at the peak, we were thrilled to have over 50 attendees in the room for the morning talks. Though it was smaller, the unconference in the afternoon had a deeply engaged crowd of folks very invested in open source infrastructures throughout our three session slots.

A common theme through the rest of the weekend was when we would be doing this next. Instead of a tiny community, we discovered through this event that there's a much broader group out there interested in furthering the development of open infrastructures. Since the event, we've seen a further increase in participation in our #openinfra channel on Freenode. You can also follow the latest news from the growing community by following us on Twitter, @OpenSourceInfra.

For more, you may also want to read Spencer Krum's article, 7 things I learned at Open Infrastructure Day @ SCaLE 15x.