The Basics of Open Source Quality Assurance

Open source depends on a sustainable community to develop code rapidly, debug code effectively, and build out new features.

Preethi ThomasOpen source depends on a sustainable community to develop code rapidly, debug code effectively, and build out new features. Because community involvement is voluntary, people's skills, levels of involvement, and time commitments can vary. Given the variable nature of these factors, along with the fact that open source often relies on a philosophy of "release early, release often," quality assurance can be become challenging.

In order to maintain the quality of the projects, the community including the developers, quality engineers, and the users of the projects have to work together.

4 factors of open source quality assurance

1. Continuous integration tools ensure defects are identified earlier in the cycle
2. Automated test frameworks encourage well-documented code and automated tests
3. Test days are proposed and executed by developers
4. Bug triage prioritizes issues and contributors / bug reporters submit use cases for future automation

Continuous integration tools

Using an open source continuous integration tool is an integral part for maintaining quality in open source. The Pulp project uses Jenkins for building, running automated tests, and monitoring results. The latest commits to Pulp are run nightly. By monitoring the results we are able to identify integration failures and regressions. Once these tests passes, the commits are promoted and nightly builds are made available for the testing team or community.

Automation test frameworks

Open source automation test frameworks are portable and flexible, allowing any contributor to write automated tests. For example, Pulp Smash is a test suite used to test Pulp. It is a GPL-licensed Python library integrated with the developer environment. It runs nightly on the development codebase and promotion of Pulp packages is based on the successful running of Pulp Smash.

Automation tests suits can be integrated with the developer environment to catch regressions early. The "release early release often" philosophy allows for fast release cycles and fewer dedicated quality resources.

Test days

Community test days are another way to engage the community in quality assurance. With clearly documented steps and tools, any user can propose a day to test a feature or area that they are interested in and can seek the help of the user base to do black box testing. A method for capturing the results should be identified as well.

For example, Fedora QA runs community Test Days on a regular basis. The test days are conducted between alpha and GA. Anyone can propose the test day by following the well-documented process. The test day-related discussions happen in the IRC channel #fedora-test-day.

Bug triage

Users can get involved by participating in public bug triage. This gives the community of users a chance to prioritize bugs that are important for them, verify or test out a fix, or even submit a fix. For example, Foreman held a community Bug Day in February this year to triage and verify existing bug reports. Unlike Fedora Test Days that focus on testing a specific feature and opening new bug reports, Foreman's Bug Day was focuses on reviewing existing bug reports across different features and closing some of them. Bug reports can be closed if the problem they describe has been fixed, is obsolete, or is described in another bug report. During this day in February, the community updated 250 bug reports, closed 133 of them, and only created 30 new bug reports.