In the fast-paced world of tech startups, quality assurance often takes a backseat to rapid development. However, Airbnb’s journey from a handful of brittle tests to a comprehensive, resilient suite demonstrates that it’s never too late to prioritize testing. This transformation not only improved code quality but also reshaped the company’s engineering culture.
The Testing Landscape at Airbnb
When a new engineer joined Airbnb in early 2023, they discovered a team committed to the idea of testing but struggling to implement it effectively. The existing test suite was slow and unreliable, with most new code shipping without any tests at all. However, within a year, the situation had dramatically improved.Airbnb’s engineering team managed to reduce build times from over an hour to just six minutes while simultaneously handling a much larger number of builds. More importantly, they fostered a culture where untested code changes were quickly identified and corrected. This shift was particularly significant for the payments team, which processes a vast number of transactions daily across multiple currencies and payment processors.
Cultivating a Testing Culture
The Power of Pull Requests
The transformation began with the introduction of pull requests (PRs). Initially, a small group of engineers started submitting PRs for their changes, a practice that quickly gained traction due to its evident benefits:
- Improved code quality through peer review
- Enhanced stylistic consistency
- A forum for discussing code structure and architectural decisions
- Increased visibility of code changes across the team
As new hires were briefed on the importance of PRs, the practice became the norm, even among long-standing team members.
Championing Testing
To further promote testing, Airbnb’s engineering leaders employed a multi-pronged strategy:
- Leading by example: Including tests with all PRs, regardless of the type of change.
- Educating the team: Conducting engineering meetings, offering guidance on test writing, and sharing testing news and PR highlights.
- Leveraging new hires: Revamping the testing bootcamp to emphasize the importance of testing in maintaining a flexible codebase.
Improving Testing Infrastructure
Recognizing that cultural change alone wasn’t sufficient, Airbnb invested in improving its testing infrastructure to make writing and running tests as painless as possible.
Local Testing Environment
Airbnb standardized its development environments using Vagrant and Chef, allowing engineers to work in sandboxed Linux instances that closely mirrored the production environment. This approach, combined with tools like Zeus for preloading the Rails environment, significantly reduced the friction in local testing.
Cloud-Based Continuous Integration
To address the challenge of running a large test suite efficiently, Airbnb’s Site Reliability Engineering (SRE) team implemented Solano, an on-premise solution that excels at parallelizing tests. This system dramatically reduced build times and improved the overall deploy workflow.
GitHub Integration
Airbnb integrated its CI server with GitHub’s commit status API, providing immediate feedback on the status of builds for open PRs. This integration increased transparency, accelerated feedback cycles, and ensured that every branch merged into master had a passing test suite.
Pragmatic Approaches to Testing Challenges
While striving for testing best practices, Airbnb’s engineers also learned to be pragmatic when dealing with legacy code and complex systems.
Testing SQL-Heavy Projects
For projects involving extensive SQL transactions, Airbnb adopted an acceptance testing approach. By focusing on the initial and final states of database tables, engineers could test complex ETL processes without getting bogged down in implementation details.
Service-Oriented Architecture Testing
As Airbnb moved towards a more service-oriented architecture, they developed strategies for testing services in isolation. One approach involved including stub functionality in client libraries, allowing tests to run without full service dependencies while still catching potential interface changes.
The Road Ahead
While Airbnb has made significant strides in its testing practices, the company acknowledges that there’s always room for improvement. The key takeaway from their experience is that it’s never too late to start prioritizing testing, even with a mature codebase.By combining grassroots efforts, improved tooling, ongoing education, and a pragmatic approach, Airbnb has created a testing culture where green builds are meaningful and code quality is a shared responsibility. This transformation has not only improved the reliability of their platform but has also empowered their engineering teams to refactor and innovate with confidence.
As Airbnb continues to grow and evolve, its commitment to testing remains a cornerstone of its engineering excellence, ensuring that the company can maintain its rapid pace of innovation while delivering a robust and reliable platform for its users worldwide.
Read more about the topic here.
Read more such articles from our newsletter here.
