Introducing the Test Sizing strategy for fast, stable feedback
At Thredd, we’re optimizing software delivery by adopting Test Sizing—categorizing tests as Small, Medium, or Large based on efficiency. This strategy ensures faster feedback, minimizes flaky tests, and enhances reliability. By thinking small first, we maintain speed without sacrificing quality. 🚀
At Thredd, we are on a journey to not only scale our technology but also optimise how we deliver high quality software. As an Issuer Processor operating in the fast-paced fintech space, we recognise that success depends on more than just innovation. Our customers and partners demand a platform that is highly reliable, performant, and adaptable to their ever-evolving needs. As we grow, these expectations only increase, requiring us to deliver faster and more frequently without sacrificing quality.
The Risks of Inefficient Quality Gates
To meet these demands, our release frequency must remain at peak levels. Our software development lifecycle (SDLC) has and continues to optimise for this through automated infrastructure and deployments.
However, in a world of frequent automated releases, how do we ensure that quality is not compromised? Automated quality gates serve as a vital checkpoint in our release pipelines, enabling us to verify the integrity of each build on its way to production.
When functioning optimally, they provide the confidence needed to maintain speed without sacrificing quality. Yet, many teams encounter a significant hurdle – automated quality gates can easily become unstable and inefficient over time, undermining their effectiveness.
Failing to prioritise test efficiency as a key element of your test strategy introduces several risks:
- Extended Feedback Loops: Slow running test suites delay feedback to engineers, reducing their ability to iterate quickly, slowing down delivery times.
- Delayed Bug Detection: Bugs caught late in the development cycle require more time and resources to address. By the time an issue surfaces, it may have already had ripple effects across other components.
- Erosion of Trust: Inefficient or flaky tests produce inconsistent results, eroding team confidence in automated quality gates, often leading to manual intervention that is costly.
- Resource Drain: Inefficient tests consume unnecessary computational and human resources.
- Missed Opportunity for Early Detection: Without efficient tests, critical issues can escape detection until later stages of the pipeline, or even worse, into production.
To mitigate these risks, we have adopted a test strategy centered on efficiency, which can deliver what we term ‘Quick Confidence’.
Test Sizing
Originally developed at Google, Test Sizing is a strategy for categorising tests based on their dependencies on external resources and execution time. In other words, categorising tests based on efficiency.
A test’s size is quantifiable (Small, Medium or Large) and is determined by how it runs, what it is allowed to do, and how many resources it consumes.
Small: Small tests have several restrictions placed upon them which ensure they don’t have access to the main sources of test slowness or nondeterminism.
Medium: Less restricted than small tests, medium tests provide a level of protection against test slowness and nondeterminism by preventing access to remote machines via the network. They are naturally slower and less stable than small tests.
Large: As the most flexible of the test sizes, large tests remove the last remaining localhost restriction and, as a result, have increased risk and the greatest chance of test slowness and nondeterminism. Large tests are, in most cases, reserved for full system, end to end testing, which are more often about validating configuration than pieces of code.
Test Sizing Matrix:
Feature | Small | Medium | Large |
Network Access | No | localhost only | Yes |
Database | No | Yes | Yes |
File System Access | No | Yes | Yes |
Use external Systems | No | Discouraged | Yes |
Multiple threads | No | Yes | Yes |
Sleep statements | No | Yes | Yes |
System properties | No | Yes | Yes |
Time Limit (seconds) | 60 | 300 | 900+ |
At Thredd, we embrace a ‘Think Small First’ philosophy in our testing approach, focusing on delivering Quick Confidence as early as possible through small, efficient tests. This mindset has driven greater testability in our designs and inspired initiatives like ‘In-a-Box’ testing. By isolating dependencies and encapsulating functionality within controlled environments, we transform traditionally large, end-to-end tests into smaller, more stable, and faster alternatives.
By combining Mike Cohn’s Test Pyramid with the Test Sizing strategy, we’ve created a robust framework to maintain efficiency and effectiveness across our test suites. This approach ensures that only the most relevant tests run at each stage of the pipeline, reducing feedback loops, conserving resources, and enhancing developer confidence. Smaller, faster tests catch issues early, while medium and large tests provide comprehensive validation where it matters most.
At Thredd, this structured and scalable testing strategy has not only improved test stability and efficiency but also fortified the reliability of our quality gates. As we continue to grow, these practices enable us to deliver with speed and confidence, meeting the high expectations of our customers and partners without compromising quality.
