Using Multi-Stage Testing To Improve Software Releases
Imagine you take your car to a local mechanic for repairs. They pop the hood, take a look, and find a loose hose. The good news is it will be quick and affordable to fix… or so they say. So they go ahead, tighten the hose and send you on your way.
No more than one hundred metres up the road you hear an enormous bang, your car grinds to a halt, and smoke is billowing out from beneath the bonnet. Your stomach sinks. This is going to be expensive!
Whether we’re talking about a greasy engine bay or your team’s instance of Github, there are four important rules to remember when troubleshooting, testing and declaring a product is fit for public use:
- Fixing one problem doesn’t necessarily fix all problems.
- Some problems can’t be diagnosed until others are fixed.
- Fixing one problem may lead to other new problems.
- Different environments can lead to different problems.
Software Testing Best Practices
How often should you test?
Naturally, that depends on the risk profile and scope of the release, but the short answer is this... you should test as many times as it takes to give you confidence that your software is working as expected, every time, within your desired specifications.
In practical terms, that equates to at least one test as early as possible in pre-production, another in pre-production if the initial test uncovers critical bugs and one in production. Scheduling testing as early as possible helps mitigate the risk of delays due to critical errors.
We call this process multi-stage testing. In keeping with the original mechanic analogy, it looks like this:
- Pre-production test #1 - is like examining the car for defects whilst on the hoist in the garage.
- Pre-production test #2 - is like driving the “fixed” car slowly around the carpark to ensure everything is fixed.
- Production test - is like driving the car on the highway during peak hour.
Of course, the closer the release gets to production, the greater the risk and the cost of remedying problems.
Professional Multi-Stage Testing By Bugwolf
Bugwolf helps digital teams improve the speed and quality of software releases. Our on-demand professional testing teams compete against the clock in accelerated six-hour challenges to provide increased coverage and test insights.
5 Reasons To Always Use Multi-Stage Testing
#1: All Bugs Should Be Uncovered As Early As Possible
According to a study conducted by IBM, the cost of a software bug discovered in the maintenance phase after release can be up to 100X greater than the same issue identified in the design phase.
Staged testing helps you discover critical issues earlier, which in turn reduces direct costs like rework, missed deadlines and lost revenue. As well as indirect costs like damaged reputation, inflated support burdens and frustrated users.
#2: Fixed Bugs Should Be Thoroughly Re-Tested
The old saying goes: when you assume, you make an ass out of u and me. That’s certainly true in the world of software testing.
Never assume that because a bug has been identified and raised, that it has been looked at by your dev team (much less fixed and tested). After all, the developers missed the original bug - there’s a chance they might do the same again. So always re-test.
I don’t say that to point the finger, but rather to stress that developers are humans too. And humans occasionally make mistakes. Your job is to implement systematic solutions that dramatically reduce the likelihood of these mistakes seeing the light of day.
#3: Resolved Bugs May Open Up New Workflows
Occasionally functional bugs will prevent testers completing workflows, which means they weren’t able to thoroughly test all areas of the software in the first round of testing. With this in mind, testers should re-test the remedied software and ensure the full workflow works correctly.
#4: Fixed Bugs Can Lead To New Bugs
Another important consideration is that code changes made while fixing bugs can create new, unrelated bugs. This is especially true when dealing with large, complex or legacy code.
We don’t do this to undermine or question the expertise of the developers behind the change, on the contrary, most veteran developers appreciate processes that provide another set of eyes over changes to fragile systems.
#5: Different Environments Can Lead To Different Bugs
Finally, different environments themselves can create new, unexpected errors.
It’s not enough to say, “it worked in pre-production, so it should work in production”. Whilst that may be the case in many instances, it’s certainly not true all of the time.
You should always test new releases once they go live, however, this step is especially important if the infrastructure behind your staging or pre-production environment is significantly different from the production environment.
Check Out This Real-World Example
We’re passionate about helping leading enterprises and growth-oriented startups win a competitive edge through better software.
Multi-environment testing isn’t just IT theory. Organisations regularly engage us to test and re-test their releases. Here’s the actual results of a recent Bugwolf challenge we conducted on behalf of a major Australian company.
We conducted a six-hour UAT challenge in their pre-production environment, which identified these bugs:
And here are the results from our test in their production environment:
Key points from this example:
- Testing conducted in the pre-production environment allowed them to identify issues that had to be resolved prior to release.
- Once these were resolved, the testers were able to re-test in the production environment, extending their coverage and identifying new issues that should quickly be addressed.
- Even more new bugs were discovered in the production environment, highlighting the importance of rigorous post-release testing.
- “Passed tests” in the production environment gave the company confidence that previous bugs had been resolved.
- Adding a second pre-production test would have allowed them to prevent more critical errors reaching the live site.
- Without these tests, significantly more bugs would have be present for longer, exacerbating the cost of software errors.
How We Help You Improve Software Quality
If you’d like our assistance bringing your testing efforts into line with best practices, we’d love to help. Bugwolf is uniquely equipped to help you test in pre-production and production without slowing down your releases. Here’s how we help you eliminate bugs and boost software quality:
Test As Fast As You Develop
Many organisations are reluctant to run multiple tests for fear of slowing down releases (that’s a scary thought!). With our help, you can run tests as you need them, on-demand. You can even schedule tests of an evening when your developers are offline, so they have actionable findings by the time they wake up the next morning.
Test What You Want, When You Want
Unlike internal teams, you don’t need to worry about conflicting schedules or capacity bottlenecks. You just tell us what you want us to test, and when you need the results by, and our professional testers will get the job done on time, every time.
Don’t Sweat The Small Stuff
Testing isn’t necessarily about perfection. Testing is about maximising quality and managing priorities. In addition to identifying as many bugs as possible, we also prioritise them all by severity. This helps your team focus on the most critical bugs first and action less important issues when the time is right. We’ll also provide reports about workflows that work as expected. This gives you confidence that previous issues have been resolved.
Faster Fixes First Time
Our testing teams use video capture software to help developers quickly replicate and resolve bugs. Our bug reports also include all important details like OS, device type, browser type, etc.. All of this information can be routed directly into your existing development suite to fast-track fixes.
Bugwolf helps you prevent critical software bugs seeing the light of day. We do this by providing flexible, on-demand access to professional testing teams.
Existing Bugwolf clients are invited to conduct their own staged testing challenges on demand. To find out more, speak to your test director.
If you are new to Bugwolf and would like to learn more about how to get started, the quickest and easiest way to find out more is to Request A Demo by clicking HERE.