Australia Post: Disruptive testing is a serious game
About Australia Post
Australia Post is an Australian government-owned enterprise, providing postal services in Australia and its overseas territories. Founded in 1809, the organisation has been keeping Australia connected for over 200 years by delivering mail, helping businesses and consumers with parcels, and providing a national retail network that supports local communities and organisations. In 2014, it reported revenue of $6.4 billion and after-tax profit of $116.2 million.
Australia Post receives as many as 10 million digital customer visits per month, from Australia and around the world, to access its products and services. Australia Post wanted to find a way to increase the speed of testing cycles, release apps bug free, and reduce the spike in help desk volumes, all while ensuring a seamless customer experience across multiple devices and web platforms.
“Disruption of the classic testing strategies was clearly in order”.
Australia Post recognised a need to transform its traditional approach to testing and quality management to ensure a better user experience in less time. While applications could be tested locally, the organisation had limited access to resources during regular releases to test how their applications responded across different operating systems, devices, and regions.
In addition to test automation, Australia Post typically engages testers onsite to run user-based test cases. In this scenario, test coverage is limited to a small pool of testers and devices using pre-scripted tests. Furthermore, this method is considered too structured to truly cover all aspects of a system. “In the real world users do not follow test scripts and the journeys our users take are varied and at times unpredictable” says Steve Maidment, General Manager - Digital Channels and Innovation at Australia Post. Disruption of the classic testing strategies was clearly in order.
Why Bugwolf Elite Teams?
Bugwolf is the leader in on-demand customer experience testing. The Bugwolf Elite Teams enabled Australia Post to leverage the benefits of having a fresh set of eyes reviewing their digital products prior to release and a team of professional testers who rigorously test applications in the same way customers use them. The testers have limited knowledge prior to testing which means they make no assumptions as to how an application might work. They do this in a fully managed, vetted, and secure environment, which is critical in ensuring their digital products remained confidential.
Bugwolf attracts the best software testers from around the world due to their potential involvement in world-class and leading software releases and not limited by their geographic location. This delivers a much faster result with greater ‘on the glass’ test coverage and more high value bugs typically discovered in the process. The Bugwolf approach reduces traditional cycles from weeks to days, lowers the cost of testing by up to 50%, and provides up to 10x accelerated test coverage than traditional testing methodologies.
“We were looking for a solution to help us better manage spikes in application user testing, and a plug-and-play type service to act as an extension of our existing teams as and when needed. We were also looking for a solution that would improve the quality of user experience over a traditional UAT and BVT process”, says Mr Maidment. “We looked at a number of companies, solutions and technologies and selected Bugwolf because they had something innovative to offer”.
Australia Post engaged Bugwolf to help transform its testing methodology and mindset to accelerate the process and increase the test coverage of customer facing applications – reducing the likelihood of poor customer experience being released to market.
Using a combination of bounty based game mechanics and crowd sourced elite teams, Australia Post achieved test coverage and engagement unlike other services they had utilised in the past.
By utilising an on-demand customer experience testing model, they were able to test across a range of different device configurations and geographic locations while keeping their new product releases confidential. Another key advantage of the Bugwolf approach was that testing could continue over the weekend during scheduled downtime. This significantly reduced the impact of testing on the performance of the asset and teams during normal business hours.
“We selected Bugwolf, because they had something innovative to offer”.
Bugwolf orchestrated a broad test, identifying 176 bugs (stack-ranked and prioritized) across the domains of usability, user experience, content, and accessibility to name a few. We provided a comprehensive report back to the leaders inside the business, across both product development and engineering, which could be utilised by their in-house test and development teams to improve development and design processes for future projects.
The report provided context relating to each bug, and included details such as browser type, device, operating system, and location of the bug hunter. We also provided explicit details for each bug including a screencast, making it easy for their teams to replicate the issues.
Uniquely, the team put the Australia Post identity verification process through its paces, allowing our elite teams to push the workflows of the system using different user accounts. We provided each professional tester with a high level view of the different customer flows and they were incentivised to perform tasks like a customer. We kept the test plan and guidelines to a minimum which allowed testers to follow unique paths and discover defects which most automation or traditional testing process may not have identified. To allow us to test in a real world environment, our teams were attacking a production version of the application and rigorously testing the on-boarding process.
By using Bugwolf, Australia Post increased their test coverage for both application workflows and devices, discovering many important bugs which had not been previously uncovered using the traditional approach. This had a direct, positive impact on Australia Post end user digital experiences, and increased overall test cycle efficiency. “Bugwolf improved the user experience, improving customer satisfaction. Prior to this, we were reliant on a small team of testers, who developed structured test scripts using a limited device pool for on-the-glass testing. This resulted in limited device coverage and a lack of confidence in the end to end user experience” says Steve Maidment.
“A considerable degree of obscurity exists surrounding the development process and few organisations recognize opportunities for compressing the testing cycle through disruptive innovations.”
Some of the key benefits Australia Post recognised:
- Elite testers who test applications like customers use them
- Access to key talent pool which are typically not available
- Accelerate testing cycles and speed up product releases
- Extend device and browser and testing in different regions
- On-demand testing which can be utilised when needed
As with so many organisations that have applied what the industry has come to accept as testing “best practices”, a considerable degree of obscurity exists surrounding the development process. Consequently, few organisations recognize opportunities for compressing the testing cycle through disruptive innovations. As Mr Maidment summarized, “We didn’t know what we didn’t know.”
The internet is always changing and few organisations implement proactive measures to ensure that software and web application quality is maintained after their initial release. In recognition of this, Australia Post have chosen Bugwolf to provide on-going testing maintenance throughout the year. Bugwolf Elite Teams will be assembled to attack the same application and report back on defects they discover regularly, while also highlighting opportunities for improvement, staying ahead of the technology adoption curve.