How We Tested 164 Mobile Applications in 10 Days Using Power of Crowdsourcing
Mobile Application Testing plays a pivotal role in the success of any app, even more so than any web or enterprise application.
The environment in which a mobile app is used is much more complex than that of web or other application types. Users access mobile applications over hundreds of different device models, operating systems and network conditions. Mobile apps also operate under harsh real-world situations.
However, due to relatively low development cost of mobile apps, the product owner finds it difficult to get sufficient budget for testing. Lack of thorough app testing results in incomplete test coverage for many critical areas of the app. Bugs in important features or functionality significantly affect image of the brand and result in loss of potential revenue.
I believe that the quality assurance (QA) model for mobile applications requires a more intensive testing through broader test coverage of devices, configurations and real-world scenarios. However, the model needs to deliver faster to enable quick iterations and at a lower cost. In short, mobile apps requires a testing model that delivers testing services better, faster and cheaper!
But is it really possible?
It is certainly not possible using traditional QA models. To enhance the test coverage, more investment would be required on both people and infrastructure, which eventually increases the testing cost.
Crowdsourced testing model is disrupting the QA industry with its unique delivery solution that allows us to engage expert testers, end-users and infrastructure on-demand under real-world conditions.
The case study I discuss here illustrates how our team leveraged the crowdsourced testing model to test 164 mobile apps for a large telecom firm in 10 days.
The Client Context
One of the largest telecom firms in the world was going to launch an app store with hundreds of entertainment apps for its users. Qualitrix is their national QA partner for apps and services. Client had planned to launch the entertainment app store portal in fourth quarter of 2016.
164 apps were to be tested for functional accuracy, usability and security before the app store launch. Owing to deployment delays from various app partners on the app portal, the client expected us to test all these apps in less than 2 weeks to meet the launch deadline.
- To complete feature, usability and security testing of 164 mobile apps within 10 days, with zero tolerance on quality and 1 week of preparation time
- Test each app on 3-4 handset models to ensure device compatibility
- Ensure that approved apps are stable and get high ratings on the live store
Clearly, traditional QA models were not going to meet these challenges.
Crowd Sourced Testing to the Rescue
We decided to use our managed crowd sourced testing model to meet client’s expectations from app testing at an acceptable cost. In this model, we leverage crowd sourced testers who are managed by our in-house experts.
- Standardize test delivery process:
- Set up an assembly line with testing teams and roles defined to focus on specific test modules only and senior test experts focusing on end-to-end scenarios
- Implemented process best practices for task breakdown, work assignment and daily collaborative meetings
- Combined structured testing with exploratory methodology to enhance test coverage
- Maximize reusability:
- Leveraged Qualitrix mobile test framework to create standard functional and security test cases customized to the app requirement
- Reused standard report templates
- Use on-demand crowd testing set up:
- Leveraged our crowdsourced community to ramp up mobile test specialists within 48 hours to support internal testing team
- Leveraged in-house device lab supported by crowd infrastructure to test apps across 20 most common mobile handset models in the target market
- Leverage technology:
- Set up a centralized test management solution on cloud for task assignment, tracking and bug reporting
How it happened:
We planned to test 16 apps per day to complete testing of 164 apps within 10 days.
Each tester was assigned a given a set of apps specific to his handset models along with a set of test scenarios he had to execute.
Senior testers conducted end-to-end testing. Test reports were discussed with client every alternate day to decide on rejection/approval of each app based on test results.
- 492 rounds (164 apps x 3 device models) of apps testing completed in 10 days
- 40 crash scenarios identified
- 750 functional issues, 450 security issues and 50 compatibility
- 20 handset models covered
- Testing completed within 10 days
- Team and infrastructure ramped up in 48 hours
- 45% reduction in testing cost
Quality of defects reported was highly appreciated which helped the client to maintain their quality standard in the market.
The crowd sourced testing model is ideally suited for the exponentially growing consumer apps market where the need of the hour is consistent app behaviour across different accessing mediums, quick periodic upgrades and controlled spend. The crowd model also provides access to a representative target market at an early stage to gain feedback on the app’s acceptance. This allows product owners to be more confident at the launch.
Mobile Test Manager, Qualitrix