Mobile Application Testing plays a pivotal role in the success of any app, even more so than any web or enterprise application.
The environment in which a mobile app is used is much more complex than that of the web or other application types.
Users access mobile applications over hundreds of different device models, operating systems and network conditions.
Mobile apps also operate under harsh real-world situations.
However, due to the relatively low development cost of mobile apps, the product owner finds it difficult to get a sufficient budget for testing.
Lack of thorough app testing results in incomplete test coverage for many critical areas of the app.
Bugs in important features or functionality significantly affect the image of the brand and result in loss of potential revenue.
I believe that the quality assurance (QA) model for mobile applications requires a more intensive testing through broader test coverage of devices, configurations, and real-world scenarios. However, the model needs to deliver faster to enable quick iterations and at a lower cost.
In short, mobile apps requires a testing model that delivers testing services better, faster and cheaper!
But is it really possible?
It is certainly not possible using traditional QA models.
To enhance the test coverage, more investment would be required on both people and infrastructure, which eventually increases the testing cost.
Crowdsourced testing model is disrupting the QA industry with its unique delivery solution that allows us to engage expert testers, end-users, and infrastructure on-demand under real-world conditions.
The case study I discuss here illustrates how our team leveraged the crowdsourced app testing model to test 164 mobile apps for a large telecom firm in 10 days.
The Client Context
One of the largest telecom firms in the world was going to launch an app store with hundreds of entertainment apps for its users.
Qualitrix is their national QA partner for apps and services.
The client had planned to launch the entertainment app store portal in the fourth quarter of 2016.
164 apps were to be tested for functional accuracy, usability, and security before the app store launch.
Owing to deployment delays from various app partners on the app portal, the client expected us to test all these apps in less than 2 weeks to meet the launch deadline.
Key Challenges:
- To complete the feature, usability and security testing of 164 mobile apps within 10 days, with zero tolerance on quality and 1 week of preparation time
- Test each app on 3-4 handset models to ensure device compatibility
- Ensure that approved apps are stable and get high ratings on the live store
Clearly, traditional QA models were not going to meet these challenges.
Crowd Based Testing to the Rescue
We decided to use our managed crowdsourced testing model to meet the client’s expectations of testing an app at affordable costs.
In this model, we leverage crowdsourced testers who are managed by our in-house experts.
-
Standardize test delivery process:
- Set up an assembly line with testing teams and roles defined to focus on specific test modules only and senior test experts focusing on end-to-end scenarios
- Implemented process best practices for task breakdown, work assignment and daily collaborative meetings
- Combined structured testing with an exploratory methodology to enhance test coverage
-
Maximize reusability:
- Leveraged Qualitrix mobile app testing framework to create standard functional and security test cases customized to the app requirement
- Reused standard report templates
-
Use on-demand crowd testing set up:
- Leveraged our crowdsourced community to ramp up mobile test specialists within 48 hours to support the internal testing team
- Leveraged in-house device lab supported by crowd infrastructure to test apps across the 20 most common mobile handset models in the target market
-
Leverage technology:
- Set up a centralized test management solution on the cloud for task assignment, tracking and bug reporting
How it happened:
We planned to test 16 apps per day to complete testing of 164 apps within 10 days.
Each tester was assigned a given a set of apps specific to his handset models along with a set of test scenarios he had to execute.
Senior testers conducted end-to-end testing.
Test reports were discussed with the client every alternate day to decide on rejection/approval of each app based on test results.
Results Achieved
- 492 rounds (164 apps x 3 device models) of apps testing completed in 10 days
- 40 crash scenarios identified
- 750 functional issues, 450 security issues, and 50 compatibility
issues reported - 20 handset models covered
- Testing completed within 10 days
- Team and infrastructure ramped up in 48 hours
- 45% reduction in testing cost
Quality of defects reported was highly appreciated which helped the client to maintain their quality standard in the market.
The crowdsourced testing model is ideally suited for the exponentially growing consumer apps market where the need of the hour is consistent app behavior across different accessing mediums, quick periodic upgrades, and controlled spend.
The crowd model also provides access to a representative target market at an early stage to gain feedback on the app’s acceptance.
This allows product owners to be more confident at the launch.
Vikas Kanago
Mobile Test Manager, Qualitrix