A unique testing solution delivered by conducting build testing on 50 devices for 8 consecutive rounds (400 test runs) to deliver enhanced UX, for one of the fastest growing regional media apps with 5M+ user base.
The Challenge
The client came to us in a dire situation. The latest version of their fast-growing OTT app, launched in Q1 2020, was facing major issues in production. The app rating had taken a big hit and had dipped to an all-time low of 2.7. The support team was receiving more than 100 issue tickets from users daily. It was October and there was a big launch event planned in front of the regional film industry along with a couple of exclusive movies dropping on the app in mid-November.
This was the time when the client reached out to Qualitrix for support in conducting a root cause analysis of the issue and helping them in their OTT/media app testing to fix the problem.
Detecting the Root Cause
On deep dive, we identified that many users were encountering a lot of ANR, Crashes and App hung issues on different device models. The client user base was diverse with extremely fragmented android users along with iOS and TV. Although there were two Quality Assurance teams working within the client ecosystem, they were able to test on only limited real / simulated devices, and couldn’t reproduce many compatibility and performance related issues at their end.
In consumer apps like OTT, recreating real-time scenarios on different user devices is very crucial to predict the app quality accurately. Typically, a lot of product firms rely on cloud-device farms and test automation to maximize their coverage for compatibility and app performance. However, such a solution has big limitation.
- Automated testing over device farms cover standard scenarios and not the edge cases – difficult to test issues like app response when a user is fast-forwarding the video, or performance on full-screen or split mode, etc.
- Network bandwidth related scenarios cannot be reproduced
- Coverage is on fixed devices, upgraded OS and device models are missed out
- If tested manually, multiple testers are required to cover more devices
Solving the Real-world Problem
The only immediate way to solving this problem was to plan a crowdsourced user testing.
The crowdsourced testing model is the one where an app is given out to a bunch of test users, who are distributed across the locations and who have an understanding of the application. These may be professional experts or maybe simple test users and cover tens of devices in live conditions.
Qualitrix being an industry leader at enterprise crowdtesting could easily pull up a team of 50 test-users with-in 24 hours using our crowd testing platform, Oprimes to run these tests. The managed crowd could be controlled by a team of senior test leads that would guide the crowd to focus on high severity issue only. The platform also allowed us to collect performance analytics from tester’s devices.
But that was not all!
A typical crowdtesting cycle is planned during a UAT or Beta phase so that the functional sprint issues are all fixed. However, the need of the hour was to fix all the compatibility and performance issues within days, and hence we had to integrate crowdtesting with the sprint cycle. This meant that we were running daily test runs for each new build with 50 test-users + 2 internal specialists, identifying and re-testing hundreds of compatibility and performance related issues on different Android, iOS and TV devices on a daily basis.
Key Highlights
- 50 testers were mixed and covered different devices, providing a good representation of at least 70-80% of the target market.
- Within first round of testing, 90-100 issues were captured, within a span of 8 hours.
- These issues were across devices and included a lot of crashes and ANR issues which could not be produced in the test labs
- The testers provided videos/screenshots, crash logs and other proofs, that allowed developers to identify and fix the problems
- This was repeated at least for 8 different builds with-in 12 days’ time
The Result
When the final version of application went live in Nov 2nd week, the play store rating went up to 4.4. The production ticket count went down drastically to less than 80%. The client acknowledged that this was their most stable build that they had released, as per their firebase analytics where the crash ratio as well as the adaptation of that build among the end users had been high.
Summary
The immediate goal of our client was successfully met after finding several key issues that helped in achieving their app stability and high ratings, which proved really useful for the client.
This case study is not just relevant for OTT app but to any kind of media, internet, eCommerce, telecom, entertainment app involving large volume of users accessing the app on diverse device models. Such apps need to work effectively for millions of such users on diverse user channels, and can easily lead to UX, compatibility and performance related issues due to limited multi-device testing done in the labs.
Our efforts in such a short span of time made a huge impact to the client and this was recognised as well. The client leadership team made a special mention for the contribution of Qualitrix QA team in their vote of thanks at the final launch event that was attended by who’s who of the film industry. Such huge applause and recognition from their side was commendable considering our involvement started only at the very last phase of the launch. Interestingly, seeing such wonderful results, crowdtesting has now been made a part of their standard release cycle ?.