Qualitrix, one of the fast-emerging global players in on-demand quality engineering and testing space, has solved one of the biggest quality challenges for mobile based start-ups, “To ensure app quality on diverse user smart phones by performing quick test runs on multiple phones.” The company has a global presence and is headquartered in Bangalore, India.
“Most of our mobility clients were raising a concern that they wanted automated testing on multiple devices but couldn’t afford a device farm to connect with more devices. Our R&D team hence designed this private and secure device farm solution built over open source technologies that can be used free of cost. The solution allows parallel testing on multiple real devices to enhance test coverage and also allows distributed testing across devices to complete test runs 5 to 10x faster. The device farm easily integrates with all common android test automation frameworks.” explains Mayank Mittal, Director at Qualitrix.
What is a device farm?
Device farm is an application testing service, used to provide QAs to remotely test their websites and mobile apps performance on numerous real devices.
Providing with the end-user’s perspective, the device farms enable developers to assess the application and website operability and performance manually. Simply by using the application or website (on desktop, mobile phone or tablet) like another user would operate on it, the device farm allows the tester to see how the application/website performs in the real world.
The developers interact with the platform and work on the appearance and operation of the user interface on various devices. This way, they can monitor their functioning and identify other issues. A considerate part of marketing budgets worldwide, device farms conceal their activity behind fresh IP addresses using several privacy features like DeviceID Reset Fraud or enabling Limit ID Tracking.
Here is an excerpt of the interview conducted with Deepak P., chief designer of the device farm solution and one of the technical architects at Qualitrix.
-
-
Could you tell us how this idea originated?
-
Yes, basically an idea that came out from our requirement. We initially ran a few pilots for our mobility clients by running automated regression testing over some of the popular real-device farms in the market. The pilots went smooth for couple of months but our clients were becoming increasingly reluctant to continue spending on device farms due to high usage cost. Eventually, we were left with nothing and were testing on our local machines over 1 or 2 devices. Each tester was executing their set of test cases and giving out scattered reports. The results were not consolidated so we thought we have to come up with a more reasonable and cheaper solution.
-
Can you explain a bit more about your solution and how it is different from existing solutions in the market?
See, you can have multi-device parallel execution even directly through Appium. But it could be unnecessarily complex. If you have to execute on ten parallel devices you will have to channelize each thread individually. Moreover, there is a need for centralized interface for device management across different testers to track device OEMs, OS versions, SIM card info, availability status and much more. On the other hand, the commercial device farms in the market have high subscription cost and permit sub-optimal usage of in-house devices.
With our private device farm, we set up an in-house server that anyone over the private network can access across the firm. Anyone can view the interface with device info, availability, and can book the devices for testing even from a remote location.
The biggest advantage of this device farm over other commercial solutions is the distributed testing feature. Most device farms today don’t allow distribution of test cases across different devices. Here you can distribute regression testing across devices. Say you have 1000 test cases and 10 devices; you can distribute 100 each and complete your entire test execution in 1/10th of the time. This could be very effective during quick releases.
-
Could you give a technical overview of your solution?
- We have built the solution over open STF project with an open source stack comprising Node.js, ADB, RethinkDB, GraphicsMagick, ZeroMQ, Protocol Buffers, yasm, pkg-config. The entire stack is set up and installed over a local server using homebrew. This server is configured to be accessible within a firm’s private network. Multiple utilities and integrations have been written to add or enhance device management experience, modular reporting, battery manipulation and many additional features.
-
What are the benefits that your solution has delivered to your clients?
- All our clients have been very excited to use this device farm solution.
Key benefits that they realized include:
Zero-cost solution
It’s a plug and play solution. Firms already procure some devices, and engineers also own some. Any or all of these devices could be easily plugged-in to the device farm on-demand at the time of test execution whenever required. You get test results on any number of devices and the cost is nil for licensing or usage. The solution is operational for android devices.
Better test coverage
The solution has been very effective in capturing both functional and device specific issues. It also recreates unique test scenarios of app performance under manipulated battery conditions. The solution is already integrated with Jenkins CI/CD pipeline, with plugins for Zephyr with Jira, email and slack notifications.
Slashed test execution time
Through distributed testing feature, you can spread your test cases by adding more parallel devices and keep reducing your test execution time manifold, thereby giving a more reliable QA sign off even for shorter test cycles for hot fixes or ad hoc releases.
Device management
Another interesting benefit is to have a firm level real-time overview of device inventory, such as model, make, OS version, SIM card, network, and availability. Multiple users can access any device remotely, at any point in time and also have the convenience of using all its features including use of hard keys, drag and drop app install feature and many more.
-
Where do you take this solution from here?
- As an organization, we follow a continuous improvement approach. There are many interesting features that we are adding on as we speak to bring this solution at par with any commercial device farms available in the market. One of them is a plugin for better experience for distributed test runs where a tester can select devices and test packages or classes from an interface in any job configuration. We are also working on capturing performance data (CPU, memory usage, battery, network info) at specific points during the test execution and append it to the reports.
As an organization our goal is to make premium quality affordable to all by mitigating such infrastructure disparity arising due to cost constraints of small firms vis-à-vis large enterprises.
