Client Overview:

Indee is an innovative online platform that enables independent filmmakers to showcase their creative work to film festivals and online audiences. With a core focus on awards, publicity events, and online content distribution, our client serves as a trusted medium for filmmakers to present, market, and monetize their creations effectively.

Given that Indee deploys multiple apps across a variety of platforms including Fire TV, Roku TV, and iPad, ensuring a consistent user experience and high standards of performance, while consistently responding to changing requirements at an accelerated pace is a top business requirement.

Key Challenges

As our client expanded its apps across Fire TV, Roku TV, and iPad, critical QA hurdles emerged:

1. Consistent User Experience Across Devices: Ensuring that all applications worked consistently across  different devices and platforms.
2. Predictable Quality : Ensuring that s new features and functionalities were added, all new and existing end to end user journeys worked seamlessly across every release
3. Increase Release Velocity : Ensuring predictable quality at speed, as the demand for user features increased across more devices
4. Integrating QA with CI/CD: Ensuring quality assurance processes were part of the CI/CD pipelines, maintaining high app quality while enabling frequent updates.

Our Solution:

To address these challenges, our Qualitrix team established a Managed QA Center with a focus on automation, cross-platform testing, and CI/CD alignment:

1. Quality: Setting clear goals and expectations for quality, including measurable targets and agreed-upon service levels.
2. QA Strategy: Creating an Automation strategy to test the app across multiple platforms like Fire TV, Roku TV, and iPad.
3. Automation: Setting up automated tests for common user actions, integrated with the continuous integration and deployment process.
4. Test Coverage: Developing and running over 500 test cases to make sure the app worked well on all platforms and covered all features.
5. Shifting Left Testing: In-sprint automation and early testing ensured that testing was done earlier in the release cycle

Technology Stack:

The QA approach utilized a combination of tools, frameworks, and systems to ensure quality:

1. Tools:
   • UIAutomator2: An Appium driver for automating tests on Android devices.
   • Appium 2: Used for automating tests on mobile apps across different platforms like iOS and Android.
   • XCUITest Driver: An Appium driver for automating tests specifically on iOS devices.
   • Roku WebDriver: A testing automation tool for Roku devices, developed by Roku’s own team.

1. Frameworks:

• Pytest with BDD (Behavior-Driven Development): A testing framework that allows writing tests in a simple, human-readable format, focusing on the behavior of the app.
• Reports: Allure Reports integrated for graphical representation of Test Results

1. Version Control and CI/CD:

• GitHub: Used for version control to manage code and track changes.
• Jenkins: A tool for automating the continuous integration and delivery process, ensuring new code is tested and deployed smoothly.
• Docker: A platform for developing, shipping, and running applications in containers, ensuring consistency across different environments.

Implementation Approach: 

1. Behavior-Driven Development (BDD)
   Test cases were documented using Gherkin, aligning with requirements and enabling better collaboration between QA, developers, and stakeholders.
2. Test Grouping and Prioritization
   Test cases were grouped by functionality and prioritized based on critical features. This allowed for parallel execution, significantly reducing test execution time.
3. Reusable Components
   A library of reusable actions, page definitions, and descriptive details was created to maintain flexibility and ensure easy updates and scalability of test scripts.
4. CI/CD Integration
   Seamless testing was integrated into the development pipeline using Jenkins and Docker, ensuring consistent execution and rapid feedback.
5. Test Data Management
   Test data was centralized and managed using master test data techniques, dynamically retrieving data from runtime parameters or a test data JSON file within the folder structure.
6. On-Device Execution
   End-to-end testing was conducted on real devices like Fire TV, Roku and iPad, ensuring that tests closely mirrored actual user experiences.

Results and Impact:

The implementation of the QA framework resulted in:

• Enhanced Quality: Achieved high reliability across platforms with over 500 automated test cases.
• Efficiency Gains: Integrated QA with CI/CD, significantly reducing manual testing efforts.
• Scalability: Reusable components made it easier to add new test cases as needed.
• Improved User Experience: Ensured seamless functionality and consistency across devices for multiple applications.
• Reduced Time-to-Market: Streamlined testing processes accelerated delivery timelines.

Challenges and Key Takeaways:

Challenges:

• Device-Specific Quirks: Addressing platform-specific quirks for Fire TV, Roku, and iPad.
• Test Data Complexity: Managing and maintaining accurate, reusable test data, including the client’s request to make all test data calls from examples.
• Initial Setup: High upfront effort required to establish reusable test libraries and action definitions, with each device requiring a different driver approach.
• Dynamic Locator Strategy: Implementing a dynamic locator strategy to handle varying UI elements across different platforms.
• Behavior Variations: Managing and addressing behavior variations across multiple platforms proved to be challenging.
• Installation Scripts for iOS: Developing installation scripts for iOS added complexity to the setup process.
• Screenshot Addition for Roku: Implementing screenshot functionality for Roku testing was particularly challenging due to platform-specific limitations.

Key Takeaways:

1. Early Collaboration: Close collaboration between the QA and client development teams greatly enhances efficiency and ensures smoother communication.
2. Reusable Libraries: Developing reusable libraries significantly reduces future maintenance costs and streamlines test creation.
3. Continuous Monitoring: Ongoing monitoring and feedback loops are critical for improving test coverage and ensuring that all scenarios are captured.
4. Local Execution Report: Ensuring that local execution reports accurately reflected the app’s behavior variations helped provide proof to the client that the scripts were working correctly during development.
5. Test Data and App-Specific Issues: Having clear documentation and proof for raising test data and app-specific issues ensured that any concerns were promptly communicated to the client.
6. Standup MOM (Minutes of Meeting): Maintaining written acknowledgment of individual or standup meetings helped keep all teams aligned and provided a clear record of actions and decisions.

Conclusion:

The QA initiative for Indee successfully overcame platform-specific challenges, ensuring smooth functionality on devices like Fire TV, Roku TV, and iPad. By using a detailed QA strategy that included automation, Behavior-driven development (BDD), and CI/CD integration, the project achieved high-quality results while maintaining efficiency. This approach simplified the testing process, improved collaboration between the client’s development and QA teams, and enhanced the overall user experience across all platforms.