Over-the-Air (OTA) updates allow you to deliver new features, fixes, and improvements to your mobile app without requiring users to download a new version from the app store. However, rolling out OTA updates requires careful testing and quality control to avoid introducing bugs or performance issues. In this article, we’ll walk through a professional workflow for running quality control (QC) and quality assurance (QA) before releasing OTA updates using Expo, including the benefits of Canary deployment.
Why OTA Updates Need Rigorous Testing
OTA updates can be a powerful tool for quickly addressing bugs or pushing small updates. However, due to their ease of deployment, it's tempting to skip rigorous QC, thinking, "It's just a minor change; it won’t break the app." But even small changes can introduce significant issues. For instance, during one of our QC processes, a minor OTA update caused a critical authentication error in the Android version of our app, underscoring the importance of thorough testing, even for "simple" updates.
Understanding Our OTA and Core Release Strategy
Our development process for OTA updates works in parallel with our major "core releases." Core releases are larger updates that require native code changes and are pushed through the App Store and Play Store. These core versions are labeled V1.2, V1.3, V1.4, and so on. OTA updates, on the other hand, are smaller patches or feature updates that don’t require native code modifications and are labeled as 1.2.1, 1.2.2, 1.3.1, and so forth.
The Workflow: How We Test and Release OTA Updates
Here’s how we manage and test OTA updates while maintaining high-quality standards:
1. Create a Mirror Build for Testing
When we prepare to release an OTA update, we don’t immediately push it live. Instead, we incorporate the update into the next core release (for example, V1.3), build the app, and run manual tests. This step is critical, as it ensures that any changes introduced in the OTA update are tested in an environment that mirrors the final production app.
2. Label OTA Updates and Track Versions
Our OTA releases are tagged with incremental version numbers such as 1.2.1, 1.2.3, or 1.3.2, depending on the major core release version they belong to. This versioning system helps us track which OTA updates are linked to which core releases, ensuring that any fixes or changes made during QC are automatically consolidated into the upcoming core release.
3. Test OTA Updates as Part of Core Releases
One challenge with OTA updates is the difficulty in isolating and testing them before they go live. By integrating OTA changes into the following core release, we effectively simulate the live environment during our testing process. This allows us to catch potential issues early, preventing bugs from reaching production.
For instance, during one of our manual QC tests, a small OTA update unexpectedly broke authentication on Android. Because this issue was caught during the testing phase, we were able to fix it before rolling out the update to users.
4. Release OTA Updates After QC
Once the OTA update has been thoroughly tested as part of the core release, we release it to the appropriate environment (Canary or production). We use a phased rollout strategy, starting with a small percentage of users before scaling up, ensuring that any unforeseen issues are detected before they impact the entire user base.
5. Consolidating OTA Changes in Core Releases
By incorporating OTA updates into the next core release, we ensure that all changes, whether minor or major, are consolidated and tracked. This helps avoid any confusion in the development process, speeds up the dev cycle, and ensures consistency between OTA and core releases. The edits we make during QC are already included in the next major release, allowing us to move faster and minimize duplicate efforts.
The Importance of a Structured QC Process for OTA Updates
Skipping QC for OTA updates may seem harmless, especially when you’re only pushing minor changes or bug fixes. However, even the smallest change can introduce unexpected bugs, as we’ve seen in our own experience. A structured QC process ensures that every update, no matter how small, is thoroughly tested before it reaches your users.
Key benefits of this approach include:
Early Detection of Bugs: By testing OTA updates as part of the core release, you catch issues early in the development cycle, preventing problems from reaching production.
Reduced Risk: The phased rollout of OTA updates minimizes the risk of widespread issues affecting your entire user base.
Streamlined Development Process: Consolidating OTA updates into core releases ensures that changes are tracked and organized, leading to a faster, more productive development cycle.
Final Thoughts
OTA updates provide a flexible and efficient way to keep your app up-to-date without requiring users to download full app updates. However, skipping the QC process for OTA updates is risky, even when dealing with minor changes. By following a structured testing workflow, incorporating Canary deployment, and consolidating updates into core releases, you can ensure that your OTA updates are stable, reliable, and free from bugs.
During our own testing, we once encountered a situation where a minor alteration in the user interface of a single page on a mobile application resulted in a catastrophic failure of the login process for Android users. This unexpected outcome underscored the potential for seemingly insignificant changes to have far-reaching consequences, which could easily go unnoticed. It is fortunate that our rigorous testing policy was in place, as it prevented a significant number of users from being locked out of the mobile app due to a small oversight.
Following these best practices not only protects your app from potential issues but also maintains a smoother, faster, and more productive development process. Ensure you don’t treat OTA updates lightly—proper QC is essential for a successful, stable release.
