Managing IoT devices at scale is inherently complex, and over-the-air (OTA) firmware updates only amplify that complexity. While deploying OTA updates to a few hundred devices may be manageable, doing so across hundreds of thousands or even millions raises the stakes dramatically.

A failed update can brick devices — rendering them unusable — requiring a recovery process that may be prohibitively expensive. Worse yet are boot loops or partial failures that are harder to detect at scale but just as critical. And while non-fatal problems like a degraded user experience during updates may seem minor, they can damage trust and disrupt operations. In industries like industrial automation, automotive, retail, or entertainment, such failures carry high reputational and financial risks.

Challenges With Device OTA Updates and Their Solutions Image

Catastrophic failures from experience

These real-world cases highlight how severe the consequences of OTA update missteps can be.

In a project serving millions of customers, a routine OTA update introduced a change to a proprietary document format in support of new features. Due to insufficient testing of the upgrade path, the update resulted in irreversible data loss across approximately 300,000 devices. The fallout included significant customer impact, brand damage, and some team member terminations.

Lesson: A lack of comprehensive testing across software and version compatibility boundaries leads to catastrophic failure.

A team managed to catch a memory leak just before an OTA update was pushed to 100,000 devices. Had it been missed, the devices would have bled with memory leaks and become unresponsive within days. Luckily, the team did a proper test and managed to catch the issue before the release, but it was a close call.

Lesson: It’s important to test not only the functionality within over-the-air update processes, but also to do non-functional testing, like verifying that the device will survive the time after the update and that performance will not degrade.

A customer chose inferior hardware before the over-the-air update requirements were fully considered. As a result, tens of thousands of devices were deployed in the field without any OTA update functionality. Initially, they did not need this capability, but as requirements evolved, the need for OTA updates became apparent. Unfortunately, the hardware was incapable of supporting OTA updates due to poor decision-making during the hardware procurement phase.

Lesson: Even if OTA updates are not immediately required, hardware choices must anticipate potential future needs and accommodate them accordingly.

The fundamental challenge lies not only in ensuring a device performs correctly after an update, but in ensuring that the update process itself is flawless and recoverable, even across diverse hardware platforms and unstable networks.

Proposed solutions

1. First principles: decisions must come first

The most critical OTA decisions must be made before development begins. Over-the-air update mechanisms define fundamental architectural constraints, and attempting to retrofit them later is often impractical or even catastrophic.

Choosing a bootloader that doesn’t support rollback, assuming an app-only update model without validating future kernel requirements, or neglecting secure boot and key management can result in limitations that cannot be resolved once devices are deployed. These are not choices to revisit later — they are foundational system requirements.

Failing to account for them early can leave teams without recovery paths, expose fleets to security risks, and significantly increase total operational cost. OTA reliability and resilience start not with code but with upfront design.

2. Strategic choices: open source vs. commercial

At the organizational level, stakeholders must decide whether to build their own OTA framework or adopt a commercial solution. Open-source frameworks offer greater control and eliminate per-device licensing costs, but they come with a high engineering burden. Commercial solutions reduce time-to-market and provide built-in support, but they can introduce significant long-term operational expenses — especially when managing hundreds of thousands of devices.

3. Update strategy: application-only vs. full system

From a technical standpoint, the OTA process can target either the application layer or the entire system image. Application-only updates reduce complexity and deployment risk but restrict future flexibility. If future applications require new kernel features or system libraries, these constraints become problematic.

Full system updates allow for complete environmental control but demand more sophisticated infrastructure and risk management. These typically involve either:

A/B partitioning, where updates are applied to an inactive partition with rollback support, requiring twice the disk space.
Rescue system models, which install updates from a temporary environment. These require less space but are more vulnerable to failed updates.

4. Hardware capabilities

A robust OTA implementation depends heavily on the hardware. It must:

Support conditional booting based on update success or failure.
Provide rollback mechanisms.
Integrate with the chosen update strategy.

These capabilities must be planned at the start. Retrofitting them later is often infeasible and expensive.

5. Managing upgrade paths

Testing isn’t just about validating the new version. Every update must also be verified as a transition from every supported prior version. As real-world devices often lag behind the latest release, this testing effort increases exponentially with each release.

To manage this complexity, options include:

Enforcing incremental updates (e.g., v1 → v2 → v3).
Limiting supported version windows (e.g., only updates from the last six months are supported).
Defining a strict upgrade policy from project inception.

These decisions must be made early. Changing policy midstream is difficult and often infeasible once devices are in the field.

6. Security architecture and key management

Security is foundational. OTA systems must ensure that:

Firmware is signed and verified before execution.
Secure boot is enforced where hardware allows.
The entire update process is cryptographically trusted, end-to-end.

Devices without secure hardware must rely on software-based safeguards and obfuscation to slow down attackers. Additionally, key rotation must be built into the design. If a signing key is compromised or access changes (e.g., when a vendor relationship ends), the update infrastructure must allow seamless migration to new keys — without exposing the fleet to attack or disruption.

Conclusion

Over-the-air updates in IoT are both essential and dangerous. As scale increases, so does the potential for catastrophic failure. What makes OTA especially unforgiving is that many of the most critical decisions must be made before the first device ships — often even before the first line of code is written.

The architecture of the update mechanism, the type of hardware used, and the decision to adopt a full-system or app-only update model all define the constraints that will govern future development. These choices are foundational. Choosing hardware without rollback capabilities or locking the update process into an inflexible delivery method can corner a project into costly and unrecoverable paths. Retrofitting features like rollback, secure boot, or system-level patching after deployment is, in many cases, either prohibitively expensive or outright impossible.

Security concerns add another layer of criticality. Key management and secure delivery paths must be planned in advance. It’s not enough to assume that updates will remain secure by design. A compromised signing key or weak authentication model can invalidate the integrity of every device in the fleet. Without a strategy for key rotation baked into the update pipeline, even identifying a compromise can become a liability instead of a recovery point.

Furthermore, update testing complexity compounds over time. Each new version does not just need validation — it must be tested as an upgrade from multiple prior versions, because devices will inevitably lag behind. Failing to plan for version support windows or incremental-only upgrade paths can lead to a scenario where updates are either blocked or dangerously unpredictable.

What might seem like engineering optimizations early in the development cycle — deferring secure boot support, limiting rollback options, or choosing a simpler OTA model — can turn into existential risks when a fleet scales to hundreds of thousands or millions of devices.

OTA is not a deploy-and-forget feature. It is a design-time commitment. Its safety, reliability, and cost are determined not by the OTA tool chosen, but by the architecture, constraints, and recovery paths put in place from the beginning. A well-designed OTA system can sustain and evolve a product for years. A poorly planned one can collapse it in a single release.

Challenges With Device OTA Updates and Their Solutions Image 2

Start a conversation with us

Oleksandr’s recent stories

Challenges With Device OTA Updates and Their Solutions