Deployment orchestration
How NervesHub coordinates firmware rollouts across a fleet: the distributed orchestrator, concurrency, queues, failure handling, and the penalty box.
Written By Josh Kalderimis
Last updated About 2 months ago
When you activate a deployment group, NervesHub doesn't push the update to every matching device at once. A background process called the orchestrator hands out updates in a controlled way, so a bad release never reaches your whole fleet at the same time.
The orchestrator
Each active deployment group has its own orchestrator process. Its job:
Watch for changes to the deployment group and for devices coming online.
Find online devices in the group that need the current release.
Tell those devices to update, up to the group's limits.
The orchestrator runs distributed across the cluster: on a multi-node NervesHub deployment, orchestrators are spread across nodes and coordinated globally, so exactly one runs per active deployment group and the work scales out. This is why self-hosting at scale involves a cluster rather than a single box.
Concurrency
The group's concurrent updates setting caps how many devices update at the same time (default 10). The orchestrator keeps that many updates in flight and starts new ones as others finish. A priority queue can run a separate, smaller concurrent lane to push devices that are far behind ahead of the rest.
Queue order
When more devices need updating than the concurrency limit allows, the rest wait in a queue. Queue management controls the order:
FIFO (default): first in, first out.
LIFO: last in, first out.
Matching devices to groups
A device is offered a release when it's in an active group, online, and running older firmware than the current release. When a device could match more than one group, NervesHub prefers the group whose tags match most specifically.
Failure handling
Rollouts watch for trouble and back off automatically:
Per-device failures. A device that fails repeatedly (within the configured count and time window) is moved to the penalty box and stops being offered the update for the penalty timeout. This keeps a struggling device from hammering downloads. Clearing the penalty box (from the device page) lets it try again immediately.
Group health. If the overall failure rate crosses the group's failure threshold, the group is marked unhealthy so a broadly failing release doesn't keep rolling out.
These thresholds are all configured per deployment group. See the rollout options in Deployment groups & releases.
Network-aware rollouts
A group can restrict updates to devices on particular network interfaces (wifi, ethernet, cellular). Combined with firmware deltas, this lets you avoid pushing large updates over expensive links.
Putting it together
A healthy rollout looks like this: you activate a group with a new release; the orchestrator offers it to online, matching devices up to the concurrency limit; devices update, verify, and reboot; failures are contained to individual devices via the penalty box; and if too many fail, the whole group pauses so you can investigate.