Phased Updates in Ubuntu

Phased updates are a critical feature in Ubuntu that help ensure a smooth and stable update experience for users. Instead of releasing updates to all users at once, phased updates gradually roll out to increasingly larger subsets of users. This approach allows issues to be identified and resolved early before impacting the entire user base.

How Phased Updates Work

When a new update is available, it goes through the phased update process:

Initially, the update is released to a small pilot group, typically around 10% of users.
If no critical bugs are reported after a set period, the update proceeds to a larger subset, usually around 50% of users.
After another period with no critical issues, it rolls out to 100% of the user base.
If issues are found at any stage, the phased update is paused for investigation and fixes before continuing.

Benefits of Phased Updates

Phased updates offer several significant benefits:

Early identification of update issues before they affect all users
Ability to pause problematic updates and minimize negative user impact
Smoother overall update experience for most users
Increased stability of the operating system over time

Phased Update Infrastructure

Ubuntu’s phased update process is powered by several key infrastructure components. These work together to manage the rollout process and monitor for issues.

Update Servers

Ubuntu’s update servers, managed by Canonical, are the central hub from which updates are distributed. These servers are spread out geographically to enable fast downloads for users worldwide. The update servers contain the logic that controls the phased rollout process, determining which users receive updates and when.

Phased Updater Client

The Phased Updater client component runs on each user’s device and communicates with the update servers. It sends information used to determine eligibility for phased update groups, such as:

Ubuntu version and architecture
Enrollment in Beta or Edge update channels
Opted-in to phased updates
Randomized user identifier The Phased Updater is responsible for downloading and installing updates that have been approved for the user’s group by the server-side rollout logic.

Error Tracker

Ubuntu uses an error tracking system to detect issues with updates. When an application crashes or encounters errors after an update, an anonymized report is sent to the Error Tracker.

Ubuntu developers and engineering teams continuously monitor Error Tracker data. If error rates exceed predefined thresholds, it automatically triggers a pause in the phased rollout. The team then investigates the errors to identify the problematic package and potential fixes before resuming the update for remaining users.

Controlling Phased Updates

Users and administrators have several options to control how phased updates are applied to their systems. Selecting the appropriate configuration helps strike a balance between early access to new features and stability.

Standard VS Edge Channels

Ubuntu provides two primary update channels:

Standard channel: This is the default stable channel that uses phased updates. Most users should stick with this channel.
Edge channel: This channel skips phased rollouts and provides updates immediately upon release. It’s primarily intended for testing and development rather than production use.

Opting Out of Phased Updates

Users who need the latest updates as quickly as possible can choose to opt-out of phased updates while staying on the standard channel. This is possible via the Software & Updates application in System Settings.

However, opting out is generally not recommended, as it bypasses the stability benefits of phased rollouts. It should only be used by advanced users who are prepared to deal with occasional issues.

Managed Rollouts for Organizations

Organizations with large-scale Ubuntu deployments often require more granular control over updates. Landscape, Canonical’s systems management solution, offers managed phased update rollouts.

Administrators can create their own phased update groups within their organization, for example:

Pilot: 1% of devices, used for initial testing
Early Adopters: 10% of devices, usually more tolerant of risk
General: The remaining 89% of devices Landscape allows setting up rules and schedules for progressing updates through these internal phases. This gives admins the ability to rigorously validate updates in their environment before wide-scale rollout.

Phased Update Workflow

Now that we understand the infrastructure and controls, let’s walk through a typical phased update rollout in greater detail.

Update Preparation

Prior to release, an update goes through rigorous testing by Canonical and Ubuntu developers. This includes:

Automated integration and acceptance testing
Manual testing of critical packages and scenarios
Limited beta testing with opt-in users Once the update has passed these checks, it’s approved for phased rollout to the standard channel.

Pilot Phase

The initial pilot rollout is essential for catching issues that weren’t found during pre-release testing. The pilot group is chosen based on several factors:

Devices enrolled in Beta or Edge channels
Users who have opted-in to receive early phased updates
A randomized sample of remaining standard channel users, up to around 10% of the total user base The pilot phase usually lasts 1-2 days to provide time for issues to surface. During this period, Error Tracker data is closely monitored. If error rates stay within acceptable levels, the update advances to the next phase.

Broad Rollout

After a successful pilot, the update is made available to a larger portion of standard channel users, typically around 50%. An additional 2-5 days of Error Tracker monitoring takes place.

If everything goes smoothly, the update reaches general availability for 100% of the standard channel. The specific timing of these phases varies based on the type and risk level of the packages included in the update.

Rollout Pausing and Resuming

At any point during the phased rollout, if Error Tracker rates exceed thresholds, the update is automatically paused. Developers investigate the issues and identify fixes, usually in the form of updated package versions.

Once fixes are ready, the phased rollout starts over from the beginning with a new pilot group. It then proceeds through the stages again until reaching general availability.

Technical Deep Dive

Let’s explore some of the technical intricacies of how phased updates are implemented in Ubuntu.

Phased Percentage Calculation

The update server’s rollout logic uses several data points to calculate each user’s phased update percentage:

Ubuntu version and architecture
Beta or Edge channel enrollment status
Opt-in status for early phased updates
A randomized, anonymized machine identifier The machine identifier enables consistent but privacy-preserving user categorization. A hash function maps the identifier to a percentage value between 0-100.

For each phase of the update rollout, the server specifies a target percentage range. For instance:

Pilot: 0-10%
Broad: 10-50%
General: 50-100% If the user’s identifier falls within the active range, and the other criteria are met, the update becomes available to that device.

Phased Update Metadata

Each update published in Ubuntu includes a set of package files and metadata describing the contents and rollout criteria. Key metadata fields related to phased updates include:

Phased-Update-Percentage: The current target percentage for the rollout
Phased-Update-Percentage-Override: Used to pause or reset phased updates
Phased-Update-Interval: The time interval between update checks for clients These fields are managed by Ubuntu developers and enable precise control over the rollout process.

Client-Server Communication

When a user’s device checks for updates, the Phased Updater client sends a request to the update server with the machine’s details. The server responds with a list of available updates that match the rollout criteria.

This communication happens securely over HTTPS, ensuring the integrity and confidentiality of the update metadata. The client then proceeds to download and install the update packages.

If an update is paused or reset while in progress, the server uses the metadata override fields to communicate this to clients. The Phased Updater halts the update and waits for the next check-in to receive updated instructions.

Phased Updates In Practice

To illustrate the phased update process, let’s walk through a hypothetical example of a significant Ubuntu release upgrade.

Preparing for the Release Upgrade

Leading up to a new Ubuntu version release, developers are hard at work preparing the upgrade packages. This includes:

Updating the core operating system components
Rebuilding packages to be compatible with the new release
Conducting integration testing across a wide range of hardware and configurations
Resolving any upgrade-related bugs discovered in testing
Documenting known issues and workarounds As the release date approaches, the upgrade packages are staged on the update servers in preparation for phased rollout.

Initial Pilot Rollout

On release day, the upgrade is made available to the pilot group. This includes:

Users enrolled in the Beta channel
A small percentage of standard channel users, typically around 0.5%
Hardware configurations known to have good upgrade compatibility Users in the pilot group are notified of the available upgrade. They can choose to initiate the upgrade process through the Software Updater application.

As pilot users upgrade, Error Tracker monitoring is in full effect. The Ubuntu engineering team keeps a close eye on error reports, especially those related to the upgrade process itself.

Analyzing Pilot Results

After the first day of the pilot rollout, the Ubuntu team analyzes the results:

Upgrade success rates
Most common error reports
User feedback on forums and social media If the upgrade is going smoothly for the majority of pilot users, with error rates within expected thresholds, the rollout progresses. If not, the team investigates the issues and develops fixes before re-starting the pilot.

Broadening the Rollout

With a successful pilot, the upgrade is rolled out to a larger audience, typically around 10-15% of standard channel users. This phase lasts for several days to a week, depending on the complexity of the upgrade.

During this phase, Error Tracker monitoring continues, with a focus on issues that may not have appeared in the smaller pilot group. The team also monitors user sentiment and feedback channels for signs of widespread problems.

If all goes well, the rollout gradually expands to 50%, 75%, and eventually 100% of standard channel users. The cadence is adjusted based on the stability and error rates seen at each stage.

Dealing with Upgrade Issues

Despite extensive preparation and testing, it’s not uncommon for issues to surface during a major release upgrade. When this happens, the phased rollout is paused for investigation.

Ubuntu developers work to diagnose the issue using:

Error Tracker reports
User-submitted bug reports
Logs from affected systems Once the root cause is identified, a fix is developed and incorporated into the upgrade packages. The phased rollout then restarts from the pilot phase to ensure the fix resolves the issue.

In some cases, issues may only affect certain hardware configurations or software combinations. In these situations, the phased rollout may continue for unaffected users while a fix is developed in parallel.

Communication is Key

Throughout the upgrade rollout, clear communication with users is essential. The Ubuntu team provides regular status updates through:

Official Ubuntu blogs and social media channels
Notifications within the Software Updater application
Release notes and known issues in the documentation Users are encouraged to provide feedback and report issues through the appropriate support channels. This helps ensure that any problems are identified and resolved quickly.

Completing the Upgrade

Once the phased rollout reaches 100% availability, the upgrade is considered complete. At this point, the focus shifts to ongoing support and maintenance.

Users who may have deferred the upgrade are notified through Software Updater. Release notes and upgrade guides are promoted to help users with the transition.

For several weeks after the upgrade completion, the Ubuntu team continues to monitor Error Tracker reports closely. Any post-upgrade issues are prioritized for investigation and resolution.

Continuous Improvement

The phased update process is not a static one. With each major release and upgrade cycle, the Ubuntu team learns and adapts.

Retrospectives

After each upgrade, the team conducts a thorough retrospective to identify:

What went well during the upgrade process
Pain points and areas for improvement
Specific issues that could have been caught earlier
Ways to streamline the rollout process Insights from these retrospectives are incorporated into the planning for the next release cycle.

Automated Testing

A key focus area for continuous improvement is expanding the breadth and depth of automated testing. This includes:

Unit testing of individual packages and components
Integration testing of package combinations
Upgrade testing from various starting configurations
Hardware compatibility testing By continually investing in automated testing, Ubuntu can catch more potential issues before they reach users.

Phased Update Refinement

The phased update process itself is also subject to ongoing refinement. This may include:

Adjusting the percentages and durations of each rollout phase
Improving the algorithms used for user grouping and rollout targeting
Enhancing the Error Tracker system to provide more actionable insights
Streamlining the rollback and re-start process when issues are encountered By continuously iterating on the phased update approach, Ubuntu can deliver an ever-smoother upgrade experience.

Looking to the Future

As Ubuntu continues to evolve and grow, phased updates will play an increasingly critical role in delivering a stable, reliable operating system to millions of users worldwide.

Expanding Phased Updates

While phased updates are currently used primarily for major release upgrades, there is potential to expand their use to other types of updates, such as:

Critical security patches
Hardware enablement kernels
Major application updates By applying the phased approach more broadly, Ubuntu can bring the benefits of gradual rollout and early issue detection to more areas of the system.

Integration with Other Tools

Phased updates can also be more tightly integrated with other Ubuntu tools and services, such as:

Canonical Livepatch for real-time kernel patching
Landscape for managed updates in enterprise environments
Ubuntu Pro for extended security maintenance By providing a cohesive update experience across all these tools, Ubuntu can further simplify and streamline the process for users and administrators.

Empowering the Community

As an open source project, Ubuntu has a vibrant community of developers, testers, and enthusiasts. Phased updates provide an opportunity to further empower and engage this community.

For example, the Ubuntu team could:

Provide more transparency into the phased update process and criteria
Engage community testers to provide early feedback on updates
Collaborate with community developers on testing and bug fixing
Incorporate community feedback into the planning and prioritization of updates By working hand-in-hand with the community, Ubuntu can ensure that phased updates are meeting the needs of its diverse user base.