This is the official wiki for the NixOS release process. All release managers should document how they're actively changing the release process here. And also use as it as a reference on how to release NixOS and engage the community.

This is not a set of rules but rather what the previous release managers did. It mainly serves as a list of tasks to remember and is not written in stone. This means that if you are a Release Manager and think you can do thing better, feel free to do so! Also, if you are not a Release Manager and think the process can be improved, open an Issue or Pull Request on the repository.

Roles

The release process currently knows two kinds of roles, whose duties should be documented in this wiki.

• Release Managers
• Release Editors

Both roles have been filled by two persons each since the 22.11 release.

One person will have already worked on the previous release, while the other one will be new to the release process, and can bring in new ideas, while benefitting from the knowledge transfer.

The process to find new team member should be started at least two and half months before the next release, so that the schedule documented in this wiki can be followed. This "Call for Release-Manager and Release -Editor" should be posted with sufficient spare time to the Discourse.

Examples:

• 22.11 Call for Release Manager and Editor
• 22.05 Call for Release Managers and Editor
• 21.11 Call for Release Manager

Release Managers

When you are nixpkgs committer, you already have the skillset necessary for release management. You just need to apply it in a different way.

— Jon Ringer

For each release there are two release managers. After each release the release manager having managed two releases steps down and the release management team of the last release appoints a new release manager.

This makes sure a release management team always consists of one release manager who already has managed one release and one release manager being introduced to their role, making it easier to pass on knowledge and experience.

Release managers for the current NixOS release are tracked by GitHub team @NixOS/nixos-release-managers.

A release manager’s role and responsibilities are:

• manage the release process

• start discussions about features and changes for a given release

• create a roadmap

• decide which bug fixes, features, etc… get backported after a release

This team has special write permissions to certain repositories in NixOS. It is very important that the new co-release-manager removes the previous manager from the team when they assign a new one.

A release manager must have the commit bit in the nixpkgs repository and must not be part of an ongoing conflict, that would impact the execution of their duties in this role.

Release Editors

The role of the release editors is a result of an action item from the 21.11 release, where the satistfaction with the release notes was low.

The first release editor role was filled in the 22.05 release with a focus on the release notes and content for the homepage.

Responsibiltiies

• Improve the legibility of the release notes

• Correct grammar and spelling issues

Release Critical Packages

These packages are deemed "sufficiently complex" in the package ecosystem to warrant treating them differently when stabilizing the package set leading up to a stable release. Generally, these packages will be closely related to stdenv or fundamental to NixOS (e.g. systemd). The implications of RFC 0085 are that these packages can not have breaking changes applied to them for six weeks leading up to a release date. This delay allows for changes in these packages to be sufficiently stabilized through usage on unstable, and allows for many staging iterations for these fixes to be applied.

List of Release Critical Packages:

• binutils
• gcc
• glibc
• llvm
• systemd

Process for Modifying Release Critical Packages

A release retrospective will take place after a release occured; one of the topics of discussion will be to modify the list of critical packages. Packages may be added if they caused a significant amount of pain, or packages may be removed if their updates cause little to no pain. Pain in this case is defined as causing failing builds, regressions in downstream packages, or regressions in user experience.

Release Process

These pages give an overview of the release process with as many details as there are available at the time of writing.

Release Schedule

With RFC85, the release schedule is largely already dictated below:

As Nixpkgs is a project with contributors spanning across many timezones, it is best to make the dates go into effect in accordance to the Anywhere on Earth (AoE) timezone.

Clarification of terms:

The purpose of these changes are mainly to push "riskier" PRs to being merged earlier, or after a release. Merging pull-requests which may have many downstream breakages should be avoided right before ZHF, as ZHF should focus on stabilization of the nixpkgs package set, and not distracted by large noise-to-signal build failures. Also, fixes to packages which required the staging process will likely require additional staging cycles to remedy the regressions.

What to do once appointed

You have been appointed as the new Release Manager? Congratulations! This is a loose list of things you could start doing now. These tasks are not urgent but can be done as early as the appointment which should start to get you into your new role. While not urgent, these task need to be completed at some point before the release process starts. The release manager of the previous release will get in contact with you and help you along the way.

This is not an actual part of the release process but rather preparations for starting the process.

• Get in contact with the previous release manager and schedule a meeting
• Join the release management room on Matrix
  • Request moderator permissions on the room, this allows to manage the topic and create video calls
• Gain access to the release-manager GitHub Team by asking any of the existing people with "Manager" rights
  • Also make sure there are only people left as members who are managing this release
• Gain access to the nixos-release Discourse Team
  • Your partner should have the permissions to invite you. Make sure to remove previous members, who are not active anymore.
• Update the NixOS Release Team members on the homepage
  • You will be the primary contact for this release, your experienced team mate will be secondary.
• Ensure you have a Hydra account which will come in handy
  • Ensure that all release managers have at least restart-jobs, bump-to-front, and cancel-build privileges (see the top-right "Preferences" button). If they are missing, reach out to the Infrastructure team
• Create a blockers project in the organization for your release and the next one
  • Also close old ones while you're at it and consider migrating remaining open issues to the current project
• Figure out a codename for the next release (not the one you are releasing but the next one) and write it down somewhere for the branch-off
• Find someone to create a logo or create one yourself for the current release and PR it into the repository
  • Usually done by creating an issue on the nixos-artwork repository pinging artists who created previous logos
• Check if the process can be improved this release
  • Find comments on the Discourse posts, GitHub issues, … of the previous release and see if the comments suggest doing something different this time
  • Read the Discourse post of the last retrospective
  • Check the Issues and Pull Requests of the release-wiki repository
• Figure out concrete dates for the week offsets outlined in schedule
• Publish a "Let's have a great XX.XX release cycle!" post on Discourse, outlining the plans for this release. Examples: 22.05
  • This links to the release schedule issue (examples: 22.11, 22.05) which must be created before posting to Discourse

Feature Freeze announcement and pre-release cleanup

The feature freeze announcement should be done 8 weeks before the release to inform subsystem maintainers about the upcoming feature freeze and to discuss potential blockers in the respective subsystems. This announcement is done on GitHub (22.05 example). It's also a good idea to cross-post this to Discourse to raise awareness about the upcoming freeze (22.05 example).

People and teams will be pinged in this issue. The list of people and teams pinged is generated from the teams list in nixpkgs. The script to generate the pings is: ./maintainers/feature-freeze-teams.pl in nixpkgs. The script needs a GitHub username and a token to read the teams from the NixOS org. Instructions are provided as a comment in the script.

You might need to add a comment that pings people again because GitHub seems to limit the amount of people that can be pinged per post.

Pre-release cleanup

The pre-release cleanup consists of multiple small cleanups to keep the release secure throughout its lifecycle while also maintaining some level of cleanliness.

• Remove attributes that we know we will not be able to support, especially if there is a stable alternative. E.g. Check that our Linux kernels’ projected end-of-life are after our release projected end-of-life.

  Many EOL dates are also tracked by the endoflife.date project and some packages might need closer supervision in that regard.

  The "Picking up the garbage" project tracks problematic packages that should be removed.

• Attributes that can also be safely removed are packages that were broken for more than 2 years.

ZERO Hydra Failures

Hydra scaling

The activity during ZHF is usually higher, so more build capacity is a good investment, to keep up with the pace of contributions.

There is an autoscaler in place for aarch64-linux and x86_64-linux while the darwin allocations are pretty much static.

If capacity issues are apparent, contact the infrastructure team on the infra repository (example: 21.11 example), and start a discussion in #infra:nixos.org.

ZHF issue

• Create an issue for tracking Zero Hydra Failures progress. ZHF is an effort to get build failures down to zero. Examples: 22.11, 22.05

• The nixpkgs/trunk jobset is ideal to track regressions, since it builds for all platforms.

• Subscribe to the issue and monitor the conversation for new blockers.

• Get in contact with the staging team through their Matrix room at #staging:nixos.org to coordinate the staging-next merges.

Branch-off

For these steps "24.05" represents the current release tag and "24.11" represents the next (6 months in the future) release tag. "23.11" is the last release which was released 6 months ago.

Jobset creation

The infrastructure team can create the necessary jobsets in advance. They will not evaluate because the release branch doesn't exist but it allows the infrastructure team and the release team to work more asynchronously. Reach out to them 1-2 days before release to create the necessary Hydra Jobsets. You can link them this section.

• https://hydra.nixos.org/project/nixos
  • release-24.05
  • release-24.05-small
• https://hydra.nixos.org/project/nixpkgs
  • nixpkgs-24.05-darwin
  • staging-next-24.05

Example configuration: nixos:release-23.11

Inputs:

Actual branch-off

Set NEWVER to the new release version:

export NEWVER=24.05

On the master branch

Pull in the final changes before performing the actal branch-off.

1. Wait for the staging team to merge the final staging-next iteration

2. Fetch and check out the master branch

3. Create the release branch:

   git switch -c release-$NEWVER
   

On the release branch

Update metadata on the release branch, create its staging branches and tag the release.

1. Update the system.defaultChannel attribute in nixos/modules/config/nix-channel.nix

2. Update the versionSuffix attribute in nixos/release.nix

   To get the commit count, use the following command:

   git rev-list --count release-$NEWVER
   

3. Add SUPPORT_END=YYYY-MM-DD to osReleaseContents in nixos/modules/misc/version.nix.

4. Commit the changes from the previous steps

   git commit -m "$NEWVER beta release" -S
   

5. Create the staging branches

   git branch staging-$NEWVER
   git branch staging-next-$NEWVER
   

6. Tag the release and push everything

   git tag --annotate --message="Release $NEWVER-beta" $NEWVER-beta
   git push upstream master release-$NEWVER $NEWVER-beta staging-$NEWVER staging-next-$NEWVER
   

7. Create jobsets on hydra by contacting the infrastructure team and start the evaluation on all new jobsets.

8. Switch back to the master branch

   git switch master
   

Back on the master branch

Now we prepare the master branch for the next release after this one.

1. Update the periodic-merge workflow to include the new release.

2. Increment the .version file in the repository root.

   # The release after $NEWVER (24.05 -> 24.11)
   echo -n "24.11" > .version
   

3. Update the codeName attribute in lib/trivial.nix This will be the name for the next release.

4. Create a new release notes file for the next release

5. Update the release versions in .github/PULL_REQUEST_TEMPLATE.md on master.

6. Update CONTRIBUTING.md on master.

7. Commit the changes (23.05 example + this commit + this commit)

8. Tag the master branch, so that git describe shows the new version as the base for commits.

   git tag --annotate 24.11-pre
   git push upstream master 24.11-pre
   git describe HEAD # should yield 24.11-pre
   

And afterwards

Now that everything on git is done, we are still missing the channels.

1. Create the necessary channels for https://channels.nixos.org in beta status, by updating channels.nix in infra and nag the infrastructure team to get these changes deployed.

   Example: 22.11

2. Create the backport labels for all new branches:

   • backport staging-24.05
   • backport release-24.05

   Use the description Backport PR automatically and the color value #0fafaa

3. Update the ZHF issue, that now that the branch-off has been performed, fixes have to be backported. Examples: 22.05

Once the channel is available

The following steps should be done after the channels have become available on channels.nixos.org.

1. Update the flake input on the nixos-search repository, and create a pull request:

   git clone git@github.com:nixos/nixos-search
   nix --extra-experimental-features "nix-command flakes" flake lock --update-input nixos-infra
   

2. Give the Marketing team a heads-up about the upcoming release

3. Get in contact with Arian van Putten

   1. They will need to update upload-legacy-ami.yml to point to the new jobset. This will automatically upload the AMIs from the jobset to AWS
   2. A PR is also welcome :)

4. Make sure the release editors have started finalizing the release notes. Only 7 days left until release!

Beta Phase

• Monitor the master branch for bugfixes and minor updates and cherry-pick them to the release branch.

• Start marking packages as broken. There is no fully defined process to do this, so look at the corresponding issue and also this discussion.

Final Release

Set the following environment variables to execute the commands below

# The version of the previous stable release
export OLDVER=22.11
# The version that is about to be released
export NEWVER=23.05

Before the final release

• Two days before expected release date, change stableBranch to true in Hydra and wait for the channel to update

• Re-check that the release editors have completed their work on the release notes

• Merge the Pull Request that marks broken packages as broken

• Update README.md on master with new stable NixOS version information. Then backport the change to the release branch.

• Update Upgrading NixOS section of the manual to match new stable release version and backport this as well. This way the manual will already reflect the new version on release. Example: 22.05

• Ensure the installer images are sufficiently tested on both aarch64 and x86_64 linux:

  • Graphical (GNOME/KDE)
  • Minimal

  All ISO images can be found on the nixos/release-$NEWVER hydra jobset.

  If you don't have any hardware to test the aarch64-linux on, QEMU emulation should allow to boot the images in a reasonable time to check the installer starts and is usable.

• Gather some information about the release for the final announcement

  • Number of non-merge commits for the release:

    git log upstream/release-$OLDVER..upstream/release-$NEWVER --no-merges --format=%an | wc -l
    

  • Number of contributors for this release:

    git log upstream/release-$OLDVER..upstream/release-$NEWVER --no-merges --format=%an | sort | uniq | wc -l
    

  • New/updated/removed packages:

    git switch release-$OLDVER
    NIX_PATH= nix-env -f $PWD -qaP --json --out-path > old.json
    
    git switch release-$NEWVER
    NIX_PATH= nix-env -f $PWD -qaP --json --out-path > new.json
    

    #!/usr/bin/env python3
    
    import json
    
    with open('old.json') as f:
        old = json.load(f)
    with open('new.json') as f:
        new = json.load(f)
    
    n_removed = 0
    n_new = 0
    n_updated = 0
    for pkgname,pkg in new.items():
        if pkgname not in old:
            n_new += 1
            continue
        if pkg["version"] != old[pkgname]["version"]:
            n_updated += 1
    for pkgname in old.keys():
        if pkgname not in new:
            n_removed += 1
    
    print(f"New: {n_new}")
    print(f"Rem: {n_removed}")
    print(f"Upd: {n_updated}")
    

    Best to check how the previous announcement post was formulated to see what needs to be included.

  • Added/removed modules:

    git diff release-$OLDVER..release-$NEWVER nixos/modules/module-list.nix | grep ^+ | wc -l
    git diff release-$OLDVER..release-$NEWVER nixos/modules/module-list.nix | grep ^- | wc -l
    

  • Added/removed options:

    Download the result of the nixos.options job in Hydra from both your release and the previous release, then do:

    # removed options:
    comm -23 <(jq -r 'keys[]' old.json | sort) <(jq -r 'keys[]' new.json | sort) | wc -l
    # new options:
    comm -13 <(jq -r 'keys[]' old.json | sort) <(jq -r 'keys[]' new.json | sort) | wc -l
    

At final release time

On the master branch

1. Create these PRs on master and backport to the release branch:

   1. Update rl-$NEWVER.section.md with the final release date.

   2. Commit and push all changes.

      git commit -m "Release NixOS $NEWVER" -S
      git push upstream master
      

On the release branch

1. Switch to the release branch

   git switch release-$NEWVER
   

2. Cherry-pick the release commit from master

3. Push the commit to the release branch

   git push
   

4. Find the commit id and tag the release on the release branch:

   git tag --annotate --message="Release $NEWVER" "branch-off-$NEWVER" <COMMIT_ID>
   git push upstream "branch-off-$NEWVER"
   

5. Update nixos-homepage for the release.

   Examples: 24.05

   This step requires the released ISOs, from the hydra evaluation with stableBranch enabled, to be available.

   1. Update the flake.nix input released-nixpkgs-stable to $NEWVER and then run:

   2. Run nix flake update (this updates flake.lock to updated channel).

   3. Add a compressed version of the NixOS logo (24.05 example). The logo should have a width of 100px.

   4. Write the announcement under src/content/blog/announcements/ (24.05 example).

   5. Create a pull request and poke the marking team. Make sure to push to a branch on the nixos-homepage repository, not your own fork, so the CI actions work properly.

   The website is hosted by a CDN so you may occasionally see the old site for a couple of minutes/hours (?) after your changes

6. Update infra to reflect the stable channel status for $NEWVER and the deprecated status for $OLDVER.

   Examples: 22.11, 22.05, 21.11

7. Create a new topic on Discourse to announce the release.

   Examples: 22.11, 22.05

8. Once the Pull Request in infra is merged, update nixos-search to mark the channel as released. This is the same process as for the creation of the beta channel in the project.

After Release

1. Add the new release to repology. Don't remove old releases, they want to keep them around.

   Examples: 22.11, 22.05

2. Update osinfo-db entries.

   Examples: 22.11, 22.05, 21.11

   nix-shell -p "python3.withPackages (p: [ p.lxml p.requests ])" -p cdrkit
   ./scripts/updates/nixos.py --release YY.MM --codename <codename> --release-date YYYY-MM-DD --next-release YY.MM
   git add .
   git commit -s -m "nixos: Add YY.MM release"
   
   # run tests to validate your changes
   nix-shell -p "python3.withPackages (p: [ p.lxml p.requests p.pytest ])" -p cdrkit osinfo-db-tools gettext --run "make check"
   

3. Check that the AMI image references have been updated in nixos/modules/virtualisation/amazon-ec2-amis.nix.

4. Close the milestone

5. Close the blockers project

6. Close all issues you opened for the release and unpin them if necessary

7. Ask someone with the appropriate permissions to update the description of the NixOS and NixOS Dev Matrix rooms to include the new release

8. Create an announcement for the retrospective, some days after the old channel is discontinued.

   Examples: 22.11, 22.05

Retrospective

The retrospective is an important part of the release process where we give every member of the community the chance to give feedback about their experience, what went well and what did not.

After the release

1. Set a date and time one week after the release date. This is still early enough, so people will remember things about the release, but late enough, so they will have had time to reflect. An hour is usually sufficient for this.

   Make sure the time slot works for people from the EU and US, where most of the current contributors reside.

2. Plan a venue. We usually rely on Jitsi Meet for conversations that include audio/video as well as screensharing.

3. Create the invitation post over on the Discourse. Examples:

   • NixOS 22.11 Retrospective
   • 22.05 Retrospective
   • 21.11 Retrospective

During the retrospective

1. Provide a collaborative document, that participants can edit. It should allow adding and classifying discussion items. In recent retrospectives we classified things as good/neutral/bad.

   Examples:

   • 22.11 Retrospective

2. Put up limits for how long a single discussion point can take place, so every topic can be addressed.

3. Identify action items from the discussion points and identify people who would want to follow up on those.

After the retrospective

1. Post the conclusions onto the Discourse thread. Also link the notes that were made during the event.

2. Follow up on assigned action items.

EOL cleanup

The old release reaches its end of life one month after the new release gets released. At that point a few cleanup tasks need to be done.

1. Set the EOL channel status from deprecated to unmaintained in infra. Examples:

   • channels: Deprecate NixOS 22.05
   • channels: Mark 21.11 as unmaintained
   • channels:21.05 is unmaintained

2. Once this is merged, update the nixos-search repository to reflect that status on search.nixos.org.

   nix --extra-experimental-features "nix-command flakes" flake lock --update-input nixos-infra
   

3. Close all pull requests that target one of the the old release- or staging-branches

4. Prune old backport labels

5. Remove the old release from the periodic merge workflows

6. Increase the oldestSupportedRelease in lib/trivial.nix to match the oldest supported release.

   Examples: 22.05

Handover to the next team

Some time after your EOL cleanup, publish a new topic to Discourse, asking for a new editor and a new manager to step up. You select the team.

It's often a good idea to schedule a meeting with the upcoming team to get to know each other.

Contrib

This section holds scripts that can be helpful in the context of the release, but are optional and would likely impact the legibility of the documentation if included inline.

QEMU aarch64 emulation

The following script should allow booting the aarch64-linux UEFI installer ISOs provided through the nixos:unstable-small and nixos:trunk jobsets, when no native hardware support exists.

Pass the ISO file as the first argument, and everything should boot up.

#!/usr/bin/env nix-shell
#!nix-shell -i bash -p qemu_full virt-viewer
# shellcheck shell=bash

set -euo pipefail

BASEDIR=$(dirname "$0")

VMNAME=nixos-aarch64
ISO="${BASEDIR}/$1"
NVRAM="${BASEDIR}/vars-pflash.raw"
CORES=6
MEMORY=4096
KEYBOARD="de"

cleanup() {
	kill "$QEMU_PID" || true
}

trap cleanup EXIT

run_vm() {
	OVMF_FD=$(nix-build '<nixpkgs>' --no-out-link -A OVMF.fd --system aarch64-linux)
	test -f "${NVRAM}" || {
		cp "${OVMF_FD}/AAVMF/vars-template-pflash.raw" "${NVRAM}"
		chmod u+w "${NVRAM}"
	}

	qemu-system-aarch64 \
		-name $VMNAME,process=$VMNAME \
		-machine virt,gic-version=max \
		--accel tcg,thread=multi \
		-cpu max \
		-smp $CORES \
		-m $MEMORY \
		-k $KEYBOARD \
		-serial stdio \
		-drive if=pflash,format=raw,file="${OVMF_FD}/AAVMF//QEMU_EFI-pflash.raw",readonly=on \
		-drive if=pflash,format=raw,file="${NVRAM}" \
		-device virtio-scsi-pci \
		-device virtio-gpu-pci \
		-device virtio-net-pci,netdev=wan \
		-netdev user,id=wan \
		-device virtio-rng-pci,rng=rng0 \
		-object rng-random,filename=/dev/urandom,id=rng0 \
		-device virtio-serial-pci \
		-drive file="${ISO}",media=cdrom \
		-boot d

	QEMU_PID=$!
}

run_vm