Introduction

About Hydra

Hydra is a tool for continuous integration testing and software release that uses a purely functional language to describe build jobs and their dependencies. Continuous integration is a simple technique to improve the quality of the software development process. An automated system continuously or periodically checks out the source code of a project, builds it, runs tests, and produces reports for the developers. Thus, various errors that might accidentally be committed into the code base are automatically caught. Such a system allows more in-depth testing than what developers could feasibly do manually:

• Portability testing: The software may need to be built and tested on many different platforms. It is infeasible for each developer to do this before every commit.
• Likewise, many projects have very large test sets (e.g., regression tests in a compiler, or stress tests in a DBMS) that can take hours or days to run to completion.
• Many kinds of static and dynamic analyses can be performed as part of the tests, such as code coverage runs and static analyses.
• It may also be necessary to build many different variants of the software. For instance, it may be necessary to verify that the component builds with various versions of a compiler.
• Developers typically use incremental building to test their changes (since a full build may take too long), but this is unreliable with many build management tools (such as Make), i.e., the result of the incremental build might differ from a full build.
• It ensures that the software can be built from the sources under revision control. Users of version management systems such as CVS and Subversion often forget to place source files under revision control.
• The machines on which the continuous integration system runs ideally provides a clean, well-defined build environment. If this environment is administered through proper SCM techniques, then builds produced by the system can be reproduced. In contrast, developer work environments are typically not under any kind of SCM control.
• In large projects, developers often work on a particular component of the project, and do not build and test the composition of those components (again since this is likely to take too long). To prevent the phenomenon of “big bang integration”, where components are only tested together near the end of the development process, it is important to test components together as soon as possible (hence continuous integration).
• It allows software to be released by automatically creating packages that users can download and install. To do this manually represents an often prohibitive amount of work, as one may want to produce releases for many different platforms: e.g., installers for Windows and Mac OS X, RPM or Debian packages for certain Linux distributions, and so on.

In its simplest form, a continuous integration tool sits in a loop building and releasing software components from a version management system. For each component, it performs the following tasks:

• It obtains the latest version of the component’s source code from the version management system.
• It runs the component’s build process (which presumably includes the execution of the component’s test set).
• It presents the results of the build (such as error logs and releases) to the developers, e.g., by producing a web page.

Examples of continuous integration tools include Jenkins, CruiseControl, Tinderbox, Sisyphus, Anthill and BuildBot. These tools have various limitations.

• They do not manage the build environment. The build environment consists of the dependencies necessary to perform a build action, e.g., compilers, libraries, etc. Setting up the environment is typically done manually, and without proper SCM control (so it may be hard to reproduce a build at a later time). Manual management of the environment scales poorly in the number of configurations that must be supported. For instance, suppose that we want to build a component that requires a certain compiler X. We then have to go to each machine and install X. If we later need a newer version of X, the process must be repeated all over again. An ever worse problem occurs if there are conflicting, mutually exclusive versions of the dependencies. Thus, simply installing the latest version is not an option. Of course, we can install these components in different directories and manually pass the appropriate paths to the build processes of the various components. But this is a rather tiresome and error-prone process.
• They do not easily support variability in software systems. A system may have a great deal of build-time variability: optional functionality, whether to build a debug or production version, different versions of dependencies, and so on. (For instance, the Linux kernel now has over 2,600 build-time configuration switches.) It is therefore important that a continuous integration tool can easily select and test different instances from the configuration space of the system to reveal problems, such as erroneous interactions between features. In a continuous integration setting, it is also useful to test different combinations of versions of subsystems, e.g., the head revision of a component against stable releases of its dependencies, and vice versa, as this can reveal various integration problems.

Hydra is a continuous integration tool that solves these problems. It is built on top of the Nix package manager, which has a purely functional language for describing package build actions and their dependencies. This allows the build environment for projects to be produced automatically and deterministically, and variability in components to be expressed naturally using functions; and as such is an ideal fit for a continuous build system.

About Us

Hydra is the successor of the Nix Buildfarm, which was developed in tandem with the Nix software deployment system. Nix was originally developed at the Department of Information and Computing Sciences, Utrecht University by the TraCE project (2003-2008). The project was funded by the Software Engineering Research Program Jacquard to improve the support for variability in software systems. Funding for the development of Nix and Hydra is now provided by the NIRICT LaQuSo Build Farm project.

About this Manual

This manual tells you how to install the Hydra buildfarm software on your own server and how to operate that server using its web interface.

License

Hydra is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Hydra is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Hydra at nixos.org

The nixos.org installation of Hydra runs at http://hydra.nixos.org/. That installation is used to build software components from the Nix, NixOS, GNU, Stratego/XT, and related projects.

If you are one of the developers on those projects, it is likely that you will be using the NixOS Hydra server in some way. If you need to administer automatic builds for your project, you should pull the right strings to get an account on the server. This manual will tell you how to set up new projects and build jobs within those projects and write a release.nix file to describe the build process of your project to Hydra. You can skip the next chapter.

If your project does not yet have automatic builds within the NixOS Hydra server, it may actually be eligible. We are in the process of setting up a large buildfarm that should be able to support open source and academic software projects. Get in touch.

Hydra on your own buildfarm

If you need to run your own Hydra installation, the installation chapter explains how to download and install the system on your own server.

Installation

This chapter explains how to install Hydra on your own build farm server.

Prerequisites

To install and use Hydra you need to have installed the following dependencies:

• Nix

• PostgreSQL

• many Perl packages, notably Catalyst, EmailSender, and NixPerl (see the Hydra expression in Nixpkgs for the complete list)

At the moment, Hydra runs only on GNU/Linux (i686-linux and x86_64_linux).

For small projects, Hydra can be run on any reasonably modern machine. For individual projects you can even run Hydra on a laptop. However, the charm of a buildfarm server is usually that it operates without disturbing the developer’s working environment and can serve releases over the internet. In conjunction you should typically have your source code administered in a version management system, such as subversion. Therefore, you will probably want to install a server that is connected to the internet. To scale up to large and/or many projects, you will need at least a considerable amount of diskspace to store builds. Since Hydra can schedule multiple simultaneous build jobs, it can be useful to have a multi-core machine, and/or attach multiple build machines in a network to the central Hydra server.

Of course we think it is a good idea to use the NixOS GNU/Linux distribution for your buildfarm server. But this is not a requirement. The Nix software deployment system can be installed on any GNU/Linux distribution in parallel to the regular package management system. Thus, you can use Hydra on a Debian, Fedora, SuSE, or Ubuntu system.

Getting Nix

If your server runs NixOS you are all set to continue with installation of Hydra. Otherwise you first need to install Nix. The latest stable version can be found on the Nix web site, along with a manual, which includes installation instructions.

Installation

The latest development snapshot of Hydra can be installed by visiting the URL http://hydra.nixos.org/view/hydra/unstable and using the one-click install available at one of the build pages. You can also install Hydra through the channel by performing the following commands:

nix-channel --add http://hydra.nixos.org/jobset/hydra/master/channel/latest
nix-channel --update
nix-env -i hydra

Command completion should reveal a number of command-line tools from Hydra, such as hydra-queue-runner.

Creating the database

Hydra stores its results in a PostgreSQL database.

To setup a PostgreSQL database with hydra as database name and user name, issue the following commands on the PostgreSQL server:

createuser -S -D -R -P hydra
createdb -O hydra hydra

Note that $prefix is the location of Hydra in the nix store.

Hydra uses an environment variable to know which database should be used, and a variable which points to a location that holds some state. To set these variables for a PostgreSQL database, add the following to the file ~/.profile of the user running the Hydra services.

export HYDRA_DATABASE_URL="postgres://hydra@dbserver.example.org/hydra"
export HYDRA_DATA=/var/lib/hydra

You can provide the username and password in the file ~/.pgpass, e.g.

dbserver.example.org:*:hydra:hydra:password

Make sure that the HYDRA_DATA directory exists and is writable for the user which will run the Hydra services.

Having set these environment variables, you can now initialise the database by doing:

hydra-init

To create projects, you need to create a user with admin privileges. This can be done using the command hydra-create-user:

$ hydra-create-user alice --full-name 'Alice Q. User' \
    --email-address 'alice@example.org' --password-prompt --role admin

Additional users can be created through the web interface.

Upgrading

If you’re upgrading Hydra from a previous version, you should do the following to perform any necessary database schema migrations:

hydra-init

Getting Started

To start the Hydra web server, execute:

hydra-server

When the server is started, you can browse to http://localhost:3000/ to start configuring your Hydra instance.

The hydra-server command launches the web server. There are two other processes that come into play:

• The evaluator is responsible for periodically evaluating job sets, checking out their dependencies off their version control systems (VCS), and queueing new builds if the result of the evaluation changed. It is launched by the hydra-evaluator command.
• The queue runner launches builds (using Nix) as they are queued by the evaluator, scheduling them onto the configured Nix hosts. It is launched using the hydra-queue-runner command.

All three processes must be running for Hydra to be fully functional, though it’s possible to temporarily stop any one of them for maintenance purposes, for instance.

Optionally, the new and still experimental hydra-ad-hoc command runs a nix daemon endpoint through which builds can be submitted to Hydra from outside, without any jobset. See Ad hoc builds.

Configuration

This chapter is a collection of configuration snippets for different scenarios.

The configuration is parsed by Config::General which has a pretty thorough documentation on their file format. Hydra calls the parser with the following options:

• -UseApacheInclude => 1
• -IncludeAgain => 1
• -IncludeRelative => 1

Including files

hydra.conf supports Apache-style includes. This is IMPORTANT because that is how you keep your secrets out of the Nix store. Hopefully this got your attention 😌

This:

<github_authorization>
NixOS = Bearer gha-secret😱secret😱secret😱
</github_authorization>

should NOT be in hydra.conf.

hydra.conf is rendered in the Nix store and is therefore world-readable.

Instead, the above should be written to a file outside the Nix store by other means (manually, using Nixops’ secrets feature, etc) and included like so:

Include /run/keys/hydra/github_authorizations.conf

Serving behind reverse proxy

To serve hydra web server behind reverse proxy like nginx or httpd some additional configuration must be made.

Edit your hydra.conf file in a similar way to this example:

using_frontend_proxy 1
base_uri example.com

base_uri should be your hydra servers proxied URL. If you are using Hydra nixos module then setting hydraURL option should be enough.

You also need to configure your reverse proxy to pass X-Request-Base to hydra, with the same value as base_uri. This also covers the case of serving Hydra with a prefix path, as in http://example.com/hydra.

For example if you are using nginx, then use configuration similar to following:

server {
    listen 433 ssl;
    server_name example.com;
    .. other configuration ..
    location /hydra/ {

        proxy_pass http://127.0.0.1:3000/;

        proxy_set_header  Host              $host;
        proxy_set_header  X-Real-IP         $remote_addr;
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
        proxy_set_header  X-Request-Base    /hydra;
    }
}

Note the trailing slash on the proxy_pass directive, which causes nginx to strip off the /hydra/ part of the URL before passing it to hydra.

Populating a Cache

A common use for Hydra is to pre-build and cache derivations which take a long time to build. While it is possible to direcly access the Hydra server’s store over SSH, a more scalable option is to upload built derivations to a remote store like an S3-compatible object store. Setting the store_uri parameter will cause Hydra to sign and upload derivations as they are built:

store_uri = s3://cache-bucket-name?compression=zstd&parallel-compression=true&write-nar-listing=1&ls-compression=br&log-compression=br&secret-key=/path/to/cache/private/key

This example uses Zstandard compression on derivations to reduce CPU usage on the server, but Brotli compression for derivation listings and build logs because it has better browser support.

See nix help stores for a description of the store URI format.

Statsd Configuration

By default, Hydra will send stats to statsd at localhost:8125. Point Hydra to a different server via:

<statsd>
  host = alternative.host
  port = 18125
</statsd>

hydra-notify’s Prometheus service

hydra-notify supports running a Prometheus webserver for metrics. The exporter does not run unless a listen address and port are specified in the hydra configuration file, as below:

<hydra_notify>
  <prometheus>
    listen_address = 127.0.0.1
    port = 9199
  </prometheus>
</hydra_notify>

hydra-queue-runner’s Prometheus service

hydra-queue-runner supports running a Prometheus webserver for metrics. The exporter’s address defaults to exposing on 127.0.0.1:9198, but is also configurable through the hydra configuration file and a command line argument, as below. A port of :0 will make the exposer choose a random, available port.

queue_runner_metrics_address = 127.0.0.1:9198
# or
queue_runner_metrics_address = [::]:9198

$ hydra-queue-runner --prometheus-address 127.0.0.1:9198
# or
$ hydra-queue-runner --prometheus-address [::]:9198

Overflow binary cache (optional)

The queue runner can upload builds of selected jobsets to a separate “overflow” S3 bucket, for example to keep large staging rebuilds out of the main cache. A step is uploaded to the overflow bucket only when every jobset referencing it is listed. Shared steps go to the default bucket.

Both buckets must live on the same S3 endpoint and use static credentials: when a later build from a regular jobset needs outputs that only exist in the overflow bucket, the queue runner copies them back to the default bucket server-side instead of rebuilding.

Configured in queue-runner.toml:

[overflowStore]
store = "s3://hydra-overflow?region=eu-west-1"
jobsets = ["nixpkgs:staging-next"]

Ad hoc builds (optional, experimental)

Hydra normally only builds what its evaluator queues. The optional hydra-ad-hoc service, new and still experimental, additionally lets a Nix client submit builds directly. It serves the nix daemon protocol on a Unix socket, so from the client’s point of view the whole Hydra deployment is one giant nix daemon. A derivation realised through it is filed as a Hydra build under a hidden adhoc/adhoc jobset and built by the queue runner and its builders. Read operations and store uploads are proxied to the upstream nix-daemon.

Nothing inside Hydra uses this socket. It exists for ad hoc jobs and ad hoc store usage from outside, and can be left disabled. Expect its interface and limitations to change.

On NixOS, enable it alongside the queue runner:

{
  services.hydra-ad-hoc-dev.enable = true;
}

This creates /run/hydra-ad-hoc/socket, with a hydra-ad-hoc group of its own. Add the users who may submit builds to that group. It is separate from hydra, the group Hydra’s services run as, on purpose: this socket is for use from outside Hydra.

Be aware of what membership grants. The daemon is a trusted user of the upstream nix-daemon, and it forwards uploads and build requests on a client’s behalf without any checks of its own yet. So, until the daemon gains its own access control, anyone in the hydra-ad-hoc group is effectively a trusted Nix user on the coordinator, and the group should be handed out as carefully as trusted-users.

The service reads /etc/hydra/ad-hoc.toml, generated from services.hydra-ad-hoc-dev.settings (dbUrl, maxDbConnections, upstreamSocket, storeDir, hydraDataDir). The defaults suit a single-host install.

A client then points its store at the socket:

$ nix-store --store unix:///run/hydra-ad-hoc/socket --realise /nix/store/...-hello.drv

The .drv must be in the coordinator’s store; a nix-build of an expression uploads it through the daemon as part of instantiation. Content-addressed and dynamic derivations work too: the daemon answers output-path queries from what the queue runner recorded, and builds a dynamic derivation’s producer first.

While the client waits, the daemon streams the build’s progress back to it the way a local build would report it. Each build step the queue runner dispatches appears as a build activity, and the step’s log lines follow as they are written. So nix build -L shows the log live, and --log-format bar-with-logs does the same for the classic commands. The daemon reads the logs from hydraDataDir, so it has to run on the same host as the queue runner.

Using LDAP as authentication backend (optional)

Instead of using Hydra’s built-in user management you can optionally use LDAP to manage roles and users.

This is configured by defining the <ldap> block in the configuration file. In this block it’s possible to configure the authentication plugin in the <config> block. All options are directly passed to Catalyst::Authentication::Store::LDAP. The documentation for the available settings can be found here.

Note that the bind password (if needed) should be supplied as an included file to prevent it from leaking to the Nix store.

Roles can be assigned to users based on their LDAP group membership. For this to work use_roles = 1 needs to be defined for the authentication plugin. LDAP groups can then be mapped to Hydra roles using the <role_mapping> block.

Example configuration:

<ldap>
  <config>
    <credential>
      class = Password
      password_field = password
      password_type = self_check
    </credential>
    <store>
      class = LDAP
      ldap_server = localhost
      <ldap_server_options>
        timeout = 30
      </ldap_server_options>
      binddn = "cn=root,dc=example"
      include ldap-password.conf
      start_tls = 0
      <start_tls_options>
        verify = none
      </start_tls_options>
      user_basedn = "ou=users,dc=example"
      user_filter = "(&(objectClass=inetOrgPerson)(cn=%s))"
      user_scope = one
      user_field = cn
      <user_search_options>
        deref = always
      </user_search_options>
      # Important for role mappings to work:
      use_roles = 1
      role_basedn = "ou=groups,dc=example"
      role_filter = "(&(objectClass=groupOfNames)(member=%s))"
      role_scope = one
      role_field = cn
      role_value = dn
      <role_search_options>
        deref = always
      </role_search_options>
    </store>
  </config>
  <role_mapping>
    # Make all users in the hydra_admin group Hydra admins
    hydra_admin = admin
    # Allow all users in the dev group to eval jobsets, restart jobs and cancel builds
    dev = eval-jobset
    dev = restart-jobs
    dev = cancel-build
  </role_mapping>
</ldap>

Then, place the password to your LDAP server in /var/lib/hydra/ldap-password.conf:

bindpw = the-ldap-password

Debugging LDAP

Set the debug parameter under ldap.config.ldap_server_options.debug:

<ldap>
  <config>
    <store>
      <ldap_server_options>
        debug = 2
      </ldap_server_options>
    </store>
  </config>
</ldap>

Legacy LDAP Configuration

Hydra used to load the LDAP configuration from a YAML file in the HYDRA_LDAP_CONFIG environment variable. This behavior is deperecated and will be removed.

When Hydra uses the deprecated YAML file, Hydra applies the following default role mapping:

<ldap>
  <role_mapping>
    hydra_admin = admin
    hydra_bump-to-front = bump-to-front
    hydra_cancel-build = cancel-build
    hydra_create-projects = create-projects
    hydra_restart-jobs = restart-jobs
  </role_mapping>
</ldap>

Note that configuring both the LDAP parameters in the hydra.conf and via the environment variable is a fatal error.

Turning off Hydra’s own user management

By default anyone with a Hydra account can sign in with a user name and a password. Set

local_auth_enabled = 0

to turn that off, for a Hydra whose users are meant to come from somewhere else.

This does not affect LDAP, which is a separate backend that happens to use the same form: with LDAP configured, the form stays, labelled “Sign in with LDAP”, and works as before. It does not affect OIDC or GitHub either, which have endpoints of their own.

Turning off the password form leaves nobody able to sign in at all if no other method is configured, which on a private Hydra means nobody gets in; Hydra warns about that at startup.

Local accounts are not deleted by this and hydra-create-user still works, so switching it back on needs no changes to the users.

Single sign-on with OIDC

Hydra can delegate login to one or more OpenID Connect providers, configured in the <oidc> block:

<oidc>
  <provider authentik>
    display_name = "Authentik"
    discovery_url = "https://authentik.example.com/application/o/hydra/.well-known/openid-configuration"
    client_id = "hydra"
    client_secret_file = "/var/lib/hydra/secrets/authentik-client-secret"
    role_claim = "groups"
    <role_mapping>
      hydra-admins = admin
      hydra-builders = create-projects
      hydra-builders = eval-jobset
    </role_mapping>
  </provider>
</oidc>

The role_mapping block translates the values the IdP sends in its role claim into Hydra roles, so the two need not use the same names. Hydra validates the mapping at startup.

See the OIDC documentation for all settings, role handling, sign-out, and examples for Authentik and GitHub.

Webhook Authentication

Hydra supports authenticating webhook requests from GitHub and Gitea to prevent unauthorized job evaluations. Webhook secrets should be stored in separate files outside the Nix store for security using Config::General’s include mechanism.

In your main hydra.conf:

<webhooks>
  Include /var/lib/hydra/secrets/webhook-secrets.conf
</webhooks>

Then create /var/lib/hydra/secrets/webhook-secrets.conf with your actual secrets:

<github>
  secret = your-github-webhook-secret
</github>
<gitea>
  secret = your-gitea-webhook-secret
</gitea>

For multiple secrets (useful for rotation or multiple environments), use an array:

<github>
  secret = your-github-webhook-secret-prod
  secret = your-github-webhook-secret-staging
</github>

Important: The secrets file should have restricted permissions (e.g., 0600) to prevent unauthorized access. See the Webhooks documentation for detailed setup instructions.

Embedding Extra HTML

Embed an analytics widget or other HTML in the <head> of each HTML document via:

tracker = <script src="...">

Single sign-on with OIDC

Hydra can delegate login to one or more OpenID Connect providers, alongside the built-in accounts and LDAP or instead of them. Each provider gets a “Sign in with <display name>” entry in the sign-in menu, or a plain Sign in button when a provider is the only way to sign in. To make it the only way, set local_auth_enabled = 0 in hydra.conf as well; see Turning off Hydra’s own user management.

How a login works

Hydra runs the OIDC Authorization Code flow with PKCE (S256), state and nonce. It requests the scopes openid profile email plus extra_scopes. The login has to finish within 10 minutes.

Hydra validates the ID token’s signature against the provider’s JWKS, and checks iss, aud (your client_id), exp, nbf and nonce. An unknown key ID triggers one JWKS refetch, so key rotation needs no restart.

Hydra logs the user in as <provider>:<sub>, with the email and name claims from the ID token. Hydra updates both on every login, so changes at the provider carry over. Hydra’s pages show OIDC users by their email address, since sub is often an opaque ID. Hydra needs a usable email claim in the ID token and does not call the userinfo endpoint. If the token has email_verified: false, Hydra refuses the login. If allowed_domains is set, the address must be in one of those domains. OIDC users have no password, so they can only sign in through their provider.

Configuring a provider

Each provider is a <provider> block inside <oidc> in hydra.conf. The block name appears in the URLs and in usernames, so pick something stable.

<oidc>
  <provider authentik>
    display_name = "Authentik"
    discovery_url = "https://authentik.example.com/application/o/hydra/.well-known/openid-configuration"
    client_id = "hydra"
    client_secret_file = "/var/lib/hydra/secrets/authentik-client-secret"
  </provider>
</oidc>

Register https://<hydra>/oidc-callback/<provider> as the redirect URI at the IdP.

Hydra checks the configuration and reads client_secret_file at startup, so a new secret needs a restart. It fetches the discovery document on the first login, so Hydra starts even when the IdP is down. Hydra keeps the discovered endpoints until it restarts. Hydra caches the JWKS for a minute.

You can configure several providers. Each has its own role settings, and its users are separate Hydra users.

Roles

Hydra reads roles from the role_claim in the ID token and replaces the user’s roles with them on every login, including roles an admin set in Hydra. If the token has no role claim at all, Hydra leaves the user’s roles unchanged, so you can manage them in Hydra instead. The claim has to be in the ID token. Hydra ignores claims only available from the userinfo endpoint.

role_mapping translates the IdP’s values into Hydra roles. Hydra’s roles are admin, bump-to-front, cancel-build, create-projects, eval-jobset and restart-jobs. Repeat a key to grant several roles. Keys are case-sensitive, and values that are not keys grant nothing.

<provider authentik>
  # ...
  role_claim = "groups"
  <role_mapping>
    hydra-admins = admin
    hydra-builders = create-projects
    hydra-builders = eval-jobset
    hydra-operators = restart-jobs
  </role_mapping>
</provider>

Without a role_mapping, the claim must contain Hydra role names, and Hydra drops unknown values. Some IdPs, like kanidm, disallow dashes in claim values, so Hydra also accepts underscores, as in restart_jobs.

Getting the claim into the ID token depends on the IdP:

• Keycloak. Add a protocol mapper to the client, such as the built-in groups mapper with Add to ID token enabled.
• kanidm. Use kanidm system oauth2 update-claim-map hydra hydra_roles hydra_admins admin.
• Authentik. See the example below.

Sign-out

Signing out clears Hydra’s session. If the provider has an end_session_endpoint, Hydra also redirects there with client_id and post_logout_redirect_uri, which signs the user out of the IdP. kanidm and some other IdPs don’t offer one, so users stay signed in there.

Examples

Authentik

1. In Authentik, create a provider: Applications → Providers → Create → OAuth2/OpenID Connect.

   • Client type: Confidential
   • Redirect URI: https://hydra.example.com/oidc-callback/authentik, where authentik is the name of the <provider> block. It has to match exactly, including the scheme and any path prefix Hydra is served under.

2. Create an application that uses that provider.

3. Add the authentik default OAuth Mapping: OpenID ‘groups’ scope mapping to the provider.

4. Write the client secret, and nothing else, to /var/lib/hydra/secrets/authentik-client-secret.

5. In Hydra:

   <oidc>
     <provider authentik>
       display_name = "Authentik"
       discovery_url = "https://authentik.example.com/application/o/hydra/.well-known/openid-configuration"
       client_id = "the-client-id-from-authentik"
       client_secret_file = "/var/lib/hydra/secrets/authentik-client-secret"
       # Only needed if the provider does not include `groups` by default.
       extra_scopes = "groups"
       role_claim = "groups"
       <role_mapping>
         hydra-admins = admin
         hydra-builders = create-projects
         hydra-builders = eval-jobset
         hydra-operators = restart-jobs
         hydra-operators = cancel-build
         hydra-operators = bump-to-front
       </role_mapping>
     </provider>
   </oidc>
   

The groups claim contains every group the user is in, so give the Hydra groups names that won’t collide with your other Authentik groups.

To avoid depending on group names, write a scope mapping that emits Hydra role names directly, and leave role_claim at its default:

{
    "hydra_roles": [
        role
        for group, role in [("Hydra Admins", "admin"), ("Hydra Operators", "restart-jobs")]
        if group in user.ak_groups
    ]
}

GitHub

GitHub’s OAuth Apps don’t issue ID tokens, so Hydra can’t use GitHub directly as an OIDC provider. Put a broker in front of it instead. With Authentik:

1. Create a GitHub OAuth app: Settings → Developer settings → OAuth Apps → New OAuth App.
   • Homepage URL: https://authentik.example.com/
   • Authorization callback URL: https://authentik.example.com/source/oauth/github/callback/
2. In Authentik, create a GitHub source: Sources → Create → Social Source → GitHub. Paste the client ID and secret, and request the user:email scope.
3. Assign the source to a stage, so that users can enroll through it.
4. Set up the Authentik provider and Hydra as in the Authentik example.

Users need a primary email address on their GitHub account, or they can’t sign in. The broker sets sub, which is part of the Hydra username. For a GitHub source, sub is the numeric GitHub user ID. Switching brokers changes usernames, so Hydra treats returning users as new ones.

Hydra also has an older, non-OIDC GitHub login, configured with github_client_id and github_client_secret. It grants no roles.

For a working setup, see the kanidm provider in subprojects/hydra-tests/Hydra/Controller/User/oidc.t and foreman/start-kanidm.pl.

Creating and Managing Projects

Once Hydra is installed and running, the next step is to add projects to the build farm. We follow the example of the Patchelf project, a software tool written in C and using the GNU Build System (GNU Autoconf and GNU Automake).

Log in to the web interface of your Hydra installation using the user name and password you inserted in the database (by default, Hydra’s web server listens on localhost:3000). Then follow the “Create Project” link to create a new project.

Project Information

A project definition consists of some general information and a set of job sets. The general information identifies a project, its owner, and current state of activity. Here’s what we fill in for the patchelf project:

Identifier: patchelf

The identifier is the identity of the project. It is used in URLs and in the names of build results.

The identifier should be a unique name (it is the primary database key for the project table in the database). If you try to create a project with an already existing identifier you’d get an error message from the database. So try to create the project after entering just the general information to figure out if you have chosen a unique name. Job sets can be added once the project has been created.

Display name: Patchelf

The display name is used in menus.

Description: A tool for modifying ELF binaries

The description is used as short documentation of the nature of the project.

Owner: eelco

The owner of a project can create and edit job sets.

Enabled: Yes

Only if the project is enabled are builds performed.

Once created there should be an entry for the project in the sidebar. Go to the project page for the Patchelf project.

Job Sets

A project can consist of multiple job sets (hereafter jobsets), separate tasks that can be built separately, but may depend on each other (without cyclic dependencies, of course). Go to the Edit page of the Patchelf project and “Add a new jobset” by providing the following “Information”:

Identifier:     trunk
Description:    Trunk
Nix expression: release.nix in input patchelfSrc

This states that in order to build the trunk jobset, the Nix expression in the file release.nix, which can be obtained from input patchelfSrc, should be evaluated. (We’ll have a look at release.nix later.)

To realize a job we probably need a number of inputs, which can be declared in the table below. As many inputs as required can be added. For patchelf we declare the following inputs.

patchelfSrc
'Git checkout' https://github.com/NixOS/patchelf

nixpkgs 'Git checkout' https://github.com/NixOS/nixpkgs

officialRelease   Boolean false

system   String value "i686-linux"

Building Jobs

Build Recipes

Build jobs and build recipes for a jobset are specified in a text file written in the Nix language. The recipe is actually called a Nix expression in Nix parlance. By convention this file is often called release.nix.

The release.nix file is typically kept under version control, and the repository that contains it one of the build inputs of the corresponding — often called hydraConfig by convention. The repository for that file and the actual file name are specified on the web interface of Hydra under the Setup tab of the jobset’s overview page, under the Nix expression heading. See, for example, the jobset overview page of the PatchELF project, and the corresponding Nix file.

Knowledge of the Nix language is recommended, but the example below should already give a good idea of how it works:

let
  pkgs = import <nixpkgs> {}; # ①

  jobs = rec { # ②

    tarball = # ③
      pkgs.releaseTools.sourceTarball { # ④
        name = "hello-tarball";
        src = <hello>; # ⑤
        buildInputs = (with pkgs; [ gettext texLive texinfo ]);
      };

    build = # ⑥
      { system ? builtins.currentSystem }:  # ⑦

      let pkgs = import <nixpkgs> { inherit system; }; in
      pkgs.releaseTools.nixBuild { # ⑧
        name = "hello";
        src = jobs.tarball;
        configureFlags = [ "--disable-silent-rules" ];
      };
  };
in
  jobs # ⑨

This file shows what a release.nix file for GNU Hello would look like. GNU Hello is representative of many GNU and non-GNU free software projects:

• it uses the GNU Build System, namely GNU Autoconf, and GNU Automake; for users, it means it can be installed using the usual ./configure && make install procedure;
• it uses Gettext for internationalization;
• it has a Texinfo manual, which can be rendered as PDF with TeX.

The file defines a jobset consisting of two jobs: tarball, and build. It contains the following elements (referenced from the figure by numbers):

1. This defines a variable pkgs holding the set of packages provided by Nixpkgs.

   Since nixpkgs appears in angle brackets, there must be a build input of that name in the Nix search path. In this case, the web interface should show a nixpkgs build input, which is a checkout of the Nixpkgs source code repository; Hydra then adds this and other build inputs to the Nix search path when evaluating release.nix.

2. This defines a variable holding the two Hydra jobs – an attribute set in Nix.

3. This is the definition of the first job, named tarball. The purpose of this job is to produce a usable source code tarball.

4. The tarball job calls the sourceTarball function, which (roughly) runs autoreconf && ./configure && make dist on the checkout. The buildInputs attribute specifies additional software dependencies for the job.

   The package names used in buildInputs – e.g., texLive – are the names of the attributes corresponding to these packages in Nixpkgs, specifically in the all-packages.nix file. See the section entitled “Package Naming” in the Nixpkgs manual for more information.

5. The tarball jobs expects a hello build input to be available in the Nix search path. Again, this input is passed by Hydra and is meant to be a checkout of GNU Hello’s source code repository.

6. This is the definition of the build job, whose purpose is to build Hello from the tarball produced above.

7. The build function takes one parameter, system, which should be a string defining the Nix system type – e.g., "x86_64-linux". Additionally, it refers to jobs.tarball, seen above.

   Hydra inspects the formal argument list of the function (here, the system argument) and passes it the corresponding parameter specified as a build input on Hydra’s web interface. Here, system is passed by Hydra when it calls build. Thus, it must be defined as a build input of type string in Hydra, which could take one of several values.

   The question mark after system defines the default value for this argument, and is only useful when debugging locally.

8. The build job calls the nixBuild function, which unpacks the tarball, then runs ./configure && make && make check && make install.

9. Finally, the set of jobs is returned to Hydra, as a Nix attribute set.

Building from the Command Line

It is often useful to test a build recipe, for instance before it is actually used by Hydra, when testing changes, or when debugging a build issue. Since build recipes for Hydra jobsets are just plain Nix expressions, they can be evaluated using the standard Nix tools.

To evaluate the tarball jobset of the above example, just run:

$ nix-build release.nix -A tarball

However, doing this with the example as is will probably yield an error like this:

error: user-thrown exception: file `hello' was not found in the Nix search path (add it using $NIX_PATH or -I)

The error is self-explanatory. Assuming $HOME/src/hello points to a checkout of Hello, this can be fixed this way:

$ nix-build -I ~/src release.nix -A tarball

Similarly, the build jobset can be evaluated:

$ nix-build -I ~/src release.nix -A build

The build job reuses the result of the tarball job, rebuilding it only if it needs to.

Adding More Jobs

The example illustrates how to write the most basic jobs, tarball and build. In practice, much more can be done by using features readily provided by Nixpkgs or by creating new jobs as customizations of existing jobs.

For instance, test coverage report for projects compiled with GCC can be automatically generated using the coverageAnalysis function provided by Nixpkgs instead of nixBuild. Back to our GNU Hello example, we can define a coverage job that produces an HTML code coverage report directly readable from the corresponding Hydra build page:

coverage =
  { system ? builtins.currentSystem }:

  let pkgs = import nixpkgs { inherit system; }; in
  pkgs.releaseTools.coverageAnalysis {
    name = "hello";
    src = jobs.tarball;
    configureFlags = [ "--disable-silent-rules" ];
  };

As can be seen, the only difference compared to build is the use of coverageAnalysis.

Nixpkgs provides many more build tools, including the ability to run build in virtual machines, which can themselves run another GNU/Linux distribution, which allows for the creation of packages for these distributions. Please see the pkgs/build-support/release directory of Nixpkgs for more. The NixOS manual also contains information about whole-system testing in virtual machine.

Now, assume we want to build Hello with an old version of GCC, and with different configure flags. A new build_exotic job can be written that simply overrides the relevant arguments passed to nixBuild:

build_exotic =
  { system ? builtins.currentSystem }:

  let
    pkgs = import nixpkgs { inherit system; };
    build = jobs.build { inherit system; };
  in
    pkgs.lib.overrideDerivation build (attrs: {
      buildInputs = [ pkgs.gcc33 ];
      preConfigure = "gcc --version";
      configureFlags =
        attrs.configureFlags ++ [ "--disable-nls" ];
    });

The build_exotic job reuses build and overrides some of its arguments: it adds a dependency on GCC 3.3, a pre-configure phase that runs gcc --version, and adds the --disable-nls configure flags.

This customization mechanism is very powerful. For instance, it can be used to change the way Hello and all its dependencies – including the C library and compiler used to build it – are built. See the Nixpkgs manual for more.

Declarative Projects

see this chapter

Email Notifications

Hydra can send email notifications when the status of a build changes. This provides immediate feedback to maintainers or committers when a change causes build failures.

The feature can be turned on by adding the following line to hydra.conf

<email_notifications>
  build = 1
</email_notifications>

Evaluation errors are a separate switch in the same block, because they go to a different person — the project’s owner, rather than whoever maintains the job:

<email_notifications>
  build = 1
  eval = 1
</email_notifications>

The owner must also have “Receive evaluation error notifications” set on their user page; unlike build notification, it is off by default.

A plain email_notification = 1 — without the s — still works and turns on both, but it is deprecated in favour of the block above. Setting it as well as the block is an error rather than one of them winning: they are two ways of saying the same thing, and which you meant is not for Hydra to guess.

By default, Hydra only sends email notifications if a previously successful build starts to fail. In order to force Hydra to send an email for each build (including e.g. successful or cancelled ones), the environment variable HYDRA_FORCE_SEND_MAIL can be declared:

services.hydra-dev.extraEnv.HYDRA_FORCE_SEND_MAIL = "1";

SASL Authentication for the email address that’s used to send notifications can be configured like this:

EMAIL_SENDER_TRANSPORT_sasl_username=hydra@example.org
EMAIL_SENDER_TRANSPORT_sasl_password=verysecret
EMAIL_SENDER_TRANSPORT_port=587
EMAIL_SENDER_TRANSPORT_ssl=starttls

Further information about these environment variables can be found at the MetaCPAN documentation of Email::Sender::Manual::QuickStart.

It’s recommended to not put this in services.hydra-dev.extraEnv as this would leak the secrets into the Nix store. Instead, it should be written into an environment file and configured like this:

{ systemd.services.hydra-notify = {
    serviceConfig.EnvironmentFile = "/etc/secrets/hydra-mail-cfg";
  };
}

The simplest approach to enable Email Notifications is to use the ssmtp package, which simply hands off the emails to another SMTP server. For details on how to configure ssmtp, see the documentation for the networking.defaultMailServer option. To use ssmtp for the Hydra email notifications, add it to the path option of the Hydra services in your /etc/nixos/configuration.nix file:

systemd.services.hydra-queue-runner.path = [ pkgs.ssmtp ];
systemd.services.hydra-server.path = [ pkgs.ssmtp ];

Gitea Integration

Hydra can notify Git servers (such as GitLab, GitHub or Gitea) about the result of a build from a Git checkout.

This section describes how it can be implemented for gitea, but the approach for gitlab is analogous:

• Obtain an API token for your user

• Add it to a file which only users in the hydra group can read like this: see including files for more information

  <gitea_authorization>
    your_username=your_token
  </gitea_authorization>
  

• Include the file in your hydra.conf like this:

  {
    services.hydra-dev.extraConfig = ''
      Include /path/to/secret/file
    '';
  }
  

• For a jobset with a Git-input which points to a gitea-instance, add the following additional inputs:

  Type	Name	Value
  String value	gitea_repo_name	Name of the repository to build
  String value	gitea_repo_owner	Owner of the repository
  String value	gitea_status_repo	Name of the Git checkout input
  String value	gitea_http_url	Public URL of gitea, optional

Content-addressed derivations

Hydra can to a certain extent use the ca-derivations experimental Nix feature. To use it, make sure that the Nix version you use is at least as recent as the one used in hydra’s flake.

Be warned that this support is still highly experimental, and anything beyond the basic functionality might be broken at that point.

Hydra Jobs

Derivation Attributes

Hydra stores the following job attributes in its database:

• nixName - the Derivation’s name attribute
• system - the Derivation’s system attribute
• drvPath - the Derivation’s path in the Nix store
• outputs - A JSON dictionary of output names and their store path.

Meta fields

• description - meta.description, a string
• license - a comma separated list of license names from meta.license, expected to be a list of attribute sets with an attribute named shortName, ex: [ { shortName = "licensename"} ].
• homepage - meta.homepage, a string
• maintainers - a comma separated list of maintainer email addresses from meta.maintainers, expected to be a list of attribute sets with an attribute named email, ex: [ { email = "alice@example.com"; } ].
• schedulingPriority - meta.schedulingPriority, an integer. Default: 100. Slightly prioritizes this job over other jobs within this jobset.
• timeout - meta.timeout, an integer. Default: 36000. Number of seconds this job must complete within.
• maxSilent - meta.maxSilent, an integer. Default: 7200. Number of seconds of no output on stderr / stdout before considering the job failed.
• isChannel - meta.isHydraChannel, bool. Default: false. Deprecated.

Plugins

This chapter describes all plugins present in Hydra.

Inputs

Hydra supports the following inputs:

• Bazaar input
• Darcs input
• Git input
• Mercurial input
• Path input

Bitbucket pull requests

Create jobs based on open bitbucket pull requests.

Configuration options

• bitbucket_authorization.<owner>

Bitbucket status

Sets Bitbucket CI status.

Configuration options

• enable_bitbucket_status
• bitbucket.username
• bitbucket.password

CircleCI Notification

Sets CircleCI status.

Configuration options

• circleci.[].jobs
• circleci.[].vcstype
• circleci.[].token

Compress build logs

Compresses build logs after a build with bzip2 or zstd.

Configuration options

• compress_build_logs

Enable log compression

• compress_build_logs_compression

Which compression format to use. Valid values are bzip2 (default) and zstd.

• compress_build_logs_silent

Whether to compress logs silently.

Example

compress_build_logs = 1

Coverity Scan

Uploads source code to coverity scan.

Configuration options

• coverityscan.[].jobs
• coverityscan.[].project
• coverityscan.[].email
• coverityscan.[].token
• coverityscan.[].scanurl

Email notification, builds

Sends email notification if build status changes.

This plugin is named just EmailNotification for historical reasons, and it should be renamed to EmailNotificationBuild.

Configuration options

• email_notifications.build — mail about finished builds, to whoever maintains the job
• email_notification — deprecated; enables this and email_notifications.eval below. Setting it as well as the block is an error.

Example

<email_notifications>
  build = 1
</email_notifications>

Email notification, evaluation errors

Tells a project’s owner when one of its jobsets fails to evaluate, or evaluates with jobs that did not.

The owner must also have “Receive evaluation error notifications” set on their user page; unlike build notification, it is off by default. Only a changed error is reported, so a jobset failing the same way is not mailed about repeatedly.

Configuration options

• email_notifications.eval
• email_notification — deprecated; enables this and email_notifications.build above. Setting it as well as the block is an error.

Example

<email_notifications>
  eval = 1
</email_notifications>

Gitea status

Sets Gitea CI status

Configuration options

• gitea_authorization.<repo-owner>

GitHub pulls

Create jobs based on open GitHub pull requests

Configuration options

• github_authorization.<repo-owner>

Github refs

Hydra plugin for retrieving the list of references (branches or tags) from GitHub following a certain naming scheme.

Configuration options

• github_endpoint (defaults to https://api.github.com)
• github_authorization.<repo-owner>

Github status

Sets GitHub CI status.

Configuration options

• githubstatus.[].jobs

Regular expression for jobs to match in the format project:jobset:job. This field is required and has no default value.

• githubstatus.[].excludeBuildFromContext

Don’t include the build’s ID in the status.

• githubstatus.[].context

Context shown in the status

• githubstatus.[].useShortContext

Renames continuous-integration/hydra to ci/hydra and removes the PR suffix from the name. Useful to see the full path in GitHub for long job names.

• githubstatus.[].description

Description shown in the status. Defaults to Hydra build #<build-id> of <jobname>

• githubstatus.[].inputs

The input which corresponds to the github repo/rev whose status we want to report. Can be repeated.

• githubstatus.[].authorization

Verbatim contents of the Authorization header. See GitHub documentation for details. This field is only used if github_authorization.<repo-owner> is not set.

Example

<githubstatus>
  jobs = test:pr:build
  ## This example will match all jobs
  #jobs = .*
  inputs = src
  authorization = Bearer gha-secret😱secret😱secret😱
  excludeBuildFromContext = 1
</githubstatus>

GitLab pulls

Create jobs based on open gitlab pull requests.

Configuration options

• gitlab_authorization.<projectId>

Gitlab status

Sets Gitlab CI status.

Configuration options

• gitlab_authorization.<projectId>

InfluxDB notification

Writes InfluxDB events when a builds finished.

Configuration options

• influxdb.url
• influxdb.db

RunCommand

Runs a shell command when the build is finished.

See The RunCommand Plugin for more information.

Configuration options:

• runcommand.[].job

Regular expression for jobs to match in the format project:jobset:job. Defaults to *:*:*.

• runcommand.[].command

Command to run. Can use the $HYDRA_JSON environment variable to access information about the build.

Example

<runcommand>
  job = myProject:*:*
  command = cat $HYDRA_JSON > /tmp/hydra-output
</runcommand>

S3 backup

Upload nars and narinfos to S3 storage.

Configuration options

• s3backup.[].jobs
• s3backup.[].compression_type
• s3backup.[].name
• s3backup.[].prefix

Slack notification

Sending Slack notifications about build results.

Configuration options

• slack.[].jobs
• slack.[].force
• slack.[].url

SoTest

Scheduling hardware tests to SoTest controller

This plugin submits tests to a SoTest controller for all builds that contain two products matching the subtypes “sotest-binaries” and “sotest-config”.

Build products are declared by the file “nix-support/hydra-build-products” relative to the root of a build, in the following format:

 file sotest-binaries /nix/store/…/binaries.zip
 file sotest-config /nix/store/…/config.yaml

Configuration options

• sotest.[].uri

URL of the controller, defaults to https://opensource.sotest.io

• sotest.[].authfile

File containing username:password

• sotest.[].priority

Optional priority setting.

Example

 <sotest>
   uri = https://sotest.example
   authfile = /var/lib/hydra/sotest.auth
   priority = 1
 </sotest>

Declarative Projects

Declarative Projects

Hydra supports declaratively configuring a project’s jobsets. This configuration can be done statically, or generated by a build job.

Note

Hydra will treat the project’s declarative input as a static definition if and only if the spec file contains a dictionary of dictionaries. If the value of any key in the spec is not a dictionary, it will treat the spec as a generated declarative spec.

Static, Declarative Projects

Hydra supports declarative projects, where jobsets are configured from a static JSON document in a repository.

To configure a static declarative project, take the following steps:

1. Create a Hydra-fetchable source like a Git repository or local path.

2. In that source, create a file called spec.json, and add the specification for all of the jobsets. Each key is jobset and each value is a jobset’s specification. For example:

   {
     "nixpkgs": {
       "enabled": 1,
       "hidden": false,
       "description": "Nixpkgs",
       "nixexprinput": "nixpkgs",
       "nixexprpath": "pkgs/top-level/release.nix",
       "checkinterval": 300,
       "schedulingshares": 100,
       "enableemail": false,
       "enable_dynamic_run_command": false,
       "emailoverride": "",
       "keepnr": 3,
       "inputs": {
         "nixpkgs": {
             "type": "git",
             "value": "git://github.com/NixOS/nixpkgs.git master",
             "emailresponsible": false
         }
       }
     },
     "nixos": {
       "enabled": 1,
       "hidden": false,
       "description": "NixOS: Small Evaluation",
       "nixexprinput": "nixpkgs",
       "nixexprpath": "nixos/release-small.nix",
       "checkinterval": 300,
       "schedulingshares": 100,
       "enableemail": false,
       "enable_dynamic_run_command": false,
       "emailoverride": "",
       "keepnr": 3,
       "inputs": {
         "nixpkgs": {
           "type": "git",
           "value": "git://github.com/NixOS/nixpkgs.git master",
           "emailresponsible": false
         }
       }
     }
   }
   

3. Create a new project, and set the project’s declarative input type, declarative input value, and declarative spec file to point to the source and JSON file you created in step 2.

Hydra will create a special jobset named .jobsets. When the .jobsets jobset is evaluated, this static specification will be used for configuring the rest of the project’s jobsets.

Generated, Declarative Projects

Hydra also supports generated declarative projects, where jobsets are configured automatically from specification files instead of being managed through the UI. A jobset specification is a JSON object containing the configuration of the jobset, for example:

{
    "enabled": 1,
    "hidden": false,
    "description": "js",
    "nixexprinput": "src",
    "nixexprpath": "release.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "enable_dynamic_run_command": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}

To configure a declarative project, take the following steps:

1. Create a jobset repository in the normal way (e.g. a git repo with a release.nix file, any other needed helper files, and taking any kind of hydra input), but without adding it to the UI. The nix expression of this repository should contain a single job, named jobsets. The output of the jobsets job should be a JSON file containing an object of jobset specifications. Each member of the object will become a jobset of the project, configured by the corresponding jobset specification.

2. In some hydra-fetchable source (potentially, but not necessarily, the same repo you created in step 1), create a JSON file containing a jobset specification that points to the jobset repository you created in the first step, specifying any needed inputs (e.g. nixpkgs) as necessary.

3. In the project creation/edit page, set declarative input type, declarative input value, and declarative spec file to point to the source and JSON file you created in step 2.

Hydra will create a special jobset named .jobsets, which whenever evaluated will go through the steps above in reverse order:

1. Hydra will fetch the input specified by the declarative input type and value.

2. Hydra will use the configuration given in the declarative spec file as the jobset configuration for this evaluation. In addition to any inputs specified in the spec file, hydra will also pass the declInput argument corresponding to the input fetched in step 1 and the projectName argument containing the project’s name.

3. As normal, hydra will build the jobs specified in the jobset repository, which in this case is the single jobsets job. When that job completes, hydra will read the created jobset specifications and create corresponding jobsets in the project, disabling any jobsets that used to exist but are not present in the current spec.

RunCommand

The RunCommand Plugin

Hydra supports executing a program after certain builds finish. This behavior is disabled by default.

Hydra executes these commands under the hydra-notify service.

Static Commands

Configure specific commands to execute after the specified matching job finishes.

Configuration

• runcommand.[].job

A matcher for jobs to match in the format project:jobset:job. Defaults to *:*:*.

Note: This matcher format is not a regular expression. The * is a wildcard for that entire section, partial matches are not supported.

• runcommand.[].command

Command to run. Can use the $HYDRA_JSON environment variable to access information about the build.

Example

<runcommand>
  job = myProject:*:*
  command = cat $HYDRA_JSON > /tmp/hydra-output
</runcommand>

Dynamic Commands

Hydra can optionally run RunCommand hooks defined dynamically by the jobset. In order to enable dynamic commands, you must enable this feature in your hydra.conf, as well as in the parent project and jobset configuration.

Behavior

Hydra will execute any program defined under the runCommandHook attribute set. These jobs must have a single output named out, and that output must be an executable file located directly at $out.

Security Properties

Safely deploying dynamic commands requires careful design of your Hydra jobs. Allowing arbitrary users to define attributes in your top level attribute set will allow that user to execute code on your Hydra.

If a jobset has dynamic commands enabled, you must ensure only trusted users can define top level attributes.

Configuration

• dynamicruncommand.enable

Set to 1 to enable dynamic RunCommand program execution.

Example

In your Hydra configuration, specify:

<dynamicruncommand>
  enable = 1
</dynamicruncommand>

Then create a job named runCommandHook.example in your jobset:

{ pkgs, ... }: {
    runCommandHook = {
        recurseForDerivations = true;

        example = pkgs.writeScript "run-me" ''
          #!${pkgs.runtimeShell}

          ${pkgs.jq}/bin/jq . "$HYDRA_JSON"
        '';
    };
}

After the runcommandHook.example build finishes that script will execute.

Using the external API

To be able to create integrations with other services, Hydra exposes an external API that you can manage projects with.

The API is accessed over HTTP(s) where all data is sent and received as JSON.

Creating resources requires the caller to be authenticated, while retrieving resources does not.

The API does not have a separate URL structure for its endpoints. Instead you request the pages of the web interface as application/json to use the API.

List projects

To list all the projects of the Hydra install:

GET /
Accept: application/json

This will give you a list of projects, where each project contains general information and a list of its job sets.

Example

curl -i -H 'Accept: application/json' \
    https://hydra.nixos.org

Note: this response is truncated

GET https://hydra.nixos.org/
HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "displayname": "Acoda",
    "name": "acoda",
    "description": "Acoda is a tool set for automatic data migration along an evolving data model",
    "enabled": 0,
    "owner": "sander",
    "hidden": 1,
    "jobsets": [
      "trunk"
    ]
  },
  {
    "displayname": "cabal2nix",
    "name": "cabal2nix",
    "description": "Convert Cabal files into Nix build instructions",
    "enabled": 0,
    "owner": "simons@cryp.to",
    "hidden": 1,
    "jobsets": [
      "master"
    ]
  }
]

Get a single project

To get a single project by identifier:

GET /project/:project-identifier
Accept: application/json

Example

curl -i -H 'Accept: application/json' \
    https://hydra.nixos.org/project/hydra

GET https://hydra.nixos.org/project/hydra
HTTP/1.1 200 OK
Content-Type: application/json

{
  "description": "Hydra, the Nix-based continuous build system",
  "hidden": 0,
  "displayname": "Hydra",
  "jobsets": [
    "hydra-master",
    "hydra-ant-logger-trunk",
    "master",
    "build-ng"
  ],
  "name": "hydra",
  "enabled": 1,
  "owner": "eelco"
}

Get a single job set

To get a single job set by identifier:

GET /jobset/:project-identifier/:jobset-identifier
Content-Type: application/json

Example

curl -i -H 'Accept: application/json' \
    https://hydra.nixos.org/jobset/hydra/build-ng

GET https://hydra.nixos.org/jobset/hydra/build-ng
HTTP/1.1 200 OK
Content-Type: application/json

{
  "errormsg": "evaluation failed due to signal 9 (Killed)",
  "fetcherrormsg": null,
  "nixexprpath": "release.nix",
  "nixexprinput": "hydraSrc",
  "emailoverride": "rob.vermaas@gmail.com, eelco.dolstra@logicblox.com",
  "jobsetinputs": {
    "officialRelease": {
      "jobsetinputalts": [
        "false"
      ]
    },
    "hydraSrc": {
      "jobsetinputalts": [
        "https://github.com/NixOS/hydra.git build-ng"
      ]
    },
    "nixpkgs": {
      "jobsetinputalts": [
        "https://github.com/NixOS/nixpkgs.git release-14.12"
      ]
    }
  },
  "enabled": 0
}

List evaluations

To list the evaluations of a job set by identifier:

GET /jobset/:project-identifier/:jobset-identifier/evals
Content-Type: application/json

Example

curl -i -H 'Accept: application/json' \
    https://hydra.nixos.org/jobset/hydra/build-ng/evals

Note: this response is truncated

GET https://hydra.nixos.org/jobset/hydra/build-ng/evals
HTTP/1.1 200 OK
Content-Type: application/json

{
  "evals": [
    {
      "jobsetevalinputs": {
        "nixpkgs": {
          "dependency": null,
          "type": "git",
          "value": null,
          "uri": "https://github.com/NixOS/nixpkgs.git",
          "revision": "f60e48ce81b6f428d072d3c148f6f2e59f1dfd7a"
        },
        "hydraSrc": {
          "dependency": null,
          "type": "git",
          "value": null,
          "uri": "https://github.com/NixOS/hydra.git",
          "revision": "48d6f0de2ab94f728d287b9c9670c4d237e7c0f6"
        },
        "officialRelease": {
          "dependency": null,
          "value": "false",
          "type": "boolean",
          "uri": null,
          "revision": null
        }
      },
      "hasnewbuilds": 1,
      "builds": [
        24670686,
        24670684,
        24670685,
        24670687
      ],
      "id": 1213758
    }
  ],
  "first": "?page=1",
  "last": "?page=1"
}

Get a single build

To get a single build by its id:

GET /build/:build-id
Content-Type: application/json

Example

curl -i -H 'Accept: application/json' \
    https://hydra.nixos.org/build/24670686

GET /build/24670686
HTTP/1.1 200 OK
Content-Type: application/json

{
  "job": "tests.api.x86_64-linux",
  "jobsetevals": [
    1213758
  ],
  "buildstatus": 0,
  "buildmetrics": null,
  "project": "hydra",
  "system": "x86_64-linux",
  "priority": 100,
  "releasename": null,
  "starttime": 1439402853,
  "nixname": "vm-test-run-unnamed",
  "timestamp": 1439388618,
  "id": 24670686,
  "stoptime": 1439403403,
  "jobset": "build-ng",
  "buildoutputs": {
    "out": {
      "path": "/nix/store/lzrxkjc35mhp8w7r8h82g0ljyizfchma-vm-test-run-unnamed"
    }
  },
  "buildproducts": {
    "1": {
      "path": "/nix/store/lzrxkjc35mhp8w7r8h82g0ljyizfchma-vm-test-run-unnamed",
      "defaultpath": "log.html",
      "type": "report",
      "sha256hash": null,
      "filesize": null,
      "name": "",
      "subtype": "testlog"
    }
  },
  "finished": 1
}

Webhooks

Hydra can be notified by github or gitea with webhooks to trigger a new evaluation when a jobset has a github repo in its input.

Webhook Authentication

Hydra supports webhook signature verification for both GitHub and Gitea using HMAC-SHA256. This ensures that webhook requests are coming from your configured Git forge and haven’t been tampered with.

Configuring Webhook Authentication

1. Create webhook configuration: Generate and store webhook secrets securely:

   # Create directory and generate secrets in one step
   mkdir -p /var/lib/hydra/secrets
   cat > /var/lib/hydra/secrets/webhook-secrets.conf <<EOF
   <github>
     secret = $(openssl rand -hex 32)
   </github>
   <gitea>
     secret = $(openssl rand -hex 32)
   </gitea>
   EOF
   
   # Set secure permissions
   chmod 0600 /var/lib/hydra/secrets/webhook-secrets.conf
   chown hydra:hydra /var/lib/hydra/secrets/webhook-secrets.conf
   

2. Configure Hydra: Add the following to your hydra.conf:

   <webhooks>
     Include /var/lib/hydra/secrets/webhook-secrets.conf
   </webhooks>
   

3. Configure your Git forge: View the generated secrets and configure them in GitHub/Gitea:

   grep "secret =" /var/lib/hydra/secrets/webhook-secrets.conf
   

Multiple Secrets Support

Hydra supports configuring multiple secrets for each platform, which is useful for:

• Zero-downtime secret rotation
• Supporting multiple environments (production/staging)
• Gradual migration of webhooks

To configure multiple secrets, use array syntax:

<github>
  secret = current-webhook-secret
  secret = previous-webhook-secret
</github>

GitHub

To set up a webhook for a GitHub repository go to https://github.com/<yourhandle>/<yourrepo>/settings and in the Webhooks tab click on Add webhook.

• In Payload URL fill in https://<your-hydra-domain>/api/push-github.
• In Content type switch to application/json.
• In the Secret field, enter the content of your GitHub webhook secret file (if authentication is configured).
• For Which events would you like to trigger this webhook? keep the default option for events on Just the push event..

Then add the hook with Add webhook.

Verifying GitHub Webhook Security

After configuration, GitHub will send webhook requests with an X-Hub-Signature-256 header containing the HMAC-SHA256 signature of the request body. Hydra will verify this signature matches the configured secret.

Gitea

To set up a webhook for a Gitea repository go to the settings of the repository in your Gitea instance and in the Webhooks tab click on Add Webhook and choose Gitea in the drop down.

• In Target URL fill in https://<your-hydra-domain>/api/push-gitea.
• Keep HTTP method POST, POST Content Type application/json and Trigger On Push Events.
• In the Secret field, enter the content of your Gitea webhook secret file (if authentication is configured).
• Change the branch filter to match the git branch hydra builds.

Then add the hook with Add webhook.

Verifying Gitea Webhook Security

After configuration, Gitea will send webhook requests with an X-Gitea-Signature header containing the HMAC-SHA256 signature of the request body. Hydra will verify this signature matches the configured secret.

Troubleshooting

If you receive 401 Unauthorized errors:

• Verify the webhook secret in your Git forge matches the content of the secret file exactly
• Check that the secret file has proper permissions (should be 0600)
• Look at Hydra’s logs for specific error messages
• Ensure the correct signature header is being sent by your Git forge

If you see warnings about webhook authentication not being configured:

• Configure webhook authentication as described above to secure your endpoints

Webhook Authentication Migration Guide

This guide helps Hydra administrators migrate from unauthenticated webhooks to authenticated webhooks to secure their Hydra instances against unauthorized job evaluations.

Why Migrate?

Currently, Hydra’s webhook endpoints (/api/push-github and /api/push-gitea) accept any POST request without authentication. This vulnerability allows:

• Anyone to trigger expensive job evaluations
• Potential denial of service through repeated requests
• Manipulation of build timing and scheduling

Step-by-Step Migration for NixOS

1. Create Webhook Configuration

Create a webhook secrets configuration file with the generated secrets:

# Create the secrets configuration file with inline secret generation
cat > /var/lib/hydra/secrets/webhook-secrets.conf <<EOF
<github>
  secret = $(openssl rand -hex 32)
</github>
<gitea>
  secret = $(openssl rand -hex 32)
</gitea>
EOF

# Set secure permissions
chmod 0440 /var/lib/hydra/secrets/webhook-secrets.conf
chown hydra:hydra /var/lib/hydra/secrets/webhook-secrets.conf

Important: Save the generated secrets to configure them in GitHub/Gitea later. You can view them with:

cat /var/lib/hydra/secrets/webhook-secrets.conf

Then update your NixOS configuration to include the webhook configuration:

{
  services.hydra-dev = {
    enable = true;
    hydraURL = "https://hydra.example.com";
    notificationSender = "hydra@example.com";

    extraConfig = ''
      <webhooks>
        Include /var/lib/hydra/secrets/webhook-secrets.conf
      </webhooks>
    '';
  };
}

For multiple secrets (useful for rotation or multiple environments), update your webhook-secrets.conf:

<github>
  secret = your-github-webhook-secret-prod
  secret = your-github-webhook-secret-staging
</github>
<gitea>
  secret = your-gitea-webhook-secret
</gitea>

2. Deploy Configuration

Apply the NixOS configuration:

nixos-rebuild switch

This will automatically restart Hydra services with the new configuration.

3. Verify Configuration

Check Hydra’s logs to ensure secrets were loaded successfully:

journalctl -u hydra-server | grep -i webhook

You should not see warnings about webhook authentication not being configured.

4. Update Your Webhooks

GitHub

1. Navigate to your repository settings: https://github.com/<owner>/<repo>/settings/hooks
2. Edit your existing Hydra webhook
3. In the “Secret” field, paste the content of /var/lib/hydra/secrets/github-webhook-secret
4. Click “Update webhook”
5. GitHub will send a ping event to verify the configuration

Gitea

1. Navigate to your repository webhook settings
2. Edit your existing Hydra webhook
3. In the “Secret” field, paste the content of /var/lib/hydra/secrets/gitea-webhook-secret
4. Click “Update Webhook”
5. Use the “Test Delivery” button to verify the configuration

5. Test the Configuration

After updating each webhook:

1. Make a test commit to trigger the webhook
2. Check Hydra’s logs for successful authentication
3. Verify the evaluation was triggered in Hydra’s web interface

Troubleshooting

401 Unauthorized Errors

If webhooks start failing with 401 errors:

• Verify the secret in the Git forge matches the file content exactly
• Check file permissions: ls -la /var/lib/hydra/secrets/
• Ensure no extra whitespace in secret files
• Check Hydra logs for specific error messages

Webhook Still Unauthenticated

If you see warnings about unauthenticated webhooks after configuration:

• Verify the configuration syntax in your NixOS module
• Ensure the NixOS configuration was successfully applied
• Check that the webhook-secrets.conf file exists and is readable by the Hydra user
• Verify the Include path is correct in your hydra.conf
• Check the syntax of your webhook-secrets.conf file

Testing Without Git Forge

You can test webhook authentication using curl:

# Read the secret
SECRET=$(cat /var/lib/hydra/secrets/github-webhook-secret)

# Create test payload
PAYLOAD='{"ref":"refs/heads/main","repository":{"clone_url":"https://github.com/test/repo.git"}}'

# Calculate signature
SIGNATURE="sha256=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | cut -d' ' -f2)"

# Send authenticated request
curl -X POST https://your-hydra/api/push-github \
  -H "Content-Type: application/json" \
  -H "X-Hub-Signature-256: $SIGNATURE" \
  -d "$PAYLOAD"

For Gitea (no prefix in signature):

# Read the secret
SECRET=$(cat /var/lib/hydra/secrets/gitea-webhook-secret)

# Create test payload
PAYLOAD='{"ref":"refs/heads/main","repository":{"clone_url":"https://gitea.example.com/test/repo.git"}}'

# Calculate signature
SIGNATURE=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | cut -d' ' -f2)

# Send authenticated request
curl -X POST https://your-hydra/api/push-gitea \
  -H "Content-Type: application/json" \
  -H "X-Gitea-Signature: $SIGNATURE" \
  -d "$PAYLOAD"

Monitoring Hydra

Webserver

The webserver exposes Prometheus metrics for the webserver itself at /metrics.

Queue Runner

The queue runner’s status is exposed at /queue-runner-status:

$ curl --header "Accept: application/json" http://localhost:63333/queue-runner-status
... JSON payload ...

Notification Daemon

The hydra-notify process can expose Prometheus metrics for plugin execution. See hydra-notify’s Prometheus service for details on enabling and configuring the exporter.

The notification exporter exposes metrics on a per-plugin, per-event-type basis: execution durations, frequency, successes, and failures.

Diagnostic Dump

The notification daemon can also dump its metrics to stderr whether or not the exporter is configured. This is particularly useful for cases where metrics data is needed but the exporter was not enabled.

To trigger this diagnostic dump, send a Postgres notification with the hydra_notify_dump_metrics channel and no payload. See Re-sending a notification.

Architecture

This is an overview of Hydra’s inner workings. You can use it as a guide to navigate the codebase or ask questions.

Components

Hydra’s components are split across a coordinator machine and any number of builder machines. The NixOS modules in nixos-modules/ reflect this split: web-app and queue-runner run on the master, while builder runs on remote machines. The optional ad-hoc module also runs on the master. For small installations, all three can run on a single host (the hydra module combines them). But most installation will want to use multiple build machines for scale.

Coordinator machine

These components all share a single Nix store and PostgreSQL database on the master:

• PostgreSQL database
  • stores configuration, the build queue (scheduled and finished builds), and results
• hydra-server (Perl, Catalyst)
  • web frontend and REST API
  • user authentication (built-in or LDAP)
• hydra-ws (Rust)
  • WebSocket service for streaming live build logs
  • reads build metadata from PostgreSQL and tails log files from the coordinator’s store
  • listens for PostgreSQL build-completion notifications and forwards events to clients
• hydra-evaluator (Rust)
  • periodically evaluates jobsets by invoking the Nix evaluator
  • writes .drv files into the coordinator’s Nix store
  • adds new builds to the queue when evaluation results change
• hydra-eval-jobset (Perl)
  • called by the evaluator to orchestrate fetching inputs and running the Nix evaluation
• hydra-queue-runner (Rust)
  • reads .drv files from the coordinator’s Nix store
  • schedules build steps across builders
  • uploads results to a destination store
  • exposes a gRPC service that builders connect to
• hydra-notify (Perl)
  • dispatches post-build notifications to plugins (email, GitHub/GitLab status, Slack, etc.)
  • listens for PostgreSQL NOTIFY events from the queue runner
• hydra-ad-hoc (Rust, optional, experimental)
  • serves the nix daemon protocol on a Unix socket, presenting Hydra as one giant nix daemon
  • a nix-build or nix-store --realise pointed at it has its derivations built by the queue runner and builders instead of locally
  • files such ad hoc jobs under a hidden adhoc/adhoc jobset; nothing inside Hydra uses it
  • streams each build step’s log back to the waiting client, via build-logs
  • see Ad hoc builds
• Plugin system (Perl)
  • input plugins extend the evaluator with new source types (Git, Mercurial, Darcs, etc.)
  • notification plugins react to build lifecycle events

Destination store

The queue runner uploads built outputs to a destination store, which is separate from the coordinator’s local Nix store. This can be an S3-compatible binary cache, or for small installations it can just be the coordinator’s own store. See Populating a Cache for configuration.

Builder machines

Builders have their own Nix store — they do not need access to the coordinator’s store or database.

• hydra-builder (Rust)
  • build execution agent that runs on remote machines
  • connects to the queue runner’s gRPC service
  • receives derivations to build, streams back logs and results

Rust crate dependencies

The following is the transitive reduction of the dependency graph between the Rust crates in this repo. Solid arrows are normal dependencies; dashed arrows are dev (test-only) dependencies.

graph BT
    binary-cache --> daemon-client-utils
    nix-support --> store-path-utils
    db --> nix-support
    hydra-proto --> nix-support
    hydra-ad-hoc --> build-logs
    hydra-ad-hoc --> db
    hydra-ad-hoc --> hydra-tracing
    hydra-evaluator --> db
    hydra-evaluator --> hydra-tracing
    hydra-ws --> build-logs
    hydra-ws --> db
    hydra-ws --> hydra-tracing
    store-transfer --> daemon-client-utils
    store-transfer --> hydra-proto
    hydra-builder --> binary-cache
    hydra-builder --> hydra-tracing
    hydra-builder --> store-transfer
    hydra-queue-runner --> binary-cache
    hydra-queue-runner --> db
    hydra-queue-runner --> hydra-tracing
    hydra-queue-runner --> store-transfer
    binary-cache -.-> hydra-tracing
    db -.-> test-utils

Shared Rust libraries

• hydra-proto: generated gRPC/protobuf code for the builder ↔ queue-runner interface (message types, client stubs, server traits)

• db: PostgreSQL database access via SQLx (models, queries, connection pooling)

• binary-cache: reading and writing Nix binary cache artifacts (NARinfo, NAR files, signatures, presigned uploads)

• build-logs: following build-step logs as the queue runner writes them, and reading its step/build notifications; shared by hydra-ws and hydra-ad-hoc

• daemon-client-utils: Various utilities for working with the daemon connection beyond what the Harmonia libraries provide.

• store-transfer: shared import/export logic for streaming store objects as AddToStoreRequest protobuf messages, used by both the builder and queue-runner

• nix-support: Infrastructure for interpreting the ${store_object}/nix-support directory convention

• store-path-utils: lightweight store path utilities built on harmonia types

• tracing: OpenTelemetry/tracing setup with optional gRPC export

• test-utils: test fixtures and helpers for integration tests

Source layout

The repository is organized into subprojects:

• subprojects/hydra/ — the Perl/Catalyst web application, evaluator scripts, SQL schema, and plugins
• subprojects/hydra-queue-runner/ — the Rust queue runner
• subprojects/hydra-builder/ — the Rust build agent
• subprojects/hydra-ad-hoc/ — the optional, experimental Rust nix-daemon-protocol endpoint for ad hoc builds
• subprojects/crates/ — shared Rust libraries
• subprojects/proto/ — Protocol Buffer .proto source files (compiled by the proto crate’s build script)
• subprojects/nix-perl/ — Perl bindings to the Nix store API
• subprojects/hydra-tests/ — integration test suite (Perl, using Test::More)
• subprojects/hydra-manual/ — this manual (mdbook)

The build system uses Meson for the Perl components and Cargo for the Rust workspace.

Database Schema

The canonical schema lives in subprojects/hydra/sql/hydra.sql. Incremental migrations are in migrations/upgrade-N.sql; see the SQL README for details on making schema changes.

The database is accessed by both language runtimes: Perl (DBI/DBIx::Class) and Rust (SQLx).

Key tables:

• Jobsets
  • populated by calling Nix evaluator
  • every Nix derivation in release.nix is a Job
  • flake
    • URL to flake, if job is from a flake
    • single-point of configuration for flake builds
    • flake itself contains pointers to dependencies
    • for other builds we need more configuration data
• JobsetInputs
  • more configuration for a Job
• JobsetInputAlts
  • historical, where you could have more than one alternative for each input
  • it would have done the cross product of all possibilities
  • not used any more, as now every input is unique
  • originally that was to have alternative values for the system parameter
    • x86-linux, x86_64-darwin
    • turned out not to be a good idea, as job set names did not uniquely identify output
• Builds
  • queue: scheduled and finished builds
  • instance of a Job
  • corresponds to a top-level derivation
    • can have many dependencies that don’t have a corresponding build
    • dependencies represented as BuildSteps
  • a Job is all the builds with a particular name, e.g.
    • git.x86_64-linux is a job
    • there maybe be multiple builds for that job
      • build ID: just an auto-increment number
  • building one thing can actually cause many (hundreds of) derivations to be built
  • for queued builds, the drv has to be present in the store
    • otherwise build will fail, e.g. after garbage collection
• BuildSteps
  • corresponds to a derivation or substitution
  • are reused through the Nix store
  • may be duplicated for unique derivations due to how they relate to Jobs
• BuildStepOutputs
  • corresponds directly to derivation outputs
    • out, dev, …
• BuildProducts
  • not a Nix concept
  • populated from a special file $out/nix-support/hydra-build-products
  • used to scrape parts of build results out to the web frontend
    • e.g. manuals, ISO images, etc.
• BuildMetrics
  • scrapes data from magic location, similar to BuildProducts to show fancy graphs
    • e.g. test coverage, build times, CPU utilization for build
  • $out/nix-support/hydra-metrics
• BuildInputs
  • probably obsolete
• JobsetEvalMembers
  • joins evaluations with jobs
  • huge table, 10k’s of entries for one nixpkgs evaluation
  • can be imagined as a subset of the eval cache
    • could in principle use the eval cache

release.nix

• hydra-specific convention to describe the build
• should evaluate to an attribute set that contains derivations
• hydra considers every attribute in that set a job
• every job needs a unique name
  • if you want to build for multiple platforms, you need to reflect that in the name
• hydra does a deep traversal of the attribute set
  • just evaluating the names may take half an hour

FAQ

Can we imagine Hydra to be a persistence layer for the build graph?

• partially, it lacks a lot of information
  • does not keep edges of the build graph

How does Hydra relate to nix build?

• reimplements the top level Nix build loop, scheduling, etc.
• Hydra has to persist build results
• Hydra has more sophisticated remote build execution and scheduling than Nix

Is it conceptually possible to unify Hydra’s capabilities with regular Nix?

• Nix does not have any scheduling, it just traverses the build graph
• Hydra has scheduling in terms of job set priorities, tracks how much of a job set it has worked on
  • makes sure jobs don’t starve each other
• Both Hydra and Nix can dynamically add build jobs at runtime
  • Hydra queued up new jobs dynamically / on-line long before Nix.
  • But now, both Nix and Hydra now have experimental support for dynamic derivations, where build jobs can produce new derivations at build time
• Hydra queue runner is a long running process
  • Nix takes a static set of jobs, working it off at once

Hacking

This section provides some notes on how to hack on Hydra. To get the latest version of Hydra from GitHub:

$ git clone git://github.com/NixOS/hydra.git
$ cd hydra

To enter a shell in which all environment variables (such as PERL5LIB) and dependencies can be found:

$ nix develop

To build Hydra, you should then do:

$ mesonConfigurePhase
$ ninja

You start a local database, the webserver, and other components with foreman:

$ foreman start

The Hydra interface will be available on port 63333, with an admin user named “alice” with password “foobar”

You can run just the Hydra web server in your source tree as follows:

$ ./src/script/hydra-server

You can run Hydra’s test suite with the following:

$ meson test
# to run as many tests as you have cores:
$ YATH_JOB_COUNT=$NIX_BUILD_CORES meson test

To run individual tests:

# Run a specific test file
$ PERL5LIB=t/lib:$PERL5LIB perl t/test.pl t/Hydra/Controller/API/checks.t

# Run all tests in a directory
$ PERL5LIB=t/lib:$PERL5LIB perl t/test.pl t/Hydra/Controller/API/

Warning: Currently, the tests can fail if run with high parallelism due to an issue in Test::PostgreSQL causing database ports to collide.

Working on the Manual

By default, foreman start runs mdbook in “watch” mode. mdbook listens at http://localhost:63332/, and will reload the page every time you save.

Building

To build Hydra and its dependencies:

$ nix build .#packages.x86_64-linux.default

Development Tasks

Connecting to the database

Assuming you’re running the default configuration with foreman start, open an interactive session with Postgres via:

$ psql -h localhost -p 64444 hydra

Running the builder locally

To build and test the queue-runner:

$ cargo build -p hydra-queue-runner
$ cargo test -p hydra-queue-runner

foreman start launches the queue-runner automatically with the right config and environment. If a previous instance crashed, you may need to remove the stale lock file:

$ rm .hydra-data/queue-runner/lock

For hydra-queue-runner to successfully build locally, your development user will need to be “trusted” by your Nix store.

Add yourself to the trusted_users option of /etc/nix/nix.conf.

On NixOS:

{
  nix.settings.trusted-users = [ "YOURUSER" ];
}

Off NixOS, change /etc/nix/nix.conf:

trusted-users = root YOURUSERNAME

hydra-notify and Hydra’s Notifications

Hydra uses a notification-based subsystem to implement some features and support plugin development. Notifications are sent to hydra-notify, which is responsible for dispatching each notification to each plugin.

Notifications are passed from hydra-queue-runner to hydra-notify through Postgres’s NOTIFY and LISTEN feature.

Notification Types

Note that the notification format is subject to change and should not be considered an API. Integrate with hydra-notify instead of listening directly.

cached_build_finished

• Payload: Exactly two values, tab separated: the ID of the evaluation which contains the finished build, followed by the ID of the finished build.
• When: Issued directly after an evaluation completes, when that evaluation includes this finished build.
• Delivery Semantics: At most once per evaluation.

cached_build_queued

• Payload: Exactly two values, tab separated: The ID of the evaluation which contains the finished build, followed by the ID of the queued build.
• When: Issued directly after an evaluation completes, when that evaluation includes this queued build.
• Delivery Semantics: At most once per evaluation.

build_queued

• Payload: Exactly one value, the ID of the build.
• When: Issued after the transaction inserting the build in to the database is committed. One notification is sent per new build.
• Delivery Semantics: Ephemeral. hydra-notify must be running to react to this event. No record of this event is stored.

build_started

• Payload: Exactly one value, the ID of the build.
• When: Issued directly before building happens, and only if the derivation’s outputs cannot be substituted.
• Delivery Semantics: Ephemeral. hydra-notify must be running to react to this event. No record of this event is stored.

step_finished

• Payload: Three values, tab separated: the ID of the build which the step is part of, the step number, and the path on disk to the log file.
• When: Issued directly after a step completes, regardless of success. Is not issued if the step’s derivation’s outputs can be substituted.
• Delivery Semantics: Ephemeral. hydra-notify must be running to react to this event. No record of this event is stored.

build_finished

• Payload: At least one value, tab separated: the ID of the build which finished, followed by IDs of all of the builds which also depended upon this build.
• When: Issued directly after a build completes, regardless of success and substitutability.
• Delivery Semantics: At least once.

hydra-notify will call buildFinished for each plugin in two ways:

• The builds table’s notificationspendingsince column stores when the build finished. On startup, hydra-notify will query all builds with a non-null notificationspendingsince value and treat each row as a received build_finished event.

• Additionally, hydra-notify subscribes to build_finished events and processes them in real time.

After processing, the row’s notificationspendingsince column is set to null.

It is possible for subsequent deliveries of the same build_finished data to imply different outcomes. For example, if the build fails, is restarted, and then succeeds. In this scenario the build_finished events will be delivered at least twice, once for the failure and then once for the success.

eval_started

• Payload: Exactly two values, tab separated: an opaque trace ID representing this evaluation, and the ID of the jobset.
• When: At the beginning of the evaluation phase for the jobset, before any work is done.
• Delivery Semantics: Ephemeral. hydra-notify must be running to react to this event. No record of this event is stored.

eval_added

• Payload: Exactly three values, tab separated: an opaque trace ID representing this evaluation, the ID of the jobset, and the ID of the JobsetEval record.
• When: After the evaluator fetches inputs and completes the evaluation successfully.
• Delivery Semantics: Ephemeral. hydra-notify must be running to react to this event. No record of this event is stored.

eval_cached

• Payload: Exactly three values: an opaque trace ID representing this evaluation, the ID of the jobset, and the ID of the previous identical evaluation.
• When: After the evaluator fetches inputs, if none of the inputs changed.
• Delivery Semantics: Ephemeral. hydra-notify must be running to react to this event. No record of this event is stored.

eval_failed

• Payload: Exactly two values: an opaque trace ID representing this evaluation, and the ID of the jobset.
• When: After fetching any input fails, or any other evaluation error occurs.
• Delivery Semantics: Ephemeral. hydra-notify must be running to react to this event. No record of this event is stored.

Development Notes

Re-sending a notification

Notifications can be experimentally re-sent on the command line with psql, with NOTIFY $notificationname, '$payload'.

Authors

• Eelco Dolstra, Delft University of Technology, Department of Software Technology
• Rob Vermaas, Delft University of Technology, Department of Software Technology
• Eelco Visser, Delft University of Technology, Department of Software Technology
• Ludovic Courtès