Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Configuration

This chapter is a collection of configuration snippets for different scenarios.

The configuration is parsed by Config::General which has a pretty thorough documentation on their file format. Hydra calls the parser with the following options:

  • -UseApacheInclude => 1
  • -IncludeAgain => 1
  • -IncludeRelative => 1

Including files

hydra.conf supports Apache-style includes. This is IMPORTANT because that is how you keep your secrets out of the Nix store. Hopefully this got your attention 😌

This:

<github_authorization>
NixOS = Bearer gha-secret😱secret😱secret😱
</github_authorization>

should NOT be in hydra.conf.

hydra.conf is rendered in the Nix store and is therefore world-readable.

Instead, the above should be written to a file outside the Nix store by other means (manually, using Nixops’ secrets feature, etc) and included like so:

Include /run/keys/hydra/github_authorizations.conf

Serving behind reverse proxy

To serve hydra web server behind reverse proxy like nginx or httpd some additional configuration must be made.

Edit your hydra.conf file in a similar way to this example:

using_frontend_proxy 1
base_uri example.com

base_uri should be your hydra servers proxied URL. If you are using Hydra nixos module then setting hydraURL option should be enough.

You also need to configure your reverse proxy to pass X-Request-Base to hydra, with the same value as base_uri. This also covers the case of serving Hydra with a prefix path, as in http://example.com/hydra.

For example if you are using nginx, then use configuration similar to following:

server {
    listen 433 ssl;
    server_name example.com;
    .. other configuration ..
    location /hydra/ {

        proxy_pass http://127.0.0.1:3000/;

        proxy_set_header  Host              $host;
        proxy_set_header  X-Real-IP         $remote_addr;
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
        proxy_set_header  X-Request-Base    /hydra;
    }
}

Note the trailing slash on the proxy_pass directive, which causes nginx to strip off the /hydra/ part of the URL before passing it to hydra.

Populating a Cache

A common use for Hydra is to pre-build and cache derivations which take a long time to build. While it is possible to direcly access the Hydra server’s store over SSH, a more scalable option is to upload built derivations to a remote store like an S3-compatible object store. Setting the store_uri parameter will cause Hydra to sign and upload derivations as they are built:

store_uri = s3://cache-bucket-name?compression=zstd&parallel-compression=true&write-nar-listing=1&ls-compression=br&log-compression=br&secret-key=/path/to/cache/private/key

This example uses Zstandard compression on derivations to reduce CPU usage on the server, but Brotli compression for derivation listings and build logs because it has better browser support.

See nix help stores for a description of the store URI format.

Statsd Configuration

By default, Hydra will send stats to statsd at localhost:8125. Point Hydra to a different server via:

<statsd>
  host = alternative.host
  port = 18125
</statsd>

hydra-notify’s Prometheus service

hydra-notify supports running a Prometheus webserver for metrics. The exporter does not run unless a listen address and port are specified in the hydra configuration file, as below:

<hydra_notify>
  <prometheus>
    listen_address = 127.0.0.1
    port = 9199
  </prometheus>
</hydra_notify>

hydra-queue-runner’s Prometheus service

hydra-queue-runner supports running a Prometheus webserver for metrics. The exporter’s address defaults to exposing on 127.0.0.1:9198, but is also configurable through the hydra configuration file and a command line argument, as below. A port of :0 will make the exposer choose a random, available port.

queue_runner_metrics_address = 127.0.0.1:9198
# or
queue_runner_metrics_address = [::]:9198

$ hydra-queue-runner --prometheus-address 127.0.0.1:9198
# or
$ hydra-queue-runner --prometheus-address [::]:9198

Overflow binary cache (optional)

The queue runner can upload builds of selected jobsets to a separate “overflow” S3 bucket, for example to keep large staging rebuilds out of the main cache. A step is uploaded to the overflow bucket only when every jobset referencing it is listed. Shared steps go to the default bucket.

Both buckets must live on the same S3 endpoint and use static credentials: when a later build from a regular jobset needs outputs that only exist in the overflow bucket, the queue runner copies them back to the default bucket server-side instead of rebuilding.

Configured in queue-runner.toml:

[overflowStore]
store = "s3://hydra-overflow?region=eu-west-1"
jobsets = ["nixpkgs:staging-next"]

Ad hoc builds (optional, experimental)

Hydra normally only builds what its evaluator queues. The optional hydra-ad-hoc service, new and still experimental, additionally lets a Nix client submit builds directly. It serves the nix daemon protocol on a Unix socket, so from the client’s point of view the whole Hydra deployment is one giant nix daemon. A derivation realised through it is filed as a Hydra build under a hidden adhoc/adhoc jobset and built by the queue runner and its builders. Read operations and store uploads are proxied to the upstream nix-daemon.

Nothing inside Hydra uses this socket. It exists for ad hoc jobs and ad hoc store usage from outside, and can be left disabled. Expect its interface and limitations to change.

On NixOS, enable it alongside the queue runner:

{
  services.hydra-ad-hoc-dev.enable = true;
}

This creates /run/hydra-ad-hoc/socket, with a hydra-ad-hoc group of its own. Add the users who may submit builds to that group. It is separate from hydra, the group Hydra’s services run as, on purpose: this socket is for use from outside Hydra.

Be aware of what membership grants. The daemon is a trusted user of the upstream nix-daemon, and it forwards uploads and build requests on a client’s behalf without any checks of its own yet. So, until the daemon gains its own access control, anyone in the hydra-ad-hoc group is effectively a trusted Nix user on the coordinator, and the group should be handed out as carefully as trusted-users.

The service reads /etc/hydra/ad-hoc.toml, generated from services.hydra-ad-hoc-dev.settings (dbUrl, maxDbConnections, upstreamSocket, storeDir, hydraDataDir). The defaults suit a single-host install.

A client then points its store at the socket:

$ nix-store --store unix:///run/hydra-ad-hoc/socket --realise /nix/store/...-hello.drv

The .drv must be in the coordinator’s store; a nix-build of an expression uploads it through the daemon as part of instantiation. Content-addressed and dynamic derivations work too: the daemon answers output-path queries from what the queue runner recorded, and builds a dynamic derivation’s producer first.

While the client waits, the daemon streams the build’s progress back to it the way a local build would report it. Each build step the queue runner dispatches appears as a build activity, and the step’s log lines follow as they are written. So nix build -L shows the log live, and --log-format bar-with-logs does the same for the classic commands. The daemon reads the logs from hydraDataDir, so it has to run on the same host as the queue runner.

Using LDAP as authentication backend (optional)

Instead of using Hydra’s built-in user management you can optionally use LDAP to manage roles and users.

This is configured by defining the <ldap> block in the configuration file. In this block it’s possible to configure the authentication plugin in the <config> block. All options are directly passed to Catalyst::Authentication::Store::LDAP. The documentation for the available settings can be found here.

Note that the bind password (if needed) should be supplied as an included file to prevent it from leaking to the Nix store.

Roles can be assigned to users based on their LDAP group membership. For this to work use_roles = 1 needs to be defined for the authentication plugin. LDAP groups can then be mapped to Hydra roles using the <role_mapping> block.

Example configuration:

<ldap>
  <config>
    <credential>
      class = Password
      password_field = password
      password_type = self_check
    </credential>
    <store>
      class = LDAP
      ldap_server = localhost
      <ldap_server_options>
        timeout = 30
      </ldap_server_options>
      binddn = "cn=root,dc=example"
      include ldap-password.conf
      start_tls = 0
      <start_tls_options>
        verify = none
      </start_tls_options>
      user_basedn = "ou=users,dc=example"
      user_filter = "(&(objectClass=inetOrgPerson)(cn=%s))"
      user_scope = one
      user_field = cn
      <user_search_options>
        deref = always
      </user_search_options>
      # Important for role mappings to work:
      use_roles = 1
      role_basedn = "ou=groups,dc=example"
      role_filter = "(&(objectClass=groupOfNames)(member=%s))"
      role_scope = one
      role_field = cn
      role_value = dn
      <role_search_options>
        deref = always
      </role_search_options>
    </store>
  </config>
  <role_mapping>
    # Make all users in the hydra_admin group Hydra admins
    hydra_admin = admin
    # Allow all users in the dev group to eval jobsets, restart jobs and cancel builds
    dev = eval-jobset
    dev = restart-jobs
    dev = cancel-build
  </role_mapping>
</ldap>

Then, place the password to your LDAP server in /var/lib/hydra/ldap-password.conf:

bindpw = the-ldap-password

Debugging LDAP

Set the debug parameter under ldap.config.ldap_server_options.debug:

<ldap>
  <config>
    <store>
      <ldap_server_options>
        debug = 2
      </ldap_server_options>
    </store>
  </config>
</ldap>

Legacy LDAP Configuration

Hydra used to load the LDAP configuration from a YAML file in the HYDRA_LDAP_CONFIG environment variable. This behavior is deperecated and will be removed.

When Hydra uses the deprecated YAML file, Hydra applies the following default role mapping:

<ldap>
  <role_mapping>
    hydra_admin = admin
    hydra_bump-to-front = bump-to-front
    hydra_cancel-build = cancel-build
    hydra_create-projects = create-projects
    hydra_restart-jobs = restart-jobs
  </role_mapping>
</ldap>

Note that configuring both the LDAP parameters in the hydra.conf and via the environment variable is a fatal error.

Turning off Hydra’s own user management

By default anyone with a Hydra account can sign in with a user name and a password. Set

local_auth_enabled = 0

to turn that off, for a Hydra whose users are meant to come from somewhere else.

This does not affect LDAP, which is a separate backend that happens to use the same form: with LDAP configured, the form stays, labelled “Sign in with LDAP”, and works as before. It does not affect OIDC or GitHub either, which have endpoints of their own.

Turning off the password form leaves nobody able to sign in at all if no other method is configured, which on a private Hydra means nobody gets in; Hydra warns about that at startup.

Local accounts are not deleted by this and hydra-create-user still works, so switching it back on needs no changes to the users.

Single sign-on with OIDC

Hydra can delegate login to one or more OpenID Connect providers, configured in the <oidc> block:

<oidc>
  <provider authentik>
    display_name = "Authentik"
    discovery_url = "https://authentik.example.com/application/o/hydra/.well-known/openid-configuration"
    client_id = "hydra"
    client_secret_file = "/var/lib/hydra/secrets/authentik-client-secret"
    role_claim = "groups"
    <role_mapping>
      hydra-admins = admin
      hydra-builders = create-projects
      hydra-builders = eval-jobset
    </role_mapping>
  </provider>
</oidc>

The role_mapping block translates the values the IdP sends in its role claim into Hydra roles, so the two need not use the same names. Hydra validates the mapping at startup.

See the OIDC documentation for all settings, role handling, sign-out, and examples for Authentik and GitHub.

Webhook Authentication

Hydra supports authenticating webhook requests from GitHub and Gitea to prevent unauthorized job evaluations. Webhook secrets should be stored in separate files outside the Nix store for security using Config::General’s include mechanism.

In your main hydra.conf:

<webhooks>
  Include /var/lib/hydra/secrets/webhook-secrets.conf
</webhooks>

Then create /var/lib/hydra/secrets/webhook-secrets.conf with your actual secrets:

<github>
  secret = your-github-webhook-secret
</github>
<gitea>
  secret = your-gitea-webhook-secret
</gitea>

For multiple secrets (useful for rotation or multiple environments), use an array:

<github>
  secret = your-github-webhook-secret-prod
  secret = your-github-webhook-secret-staging
</github>

Important: The secrets file should have restricted permissions (e.g., 0600) to prevent unauthorized access. See the Webhooks documentation for detailed setup instructions.

Embedding Extra HTML

Embed an analytics widget or other HTML in the <head> of each HTML document via:

tracker = <script src="...">