Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand manual on derivation outputs #12442

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
+326 −172
Draft

Expand manual on derivation outputs #12442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0678ddf
WIP More on derivation outputs
Ericson2314 Feb 10, 2025
ddee560
Add some text from Eelco's dissertation
Ericson2314 Feb 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/manual/source/SUMMARY.md.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@
- [Store Object](store/store-object.md)
- [Content-Addressing Store Objects](store/store-object/content-address.md)
- [Store Path](store/store-path.md)
- [Store Derivation and Deriving Path](store/drv.md)
- [Store Derivation and Deriving Path](store/derivation/index.md)
- [Derivation Outputs and Types of Derivations](store/derivation/outputs.md)
- [Building](store/building.md)
- [Store Types](store/types/index.md)
{{#include ./store/types/SUMMARY.md}}
Expand Down
6 changes: 3 additions & 3 deletions doc/manual/source/glossary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@
- [store derivation]{#gloss-store-derivation}

A single build task.
See [Store Derivation](@docroot@/store/drv.md#store-derivation) for details.
See [Store Derivation](@docroot@/store/derivation/index.md#store-derivation) for details.

[store derivation]: #gloss-store-derivation

- [derivation path]{#gloss-derivation-path}

A [store path] which uniquely identifies a [store derivation].

See [Referencing Store Derivations](@docroot@/store/drv.md#derivation-path) for details.
See [Referencing Store Derivations](@docroot@/store/derivation/index.md#derivation-path) for details.

Not to be confused with [deriving path].

Expand Down Expand Up @@ -252,7 +252,7 @@

Deriving paths are a way to refer to [store objects][store object] that might not yet be [realised][realise].

See [Deriving Path](./store/drv.md#deriving-path) for details.
See [Deriving Path](./store/derivation/index.md#deriving-path) for details.

Not to be confused with [derivation path].

Expand Down
259 changes: 127 additions & 132 deletions doc/manual/source/language/advanced-attributes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,8 @@ Derivations can declare some infrequently used optional attributes.
to make it use the proxy server configuration specified by the user
in the environment variables `http_proxy` and friends.

This attribute is only allowed in *fixed-output derivations* (see
below), where impurities such as these are okay since (the hash
This attribute is only allowed in [fixed-output derivations][fixed-output derivation],
where impurities such as these are okay since (the hash
of) the output is known in advance. It is ignored for all other
derivations.

Expand All @@ -119,135 +119,6 @@ Derivations can declare some infrequently used optional attributes.
[`impure-env`](@docroot@/command-ref/conf-file.md#conf-impure-env)
configuration setting.

- [`outputHash`]{#adv-attr-outputHash}; [`outputHashAlgo`]{#adv-attr-outputHashAlgo}; [`outputHashMode`]{#adv-attr-outputHashMode}\
These attributes declare that the derivation is a so-called *fixed-output derivation* (FOD), which means that a cryptographic hash of the output is already known in advance.

As opposed to regular derivations, the [`builder`] executable of a fixed-output derivation has access to the network.
Nix computes a cryptographic hash of its output and compares that to the hash declared with these attributes.
If there is a mismatch, the derivation fails.

The rationale for fixed-output derivations is derivations such as
those produced by the `fetchurl` function. This function downloads a
file from a given URL. To ensure that the downloaded file has not
been modified, the caller must also specify a cryptographic hash of
the file. For example,

```nix
fetchurl {
url = "http://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz";
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
}
```

It sometimes happens that the URL of the file changes, e.g., because
servers are reorganised or no longer available. We then must update
the call to `fetchurl`, e.g.,

```nix
fetchurl {
url = "ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz";
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
}
```

If a `fetchurl` derivation was treated like a normal derivation, the
output paths of the derivation and *all derivations depending on it*
would change. For instance, if we were to change the URL of the
Glibc source distribution in Nixpkgs (a package on which almost all
other packages depend) massive rebuilds would be needed. This is
unfortunate for a change which we know cannot have a real effect as
it propagates upwards through the dependency graph.

For fixed-output derivations, on the other hand, the name of the
output path only depends on the `outputHash*` and `name` attributes,
while all other attributes are ignored for the purpose of computing
the output path. (The `name` attribute is included because it is
part of the path.)

As an example, here is the (simplified) Nix expression for
`fetchurl`:

```nix
{ stdenv, curl }: # The curl program is used for downloading.

{ url, sha256 }:

stdenv.mkDerivation {
name = baseNameOf (toString url);
builder = ./builder.sh;
buildInputs = [ curl ];

# This is a fixed-output derivation; the output must be a regular
# file with SHA256 hash sha256.
outputHashMode = "flat";
outputHashAlgo = "sha256";
outputHash = sha256;

inherit url;
}
```
Comment on lines -135 to -188
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This motivation is taken from (going back far enough) Eelco's dissertation; it should be reworded to not use the language, since in this new context, we are describing the store without reference to the language.


The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings, and the [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.

The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
It can currently be `"blake3", "sha1"`, `"sha256"`, `"sha512"`, or `null`.
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format.

The `outputHashMode` attribute determines how the hash is computed.
It must be one of the following values:

- [`"flat"`](@docroot@/store/store-object/content-address.md#method-flat)

This is the default.

- [`"recursive"` or `"nar"`](@docroot@/store/store-object/content-address.md#method-nix-archive)

> **Compatibility**
>
> `"recursive"` is the traditional way of indicating this,
> and is supported since 2005 (virtually the entire history of Nix).
> `"nar"` is more clear, and consistent with other parts of Nix (such as the CLI),
> however support for it is only added in Nix version 2.21.

- [`"text"`](@docroot@/store/store-object/content-address.md#method-text)

> **Warning**
>
> The use of this method for derivation outputs is part of the [`dynamic-derivations`][xp-feature-dynamic-derivations] experimental feature.

- [`"git"`](@docroot@/store/store-object/content-address.md#method-git)

> **Warning**
>
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.

- [`__contentAddressed`]{#adv-attr-__contentAddressed}

> **Warning**
> This attribute is part of an [experimental feature](@docroot@/development/experimental-features.md).
>
> To use this attribute, you must enable the
> [`ca-derivations`][xp-feature-ca-derivations] experimental feature.
> For example, in [nix.conf](../command-ref/conf-file.md) you could add:
>
> ```
> extra-experimental-features = ca-derivations
> ```

If this attribute is set to `true`, then the derivation
outputs will be stored in a content-addressed location rather than the
traditional input-addressed one.

Setting this attribute also requires setting
[`outputHashMode`](#adv-attr-outputHashMode)
and
[`outputHashAlgo`](#adv-attr-outputHashAlgo)
like for *fixed-output derivations* (see above).

It also implicitly requires that the machine to build the derivation must have the `ca-derivations` [system feature](@docroot@/command-ref/conf-file.md#conf-system-features).

- [`passAsFile`]{#adv-attr-passAsFile}\
A list of names of attributes that should be passed via files rather
than environment variables. For example, if you have
Expand Down Expand Up @@ -370,6 +241,130 @@ Derivations can declare some infrequently used optional attributes.

ensures that the derivation can only be built on a machine with the `kvm` feature.

[xp-feature-ca-derivations]: @docroot@/development/experimental-features.md#xp-feature-ca-derivations
## Setting the derivation type

As discussed in [Derivation Outputs and Types of Derivations](@docroot@/store/derivation/outputs.md), there are multiples kinds of derivations / kinds of derivation outputs.
The choice of the following attributes determines which kind of derivation we are making.

- [`__contentAddressed`]

- [`outputHash`]

- [`outputHashAlgo`]

- [`outputHashMode`]

The three types of derivations are chosen based on the following combinations of these attributes.
All other combinations are invalid.

- [Input-addressing derivations](http://TODO)

This is the default for `builtins.derivation`.
Nix only currently supports one kind of input-addressing, so no other information is needed.

`__contentAddressed = false;` may also be included, but is not needed, and will trigger the experimental feature check.

- [Fixed-output derivations][fixed-output derivation]

All of [`outputHash`], [`outputHashAlgo`], and [`outputHashMode`].

`__contentAddressed` is ignored, becaused fixed-output derivations always content-address their outputs, by definition.

**TODO CHECK**

- [(Floating) content-addressing derivations](http://TODO)

Both [`outputHashAlgo`] and [`outputHashMode`], `__contentAddressed = true;`, and *not* `outputHash`.

If an output hash was given, then the derivation output would be "fixed" not "floating".

Here is more information on the `output*` attributes, and what values they may be set to:

- [`outputHashMode`]{#adv-attr-outputHashMode}

This specifies how the files of a content-addressing derivation output are digested to produce a content address.

This works in conjunction with [`outputHashAlgo`](#adv-attr-outputHashAlgo).
Specifying one without the other is an error (unless [`outputHash` is also specified and includes its own hash algorithm as described below).

The `outputHashMode` attribute determines how the hash is computed.
It must be one of the following values:

- [`"flat"`](@docroot@/store/store-object/content-address.md#method-flat)

This is the default.

- [`"recursive"` or `"nar"`](@docroot@/store/store-object/content-address.md#method-nix-archive)

> **Compatibility**
>
> `"recursive"` is the traditional way of indicating this,
> and is supported since 2005 (virtually the entire history of Nix).
> `"nar"` is more clear, and consistent with other parts of Nix (such as the CLI),
> however support for it is only added in Nix version 2.21.

- [`"text"`](@docroot@/store/store-object/content-address.md#method-text)

> **Warning**
>
> The use of this method for derivation outputs is part of the [`dynamic-derivations`][xp-feature-dynamic-derivations] experimental feature.

- [`"git"`](@docroot@/store/store-object/content-address.md#method-git)

> **Warning**
>
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.

See [content-addressing store objects](@docroot@/store/store-object/content-address.md) for more information about the process this flag controls.

- [`outputHashAlgo`]{#adv-attr-outputHashAlgo}

This specifies the hash alorithm used to digest the [file system object] data of a content-addressing derivation output.

This works in conjunction with [`outputHashMode`](#adv-attr-outputHashAlgo).
Specifying one without the other is an error (unless [`outputHash` is also specified and includes its own hash algorithm as described below).

The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
It can currently be `"blake3"`, "sha1"`, `"sha256"`, `"sha512"`, or `null`.

`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format, because in that case the choice of hash algorithm is determined by `outputHash`.

- [`outputHash`]{#adv-attr-outputHashAlgo}; [`outputHash`]{#adv-attr-outputHashMode}\

This will specify the output hash of the single output of a [fixed-output derivation].

The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.

> **Note**
>
> The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings.
> The [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.

- [`__contentAddressed`]{#adv-attr-__contentAddressed}

> **Warning**
>
> This attribute is part of an [experimental feature](@docroot@/development/experimental-features.md).
>
> To use this attribute, you must enable the
> [`ca-derivations`][xp-feature-ca-derivations] experimental feature.
> For example, in [nix.conf](../command-ref/conf-file.md) you could add:
>
> ```
> extra-experimental-features = ca-derivations
> ```

This is a boolean with a default of `false`.
It determines whether the derivation is floating content-addressing.

[`__contentAddressed`]: #adv-attr-__contentAddressed
[`outputHash`]: #adv-attr-outputHash
[`outputHashAlgo`]: #adv-attr-outputHashAlgo
[`outputHashMode`]: #adv-attr-outputHashMode

[fixed-output derivation]: @docroot@/glossary.md#gloss-fixed-output-derivation
[file system object]: @docroot@/store/file-system-object.md
[store object]: @docroot@/store/store-object.md
[xp-feature-dynamic-derivations]: @docroot@/development/experimental-features.md#xp-feature-dynamic-derivations
[xp-feature-git-hashing]: @docroot@/development/experimental-features.md#xp-feature-git-hashing
10 changes: 5 additions & 5 deletions doc/manual/source/language/derivations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Derivations

The most important built-in function is `derivation`, which is used to describe a single store-layer [store derivation].
Consult the [store chapter](@docroot@/store/drv.md) for what a store derivation is;
Consult the [store chapter](@docroot@/store/derivation/index.md) for what a store derivation is;
this section just concerns how to create one from the Nix language.

This builtin function takes as input an attribute set, the attributes of which specify the inputs to the process.
Expand All @@ -16,7 +16,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
- [`name`]{#attr-name} ([String](@docroot@/language/types.md#type-string))

A symbolic name for the derivation.
See [derivation outputs](@docroot@/store/drv.md#outputs) for what this is affects.
See [derivation outputs](@docroot@/store/derivation/index.md#outputs) for what this is affects.

[store path]: @docroot@/store/store-path.md

Expand All @@ -34,7 +34,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect

- [`system`]{#attr-system} ([String](@docroot@/language/types.md#type-string))

See [system](@docroot@/store/drv.md#system).
See [system](@docroot@/store/derivation/index.md#system).

> **Example**
>
Expand Down Expand Up @@ -64,7 +64,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect

- [`builder`]{#attr-builder} ([Path](@docroot@/language/types.md#type-path) | [String](@docroot@/language/types.md#type-string))

See [builder](@docroot@/store/drv.md#builder).
See [builder](@docroot@/store/derivation/index.md#builder).

> **Example**
>
Expand Down Expand Up @@ -113,7 +113,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect

Default: `[ ]`

See [args](@docroot@/store/drv.md#args).
See [args](@docroot@/store/derivation/index.md#args).

> **Example**
>
Expand Down
2 changes: 1 addition & 1 deletion doc/manual/source/store/building.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

## Builder Execution

The [`builder`](./drv.md#builder) is executed as follows:
The [`builder`](./derivation/index.md#builder) is executed as follows:

- A temporary directory is created under the directory specified by
`TMPDIR` (default `/tmp`) where the build will take place. The
Expand Down
Loading
Loading