Skip to content

Commit

Permalink
WIP More on derivation outputs
Browse files Browse the repository at this point in the history
  • Loading branch information
@Ericson2314
Ericson2314 committed Feb 10, 2025
1 parent cafefed commit 8c25607
Show file tree
Hide file tree
Showing 8 changed files with 298 additions and 171 deletions.
3 changes: 2 additions & 1 deletion doc/manual/source/SUMMARY.md.in
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@
- [Store Object](store/store-object.md)
- [Content-Addressing Store Objects](store/store-object/content-address.md)
- [Store Path](store/store-path.md)
- [Store Derivation and Deriving Path](store/drv.md)
- [Store Derivation and Deriving Path](store/derivation/index.md)
- [Derivation Outputs and Types of Derivations](store/derivation/outputs.md)
- [Building](store/building.md)
- [Store Types](store/types/index.md)
{{#include ./store/types/SUMMARY.md}}
Expand Down
6 changes: 3 additions & 3 deletions doc/manual/source/glossary.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@
- [store derivation]{#gloss-store-derivation}

A single build task.
See [Store Derivation](@docroot@/store/drv.md#store-derivation) for details.
See [Store Derivation](@docroot@/store/derivation/index.md#store-derivation) for details.

[store derivation]: #gloss-store-derivation

- [derivation path]{#gloss-derivation-path}

A [store path] which uniquely identifies a [store derivation].

See [Referencing Store Derivations](@docroot@/store/drv.md#derivation-path) for details.
See [Referencing Store Derivations](@docroot@/store/derivation/index.md#derivation-path) for details.

Not to be confused with [deriving path].

Expand Down Expand Up @@ -252,7 +252,7 @@

Deriving paths are a way to refer to [store objects][store object] that might not yet be [realised][realise].

See [Deriving Path](./store/drv.md#deriving-path) for details.
See [Deriving Path](./store/derivation/index.md#deriving-path) for details.

Not to be confused with [derivation path].

Expand Down
259 changes: 127 additions & 132 deletions doc/manual/source/language/advanced-attributes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,8 @@ Derivations can declare some infrequently used optional attributes.
to make it use the proxy server configuration specified by the user
in the environment variables `http_proxy` and friends.
This attribute is only allowed in *fixed-output derivations* (see
below), where impurities such as these are okay since (the hash
This attribute is only allowed in [fixed-output derivations][fixed-output derivation],
where impurities such as these are okay since (the hash
of) the output is known in advance. It is ignored for all other
derivations.
Expand All @@ -119,135 +119,6 @@ Derivations can declare some infrequently used optional attributes.
[`impure-env`](@docroot@/command-ref/conf-file.md#conf-impure-env)
configuration setting.
- [`outputHash`]{#adv-attr-outputHash}; [`outputHashAlgo`]{#adv-attr-outputHashAlgo}; [`outputHashMode`]{#adv-attr-outputHashMode}\
These attributes declare that the derivation is a so-called *fixed-output derivation* (FOD), which means that a cryptographic hash of the output is already known in advance.
As opposed to regular derivations, the [`builder`] executable of a fixed-output derivation has access to the network.
Nix computes a cryptographic hash of its output and compares that to the hash declared with these attributes.
If there is a mismatch, the derivation fails.
The rationale for fixed-output derivations is derivations such as
those produced by the `fetchurl` function. This function downloads a
file from a given URL. To ensure that the downloaded file has not
been modified, the caller must also specify a cryptographic hash of
the file. For example,
```nix
fetchurl {
url = "http://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz";
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
}
```
It sometimes happens that the URL of the file changes, e.g., because
servers are reorganised or no longer available. We then must update
the call to `fetchurl`, e.g.,
```nix
fetchurl {
url = "ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz";
sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
}
```
If a `fetchurl` derivation was treated like a normal derivation, the
output paths of the derivation and *all derivations depending on it*
would change. For instance, if we were to change the URL of the
Glibc source distribution in Nixpkgs (a package on which almost all
other packages depend) massive rebuilds would be needed. This is
unfortunate for a change which we know cannot have a real effect as
it propagates upwards through the dependency graph.
For fixed-output derivations, on the other hand, the name of the
output path only depends on the `outputHash*` and `name` attributes,
while all other attributes are ignored for the purpose of computing
the output path. (The `name` attribute is included because it is
part of the path.)
As an example, here is the (simplified) Nix expression for
`fetchurl`:
```nix
{ stdenv, curl }: # The curl program is used for downloading.
{ url, sha256 }:
stdenv.mkDerivation {
name = baseNameOf (toString url);
builder = ./builder.sh;
buildInputs = [ curl ];
# This is a fixed-output derivation; the output must be a regular
# file with SHA256 hash sha256.
outputHashMode = "flat";
outputHashAlgo = "sha256";
outputHash = sha256;
inherit url;
}
```
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings, and the [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.
The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
It can currently be `"blake3", "sha1"`, `"sha256"`, `"sha512"`, or `null`.
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format.
The `outputHashMode` attribute determines how the hash is computed.
It must be one of the following values:
- [`"flat"`](@docroot@/store/store-object/content-address.md#method-flat)
This is the default.
- [`"recursive"` or `"nar"`](@docroot@/store/store-object/content-address.md#method-nix-archive)
> **Compatibility**
>
> `"recursive"` is the traditional way of indicating this,
> and is supported since 2005 (virtually the entire history of Nix).
> `"nar"` is more clear, and consistent with other parts of Nix (such as the CLI),
> however support for it is only added in Nix version 2.21.
- [`"text"`](@docroot@/store/store-object/content-address.md#method-text)
> **Warning**
>
> The use of this method for derivation outputs is part of the [`dynamic-derivations`][xp-feature-dynamic-derivations] experimental feature.
- [`"git"`](@docroot@/store/store-object/content-address.md#method-git)
> **Warning**
>
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.
- [`__contentAddressed`]{#adv-attr-__contentAddressed}
> **Warning**
> This attribute is part of an [experimental feature](@docroot@/development/experimental-features.md).
>
> To use this attribute, you must enable the
> [`ca-derivations`][xp-feature-ca-derivations] experimental feature.
> For example, in [nix.conf](../command-ref/conf-file.md) you could add:
>
> ```
> extra-experimental-features = ca-derivations
> ```
If this attribute is set to `true`, then the derivation
outputs will be stored in a content-addressed location rather than the
traditional input-addressed one.
Setting this attribute also requires setting
[`outputHashMode`](#adv-attr-outputHashMode)
and
[`outputHashAlgo`](#adv-attr-outputHashAlgo)
like for *fixed-output derivations* (see above).
It also implicitly requires that the machine to build the derivation must have the `ca-derivations` [system feature](@docroot@/command-ref/conf-file.md#conf-system-features).
- [`passAsFile`]{#adv-attr-passAsFile}\
A list of names of attributes that should be passed via files rather
than environment variables. For example, if you have
Expand Down Expand Up @@ -370,6 +241,130 @@ Derivations can declare some infrequently used optional attributes.

ensures that the derivation can only be built on a machine with the `kvm` feature.

[xp-feature-ca-derivations]: @docroot@/development/experimental-features.md#xp-feature-ca-derivations
## Setting the derivation type

As discussed in [Derivation Outputs and Types of Derivations](@docroot@/store/derivation/outputs.md), there are multiples kinds of derivations / kinds of derivation outputs.
The choice of the following attributes determines which kind of derivation we are making.

- [`__contentAddressed`]

- [`outputHash`]

- [`outputHashAlgo`]

- [`outputHashMode`]

The three types of derivations are chosen based on the following combinations of these attributes.
All other combinations are invalid.

- [Input-addressing derivations](http://TODO)

This is the default for `builtins.derivation`.
Nix only currently supports one kind of input-addressing, so no other information is needed.

`__contentAddressed = false;` may also be included, but is not needed, and will trigger the experimental feature check.

- [Fixed-output derivations][fixed-output derivation]

All of [`outputHash`], [`outputHashAlgo`], and [`outputHashMode`].

`__contentAddressed` is ignored, becaused fixed-output derivations always content-address their outputs, by definition.

**TODO CHECK**

- [(Floating) content-addressing derivations](http://TODO)

Both [`outputHashAlgo`] and [`outputHashMode`], `__contentAddressed = true;`, and *not* `outputHash`.

If an output hash was given, then the derivation output would be "fixed" not "floating".

Here is more information on the `output*` attributes, and what values they may be set to:

- [`outputHashMode`]{#adv-attr-outputHashMode}

This specifies how the files of a content-addressing derivation output are digested to produce a content address.

This works in conjunction with [`outputHashAlgo`](#adv-attr-outputHashAlgo).
Specifying one without the other is an error (unless [`outputHash` is also specified and includes its own hash algorithm as described below).

The `outputHashMode` attribute determines how the hash is computed.
It must be one of the following values:

- [`"flat"`](@docroot@/store/store-object/content-address.md#method-flat)

This is the default.

- [`"recursive"` or `"nar"`](@docroot@/store/store-object/content-address.md#method-nix-archive)

> **Compatibility**
>
> `"recursive"` is the traditional way of indicating this,
> and is supported since 2005 (virtually the entire history of Nix).
> `"nar"` is more clear, and consistent with other parts of Nix (such as the CLI),
> however support for it is only added in Nix version 2.21.

- [`"text"`](@docroot@/store/store-object/content-address.md#method-text)

> **Warning**
>
> The use of this method for derivation outputs is part of the [`dynamic-derivations`][xp-feature-dynamic-derivations] experimental feature.

- [`"git"`](@docroot@/store/store-object/content-address.md#method-git)

> **Warning**
>
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.

See [content-addressing store objects](@docroot@/store/store-object/content-address.md) for more information about the process this flag controls.

- [`outputHashAlgo`]{#adv-attr-outputHashAlgo}

This specifies the hash alorithm used to digest the [file system object] data of a content-addressing derivation output.

This works in conjunction with [`outputHashMode`](#adv-attr-outputHashAlgo).
Specifying one without the other is an error (unless [`outputHash` is also specified and includes its own hash algorithm as described below).

The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
It can currently be `"blake3"`, "sha1"`, `"sha256"`, `"sha512"`, or `null`.

`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format, because in that case the choice of hash algorithm is determined by `outputHash`.

- [`outputHash`]{#adv-attr-outputHashAlgo}; [`outputHash`]{#adv-attr-outputHashMode}\

This will specify the output hash of the single output of a [fixed-output derivation].

The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.

> **Note**
>
> The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings.
> The [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.
- [`__contentAddressed`]{#adv-attr-__contentAddressed}

> **Warning**
>
> This attribute is part of an [experimental feature](@docroot@/development/experimental-features.md).
>
> To use this attribute, you must enable the
> [`ca-derivations`][xp-feature-ca-derivations] experimental feature.
> For example, in [nix.conf](../command-ref/conf-file.md) you could add:
>
> ```
> extra-experimental-features = ca-derivations
> ```
This is a boolean with a default of `false`.
It determines whether the derivation is floating content-addressing.
[`__contentAddressed`]: #adv-attr-__contentAddressed
[`outputHash`]: #adv-attr-outputHash
[`outputHashAlgo`]: #adv-attr-outputHashAlgo
[`outputHashMode`]: #adv-attr-outputHashMode
[fixed-output derivation]: @docroot@/glossary.md#gloss-fixed-output-derivation
[file system object]: @docroot@/store/file-system-object.md
[store object]: @docroot@/store/store-object.md
[xp-feature-dynamic-derivations]: @docroot@/development/experimental-features.md#xp-feature-dynamic-derivations
[xp-feature-git-hashing]: @docroot@/development/experimental-features.md#xp-feature-git-hashing
10 changes: 5 additions & 5 deletions doc/manual/source/language/derivations.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Derivations

The most important built-in function is `derivation`, which is used to describe a single store-layer [store derivation].
Consult the [store chapter](@docroot@/store/drv.md) for what a store derivation is;
Consult the [store chapter](@docroot@/store/derivation/index.md) for what a store derivation is;
this section just concerns how to create one from the Nix language.

This builtin function takes as input an attribute set, the attributes of which specify the inputs to the process.
Expand All @@ -16,7 +16,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
- [`name`]{#attr-name} ([String](@docroot@/language/types.md#type-string))

A symbolic name for the derivation.
See [derivation outputs](@docroot@/store/drv.md#outputs) for what this is affects.
See [derivation outputs](@docroot@/store/derivation/index.md#outputs) for what this is affects.

[store path]: @docroot@/store/store-path.md

Expand All @@ -34,7 +34,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
- [`system`]{#attr-system} ([String](@docroot@/language/types.md#type-string))
See [system](@docroot@/store/drv.md#system).
See [system](@docroot@/store/derivation/index.md#system).
> **Example**
>
Expand Down Expand Up @@ -64,7 +64,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
- [`builder`]{#attr-builder} ([Path](@docroot@/language/types.md#type-path) | [String](@docroot@/language/types.md#type-string))
See [builder](@docroot@/store/drv.md#builder).
See [builder](@docroot@/store/derivation/index.md#builder).
> **Example**
>
Expand Down Expand Up @@ -113,7 +113,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
Default: `[ ]`
See [args](@docroot@/store/drv.md#args).
See [args](@docroot@/store/derivation/index.md#args).
> **Example**
>
Expand Down
2 changes: 1 addition & 1 deletion doc/manual/source/store/building.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

## Builder Execution

The [`builder`](./drv.md#builder) is executed as follows:
The [`builder`](./derivation/index.md#builder) is executed as follows:

- A temporary directory is created under the directory specified by
`TMPDIR` (default `/tmp`) where the build will take place. The
Expand Down
Loading

0 comments on commit 8c25607

Please sign in to comment.