Document Store Derivations and Deriving Paths (#12290)

This is a big step documenting the store layer on its own, separately from the evaluator (and `builtins.derivation`).

Co-authored-by: Robert Hensing <[email protected]>

doc/manual/redirects.js

@@ -346,6 +346,9 @@ const redirects = {
     "scoping-rules": "scoping.html",
     "string-literal": "string-literals.html",
   },
+  "language/derivations.md": {
+    "builder-execution": "store/drv/building.md#builder-execution",
+  },
   "installation/installing-binary.html": {
     "linux": "uninstall.html#linux",
     "macos": "uninstall.html#macos",

doc/manual/source/SUMMARY.md.in

@@ -22,6 +22,8 @@
   - [Store Object](store/store-object.md)
     - [Content-Addressing Store Objects](store/store-object/content-address.md)
   - [Store Path](store/store-path.md)
+  - [Store Derivation and Deriving Path](store/drv.md)
+  - [Building](store/building.md)
   - [Store Types](store/types/index.md)
 {{#include ./store/types/SUMMARY.md}}
 - [Nix Language](language/index.md)

doc/manual/source/architecture/architecture.md

@@ -69,7 +69,7 @@ It can also execute build plans to produce new data, which are made available to
 A build plan itself is a series of *build tasks*, together with their build inputs.
 
 > **Important**
-> A build task in Nix is called [derivation](@docroot@/glossary.md#gloss-derivation).
+> A build task in Nix is called [store derivation](@docroot@/glossary.md#gloss-store-derivation).
 
 Each build task has a special build input executed as *build instructions* in order to perform the build.
 The result of a build task can be input to another build task.

doc/manual/source/command-ref/nix-env/install.md

@@ -22,11 +22,11 @@ It is based on the current generation of the active [profile](@docroot@/command-
 
 The arguments *args* map to store paths in a number of possible ways:
 
-- By default, *args* is a set of [derivation] names denoting derivations in the [default Nix expression].
+- By default, *args* is a set of names denoting derivations in the [default Nix expression].
   These are [realised], and the resulting output paths are installed.
   Currently installed derivations with a name equal to the name of a derivation being added are removed unless the option `--preserve-installed` is specified.
 
-  [derivation]: @docroot@/glossary.md#gloss-derivation
+  [derivation expression]: @docroot@/glossary.md#gloss-derivation-expression
   [default Nix expression]: @docroot@/command-ref/files/default-nix-expression.md
   [realised]: @docroot@/glossary.md#gloss-realise
 
@@ -66,11 +66,11 @@ The arguments *args* map to store paths in a number of possible ways:
   This can be used to override the priority of the derivations being installed.
   This is useful if *args* are [store paths], which don't have any priority information.
 
-- If *args* are [store derivations](@docroot@/glossary.md#gloss-store-derivation), then these are [realised], and the resulting output paths are installed.
+- If *args* are [store paths] that point to [store derivations][store derivation], then those store derivations are [realised], and the resulting output paths are installed.
 
-- If *args* are [store paths] that are not store derivations, then these are [realised] and installed.
+- If *args* are [store paths] that do not point to store derivations, then these are [realised] and installed.
 
-- By default all [outputs](@docroot@/language/derivations.md#attr-outputs) are installed for each [derivation].
+- By default all [outputs](@docroot@/language/derivations.md#attr-outputs) are installed for each [store derivation].
   This can be overridden by adding a `meta.outputsToInstall` attribute on the derivation listing a subset of the output names.
 
   Example:
@@ -122,6 +122,8 @@ The arguments *args* map to store paths in a number of possible ways:
   manifest.nix
   ```
 
+[store derivation]: @docroot@/glossary.md#gloss-store-derivation
+
 # Options
 
 - `--prebuilt-only` / `-b`

doc/manual/source/command-ref/nix-env/query.md

@@ -125,7 +125,10 @@ derivation is shown unless `--no-name` is specified.
 
   - `--drv-path`
 
-    Print the path of the [store derivation](@docroot@/glossary.md#gloss-store-derivation).
+    Print the [store path] to the [store derivation].
+
+    [store path]: @docroot@/glossary.md#gloss-store-path
+    [store derivation]: @docroot@/glossary.md#gloss-derivation
 
   - `--out-path`
 

doc/manual/source/command-ref/nix-instantiate.md

@@ -42,8 +42,8 @@ standard input.
 - `--eval`
 
   Just parse and evaluate the input files, and print the resulting
-  values on standard output. No instantiation of store derivations
-  takes place.
+  values on standard output.
+  Store derivations are not serialized and written to the store, but instead just hashed and discarded.
 
   > **Warning**
   >

doc/manual/source/glossary.md

@@ -19,31 +19,35 @@
 
   Besides content addressing, the Nix store also uses [input addressing](#gloss-input-addressed-store-object).
 
-- [derivation]{#gloss-derivation}
+- [store derivation]{#gloss-store-derivation}
 
-  A description of a build task. The result of a derivation is a
-  store object. Derivations declared in Nix expressions are specified
-  using the [`derivation` primitive](./language/derivations.md). These are
-  translated into low-level *store derivations* (implicitly by
-  `nix-build`, or explicitly by `nix-instantiate`).
+  A single build task.
+  See [Store Derivation](@docroot@/store/drv.md#store-derivation) for details.
 
-  [derivation]: #gloss-derivation
+  [store derivation]: #gloss-store-derivation
 
-- [store derivation]{#gloss-store-derivation}
+- [derivation path]{#gloss-derivation-path}
 
-  A [derivation] represented as a `.drv` file in the [store].
-  It has a [store path], like any [store object].
-  It is the [instantiated][instantiate] form of a derivation.
+  A [store path] which uniquely identifies a [store derivation].
 
-  Example: `/nix/store/g946hcz4c8mdvq2g8vxx42z51qb71rvp-git-2.38.1.drv`
+  See [Referencing Store Derivations](@docroot@/store/drv.md#derivation-path) for details.
 
-  See [`nix derivation show`](./command-ref/new-cli/nix3-derivation-show.md) (experimental) for displaying the contents of store derivations.
+  Not to be confused with [deriving path].
 
-  [store derivation]: #gloss-store-derivation
+  [derivation path]: #gloss-derivation-path
+
+- [derivation expression]{#gloss-derivation-expression}
+
+  A description of a [store derivation] in the Nix language.
+  The output(s) of a derivation are store objects.
+  Derivations are typically specified in Nix expressions using the [`derivation` primitive](./language/derivations.md).
+  These are translated into store layer *derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`).
+
+  [derivation expression]: #gloss-derivation-expression
 
 - [instantiate]{#gloss-instantiate}, instantiation
 
-  Save an evaluated [derivation] as a [store derivation] in the Nix [store].
+  Translate a [derivation expression] into a [store derivation].
 
   See [`nix-instantiate`](./command-ref/nix-instantiate.md), which produces a store derivation from a Nix expression that evaluates to a derivation.
 
@@ -55,7 +59,7 @@
 
   This can be achieved by:
   - Fetching a pre-built [store object] from a [substituter]
-  - Running the [`builder`](@docroot@/language/derivations.md#attr-builder) executable as specified in the corresponding [derivation]
+  - Running the [`builder`](@docroot@/language/derivations.md#attr-builder) executable as specified in the corresponding [store derivation]
   - Delegating to a [remote machine](@docroot@/command-ref/conf-file.md#conf-builders) and retrieving the outputs
   <!-- TODO: link [running] to build process page, #8888 -->
 
@@ -73,7 +77,7 @@
 
 - [fixed-output derivation]{#gloss-fixed-output-derivation} (FOD)
 
-  A [derivation] where a cryptographic hash of the [output] is determined in advance using the [`outputHash`](./language/advanced-attributes.md#adv-attr-outputHash) attribute, and where the [`builder`](@docroot@/language/derivations.md#attr-builder) executable has access to the network.
+  A [store derivation] where a cryptographic hash of the [output] is determined in advance using the [`outputHash`](./language/advanced-attributes.md#adv-attr-outputHash) attribute, and where the [`builder`](@docroot@/language/derivations.md#attr-builder) executable has access to the network.
 
 - [store]{#gloss-store}
 
@@ -188,7 +192,7 @@
   >
   > The contents of a `.nix` file form a Nix expression.
 
-  Nix expressions specify [derivations][derivation], which are [instantiated][instantiate] into the Nix store as [store derivations][store derivation].
+  Nix expressions specify [derivation expressions][derivation expression], which are [instantiated][instantiate] into the Nix store as [store derivations][store derivation].
   These derivations can then be [realised][realise] to produce [outputs][output].
 
   > **Example**
@@ -230,14 +234,14 @@
 
 - [output]{#gloss-output}
 
-  A [store object] produced by a [derivation].
+  A [store object] produced by a [store derivation].
   See [the `outputs` argument to the `derivation` function](@docroot@/language/derivations.md#attr-outputs) for details.
 
   [output]: #gloss-output
 
 - [output path]{#gloss-output-path}
 
-  The [store path] to the [output] of a [derivation].
+  The [store path] to the [output] of a [store derivation].
 
   [output path]: #gloss-output-path
 
@@ -246,14 +250,11 @@
 
 - [deriving path]{#gloss-deriving-path}
 
-  Deriving paths are a way to refer to [store objects][store object] that ar not yet [realised][realise].
-  This is necessary because, in general and particularly for [content-addressed derivations][content-addressed derivation], the [output path] of an [output] is not known in advance.
-  There are two forms:
+  Deriving paths are a way to refer to [store objects][store object] that might not yet be [realised][realise].
 
-  - *constant*: just a [store path]
-    It can be made [valid][validity] by copying it into the store: from the evaluator, command line interface or another store.
+  See [Deriving Path](./store/drv.md#deriving-path) for details.
 
-  - *output*: a pair of a [store path] to a [derivation] and an [output] name.
+  Not to be confused with [derivation path].
 
 - [deriver]{#gloss-deriver}
 

doc/manual/source/language/derivations.md

@@ -1,9 +1,10 @@
 # Derivations
 
-The most important built-in function is `derivation`, which is used to describe a single derivation:
-a specification for running an executable on precisely defined input files to repeatably produce output files at uniquely determined file system paths.
+The most important built-in function is `derivation`, which is used to describe a single store-layer [store derivation].
+Consult the [store chapter](@docroot@/store/drv.md) for what a store derivation is;
+this section just concerns how to create one from the Nix language.
 
-It takes as input an attribute set, the attributes of which specify the inputs to the process.
+This builtin function takes as input an attribute set, the attributes of which specify the inputs to the process.
 It outputs an attribute set, and produces a [store derivation] as a side effect of evaluation.
 
 [store derivation]: @docroot@/glossary.md#gloss-store-derivation
@@ -15,7 +16,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
 - [`name`]{#attr-name} ([String](@docroot@/language/types.md#type-string))
 
   A symbolic name for the derivation.
-  It is added to the [store path] of the corresponding [store derivation] as well as to its [output paths](@docroot@/glossary.md#gloss-output-path).
+  See [derivation outputs](@docroot@/store/drv.md#outputs) for what this is affects.
 
   [store path]: @docroot@/store/store-path.md
 
@@ -28,17 +29,12 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
   > }
   > ```
   >
-  > The store derivation's path will be `/nix/store/<hash>-hello.drv`.
+  > The derivation's path will be `/nix/store/<hash>-hello.drv`.
   > The [output](#attr-outputs) paths will be of the form `/nix/store/<hash>-hello[-<output>]`
 
 - [`system`]{#attr-system} ([String](@docroot@/language/types.md#type-string))
 
-  The system type on which the [`builder`](#attr-builder) executable is meant to be run.
-
-  A necessary condition for Nix to build derivations locally is that the `system` attribute matches the current [`system` configuration option].
-  It can automatically [build on other platforms](@docroot@/language/derivations.md#attr-builder) by forwarding build requests to other machines.
-
-  [`system` configuration option]: @docroot@/command-ref/conf-file.md#conf-system
+  See [system](@docroot@/store/drv.md#system).
 
   > **Example**
   >
@@ -68,7 +64,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
 
 - [`builder`]{#attr-builder} ([Path](@docroot@/language/types.md#type-path) | [String](@docroot@/language/types.md#type-string))
 
-  Path to an executable that will perform the build.
+  See [builder](@docroot@/store/drv.md#builder).
 
   > **Example**
   >
@@ -117,7 +113,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
 
   Default: `[ ]`
 
-  Command-line arguments to be passed to the [`builder`](#attr-builder) executable.
+  See [args](@docroot@/store/drv.md#args).
 
   > **Example**
   >
@@ -239,77 +235,3 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
       passed as an empty string.
 
 <!-- FIXME: add a section on output attributes -->
-
-## Builder execution
-
-The [`builder`](#attr-builder) is executed as follows:
-
-- A temporary directory is created under the directory specified by
-  `TMPDIR` (default `/tmp`) where the build will take place. The
-  current directory is changed to this directory.
-
-- The environment is cleared and set to the derivation attributes, as
-  specified above.
-
-- In addition, the following variables are set:
-
-  - `NIX_BUILD_TOP` contains the path of the temporary directory for
-    this build.
-
-  - Also, `TMPDIR`, `TEMPDIR`, `TMP`, `TEMP` are set to point to the
-    temporary directory. This is to prevent the builder from
-    accidentally writing temporary files anywhere else. Doing so
-    might cause interference by other processes.
-
-  - `PATH` is set to `/path-not-set` to prevent shells from
-    initialising it to their built-in default value.
-
-  - `HOME` is set to `/homeless-shelter` to prevent programs from
-    using `/etc/passwd` or the like to find the user's home
-    directory, which could cause impurity. Usually, when `HOME` is
-    set, it is used as the location of the home directory, even if
-    it points to a non-existent path.
-
-  - `NIX_STORE` is set to the path of the top-level Nix store
-    directory (typically, `/nix/store`).
-
-  - `NIX_ATTRS_JSON_FILE` & `NIX_ATTRS_SH_FILE` if `__structuredAttrs`
-    is set to `true` for the derivation. A detailed explanation of this
-    behavior can be found in the
-    [section about structured attrs](./advanced-attributes.md#adv-attr-structuredAttrs).
-
-  - For each output declared in `outputs`, the corresponding
-    environment variable is set to point to the intended path in the
-    Nix store for that output. Each output path is a concatenation
-    of the cryptographic hash of all build inputs, the `name`
-    attribute and the output name. (The output name is omitted if
-    it’s `out`.)
-
-- If an output path already exists, it is removed. Also, locks are
-  acquired to prevent multiple Nix instances from performing the same
-  build at the same time.
-
-- A log of the combined standard output and error is written to
-  `/nix/var/log/nix`.
-
-- The builder is executed with the arguments specified by the
-  attribute `args`. If it exits with exit code 0, it is considered to
-  have succeeded.
-
-- The temporary directory is removed (unless the `-K` option was
-  specified).
-
-- If the build was successful, Nix scans each output path for
-  references to input paths by looking for the hash parts of the input
-  paths. Since these are potential runtime dependencies, Nix registers
-  them as dependencies of the output paths.
-
-- After the build, Nix sets the last-modified timestamp on all files
-  in the build result to 1 (00:00:01 1/1/1970 UTC), sets the group to
-  the default group, and sets the mode of the file to 0444 or 0555
-  (i.e., read-only, with execute permission enabled if the file was
-  originally executable). Note that possible `setuid` and `setgid`
-  bits are cleared. Setuid and setgid programs are not currently
-  supported by Nix. This is because the Nix archives used in
-  deployment have no concept of ownership information, and because it
-  makes the build result dependent on the user performing the build.

doc/manual/source/language/import-from-derivation.md

@@ -71,8 +71,9 @@ Boxes are data structures, arrow labels are transformations.
 |       evaluate       |             |                        |
 |          |           |             |                        |
 |          V           |             |                        |
-|    .------------.    |             |  .------------------.  |
-|    | derivation |----|-instantiate-|->| store derivation |  |
+|    .------------.    |             |                        |
+|    | derivation |    |             |  .------------------.  |
+|    | expression |----|-instantiate-|->| store derivation |  |
 |    '------------'    |             |  '------------------'  |
 |                      |             |           |            |
 |                      |             |        realise         |

doc/manual/source/language/string-interpolation.md

@@ -22,9 +22,9 @@ Rather than writing
 "--with-freetype2-library=" + freetype + "/lib"
 ```
 
-(where `freetype` is a [derivation]), you can instead write
+(where `freetype` is a [derivation expression]), you can instead write
 
-[derivation]: @docroot@/glossary.md#gloss-derivation
+[derivation expression]: @docroot@/glossary.md#gloss-derivation-expression
 
 ```nix
 "--with-freetype2-library=${freetype}/lib"
@@ -148,7 +148,7 @@ An expression that is interpolated must evaluate to one of the following:
   - `__toString` must be a function that takes the attribute set itself and returns a string
   - `outPath` must be a string
 
-  This includes [derivations](./derivations.md) or [flake inputs](@docroot@/command-ref/new-cli/nix3-flake.md#flake-inputs) (experimental).
+  This includes [derivation expressions](./derivations.md) or [flake inputs](@docroot@/command-ref/new-cli/nix3-flake.md#flake-inputs) (experimental).
 
 A string interpolates to itself.