Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New store settings system #11139

Draft
wants to merge 100 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
f90e771
feat: Refactor unix domain socket store config
fzakaria Jul 15, 2024
1f72267
uds-remote-store is now clang-formatted
fzakaria Jul 15, 2024
660d868
Removed clang-format changes
fzakaria Jul 15, 2024
6585c6e
Reference public variable
fzakaria Jul 15, 2024
10c5c65
Update src/libstore/uds-remote-store.hh
fzakaria Jul 15, 2024
030d91d
Move constant to within Config object
fzakaria Jul 15, 2024
7169905
Lowercase constexpr
fzakaria Jul 15, 2024
4af5092
Update src/libstore/store-api.cc
fzakaria Jul 15, 2024
dd74b72
Revert back returning daemon for URI if path empty
fzakaria Jul 15, 2024
01fbb4b
Added unit test
fzakaria Jul 15, 2024
6eb3303
Added to meson
fzakaria Jul 15, 2024
d3275e8
Changed to protected and added another test
fzakaria Jul 16, 2024
0087d7d
Clang formatted test file
fzakaria Jul 16, 2024
1e97346
Merge remote-tracking branch 'upstream/master' into issue-10766-unix-…
Ericson2314 Jul 16, 2024
e733f6d
Move `UdsRemoteStore{,Config}` defs out of headers, and other cleanups
Ericson2314 Jul 16, 2024
e435f3e
Add two more headers to try to provide missing template instantiations
Ericson2314 Jul 16, 2024
ed2ccdc
Add globals.hh to uds-remote-store.cc
fzakaria Jul 16, 2024
e9c0636
Fix linking issue, convert many more
Ericson2314 Jul 16, 2024
8162dbd
Merge remote-tracking branch 'upstream/master' into issue-10766-unix-…
Ericson2314 Jul 16, 2024
f23d168
`make format`
Ericson2314 Jul 16, 2024
c4c148f
Declare some template specializations in the header
Ericson2314 Jul 16, 2024
4e79b4e
Move `uriSchemes` to `*StoreConfig`
Ericson2314 Jul 16, 2024
1e25c22
Revert "Declare some template specializations in the header"
Ericson2314 Jul 16, 2024
ea3a420
Merge branch 'move-schemes-set' into new-store-settings
Ericson2314 Jul 16, 2024
414d6ad
`make format`
Ericson2314 Jul 17, 2024
fcf3e32
Merge branch 'move-schemes-set' into HEAD
Ericson2314 Jul 17, 2024
3b0edea
WIP Start getting store settings off config.hh
Ericson2314 Jul 17, 2024
d095fa3
WIP
Ericson2314 Jul 17, 2024
121e607
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 17, 2024
7b135d5
Comment out tests
fzakaria Jul 17, 2024
596a682
Dummy store compiles
Ericson2314 Jul 17, 2024
95b36f4
Merge branch 'issue-10766-unix-socket' of github.com:fzakaria/nix int…
Ericson2314 Jul 17, 2024
0e65638
Merge remote-tracking branch 'upstream/master' into issue-10766-unix-…
Ericson2314 Jul 17, 2024
34f876f
Format, CI hopefully happy now
Ericson2314 Jul 17, 2024
a7d9394
Merge branch 'issue-10766-unix-socket' into new-store-settings
Ericson2314 Jul 17, 2024
4909f9a
Get `LocalFSStore` building
Ericson2314 Jul 18, 2024
9052c09
Merge commit '17051ca80ab9bcee896e040cfe0ea699702ac0ff' into new-stor…
Ericson2314 Jul 18, 2024
118b050
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 18, 2024
307440b
Convert `LocalStoreConfig` to new system
Ericson2314 Jul 18, 2024
de1ec5a
A bunch of small fixes
Ericson2314 Jul 18, 2024
9b30c20
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 18, 2024
bb7bf73
More progress converting
Ericson2314 Jul 18, 2024
ca1e895
WIP
Ericson2314 Jul 18, 2024
901b34f
WIP
Ericson2314 Jul 18, 2024
6e5d7fc
Convert ssh:// to new system
Ericson2314 Jul 18, 2024
74620b9
WIP
Ericson2314 Jul 18, 2024
469ac00
Convert `UDSRemoteStore`
Ericson2314 Jul 18, 2024
2fcb196
Fix `StoreReference`
Ericson2314 Jul 18, 2024
103c90b
More progress
Ericson2314 Jul 18, 2024
3a149b5
Getting close!
Ericson2314 Jul 18, 2024
68ae4dc
It builds, debugging
Ericson2314 Jul 18, 2024
387767b
Fix some bugs
Ericson2314 Jul 18, 2024
b53a4a0
More tests
Ericson2314 Jul 19, 2024
34f9b13
Inline defaults
Ericson2314 Jul 19, 2024
ae93776
Avoid potentially problematic move
Ericson2314 Jul 19, 2024
40955fa
WIP hack so it builds
Ericson2314 Jul 19, 2024
736dd76
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 19, 2024
1b027f7
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 19, 2024
1c5fac5
Adjust registration infra, fix Meson build
Ericson2314 Jul 19, 2024
a6c6852
WIP Store doesn't inherit from config
Ericson2314 Jul 20, 2024
d8856f4
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 20, 2024
48aa3bb
WIP
Ericson2314 Jul 20, 2024
1344bcd
WIP
Ericson2314 Jul 21, 2024
5a1b109
WIP
Ericson2314 Jul 21, 2024
0c8330c
WIP
Ericson2314 Jul 21, 2024
f5df5e7
WIP
Ericson2314 Jul 21, 2024
6ab4817
WIP
Ericson2314 Jul 21, 2024
cc1ce6a
WIP
Ericson2314 Jul 21, 2024
53f692b
WIP
Ericson2314 Jul 21, 2024
e621208
WIP
Ericson2314 Jul 21, 2024
05861c4
WIP
Ericson2314 Jul 21, 2024
b0c6f18
Build `S3BinaryCacheStore` with Meson
Ericson2314 Jul 21, 2024
40ebe0f
S3 binary cache store
Ericson2314 Jul 21, 2024
dd6c7f0
Merge branch 'master' into new-store-settings
Ericson2314 Jul 22, 2024
c9b772f
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Aug 27, 2024
3829f14
Merge branch 'master' into new-store-settings
Ericson2314 Aug 28, 2024
c5fd7d9
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Oct 10, 2024
b601117
Merge branch 'master' into new-store-settings
Ericson2314 Feb 2, 2025
0de6c96
Merge branch 'master' into new-store-settings
Ericson2314 Feb 2, 2025
3229211
It links now
Ericson2314 Feb 2, 2025
c5b11c5
Fix an issue with `BinaryCacheStore::init`
Ericson2314 Feb 2, 2025
1146db3
Fix store reference test parsed
Ericson2314 Feb 2, 2025
cb60d29
Fix dummy store URL in test
Ericson2314 Feb 2, 2025
643620d
`openUncachedStore`: Use number not string for JSON now
Ericson2314 Feb 2, 2025
2312356
Fix some manual mangling of store reference params
Ericson2314 Feb 2, 2025
1b370d9
Fix appending features
Ericson2314 Feb 2, 2025
b8cdb21
Fix mounted ssh store test
Ericson2314 Feb 2, 2025
cf5d50b
Fix opening the local overlay store
Ericson2314 Feb 2, 2025
b19aa37
Get the store docs to render again
Ericson2314 Feb 2, 2025
4dbb0db
Make all config description `constexpr`
Ericson2314 Feb 2, 2025
287d0ef
Fix `documentDefault`, it was inverted
Ericson2314 Feb 2, 2025
880c1e2
Make `JSON_IMPL(StoreReference)`, use for `JSON_IMPL(ref<StoreConfig>)`
Ericson2314 Feb 2, 2025
aae2c8e
Test store reference to/from JSON roundtrip
Ericson2314 Feb 2, 2025
34af9ca
Fix settings to JSON, write from JSON for completeness
Ericson2314 Feb 2, 2025
5f79489
Fix docs, add test
Ericson2314 Feb 2, 2025
8626dfe
Add test showing bug
Ericson2314 Feb 2, 2025
1d51e95
Render nested setting descriptions
Ericson2314 Feb 3, 2025
c431542
Require store settings experimental feature
Ericson2314 Feb 3, 2025
01d1acc
Update tests given XP feature checks
Ericson2314 Feb 3, 2025
ede1c2b
Back compat for boolean fields
Ericson2314 Feb 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 38 additions & 8 deletions doc/manual/generate-settings.nix
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,32 @@ in
prefix,
inlineHTML ? true,
}:
settingsInfo:

let

showSetting =
prefix: setting:
{
description,
documentDefault,
defaultValue,
aliases,
value,

experimentalFeature,

# Whether we document the default, because it is machine agostic,
# or don't because because it is machine-specific.
documentDefault ? true,

# The default value is JSON for new-style config, rather than then
# a string or boolean, for old-style config.
json ? false,

defaultValue ? null,

subSettings ? null,

aliases ? [ ],

# The current value for this setting. Purposefully unused.
value ? null,
}:
let
result = squash ''
Expand All @@ -50,7 +63,7 @@ let

${description}

**Default:** ${showDefault documentDefault defaultValue}
${showDefaultOrSubSettings}

${showAliases aliases}
'';
Expand All @@ -72,9 +85,24 @@ let
> ```
'';

showDefaultOrSubSettings =
if !isAttrs subSettings then
# No subsettings, instead single setting. Show the default value.
''
**Default:** ${showDefault}
''
else
# Indent the nested sub-settings, and append the outer setting name onto the prefix
indent " " ''
**Nullable sub-settings**: ${if subSettings.nullable then "true" else "false"}
${builtins.trace prefix (showSettings "${prefix}-${setting}" subSettings.map)}
'';

showDefault =
documentDefault: defaultValue:
if documentDefault then
if json then
"`${builtins.toJSON defaultValue}`"
else
# a StringMap value type is specified as a string, but
# this shows the value type. The empty stringmap is `null` in
# JSON, but that converts to `{ }` here.
Expand All @@ -95,5 +123,7 @@ let
in
result;

showSettings =
prefix: settingsInfo: concatStrings (attrValues (mapAttrs (showSetting prefix) settingsInfo));
in
concatStrings (attrValues (mapAttrs (showSetting prefix) settingsInfo))
showSettings prefix
1 change: 1 addition & 0 deletions doc/manual/generate-store-info.nix
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ let
{
settings,
doc,
uri-schemes,
experimentalFeature,
}:
let
Expand Down
6 changes: 3 additions & 3 deletions src/build-remote/build-remote.cc
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
#include "globals.hh"
#include "serialise.hh"
#include "build-result.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "strings.hh"
#include "derivations.hh"
#include "local-store.hh"
Expand Down Expand Up @@ -44,7 +44,7 @@ static AutoCloseFD openSlotLock(const Machine & m, uint64_t slot)

static bool allSupportedLocally(Store & store, const std::set<std::string>& requiredFeatures) {
for (auto & feature : requiredFeatures)
if (!store.systemFeatures.get().count(feature)) return false;
if (!store.config.systemFeatures.get().count(feature)) return false;
return true;
}

Expand Down Expand Up @@ -85,7 +85,7 @@ static int main_build_remote(int argc, char * * argv)
that gets cleared on reboot, but it wouldn't work on macOS. */
auto currentLoadName = "/current-load";
if (auto localStore = store.dynamic_pointer_cast<LocalFSStore>())
currentLoad = std::string { localStore->stateDir } + currentLoadName;
currentLoad = std::string { localStore->config.stateDir } + currentLoadName;
else
currentLoad = settings.nixStateDir + currentLoadName;

Expand Down
2 changes: 1 addition & 1 deletion src/libcmd/command.cc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

#include "command.hh"
#include "markdown.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "local-fs-store.hh"
#include "derivations.hh"
#include "nixexpr.hh"
Expand Down
2 changes: 1 addition & 1 deletion src/libcmd/command.hh
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ extern char ** savedArgv;
class EvalState;
struct Pos;
class Store;
class LocalFSStore;
struct LocalFSStore;

static constexpr Command::Category catHelp = -1;
static constexpr Command::Category catSecondary = 100;
Expand Down
2 changes: 1 addition & 1 deletion src/libcmd/common-eval-args.cc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
#include "registry.hh"
#include "flake/flakeref.hh"
#include "flake/settings.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "command.hh"
#include "tarball.hh"
#include "fetch-to-store.hh"
Expand Down
2 changes: 1 addition & 1 deletion src/libcmd/repl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
#include "eval-settings.hh"
#include "attr-path.hh"
#include "signals.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "log-store.hh"
#include "common-eval-args.hh"
#include "get-drvs.hh"
Expand Down
2 changes: 1 addition & 1 deletion src/libexpr/primops/fetchClosure.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#include "primops.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "realisation.hh"
#include "make-content-addressed.hh"
#include "url.hh"
Expand Down
3 changes: 2 additions & 1 deletion src/libstore-c/nix_api_store.cc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

#include "path.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "build-result.hh"

#include "globals.hh"
Expand Down Expand Up @@ -42,7 +43,7 @@ Store * nix_store_open(nix_c_context * context, const char * uri, const char ***
if (!params)
return new Store{nix::openStore(uri_str)};

nix::Store::Params params_map;
nix::StoreReference::Params params_map;
for (size_t i = 0; params[i] != nullptr; i++) {
params_map[params[i][0]] = params[i][1];
}
Expand Down
1 change: 1 addition & 0 deletions src/libstore-test-support/tests/libstore.hh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
#include <gmock/gmock.h>

#include "store-api.hh"
#include "store-open.hh"

namespace nix {

Expand Down
3 changes: 3 additions & 0 deletions src/libstore-tests/data/store-reference/auto.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"scheme": "auto"
}
4 changes: 4 additions & 0 deletions src/libstore-tests/data/store-reference/auto_param.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"root": "/foo/bar/baz",
"scheme": "auto"
}
5 changes: 5 additions & 0 deletions src/libstore-tests/data/store-reference/local_1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"authority": "",
"root": "/foo/bar/baz",
"scheme": "local"
}
5 changes: 5 additions & 0 deletions src/libstore-tests/data/store-reference/local_2.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"authority": "/foo/bar/baz",
"scheme": "local",
"trusted": true
}
4 changes: 4 additions & 0 deletions src/libstore-tests/data/store-reference/ssh.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"authority": "localhost",
"scheme": "ssh"
}
6 changes: 6 additions & 0 deletions src/libstore-tests/data/store-reference/unix.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"authority": "",
"max-connections": 7,
"scheme": "unix",
"trusted": true
}
20 changes: 20 additions & 0 deletions src/libstore-tests/dummy-store.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#include <gtest/gtest.h>

#include "dummy-store.hh"
#include "globals.hh"

namespace nix {

TEST(DummyStore, constructConfig)
{
DummyStoreConfig config{"dummy", "", {}};

EXPECT_EQ(config.storeDir, settings.nixStore);
}

TEST(DummyStore, constructConfigNoAuthority)
{
EXPECT_THROW(DummyStoreConfig("dummy", "not-allowed", {}), UsageError);
}

} // namespace nix
8 changes: 5 additions & 3 deletions src/libstore-tests/legacy-ssh-store.cc
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,13 @@ TEST(LegacySSHStore, constructConfig)
LegacySSHStoreConfig config{
"ssh",
"localhost",
StoreConfig::Params{
StoreReference::Params{
{
"remote-program",
// TODO #11106, no more split on space
"foo bar",
{
"foo",
"bar",
},
},
}};
EXPECT_EQ(
Expand Down
8 changes: 2 additions & 6 deletions src/libstore-tests/local-overlay-store.cc
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
// FIXME: Odd failures for templates that are causing the PR to break
// for now with discussion with @Ericson2314 to comment out.
#if 0
# include <gtest/gtest.h>
#include <gtest/gtest.h>

# include "local-overlay-store.hh"
#include "local-overlay-store.hh"

namespace nix {

Expand Down Expand Up @@ -31,4 +28,3 @@ TEST(LocalOverlayStore, constructConfig_rootPath)
}

} // namespace nix
#endif
14 changes: 2 additions & 12 deletions src/libstore-tests/local-store.cc
Original file line number Diff line number Diff line change
@@ -1,15 +1,6 @@
// FIXME: Odd failures for templates that are causing the PR to break
// for now with discussion with @Ericson2314 to comment out.
#if 0
# include <gtest/gtest.h>
#include <gtest/gtest.h>

# include "local-store.hh"

// Needed for template specialisations. This is not good! When we
// overhaul how store configs work, this should be fixed.
# include "args.hh"
# include "config-impl.hh"
# include "abstract-setting-to-json.hh"
#include "local-store.hh"

namespace nix {

Expand Down Expand Up @@ -37,4 +28,3 @@ TEST(LocalStore, constructConfig_rootPath)
}

} // namespace nix
#endif
1 change: 1 addition & 0 deletions src/libstore-tests/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ sources = files(
'derivation-advanced-attrs.cc',
'derivation.cc',
'derived-path.cc',
'dummy-store.cc',
'downstream-placeholder.cc',
'http-binary-cache-store.cc',
'legacy-ssh-store.cc',
Expand Down
2 changes: 1 addition & 1 deletion src/libstore-tests/nix_api_store.cc
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ TEST_F(nix_api_util_context, nix_store_open_dummy)
nix_libstore_init(ctx);
Store * store = nix_store_open(ctx, "dummy://", nullptr);
ASSERT_EQ(NIX_OK, ctx->last_err_code);
ASSERT_STREQ("dummy", store->ptr->getUri().c_str());
ASSERT_STREQ("dummy://", store->ptr->getUri().c_str());

std::string str;
nix_store_get_version(ctx, store, OBSERVE_STRING(str));
Expand Down
Loading
Loading