Overhaul flake ignore system

[thislooksfun]

Is your feature request related to a problem?

Currently the way flakes handle ignored files (via a git repo) is confusing at best and actively detrimental at worst. I'm a brand new user to nixos, but I've been around computers for many years, and I'm finding the flake defaults to be very annoying to work with, and I know I'm not the only one.

My main issue is that I see no intuitive reason why the files copied over by nixos-rebuild switch --flake would be affected by the existence git repo. It is, as another user pointed out, "spooky action at a distance", which makes it really hard to reason about what is actually happening and why. This behavior is documented at least, so it's findable, but it's still confusing.

Additionally it makes my personal use-case harder to achieve cleanly. I'd like to have one central source-of-truth config for all my system settings and installed packages, but still have a small local config file for some overrides (username, hostname, etc.). These overrides are extremely machine-specific and don't make sense (to me) to be committed in the repo. And I'm not the only one wanting to set things up this way. This user on the forum and the lovely individual who put up #6858 both are just as annoyed and confused by this as I am.

On the flip side, the git approach is also currently the only way to exclude files from being copied. So if someone has some secret file(s) they don't want included in the flake copy they currently have to make a git repo. This is also silly and confusing, and leads to issues like #9996.

Proposed solution

I propose using a clear and explicit hierarchy of .ignore files to control which files do and don't get included in the flake copy, and ditching the git integration entirely.

What I would like to see is the following hierarchy:

1. If there is a .nixignore in the same directory as the running flake, then its contents (and exclusively its contents) are used to control which files are copied over to the system copy.
2. If there is no .nixignore, then it should check for a standard .ignore file.
3. If neither .nixignore nor .ignore is found, it should check for a .gitignore
4. If none of these three files is found, then all files should be included. At no point should more than one of these files be used simultaneously. If multiple are found, only the first in the above list should be used.

Alternative solutions

An alternative solution is to go with the approach posed in #6858, but that still doesn't allow for having nix use files that are explicitly .gitignored. It also doesn't allow for ignoring files without using a git repo.

Additional context

N/A

Checklist

• checked latest Nix manual (source)
• checked open feature issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[thislooksfun]

For the record, I am willing to put effort into making this happen as long as such a PR wouldn't get rejected out of the gate, so I want to gather community and maintainer support and consensus before I waste everyone's time making a PR.

[roberth]

Forcing type = "file"; locally

and ditching the git integration entirely.

This ditching would have to be opt-in, at least imo.
I think a good way to achieve this is by implementing fetch configuration in flake.nix.
The way that's done could look a lot like a rework of #7862

  inputs.self.type = "file";

This declaration should have the effect that

• the local flake is fetched (or re-fetched) as a file: source, as if ignoring the presence of .git
• when such a flake is used as a remote type = "git" input in another flake's inputs, an error is thrown, based on the assumption that non-git files are required in the dependency
  • I think type = "file"; flakes should be considered "not for distributed use", but correct me if I'm wrong. I guess the type attr could be ignored if it's file. This design does not make it possible to then still specify git-related flags like submodules = true;

---

Ignore feature

.nixignore

This is sort of the dual of the git ditching part of your proposal: removing things.

We've been making progress to significantly improve laziness in the "fetching" and source processing parts of the evaluator for the benefit of basically your use case.
The goal is that as long as you don't have any unfiltered source inputs at the expression level (i.e. no raw src = ./.;) then Nix won't perform any unnecessary copying of the flake sources.

In the current evaluation cache design it does read all of it, for the cache key, but I'm planning to try and implement an alternate solution that does not require that this year.

I'm a little concerned about a .nixignore feature being implemented as part of Nix because

• we may not need it, as just explained
• it adds another language to the system, which is a lot of work for everyone involved
• it will have to be implemented correctly from the get go, because any changes in behavior in later versions will cause reproducibility issues, everything's hashed, so it's very sensitive to that

So I think many flakes won't need the .nixignore feature, and by the looks of it, it will be possible to implement as a library, which does not suffer from the issues I just mentioned.
Usage of such a library would feasibly look like:

  outputs = inputs: inputs.nixignore.lib.filterFlake inputs {
    packages = ...
      # path literal is filtered, because filterFlake re-imports flake.nix from the filtered flake source
      ./src
      ...;
  };

Furthermore I think we can make ergonomic improvements to flake.nix that benefit all flake frameworks.

---

Conclusion

I would propose to view your proposal as a two-parter

1. implement new flake attributes that combine with feat: support managing submodules via inputs.self #7862
2. when flake sources are lazy, implement filtering as a flake framework

Related

• Laziness
  • Lazy paths #11367
  • Lazy trees #6530
• Source control attributes
  • feat: support managing submodules via inputs.self #7862

[thislooksfun]

We've been making progress to significantly improve laziness in the "fetching" and source processing parts of the evaluator for the benefit of basically your use case. The goal is that as long as you don't have any unfiltered source inputs at the expression level (i.e. no raw src = ./.;) then Nix won't perform any unnecessary copying of the flake sources.

Ah! I didn't realize that was in the works. That would indeed completely negate the need for an ignore mechanism at all in my opinion. The only case I could think of would be to prevent you from somehow accidentally pulling a file you didn't mean to into the nix file, but I think that's a small enough edge case (with an easy enough prevention mechanism -- just don't include the file if you don't want it copied) that it wouldn't be worth the effort.

---

Now back to the git handling. In reading your reply I finally think I understand why the git handling was designed the way it was (and please correct me if my understanding is flawed); it's done that way to make sure that running a git-controlled flake will behave consistently whether it is run from disk or a git url. If that's the case, then I understand how you (collectively) made the design decisions that you did.

That being said, I still disagree with them. This kind of protection really only helps a small number of users--those who are not already familiar with version control software. For anyone who is even passingly familiar though we know very well that any files/changes that aren't committed won't be available in the remote. And especially for a project with as much technical overhead in the initial learning curve as nix has, that really feels like protecting a very small minority of people at the expense of the vast majority who might want to break out of the bubble occasionally.

Additionally, this also means that there is different behavior for anyone who downloads a flake via git vs downloading the same flake via a zip of the repo contents, which seems undesirable, if I understand the motivation correctly.

I think that the inputs.self.type = "file"; suggestion is a fine stop-gap solution, as it at least gives us an escape hatch when we need one, but I would still prefer to have the default behavior changed if possible. I think this could even likely be done as a non-breaking change after lazy fetching is released as unless someone is doing a full-directory include of previously-ignored files (which seems like a bad idea in the first place) then only the files they had included will be pulled in, which will end up having the same effect on the final store entry.

If we do go forwards with the inputs.self.type = "file"; solution though, whether as a temporary stop-gap or as a final solution, I would want to make running it from a git repo a warning, not a hard error. That way it will still work, but with a clear attribution of why something might break. Something like this (wording very much not final):

WARNING: This flake was defined as a "file"-type flake, but it is being used from a git url. Things might break.

I also think that the rationale behind why having a git repo changes the behavior would be a good thing to have in the docs. Having this behavior felt really counter-intuitive to me until I finally understood that it was trying to keep consistency with running flakes directly from git urls.

---

To conclude, I counter-propose the following:

1. Add the inputs.self.type = "file"; override
2. Document why having a git repo changes the behavior, and that using type = "file"; will disable that behavior.
3. [Optionally] make it a warning to load a type = "file" flake via git (but NOT an error)
4. Wait for lazy paths to be merged
5. Either:
   1. Remove the special-handling of git repos, and just let lazy paths handle it (my preferred solution)
   2. Make loading files from outside the git repo a warning, but still allow it

Unless someone is using src = ./.; this will be a non-breaking change (and imo anyone using src = ./.; and relying on git to prevent copying certain local files is playing with fire anyway, but that's my subjective opinion). Plus flakes are still technically experimental, so I think we can get away with this change with minimal problems.