Try to rewrite emacs packages updater.

[Vonfry]

I tried to rewrite the emacs packages updater in pure emacs lisp without third party dependencies.

But I have less time to fix it recently.
Therefore, I public my code here and hope someone could make it as a pr.

source code

Goals

1. Get rid of the inactive emacs2nix project.
2. Only use emacs builtin functions to reduce maintance costs.

[jian-lin]

Improving Nixpkgs Emacs infra is always on my TODO list, such as correcting the generated meta.homepage, replacing sha256 to hash and not adding emacs to packageRequires. So I am glad to see people working towards similar goals!

cc @AndersonTorres

---

Reducing maintenance cost seems to be your main motivation.

You approach it by reducing dependencies. However, maintainability is more then just reducing dependencies and reducing dependencies may not improve maintainability. For example, promise is better than callback (used by url-http) and sleep-for is a code smell to me. Abstraction is added for a reason. That being said, unifying the update code for ELPA and MELPA in your work is good, I think.

About the inactivity of emacs2nix, maybe it is because it has always been working well. So there is nothing need to be changed there. I have not touched its codebase yet, so I am not sure about its extensibility.

Last but not least, the people who actually do the maintenance are also important. Currently, I am one of these people and I personally think Haskell (used by emacs2nix) is better than elisp for this not-very-trivial task. And according to your words, you may not do maintenance.

[adisbladis]

I have no comment on the code as-is. I'm commenting on the overall state/implementation of Emacs updaters.

I've also been interested to rewrite our generators, but in type-annotated Python.
Very few contributors are actually competent elisp developers (myself included), and these updaters are somewhat nontrivial.
My main concern with a new implementation, as @jian-lin also alludes to, comes down to who is able to maintain it.

[AndersonTorres]

While I was writing this answer (and losing it again due to sleep), I discovered that I was not the first one to notice that annoying bug on emacs2nix that duplicates shell-command-plus.

nix-community/emacs2nix#60

About the maintenance issue, I was thinking in Common Lisp initially, but the CL ecosystem in Nixpkgs is too confusing!

Further, Haskell would require a crash course of many potential maintainers here, way more than elisp ones. The single fact Haskell is not bootstrappable makes things worse.

Python looks more promising.

[jian-lin]

MELPA, like GNU ELPA, also follows "the package.el protocol", which means MELPA provides a archive-contents file and "built" package tarballs. As a result, packages in MELPA can be generated in the same way as GNU ELPA. I just tried elpa2nix https://melpa.org/packages/ --names ./names.nix -o /tmp/melpa-as-elpa-generated.nix and it works! (Well, the generated homepage link is wrong, but that's not a big issue.)

However, currently, we do not treat MELPA the same as GNU ELPA when generating packages. This is confusing to me. Maybe I missed something. As the author of package generation for GNU ELPA @ttuegel and MELPA @bendlas, could you provide some context?

[AndersonTorres]

Like the old music, "it was always burning". Even that strange problem of not being able to use arbitrary strings in version is too mysterious.
If I would guess, it is because usually recipes are more precise and they are suitable for building from pristine sources.

More to the point, my first question would be: how exactly the custom manual-packages can be built by elpaBuild?

[jian-lin]

how exactly the custom manual-packages can be built by elpaBuild?

To be clear, I am talking about using the same process to generate MELPA and GNU ELPA packages. I am not talking about replacing melpaBuild with elpaBuild. melpaBuild is the perfect choice for manualPackages.

[AndersonTorres]

I asked most of curiousity, because melpaBuild is by far the most dissonant from the elisp repos we track. Unifying them would be an interesting end goal.
Further, it always bugged me that necessity of a recipe file for MELPA but not for the others.

[jian-lin]

For multi-file manualPackages, a build phase is needed to make a valid multi-file package tarball from their source repo. melpaBuild has a build phase and elpaBuild does not. This is the key difference between them. What do you mean by "unifying them"?

How to actually do the build phase is another question. Currently, we use MELPA's package-build package for that, so a recipe is needed. If you roll your own build phase, you may not need a recipe but you still need a way to pass the related info, such as which files should be included in the package. I would say a thing like recipe is necessary for a build phase, but it can be more nix-friendly.

---

BTW, this issue is about generating emacs packages. Discussions about elpaBuild and melpaBuild seem a bit offtopic. Maybe they should be discussed in their own issue.

[bendlas]

As the author of package generation for GNU ELPA @ttuegel and MELPA @bendlas, could you provide some context?

Whew, that's been a minute ... so when I rewrote the melpa updater, it was mainly because it was very unreliable and slow. I did it in elisp, because a) I don't know enough haskell to have fun debugging it and b) I wanted to maximize the chance of its users being able to hack on it in a language they're fluent in (seems to have been working so far).

I did originially want to tackle elpa and org packages as well, but those sets are tiny compared to melpa - hence did't have the same problems, and since my PR solved the immediate problem, it got merged for just melpa.

Not a fan of python any more, FWIW

[jian-lin]

Thanks for the context.

According to #59189, it seems not using "the package.el protocol" to generate MELAP packages dates back to melpa2nix of emacs2nix. I failed to find the reason in a quick search though.

To be clear, using "the package.el protocol" to generate MELAP packages basically means using elpa2nix of emacs2nix for the generation instead of dealing with recipes and building valid package tarballs.

[jian-lin]

Found nix-community/emacs2nix#5 (comment)

I guess dealing with recipes and building valid package tarballs for MELPA packages is better for reproducible builds assuming MELPA also not keeps old tarballs.

However, assuming cache.nixos.org keeps all FODs, then it is not a reproducibility issue for packages in Nixpkgs. Only third party projects re-using package generation infra of Nixpkgs, such as emacs-overlay, have this reproducibility issue. To reproduce, apply an old emacs-overlay and build a GNU-devel ELPA package. If you are "lucky" enough, you will get a 404 for the tarball.

It is still a question whether it is worth adding complexity of package generation infra to get better reproducibility (only for third party projects).