compress-man-pages: Parallelise compression #380948
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| do | ||
| if gzip -c -n "$f" > "$f".gz; then | ||
| rm "$f" | ||
| else | ||
| rm "$f".gz | ||
| fi | ||
| done | ||
| done' -- |
There was a problem hiding this comment.
The intention is to terminate option processing for the shell. The path names are passed as arguments by xargs and without -- arguments with a leading - would be interpreted as options.
|
On my local machine, parallelisation reduces (real‐time) fixup duration for I haven’t looked at CPU and I/O wait times. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Parallelise compression of man pages using
xargs -P.This can significantly reduce fixup times for packages with a large number of man pages like
opensslorlinux-manualon multi‐core build machines.Since the compressing involves a bit of shell logic, I decided to process the pages in batches to balance the gains of parallelism against the cost of spawning a shell and parsing a brief script. The batch size is supplied through the
-nargument toxargs.This overhead could be avoided with a construction that spawns one process per build core and feeds the file paths in through pipes, but I feel that the programmatic complexity may not be worth the benefits.
With a batch size of 128 as proposed in this pull request, the hook should behave exactly the same as before for most packages, except for the extra
xargsand shell process.Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.