Preliminary steps to save the CI infrastructure #39009
Conversation
kwankyu
changed the title
|
Documentation preview for this PR (built with commit ff02cd0; changes) is ready! 🎉 |
kwankyu
force-pushed
the
p/add-comments-to-scripts
branch
from
ff6bdcd to
7da5efe
Compare
kwankyu
changed the title
kwankyu
force-pushed
the
p/add-comments-to-scripts
branch
from
db666ee to
6df7a6a
Compare
kwankyu
changed the title
kwankyu
changed the title
dimpase
added
the
disputed
|
before proceeding it's good to decide whether we rather go with the other PR |
I thought By the way, @dimpase, do you think it makes sense to support building packages which are more obscure than compilers or python, e.g. gap or fplll from source on operating systems that has them in system packages? (personal experience, though this is a long time ago I might have misremember the exact detail, I moved to meson already: there's a time where my operating system upgrades GAP to 2.14, but sage is still on GAP 2.13, so it cannot use system GAP and have to build from source — even though I have system GAP package installed.) |
Yes, |
This is not a good fix in my opinion. "maximal" means that optional sage packages are installed from the system package manager before configure + make; this is what the documentation recommends and should be tested in CI. Moreover, I find this PR very hard to review since it does many different things at the same time. I suggest you split it in multiple smaller PRs. |
There is no change in "maximal" job. In my opinion, the purpose of "optional" job is to test optional sage packages. No? So at least optional packages should be installed before test. On "maximal" docker image, they have no chance to be installed.
I made many small changes, as in PR description, for one goal, to make the CI infrastructure in good shape. I see no point in splitting it. |
so I convinced you that minimal configs must be removed, right? |
none of the minimal configurations installs gfortan. This is just sabotage, aimed at keeping Sage in its present hard to install, user unfriendly, monster shape. Then, prereqs are only for building Sage, not for running it. Some time ago it turned out that a certain utility, needed for a minimal Debian config, became a part of a Debian package not listed among the minimal config packages.
Only truly obscure packages should be a part of Sage, apart from sagelib itself. Sage should not be a replacement of a missing macOS package system, and Sage should not be held hostage of laziness of macOS folks, who can't be bothered to supply missing in Homebrew formulas for e.g. GAP (GAP does have an unofficial Homebrew formula, it's unfortunately missing libgap). |
No. I think that you are in a "religious war against the minimal configs", which is off topic for this PR. Further discussion with you on the off-topic just ruins the purpose of this PR setting up the crumbling CI infrastructure right. |
If you believe gfortran to be in The present PR has nothing to do with the exact list of the prerequisites, that is a minimal set of system packages required to build sage from source. This topic belongs to the sage build system, not the CI infrastructure. |
Just checking if I understood you correctly:
Maybe there's some misunderstanding here. Just because the build is called Let's define minimal to mean "minimal supported configuration" and standard to mean "maximal supported configuration (without optional packages)". As long as these two differs, you need two differently-named configurations, and I think |
no, why? We'll tell the user to run along with a host of other
There is no misunderstanding. Now @kwankyu wants to preserve all these spaghetti, even though mkoeppe is gone and people don't see much point in preserving all that.
Why on Earth do you need to even think about that "minimal" thing? Why not just tell the user to install "standard" in no uncertain terms? This is much easier for the user and for support. |
would you avoid phrases that may be perceived as aggression, thanks.
I don't have a conceptual objection to this, but what's the availability of various packages on various systems? This is all packages, sorted by decreasing availability: which fraction of these do you propose to support building from source, and which fraction do you propose to require the user to install by system package manager? (if you're proposing requiring some package to be system-package-manager-only, it would be time to discuss which.) I suppose if I'm on Arch, I'd still need to build it from source? |
Big no. Let me repeat. I don't care what is in "_prereq" as far as it contains reasonable minimal requirements that we can request users to install before starting to build sage from source. Go ahead and make your own PR to define your own version of minimal configuration and seek approval. In the other PR, just admit that distinction of "minimal", "standard" , and "maximal"configurations is a good idea.
Do you want "minimal" = "standard", and require users to install a big bunch of system packages? Then discuss on sage-devel about that idea. This PR does not prevent you from doing that. |
@user202729 |
not until we have a consensus on what this is. I'm not going to waste my time making PRs which get disputed |
|
To help you understand the effect of this change
here is a diagram of the jobs (configurations "minimal", "standard", etc.) showing how packages are installed: where "S" represents system package and dash "-" represents Sage package. All configurations are useful to detect what Sage package or system package has issues in the build system. |
The portion of standard packages I marked with |
In principle I agree that saves electricity. But in practice I can see a problem that may come up. How likely/frequent is this case in your opinion, and how do you propose to deal with them?
|
OK. You want "_prereq == standard package". If that is your vision, then go ahead with your own PR to achieve that. When that PR would get merged, we may remove "minimal" configuration from the CI. That vision is about the build system. Until the build system get changed, we should keep the current CI that support the current build system. The aim of this PR is to improve the current CI. By the way, the other PR is trying to destroy the current CI, with no change to the current build system. |
Did I correctly describe your position? Perhaps I misunderstood... If you just want to move some packages from "standard packages" to "_prereq", then that is just minor adjustments in the build system, such that the distinction of "minimal" and "standard" in CI is still useful. |
it's not a minor adjustment if you only move such packages as gcc and python in "_prereq", it's a drastic simplification of the build system, as there is a huge extra complexity involved into building them from source. |
typically users without root access are using some kind of HPC system, with a toolchain not capable to install Sage from source. Yes, then conda is the only choice.
All the bets are off as soon as you try to do anything with Sage version Well, I want building from source support for common packages removed, in particular such complex packages as compilers and Python. I don't want to work on it, I don't think it's time well-spent, it's better left to the experts. If a common package on your OS got broken, install a better version yourself, or move to a better OS. |
Please refrain from making such exaggerated and unfounded statements. |
OK. I respect that. Please note that the current CI infrastructure should be preserved for your "drastic simplification of the build system". |
|
On 21 February 2025 17:07:08 GMT-06:00, Kwankyu Lee ***@***.***> wrote:
kwankyu left a comment (sagemath/sage#39009)
> it's not a minor adjustment if you only move such packages as gcc and python in "_prereq", it's a drastic simplification of the build system, as there is a huge extra complexity involved into building them from source.
OK. I respect that. Please note that the current CI infrastructure should be preserved for your "drastic simplification of the build system".
after we remove gcc and python spkgs, CI will get simpler and will run much faster (these runs which build them)
|
Such changes in the build system are independent from the CI infrastructure itself. Only running speed of the Ci workflows would get affected. |
Sorry. I meant your removing "minimal" and "standard" and other failing configurations at the point of Dima's positive review. That looked "destructive" to me. Afterwards you restored "standard", but still removed "minimal" and other failing configurations. If you accept my proposal that you discard those changes in CI, which I cannot accept, from your PR, then both PRs may go hand in hand. |
kwankyu
force-pushed
the
p/add-comments-to-scripts
branch
from
caf7a85 to
ff02cd0
Compare
To improve the situation with the CI infrastructure, this PR:
added comments untangling obscure code in CI-related files, for those poor guys who ever attempt to read the files for whatever reasons.
while doing the cosmetic changes, a bug (about
-uninstalltargets) was foundbuild/make/Makefile.in, which is fixed here.to test, do
fixed some jobs in the CI-linux workflow that fail because of duplicate artifact names.
removed ubuntu-lunar, ubuntu-mantic, conda-forge-python3.11, ubuntu-bionic-gcc_8-i386, debian-bullseye-i386 from the list of the default systems that CI runs for. This is how to properly modify the list:
tox.ini(find DEFAULT_SYSTEM_FACTORS)tox -e update_docker_platformsremoved old versions of linuxmint and added new versions.
"optional" and "experimental" jobs now run upon "standard" docker images, instead of "maximal" ones, to avoid "out of runner space" error.
renamed "Reusable workflow for Docker-based portability CI" to "Workflow for Linux portability CI" for short name and made it runnable through github interface to facilitate testing specific platform.
test: https://github.com/kwankyu/sage/actions/workflows/docker.yml
added helpful comments and updated the developer doc
reimplemented
.ci/write-dockerfile.shso that simplified Dockerfile is generated for present and future stabilityturned off failing jobs in "CI Linux incremental"
removed seemingly useless
subprojects/factorydirectory to eliminate certain git warnings.test CI run: https://github.com/kwankyu/sage/actions/runs/13431349016
compare with the status quo: https://github.com/sagemath/sage/actions/runs/13252848846
test CI run after the above change: https://github.com/kwankyu/sage/actions/runs/13467567390
test CI with a PR: kwankyu#81
The main objective of this PR is to solve issues with the workflow "CI Linux" such that a failure on a platform reveals solely some problem of sage built on the platform, but not a problem of the CI infrastructure. After this PR, hopefully, each of failing platforms should be tackled individually. If a platform fails, perhaps we should
Only decent platforms according to the CI results should be listed in https://github.com/sagemath/sage/wiki/Sage-10.6-Release-Tour#availability-and-installation-help.
📝 Checklist
⌛ Dependencies