Re: Use gen-libm-test.py to generate ulps table for manual

[Michael Matz <matz at suse dot de>]

Hi,

[answering to a bounced mail, I'm not on libc-alpha@, so keep me CCed]

I'm randomly using one mail of this thread as it covers most topics, but I 
use it for making general points, not only specifically answering to your 
mail Carlos.

On Tue, 14 Aug 2018, Carlos O'Donell wrote:

> > This is a capability that deteriorates very quickly if you do not 
> > continuously exercise it.
> 
> Agreed. How does adding python prevent you from continuously exercising 
> the bootstrap? Do you argue that it makes the whole build cycle too 
> long?

It's technically not (very) difficult to include python into the bootstrap 
set.  But that's not the most important part.  All SUSE distros since 
forever use this scheme of building the distro, a bootstrap set (iterating 
builds until stability), and once the bootstrap set is finished building 
all dependend packages in a topological order (breaking cycles in that 
other non-bootstrap set, if they exist, at defined points).

That implies the following thing: every change in any package of the 
bootstrap set implies a whole-distro rebuild (any change in any of the 
other packages implies only a rebuild on the depending packages).  So 
enlarging the bootstrap set makes full distro rebuild more common, and 
given that this takes about two to four days (depending on architecture) 
it's something you don't want to do lightly.  So changes to packages in 
the bootstrap set are done more seldom than for other packages.

> > We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
> > know that there are uses for that, such as generating accurate markup
> > (be it annobin or CET) based on the latest toolchain fixes.
> 
> Either we can't do it technically (not true, we can).

I bet you theoretically can, in reality you don't.  Such bootstrap 
capability detoriates within weeks if not done regularly (we for instance 
see this with old enterprise products where this is not regularly done; 
once you try this you go "oh my god, what is this, what happened here?").

> I don't know why Andreas wants the core bootstrap small.

It's not Andreas, it's all (open)SUSE distros.  It's the way we do this 
since ages and it's not small work (occasionally) to keep the bootstrap 
set small.  Which is why we try very hard to avoid extending it, including 
having these very discussions.

> I don't understand Andreas' motivation.

I can bring up the original philosophy of how this came to be in the SUSE 
distros: whenever there's a change in any package, all other packages 
depending on it (per buildrequires, and transitively so) should be rebuilt 
(I won't got into the reasoning for this policy here).  That requires a 
topological sort, and as such can't handle cycles.  So we allowed exactly 
one cycle of packages, the bootstrap cycle.  That one builds until a fixed 
point (or well, in reality all packages a max of three times).  But all 
other packages are build in topological order.  Obviously it's a good idea 
to let this cycle be small, so that at least the rest of packages can be 
built according to the golden rule.  So, that's where we're coming from.

As you can see in Andreas' list the bootstrap cycle is small (and with 
minor additions over time contains still the same things since 20 years; 
though I think we could remove some more things from it with a little 
work).

Over time we of course got more cycles in BuildRequires than this (and the 
reason is often programmers not taking these things into account, e.g. 
adding a dependency on java from within gettext; just an example, not 
necessarily what happened), but still only the bootstrap cycle is iterated 
and implies a full distro rebuild.

> Using python in upstream glibc makes it easier to maintain, and the only
> downside is the bootstrap issue.

(sidenote: if I were a glibc maintainer I'd disagree; I know perl better 
than python.  Alas, I'm not :)  Though one point I'll make is that 
everybody sufficiently intelligent can learn any reasonable language; and 
hence the like or dislike for specific languages should never in itself be 
a reason to choose one or the other.  Rather all other reasons to prefer 
one over the other should trump such personal preferences.  Such 
considerations include: preexisting use of language $X, wideness of 
availability for language $X interpreters/compilers and so on)

Another point I made over at gcc@ is that for base tools like gcc and 
glibc the list of dependencies should be as minimal as possible, even at 
the expense of a little more work.  In my world the former is of much 
higher importance than a little convenience for authors of helper scripts.
To my liking even depending on perl should be avoided (surely a little 
text processing can just as well be done with awk).

> The solution *today* is to not build the subsystems in glibc that need 
> python (manual, benchtests) during bootstrap.

To note, for all of the above I'm not talking about use of anything for 
building optional stuff.  We do the above, and if python is to be used for 
building the manual we'll split that off as well from the main glibc 
build.  All my points above only apply to dependencies for 
building essential part, which for glibc would be, ... well, libc ;-)

> If more things require python then we may need a minimal python for the 
> bootstrap. Investing in that is a good idea only if you value what we 
> have just discussed as a bootstrap.

We split python already into a base and a full python (which requires 
tk and hence all of X).  It might be that we "only" need to add openssl to 
the bootstrap cycle in addition to python3-base.  It would be good if we 
can avoid all of this though.

> Other more useful bootstraps are IMO something like "Build all C++ 
> programs with the latest C++ compiler changes, but do it in a side tag" 
> which would allow the Fedora toolchain team to test C++ changes. These 
> kinds of CI-esque workflows don't need a minimal python or a minimal 
> bootstrap since the C++ applications you are building will likely 
> already require much of the distro to be present.

Actually at SUSE we do something similar to this as well.  All compiler 
changes are going into a staging project where many packages are rebuilt.  
A new compiler version (and new glibc and similar things) goes through 
larger stagings.  That is indeed nice, but orthogonal to the bootstrap 
topic.

> Agreed. This still doesn't convince me that you have a compelling enough 
> use case for minimal bootstraps that upstream glibc must not adopt 
> python as a bootstrap requirement (which it isn't yet).

What I'm always wondering in these discussion about switching languages 
for helper scripts (of all!) are compelling reasons to switch _to_ new 
infrastructures.  Shouldn't the onus of finding such reasons be on those 
suggesting a change?  (I realize that one stated reason is "easier 
maintenance", I merely disagree that even if this is true (which is at 
least questionable) that it'd be a very compelling reason).

> To convince myself that the real solution is a python3-minimal solution 
> I started with the python3 in Fedora Rawhide, and with maybe 5 minutes 
> of spec file hacking I had a Python3 which built (in a clean rawhide 
> mock chroot) *without* all the dependencies enabled.

As without any dependencies no C compiler would be installed I assume the 
mock chroot contains a multitude of things by default.  So is your 
experiment for creating a -minimal variant valid?

> Is there a technical problem with needing python3-minimal? It seems like 
> such a package would solve all sorts of early bootstrap issues with 
> needing python?

As we currently don't have any such problems including it wouldn't solve 
anything.  If you chose to create such problem, then yes, including a 
python3-base will be the technical solution to it.


Ciao,
Michael.