Re: [XEN PATCH] tools: Have flex and bison mandatory

[Elliott Mitchell <ehem+xen@xxxxxxx>]

On Thu, Feb 09, 2023 at 09:23:43AM +0100, Jan Beulich wrote:
> On 08.02.2023 19:51, Elliott Mitchell wrote:
> > On Wed, Feb 08, 2023 at 09:02:35AM +0100, Jan Beulich wrote:
> >> On 07.02.2023 19:02, Anthony PERARD wrote:
> >>>
> >>> I think the issue that Andrew saw in the CI with "libxlu_cfg_y.o"
> >>> failing to build while "libxlu_cfg_l.[ch]" are been regenerated is
> >>> probably because make doesn't consider "libxlu_cfg_l.[ch]" as a group of
> >>> targets.
> >>>
> >>> If for some reason we have:
> >>>     libxlu_cfg_l.h newer than libxlu_cfg_l.l newer than libxlu_cfg_l.c
> >>>
> >>> Then make seems to take two parallel decision:
> >>>     When building libxlu_cfg_y.o:
> >>>         libxlu_cfg_l.h is newer than libxlu_cfg_l.l
> >>>         -> libxlu_cfg_l.h is up-to-date, start building libxlu_cfg_y.o
> >>>     When building libxlu_cfg_l.o:
> >>>         libxlu_cfg_l.c is older than libxlu_cfg_l.l
> >>>         -> we need to generate libxlu_cfg_l.c first
> >>> Then, libxlu_cfg_y.o fails to build because libxlu_cfg_l.h is been
> >>> updated do to the parallel build of libxlu_cfg_l.o.
> >>>
> >>> I can easily reproduce the issue with:
> >>>     touch libxlu_cfg_l.c; sleep .1; touch libxlu_cfg_l.l; sleep .1;
> >>>     touch libxlu_cfg_l.h; sleep .1; make -j
> >>> And having "sleep 1" in "%.c %.h: %.l" recipe, between `rm` and `flex`.
> >>
> >> Interesting. In my somewhat similar experiments I found that an issue
> >> exists only when one of the .c files is missing. Things looked to work
> >> properly when both were there, but one was outdated. Clearly that's
> >> yet more surprising behavior than, as you appear to observe, the issue
> >> simply being tied too "needs updating".
> > 
> > The actual problem is flex and bison output both files at the same time.
> > One process sees all of its prerequisites are up to do and therefore
> > proceeds to the next step.  A second process sees an out of date
> > prerequisite, and starts to run flex which wipes out the prerequisite of
> > the first process.
> > 
> > 
> > Crucially libxlu_cfg_y.o does NOT depend on libxlu_cfg_l.c, therefore GNU
> > Make is allowed to proceed with building libxlu_cfg_y.o even if
> > libxlu_cfg_l.c is out of date.
> > 
> > The problem is the rule to update libxlu_cfg_l.c ALSO updates
> > libxlu_cfg_l.h as a side-effect.


> According to my reading, an order-only prereq isn't what would help
> us here: "Occasionally you may want to ensure that a prerequisite is
> built before a target, but without forcing the target to be updated
> if the prerequisite is updated." This isn't the situation we're
> dealing with, I would say. We want the prereq updating to occur and
> the building of the dependent target to be deferred until all the
> prereqs were updated.

You could create a parser which handled all the tokenization itself and
only needed the constants for the tokens.  The result would need the
Lexer header (for the token constants), but not the Lexer itself.  In
such case case foo_y.o wouldn't need foo_l.c to be rebuilt if foo_l.h
was up to date.

"libxlu_cfg_y.o" does NOT require "libxlu_cfg_l.c".  What "libxlu_cfg_y.o"
requires is that IF "libxlu_cfg_l.c" (and thereby "libxlu_cfg_l.h") is
rebuilt, it must be rebuilt BEFORE "libxlu_cfg_y.o" is built.

"libxlu_cfg_y.o" needs "libxlu_cfg_l.c" to be built before it, but does
not itself require "libxlu_cfg_l.c" to be rebuilt.

Hopefully this is sounding rather more similar to what GNU Make's manual
says?


> >>> I don't know how to properly fix this yet.
> >>> Event writing "%.c %.h &: %.l" doesn't work, with make 4.3, which is
> >>> supposed to be grouped targets. But that's is maybe fixed in 4.4.
> >>
> >> Well, in my tests while 4.4 has fixed one issue with intermediate files
> >> affecting the behavior here, the issue we're possibly running into is
> >> still there. I've sent an inquiry to bug-make@xxxxxxx.
> > 
> > I do not believe this is a bug in GNU Make.  Merely double-output rules
> > cause precisely this sort of trouble in Make when run in parallel.
> 
> I suppose it can be viewed both ways: Make has all information to know
> that the file _is_ going to be touched, so it _could_ order operations
> accordingly.

Indeed, the syntax of the Makefile implies this, but the parser isn't
intelligent enough to add this extra constraint.  This makes it harder
for humans to write correct Makefiles, but it meets the requirements of
the specifications.  (may also yield faster builds in some situations)


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@xxxxxxx  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445