Re: [XEN PATCH] tools: Have flex and bison mandatory

[Jan Beulich <jbeulich@xxxxxxxx>]

On 08.02.2023 19:51, Elliott Mitchell wrote:
> On Wed, Feb 08, 2023 at 09:02:35AM +0100, Jan Beulich wrote:
>> On 07.02.2023 19:02, Anthony PERARD wrote:
>>>
>>> I think the issue that Andrew saw in the CI with "libxlu_cfg_y.o"
>>> failing to build while "libxlu_cfg_l.[ch]" are been regenerated is
>>> probably because make doesn't consider "libxlu_cfg_l.[ch]" as a group of
>>> targets.
>>>
>>> If for some reason we have:
>>>     libxlu_cfg_l.h newer than libxlu_cfg_l.l newer than libxlu_cfg_l.c
>>>
>>> Then make seems to take two parallel decision:
>>>     When building libxlu_cfg_y.o:
>>>         libxlu_cfg_l.h is newer than libxlu_cfg_l.l
>>>         -> libxlu_cfg_l.h is up-to-date, start building libxlu_cfg_y.o
>>>     When building libxlu_cfg_l.o:
>>>         libxlu_cfg_l.c is older than libxlu_cfg_l.l
>>>         -> we need to generate libxlu_cfg_l.c first
>>> Then, libxlu_cfg_y.o fails to build because libxlu_cfg_l.h is been
>>> updated do to the parallel build of libxlu_cfg_l.o.
>>>
>>> I can easily reproduce the issue with:
>>>     touch libxlu_cfg_l.c; sleep .1; touch libxlu_cfg_l.l; sleep .1;
>>>     touch libxlu_cfg_l.h; sleep .1; make -j
>>> And having "sleep 1" in "%.c %.h: %.l" recipe, between `rm` and `flex`.
>>
>> Interesting. In my somewhat similar experiments I found that an issue
>> exists only when one of the .c files is missing. Things looked to work
>> properly when both were there, but one was outdated. Clearly that's
>> yet more surprising behavior than, as you appear to observe, the issue
>> simply being tied too "needs updating".
> 
> The actual problem is flex and bison output both files at the same time.
> One process sees all of its prerequisites are up to do and therefore
> proceeds to the next step.  A second process sees an out of date
> prerequisite, and starts to run flex which wipes out the prerequisite of
> the first process.
> 
> 
> Crucially libxlu_cfg_y.o does NOT depend on libxlu_cfg_l.c, therefore GNU
> Make is allowed to proceed with building libxlu_cfg_y.o even if
> libxlu_cfg_l.c is out of date.
> 
> The problem is the rule to update libxlu_cfg_l.c ALSO updates
> libxlu_cfg_l.h as a side-effect.
> 
> 
> As such libxlu_cfg_y.o needs an order-only dependency (assuming GNU Make
> syntax is allowed) on libxlu_cfg_l.c, so the two rules DON'T run in
> parallel and cause havoc.

Since you keep referring to this: Yes, GNU make syntax is allowed. We
use pattern rules, which themselves are an extension of the original
suffix rules. The issue is that we can't (unconditionally) use stuff
which was added only in newer versions of GNU make. (See ./README
both for a statement on us requiring GNU make as well as one on the
minimum version we expect to be in use.)

According to my reading, an order-only prereq isn't what would help
us here: "Occasionally you may want to ensure that a prerequisite is
built before a target, but without forcing the target to be updated
if the prerequisite is updated." This isn't the situation we're
dealing with, I would say. We want the prereq updating to occur and
the building of the dependent target to be deferred until all the
prereqs were updated.

>>> I don't know how to properly fix this yet.
>>> Event writing "%.c %.h &: %.l" doesn't work, with make 4.3, which is
>>> supposed to be grouped targets. But that's is maybe fixed in 4.4.
>>
>> Well, in my tests while 4.4 has fixed one issue with intermediate files
>> affecting the behavior here, the issue we're possibly running into is
>> still there. I've sent an inquiry to bug-make@xxxxxxx.
> 
> I do not believe this is a bug in GNU Make.  Merely double-output rules
> cause precisely this sort of trouble in Make when run in parallel.

I suppose it can be viewed both ways: Make has all information to know
that the file _is_ going to be touched, so it _could_ order operations
accordingly.

Jan