[Xen-devel] rochester (Thunder-X) bootloader lost output issue

[Ian Jackson <ian.jackson@xxxxxxxxxx>]

Hi.  A couple of weeks ago I wrote a writeup about this kind of
problem:

 test-arm64-arm64-examine 11 examine-serial/bootloader fail REGR. vs. 119195

This is going to affect all of osstest's branches and it will be
tiresome to force push it.  Particularly, I think it may cause the
examine jobs to become stuck on the rochesters and (even worse) if it
doesn't it will pass by accident requiring another force push.

I would like to either fix this, or write a programmatic workaround.
Would someone from the Xen ARM community be willing to debug this any
time soon ?

Failing that I will probably make a hostflag that causes this test to
be skipped, or something.

(Bottom-quoting my previous writeup for context.)

Thanks,
Ian.

Ian Jackson writes ("Re: [Xen-devel] [linux-arm-xen test] 134708: regressions - 
all pass"):
> Julien Grall writes ("Re: [Xen-devel] [linux-arm-xen test] 134708: 
> regressions - all pass"):
> > On 4/15/19 7:28 PM, osstest service owner wrote:
> > > flight 134708 linux-arm-xen real [real]
> > > http://logs.test-lab.xenproject.org/osstest/logs/134708/
> ...
> > >   test-arm64-arm64-examine    11 examine-serial/bootloader fail REGR. vs. 
> > > 119195
> > 
> > I am a bit confused with the failure here [1]:
> > 
> > 2019-04-13 22:36:55 Z ---------- substep 11 examine-serial/bootloader fail 
> > ---------- 
> > 
> > IIUC, we are looking for "osstest cookie". However this does not seem to
> > appear on rochester0's logs. Any insights how "osstest cookie" will be 
> > output?
> 
> Hi.  Yes, this is an issue I already discovered...
> 
> Ian Jackson writes ("[OSSTEST PATCH 00/62] Update to Debian stable 
> (stretch)"):
> > There seem to be a number of outstanding problems which I decided were
> > not blockers:
> ...
> >  * The serial output from the new rochester arm64 machines is missing
> >    some of the grub bootloader messages (and this is detected by one
> >    of the examine jobs).  I may need help from someone familiar with
> >    these machines' hardware/firmware.
> 
> I decided that this issue was not serious enough to be a blocker.  It
> may need some force pushes, especially if we can't fix it soon,
> because osstest will try this test, in other flights, on the
> rochesters, where it fails, and that looks like a regression.
> 
> In particular, this is not a regression in the linux-arm-xen branch.
> So we could force push it.
> 
> 
> As for the bug:
> 
> This check is checking that the osstest serial log captures the output
> from the bootloader.  This can be quite important for debugging,
> especially to distinguish hardware or configuration problems from
> software problems.  Seeing the bootloader output also provides
> a useful reference point to people unfamiliar with osstest.
> 
> This check is primarily a test on osstest itself; it is also part of
> machine commissioning.  The test works by putting a magic string in
> the bootloader output.
> 
> Compare the failure above
>   
> http://logs.test-lab.xenproject.org/osstest/logs/134708/test-arm64-arm64-examine/info.html
> with this one which ran on laxton1:
>   
> http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/info.html
> 
> If you look at the final grub config
>   
> http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/laxton1--grub.cfg.new
> you can see things like this
>   menuentry 'osstest cookie 0b4b9311157321d55cca9d25ff49159d Debian 
> GNU/Linux' --class debian --class gnu-linux --class gnu --class os 
> $menuentry_id_option 'gnulinux-simple-a8e4e9c6-4829-402b-93aa-964f9cf39c41' {
> 
> In
>   
> http://logs.test-lab.xenproject.org/osstest/logs/134643/test-arm64-arm64-examine/serial-laxton1.log
> near Apr 11 17:07:07.787491, you can see the osstest cookie coming
> out in the serial representation, in amongst the cursor positioning
> codes.
> 
> In
>   
> http://logs.test-lab.xenproject.org/osstest/logs/134708/test-arm64-arm64-examine/serial-rochester0.log
> near Apr 13 22:34:32.442724 you see a banner from grub and then a
> bunch of "invalid utf-8 sequence: \304", and at the end some actual
> messages from grub.  But the meat of the menu, including the cookie,
> is missing.
> 
> So the osstest considers the test a failure: the thing it arranged to
> appear in the bootloader output was not visible in the serial log.
> 
> 
> I have not yet investigated this beyond making these observations.  It
> is possible that there is a disagreement over character encoding or
> something, and what is happening is that line drawing characters are
> being misinterpreted by sympathy.  It might be a firmware bug or a
> grub bug or a sympathy bug or a hardware design issue.  I think the
> next steps would be to determine exactly what sequence of bytes are
> being sent by the host, and to investigate its BIOS options for
> controlling serial terminal output.
> 
> 
> HTH.
> 
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel