Re: [Xen-devel] [xen-unstable test] 65141: regressions - FAIL

["Hu, Robert" <robert.hu@xxxxxxxxx>]

> -----Original Message-----
> From: Ian Campbell [mailto:ian.campbell@xxxxxxxxxx]
> Sent: Saturday, December 5, 2015 4:10 PM
> To: Hu, Robert <robert.hu@xxxxxxxxx>; Ian Jackson
> <Ian.Jackson@xxxxxxxxxxxxx>; Nakajima, Jun <jun.nakajima@xxxxxxxxx>;
> Tian, Kevin <kevin.tian@xxxxxxxxx>
> Cc: xen-devel@xxxxxxxxxxxxxxxxxxx; osstest service owner
> <osstest-admin@xxxxxxxxxxxxxx>; Jan Beulich <jbeulich@xxxxxxxx>;
> Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
> Subject: Re: [xen-unstable test] 65141: regressions - FAIL
> 
> On Wed, 2015-12-02 at 13:51 +0000, Ian Campbell wrote:
> 
> > http://osstest.test-lab.xenproject.org/~osstest/pub/logs/65301/
> >
> > I think that ought to give a baseline for the bisector to work with. I'll
> > prod it to do so.
> 
> Results are below. TL;DR: d02e84b9d9d "vVMX: use latched VMCS machine
> address" is somehow at fault.
> 
> It appears to be somewhat machine specific, the one this has been
> failing on is godello* which says "CPU0: Intel(R) Xeon(R) CPU E3-1220
> v3 @ 3.10GHz stepping 03" in its serial log.
> 
> Andy suggested this might be related to cpu_has_vmx_vmcs_shadowing
> so Haswell and newer vs IvyBridge and older.
> 
> Ian.
> 
> branch xen-unstable
> xenbranch xen-unstable
> job test-amd64-amd64-qemuu-nested-intel
> testid debian-hvm-install/l1/l2
> 
> Tree: linux git://xenbits.xen.org/linux-pvops.git
> Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
> Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> Tree: xen git://xenbits.xen.org/xen.git
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  xen git://xenbits.xen.org/xen.git
>   Bug introduced:  d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3
>   Bug not present: 3b47431691409004c7218f6a6ba5c9c0bcf483ea
>   Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/65388/
> 
> 
>   commit d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3
>   Author: Jan Beulich <jbeulich@xxxxxxxx>
>   Date:   Tue Nov 24 12:07:27 2015 +0100
> 
>       vVMX: use latched VMCS machine address
> 
>       Instead of calling domain_page_map_to_mfn() over and over, latch
> the
>       guest VMCS machine address unconditionally (i.e. independent of
> whether
>       VMCS shadowing is supported by the hardware).
> 
>       Since this requires altering the parameters of __[gs]et_vmcs{,_real}()
>       (and hence all their callers) anyway, take the opportunity to also drop
>       the bogus double underscores from their names (and from
>       __[gs]et_vmcs_virtual() as well).
> 
>       Signed-off-by: Jan Beulich <jbeulich@xxxxxxxx>
>       Acked-by: Kevin Tian <kevin.tian@xxxxxxxxx>
> 
> 
> For bisection revision-tuple graph see:
> 
> http://logs.test-lab.xenproject.org/osstest/results-adhoc/bisect/xen-unstabl
> e/test-amd64-amd64-qemuu-nested-intel.debian-hvm-install--l1--l2.html
> Revision IDs in each graph node refer, respectively, to the Trees above.
> 
> ----------------------------------------
> Running cs-bisection-step
> --graph-out=/home/logs/results-adhoc/bisect/xen-unstable/test-amd64-am
> d64-qemuu-nested-intel.debian-hvm-install--l1--l2
> --summary-out=tmp/65388.bisection-summary
> --blessings=real,real-bisect,adhoc-bisect --basis-template=65301
> --basis-flight=65301 xen-unstable test-amd64-amd64-qemuu-nested-intel
> debian-hvm-install/l1/l2
> Searching for failure / basis pass:
>  65314 fail [host=godello1] / template as basis? using template as basis.
> Failure / basis pass flights: 65314 / 65301
> (tree with no url: ovmf)
> (tree with no url: seabios)
> Tree: linux git://xenbits.xen.org/linux-pvops.git
> Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
> Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> Tree: xen git://xenbits.xen.org/xen.git
> Latest 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 2c4f313a7e62c7e559a469d4af4c3d03c49afa43
> Basis pass 1230ae0e99e05ced8a945a1a2c5762ce5c6c97c9
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 816609b2841297925a223ec377c336360e044ee5
> d07f63fa6e70350b23e7acbde06129247c4e655d
> Generating revisions with ./adhoc-revtuple-generator
> git://xenbits.xen.org/linux-pvops.git#1230ae0e99e05ced8a945a1a2c5762ce
> 5c6c97c9-769b79eb206ad5b0249a08665fefb913c3d1998e
> git://xenbits.xen.org/osstest/linux-firmware.git#c530a75c1e6a472b0eb955
> 8310b518f0dfcd8860-c530a75c1e6a472b0eb9558310b518f0dfcd8860
> git://xenbits.xen.org/qemu-xen-traditional.git#bc00cad75d8bcc3ba696992b
> ec219c21db8406aa-bc00cad75d8bcc3ba696992bec219c21db8406aa
> git://xenbits.xen.org/qemu-xen.git#816609b2841297925a223ec377c336360
> e044ee5-3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> git://xenbits.xen.org/xen.git#d07f63fa6e70350b23e7acbde06129247c4e655
> d-2c4f313a7e62c7e559a469d4af4c3d03c49afa43
> Loaded 17133 nodes in revision graph
> Searching for test results:
>  65114 [host=italia0]
>  65141 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> b1d398b67781140d1c6efd05778d0ad4103b2a32
>  65162 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 713b7e4ef2aa4ec3ae697cde9c81d5a57548f9b1
>  65164 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> b1d398b67781140d1c6efd05778d0ad4103b2a32
>  65186 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> b1d398b67781140d1c6efd05778d0ad4103b2a32
>  65217 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> b1d398b67781140d1c6efd05778d0ad4103b2a32
>  65233 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> b1d398b67781140d1c6efd05778d0ad4103b2a32
>  65287 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> c8eb0ec277ae387e78d685523e0fee633e46f046
>  65333 pass 1230ae0e99e05ced8a945a1a2c5762ce5c6c97c9
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 816609b2841297925a223ec377c336360e044ee5
> d07f63fa6e70350b23e7acbde06129247c4e655d
>  65314 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 2c4f313a7e62c7e559a469d4af4c3d03c49afa43
>  65354 pass 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 816609b2841297925a223ec377c336360e044ee5
> c35eefded2992fc9b979f99190422527650872fd
>  65267 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 4c6cd64519f9bc270a7278128c94e4b66e3d2077
>  65301 pass 1230ae0e99e05ced8a945a1a2c5762ce5c6c97c9
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 816609b2841297925a223ec377c336360e044ee5
> d07f63fa6e70350b23e7acbde06129247c4e655d
>  65351 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 0ba0faa7f02fb310a330579adeef3534e6eca5a8
>  65343 pass 997badf1e03dfd39854094f7767e1a4cf5ed310b
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 816609b2841297925a223ec377c336360e044ee5
> d07f63fa6e70350b23e7acbde06129247c4e655d
>  65325 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> c8eb0ec277ae387e78d685523e0fee633e46f046
>  65345 pass e7a10d9297c1abfd27138d86a13f7b6435634a46
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 816609b2841297925a223ec377c336360e044ee5
> d07f63fa6e70350b23e7acbde06129247c4e655d
>  65341 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 2c4f313a7e62c7e559a469d4af4c3d03c49afa43
>  65342 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> c87303c04738b0e837da6e891eb561de0bf1b64e
>  65362 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3
>  65367 pass 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 816609b2841297925a223ec377c336360e044ee5
> 827db7b26384ce083df7154d77f13379b2cf4121
>  65370 pass 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 827db7b26384ce083df7154d77f13379b2cf4121
>  65374 pass 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 3b47431691409004c7218f6a6ba5c9c0bcf483ea
>  65379 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3
>  65380 pass 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 3b47431691409004c7218f6a6ba5c9c0bcf483ea
>  65383 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3
>  65384 pass 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 3b47431691409004c7218f6a6ba5c9c0bcf483ea
>  65388 fail 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3
> Searching for interesting versions
>  Result found: flight 65301 (pass), for basis pass
>  Result found: flight 65314 (fail), for basis failure
>  Repro found: flight 65333 (pass), for basis pass
>  Repro found: flight 65341 (fail), for basis failure
>  0 revisions at 769b79eb206ad5b0249a08665fefb913c3d1998e
> c530a75c1e6a472b0eb9558310b518f0dfcd8860
> bc00cad75d8bcc3ba696992bec219c21db8406aa
> 3fb401edbd8e9741c611bfddf6a2032ca91f55ed
> 3b47431691409004c7218f6a6ba5c9c0bcf483ea
> No revisions left to test, checking graph state.
>  Result found: flight 65374 (pass), for last pass
>  Result found: flight 65379 (fail), for first failure
>  Repro found: flight 65380 (pass), for last pass
>  Repro found: flight 65383 (fail), for first failure
>  Repro found: flight 65384 (pass), for last pass
>  Repro found: flight 65388 (fail), for first failure
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  xen git://xenbits.xen.org/xen.git
>   Bug introduced:  d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3
>   Bug not present: 3b47431691409004c7218f6a6ba5c9c0bcf483ea
>   Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/65388/
> 
> 
>   commit d02e84b9d9d16b6b56186f0dfdcb3c90b83c82a3
>   Author: Jan Beulich <jbeulich@xxxxxxxx>
>   Date:   Tue Nov 24 12:07:27 2015 +0100
> 
>       vVMX: use latched VMCS machine address
> 
>       Instead of calling domain_page_map_to_mfn() over and over, latch
> the
>       guest VMCS machine address unconditionally (i.e. independent of
> whether
>       VMCS shadowing is supported by the hardware).
> 
>       Since this requires altering the parameters of __[gs]et_vmcs{,_real}()
>       (and hence all their callers) anyway, take the opportunity to also drop
>       the bogus double underscores from their names (and from
>       __[gs]et_vmcs_virtual() as well).
> 
>       Signed-off-by: Jan Beulich <jbeulich@xxxxxxxx>
>       Acked-by: Kevin Tian <kevin.tian@xxxxxxxxx>
> 
> Revision graph left in
> /home/logs/results-adhoc/bisect/xen-unstable/test-amd64-amd64-qemuu-n
> ested-intel.debian-hvm-install--l1--l2.{dot,ps,png,html,svg}.
> ----------------------------------------
> 65388: tolerable ALL FAIL
> 
> flight 65388 xen-unstable adhoc-bisect [real]
> http://logs.test-lab.xenproject.org/osstest/logs/65388/
> 
> Failures :-/ but no regressions.
> 
> Tests which did not succeed,
> including tests which could not be run:
>  test-amd64-amd64-qemuu-nested-intel 16 debian-hvm-install/l1/l2 fail
> baseline untested
> 
> 
> jobs:
>  test-amd64-amd64-qemuu-nested-intel                          fail
[Hu, Robert] 

I tried to reproduce the failure in my environment, with your Qemu/Xen/Dom0
/Seabios/OVMF designation, but failed at even earlier stage -- normal L1 guest
installation.

xen be: vkbd-0: initialise() failed
xen be: vkbd-0: initialise() failed
xen be: vkbd-0: initialise() failed

Not sure if this related to linux-pvops tree, for I previously using 
linux-stable tree.

For your failure, as Kevin mentioned in other mail, we will find someone to 
look into.
Would you find out the detailed log of ' debian-hvm-install--l1--l2' step? so 
that he
can start analyze, as I cannot reproduce it right now.
(I didn't managed to find out the log in your web links)

> 
> 
> ------------------------------------------------------------
> sg-report-flight on osstest.test-lab.xenproject.org
> logs: /home/logs/logs
> images: /home/logs/images
> 
> Logs, config files, etc. are available at
>     http://logs.test-lab.xenproject.org/osstest/logs
> 
> Explanation of these reports, and of osstest in general, is at
> 
> http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=m
> aster
> 
> http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
> 
> Test harness code can be found at
>     http://xenbits.xen.org/gitweb?p=osstest.git;a=summary

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel