Re: [Xen-devel] Linux 3.18 no longer boots under Xen, please fix

[Wei Liu <wei.liu2@xxxxxxxxxx>]

On Mon, May 20, 2019 at 11:46:16AM +0100, Ian Jackson wrote:
> This has been true for some time.  osstest has been sending automatic
> mails including bisection reports like the one below, for about a
> month.
> 
> osstest's "tested and good" branch of linux-3.18 is 190 days behind
> the upstream tip.
> 
> Do we care about this at all ?
> 
> Note that the Xen Project does not have a dedicated team of QA
> engineers who do triage of test reports.  It is up to every maintainer
> of every Xen-related component to keep an eye on these test reports
> and investigate failures.
> 
> For the avoidance of any doubt, osstest's "bisection complete" reports
> can always be relied on to accurately identify a real, repeatable,
> regression.  osstest will only make such a report if it has reproduced
> several times both the failure at the commit it blames, and the
> success at the commit it says is OK.  (In trees with a lot of merging,
> the commit that is blamed is sadly often a merge, but one of its
> parents will be identified as good.)
> 
> In this case osstest has sent a significant number of functionally
> identical reports saying it has identified a regression in lots of
> different tests.  But this has not resulted in any action AFAIAA.
> 
> Please would someone involved with the Linux kernel end of things pick
> this up.
> 
> Thanks,
> Ian.
> 
> osstest service owner writes ("[linux-3.18 bisection] complete 
> test-amd64-amd64-xl"):
> > branch xen-unstable
> > xenbranch xen-unstable
> > job test-amd64-amd64-xl
> > testid xen-boot
> > 
> > Tree: linux 
> > git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> > Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> > Tree: ovmf git://xenbits.xen.org/osstest/ovmf.git
> > Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
> > Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> > Tree: xen git://xenbits.xen.org/xen.git
> > 
> > *** Found and reproduced problem changeset ***
> > 
> >   Bug is in tree:  linux 
> > git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> >   Bug introduced:  6b1ae527b1fdee86e81da0cb26ced75731c6c0fa
> >   Bug not present: ba6984fc0162f24a510ebc34e881b546b69c553b
> >   Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/136574/
> > 


It seems that there is something wrong with the IGB driver.

Seen in one of the normal flight:

http://logs.test-lab.xenproject.org/osstest/logs/136390/test-amd64-amd64-xl/serial-debina1.log

May 19 00:12:40.076089 [   12.671451] ------------[ cut here ]------------
May 19 00:12:40.076107 [   12.671463] WARNING: CPU: 7 PID: 1519 at 
drivers/pci/msi.c:975 pci_enable_msix+0x364/0x3b0()
May 19 00:12:40.088095 [   12.671470] Modules linked in: igb(+) i2c_algo_bit
May 19 00:12:40.088115 [   12.671481] CPU: 7 PID: 1519 Comm: systemd-udevd Not 
tainted 3.18.140 #1
May 19 00:12:40.100092 [   12.671487] Hardware name: Supermicro 
SYS-5018D-FN8T/X10SDV-TP8F, BIOS 1.0b 11/21/2016
May 19 00:12:40.112083 [   12.671493]  0000000000000000 ffff880005147a78 
ffffffff817f4db9 0000000000000000
May 19 00:12:40.112108 [   12.671502]  0000000000000009 ffff880005147ab8 
ffffffff810b740c ffff8800196d3000
May 19 00:12:40.124110 [   12.671511]  ffff8800196d3000 000000000000000a 
ffff88000409e998 0000000000000009
May 19 00:12:40.124132 [   12.671521] Call Trace:
May 19 00:12:40.136087 [   12.671528]  [<ffffffff817f4db9>] dump_stack+0x7a/0x96
May 19 00:12:40.136108 [   12.671534]  [<ffffffff810b740c>] 
warn_slowpath_common+0x7c/0xa0
May 19 00:12:40.148084 [   12.671540]  [<ffffffff810b74d5>] 
warn_slowpath_null+0x15/0x20
May 19 00:12:40.148107 [   12.671547]  [<ffffffff81364344>] 
pci_enable_msix+0x364/0x3b0
May 19 00:12:40.148121 [   12.671563]  [<ffffffffa0005b88>] ? 
igb_alloc_q_vector.isra.57+0x98/0x280 [igb]
May 19 00:12:40.160098 [   12.671570]  [<ffffffff813643bd>] 
pci_enable_msix_range+0x2d/0x70
May 19 00:12:40.172086 [   12.671580]  [<ffffffffa0005e76>] 
igb_set_interrupt_capability+0x86/0x1b0 [igb]
May 19 00:12:40.172111 [   12.671591]  [<ffffffffa0009d63>] 
igb_probe+0x4d3/0x1270 [igb]
May 19 00:12:40.184090 [   12.671600]  [<ffffffff813530e9>] 
pci_device_probe+0x69/0xc0
May 19 00:12:40.184111 [   12.671608]  [<ffffffff81414eec>] 
really_probe+0x6c/0x240
May 19 00:12:40.196087 [   12.671613]  [<ffffffff814151a9>] 
__driver_attach+0x99/0xa0
May 19 00:12:40.196108 [   12.671619]  [<ffffffff81415110>] ? 
__device_attach+0x50/0x50
May 19 00:12:40.208087 [   12.671625]  [<ffffffff81413273>] 
bus_for_each_dev+0x63/0xa0
May 19 00:12:40.208109 [   12.671631]  [<ffffffff81414a89>] 
driver_attach+0x19/0x20
May 19 00:12:40.220087 [   12.671637]  [<ffffffff8141469c>] 
bus_add_driver+0x10c/0x210
May 19 00:12:40.220109 [   12.671643]  [<ffffffffa0029000>] ? 0xffffffffa0029000
May 19 00:12:40.220123 [   12.671648]  [<ffffffff81415a8f>] 
driver_register+0x5f/0xf0
May 19 00:12:40.232095 [   12.671655]  [<ffffffff81351aa6>] 
__pci_register_driver+0x46/0x50
May 19 00:12:40.232123 [   12.671665]  [<ffffffffa002904f>] 
igb_init_module+0x4f/0x51 [igb]
May 19 00:12:40.244091 [   12.671671]  [<ffffffff81002128>] 
do_one_initcall+0x88/0x1e0
May 19 00:12:40.244112 [   12.671678]  [<ffffffff811c0412>] ? 
__vunmap+0xa2/0x100
May 19 00:12:40.256090 [   12.671685]  [<ffffffff8112d145>] 
load_module+0x1bd5/0x2320
May 19 00:12:40.256111 [   12.671692]  [<ffffffff8112da16>] 
SYSC_finit_module+0x96/0xa0
May 19 00:12:40.268093 [   12.671699]  [<ffffffff8112da39>] 
SyS_finit_module+0x9/0x10
May 19 00:12:40.268113 [   12.671705]  [<ffffffff817fd789>] 
system_call_fastpath+0x12/0x17
May 19 00:12:40.280089 [   12.671710] ---[ end trace 723a474d0c2ed165 ]---
May 19 00:12:40.280109 (XEN) d0: Forcing read-only access to MFN fb18c
May 19 00:12:40.292086 [   12.672504] BUG: unable to handle kernel paging 
request at ffffc900000fc00c
May 19 00:12:40.292110 [   12.672511] IP: [<ffffffff81364105>] 
pci_enable_msix+0x125/0x3b0
May 19 00:12:40.304086 [   12.672519] PGD 1a833067 PUD 1a834067 PMD 1a835067 
PTE 0
May 19 00:12:40.304106 [   12.672527] Oops: 0000 [#1] SMP 
May 19 00:12:40.304118 [   12.672532] Modules linked in: igb(+) i2c_algo_bit
May 19 00:12:40.316091 [   12.672540] CPU: 7 PID: 1519 Comm: systemd-udevd 
Tainted: G        W      3.18.140 #1
May 19 00:12:40.316116 [   12.672546] Hardware name: Supermicro 
SYS-5018D-FN8T/X10SDV-TP8F, BIOS 1.0b 11/21/2016
May 19 00:12:40.328096 [   12.672552] task: ffff88001abf0000 ti: 
ffff880005144000 task.ti: ffff880005144000
May 19 00:12:40.340096 [   12.672558] RIP: e030:[<ffffffff81364105>]  
[<ffffffff81364105>] pci_enable_msix+0x125/0x3b0
May 19 00:12:40.340122 [   12.672566] RSP: e02b:ffff880005147ad8  EFLAGS: 
00010286
May 19 00:12:40.352090 [   12.672570] RAX: ffffc900000fc00c RBX: 
ffff8800196d3000 RCX: 00000000fffffffa
May 19 00:12:40.352112 [   12.672576] RDX: 0000000000000000 RSI: 
ffff880004fb0200 RDI: ffff880004fb0200
May 19 00:12:40.364097 [   12.672582] RBP: ffff880005147b38 R08: 
ffff880004fb0200 R09: ffff88001a400658
May 19 00:12:40.376087 [   12.672587] R10: 0000000000000000 R11: 
000000000001a0f1 R12: 000000000000000d
May 19 00:12:40.376108 [   12.672593] R13: ffff880005344480 R14: 
000000000008800c R15: ffff88000409e998
May 19 00:12:40.388029 [   12.672607] FS:  00007fc8fb4ed8c0(0000) 
GS:ffff88001fdc0000(0000) knlGS:0000000000000000
May 19 00:12:40.400084 [   12.672613] CS:  e033 DS: 0000 ES: 0000 CR0: 
0000000080050033
May 19 00:12:40.400104 [   12.672618] CR2: ffffc900000fc00c CR3: 
0000000000027000 CR4: 0000000000040660
May 19 00:12:40.412088 [   12.672627] Stack:
May 19 00:12:40.412105 [   12.672629]  ffff880005147b28 ffff8800196d37a0 
0000000000000009 ffff8800196d37a0
May 19 00:12:40.412122 [   12.672638]  ffff88000409e780 0000000000090007 
0000000000000007 0000000000000009
May 19 00:12:40.424096 [   12.672646]  ffff8800196d3000 ffff88000409e998 
0000000000000009 ffff8800196d3000
May 19 00:12:40.436098 [   12.672655] Call Trace:
May 19 00:12:40.436114 [   12.672659]  [<ffffffff813643bd>] 
pci_enable_msix_range+0x2d/0x70
May 19 00:12:40.436130 [   12.672670]  [<ffffffffa0005e76>] 
igb_set_interrupt_capability+0x86/0x1b0 [igb]
May 19 00:12:40.448097 [   12.672680]  [<ffffffffa0009d63>] 
igb_probe+0x4d3/0x1270 [igb]
May 19 00:12:40.448117 [   12.672688]  [<ffffffff813530e9>] 
pci_device_probe+0x69/0xc0
May 19 00:12:40.460094 [   12.672694]  [<ffffffff81414eec>] 
really_probe+0x6c/0x240
May 19 00:12:40.460114 [   12.672700]  [<ffffffff814151a9>] 
__driver_attach+0x99/0xa0
May 19 00:12:40.472093 [   12.672705]  [<ffffffff81415110>] ? 
__device_attach+0x50/0x50
May 19 00:12:40.472114 [   12.672711]  [<ffffffff81413273>] 
bus_for_each_dev+0x63/0xa0
May 19 00:12:40.484090 [   12.672717]  [<ffffffff81414a89>] 
driver_attach+0x19/0x20
May 19 00:12:40.484110 [   12.672722]  [<ffffffff8141469c>] 
bus_add_driver+0x10c/0x210
May 19 00:12:40.496091 [   12.672727]  [<ffffffffa0029000>] ? 0xffffffffa0029000
May 19 00:12:40.496110 [   12.672733]  [<ffffffff81415a8f>] 
driver_register+0x5f/0xf0
May 19 00:12:40.508095 [   12.672739]  [<ffffffff81351aa6>] 
__pci_register_driver+0x46/0x50
May 19 00:12:40.508117 [   12.672748]  [<ffffffffa002904f>] 
igb_init_module+0x4f/0x51 [igb]
May 19 00:12:40.520089 [   12.672754]  [<ffffffff81002128>] 
do_one_initcall+0x88/0x1e0
May 19 00:12:40.520111 [   12.672763]  [<ffffffff811c0412>] ? 
__vunmap+0xa2/0x100
May 19 00:12:40.532086 [   12.672769]  [<ffffffff8112d145>] 
load_module+0x1bd5/0x2320
May 19 00:12:40.532107 [   12.672775]  [<ffffffff8112da16>] 
SYSC_finit_module+0x96/0xa0
May 19 00:12:40.544087 [   12.672782]  [<ffffffff8112da39>] 
SyS_finit_module+0x9/0x10
May 19 00:12:40.544108 [   12.672787]  [<ffffffff817fd789>] 
system_call_fastpath+0x12/0x17
May 19 00:12:40.556083 [   12.672792] Code: 0c 4c 89 ee 49 8d 04 c7 48 89 55 b8 
44 0f b7 70 04 89 08 41 8b 7d 0c 41 c1 e6 04 41 83 c6 0c e8 d2 19 da ff 49 63 
c6 49 03 45 28 <8b> 00 be 01 00 00 00 41 89 45 08 4c 89 ef e8 58 83 d0 ff 48 8b 
May 19 00:12:40.568096 [   12.672858] RIP  [<ffffffff81364105>] 
pci_enable_msix+0x125/0x3b0
May 19 00:12:40.580087 [   12.672865]  RSP <ffff880005147ad8>
May 19 00:12:40.580106 [   12.672868] CR2: ffffc900000fc00c
May 19 00:12:40.580117 [   12.672873] ---[ end trace 723a474d0c2ed166 ]---


> > 
> >   (Revision log too long, omitted.)
> > 
> > 
> > For bisection revision-tuple graph see:
> >    
> > http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-amd64-xl.xen-boot.html
> > Revision IDs in each graph node refer, respectively, to the Trees above.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel