Re: [Xen-devel] [linux-linus test] 30356: regressions - FAIL

[Julien Grall <julien.grall@xxxxxxxxxx>]

Hi,

On 09/23/2014 04:20 PM, Julien Grall wrote:
> On 09/23/2014 12:45 PM, Ian Campbell wrote:
>> On Tue, 2014-09-23 at 12:30 +0100, xen.org wrote:
>>> flight 30356 linux-linus real [real]
>>> http://www.chiark.greenend.org.uk/~xensrcts/logs/30356/
>>>
>>> Regressions :-(
>>>
>>> Tests which did not succeed and are blocking,
>>> including tests which could not be run:
>>>  test-armhf-armhf-xl           9 guest-start               fail REGR. vs. 
>>> 30019
>>
>> http://www.chiark.greenend.org.uk/~xensrcts/logs/30356/test-armhf-armhf-xl/info.html
>>
>> This has failed in everyone of a couple of dozen runs since the end of
>> August.
>>
>> 30019 was OK, so was 30032 but from 30050 onwards it is consistently
>> failing.
>>
>> Failure is:
>> 2014-09-23 05:39:03 Z guest debian.guest.osstest 5a:36:0e:94:00:07 22 
>> link/ip/tcp: waiting 40s...
>> 2014-09-23 05:39:03 Z guest debian.guest.osstest 5a:36:0e:94:00:07 22 
>> link/ip/tcp: no active lease (waiting) ...
>> ...
>> 2014-09-23 05:39:44 Z FAILURE: guest debian.guest.osstest 5a:36:0e:94:00:07 
>> 22 link/ip/tcp: wait timed out: no active lease.
>> failure: guest debian.guest.osstest 5a:36:0e:94:00:07 22 link/ip/tcp: wait 
>> timed out: no active lease.
>>
>> Guest console logs are empty, host console log is uninteresting. The
>> xenstore dump shows that none of the devices are connected, all of which
>> suggests an early crash of some sort.
>>
>> Baseline was f1bd473f aka v3.17-rc2-42-gf1bd473 and this is b0e2a55c6536
>> aka v3.17-rc6-7-gb0e2a55
>>
>> Perhaps more interestingly the first failure in 30050 was 
>> fd5984d7c8e aka v3.17-rc2-227-gfd5984d, which is a much smaller range.
>>
>> Nothing in the log (below) looks terribly exciting to me and there's not
>> a lot to go on. Anyone got any ideas?
> 
> It looks like to be an issue with the .config. I have a working linux if
> I disable most the CONFIG_MACH_* and CONFIG_ARCH_*.
> 
> Touching those CONFIGs also modify CONFIG_DEBUG_LL_INCLUDE.
> 
> The current value of CONFIG_DEBUG_LL_INCLUDE looks wrong to me but I
> still need to figure out if it's really an issue or not.

I've spent more time to debug this issue and found another one, which is
finally related.

When multi_v7 config (+ Xen options) is used, DOM0 will crash [1] in
the swiotlb code.

The config is used short page table, which make linux using 32 bits
for the physical address. If we choose to use 64 bits for the DMA
(enabled when Xen is selected), BUG(dma != phys) will likely hit with
the recent change in swiotlb (i.e handle multiple grant reference on
the same mapping).

We now require to use LPAE by default. Enable CONFIG_LPAE=y also
solve guest boot. I haven't yet figured out if it's related or not.

I guess we will have to select LPAE when XEN is enabled, right? If
it's the case that would mean the user won't be able to compile a
Linux guest with short page table and Xen.

Any though?

Regards,


[1] DOM0 crash:

[  110.968052] kernel BUG at 
/local/home/julien/works/linux/drivers/xen/swiotlb-xen.c:101!
[  110.976124] Internal error: Oops - BUG: 0 [#1] SMP ARM
[  110.981331] Modules linked in:
[  110.984459] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.17.0-rc6+ #84
[  110.990968] task: c0bbdaa0 ti: c0bb2000 task.ti: c0bb2000
[  110.996449] PC is at xen_unmap_single+0xc4/0xc8
[  111.001037] LR is at xen_unmap_single+0xc4/0xc8
[  111.005637] pc : [<c04c4670>]    lr : [<c04c4670>]    psr: 20010193
[  111.005637] sp : c0bb3d90  ip : 00000001  fp : b679a000
[  111.017274] r10: 00000200  r9 : 00000002  r8 : 00000002
[  111.022562] r7 : 00000000  r6 : 002b679a  r5 : 00000002  r4 : b679a000
[  111.029159] r3 : 0000075d  r2 : c0bba968  r1 : 20010193  r0 : 00000020
[  111.035757] Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment 
kernel
[  111.043221] Control: 10c5387d  Table: 39fdc06a  DAC: 00000015
[  111.049036] Process swapper/0 (pid: 0, stack limit = 0xc0bb2250)
[  111.055112] Stack: (0xc0bb3d90 to 0xc0bb4000)
[  111.059540] 3d80:                                     c0bb3da4 c0268948 
d9d8f980 dbbb2cc0
[  111.067786] 3da0: c0bb3dcc db0e6210 00208040 db717f00 00000001 00000001 
00000002 db0e6210
[  111.076032] 3dc0: 00000000 db439c90 0000259c c04c48b0 00000200 00000002 
00000000 d80b7f18
[  111.084278] 3de0: d8042058 db465088 db717f00 db464000 00000001 000000b0 
00000001 c0575b1c
[  111.092533] 3e00: 00000000 c0cbfb40 c0cbfb40 c0295ca8 40010193 db465088 
db464000 db4656c8
[  111.100770] 3e20: 00000001 c0575df0 00000000 00000017 db464000 c05761b8 
db464000 db4656c8
[  111.109016] 3e40: 00000000 db464000 00000008 e0880100 00000001 db443810 
db439b90 c058c2ac
[  111.117267] 3e60: 8007a120 00009896 00000000 00989680 00000000 00989680 
0016e360 00000000
[  111.125517] 3e80: db439c90 e0880000 00000001 00000001 00000001 db439c90 
0000259c c058ca40
[  111.133754] 3ea0: c0bba54c 80010193 d3d7ef03 c0bba44c 00000001 db447780 
c0c065d0 00000000
[  111.142006] 3ec0: 00000000 00000073 db00b780 c0c9841c 00000001 c0282478 
c0cbff98 d56e6afc
[  111.150246] 3ee0: 00000019 db00b780 c0c065d0 00000000 e0804000 c0bb0060 
c0bb2000 00000000
[  111.158501] 3f00: c0842468 c02825b0 00000000 db00b780 c0c065d0 c02850d0 
c0285028 00000073
[  111.166738] 3f20: 00000073 c0281cc8 c0bafc78 c020fb40 e080400c c0bba960 
c0bb3f58 c0208910
[  111.174984] 3f40: c020fe44 c020fe48 60010013 ffffffff c0bb3f8c c02132c0 
ffffffed 00000000
[  111.183230] 3f60: ffffffed c0220260 c0bba504 c0bba4a0 00000000 00000000 
c0bb0060 c0bb2000
[  111.191485] 3f80: 00000000 c0842468 00000020 c0bb3fa0 c020fe44 c020fe48 
60010013 ffffffff
[  111.199722] 3fa0: 00000000 c0277e5c c0bb3fb4 c0c9841a 00000000 c0c985c0 
00000000 c0affba8
[  111.207973] 3fc0: ffffffff ffffffff c0aff5ec 00000000 00000000 c0b73eb8 
c0c99214 c0bba484
[  111.216214] 3fe0: c0b73eb4 c0bbeb94 2020406a 413fc0f2 00000000 20208074 
00000000 00000000
[  111.224472] [<c04c4670>] (xen_unmap_single) from [<c04c48b0>] 
(xen_swiotlb_unmap_sg_attrs+0x48/0x68)



-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel