Re: [Xen-devel] HVM support for e820_host (Was: Bug: Limitation of <=2GB RAM in domU persists with 4.3.0)

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

> >>Then there seems to be a hole in dom0:
> >>40000000-fedfffff which talles up with the dom0 dmesg output above
> >>about it being for the PCI devices, i.e. that's the IOMEM region
> >>(from 1GB to a lilttle under 4GB).
> >>
> >>But in domU, the 40000000-a77fffff is available as RAM.
> >
> >OK, so that is the goal - make hvmloader construct the E820 memory
> >layout and all of its pieces to fit that layout.
> 
> I am actually leaning toward only copying the holes from the
> host E820. The domU already seems to be successfully using various
> memory ranges that correspond to reserved and acpi ranges, so
> it doesn't look like these are a problem.

OK.
> 
> >>On the face of it, that's actually fine - my PCI IOMEM mappings show
> >>the lowest mapping (according to lspci -vvv) starts at a8000000,
> >
> ><surprise>
> 
> Indeed - on the host, the hole is 1GB-4GB, but there is no IOMEM
> mapped between 1024M and 2688MB. Hence why I can get away with a
> domU memory allocation up to 2688MB.

When you say 'IOMEM' you mean /proc/iomem output?

> 
> >>which falls into the domU area marked as "HOLE" (a7800000-fc000000).
> >>And this does in fact appears to be where domU maps the GPU in both
> >>of my VMs:
> >>
> >>E0000000-E7FFFFFF
> >>E8000000-EBFFFFFF
> >>EC000000-EDFFFFFF
> >>
> >>and this doesn't overlap with any mapped PCI IOMEM according to
> >>lspci.
> >>
> >>If we assume that anything below a8000000 doesn't actually matter in
> >>this case (since if I give up to a8000000 memory to a domU
> >>everything works absolutely fine indefinitely, I am at a loss to
> >
> >
> >Just to make sure I am not leading you astray. You are getting
> >_no_ crashes
> >when you have a guest with 1GB?
> 
> I haven't tried limiting a guest to 1GB recently. My PCI passthrough
> domUs all have 2688MB assigned, and this works fine. More than that
> and they crash eventually. Does that answer your question? Or were
> you after something very specific to the 1GB domU case?

No no. I just was too lazy to compute what a800000 came out in
decimal.
> 
> >>explain what is actually going wrong and why the crash is still
> >>occuring - unless some other piece of hardware is having it's domU
> >>IOMEM mapped somewhere in the range f3df4000-fec8b000 and that is
> >>causing a memory overwrite.
> >>
> >>I am just not seeing any obvious memory stomp at the moment...
> >
> >Neither am I.
> 
> I may have pasted the wrong domU e820. I have a sneaky suspicion
> that this above map was from a domU with 2688MB of RAM assigned,
> hence why there is on domU RAM in the map above a7800000. I'll
> re-check when I'm in front of that machine again.
> 
> Are you OK with the plan to _only_ copy the holes from host E820
> to the hvmloader E820? I think this would be sufficient and not
> cause any undue problems. The only things that would need to
> change are:
> 1) Enlarge the domU hole
> 2) Do something with the top reserved block, starting at
> RESERVED_MEMBASE=0xFC000000. What is this actually for? It
> overlaps with the host memory hole which extends all the way up
> to 0xfee00000. If it must be where it is, this could be
> problematic. What to do in this case?

I would do a git log or git annotate to find it. I recall
some patches to move that - but I can't recall the details.

> 
> This does, also bring up another question - is there any point
> in bothering with matching the host holes? I would hazard a
> guess that no physical hardware is likely to have a memory
> hole bigger than 3GB under the 4GB limit.

3GB is about the max I have seen.

> 
> So would it perhaps be neater, easier, more consistent and
> more debuggable to just make the hvmloader put in a hole
> between 0x40000000-0xffffffff (the whole 3GB) by default?
> Or is that deemed to be too crippling for 32-bit non-PAE
> domUs (and are there enough of these aroudn to matter?)?

Correct. Also it would wreak havoc when migrating to other
hvmloader's which have a different layout.

> 
> Caveat - this alone wouldn't cover any other weirdness such as
> the odd memory hole 0x3f7e0000-0x3f7e7000 on my hardware. Was
> this what you were thinking about when asking whether my domUs
> work OK with 1GB of RAM? Since that is just under the 1GB
> limit.

So there are some issues with i915 IGD having to have a 'flush
page'. Mainly some non-RAM region that they can tell the IGD
to flush its pages. And it had to be non-RAM and somehow
via magic IGD registers you can program the physical address
in the card - so the card has it remapped to itself.

Usually it is some gap (aka hole) that ends has to be
faithfully reproduced in the guest. But you are using
nvidia and are not playing  those nasty tricks.


> 
> To clarify, I am not suggesting just hard coding a 3GB memory
> hole - I am suggesting defaulting to at least that and them
> mapping in any additional memory holes as well. My reasoning
> behind this suggestion is that it would make things more
> consistent between different (possibly dissimilar) hosts.

Potentially. The other option when thinking about migration
and PCI - is to interogate _All_ of the hosts that will be involved
in the migration and construct an E820 that covers all the
right regions. Then use that for the guests and then you
can unplug/plug the PCI devices without much trouble.

That is where the e820_host=1 parameter can be used and
also some extra code to slurp up an XML of the E820 could be
implemented.

The 3GB HOLE could do it, but what if the host has some
odd layout where the HOLE is above 4GB? Then we are back at
remapping.

I think Stefano had some thoughts about enlaring the HOLE
and it might be good to include him here.
> 
> Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel