Re: [Xen-devel] [PATCH v2] x86/hvmloader: round up memory BAR size to 4K

[Jason Andryuk <jandryuk@xxxxxxxxx>]

On Wed, Jan 22, 2020 at 5:52 AM Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>
> On Wed, Jan 22, 2020 at 11:27:24AM +0100, Jan Beulich wrote:
> > On 21.01.2020 17:57, Roger Pau Monné wrote:
> > > On Tue, Jan 21, 2020 at 05:15:20PM +0100, Jan Beulich wrote:
> > >> On 21.01.2020 16:57, Roger Pau Monné wrote:
> > >>> On Tue, Jan 21, 2020 at 11:43:58AM +0100, Jan Beulich wrote:
> > >>>> On 21.01.2020 11:29, Roger Pau Monné wrote:
> > >>>>> So I'm not sure how to progress with this patch, are we fine with
> > >>>>> those limitations?
> > >>>>
> > >>>> I'm afraid this depends on ...
> > >>>>
> > >>>>> As I said, Xen hasn't got enough knowledge to correctly isolate the
> > >>>>> BARs, and hence we have to rely on dom0 DTRT. We could add checks in
> > >>>>> Xen to make sure no BARs share a page, but it's a non-trivial amount
> > >>>>> of scanning and sizing each possible BAR on the system.
> > >>>>
> > >>>> ... whether Dom0 actually "DTRT", which in turn is complicated by there
> > >>>> not being a specific Dom0 kernel incarnation to check against. Perhaps
> > >>>> rather than having Xen check _all_ BARs, Xen or the tool stack could
> > >>>> check BARs of devices about to be handed to a guest? Perhaps we need to
> > >>>> pass auxiliary information to hvmloader to be able to judge whether a
> > >>>> BAR shares a page with another one? Perhaps there also needs to be a
> > >>>> way for hvmloader to know what offset into a page has to be maintained
> > >>>> for any particular BAR, as follows from Jason's recent reply?
> > >>>
> > >>> Linux has an option to force resource alignment (as reported by
> > >>> Jason), maybe we could force all BARs to be aligned to page size in
> > >>> order to be passed through?
> > >>>
> > >>> That would make it easier to check (as Xen/Qemu would only need to
> > >>> assert that the BAR address is aligned), and won't require much extra
> > >>> work in Xen apart from the check itself.
> > >>>
> > >>> Do you think this would be an acceptable solution?
> > >>
> > >> In principle yes, but there are loose ends:
> > >> - What do you mean by "we could force"? We have no control over the
> > >>   Dom0 kernel.
> > >
> > > I should rephrase:
> > >
> > > ... maybe we should require dom0 to align all memory BARs to page size
> > > in order to be passed through?
> > >
> > > Ie: Xen should refuse to pass through any memory BAR that's not page
> > > aligned. How the alignment is accomplished is out of the scope to Xen,
> > > as long as memory BARs are aligned.
> >
> > That's an acceptable model, as long as it wouldn't typically break
> > existing configurations, and as long as for those who we would
> > break there are easy to follow steps to unbreak their setups.
>
> Jason, do you think you could take a stab at adding a check in order
> to make sure memory BAR addresses are 4K aligned when assigning a
> device to a guest?

I can take a look.  You want the hypervisor to make the enforcement
and not the toolstack?

Waving my hands a little bit, but it might be possible to have `xl
pci-assignable-add` trigger the linux pci resource_alignment at
runtime.

It may also be possible for the Linux kernel, when running as the
initial domain, to set the minimum alignment for the PCI bus.

> > >> - What about non-Linux Dom0?
> > >
> > > Other OSes would have to provide similar functionality in order to
> > > align the memory BARs. Right now Linux is the only dom0 that supports
> > > PCI passthrough AFAIK.
> > >
> > >> Also, apart from extra resource (address space) consumption,
> > >
> > > The PCI spec actually recommends memory BARs to be at least of page
> > > size, but that's not a strict requirement. I would hope there aren't
> > > that many devices with memory BARs smaller than a page.
> >
> > I've simply gone and grep-ed all the stored lspci output I have
> > for some of the test systems I have here:
> >
> > 0/12
> > 3/31 (all 4k-aligned)
> > 6/13 (all 4k-aligned)
> > 3/12
> > 6/19 (all 4k-aligned)
> > 3/7 (all 4k-aligned)
>
> What does X/Y at the beginning of the line stand for?

I think it's BARs smaller than 4k out of total BARs.

Ivy Bridge HP laptop: 7/15 (all 4k aligned)
Sandy Bridge Dell desktop: 5/13 (all 4k-aligned)
Kaby Lake Dell laptop: 2/18 (all 4k-aligned)

> > This is without regard to what specific devices these are, and
> > hence whether there would be any point in wanting to pass it to
> > a guest in the first place. I'd like to note though that there
> > are a fair amount of USB controllers among the ones with BARs
> > smaller than a page's worth.

The Intel EHCI USB controllers on my Sandy Bridge and Ivy Bridge
systems have 1K BARs.  USB controllers for their USB vm  may have
motivated Qubes's original work with handling small BARs.

Regards,
Jason

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel