Re: [Xen-devel] [Draft D] Xen on ARM vITS Handling

[Vijay Kilari <vijay.kilari@xxxxxxxxx>]

On Fri, Jun 5, 2015 at 3:19 PM, Ian Campbell <ian.campbell@xxxxxxxxxx> wrote:
> On Fri, 2015-06-05 at 11:37 +0530, Vijay Kilari wrote:
>> > ## Virtual LPI interrupt injection
>> >
>> > A physical interrupt which is routed to a guest vCPU has the
>> > `_IRQ_GUEST` flag set in the `irq_desc` status mask. Such interrupts
>> > have an associated instance of `struct irq_guest` which contains the
>> > target `struct domain` pointer and virtual interrupt number.
>> >
>> > In Xen a virtual interrupt (either arising from a physical interrupt
>> > or completely virtual) is ultimately injected to a VCPU using the
>> > `vgic_vcpu_inject_irq` function, or `vgic_vcpu_inject_spi`.
>> >
>> > This mechanism will likely need updating to handle the injection of
>> > virtual LPIs. In particular rather than `GICD_ITARGERRn` or
>> > `GICD_IROUTERn` routing of LPIs is performed via the ITS collections
>> > mechanism. This is discussed below (In _vITS_:_Virtual LPI injection_).
>> >
>>
>> For 4.6 scope, we can inject LPIs always to VCPU0?
>
> If that isn't going to be too upsetting for guest OSes we might be able
> to get away with it. I can't remember if we did this for SPIs in the
> early days also.
>
>
>> > During assignment all LPIs associated with the device will be routed
>> > to the guest (i.e. `route_irq_to_guest` will be called for each LPI in
>> > the `struct its_device.events` array).
>>
>>  Is there any hypercall where in we can assign & route LPI's to the guest?
>
> There's:
>         /*
>          * Dom0 should use these two to announce MMIO resources assigned to
>          * MSI-X capable devices won't (prepare) or may (release) change.
>          */
>         #define PHYSDEVOP_prepare_msix          30
>         #define PHYSDEVOP_release_msix          31
>         struct physdev_pci_device {
>             /* IN */
>             uint16_t seg;
>             uint8_t bus;
>             uint8_t devfn;
>         };
> which sort of is?
>
> But, the paragraph above is proposing to do the routing on _device_
> assignment, e.g. on XEN_DOMCTL_assign_device route all LPIs which we
> have prellocated to the device to the designated domain.
>
>> > ## vITS to pITS mapping
>> >
>> > A physical system may have multiple physical ITSs.
>> >
>> > With the simplified vits command model presented here only a single
>> > `vits` is required.
>> >
>> > In the future a more complex arrangement may be desired. Since the
>> > choice of model is internal to the hypervisor/tools and is
>> > communicated to the guest via firmware tables we are not tied to this
>> > model as an ABI if we decide to change.
>>
>> Do you mean that for 4.6 1:N model is followed? why not
>> 1:1 vITS to pITS mapping?
>
> In the model here there isn't really a mapping from vITS to pITS at all,
> that was one of the simplifications I tried to make, since it avoids all
> the stuff about scheduling etc.
>
>> > Since everything is based upon IPA (guest addresses) a malicious guest
>> > can only reference its own RAM here.
>>
>>    Here device table memory allocated by guest is used to lookup for the 
>> device.
>> Why can't we avoid using the guest memory all together and just only emulate
>> read and write of GITS_BASER0.
>
> Emulate how? All as GITS_BASERn.Valid == 0 for all?

Yes, this avoids guest allocating device table for ITS which
is not used.

>
>> Let Xen manage device information using its_device structure based on
>> MAPD command.
>
> Between C and D I had a version which was like this, and used the
> per-domain device map to lookup the its_device.
>
> The problem with it is that the INT command can be given a DeviceID
> which doesn't really exist (in order to generate completion interrupts
> etc).
>
> But a non-existent DeviceID won't have a struct its device to go with
> it, hence there was no way to translate the Event into an LPI.
>
>> am I missing any spec where in it specifies format of Device table
>> managed by ITS hw which software can read/update directly?
>
> AIUI the software is never supposed to read/update the table directly,
> irrespective of whether it is technically able to. Accessing those bits
> directly would be a bug and have UNPREDICTABLE results. The whole
> GITS_BASERn stuff is just a mechanism to avoid the h/w having to
> provision RAM specifically for the ITS by allowing it to take over
> portions of the real RAM.
>
> I think you are correct that in principal a hardware designer would be
> allowed to throw some RAM into the ITS h/w block itself and not request
> anything via GITS_BASERn, as I say I had an interim version which did
> this (using the per domain device map rather than RAM in the ITS h/w
> block), but this didn't handle the INT on a non-existing/invented device
> case.
>
> I decided to handle that case by moving the data structure into guest
> RAM (using the mechanisms provided to do so), which I considered to have
> the benefit of being reasonably simple, but it's not the only possible
> answer.
>
> As an alternative I suppose we could e.g. insert dummy struct its_device
> entries into the per-domain virtual deviceid mapping table in response
> to `MPAD` requests for guest invented devices. Then we would need a
> mechanism to stop a guest from populating all 2^32 entries in the tree
> (which would use a lot of Xen guest RAM) and ideally we would want to
> avoid allocating dummy struct its_device during emulation.

If dom0 is inventing all the devices in the platform and Xen is preallocating
memory for its_device struct that are added using PHYSDEV_add_device op,
 there is will be no chance of domain allocating 2^32 devices.

So deviceid of any MAPD command for all the guests should be have been
within pre-identified list.

>
> Perhaps each guest could have a small pool of "invented its_devices"
> allocated at start of day (small == say, 4) which could be inserted into
> the device table on MAPD, once they are all used further MAPD commands
> for invented devices would be ignored (this assumes that most guests
> will only want one such device for completion purposes), which also
> solves the problem of filling the tree completely (or does it: a
> sequence of MAPD commands setting and clearing all 2^32 devices might
> cause us to populate the whole tree).

If guest is creating this new device for its completion
INT purpose by sending MAPD command, then Xen can create dummy device.

I think we can limit number of dummy devices that guest can create to
small (2 or 4)
and number of vectors say 32. Xen can mark this devices as dummy and
allow all MAPVI/MAPI commands
on this device, but does not translate to physical ITS commands. Also
multiple guests can create dummy device with the same deviceID for which
Xen cannot translate.

On receiving INT command, Xen will just inject LPI mapped for the (device, ID)
(INT device, ID) to domain. So effectively no physical commands are _not_ sent
to ITS hardware that are related to dummy devices.

- Vijay

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel