Re: [Xen-devel] [PATCH v7 00/34] arm64: Dom0 ITS emulation

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

Thanks Andre, I have applied patches 1-9.

On Fri, 7 Apr 2017, Andre Przywara wrote:
> Hi,
> 
> an only slightly modified repost of the last version.
> I added the Reviewed-by: and Acked-by: tags from Stefano and Julien
> and rebased on top of the latest staging tree:
> commit 89216c7999eb5b8558bfac7d61ae0d5ab844ce3f
> Author: Dario Faggioli <dario.faggioli@xxxxxxxxxx>
> Date:   Fri Apr 7 18:57:14 2017 +0200
> 
>     xen: credit1: treat pCPUs more evenly during balancing.
> 
> Other than that and one typo and comment fix the first 10 patches have
> not been changed.
> I dropped the addition of the GIC_IRQ_GUEST_LPI_PENDING bit in patch 12/36
> and followed Stefano's suggestion, which led to the removal of former
> patches 17/36 and 23/36 and some simplification in later patches.
> I haven't been able to address most review comments from the last part of
> v5 yet, but will definitely still fix them.
> Detailed changelog below.
> 
> Cheers,
> Andre
> 
> ----------------------------------
> This series adds support for emulation of an ARM GICv3 ITS interrupt
> controller. For hardware which relies on the ITS to provide interrupts for
> its peripherals this code is needed to get a machine booted into Dom0 at
> all. ITS emulation for DomUs is only really useful with PCI passthrough,
> which is not yet available for ARM. It is expected that this feature
> will be co-developed with the ITS DomU code. However this code drop here
> considered DomU emulation already, to keep later architectural changes
> to a minimum.
> 
> This is technical preview version to allow early testing of the feature.
> Things not (properly) addressed in this release:
> - The MOVALL command is not emulated. In our case there is really nothing
> to do here. We might need to revisit this in the future for DomU support.
> - The INVALL command might need some rework to be more efficient. Currently
> we iterate over all mapped LPIs, which might take a bit longer.
> - Indirect tables are not supported. This affects both the host and the
> virtual side.
> - The command queue locking is currently suboptimal and should be made more
> fine-grained in the future, if possible.
> - We need to properly investigate the possible interaction when devices get
> removed. This requires to properly clean up and remove any associated
> resources like pending or in-flight LPIs, for instance.
> 
> 
> Some generic design principles:
> 
> * The current GIC code statically allocates structures for each supported
> IRQ (both for the host and the guest), which due to the potentially
> millions of LPI interrupts is not feasible to copy for the ITS.
> So we refrain from introducing the ITS as a first class Xen interrupt
> controller, also we don't hold struct irq_desc's or struct pending_irq's
> for each possible LPI.
> Fortunately LPIs are only interesting to guests, so we get away with
> storing only the virtual IRQ number and the guest VCPU for each allocated
> host LPI, which can be stashed into one uint64_t. This data is stored in
> a two-level table, which is both memory efficient and quick to access.
> We hook into the existing IRQ handling and VGIC code to avoid accessing
> the normal structures, providing alternative methods for getting the
> needed information (priority, is enabled?) for LPIs.
> Whenever a guest maps a device, we allocate the maximum required number
> of struct pending_irq's, so that any triggering LPI can find its data
> structure. Upon the guest actually mapping the LPI, this pointer to the
> corresponding pending_irq gets entered into a radix tree, so that it can
> be quickly looked up.
> 
> * On the guest side we (later will) have to deal with malicious guests
> trying to hog Xen with mapping requests for a lot of LPIs, for instance.
> As the ITS actually uses system memory for storing status information,
> we use this memory (which the guest has to provide) to naturally limit
> a guest. Whenever we need information from any of the ITS tables, we
> temporarily map them (which is cheap on arm64) and copy the required data.
> * An obvious approach to handling some guest ITS commands would be to
> propagate them to the host, for instance to map devices and LPIs and
> to enable or disable LPIs.
> However this (later with DomU support) will create an attack vector, as
> a malicious guest could try to fill the host command queue with
> propagated commands.
> So we try to avoid this situation: Dom0 sending a device mapping (MAPD)
> command is the only time we allow queuing commands to the host ITS command
> queue, as this seems to be the only reliable way of getting the
> required information at the moment. However at the same time we map all
> events to LPIs already, also enable them. This avoids sending commands
> later at runtime, as we can deal with mappings and LPI enabling/disabling
> internally.
> 
> To accomodate the tech preview nature of this feature at the moment, there
> is a Kconfig option to enable it. Also it is supported on arm64 only, which
> will most likely not change in the future.
> This leads to some hideous constructs like an #ifdef'ed header file with
> empty function stubs to accomodate arm32 and non-ITS builds, which share
> some generic code paths with the ITS emulation.
> The number of supported LPIs can be limited on the command line, in case
> the number reported by the hardware is too high. As Xen cannot foresee how
> many interrupts the guests will need, we cater for as many as possible.
> The command line parameter is called max-lpi-bits and expresses the number
> of bits required to hold an interrupt ID. It defaults to 20, if that is
> lower than the number supported by the hardware.
> This code boots Dom0 on an ARM Fast Model with ITS support. I tried to
> address the issues seen by people running the previous versions on real
> hardware, though couldn't verify this here for myself.
> So any testing, bug reports (and possibly even fixes) are very welcome.
> 
> The code can also be found on the its/v7 branch here:
> git://linux-arm.org/xen-ap.git
> http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/its/v7
> 
> Cheers,
> Andre
> 
> Changelog v6 ... v7:
> - got rid of GIC_IRQ_GUEST_LPI_PENDING
> - typo and comment fix in 09/36
> - rebased against latest staging
> - added Reviewed-by: and Acked-by: tags
> 
> Changelog v5 ... v6:
> - reordered patches to allow splitting the series
> - introduced functions later to avoid warnings on intermediate builds
> - refactored common code changes into separate patches
> - removed bogus host_its_list from non-ITS build
> - dropped GENMASK_ULL and BIT_ULL (both patches and their usage later)
> - rework locking in MMIO register reads and writes
> - protect new code from being executed without an ITS being configured
> - fix vgic_access_guest_memory (now a separate patch)
> - some more comments and TODOs
> 
> 
> Changelog v4 ... v5:
> - adding many comments
> - spinlock asserts
> - rename r_host_lpis to max_host_lpi_ids
> - remove max_its_device_bits command line
> - add warning on high number of LPIs
> - avoid potential leak on host MAPD
> - properly handle nr_events rounding
> - remove unmap_all_devices(), replace with ASSERT
> - add barriers for (lockless) host LPI lookups
> - add proper locking in ITS and redist MMIO register handling
> - rollback failing device mapping
> - fix various printks
> - add vgic_access_guest_memory() and use it
> - (getting rid of page mapping functions and helpers)
> - drop table mapping / unmapping on redist/ITS enable/disable
> - minor reworks in functions as per review comments
> - fix ITS enablement check
> - move lpi_to_pending() and lpi_get_priority() to vgic_ops
> - move do_LPI() to gic_hw_ops
> - whitespace and hard tabs fixes
> - introduce ITS domain init function (and use it for the rbtree)
> - enable IRQs around do_LPI
> - implement TODOs for later optimizations
> - add "v" prefix to variables holding virtual properties
> - provide locked and normal versions of read/write_itte
> - only CLEAR LPI if not already guest visible (plus comment)
> - update LPI property on MAPTI
> - store vcpu_id in pending_irq for LPIs (helps INVALL)
> - improve INVALL implementation to only cover LPIs on this VCPU
> - improve virtual BASE register initialization
> - limit number of virtual LPIs to 24 bits (Linux bug at 32??)
> - only inject LPIs if redistributor is actually enabled
> 
> Changelog v3 .. v4:
> - make HAS_ITS depend on EXPERT
> - introduce new patch 02 to initialize host ITS early
> - fix cmd_lock init position
> - introduce warning on high number of LPI allocations
> - various int -> unsigned fixes
> - adding and improving comments
> - rate limit ITS command queue full msg
> - drop unneeded checks
> - validate against allowed number of device IDs
> - avoid memory leaks when removing devices
> - improve algorithm for finding free host LPI
> - convert unmap_all_devices from goto to while loop
> - add message on remapping ITS device
> - name virtual device / event IDs properly
> - use atomic read when reading ITT entry
> 
> Changelog v2 .. v3:
> - preallocate struct pending_irq's
> - map ITS and redistributor tables only on demand
> - store property, enable and pending bit in struct pending_irq
> - improve error checking and handling
> - add comments
> 
> Changelog v1 .. v2:
> - clean up header file inclusion
> - rework host ITS table allocation: observe attributes, many fixes
> - remove patch 1 to export __flush_dcache_area, use existing function instead
> - use number of LPIs internally instead of number of bits
> - keep host_its_list as private as possible
> - keep struct its_devices private
> - rework gicv3_its_map_guest_devices
> - fix rbtree issues
> - more error handling and propagation
> - cope with GICv4 implementations (but no virtual LPI features!)
> - abstract host and guest ITSes by using doorbell addresses
> - join per-redistributor variables into one per-CPU structure
> - fix data types (unsigned int)
> - many minor bug fixes
> 
> (Rough) changelog RFC-v2 .. v1:
> - split host ITS driver into gic-v3-lpi.c and gic-v3-its.c part
> - rename virtual ITS driver file to vgic-v3-its.c
> - use macros and named constants for all magic numbers
> - use atomic accessors for accessing the host LPI data
> - remove leftovers from connecting virtual and host ITSes
> - bail out if host ITS is disabled in the DT
> - rework map/unmap_guest_pages():
>     - split off p2m part as get/put_guest_pages (to be done on allocation)
>     - get rid of vmap, using map_domain_page() instead
> - delay allocation of virtual tables until actual LPI/ITS enablement
> - properly size both virtual and physical tables upon allocation
> - fix put_domain() locking issues in physdev_op and LPI handling code
> - add and extend comments in various areas
> - fix lotsa coding style and white space issues, including comment style
> - add locking to data structures not yet covered
> - fix various locking issues
> - use an rbtree to deal with ITS devices (instead of a list)
> - properly handle memory attributes for ITS tables
> - handle cacheable/non-cacheable ITS table mappings
> - sanitize guest provided ITS/LPI table attributes
> - fix breakage on non-GICv2 compatible host GICv3 controllers
> - add command line parameters on top of Kconfig options
> - properly wait for an ITS to become quiescient before enabling it
> - handle host ITS command queue errors
> - actually wait for host ITS command completion (READR==WRITER)
> - fix ARM32 compilation
> - various patch splits and reorderings
> 
> 
> Andre Przywara (33):
>   ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT
>   ARM: GICv3 ITS: initialize host ITS
>   ARM: GICv3: allocate LPI pending and property table
>   ARM: GICv3 ITS: allocate device and collection table
>   ARM: GICv3 ITS: map ITS command buffer
>   ARM: GICv3 ITS: introduce ITS command handling
>   ARM: GICv3 ITS: introduce host LPI array
>   ARM: vGICv3: introduce ITS emulation stub
>   ARM: GICv3 ITS: introduce device mapping
>   ARM: GIC: Add checks for NULL pointer pending_irq's
>   ARM: GICv3: introduce separate pending_irq structs for LPIs
>   ARM: GICv3: forward pending LPIs to guests
>   ARM: GICv3: enable ITS and LPIs on the host
>   ARM: vGICv3: handle virtual LPI pending and property tables
>   ARM: vGICv3: re-use vgic_reg64_check_access
>   ARM: GIC: export vgic_init_pending_irq()
>   ARM: VGIC: add vcpu_id to struct pending_irq
>   ARM: vGICv3: add virtual ITS list head and comment about iteration
>   ARM: GICv3: prepare for virtual ITS subnodes
>   ARM: vGIC: advertise LPI support
>   ARM: vITS: add command handling stub and MMIO emulation
>   ARM: vITS: introduce translation table walks
>   ARM: vITS: handle CLEAR command
>   ARM: vITS: handle INT command
>   ARM: vITS: handle MAPC command
>   ARM: vITS: handle MAPD command
>   ARM: vITS: handle MAPTI command
>   ARM: vITS: handle MOVI command
>   ARM: vITS: handle DISCARD command
>   ARM: vITS: handle INV command
>   ARM: vITS: handle INVALL command
>   ARM: vITS: create and initialize virtual ITSes for Dom0
>   ARM: vITS: create ITS subnodes for Dom0 DT
> 
> Vijaya Kumar K (1):
>   ARM: introduce vgic_access_guest_memory()
> 
>  docs/misc/xen-command-line.markdown |    9 +
>  xen/arch/arm/Kconfig                |    5 +
>  xen/arch/arm/Makefile               |    3 +
>  xen/arch/arm/gic-v2.c               |    7 +
>  xen/arch/arm/gic-v3-its.c           | 1020 +++++++++++++++++++++++++++
>  xen/arch/arm/gic-v3-lpi.c           |  584 +++++++++++++++
>  xen/arch/arm/gic-v3.c               |   81 ++-
>  xen/arch/arm/gic.c                  |   31 +-
>  xen/arch/arm/vgic-v2.c              |   15 +
>  xen/arch/arm/vgic-v3-its.c          | 1325 
> +++++++++++++++++++++++++++++++++++
>  xen/arch/arm/vgic-v3.c              |  274 +++++++-
>  xen/arch/arm/vgic.c                 |   62 +-
>  xen/include/asm-arm/domain.h        |   14 +-
>  xen/include/asm-arm/gic.h           |    2 +
>  xen/include/asm-arm/gic_v3_defs.h   |   54 ++
>  xen/include/asm-arm/gic_v3_its.h    |  263 +++++++
>  xen/include/asm-arm/irq.h           |   16 +
>  xen/include/asm-arm/vgic-emul.h     |    9 +
>  xen/include/asm-arm/vgic.h          |   11 +
>  19 files changed, 3748 insertions(+), 37 deletions(-)
>  create mode 100644 xen/arch/arm/gic-v3-its.c
>  create mode 100644 xen/arch/arm/gic-v3-lpi.c
>  create mode 100644 xen/arch/arm/vgic-v3-its.c
>  create mode 100644 xen/include/asm-arm/gic_v3_its.h
> 
> -- 
> 2.8.2
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel