[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches

On 09/07/17 09:03, Kai Huang wrote:
Hi all,

This series is RFC Xen SGX virtualization support design and RFC draft patches.

Thankyou very much for this design doc.

2. SGX Virtualization Design

2.1 High Level Toolstack Changes:

2.1.1 New 'epc' parameter

EPC is limited resource. In order to use EPC efficiently among all domains,
when creating guest, administrator should be able to specify domain's virtual
EPC size. And admin
alao should be able to get all domain's virtual EPC size.

For this purpose, a new 'epc = <size>' parameter is added to XL configuration
file. This parameter specifies guest's virtual EPC size. The EPC base address
will be calculated by toolstack internally, according to guest's memory size,
MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted.

How will this interact with multi-package servers? Even though its fine to implement the single-package support first, the design should be extensible to the multi-package case.

First of all, what are the implications of multi-package SGX?

(Somewhere) you mention changes to scheduling. I presume this is because a guest with EPC mappings in EPT must be scheduled on the same package, or ENCLU[EENTER] will fail. I presume also that each package will have separate, unrelated private keys?

I presume there is no sensible way (even on native) for a single logical process to use multiple different enclaves? By extension, does it make sense to try and offer parts of multiple enclaves to a single VM?

2.1.3 Notify domain's virtual EPC base and size to Xen

Xen needs to know guest's EPC base and size in order to populate EPC pages for
it. Toolstack notifies EPC base and size to Xen via XEN_DOMCTL_set_cpuid.

I am currently in the process of reworking the Xen/Toolstack interface when it comes to CPUID handling. The latest design is available here: https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg00378.html but the end result will be the toolstack expressing its CPUID policy in terms of the architectural layout.

Therefore, I would expect that, however the setting is represented in the configuration file, xl/libxl would configure it with the hypervisor by setting CPUID.0x12[2] with the appropriate base and size.

2.1.4 Launch Control Support (?)

Xen Launch Control Support is about to support running multiple domains with
each running its own LE signed by different owners (if HW allows, explained
below). As explained in 1.4 SGX Launch Control, EINIT for LE (Launch Enclave)
only succeeds when SHA256(SIGSTRUCT.modulus) matches IA32_SGXLEPUBKEYHASHn,
and EINIT for other enclaves will derive EINITTOKEN key according to
IA32_SGXLEPUBKEYHASHn. Therefore, to support this, guest's virtual
IA32_SGXLEPUBKEYHASHn must be updated to phyiscal MSRs before EINIT (which
also means the physical IA32_SGXLEPUBKEYHASHn need to be *unlocked* in BIOS
before booting to OS).

For physical machine, it is BIOS's writer's decision that whether BIOS would
provide interface for user to specify customerized IA32_SGXLEPUBKEYHASHn (it
is default to digest of Intel's signing key after reset). In reality, OS's SGX
driver may require BIOS to make MSRs *unlocked* and actively write the hash
value to MSRs in order to run EINIT successfully, as in this case, the driver
will not depend on BIOS's capability (whether it allows user to customerize

The problem is for Xen, do we need a new parameter, such as 'lehash=<SHA256>'
to specify the default value of guset's virtual IA32_SGXLEPUBKEYHASHn? And do
we need a new parameter, such as 'lewr' to specify whether guest's virtual MSRs
are locked or not before handling to guest's OS?

I tends to not introduce 'lehash', as it seems SGX driver would actively update
the MSRs. And new parameter would add additional changes for upper layer
software (such as openstack). And 'lewr' is not needed either as Xen can always
*unlock* the MSRs to guest.

Please give comments?

Currently in my RFC patches above two parameters are not implemented.
Xen hypervisor will always *unlock* the MSRs. Whether there is 'lehash'
parameter or not doesn't impact Xen hypervisor's emulation of
IA32_SGXLEPUBKEYHASHn. See below Xen hypervisor changes for details.

Reading around, am I correct with the following?

1) Some processors have no launch control. There is no restriction on which enclaves can boot.

2) Some Skylake client processors claim to have launch control, but the MSRs are unavailable (is this an erratum?). These are limited to booting enclaves matching the Intel public key.

3) Launch control may be locked by the BIOS. There may be a custom hash, or it might be the Intel default. Xen can't adjust it at all, but can support running any number of VMs with matching enclaves.

4) Launch control may be unlocked by the BIOS. In this case, Xen can context switch a hash per domain, and run all enclaves.

The eventual plans for CPUID and MSR levelling should allow all of these to be expressed in sensible ways, and I don't forsee any issues with supporting all of these scenarios.

2.2 High Level Xen Hypervisor Changes:

2.2.1 EPC Management (?)

Xen hypervisor needs to detect SGX, discover EPC, and manage EPC before
supporting SGX to guest. EPC is detected via SGX CPUID 0x12.0x2. It's possible
that there are multiple EPC sections (enumerated via sub-leaves 0x3 and so on,
until invaid EPC is reported), but this is only true on multiple-socket server
machines. For server machines there are additional things also needs to be done,
such as NUMA EPC, scheduling, etc. We will support server machine in the future
but currently we only support one EPC.

EPC is reported as reserved memory (so it is not reported as normal memory).
EPC must be managed in 4K pages. CPU hardware uses EPCM to track status of each
EPC pages. Xen needs to manage EPC and provide functions to, ie, alloc and free
EPC pages for guest.

There are two ways to manage EPC: Manage EPC separately; or Integrate it to
existing memory management framework.

It is easy to manage EPC separately, as currently EPC is pretty small (~100MB),
and we can even put them in a single list. However it is not flexible, for
example, you will have to write new algorithms when EPC becomes larger, ex, GB.
And you have to write new code to support NUMA EPC (although this will not come
in short time).

Integrating EPC to existing memory management framework seems more reasonable,
as in this way we can resume memory management data structures/algorithms, and
it will be more flexible to support larger EPC and potentially NUMA EPC. But
modifying MM framework has a higher risk to break existing memory management
code (potentially more bugs).

In my RFC patches currently we choose to manage EPC separately. A new
structure epc_page is added to represent a single 4K EPC page. A whole array
of struct epc_page will be allocated during EPC initialization, so that given
the other, one of PFN of EPC page and 'struct epc_page' can be got by adding

But maybe integrating EPC to MM framework is more reasonable. Comments?

2.2.2 EPC Virtualization (?)

It looks like managing the EPC is very similar to managing the NVDIMM ranges. We have a (set of) physical address ranges which need 4k ownership granularity to different domains.

I think integrating this into struct page_struct is the better way to go.

This part is how to populate EPC for guests. We have 3 choices:
     - Static Partitioning
     - Oversubscription
     - Ballooning

Static Partitioning means all EPC pages will be allocated and mapped to guest
when it is created, and there's no runtime change of page table mappings for EPC
pages. Oversubscription means Xen hypervisor supports EPC page swapping between
domains, meaning Xen is able to evict EPC page from another domain and assign it
to the domain that needs the EPC. With oversubscription, EPC can be assigned to
domain on demand, when EPT violation happens. Ballooning is similar to memory
ballooning. It is basically "Static Partitioning" + "Balloon driver" in guest.

Static Partitioning is the easiest way in terms of implementation, and there
will be no hypervisor overhead (except EPT overhead of course), because in
"Static partitioning", there is no EPT violation for EPC, and Xen doesn't need
to turn on ENCLS VMEXIT for guest as ENCLS runs perfectly in non-root mode.

Ballooning is "Static Partitioning" + "Balloon driver" in guest. Like "Static
Paratitioning", ballooning doesn't need to turn on ENCLS VMEXIT, and doesn't
have EPT violation for EPC either. To support ballooning, we need ballooning
driver in guest to issue hypercall to give up or reclaim EPC pages. In terms of
hypercall, we have two choices: 1) Add new hypercall for EPC ballooning; 2)
Using existing XENMEM_{increase/decrease}_reservation with new memory flag, ie,
XENMEMF_epc. I'll discuss more regarding to adding dedicated hypercall or not

Oversubscription looks nice but it requires more complicated implemetation.
Firstly, as explained in 1.3.3 EPC Eviction & Reload, we need to follow specific
steps to evict EPC pages, and in order to do that, basically Xen needs to trap
ENCLS from guest and keep track of EPC page status and enclave info from all
guest. This is because:
     - To evict regular EPC page, Xen needs to know SECS location
     - Xen needs to know EPC page type: evicting regular EPC and evicting SECS,
       VA page have different steps.
     - Xen needs to know EPC page status: whether the page is blocked or not.

Those info can only be got by trapping ENCLS from guest, and parsing its
parameters (to identify SECS page, etc). Parsing ENCLS parameters means we need
to know which ENCLS leaf is being trapped, and we need to translate guest's
virtual address to get physical address in order to locate EPC page. And once
ENCLS is trapped, we have to emulate ENCLS in Xen, which means we need to
reconstruct ENCLS parameters by remapping all guest's virtual address to Xen's
virtual address (gva->gpa->pa->xen_va), as ENCLS always use *effective address*
which is able to be traslated by processor when running ENCLS.

                 |   ENCLS   |
                 |          /|\
     ENCLS VMEXIT|           | VMENTRY
                 |           |
                \|/          |

                1) parse ENCLS parameters
                2) reconstruct(remap) guest's ENCLS parameters
                3) run ENCLS on behalf of guest (and skip ENCLS)
                4) on success, update EPC/enclave info, or inject error

And Xen needs to maintain each EPC page's status (type, blocked or not, in
enclave or not, etc). Xen also needs to maintain all Enclave's info from all
guests, in order to find the correct SECS for regular EPC page, and enclave's
linear address as well.

So in general, "Static Partitioning" has simplest implementation, but obviously
not the best way to use EPC efficiently; "Ballooning" has all pros of Static
Partitioning but requies guest balloon driver; "Oversubscription" is best in
terms of flexibility but requires complicated hypervisor implemetation.

We have implemented "Static Partitioning" in RFC patches, but needs your
feedback on whether it is enough. If not, which one should we do at next stage
-- Ballooning or Oversubscription. IMO Ballooning may be good enough, given fact
that currently memory is also "Static Partitioning" + "Ballooning".


Definitely go for static partitioning to begin with. This is far simpler to implement.

I can't see a pressing usecase for oversubscription or ballooning. Any datacenter work will be using exclusively static, and I expect static will fine for all (or at least, most) client usecases.

2.2.3 Populate EPC for Guest

Toolstack notifies Xen about domain's EPC base and size by XEN_DOMCTL_set_cpuid,
so currently Xen populates all EPC pages for guest in XEN_DOMCTL_set_cpuid,
particularly, in handling XEN_DOMCTL_set_cpuid for CPUID.0x12.0x2. Once Xen
checks the values passed from toolstack is valid, Xen will allocate all EPC
pages and setup EPT mappings for guest.

2.2.4 New Dedicated Hypercall (?)

All this information should (eventually) be available via the appropriate SYSCTL_get_{cpuid,msr}_policy hypercalls. I don't see any need for dedicated hypercalls.

2.2.9 Guest Suspend & Resume

On hardware, EPC is destroyed when power goes to S3-S5. So Xen will destroy
guest's EPC when guest's power goes into S3-S5. Currently Xen is notified by
Qemu in terms of S State change via HVM_PARAM_ACPI_S_STATE, where Xen will
destroy EPC if S State is S3-S5.

Specifically, Xen will run EREMOVE for guest's each EPC page, as guest may
not handle EPC suspend & resume correctly, in which case physically guest's EPC
pages may still be valid, so Xen needs to run EREMOVE to make sure all EPC
pages are becoming invalid. Otherwise further operation in guest on EPC may
fault as it assumes all EPC pages are invalid after guest is resumed.

For SECS page, EREMOVE may fault with SGX_CHILD_PRESENT, in which case Xen will
keep this SECS page into a list, and call EREMOVE for them again after all EPC
pages have been called with EREMOVE. This time the EREMOVE on SECS will succeed
as all children (regular EPC pages) have already been removed.

2.2.10 Destroying Domain

Normally Xen just frees all EPC pages for domain when it is destroyed. But Xen
will also do EREMOVE on all guest's EPC pages (described in above 2.2.7) before
free them, as guest may shutdown unexpected (ex, user kills guest), and in this
case, guest's EPC may still be valid.

2.3 Additional Point: Live Migration, Snapshot Support (?)

How big is the EPC? If we are talking MB rather than GB, movement of the EPC could be after the pause, which would add some latency to live migration but should work. I expect that people would prefer to have the flexibility of migration even at the cost of extra latency.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.