[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] kexec -e in PVHVM guests (and in PV).
Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx> writes: > Hey, > > I had on my todo list an patch from Olaf patch that shuffles > the shared_page to be in the 0xFE700000 addr (in the "gap" > with newer QEMU's) which unfortunately did not work when > migrating on 32-bit PVHVM guests on Xen 4.1. > > The commit is 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f > "xen PVonHVM: use E820_Reserved area for shared_info" and it > ended up being reverted. I dusted it off and I think I found > the original bug (and fixed it), but while digging in this > the more I discovered a ton more of issues. > > A bit about the use case - the 'kexec -e' allows one to > restart the Linux kernel without a reboot. It is not a crash kernel > so it is just meant to restart and work, and then restart, etc. > > The 'kdump -c' (crash) is a different use case and I had not > thought much about it. But I think that all of the solutions > I am thinking of will make it also work. (so you could > do kexec-crash -> kexec-e->kexec-e>kexec-crash->kexec-e, and > so, if you would want to). > > The problem I uncovered was that the memory region where > the new kernel would be executed had bits of memory changed - which > meant that the purgatory code in kexec would detect the SHA1SUM > being incorrect and not load. That lead me to find out that > VCPUOP_register_vcpu_info was the culprit (well, the xen_vcpu_info > was being modified, and its PFN was in the 'new' kernel image area). > > Anyhow, the end result of that is that I think to get this > working we would need to have: > > 1). A symmetrical VCPUOP_register_vcpu_info call, say > VCPUOP_unregister_vcpu_info, which would for a provided vpuid > set 'vcpu_info' to the shared_info, and 'vcpu_info_mfn' to > INVALID_MFN. Naturally the vcpu_id has to be down (_VPF_down). > A prototype patch along with an naive implementation in > the Linux kernel made this work surprisingly well! > > The Linux kernel had to call on the shutdown the: > disable_nonboot_cpus() which would bring all the AP CPUs down. > Each AP CPU would call said hypercall. Also on each CPU > bringup we would call this (that is the BSP would make this > call before bringing the AP CPUs up - on bootup paths it > would result in nothing, while for an kexec -c type kernel > it would allow us to use the CPUs). > > 2). Ditto for VCPUOP_register_runtime and > VCPUOP_register_runstate_memory_area. They would need a > similar 'unregister' call with similar semantics as the > one above. > > 3). The shared_info. Olaf's patch stuck the shared_info in the > "gaps" of the E820 or the E820_RSRV region. But the recent patches > for PCI passthrough are making me twitchy and I think we would > need to parse the E820 and /proc/ioports (so 'resource API in > Linux kernel' to figure out a good place to stash this. Or on > shutdown (kexec -e) balloon out the shared region (need to > double check that this possible in the first place). > > 4). Balloon memory. I am not really sure how to deal with that. The > guest might have ballooned out tons of memory but the new kernel > won't know about it until the xen/balloon driver kicks in and > figures this out based on XenStore. Then it will try to balloon > out.. and depending on its luck - balloon out memory that was > already ballooned out, or not. Also during the bootup of > the 'kexec -e' kernel it might touch pages that had been > ballooned out - and try to use them! > > 5). Events. Olaf had written code long time ago that would poke the > events to see if they were already in use (-EEXIST) and if so > re-use them - it works great albeit there are tons of messages > in the Xen ring buffer. The Linux patch I wrote did an > 'disable_nonboot_cpus' and also tore down the BSP interrupts - that > meant that all of the events were nicely torn down. This all works > on non-FIFO event. David Vrabel says that to make this work > (re-use or teardown and bring up) would be hard. > > 6). QEMU PnP typ devices. Such as 'serial,'i8042', and 'rtc' end up > going through the EVTCHNOP_bind_pirg. Somehow on the 'kexec -e' > kernel we end up doing OK, but the devices don't work anymore. > That is - the serial input does not accept any more input (but > it can output alright). > > 7). Grants. Andrew Cooper hinted at this and a bit of experimentation > shows that Xen hypervisor will indeed smack down any guest that > tries to re-use its "old" grants. I am not even sure if the > GNTTAB_setup call is returning the "old" grant frames. > His suggestion was 'GNTTAB_reset' to well, reset everything. > > My thinking is that a lot of this code is shared with PV (and PVH) > once this is fixed we could do full scale 'kexec -e' in an PV > (or PVH) type guest. Doing dom0 kexec -e would be an interesting > experiment :-( > > I am unable to fix this for Xen 4.5 and I am not sure what other > issues there are present. If folks have some ideas or would like to > chime in (or even pick some of these up!)- please do respond. > I have one more issue related to kexec/kdump topic I'm investigating right now. When kdump happens and new kernel boots we have /proc/vmcore device. There is no problem in reading from this device, however makedumpfile reads it with mmap() by default and that doesn't work for me. I figured out the following: there are several pages (2 in my case) in vmcore which are not ram. read_from_oldmem() calls special pfn_is_ram() check (which does HVMOP_get_mem_type and these pages are reported as HVMMEM_mmio_dm) and skips them. mmap_vmcore() doesn't have this check and we got these pages mapped. When we do memcpy() from them we get stuck in case we try reading more than 16 bytes (that's weird). I have 'quick and dirty' patch which brings pfn_is_ram() check to mmap_vmcore() and replaces all HVMMEM_mmio_dm pages with an empty page. I'm going to investigate a bit more here. I can try looking at something from the above as well. E.g. I was able to solve no.6 with the following (yes, dirty again) patch: commit 23a224c4ad664dfc6fe672f74f83549387efebda Author: Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> Date: Wed Jun 18 14:12:15 2014 +0200 wip: unmap all pirqs Signed-off-by: Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c index dfa12a4..16af7e4 100644 --- a/drivers/xen/events/events_base.c +++ b/drivers/xen/events/events_base.c @@ -1658,6 +1719,35 @@ void xen_callback_vector(void) {} static bool fifo_events = true; module_param(fifo_events, bool, 0); +static void unmap_all_pirqs(void) +{ + struct evtchn_status status; + int port, rc = -ENOENT; + struct physdev_unmap_pirq unmap_irq; + struct evtchn_close close; + + memset(&status, 0, sizeof(status)); + for (port = 0; port < xen_evtchn_max_channels(); port++) { + status.dom = DOMID_SELF; + status.port = port; + rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status); + if (rc < 0) + continue; + pr_warn("unmap_all_pirqs: port: %d, status: %d\n", status.port, status.status); + if (status.status == EVTCHNSTAT_pirq) { + close.port = port; + if (HYPERVISOR_event_channel_op(EVTCHNOP_close, &close) != 0) + pr_warn("EVTCHNSTAT_pirq: failed to close event channel %d\n", port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid = DOMID_SELF; + pr_warn("unmapping previously mapped pirq %d\n", unmap_irq.pirq); + if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq) != 0) + pr_warn("failed to unmap pirq %d\n", unmap_irq.pirq); + } + } +} + + void __init xen_init_IRQ(void) { int ret = -EINVAL; @@ -1686,6 +1776,8 @@ void __init xen_init_IRQ(void) xen_callback_vector(); if (xen_hvm_domain()) { + unmap_all_pirqs(); + native_init_IRQ(); /* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ -- Vitaly _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |