[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time
On 06/11/2015 04:32 PM, Paul Durrant wrote: >> -----Original Message----- >> From: Wen Congyang [mailto:wency@xxxxxxxxxxxxxx] >> Sent: 11 June 2015 02:14 >> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-devel@xxxxxxxxxxxxx >> Cc: Wei Liu; Ian Campbell; guijianfeng@xxxxxxxxxxxxxx; >> yunhong.jiang@xxxxxxxxx; Eddie Dong; rshriram@xxxxxxxxx; Ian Jackson >> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq >> page only one time >> >> On 06/10/2015 07:47 PM, Paul Durrant wrote: >>>> -----Original Message----- >>>> From: xen-devel-bounces@xxxxxxxxxxxxx [mailto:xen-devel- >>>> bounces@xxxxxxxxxxxxx] On Behalf Of Wen Congyang >>>> Sent: 10 June 2015 12:38 >>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen- >> devel@xxxxxxxxxxxxx >>>> Cc: Wei Liu; Ian Campbell; guijianfeng@xxxxxxxxxxxxxx; >>>> yunhong.jiang@xxxxxxxxx; Eddie Dong; rshriram@xxxxxxxxx; Ian Jackson >>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero >> ioreq >>>> page only one time >>>> >>>> On 06/10/2015 06:58 PM, Paul Durrant wrote: >>>>>> -----Original Message----- >>>>>> From: Wen Congyang [mailto:wency@xxxxxxxxxxxxxx] >>>>>> Sent: 10 June 2015 11:55 >>>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen- >>>> devel@xxxxxxxxxxxxx >>>>>> Cc: Wei Liu; Ian Campbell; yunhong.jiang@xxxxxxxxx; Eddie Dong; >>>>>> guijianfeng@xxxxxxxxxxxxxx; rshriram@xxxxxxxxx; Ian Jackson >>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero >>>> ioreq >>>>>> page only one time >>>>>> >>>>>> On 06/10/2015 06:40 PM, Paul Durrant wrote: >>>>>>>> -----Original Message----- >>>>>>>> From: Wen Congyang [mailto:wency@xxxxxxxxxxxxxx] >>>>>>>> Sent: 10 June 2015 10:06 >>>>>>>> To: Andrew Cooper; Yang Hongyang; xen-devel@xxxxxxxxxxxxx; Paul >>>>>> Durrant >>>>>>>> Cc: Wei Liu; Ian Campbell; yunhong.jiang@xxxxxxxxx; Eddie Dong; >>>>>>>> guijianfeng@xxxxxxxxxxxxxx; rshriram@xxxxxxxxx; Ian Jackson >>>>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: >> zero >>>>>> ioreq >>>>>>>> page only one time >>>>>>>> >>>>>>>> Cc: Paul Durrant >>>>>>>> >>>>>>>> On 06/10/2015 03:44 PM, Andrew Cooper wrote: >>>>>>>>> On 10/06/2015 06:26, Yang Hongyang wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 06/09/2015 03:30 PM, Andrew Cooper wrote: >>>>>>>>>>> On 09/06/2015 01:59, Yang Hongyang wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 06/08/2015 06:15 PM, Andrew Cooper wrote: >>>>>>>>>>>>> On 08/06/15 10:58, Yang Hongyang wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 06/08/2015 05:46 PM, Andrew Cooper wrote: >>>>>>>>>>>>>>> On 08/06/15 04:43, Yang Hongyang wrote: >>>>>>>>>>>>>>>> ioreq page contains evtchn which will be set when we >>>> resume >>>>>> the >>>>>>>>>>>>>>>> secondary vm the first time. The hypervisor will check if >> the >>>>>>>>>>>>>>>> evtchn is corrupted, so we cannot zero the ioreq page >> more >>>>>>>>>>>>>>>> than one time. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The ioreq->state is always STATE_IOREQ_NONE after the >> vm >>>> is >>>>>>>>>>>>>>>> suspended, so it is OK if we only zero it one time. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Signed-off-by: Yang Hongyang <yanghy@xxxxxxxxxxxxxx> >>>>>>>>>>>>>>>> Signed-off-by: Wen congyang <wency@xxxxxxxxxxxxxx> >>>>>>>>>>>>>>>> CC: Andrew Cooper <andrew.cooper3@xxxxxxxxxx> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The issue here is that we are running the restore algorithm >>>> over >>>>>> a >>>>>>>>>>>>>>> domain which has already been running in Xen for a while. >>>> This >>>>>> is a >>>>>>>>>>>>>>> brand new usecase, as far as I am aware. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Exactly. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Does the qemu process associated with this domain get >>>> frozen >>>>>>>>>>>>>>> while the >>>>>>>>>>>>>>> secondary is being reset, or does the process get destroyed >>>> and >>>>>>>>>>>>>>> recreated. >>>>>>>>>>>>>> >>>>>>>>>>>>>> What do you mean by reset? do you mean secondary is >>>>>> suspended >>>>>>>> at >>>>>>>>>>>>>> checkpoint? >>>>>>>>>>>>> >>>>>>>>>>>>> Well - at the point that the buffered records are being >>>> processed, >>>>>> we >>>>>>>>>>>>> are in the process of resetting the state of the secondary to >>>> match >>>>>>>>>>>>> the >>>>>>>>>>>>> primary. >>>>>>>>>>>> >>>>>>>>>>>> Yes, at this point, the qemu process associated with this >> domain is >>>>>>>>>>>> frozen. >>>>>>>>>>>> the suspend callback will call libxl__qmp_stop(vm_stop() in >>>> qemu) >>>>>> to >>>>>>>>>>>> pause >>>>>>>>>>>> qemu. After we processed all records, qemu will be restored >> with >>>>>> the >>>>>>>>>>>> received >>>>>>>>>>>> state, that's why we add a >>>>>> libxl__qmp_restore(qemu_load_vmstate() >>>>>>>> in >>>>>>>>>>>> qemu) >>>>>>>>>>>> api to restore qemu with received state. Currently in libxl, >> qemu >>>> only >>>>>>>>>>>> start >>>>>>>>>>>> with the received state, there's no api to load received state >> while >>>>>>>>>>>> qemu is >>>>>>>>>>>> running for a while. >>>>>>>>>>> >>>>>>>>>>> Now I consider this more, it is absolutely wrong to not zero the >>>> page >>>>>>>>>>> here. The event channel in the page is not guaranteed to be the >>>>>> same >>>>>>>>>>> between the primary and secondary, >>>>>>>>>> >>>>>>>>>> That's why we don't zero it on secondary. >>>>>>>>> >>>>>>>>> I think you missed my point. Apologies for the double negative. It >>>>>>>>> must, under all circumstances, be zeroed at this point, for safety >>>>>> reasons. >>>>>>>>> >>>>>>>>> The page in question is subject to logdirty just like any other guest >>>>>>>>> pages, which means that if the guest writes to it naturally (i.e. not >>>>>>>>> a >>>>>>>>> Xen or Qemu write, both of whom have magic mappings which are >>>> not >>>>>>>>> subject to logdirty), it will be transmitted in the stream. As the >>>>>>>>> event channel could be different, the lack of zeroing it at this point >>>>>>>>> means that the event channel would be wrong as opposed to >> simply >>>>>>>>> missing. This is a worse position to be in. >>>>>>>> >>>>>>>> The guest should not access this page. I am not sure if the guest can >>>>>>>> access the ioreq page. >>>>>>>> >>>>>>>> But in the exceptional case, the ioreq page is dirtied, and is copied >>>>>>>> to >>>>>>>> the secondary vm. The ioreq page will contain a wrong event >> channel, >>>> the >>>>>>>> hypervisor will check it: if the event channel is wrong, the guest will >>>>>>>> be crashed. >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> and we don't want to unexpectedly >>>>>>>>>>> find a pending/in-flight ioreq. >>>>>>>>>> >>>>>>>>>> ioreq->state is always STATE_IOREQ_NONE after the vm is >>>> suspended, >>>>>>>> there >>>>>>>>>> should be no pending/in-flight ioreq at checkpoint. >>>>>>>>> >>>>>>>>> In the common case perhaps, but we must consider the >> exceptional >>>>>> case. >>>>>>>>> The exceptional case here is some corruption which happens to >>>> appear >>>>>> as >>>>>>>>> an in-flight ioreq. >>>>>>>> >>>>>>>> If the state is STATE_IOREQ_NONE, it may be hypervisor's bug. If the >>>>>>>> hypervisor >>>>>>>> has a bug, anything can happen. I think we should trust the >> hypervisor. >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Either qemu needs to take care of re-initialising the event >> channels >>>>>>>>>>> back to appropriate values, or Xen should tolerate the channels >>>>>>>>>>> disappearing. >>>>>>>>> >>>>>>>>> I still stand by this statement. I believe it is the only safe way of >>>>>>>>> solving the issue you have discovered. >>>>>>>> >>>>>>>> Add a new qemu monitor command to update ioreq page? >>>>>>>> >>>>>>> >>>>>>> If you're attaching to a 'new' VM (i.e one with an updated image) then >> I >>>>>> suspect you're going to have to destroy and re-create the ioreq server >> so >>>>>> that the shared page gets re-populated with the correct event >> channels. >>>>>> Either that or you're going to have to ensure that the page is not part >>>>>> of >>>>>> restored image and sample the new one that Xen should have set up. >>>>>> >>>>>> >>>>>> I agree with it. I will try to add a new qemu monitor command(or do it >>>> when >>>>>> updating qemu's state) to destroy and re-create it. >>>>> >>>>> The slightly tricky part of that is that you're going to have to cache and >>>> replay all the registrations that were done on the old instance, but you >> need >>>> to do that in any case as it's not state that is transferred in the VM save >>>> record. >>>> >>>> Why do we have to cache and replay all the registrations that were done >> on >>>> the old instance? >>> >>> Do you not have device models that you need to continue to function? >> When the ioreq server is torn down then all MMIO, port IO and PCI config >> ranges that were mapped to it will disappear. >> >> Yes, I don't known which should be done unless I implement and test it. >> >> I have some questions about it: >> 1. Can guest access the ioreq page? If the page is modified by the guest >> unexpectedly, >> what will happen? > > No, the guest cannot modify the pages once a non-default ioreq server is > active. The pages are removed from the guest P2M when it is activated, which > is one of the reasons for modifying QEMU to not behave as a legacy default > server. > >> 2. If the ioreq page is dirtied by the guest, it will be transfered from >> primary >> to secondary during checkpoint. The evtchn is invalid, I think the best >> behavior >> is that: make the guest crashed, not continue to run. > > As I said, the pages are not in the P2M if the server is active so they will > not be transferred as part of the VM state. However, this presents a problem; > at the far end, the emulator will not be able to hook into the guest. So, > when the source domain is paused, the ioreq server needs to be torn down (so > that its pages are re-inserted into the P2M and marked dirty for transfer). > This is what happens in a normal migration. One extra problem you have is > that the source domain is not then killed, it is resumed along with the > emulator. Thus, on resume, the emulator needs to create a new ioreq server > and re-register all its device models with that new server. > I don't know the detail of what you do at the far end, but if you always > start a new emulator instance using the QEMU save record then you should be > fine (just like with a normal migration). I don't find the codes where the ioreq server is torn down when the source domain is paused. Which function? In our implementation, we don't start a new emulator. The codes can work, but some bugs may be not triggered. Thanks Wen Congyang > > Paul > >> >> Thanks >> Wen Congyang >> >>> >>> Paul >>> >>>> We will set to the guest to a new state, the old state should be dropped. >>>> >>>> Thanks >>>> Wen Congyang >>>> >>>>> >>>>> Paul >>>>> >>>>>> >>>>>> Thanks >>>>>> Wen Congyang >>>>>> >>>>>>> >>>>>>> Paul >>>>>>> >>>>>>> >>>>>>>> Thanks >>>>>>>> Wen Congyang >>>>>>>> >>>>>>>>> >>>>>>>>> ~Andrew >>>>>>>>> . >>>>>>>>> >>>>>>> >>>>>>> . >>>>>>> >>>>> >>>>> . >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@xxxxxxxxxxxxx >>>> http://lists.xen.org/xen-devel >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@xxxxxxxxxxxxx >>> http://lists.xen.org/xen-devel >>> . >>> > > . > _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |