Re: [Xen-users] Error accessing memory mapped by xenforeignmemory_map()

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Mon, 30 Oct 2017, Brett Stahlman wrote:
> Julien,
> 
> On Mon, Oct 30, 2017 at 1:37 PM, Julien Grall <julien.grall@xxxxxxx> wrote:
> > On 30/10/17 16:26, Brett Stahlman wrote:
> >>
> >> Hello Julien,
> >
> >
> > Hello Brett,
> >
> >> On Sun, Oct 29, 2017 at 3:37 PM, Julien Grall <julien.grall@xxxxxxx>
> >> wrote:
> >>>
> >>> Hello Brett,
> >>>
> >>> On 27/10/2017 22:58, Brett Stahlman wrote:
> >>>>
> >>>>
> >>>> On Fri, Oct 27, 2017 at 3:22 PM, Stefano Stabellini
> >>>> <sstabellini@xxxxxxxxxx> wrote:
> >>>>>
> >>>>>
> >>>>> CC'ing the tools Maintainers and Paul
> >>>>>
> >>>>> On Fri, 27 Oct 2017, Brett Stahlman wrote:
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Oct 27, 2017 at 9:31 AM, Roger Pau Monné
> >>>>>> <roger.pau@xxxxxxxxxx>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> Adding the ARM maintainers.
> >>>>>>>
> >>>>>>> On Wed, Oct 25, 2017 at 11:54:59AM -0500, Brett Stahlman wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I'm trying to use the "xenforeignmemory" library to read arbitrary
> >>>>>>>> memory ranges from a Xen domain. The code performing the reads is
> >>>>>>>> designed to run in dom0 on a Zynq ultrascale MPSoC (ARM64), though
> >>>>>>>> I'm
> >>>>>>>> currently testing in QEMU. I constructed a simple test program,
> >>>>>>>> which
> >>>>>>>> reads an arbitrary domid/address pair from the command line,
> >>>>>>>> converts
> >>>>>>>> the address (assumed to be physical) to a page frame number, and
> >>>>>>>> uses
> >>>>>>>> xenforeignmemory_map() to map the page into the test app's virtual
> >>>>>>>> memory space. Although xenforeignmemory_map() returns a non-NULL
> >>>>>>>> pointer, my attempt to dereference it fails with the following
> >>>>>>>> error:
> >>>>>>>>
> >>>>>>>> (XEN) traps.c:2508:d0v1 HSR=0x93810007 pc=0x400a20 gva=0x7f965f7000
> >>>>>>>> gpa=0x00000030555000
> >>>>>>>>
> >>>>>>>> [   74.361735] Unhandled fault: ttbr address size fault (0x92000000)
> >>>>>>>> at 0x0000007f965f7000
> >>>>>>>> Bus error
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> I'm not sure what a Bus error means on ARM, have you tried to look
> >>>>>>> at traps.c:2508 to see if there's some comment explaining why this
> >>>>>>> fault is triggered?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I believe the fault is occurring because mmap() failed to map the
> >>>>>> page.
> >>>>>> Although xenforeignmemory_map() is indeed returning a non-NULL
> >>>>>> pointer,
> >>>>>> code comments indicate that this does not imply success: page-level
> >>>>>> errors might still be returned in the provided "err" array. In my
> >>>>>> case,
> >>>>>> it appears that an EINVAL is produced by mmap(): specifically, I
> >>>>>> believe
> >>>>>> it's coming from privcmd_ioctl_mmap_batch() (drivers/xen/privcmd.c),
> >>>>>> but
> >>>>>> there are a number of conditions that can produce this error code, and
> >>>>>> I
> >>>>>> haven't yet determined which is to blame...
> >>>>>>
> >>>>>> So although I'm not sure why I would get an "address size" fault, it
> >>>>>> makes sense that the pointer dereference would generate some sort of
> >>>>>> paging-related fault, given that the page mapping was unsuccessful.
> >>>>>> Hopefully, ARM developers will be able to explain why it was
> >>>>>> unsuccessful, or at least give me an idea of what sorts of things
> >>>>>> could
> >>>>>> cause a mapping attempt to fail... At this point, I'm not particular
> >>>>>> about what address I map. I just want to be able to read known data at
> >>>>>> a
> >>>>>> fixed (non-paged) address (e.g., kernel code/data), so I can prove to
> >>>>>> myself that the page is actually mapped.
> >>>>>
> >>>>>
> >>>>>
> >>>>> The fault means "Data Abort from a lower Exception level". It could be
> >>>>> an MMU fault or an alignment fault, according to the ARM ARM.
> >>>>>
> >>>>> I guess that the address range is not good. What DomU addresses are you
> >>>>> trying to map?
> >>>>
> >>>>
> >>>>
> >>>> The intent was to map fixed "guest physical" addresses corresponding to
> >>>> (e.g) the "zero page" of a guest's running kernel. Up until today, I'd
> >>>
> >>>
> >>>
> >>> What do you mean by "zero page"? Is it the guest physical address 0? If
> >>> so,
> >>> the current guest memory layout does not have anything mapped at the
> >>> address.
> >>
> >>
> >> No. I didn't mean guest physical address 0, but rather the start of the
> >> linux kernel itself: specifically, the code in head.S. IIUC, the kernel
> >> bootstrap code responsible for decompressing the kernel typically loads
> >> this code at a fixed address, which on x86 architectures, happens to be
> >> 0x100000. Thus, my assumption has been that if an unmodified Linux OS
> >> were run in an x86 Xen guest, Xen would need to map guest physical
> >> address 0x100000 to the machine physical address where the guest Linux
> >> kernel is actually loaded. I'd also been assuming that if code running
> >> in dom0 wished to use the foreignmemory interface to read the first page
> >> of such a guest's kernel, it would need to provide the "guest physical"
> >> address 0x100000 to xenforeignmemory_map(). I'm still thinking this may
> >> be true for an *unmodified* guest (i.e., HVM), but having read more
> >> about Xen's paravirtualized memory over the weekend, I'm thinking it
> >> would not hold true for a paravirtualized (PV) guest, which doesn't have
> >> the same concept of "guest physical" addresses.
> >
> >
> > I am not x86 expert and will let x86 folks answered to that.
> >
> > To give the Arm64 view, the image headers allow to specify whether the
> > kernel needs to be close to the beginning of the DRAM. But it is still not a
> > fixed address.
> >
> > In practice, the toolstack will always load the Image at the ram base + text
> > offset (specified in the kernel). But that's just for convenience and may
> > change in the future.
> 
> Ok. Perhaps it would help if I examined this code...
> 
> >
> >>
> >>>
> >>>> assumed that a PV guest's kernel would be loaded at a known "guest
> >>>> physical" address (like 0x100000 on i386), and that such addresses
> >>>> corresponded to the gfn's expected by xenforeignmemory_map(). But now I
> >>>> suspect this was an incorrect assumption, at least for the PV case. I've
> >>>> had trouble finding relevant documentation on the Xen site, but I did
> >>>> find a presentation earlier today suggesting that for PV's, gfn == mfn,
> >>>> which IIUC, would effectively preclude the use of fixed addresses in a
> >>>> PV guest. IOW, unlike an HVM's kernel, a PV's kernel cannot be loaded at
> >>>> a "known" address (e.g., 0x100000 on i386).
> >>>>
> >>>> Perhaps my use case (reading a guest kernel's code/data from dom0) makes
> >>>> sense for an HVM, but not a PV? Is it not possible for dom0 to use the
> >>>> foreignmemory interface to map PV guest pages read-only, without knowing
> >>>> in advance what, if anything, those pages represent in the guest? Or is
> >>>> the problem that the very concept of "guest physical" doesn't exist in a
> >>>> PV? I guess it would help if I had a better understanding of what sort
> >>>> of frame numbers are expected by xenforeignmemory_map() when the target
> >>>> VM is a PV. Is the Xen code the only documentation for this sort of
> >>>> thing, or is there some place I could get a high-level overview?
> >>>
> >>>
> >>>
> >>> I am a bit confused with the rest of this e-mail. There are no concept of
> >>> HVM or PV on Arm. This is x86 specific. For Arm, there is a single type
> >>> of
> >>> guest that borrow the goods of both HVM and PV.
> >>>
> >>> For instance, like HVM, the hardware is used to provide a separate
> >>> address
> >>> space for each virtual machine. Arm calls that stage-2 translation. So
> >>> gfn
> >>> != mfn.
> >>
> >>
> >> I was not aware that the HVM/PV concept didn't apply directly to ARM.
> >> Is there a document that summarizes the way Xen's address translation
> >> works on ARM? The document I've been looking at is...
> >>
> >> https://wiki.xen.org/wiki/X86_Paravirtualised_Memory_Management
> >
> >
> > The closest I would find explaining Xen's address translation scheme would
> > be my talk at Xen Developer Summit:
> >
> > https://www.slideshare.net/xen_com_mgr/keeping-coherency-on-arm-julien-grall-arm
> 
> After watching your talk and reading through the "Xen ARM with
> Virtualization Extensions" whitepaper, I think I have a slightly better
> understanding of how Xen ARM handles address translations. So which sort
> of address does xenforeignmemory_map() expect?
> 
> 1. stage 1 input (VA)
> 2. stage 1 output / stage 2 input (IPA)
> 3. stage 2 output (PA)
> 
> (I'm assuming #1 or #2...)

#2, guest physical addresses (also called psudo-physical addresses in
the Arm manuals)

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users