Re: [Xen-devel] status of PV drivers in Xen 4.0

[Pasi Kärkkäinen <pasik@xxxxxx>]

On Mon, Apr 12, 2010 at 04:40:49PM +0100, Andrew Lyon wrote:
> On Mon, Apr 12, 2010 at 12:27 PM, Guillaume Rousse
> <Guillaume.Rousse@xxxxxxxx> wrote:
> > Le 10/04/2010 14:58, Pasi Kärkkäinen a écrit :
> >>> Have you been successful building the drivers for .31 or .32?
> > I just tried 2.6.33 so far, without success.
> >
> >>> I'd like to make a set of patches available to apply to Xen 3.4.2 or
> >>> 4.0 to enable building the drivers with recent kernels and the forward
> >>> ported kernels I usually use, but while I've been able to rebuild them
> >>> on a opensuse 11.2 system running 2.6.31.12 I am finding it hard to
> >>> work out which patches need to be applied and what else the rpm build
> >>> process does to enable compilation.
> >>>
> >>> Any pointers would be much appreciated.
> >>>
> >>
> >> Did you take a look at novell's sles11 sp1 kernel? it's 2.6.32 and
> >> should have pv-on-hvm drivers included?
> > Yes, that's Jan Beulich said on this list. However, I've buid a
> > 2.6.32.11 kernel, with all xen patches available from SLE11-SP1 (as well
> > as a few others needed for applying xen patches properly), and used the
> > corresponding defconfig file. But I can't find any reference to those
> > drivers, at least with the names given in README file of xen tarball
> > (xen-platform-pci.ko, xen-vbd.ko, xen-vnif.ko and xen-balloon.ko).
> > That's probably just because of a name change.
> >
> > Anyway (unless I'm wrong), the actual usefulness of pv-on-hvm is on
> > standard linux kernels, not on dom0 itself, and the issue seems to be on
> > xen Makefile side.
> > --
> 
> I've been looking at this over the weekend and I've made some
> progress, you are correct about the usefulness of the pv-on-hvm
> drivers, however you need to bear in mind that the openSUSE/SLES11
> kernels *always* include the xen patches even if they are not
> activated for a particular kernel flavour (e.g. desktop, debug etc),
> so there are some differences in how the drivers are built,
> specifically the build process uses header files from the xenified
> kernel instead of from Xen itself.
> 
> You will not find the pv-on-hvm drivers in the kernel configuration
> options of any xenified kernel, they are built separately as part of
> Xen, I decided to have a look at 2.6.31 so I downloaded the latest Xen
> source rpm for openSUSE 11.2
> (http://download.opensuse.org/source/distribution/11.2/repo/oss/suse/src/xen-3.4.1_19718_04-2.1.src.rpm),
> you need to extract the contents of the rpm, then extract the
> xen-3.4.1-testing-src.tar.bz2, then you need to apply the same patches
> that the rpm build process would apply, you can work out which patches
> by reading xen.spec or by running rpmbuild -bb xen.spec on a opensuse
> system and logging the console output during the build process, then
> parsing the log for patch commands and running them on the extracted
> xen source tree.
> 

There also the "-bp" option for rpmbuild:

-bp    Executes  the  "%prep"  stage  from  the  spec  file. 
       Normally this involves unpacking the sources and applying any patches.



> The unmodified_drivers build process is designed to work with a
> separate kernel "objects" folder, so instead of configuring and
> compiling your kernel in-place you need to do something like this,
> note that while this step uses a xenified kernel you should not enable
> Xen in the processor type options, configure the kernel as you would
> for the hvm guest.
> 
> cd /usr/src/linux-2.6.31-xen-r14
> mkdir ../linux-2.6.31-xen-r14-obj
> mv .config ../linux-2.6.31-xen-r14-obj/ (or copy in your usual default
> config file)
> make mrproper
> cd ../linux-2.6.31-xen-r14-obj/
> make -C /usr/src/linux-2.6.31-xen-r14 menuconfig
> O=/usr/src/linux-2.6.31-xen-r14-obj/
> make -C /usr/src/linux-2.6.31-xen-r14 bzImage (or vmlinuz) modules
> modules_install O=/usr/src/linux-2.6.31-xen-r14-obj/
> 
> Now change to wherever you put the unmodified_drivers/linux-2.6/:
> 
> export XL=/usr/src/linux-2.6.31-xen-r14
> export XEN=/usr/src/linux-2.6.31-xen-r14/include/xen/
> 
> Notice how XEN is set to linux/include/xen ? the Xen documentation
> states that it should be set to a folder containing the Xen sources,
> I've tried that and it does not work, I think the folder is only used
> for header files so they could in theory come from Xen or from a
> Xenified kernel, but on openSUSE the kernel include files are used
> instead of the Xen ones, and that is the only way that seems to work.
> 
> ./mkbuildtree
> Defaulting to this machine's architecture, x86_64, for linking.
> This may be overridden on the command line (i386,x86_64,ia64).
> Xen tree: /usr/src/linux-2.6.31-xen-r14/include/xen
> Linux tree: /usr/src/linux-2.6.31-xen-r14
> 
> You can now compile the drivers:
> 
> make -C /usr/src/linux-2.6.31-xen-r14-obj/ modules M=$PWD
> 
> If you attempt to do the same process without putting the kernel
> object files into a separate folder the build will fail, I believe
> this is because declarations in the kernel override those in the
> unmodified_drivers.
> 
> Likewise the process fails if you try to use a generic kernel source
> tree which has not been patched for Xen dom0, in which case the header
> files are missing various Xen declarations required for successful
> compilation.
> 
> And here is a final proof, Xenified 2.6.31 with all XEN options
> disabled, with pv-on-hvm drivers:
> 
> localhost ~ # uname -a
> Linux localhost 2.6.31-xen-r14 #2 SMP Mon Apr 12 13:34:27 GMT 2010
> x86_64 Intel(R) Xeon(R) CPU E5420 @ 2.50GHz GenuineIntel GNU/Linux
> localhost ~ # zgrep XEN /proc/config.gz
> # CONFIG_X86_64_XEN is not set
> localhost ~ # lsmod
> Module                  Size  Used by
> xen_vnif               28288  0
> xen_balloon            14496  1 xen_vnif
> xen_vbd                19084  3
> xen_platform_pci       78808  3 xen_vnif,xen_balloon,xen_vbd,[permanent]
> 
> I'm not sure where to go next with this but I hope this info is useful.
> 

Congratulations!

This is valuable information, thanks for going through that.

Can you reproduce this with Xen 4.0.0 ? ie. does opensuse xen have 
some patches related to pv-on-hvm drivers? 

If there are some patches, those should be upstreamed to Xen.

Also we could try to make the compilation process easier.. somehow.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel