Re: [Xen-devel] status of PV drivers in Xen 4.0

[Andrew Lyon <andrew.lyon@xxxxxxxxx>]

On Mon, Apr 12, 2010 at 12:27 PM, Guillaume Rousse
<Guillaume.Rousse@xxxxxxxx> wrote:
> Le 10/04/2010 14:58, Pasi Kärkkäinen a écrit :
>>> Have you been successful building the drivers for .31 or .32?
> I just tried 2.6.33 so far, without success.
>
>>> I'd like to make a set of patches available to apply to Xen 3.4.2 or
>>> 4.0 to enable building the drivers with recent kernels and the forward
>>> ported kernels I usually use, but while I've been able to rebuild them
>>> on a opensuse 11.2 system running 2.6.31.12 I am finding it hard to
>>> work out which patches need to be applied and what else the rpm build
>>> process does to enable compilation.
>>>
>>> Any pointers would be much appreciated.
>>>
>>
>> Did you take a look at novell's sles11 sp1 kernel? it's 2.6.32 and
>> should have pv-on-hvm drivers included?
> Yes, that's Jan Beulich said on this list. However, I've buid a
> 2.6.32.11 kernel, with all xen patches available from SLE11-SP1 (as well
> as a few others needed for applying xen patches properly), and used the
> corresponding defconfig file. But I can't find any reference to those
> drivers, at least with the names given in README file of xen tarball
> (xen-platform-pci.ko, xen-vbd.ko, xen-vnif.ko and xen-balloon.ko).
> That's probably just because of a name change.
>
> Anyway (unless I'm wrong), the actual usefulness of pv-on-hvm is on
> standard linux kernels, not on dom0 itself, and the issue seems to be on
> xen Makefile side.
> --

I've been looking at this over the weekend and I've made some
progress, you are correct about the usefulness of the pv-on-hvm
drivers, however you need to bear in mind that the openSUSE/SLES11
kernels *always* include the xen patches even if they are not
activated for a particular kernel flavour (e.g. desktop, debug etc),
so there are some differences in how the drivers are built,
specifically the build process uses header files from the xenified
kernel instead of from Xen itself.

You will not find the pv-on-hvm drivers in the kernel configuration
options of any xenified kernel, they are built separately as part of
Xen, I decided to have a look at 2.6.31 so I downloaded the latest Xen
source rpm for openSUSE 11.2
(http://download.opensuse.org/source/distribution/11.2/repo/oss/suse/src/xen-3.4.1_19718_04-2.1.src.rpm),
you need to extract the contents of the rpm, then extract the
xen-3.4.1-testing-src.tar.bz2, then you need to apply the same patches
that the rpm build process would apply, you can work out which patches
by reading xen.spec or by running rpmbuild -bb xen.spec on a opensuse
system and logging the console output during the build process, then
parsing the log for patch commands and running them on the extracted
xen source tree.

The unmodified_drivers build process is designed to work with a
separate kernel "objects" folder, so instead of configuring and
compiling your kernel in-place you need to do something like this,
note that while this step uses a xenified kernel you should not enable
Xen in the processor type options, configure the kernel as you would
for the hvm guest.

cd /usr/src/linux-2.6.31-xen-r14
mkdir ../linux-2.6.31-xen-r14-obj
mv .config ../linux-2.6.31-xen-r14-obj/ (or copy in your usual default
config file)
make mrproper
cd ../linux-2.6.31-xen-r14-obj/
make -C /usr/src/linux-2.6.31-xen-r14 menuconfig
O=/usr/src/linux-2.6.31-xen-r14-obj/
make -C /usr/src/linux-2.6.31-xen-r14 bzImage (or vmlinuz) modules
modules_install O=/usr/src/linux-2.6.31-xen-r14-obj/

Now change to wherever you put the unmodified_drivers/linux-2.6/:

export XL=/usr/src/linux-2.6.31-xen-r14
export XEN=/usr/src/linux-2.6.31-xen-r14/include/xen/

Notice how XEN is set to linux/include/xen ? the Xen documentation
states that it should be set to a folder containing the Xen sources,
I've tried that and it does not work, I think the folder is only used
for header files so they could in theory come from Xen or from a
Xenified kernel, but on openSUSE the kernel include files are used
instead of the Xen ones, and that is the only way that seems to work.

./mkbuildtree
Defaulting to this machine's architecture, x86_64, for linking.
This may be overridden on the command line (i386,x86_64,ia64).
Xen tree: /usr/src/linux-2.6.31-xen-r14/include/xen
Linux tree: /usr/src/linux-2.6.31-xen-r14

You can now compile the drivers:

make -C /usr/src/linux-2.6.31-xen-r14-obj/ modules M=$PWD

If you attempt to do the same process without putting the kernel
object files into a separate folder the build will fail, I believe
this is because declarations in the kernel override those in the
unmodified_drivers.

Likewise the process fails if you try to use a generic kernel source
tree which has not been patched for Xen dom0, in which case the header
files are missing various Xen declarations required for successful
compilation.

And here is a final proof, Xenified 2.6.31 with all XEN options
disabled, with pv-on-hvm drivers:

localhost ~ # uname -a
Linux localhost 2.6.31-xen-r14 #2 SMP Mon Apr 12 13:34:27 GMT 2010
x86_64 Intel(R) Xeon(R) CPU E5420 @ 2.50GHz GenuineIntel GNU/Linux
localhost ~ # zgrep XEN /proc/config.gz
# CONFIG_X86_64_XEN is not set
localhost ~ # lsmod
Module                  Size  Used by
xen_vnif               28288  0
xen_balloon            14496  1 xen_vnif
xen_vbd                19084  3
xen_platform_pci       78808  3 xen_vnif,xen_balloon,xen_vbd,[permanent]

I'm not sure where to go next with this but I hope this info is useful.

Andy

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel