Re: [Xen-users] Remus: network buffering problem

[Shriram Rajagopalan <rshriram@xxxxxxxxx>]

Remus currently works with PV (2.6.18) domU or HVM. Not with 2.6.32.x
domU kernel.
Dont use a bootloader if you want a pv kernel. remove the bootloader line.

 I am not sure a ubuntu 10.10 domU would run comfortably on 2.6.18 kernel.
So, use an older version of ubuntu (hardy heron 8.04 probably?) or
something like
centos.

shriram
On Mon, Feb 21, 2011 at 1:08 PM, Wenda Ni <wonda.ni@xxxxxxxxx> wrote:
> Thanx, Shriram.
>
> This is the most confusing part and the biggest worry in my mind. So far, I
> have not managed to figure out what the difference between PVOPS and HVM is,
> particularly to the extent that how I can identify or create a domain with
> clear view of whether it is PVOPS or HVM. I understand that in CentOS 5.5,
> the first question of "virt-install --prompt" is on whether your want your
> domU to be created fully virtualized or not. But when using xen-tools (on
> Ubuntu), it is not so obvious.
>
> I have an previous post on xen-users discussing my confusion, and soliciting
> knowledge. Unfortunately, it is still left unanswered so far. The post is
> attached as follows (with modifications):
>
> I have Xen 4.0.1 pvops on Ubuntu 10.10 (The original Ubuntu kernel is
> 2.6.35-25-generic) following the steps on
> https://help.ubuntu.com/community/Xen
> I use xen-tools (debootstrap) to configure domU. In xen-tools.config, I
> specify
> "
> kernel = /boot/vmlinuz-2.6.18.8
> initrd = /boot/initrd.img-2.6.18.8
> "
> It seems that the actual .cfg file generated for domU uses
> "
> bootloader = '/usr/bin/pygrub'
> "
> There are some explanations about pygrub on xenwiki, so I believe this
> should be fine. The .cfg created is as follows:
>
> #
> # Configuration file for the Xen instance Alice, created
> # by xen-tools 4.2 on Mon Feb 21 00:30:22 2011.
> #
>
> #
> #  Kernel + memory size
> #
>
>
> bootloader = '/usr/bin/pygrub'
>
> vcpus       = '1'
> memory      = '128'
>
> #
> #  Disk device(s).
> #
> root        = '/dev/sda2 ro'
> disk        = [
>                   'file:/mnt/vm-nfs-storage/domains/Alice/disk.img,sda2,w',
>                   'file:/mnt/vm-nfs-storage/domains/Alice/swap.img,sda1,w',
>               ]
>
>
> #
> #  Physical volumes
> #
>
>
> #
> #  Hostname
> #
> name        = 'Alice'
>
> #
> #  Networking
> #
> vif         = [ 'ip=155.98.38.230,mac=00:16:3E:B1:BE:76' ]
>
> #
> #  Behaviour
> #
> on_poweroff = 'destroy'
> on_reboot   = 'restart'
> on_crash    = 'restart'
>
> And when the domU starts, the "uname -a" gives kernel as 2.6.35-25-generic,
> which is exactly the same as the original Ubuntu kernel version.
>
> So my question is 1) how to find whether domU is pv-ops or HVM?
>
>
> Next, According to http://wiki.xen.org/xenwiki/XenParavirtOps?, I checked
> the ..../build-linux-2.6-pvops_x86_32/./config,
>
> If building 32 bit kernel make sure you have CONFIG_X86_PAE enabled (which
> is set by selecting CONFIG_HIGHMEM64G)
>
> non-PAE mode doesn't work in 2.6.25, and has been dropped altogether from
> 2.6.26 and newer kernel versions.
>
> Enable these core options:
>
> CONFIG_PARAVIRT_GUEST
> CONFIG_XEN
>
> And Xen pv device support
>
> CONFIG_HVC_DRIVER and CONFIG_HVC_XEN
> CONFIG_XEN_BLKDEV_FRONTEND
> CONFIG_XEN_NETDEV_FRONTEND
>
> These options are all enabled, so I believe the dom0 built does support
> pv-ops domU. Then 2) how do I build a pv-ops domU?
>
> I have been struggling to find out the answer, but no progress so far in
> part due to my limited knowledge on OS. I am new to this area, so please
> forgive any of my simple questions, and hope the reply can be easy to
> follow.
>
>
>
>
>
>
> On Mon, Feb 21, 2011 at 11:11 AM, Shriram Rajagopalan <rshriram@xxxxxxxxx>
> wrote:
>>
>> Yes absolutely. As the Remus wiki in xensource says, it still needs 2.6.18
>> domU kernel (or novel suse 2.6.27) to operate in fast mode.
>>  What type of guest are u running? PV or hvm?
>>
>> Shriram
>> On 2011-02-20, at 11:44 PM, Wenda Ni <wonda.ni@xxxxxxxxx> wrote:
>>
>> Thank you so much, Shriram. This error has been well handled. Now I meet
>> with the following error:
>>
>> WARNING: suspend event channel unavailable, falling back to slow xenstore
>> signalling
>> Had 0 unexplained entries in p2m table
>>  1: sent 32768, skipped 0, delta 1477ms, dom0 38%, target 0%, sent
>> 726Mb/s, dirtied 0Mb/s 0 pages
>>  2: sent 0, skipped 0, Start last iteration
>> PROF: suspending at 1298269217.559698
>> timeout polling fd
>> ERROR Internal error: Suspend request failed
>> ERROR Internal error: Domain appears not to have suspended
>> Save exit rc=1
>> tc filter del dev vif2.0 parent ffff: proto ip pref 10 u32
>> RTNETLINK answers: Invalid argument
>> We have an error talking to the kernel
>> Exception xen.remus.util.PipeException: PipeException('tc failed: 2, No
>> such file or directory',) in <bound method BufferedNIC.__del__ of
>> <xen.remus.device.BufferedNIC object at 0x228cd90>> ignored
>>
>> It is related to the guest kernel?
>>
>>
>> On Sun, Feb 20, 2011 at 11:56 PM, Shriram Rajagopalan <rshriram@xxxxxxxxx>
>> wrote:
>>>
>>> On Sun, Feb 20, 2011 at 8:39 PM, Wenda Ni <wonda.ni@xxxxxxxxx> wrote:
>>> > Hi Shriram,
>>> >
>>> > I have disabled NET_SCH_PLUG.
>>> >
>>> > I have modified the tools/remus/kmod/Makefile by removing "grep -q
>>> > ^CONFIG_IMQ= $(KERNELDIR)/.config 2>/dev/null" as
>>> >
>>> > XEN_ROOT=../../..
>>> > include $(XEN_ROOT)/tools/Rules.mk
>>> >
>>> > # Should make makefiles export linux build directory!
>>> > # This is a fragile hack to tide us over
>>> > ifeq ($(KERNELS),linux-2.6-xen)
>>> > LINUX_VER=2.6.18-xen
>>> > endif
>>> > ifeq ($(KERNELS),linux-2.6-xen0)
>>> > LINUX_VER=2.6.18-xen0
>>> > endif
>>> > ifeq ($(KERNELS),linux-2.6-pvops)
>>> > LINUX_VER=2.6-pvops
>>> > endif
>>> >
>>> > KERNELDIR ?= $(XEN_ROOT)/build-linux-$(LINUX_VER)_$(XEN_TARGET_ARCH)
>>> >
>>> > .PHONY: all
>>> > all:
>>> >         if test -d $(KERNELDIR); then $(MAKE) -C $(KERNELDIR)
>>> > SUBDIRS=`pwd`
>>> > modules; fi
>>> >
>>> > .PHONY: install
>>> > install: all
>>> >         if test -d $(KERNELDIR); then $(MAKE) -C $(KERNELDIR)
>>> > SUBDIRS=`pwd`
>>> > INSTALL_MOD_PATH=$(DESTDIR) modules_install; fi
>>> >
>>> > clean::
>>> >         -rm -rf *.o *.ko *.mod.c *.mod.o Module.symvers .*.cmd
>>> > .tmp_versions
>>> >
>>> My bad. Just edit the Kbuild file to
>>> obj-m := sch_queue.o
>>>
>>> This will prevent the ebt_imq modules from being built, since we are
>>> using
>>> IFB (not IMQ). The rest is usual make and make install stuff.
>>> > When I do a "sudo make" in the same directory, it gives the following
>>> > error.
>>> > Is there a a model called ebt_imq in the menuconfig? Where can i find
>>> > it? I
>>> > have searched a lot, but failed.
>>> >
>>> > /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c: In function
>>> > ‘ebt_imq_tg’:
>>> > /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c:62: error: ‘struct
>>> > sk_buff’
>>> > has no member named ‘imq_flags’
>>> > /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c:62: error:
>>> > ‘IMQ_F_ENQUEUE’
>>> > undeclared (first use in this function)
>>> > /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c:62: error: (Each
>>> > undeclared
>>> > identifier is reported only once
>>> > /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c:62: error: for each
>>> > function
>>> > it appears in.)
>>> > make[4]: *** [/mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.o] Error 1
>>> > make[3]: *** [_module_/mnt/local/xen-4.0.1/tools/remus/kmod] Error 2
>>> > make[2]: *** [sub-make] Error 2
>>> > make[1]: *** [all] Error 2
>>> > make[1]: Leaving directory
>>> > `/mnt/local/xen-4.0.1/build-linux-2.6-pvops_x86_64'
>>> > make: *** [all] Error 2
>>> >
>>> > You said I need to manually configure the KERNELDIR. Could you be more
>>> > specific? I am new to this area. please make it easy to implement.
>>> > Thank you
>>> > very much.
>>> >
>>> >
>>> > On Sun, Feb 20, 2011 at 1:10 AM, Shriram Rajagopalan
>>> > <rshriram@xxxxxxxxx>
>>> > wrote:
>>> >>
>>> >> On Sat, Feb 19, 2011 at 9:14 PM, Wenda Ni <wonda.ni@xxxxxxxxx> wrote:
>>> >> > Hi Shriram,
>>> >> >
>>> >> > Thank you so much for your advice.
>>> >> >
>>> >> > I have enabled IFB (Intermediate functional block) module. I have
>>> >> > enabled
>>> >> > Ingress, redirecting, mirroring, NET_SCH_PLUG, and Netfilter. I have
>>> >> > moduled
>>> >> > or enabled the rest of the selections under QoS and/or fair
>>> >> > queueing.
>>> >> > Now it
>>> >> > give the following error.
>>> >> >
>>> >> > I cannot find the sch_queue module under QoS and/or fair queueing. I
>>> >> > find a
>>> >> > few previous discussions about the sch_queue module, but have not
>>> >> > found
>>> >> > a
>>> >> > good solution. Could you help a bit further? I am using XEN 4.0.1
>>> >> > release
>>> >> > pvops on Ubuntu 10.10 (2.6.32.27).
>>> >> >
>>> >> You only need either sch_plug or the sch_queue. not both.
>>> >>  this issue has been fixed in the latest xen-unstable. But it still
>>> >> lingers in 4.0.1. Here is what I suggest
>>> >>
>>> >> option 1:
>>> >> Go to tools/remus/kmod/
>>> >>  Hack the Makefile. I mean, manually set the KERNELDIR,
>>> >>  remove the "grep CONFIG_IMQ..".
>>> >>  if you do a make now, you should be able to compile the sch_queue
>>> >> module
>>> >>  and then a make install.
>>> >>  Basically, fix the makefile and compile/install the sch_queue module
>>> >>
>>> >> Option 2:
>>> >>  (I suggest you avoid this). Look at device.py and qdisc.py from
>>> >> xen-unstable
>>> >> and fix your local repo's version (from xen 4.0.1).
>>> >>
>>> >> BTW, you are probably going to be raising the next issue - Disk
>>> >> replication fails.
>>> >>  This is a bad hack but probably saves u time, for the moment, in
>>> >> 4.0.1. Edit the device.py near
>>> >>  if disk.uname.startswith("tap:remus..") to "tap2:remus"...
>>> >>  Unfortunately, 4.0.1 decided that users should explicitly specify
>>> >> tap2:... to use blktap2 driver,
>>> >> which meant that using tap:aio leads to loading blktap1
>>> >> driver(deprecated/doesnt work in pvops).
>>> >> So, one would have to use tap2:aio (loads blktap2 driver), and thus
>>> >> similarly tap2:remus:..
>>> >>        - but the remus scripts werent fixed soon enough to reflect
>>> >> this, in 4.0.1. :(
>>> >>
>>> >> shriram
>>> >> >
>>> >> > modprobe -q sch_queue
>>> >> >
>>> >> > Traceback (most recent call last):
>>> >> >   File "/usr/bin/remus", line 210, in <module>
>>> >> >     run(cfg)
>>> >> >   File "/usr/bin/remus", line 119, in run
>>> >> >     bufs.append(BufferedNIC(vif))
>>> >> >   File "/usr/local/lib/python2.6/dist-packages/xen/remus/device.py",
>>> >> > line
>>> >> > 275, in __init__
>>> >> >     self.setup()
>>> >> >   File "/usr/local/lib/python2.6/dist-packages/xen/remus/device.py",
>>> >> > line
>>> >> > 314, in setup
>>> >> >     raise BufferedNICException('could not load sch_queue module')
>>> >> > xen.remus.device.BufferedNICException: could not load sch_queue
>>> >> > module
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Sat, Feb 19, 2011 at 6:25 PM, Shriram Rajagopalan
>>> >> > <rshriram@xxxxxxxxx>
>>> >> > wrote:
>>> >> >>
>>> >> >> On 2011-02-18, at 7:04 PM, Wenda Ni <wonda.ni@xxxxxxxxx> wrote:
>>> >> >>
>>> >> >> Hi, Todd
>>> >> >>
>>> >> >> I realize what you mean. <330.gif> Hope Shriram can shed some light
>>> >> >> to
>>> >> >> us!
>>> >> >>
>>> >> >> On Fri, Feb 18, 2011 at 9:44 PM, Todd Deshane
>>> >> >> <todd.deshane@xxxxxxx>
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> Adding our new remus maintainer to the CC
>>> >> >>>
>>> >> >>> On Fri, Feb 18, 2011 at 6:24 AM, Wenda Ni <wonda.ni@xxxxxxxxx>
>>> >> >>> wrote:
>>> >> >>> > Hello,
>>> >> >>> >
>>> >> >>> > Many thanks to Simon for his trouble shooting on live migration.
>>> >> >>> > I
>>> >> >>> > have
>>> >> >>> > now
>>> >> >>> > started testing remus according to the remus link
>>> >> >>> > http://nss.cs.ubc.ca/remus/doc.html
>>> >> >>> >
>>> >> >>> > 1) Remus in its simplest mode (--no-net option), without disk
>>> >> >>> > replication or
>>> >> >>> > network protection, is now working good.
>>> >> >>> >
>>> >> >>> > 2) When we add the network buffering protection (remove the
>>> >> >>> > --no-net
>>> >> >>> > option), remus gives the error as follows:
>>> >> >>> >
>>> >> >>> > modprobe -q ifb
>>> >> >>> >
>>> >> >>> > modprobe -q imq
>>> >> >>> >
>>> >> >>
>>> >> >> Have you installed the ifb module?
>>> >> >> You have to enable it while configuring the dom0 kernel.
>>> >> >>  Device Drivers-->network devices-->Intermediate Frame Buffer
>>> >> >> You also need to enable a whole bunch of other things
>>> >> >>  Networking ->QoS ->Ingress, actions, mirred, redirect, etc
>>> >> >>   and Plug Queue (NET_SCH_PLUG).
>>> >> >>
>>> >> >> You are better off at the moment to just enable all or most of the
>>> >> >> stuff
>>> >> >> under Netfilter and Qos as modules. Might make your task easier.
>>> >> >>
>>> >> >> Btw thanks for pointing this out. These should probably be enabled
>>> >> >> by
>>> >> >> the
>>> >> >> kernel build scripts (buildconfigs/enable-xen-config ) or some such
>>> >> >> place.
>>> >> >>
>>> >> >> Shriram
>>> >> >>>
>>> >> >>> > Traceback (most recent call last):
>>> >> >>> >   File "/usr/bin/remus", line 210, in <module>
>>> >> >>> >     run(cfg)
>>> >> >>> >   File "/usr/bin/remus", line 119, in run
>>> >> >>> >     bufs.append(BufferedNIC(vif))
>>> >> >>> >   File
>>> >> >>> > "/usr/local/lib/python2.6/dist-packages/xen/remus/device.py",
>>> >> >>> > line
>>> >> >>> > 272, in __init__
>>> >> >>> >     self.pool = Netbufpool(selectnetbuf())
>>> >> >>> >   File
>>> >> >>> > "/usr/local/lib/python2.6/dist-packages/xen/remus/device.py",
>>> >> >>> > line
>>> >> >>> > 174, in selectnetbuf
>>> >> >>> >     raise BufferedNICException('no net buffer available')
>>> >> >>> > xen.remus.device.BufferedNICException: no net buffer available
>>> >> >>> > Exception AttributeError: "'BufferedNIC' object has no attribute
>>> >> >>> > 'bufdev'"
>>> >> >>> > in <bound method BufferedNIC.__del__ of
>>> >> >>> > <xen.remus.device.BufferedNIC
>>> >> >>> > object
>>> >> >>> > at 0xd9dd90>> ignored
>>> >> >>> >
>>> >> >>> > I googled, but find little useful info. Could anyone know my
>>> >> >>> > problem?
>>> >> >>> >
>>> >> >>> > --
>>> >> >>> > Cheers,
>>> >> >>> >
>>> >> >>> > Wenda Ni, Ph.D.
>>> >> >>> > Dept. of Computer Science and Engineering
>>> >> >>> > State University of New York (SUNY) at Buffalo
>>> >> >>> >
>>> >> >>> > _______________________________________________
>>> >> >>> > Xen-users mailing list
>>> >> >>> > Xen-users@xxxxxxxxxxxxxxxxxxx
>>> >> >>> > http://lists.xensource.com/xen-users
>>> >> >>> >
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> Cheers,
>>> >> >>
>>> >> >> Wenda Ni, Ph.D.
>>> >> >> Dept. of Computer Science and Engineering
>>> >> >> State University of New York (SUNY) at Buffalo
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Cheers,
>>> >> >
>>> >> > Wenda Ni, Ph.D.
>>> >> > Dept. of Computer Science and Engineering
>>> >> > State University of New York (SUNY) at Buffalo
>>> >> >
>>> >> >
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> >
>>> > Wenda Ni, Ph.D.
>>> > Dept. of Computer Science and Engineering
>>> > State University of New York (SUNY) at Buffalo
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> >
>>> > Wenda Ni, Ph.D.
>>> > Dept. of Computer Science and Engineering
>>> > State University of New York (SUNY) at Buffalo
>>> >
>>> >
>>
>>
>>
>> --
>> Cheers,
>>
>> Wenda Ni, Ph.D.
>> Dept. of Computer Science and Engineering
>> State University of New York (SUNY) at Buffalo
>>
>
>
>
>

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users