Re: [Xen-users] vif-route issue with HVM domU only

[Florian Heigl <florian.heigl@xxxxxxxxx>]

For the record  / sake of anyone else running into this:
I had to resort to moving away from vif-route.
The means putting [ '..., script=vif-bridge,bridge=br0' ] into my vif
config and remove the point-to-point tunneling from the domUs.
A mixed use of vif-route and vif-bridge on the same host also seemed
to be problematic.
The vif-route systems could not reach the vif-bridged' systems due to,
i suspect different gateways.

Lessons learned:
Obvious, the amount of time sunk into this would lease me another
subnet for two years or buy a short holiday trip.
If you're at Hetzner and consider doing a vif-route approach with
point-to-point tunnels to gain one extra IP address per Subnet:
Don't do it.
I've been using this for, I think, 4 years now, and on every upgrade
either Debian or Xen broke it, someone submits fixes and then it's
being broken again.
It's not worth it, so even if you're using vif-route successfully now,
I can just recommend to stop using it.


Florian

2012/12/24 Florian Heigl <florian.heigl@xxxxxxxxx>:
> Hi,
>
> I seem to have an interesting issue with vif-route.
> This is after an update to Xen 4.2.1, switching from xm to xl.
>
> I have 10 PV domUs on the host and two FreeBSD ones.
> All the PV domUs are now working nicely.
> Since FreeBSD has always been just slightly broken as PV I chose a HVM domU 
> for those, but with PV drivers.
> Those PV drivers all blew up now after the upgrade. I'm now trying to switch 
> the VMs to use normal emulated nice so I can *use* them again.
> I'm unable to launch the VMs due to an error that only appears for the HV 
> domUs.
>
> It looks as the following. The output also contains the arguments given to 
> vif-route on each run.
>
> waxh0002:~# xl create /etc/xen/xen08
> Parsing config from /etc/xen/xen08
> xc: info: VIRTUAL MEMORY ARRANGEMENT:
>   Loader:        0000000000100000->000000000019e148
>   TOTAL:         0000000000000000->000000001ec00000
>   ENTRY ADDRESS: 0000000000100000
> xc: info: PHYSICAL MEMORY ALLOCATION:
>   4KB PAGES: 0x0000000000000200
>   2MB PAGES: 0x00000000000000f5
>   1GB PAGES: 0x0000000000000000
> online type_if=vif
> add type_if=tap
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: 
> /etc/xen/scripts/vif-route add [22326] exited with error status 1
> libxl: error: libxl_device.c:979:device_hotplug_child_death_cb: script: 
> /etc/xen/scripts/vif-route failed; error detected.
> libxl: error: libxl_create.c:1097:domcreate_attach_pci: unable to add nic 
> devices
> offline type_if=vif
> remove type_if=tap
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: 
> /etc/xen/scripts/vif-route remove [22432] exited with error status 1
> libxl: error: libxl_device.c:979:device_hotplug_child_death_cb: script: 
> /etc/xen/scripts/vif-route failed; error detected.
>
>
> The actual error shows in xen-hotplug.log as follows:
> Command "188.40.175.7" is unknown, try "ip route help".
> Command "188.40.175.7" is unknown, try "ip route help".
>
>
> With some debugging it seems like it's actually passing in the IP address, 
> but not a $ipcmd telling either add or remove:
> + echo remove type_if=tap
> ++ dom0_ip
> ++ local nd=eth0
> +++ ip_of eth0
> +++ ip -4 -o addr show primary dev eth0
> +++ awk '$3 == "inet" {split($4,i,"/"); print i[1]; exit}'
> ++ local result=188.40.114.136
> ++ '[' -z 188.40.114.136 ']'
> ++ echo 188.40.114.136
> + main_ip=188.40.114.136
> + '[' -z 188.40.114.136 ']'
> + case "${command}" in
> + '[' 188.40.175.7 ']'
> + for addr in '${ip}'
> + ip route 188.40.175.7 dev vif143.0-emu src 188.40.114.136
> Command "188.40.175.7" is unknown, try "ip route help".
>
>
> I've tried for some while to understand whats really causing the "add" be 
> lost here, but no luck so far.
>
> The domU config is like this, the second domU is quite similar, except that 
> it only has one vCPU.
> builder = 'hvm'
> memory = 500
> name = "xen08"
> vcpus = "4"
> # added the type to avoid some error. bridge was there for some reason i do 
> not remember
> #vif = [ 'mac=00:16:3E..., ip=188.40.175.7, bridge=eth0, 
> type=paravirtualized' ]
> # 'type=ioemu, mac=00:16:3e:..., bridge=xenbr2, model=e1000']
> vif = [ 'type=ioemu,mac=00:16:3E:….,ip=188.40.175.7,model=e1000' ]
> disk=[
> #    'file:/mnt/FreeBSD-9.1-RELEASE-amd64-disc1.iso,hdc:cdrom,r',
>     'phy:/dev/vgxen/xen08,ioemu:hda,w',
>     'phy:/dev/vgxen/xen08swap,ioemu:hdb,w', ]
> # vncdisplay is broken in xl?
> vfb = [ "type=vnc,vncdisplay=08,vncpasswd=blah" ]
> nographic=0
> vnc=1
> stdvga=1
> #cdrom="/mnt/FreeBSD-9.1-RELEASE-amd64-disc1.iso"
> boot="c"
> serial='pty'
> cpus = '^0,^1,2-7'
>
>
>
> Version info:
> host                   : waxh0002
> release                : 3.6.11-grsec
> version                : #1-Alpine SMP Fri Dec 21 18:30:37 UTC 2012
> machine                : x86_64
> nr_cpus                : 8
> max_cpu_id             : 7
> nr_nodes               : 1
> cores_per_socket       : 4
> threads_per_core       : 2
> cpu_mhz                : 2673
> hw_caps                : 
> bfebfbff:28100800:00000000:00003b40:0098e3bd:00000000:00000001:00000000
> virt_caps              : hvm
> total_memory           : 8183
> free_memory            : 1747
> sharing_freed_memory   : 0
> sharing_used_memory    : 0
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 2
> xen_extra              : .1
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> xen_commandline        : dom0_mem=768M tmem tmem_compress tmem_dedup iommu=1 
> dom0_vcpus_pin dom0_max_vcpus=2 numa
> cc_compiler            : gcc (Alpine 4.7.2-r2) 4.7.2
> cc_compile_by          : buildozer
> cc_compile_domain      : [unknown]
> cc_compile_date        : Tue Dec 18 12:48:52 UTC 2012
> xend_config_format     : 4
>
>
> Would love if someone helped me a little with this. :)
> (And would love if those VMs hadn't blown up in the first place tehehe)
>
>
> Greetings & merry Xmas if applicable,
> Florian
>
>



-- 
the purpose of libvirt is to provide an abstraction layer hiding all
xen features added since 2006 until they were finally understood and
copied by the kvm devs.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users