Re: [Xen-API] nova-xcp-network plugin searching for a xapi0 interface

[Salvatore Orlando <Salvatore.Orlando@xxxxxxxxxxxxx>]

Hi Thomas, 

I can probably help you somehow with Openstack networking on XenServer, as I 
did some work on it in the past.
I see you are trying to use the VLAN manager, but the behaviour is not the 
expected one.
However, since you can spin up instances, and they appear to be attached to the 
appropriate bridge, I'd dare to say you're very close!

In a nutshell, when using the VLAN manager on XenServer:
1) A PIF identified either by (A) the vlan_interface flag or (B) the 
bridge_interface column  in the networks db table will be used for creating a 
XenServer VLAN network. 
     The VLAN tag is found in the vlan column, still in the networks table, and 
by default the first tag is 100, if my memory does not fail me.
2) VIF for VM instances within this network will be plugged in this VLAN 
network. As you said, you won't see the bridge until a VIF is plugged in it. 
This behaviour is the same in XS, XCP, and Kronos.
3) The 'Openstack domU', i.e. the VM running the nova network node, instead 
will not be plugged into this network; since it acts as a gateway for multiple 
VLAN networks, it has to be attached on a VLAN trunk. For this reason it must 
have an interface on the parent bridge of the VLAN bridge where VM instances 
are plugged. I realized this is quite obscure, so to cut a long story short, if 
vlan_interface is eth0 it must be plugged in xenbr0, eth1 --> xenbr1, and so on 
(on Kronos you might also end up with brwlan0).  
4) Within the Openstack domU, 'ip link' is then used to configure VLAN 
interfaces on the 'trunk' port. Each of this vlan interfaces is associated with 
a dnsmasq instance, which will distribute IP addresses to instances. The lease 
file for dnsmasq is constantly updated by nova-network, thus ensuring VMs get 
the IP address specified by the 

With this configuration, VM instances should be able to get the IP address 
assigned to them from the appropriate dnsmasq instance, and should be able to 
communicate without any problem with other VMs on the same network and with the 
their gateway.
The above point (3) probably needs some more explanations. With Open vSwitch, 
we don't really have distinct bridges for different VLANs; even if they appear 
as distinct bridges to linux and xen server, they are actually the same OVS 
instance, which runs a distinct 'fake-bridge' for each VLAN. The 'real' bridge 
is the 'parent' of the fake one. You can easily navigate fake and real bridges 
with ovs-vsctl. 
As you can see I am referring to Openvswitch only. This is for a specific 
reason: the fake-parent mechanism automatically imply that ports which are not 
on a fake bridge are trunk ports. This does not happen with linux bridge. A 
packet forwarded on a VLAN interfaces does not get back in the xenbrX bridge 
for ethX. 

So, coming back to your problem I would check that:
1) The XenServer network whose bridge is xapi0 is configured correctly (check 
PIF, VLAN tag)
2) The Openstack domU is connected to the appropriate bridge according to the 
value of vlan_interface (which seems wrong in your conf file)
3) Open vSwitch is enabled 
4) Check the networks table in your database

I hope I have been exhaustive enough to not become pedant... At this point you 
might wonder why this has not been documented anywhere.
Well, my answer is that it was documented, I am very sure it was. However I 
cannot find the wiki page anymore. I have the sources on my laptop, and I will 
make sure that VLAN networking and possibly all the other network managers on 
xenapi backend are documented; including Quantum.

Regards,
Salvatore


> -----Original Message-----
> From: openstack-
> bounces+salvatore.orlando=eu.citrix.com@xxxxxxxxxxxxxxxxxxx
> [mailto:openstack-
> bounces+salvatore.orlando=eu.citrix.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of
> John Garbutt
> Sent: 19 March 2012 18:01
> To: 'Thomas Goirand'
> Cc: PKG OpenStack; Dave Scott; openstack@xxxxxxxxxxxxxxxxxxx; xen-api
> Subject: Re: [Openstack] [Xen-API] nova-xcp-network plugin searching for a
> xapi0 interface
> 
> Hi,
> 
> Looks like the network configuration is not quite right.
> 
> Have a look at this, for an example of how things could look networking wise
> (when using DevStack and XenServer with two nics):
> http://wiki.openstack.org/XenServer/XenXCPAndXenServer
> 
> The manuals have a good description, although it is a little KVM specific:
> http://docs.openstack.org/trunk/openstack-
> compute/admin/content/configuring-flat-dhcp-networking.html
> 
> I suggested using DevStack because it is the best "documentation" for a
> working set of flags right now (yes, not ideal, we must fix that asap!). Take 
> a
> look:
> https://github.com/openstack-dev/devstack/blob/master/stack.sh#L291
> Note the defaults in the nova code might work for KVM, but will not work for
> XenServer, so you will need to set those flags with more appropriate values.
> 
> Hope that helps,
> John
> 
> -----Original Message-----
> From: Thomas Goirand [mailto:thomas@xxxxxxxxxx]
> Sent: 19 March 2012 5:25
> To: John Garbutt
> Cc: Dave Scott; xen-api; PKG OpenStack; openstack@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Xen-API] nova-xcp-network plugin searching for a xapi0
> interface
> 
> On 03/20/2012 12:05 AM, John Garbutt wrote:
> > This sounds a lot like a configuration issue with your guest network bridge.
> Are you OK to supply your nova.conf file?
> 
> Sure! Here it is:
> 
> --sql_connection=mysql://nova:admin@xxxxxxxxx/nova
> --novncproxy_base_url=http://<domU-IP>:6080/vnc_auto.html
> --rabbit_host=<domU-IP>
> --glance_api_servers=<domU-IP>:9292
> --network_manager=nova.network.manager.VlanManager
> --vlan_interface=br100
> --dhcpbridge=/usr/bin/nova-dhcpbridge
> --firewall_driver=nova.virt.xenapi.firewall.Dom0IptablesFirewallDriver
> --connection_type=xenapi
> --xenapi_connection_url=https://<dom0-ip>
> --xenapi_connection_username=root
> --xenapi_connection_password=XXXXXXX
> --reboot_timeout=600
> --rescue_timeout=86400
> --resize_confirm_window=86400
> --auth_strategy=keystone
> --allow_admin_api
> --allow_resize_to_same_host
> --logdir=/var/log/nova
> --state_path=/var/lib/nova
> --lock_path=/var/lock/nova
> --vncserver_listen=0.0.0.0
> --force_dhcp_release
> --use_deprecated_auth
> --use_project_ca
> --verbose
> 
> Note that since I restarted xcp-xapi, it seems that the xapi0 bridge is 
> created
> each time I create a VM, which is what I was expecting. So that parts seems
> to work now, but the issue is that now, I have no networking at all (eg: I 
> can't
> reach the instance from the Openstack domU).
> 
> Here's what "xe network-list" returns on my dom0:
> 
> uuid ( RO)                : <my-uuid>
>           name-label ( RW): br100
>     name-description ( RW): network for nova bridge br100
>               bridge ( RO): xapi0
> 
> So it really is nova who created this. But shouldn't it be linked to my
> xenbr0 as well? What is the normal networking setup that should be done?
> 
> There's absolutely zero documentation which I could find about this. Of
> course, I'll write one directly in our Debian packages as soon as this works.
> 
> > Also, have you tried using DevStack first? It is a good way to get the hang 
> > of
> how the flags work.
> 
> No it's not! DevStack is for testing with XenServer, and assumes that you'd
> be working with Ubuntu. Here, I'm testing the Debian packages that we are
> working on in Debian. Please don't direct me to DevStack, this wont help.
> 
> > I'm not familiar with the design of the Nova plugins but my guess is that
> they're trying to use the "Host internal management network" -- a network
> from which VMs can access the XenAPI (i.e. talk to xapi in domain 0).
> >
> > On recent XCP versions the default bridge name changed to "xenapi" to
> make it easier to distinguish from a firewall rules PoV.
> 
> As per Mike on IRC, (from Citrix, upstream for XCP, XenServer and Kronos),
> each time there's a new network being created, XCP create a xapiX bridge
> automatically. I shouldn't have even try to bring the xapi0 bridge myself, XCP
> does it (if it doesn't, then there's an issue).
> 
> > Note that the actual linux bridge (or openvswitch) is created on demand so
> you won't see it in "brctl show" or "ovs-vsctl show" until it's being used.
> >
> > Try changing "xapi0" to "xenapi" in the plugins and see if they work better?
> 
> Nop, that wont help, xapi0 is really what to expect. My only issue now is to
> have it connected to the br100 of Openstack in my domU.
> 
> Thanks for the help already,
> Cheers,
> 
> Thomas
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

_______________________________________________
xen-api mailing list
xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api