Re: [Xen-API] nova-xcp-network plugin searching for a xapi0 interface

[John Garbutt <John.Garbutt@xxxxxxxxxx>]

I have tried to throw together some pictures that help describe the guest 
networking modes with XenServer:
 http://wiki.openstack.org/XenServer/NetworkingFlags

The idea is to complement the existing diagrams currently in the manuals.

It would be great if someone can review these diagrams for me.
I can then maybe look at polishing them, and getting them into the manuals.

Thanks,
John

> -----Original Message-----
> From: Salvatore Orlando
> Sent: 21 March 2012 00:51
> To: John Garbutt; 'Thomas Goirand'
> Cc: PKG OpenStack; Dave Scott; openstack@xxxxxxxxxxxxxxxxxxx; xen-api
> Subject: RE: [Xen-API] nova-xcp-network plugin searching for a xapi0
> interface
> 
> Hi Thomas,
> 
> I can probably help you somehow with Openstack networking on XenServer,
> as I did some work on it in the past.
> I see you are trying to use the VLAN manager, but the behaviour is not the
> expected one.
> However, since you can spin up instances, and they appear to be attached to
> the appropriate bridge, I'd dare to say you're very close!
> 
> In a nutshell, when using the VLAN manager on XenServer:
> 1) A PIF identified either by (A) the vlan_interface flag or (B) the
> bridge_interface column  in the networks db table will be used for creating a
> XenServer VLAN network.
>      The VLAN tag is found in the vlan column, still in the networks table, 
> and
> by default the first tag is 100, if my memory does not fail me.
> 2) VIF for VM instances within this network will be plugged in this VLAN
> network. As you said, you won't see the bridge until a VIF is plugged in it. 
> This
> behaviour is the same in XS, XCP, and Kronos.
> 3) The 'Openstack domU', i.e. the VM running the nova network node,
> instead will not be plugged into this network; since it acts as a gateway for
> multiple VLAN networks, it has to be attached on a VLAN trunk. For this
> reason it must have an interface on the parent bridge of the VLAN bridge
> where VM instances are plugged. I realized this is quite obscure, so to cut a
> long story short, if vlan_interface is eth0 it must be plugged in xenbr0, 
> eth1 -->
> xenbr1, and so on (on Kronos you might also end up with brwlan0).
> 4) Within the Openstack domU, 'ip link' is then used to configure VLAN
> interfaces on the 'trunk' port. Each of this vlan interfaces is associated 
> with a
> dnsmasq instance, which will distribute IP addresses to instances. The lease
> file for dnsmasq is constantly updated by nova-network, thus ensuring VMs
> get the IP address specified by the
> 
> With this configuration, VM instances should be able to get the IP address
> assigned to them from the appropriate dnsmasq instance, and should be
> able to communicate without any problem with other VMs on the same
> network and with the their gateway.
> The above point (3) probably needs some more explanations. With Open
> vSwitch, we don't really have distinct bridges for different VLANs; even if
> they appear as distinct bridges to linux and xen server, they are actually the
> same OVS instance, which runs a distinct 'fake-bridge' for each VLAN. The
> 'real' bridge is the 'parent' of the fake one. You can easily navigate fake 
> and
> real bridges with ovs-vsctl.
> As you can see I am referring to Openvswitch only. This is for a specific
> reason: the fake-parent mechanism automatically imply that ports which are
> not on a fake bridge are trunk ports. This does not happen with linux bridge.
> A packet forwarded on a VLAN interfaces does not get back in the xenbrX
> bridge for ethX.
> 
> So, coming back to your problem I would check that:
> 1) The XenServer network whose bridge is xapi0 is configured correctly
> (check PIF, VLAN tag)
> 2) The Openstack domU is connected to the appropriate bridge according to
> the value of vlan_interface (which seems wrong in your conf file)
> 3) Open vSwitch is enabled
> 4) Check the networks table in your database
> 
> I hope I have been exhaustive enough to not become pedant... At this point
> you might wonder why this has not been documented anywhere.
> Well, my answer is that it was documented, I am very sure it was. However I
> cannot find the wiki page anymore. I have the sources on my laptop, and I
> will make sure that VLAN networking and possibly all the other network
> managers on xenapi backend are documented; including Quantum.
> 
> Regards,
> Salvatore
> 
> 
> > -----Original Message-----
> > From: openstack-
> > bounces+salvatore.orlando=eu.citrix.com@xxxxxxxxxxxxxxxxxxx
> > [mailto:openstack-
> > bounces+salvatore.orlando=eu.citrix.com@xxxxxxxxxxxxxxxxxxx] On Behalf
> > bounces+Of
> > John Garbutt
> > Sent: 19 March 2012 18:01
> > To: 'Thomas Goirand'
> > Cc: PKG OpenStack; Dave Scott; openstack@xxxxxxxxxxxxxxxxxxx; xen-api
> > Subject: Re: [Openstack] [Xen-API] nova-xcp-network plugin searching
> > for a
> > xapi0 interface
> >
> > Hi,
> >
> > Looks like the network configuration is not quite right.
> >
> > Have a look at this, for an example of how things could look
> > networking wise (when using DevStack and XenServer with two nics):
> > http://wiki.openstack.org/XenServer/XenXCPAndXenServer
> >
> > The manuals have a good description, although it is a little KVM specific:
> > http://docs.openstack.org/trunk/openstack-
> > compute/admin/content/configuring-flat-dhcp-networking.html
> >
> > I suggested using DevStack because it is the best "documentation" for
> > a working set of flags right now (yes, not ideal, we must fix that
> > asap!). Take a
> > look:
> > https://github.com/openstack-dev/devstack/blob/master/stack.sh#L291
> > Note the defaults in the nova code might work for KVM, but will not
> > work for XenServer, so you will need to set those flags with more
> appropriate values.
> >
> > Hope that helps,
> > John
> >
> > -----Original Message-----
> > From: Thomas Goirand [mailto:thomas@xxxxxxxxxx]
> > Sent: 19 March 2012 5:25
> > To: John Garbutt
> > Cc: Dave Scott; xen-api; PKG OpenStack; openstack@xxxxxxxxxxxxxxxxxxx
> > Subject: Re: [Xen-API] nova-xcp-network plugin searching for a xapi0
> > interface
> >
> > On 03/20/2012 12:05 AM, John Garbutt wrote:
> > > This sounds a lot like a configuration issue with your guest network 
> > > bridge.
> > Are you OK to supply your nova.conf file?
> >
> > Sure! Here it is:
> >
> > --sql_connection=mysql://nova:admin@xxxxxxxxx/nova
> > --novncproxy_base_url=http://<domU-IP>:6080/vnc_auto.html
> > --rabbit_host=<domU-IP>
> > --glance_api_servers=<domU-IP>:9292
> > --network_manager=nova.network.manager.VlanManager
> > --vlan_interface=br100
> > --dhcpbridge=/usr/bin/nova-dhcpbridge
> > --firewall_driver=nova.virt.xenapi.firewall.Dom0IptablesFirewallDriver
> > --connection_type=xenapi
> > --xenapi_connection_url=https://<dom0-ip>
> > --xenapi_connection_username=root
> > --xenapi_connection_password=XXXXXXX
> > --reboot_timeout=600
> > --rescue_timeout=86400
> > --resize_confirm_window=86400
> > --auth_strategy=keystone
> > --allow_admin_api
> > --allow_resize_to_same_host
> > --logdir=/var/log/nova
> > --state_path=/var/lib/nova
> > --lock_path=/var/lock/nova
> > --vncserver_listen=0.0.0.0
> > --force_dhcp_release
> > --use_deprecated_auth
> > --use_project_ca
> > --verbose
> >
> > Note that since I restarted xcp-xapi, it seems that the xapi0 bridge
> > is created each time I create a VM, which is what I was expecting. So
> > that parts seems to work now, but the issue is that now, I have no
> > networking at all (eg: I can't reach the instance from the Openstack domU).
> >
> > Here's what "xe network-list" returns on my dom0:
> >
> > uuid ( RO)                : <my-uuid>
> >           name-label ( RW): br100
> >     name-description ( RW): network for nova bridge br100
> >               bridge ( RO): xapi0
> >
> > So it really is nova who created this. But shouldn't it be linked to
> > my
> > xenbr0 as well? What is the normal networking setup that should be done?
> >
> > There's absolutely zero documentation which I could find about this.
> > Of course, I'll write one directly in our Debian packages as soon as this
> works.
> >
> > > Also, have you tried using DevStack first? It is a good way to get
> > > the hang of
> > how the flags work.
> >
> > No it's not! DevStack is for testing with XenServer, and assumes that
> > you'd be working with Ubuntu. Here, I'm testing the Debian packages
> > that we are working on in Debian. Please don't direct me to DevStack, this
> wont help.
> >
> > > I'm not familiar with the design of the Nova plugins but my guess is
> > > that
> > they're trying to use the "Host internal management network" -- a
> > network from which VMs can access the XenAPI (i.e. talk to xapi in domain
> 0).
> > >
> > > On recent XCP versions the default bridge name changed to "xenapi"
> > > to
> > make it easier to distinguish from a firewall rules PoV.
> >
> > As per Mike on IRC, (from Citrix, upstream for XCP, XenServer and
> > Kronos), each time there's a new network being created, XCP create a
> > xapiX bridge automatically. I shouldn't have even try to bring the
> > xapi0 bridge myself, XCP does it (if it doesn't, then there's an issue).
> >
> > > Note that the actual linux bridge (or openvswitch) is created on
> > > demand so
> > you won't see it in "brctl show" or "ovs-vsctl show" until it's being used.
> > >
> > > Try changing "xapi0" to "xenapi" in the plugins and see if they work 
> > > better?
> >
> > Nop, that wont help, xapi0 is really what to expect. My only issue now
> > is to have it connected to the br100 of Openstack in my domU.
> >
> > Thanks for the help already,
> > Cheers,
> >
> > Thomas
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp

_______________________________________________
xen-api mailing list
xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api