I have refined my howto a bit. I fixed a
few typos and had a few things in the wrong order.
Although they work, I still have a few
questions on these steps.
1) Do we need to create a VIF on DOM0 in order to set up the
management interface? I am starting to think not.
2) When setting up the VLAN should I connect to the BOND (as I
indicated below) or to the network that initially gets connected to the bond?
Any help on these would be great.
Thanks;
James
While I am still in the process of working
through this here are the steps I am taking to set up each machine.
1) Since each of my servers have multiple NIC ports I bond them
together (see page 34 of the Administrators Guide) This is best done from the
physical server and not via the console. If you do it remotely you will then
have to reconfigure the management port before the next step. I think it is
cleaner if you do not have to do that.
a. Shutdown all VM’s (this is easy since I am pretty much a new
installation)
i.
xe
vm-list
ii.
xe
vm-shutdown uuid=
b. Create the network (this is like a virtual switch), write down the
first part of the UUID that is returned after this command since the next step
may cause it to scroll off the page
i.
xe
network-create name-label=bond0
c. Create the actual bond (keep track of the uuid of this bond since
it will be used in step 2)
i.
xe
pif-list
ii.
xe bond-create
network-uuid=<UUID from above> pif-uuids=<UUID of the first interface
from the last step>,<UUID of the second interface from the last step>
2) Next for the VLAN work. First, of course, each NIC port must be on
a trunked switch port. IN our case, we forced the port encapsulation to dot1q
and the mode to trunk with a native VLAN of 1. We did this using ranges so that
we know everything is configured the same. I am not sure if allowing the ports
to negotiate the encapsulation or making the trunk mode dynamic would have
worked but, we were not taking any chances. This way we had two fewer things to
troubleshoot if we had problems.
a. Create a new network. This will be like a VLAN specific switch for
all of your DOM’s
i.
xe
network-create name-label=vlan103
b. Next tie the network to the network interface on the physical
server (in our case, the bonded interfaces) and specify the VLAN. Keep track of
the UUID returned as you will need it when we create VIF’s for the
DOM’s
i.
xe
vlan-create network-uuid=<UUID from above> pif-uuid=<UUID of the bond
in step 1.c.ii> vlan=103
c. Reconfigure the IP address info. I do not use DHCP for my servers
so I make this static. Also, I do not know for sure how to enter multiple DNS
servers. You may just separate them with a comma but I have not tested that
yet. Finally, the case seems to be important here. I noticed it on the DNS
keyword. If it is lowercase it does not seem to work.
i.
xe pif-reconfigure-ip
uuid=<UUID of the bond created above> mode=static gateway=<GATEWAY>
IP=<IP ADDRESS> netmask=<NETMASK> DNS=<DNS SERVER ADDRESS>
3) The next step is to get the management port back up and running. I
am not sure if you have to actually set up a VIF (steps a through d. Step e may
be enough.) These are the same steps for setting up VIF’s on other
VM’s.
a. get the UUID of DOM0
i.
xe
vm-list
b. Create a VIF to tie DOM0 to the management VLAN (this is the VLAN I
created above). In this case I had one interface (BOND0) so I set the device
to 1. Keep track of this UUID as you will need it to configure the management
interface.
i.
xe
vif-create vm-uuid=<UUID of DOM0> network-uuid=<UUID of the network created
just above in 2.b.i> device=1
c. As long as a VM is not running, you can just start it and the new
interface will be ready. For DOM0 we will have to plug it in or reboot the
server.
i.
xe
vif-plug uuid=<UUID of the VIF created above>
d. Configure IP information of the VIF. First run ifconfig to get the
device and then run it again to configure it
i.
ifconfig
ii.
ifconfig
xapi2 <IP ADDRESS> netmask <NETMASK>
e. Configure the VLAN PIF as the management interface
i.
xe
host-management-reconfigure pif-uuid=<UUID of the PIF created above 2.b.i>
f.
Clean up
after yourself. This will help to keep you from getting confused later when
you look at the settings.
i.
xe
pif-reconfigure-ip uuid=<old mgmt PIF UUID> mode=None
Notes
1) The XenNetworking WiKi page helped me wrap my head around what was
going on. http://wiki.xensource.com/xenwiki/XenNetworking
2) I found this thread on the forums that basically became my
template. http://forums.xensource.com/thread.jspa?messageID=15451㱛
3) The above thread led me to the knowledge that there is more about VLANS
in the admin guide than the index would have you believe. Page 33 – 34
tells you how to set them up. I hope that this omission can be fixed in the
next document release.
4) I found that the UUID’s work with tab completion. This may
be obvious to everyone else but I never saw it mentioned in the manual (it may
be there but I never saw it).
5) One problem I had with configuring bridges manually in Linux instead
of doing it the Xensource way above, was that Xensource automatically removes
interfaces it does not know about. Since I was not sure how to make it aware
of things I had manually created, my interfaces and bridges would stay up for
30 – 60 seconds or so before being torn down.
6) Once you get the management interface setup, you can create links
to other VLANS using the XenCenter.
7) I have not tested yet but, my hope is that this information will
travel to each of the other machines I add to the resource pool. That is why I
took the above steps on the master server.
8) If you dig through this entire thread you will find a nugget of
information that sounds like it could cause problems if set incorrectly.
“The secret seems to be that the native VLAN on the trunkport must be 1
(which is the default).” http://forums.xensource.com/thread.jspa?messageID=15780
9) Before you can take all of the above steps you must install your
license. This presents a Catch-22 since you would normally install it using
the XenCenter, however, this requires the network already be working. To
install via the command line you need to:
1. copy the license to a thumbdrive, take it to the server and mount
it.
2. mv /etc/xensource/license /etc/xensource/license.old
3. copy the license from the thumb drive to /etc/xensource/license
4. If necessary, chmod a+x /etc/xensource/license to make the
permissions match the original file.
5. reboot the box
James
Alspach
Systems
Analyst II
Shasta County Office of Education
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx
[mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of James Alspach
Sent: Monday, June 09, 2008 3:37
PM
To: augusto lopes
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: RE: RE: [Xen-users] VLAN
and BRIDGE HELP
You are very welcome.
One thing that I am working on now
involves removing the step toward the beginning:
xe pif-reconfigure-ip
uuid=<UUID of the bond created above> mode=static gateway=<GATEWAY>
IP=<IP ADDRESS> netmask=<NETMASK> DNS=<DNS SERVER ADDRESS>
I do not believe that this step is
necessary since we will not use the pif directly. We will use vif’s
connected to this pif (actually to the network connected to the pif) and the
vif’s will each get their own IP addresses.
I am working on this right now and will let you know how it goes.
Thanks;
James
James
Alspach
Systems
Analyst II
Shasta County Office of Education
From: augusto lopes
[mailto:nhanonme@xxxxxxxxxxxx]
Sent: Monday, June 09, 2008 3:29
PM
To: James Alspach
Cc: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: RE: RE: [Xen-users] VLAN
and BRIDGE HELP
Thank you VERY, VERY MUCH JAMES!
This is a great detail oriented document that answers all my noobie's doubts:
now I should be able to set without issues. I will let you know how it goes for
me, but I truly have a good feeling about it now that
you provided me with this feed...
Thanks again for you help,
Augusto Lopes
Intern Systems Admin
Apollo System
James Alspach <jalspach@xxxxxxxxxxxxx> escreveu:
While
I am still in the process of working through this here are the steps I am
taking to set up each machine.
Since
each of my servers have multiple NIC ports I bond them together (see page 34 of
the Administrators Guide) first. This is best down from the physical server and
not via the remote console. If you do it remotely you will then have to
reconfigure the management port before the next step. I think it is
cleaner if you do not have to do that.
Shutdown
all VM’s (this is easy since I am pretty much a new installation)
Create
the network (this is like a virtual switch), write down the first part of the
UUID that is returned after this command since the next step may cause it to
scroll off the page
xe
network-create name-label=bond0
Create
the actual bond (keep track of the uuid of this bond since it will be used in
step 2)
xe
bond-create network-uuid=<UUID from above> pif-uuids=<UUID if the
first interface from the last step>,<UUID if the second interface from
the last step>
Reconfigure
the IP address info. I do not use DHCP for my servers so I make this
static. Also, I do not know for sure how to enter multiple DNS servers.
You may just separate them with a comma but I have not tested that yet.
Finally, the case seems to be important here. I noticed it on the DNS
keyword. If it is lowercase it does not seem to work.
xe
pif-reconfigure-ip uuid=<UUID of the bond created above> mode=static
gateway=<GATEWAY> IP=<IP ADDRESS> netmask=<NETMASK>
DNS=<DNS SERVER ADDRESS>
Next for
the VLAN work (see page 33 of the Administrators Guide). First, of
course, each NIC port must be on a trunked switch port. IN our case, we forced
the port encapsulation to dot1q and the mode to trunk with a native VLAN of 1.
We did this using ranges so that we know everything is configured the same. I
am not sure if allowing the ports to negotiate the encapsulation or making the
trunk mode dynamic would have worked but, we were not taking any chances. This
way we had two fewer things to troubleshoot if we had problems.
Create a
new network. This will be like a VLAN specific switch for all of your
DOM’s
xe
network-create name-label=vlan103
Next tie
the network to the network interface on the physical server (in our case, the
bonded interfaces) and specify the VLAN. Keep track of the UUID returned as you
will need it when we create VIF’s for the DOM’s
xe
vlan-create network-uuid=<UUID from above> pif-uuid=<UUID of the bond
in step 1.c.ii> vlan=103
The next
step is to get the management port back up and running. These are the
same steps for setting up VIF’s on other VM’s.
Create a
VIF to tie DOM0 to the management VLAN (this is the VLAN I created
above). In this case I had one interface (BOND0) so I set the device to
1. Keep track of this UUID as you will need it to configure the
management interface.
xe
vif-create vm-uuid=<UUID of DOM0> network-uuid=<UUID of the network
created just above in 2.b.i> device=1
Configure
IP information of the VIF. First run ifconfig to get the device name and
then run it again to configure it
ifconfig
eth1 <IP ADDRESS> netmask <NETMASK>
Configure
it as the management interface
xe host-management-reconfigure
pif-uuid=<UUID of the VIF created above>
Clean up
after yourself. This will help to keep you from getting confused later
when you look at the settings.
xe
pif-reconfigure-ip uuid=<old mgmt PIF UUID> mode=None
As long
as a VM is not running, you can just start it and the new interface will be
ready. For DOM0 we will have to plug it in.
xe
vif-plug uuid=<UUID of the VIF created above>
The
above thread led me to the knowledge that there is more about VLANS in the
admin guide than the index would have you believe. Page 33 – 34 tells you
how to set them up. I hope that this omission can be fixed in the next
document release.
I found
that the UUID’s work with tab completion from the command line.
This may be obvious to everyone else but I never saw it mentioned in the manual
(it may be there but I never saw it).
One
problem I had with configuring bridges manually in Linux instead of doing it
the Xensource way above was that Xensource automatically removes interfaces it
does not know about. Since I was not sure how to make it aware of things
I had manually created, my interfaces and bridges would stay up and running for
30 – 60 seconds or so before being torn down.
Once you
get the management interface setup, you can create links to other VLANS using
the XenCenter and skip much of the above.
I have not
tested yet but, my hope is that this information will travel to each of the
other machines I add to the resource pool. That is why I took the above steps
on the master server.
I also attached the above info as a PDF. If anyone sees any
problems with what I have described or better ways to go about it, please let
me know so that I can update this in the hope that it helps somebody somewhere.
P.S. Thank you for everyone who provided suggestions and help while I was
(and still am) trying to figure this all out.
Systems
Analyst II
Shasta County Office of Education
From: augusto lopes
[mailto:nhanonme@xxxxxxxxxxxx]
Sent: Monday, June 09, 2008 9:17
AM
To: James Alspach
Subject: En: RE: [Xen-users] VLAN
and BRIDGE HELP
Hello James;
I was just checking all the recent help emails and found this particular one
which is basically referring to a similar environment I would like to set up.
Basically, in my scenario I am asked to set up three guest domains (domU1 -
domU3). The first tow will provide web and mail services sequentially. And the
last one will basically be the DBM server.
Since in a normal network environment domU3 should be on a protected subnet and
the other tow on a dmz subnet, I have not been able to set up VLANs
appropriately for them. I have practically set up netfilter (iptables firewall
) on the Dom0, but do not understand VLAN concept in xen's virtual environment
well enough to accomplish the main goal of separating each service to its own
domain.
After reading your help, I can picture the all thing better, but I am still not
sure how to bring it all about. Would you please give me some ropes regarding
this topic?
I am working with RHEL5 as the Dom0 and guest domains will be various win
server 2k3 as well as rhel5.
Thank you in advance for you help.
Augusto Lopes
Intern System Admin
Apollo System
James Alspach <jalspach@xxxxxxxxxxxxx>
escreveu:
Data: Thu, 5 Jun 2008
09:24:04 -0700
De: "James Alspach" <jalspach@xxxxxxxxxxxxx>
Para: "Emil Palm"
<empa@xxxxxxxxxx>,
<xen-users@xxxxxxxxxxxxxxxxxxx>
Assunto: RE: [Xen-users] VLAN help
Thank you! This gives me another avenue to
explore.
So what you are saying is that one does
not have to explicitly create the new interface on the VLAN by making changes
in /etc/sysconfig/network-scripts/ ? By configuring the vlan in vconfig
it builds the new interface automagically? Will this setup survive a
reboot or should I build a script to set this up each time?
In my case I have bonded two NICs (well
actually two ports on the same NIC but no need to split hairs) so I would
imagine I would replace eth0 in your example with bond0.
The host OS I am using is, I believe,
CentOS. I am using the Xen Enterprise version installed right off of the
CD which uses some RedHat derived distro.
Systems
Analyst II
Shasta County Office of Education
From: Emil Palm
[mailto:empa@xxxxxxxxxx]
Sent: Thursday, June 05, 2008
12:35 AM
To: James Alspach; xen-users@xxxxxxxxxxxxxxxxxxx
Subject: RE: [Xen-users] VLAN help
I'v just recently set that kind of eviroment up. What Host
OS are you running becouse i wrote 2 diffrent howtos for Red Hat and one for
SuSe.
But in theory you just have 1 interface for ex: eth0
then you do vconfig add eth0 VLANID
when that is done you create a bridge something like this:
brctl addbr brVLANID
brctl addif brVLANID eth0.VLANDID
when that is setup you should change
"network-script=network-bridge" to "network-script=network-dummy"
within your xend-config.sxp so Xend doesnt screw up your real physical
interface.
When that is done just put:
vif=["mac=XX:XX:XX:XX:XX:XX,bridge=brVLANID",] in your domU
configuration file.
If you want more info just give me a mail and i will help
you as the best as i can.
-----Original message-----
From:
James Alspach <jalspach@xxxxxxxxxxxxx>
Sent:
Wed 06/04/08 19:56:40
To:
xen-users@xxxxxxxxxxxxxxxxxxx;
Subject:
[Xen-users] VLAN help
We are in the process of setting up a few
Xensource servers whose initial function will be to run Exchange 2007. As
part of this (and for future VM’s) I need to be able to provide access to
various VLANS to the various DOM’s.
In theory this sounds fairly straight forward:
DOM0 gets a PIF for each VLAN. This PIF connects to a VLAN specific
bridge and then, for each DOM that needs one, a VIF is created and connected to
the bridge.
If so, my question is how to specify the VLAN
for a PIF. I can list it but I am not able to set it since it is read
only.
How do virtual networks fit into the above and
how is a virtual network different from a virtual bridge?
Any help or pointers to information are greatly
appreciated.
James
Alspach
Systems Analyst II
Shasta
County Office of
Education
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
Abra sua conta no Yahoo!
Mail, o único sem limite de espaço para armazenamento!
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users
Abra sua conta no Yahoo!
Mail, o único sem limite de espaço para armazenamento!
