Re: [Xen-users] Strange failures of Xen 4.3.1, PVHVM storage VM, iSCSI and Windows+GPLPV VM combination

[Adam Goryachev <mailinglists@xxxxxxxxxxxxxxxxxxxxxx>]

On 31/01/14 12:39, James Harper wrote:
> > A 1Gbps ethernet provides 100MB/s (actually closer to 130MB/s), so
> > simply bonding 2 x 1Gbps ethernet can usually provide more disk
> > bandwidth than the disks can provide.
> >
> > In my setup, the iSCSI server uses 8 x 1Gbps ethernet bonded, and the
> > xen machines use 2 x 1Gbps bonded for iSCSI, plus 1 x 1Gbps which is
> > bridged to the domU's and for dom0 "management". You can get a couple of
> > cheap-ish 1Gbps network cards easily enough, and your disk subsystem
> > probably won't provide more than 200MB/s anyway (we can get a max of
> > 2.5GB/s read from the disk subsystem, but the limited bandwidth for each
> > dom0 helps to stop any one domU from stealing all the disk IO). In
> > practice, I can run 4 x dom0 and obtain over 180MB/s on each of them in
> > parallel.
> If you workload is only iSCSI then can you comment on the choice of bonding 
> instead of multipath?
>
> James
I use both bonding (on the iSCSI server to join the 8 x 1Gbps) and 
multipath (on the dom0, one path on each interface). That way the server 
side can load balance data over any of the 8 ports, and the clients can 
load balance over their two ports.
There is probably additional tweaking I could do to improve performance 
in all cases, but it has been stable and works, and there are other 
(more pressing) demands on time (that very limited resource).
In actual fact, there are two identical iSCSI server, each with 5 x 
480GB Intel SSD. They each have 8 x 1Gbps ethernet bonded for their 
iSCSI, plus 1 x 1Gbps for a crossover for DRBD to sync the content. An 
additional 1 x 1Gbps is for management. The iSCSI servers use heartbeat, 
and will failover within about 5 seconds. The dom0 uses multipath, and 
if it gets errors on both paths will wait forever until success (which 
means windows sees the SCSI layer as being non-responsive). In practice, 
the failover time is transparent to windows, and works really well.
Of course, a "nice" failover takes less than one second, and has no 
noticeable impact from users etc (ie, if you aren't looking for the 
pause, then you won't notice).
This has the added advantage of being able to live migrate domU's from 
one physical host to another, which also works well.
The only thing I don't like about the current setup is that LVM layer 
(above DRBD and below iSCSI) seems to massively degrade performance.
If you want more details, please ask, but I may have forgotten some of 
the details because it was setup over 9 months ago. You can see a lot of 
the discussions on the linux-raid mailing list where I got a lot of 
assistance getting the networking and disk layers behaving properly (see 
the period from January 2013 when debugging started to around April 2013 
when I mostly finished).
(PS, thanks to all the users on the linux-raid list, esp Stan Hoeppner 
for all your help!)
Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users