Re: [Xen-devel] Degregated I/O Performance since 3.4 - Regression in 3.4?

[Tobias Geiger <tobias.geiger@xxxxxxxxx>]

Am Dienstag, 24. April 2012, 09:27:42 schrieb Jan Beulich:
> >>> On 23.04.12 at 22:53, Tobias Geiger <tobias.geiger@xxxxxxxxx> wrote:
> > Am 23.04.2012 17:24, schrieb Konrad Rzeszutek Wilk:
> >> On Mon, Apr 23, 2012 at 12:53:03PM +0100, Stefano Stabellini wrote:
> >>> On Mon, 23 Apr 2012, Tobias Geiger wrote:
> >>>> Hello!
> >>>> 
> >>>> i noticed a considerable drop in I/O Performance when using 3.4 (rc3
> >>>> and rc4 tested) as Dom0 Kernel;
> >>>> 
> >>>> With 3.3 i get over 100mb/s in a HVM DomU (win64) with PV Drivers
> >>>> (gplpv_Vista2008x64_0.11.0.357.msi);
> >>>> With 3.4 it drops to about a third of that.
> >>>> 
> >>>> Xen Version is xen-unstable:
> >>>> xen_changeset          : Tue Apr 17 19:13:52 2012 +0100
> >>>> 25209:e6b20ec1824c
> >>>> 
> >>>> Disk config line is:
> >>>> disk = [ '/dev/vg_ssd/win7system,,hda' ]
> >>>> - it uses blkback.
> >>> 
> >>> I fail to see what could be the cause of the issue: nothing on the
> >>> blkback side should affect performances significantly.
> >>> You could try reverting the four patches to blkback that were applied
> >>> between 3.3 and 3.4-rc3 just to make sure it is not a blkback
> >>> regression:
> >>> 
> >>> $ git shortlog v3.3..v3.4-rc3 drivers/block/xen-blkback
> >>> 
> >>> Daniel De Graaf (2):
> >>>        xen/blkback: use grant-table.c hypercall wrappers
> >> 
> >> Hm.. Perhaps this patch fixes it a possible perf (I would think that
> >> the compiler would have kept the result of the first call to vaddr(req,
> >> i) somewhere.. but not sure) lost with the mentioned patch:
> >> 
> >> diff --git a/drivers/block/xen-blkback/blkback.c
> > 
> > b/drivers/block/xen-blkback/blkback.c
> > 
> >> index 73f196c..65dbadc 100644
> >> --- a/drivers/block/xen-blkback/blkback.c
> >> +++ b/drivers/block/xen-blkback/blkback.c
> >> @@ -327,13 +327,15 @@ static void xen_blkbk_unmap(struct pending_req
> >> *req)
> >> 
> >>    int ret;
> >>    
> >>    for (i = 0; i<  req->nr_pages; i++) {
> >> 
> >> +          unsigned long addr;
> >> 
> >>            handle = pending_handle(req, i);
> >>            if (handle == BLKBACK_INVALID_HANDLE)
> >>            
> >>                    continue;
> >> 
> >> -          gnttab_set_unmap_op(&unmap[invcount], vaddr(req, i),
> >> +          addr = vaddr(req, i);
> >> +          gnttab_set_unmap_op(&unmap[invcount], addr,
> >> 
> >>                                GNTMAP_host_map, handle);
> >>            
> >>            pending_handle(req, i) = BLKBACK_INVALID_HANDLE;
> >> 
> >> -          pages[invcount] = virt_to_page(vaddr(req, i));
> >> +          pages[invcount] = virt_to_page(addr);
> >> 
> >>            invcount++;
> >>    
> >>    }
> >>    
> >>>        xen/blkback: Enable blkback on HVM guests
> >>> 
> >>> Konrad Rzeszutek Wilk (2):
> >>>        xen/blkback: Squash the discard support for 'file' and 'phy'
> >>>        type. xen/blkback: Make optional features be really optional.
> >>> 
> >>> _______________________________________________
> >>> Xen-devel mailing list
> >>> Xen-devel@xxxxxxxxxxxxx
> >>> http://lists.xen.org/xen-devel
> > 
> > that made it even worse :)
> > Write Performance is down to about 7mb/s (with 3.3: ~130mb/s)
> > Read "only" down to 40mb/s (with 3.3: ~140mb/s)
> 
> I doubt this patch can have any meaningful positive or negative
> performance effect at all - are you sure you're doing comparable
> runs? After all this is all just about a few arithmetic operations
> and an array access, which I'd expect to hide in the noise.
> 
> Jan

I redid the test; 

a) with 3.3.0 kernel 
b) with 3.4.0-rc4
c) with 3.40-rc4 and above patch

everything else remained the same, i.e. test-program and test-scenario was not 
changed and started after about 5min of domu bootup (so that no strange 
bootup-effects become relevant); same phy-backend (lvm on ssd), same everything 
else; so i cant see what else except the used dom0 kernel is causing this 
issue; but here are the numbers:

a) read: 135mb/s write: 142mb/s
b) read: 39mb/s  write: 39mb/s
c) read: 40mb/s  write: 40mb/s

Only thing that may become relevant is the difference in kernel-config betwen 
3.3 and 3.4 - here's the diff :
http://pastebin.com/raw.php?i=Dy71Fegq

Jan, it seems you're right: The patch doesn't add extra performance regression 
- i guess i had an i/o intensive task running in dom0 while doing the 
benchmark yesterday, so that the write performance got so bad. sorry for that.

Still there's a significant performance penalty from 3.3 to 3.4 

Greetings
Tobias





_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel