|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] Poor HVM performance with 8 vcpus
Tim Deegan wrote: > At 10:40 +0100 on 07 Oct (1254912041), Juergen Gross wrote: >> calls to shadow_alloc 438 427 424 480 436 >> 422 >> number of shadow pages in use 2765 2151 2386 2509 4885 >> 1391 >> calls to shadow_free 168 132 185 144 181 >> 105 >> calls to shadow_fault 65271 69132 60495 53756 73363 >> 52449 >> shadow_fault fast path n/p 7347 8081 6713 6134 8521 >> 6112 >> shadow_fault fast path error 14 12 15 3 13 >> 11 >> shadow_fault really guest fault 24004 25723 22815 19709 27049 >> 19190 >> shadow_fault emulates a write 1045 949 1018 995 1015 >> 901 >> shadow_fault fast emulate 424 361 449 348 387 >> 314 >> shadow_fault fixed fault 32503 34264 29624 26689 36641 >> 26096 >> calls to shadow_validate_gl2e 875 748 917 731 795 >> 667 >> calls to shadow_validate_gl3e 481 456 443 491 489 >> 446 >> calls to shadow_validate_gl4e 104 97 95 112 105 >> 95 >> calls to shadow_hash_lookup 2109654 2203254 2228896 2245849 2164727 >> 2309059 >> shadow hash hit in bucket head 2012828 2111164 2161113 2177591 2104099 >> 2242458 >> shadow hash misses 851 840 841 910 852 >> 838 >> calls to get_shadow_status 2110031 2202828 2228769 2246689 2164213 >> 2309241 >> calls to shadow_hash_insert 438 436 428 481 437 >> 430 >> calls to shadow_hash_delete 168 150 185 154 202 >> 128 >> shadow removes write access 335 324 329 385 330 >> 336 >> shadow writeable: linux high 130 139 152 155 138 >> 149 >> shadow writeable: sl1p 14508 15402 12961 11823 16474 >> 11472 >> shadow writeable brute-force 205 185 177 230 192 >> 187 >> shadow unshadows for fork/exit 9 12 12 12 18 >> 12 >> shadow unshadows a page 10 13 13 13 19 >> 13 >> shadow walks guest tables 647527 727336 649397 646601 659655 >> 621289 >> shadow checks gwalk 526 544 535 550 614 >> 554 >> shadow flush tlb by rem wr perm 235 233 229 268 238 >> 237 >> shadow emulates invlpg 14688 15499 14604 12630 16627 >> 11370 >> shadow OOS fixup adds 14467 15335 13059 11840 16624 >> 11339 >> shadow OOS unsyncs 14467 15335 13058 11840 16624 >> 11339 >> shadow OOS evictions 566 449 565 369 589 >> 336 >> shadow OOS resyncs 14510 15407 12964 11828 16478 >> 11481 >> >> I don't think the "shadow writable brute-force" is the problem. >> get_shadow_status seems to be a more critical candidate. > > get_shadow_status is a simple hash lookup to find the shadow of a frame; > it's expected to happen multiple times per pagefault. Even so those > numbers look high. ~10k guest PT walks per CPU per second, each causing > 3-4 shadow hash lookups. That's much higher than the number of > pagefaults. > > I take it you reset the performance counters at the start of that run? Yes. > > Are there any other numbers (outside the shadow stats) that are up > around 600k/cpu? No. All other counts are below 100k. > > I wonder whether this is caused by pagetables changing under out feet in > the shadow fault handler -- in order to avoid taking the shadow lock too > often, the fault handler walks the pagetables first, then takes the lock > and double-checks its work. If the other cpus are aggressively writing > to the pagetables that this CPU is running on that could cause the > pagefault handler to spin, locking and unlocking the shadow lock as it > goes. I would expect to see the OOS unsyncs number much higher if that > was the case though. > > Can you try the attached patch and see if it makes any difference? Not really. Still 4 vcpus waiting at the shadow lock, performance counters as follows (test ran with 8 vcpus this time, performance was 2% higher than without patch, but uncertainty is in the same order). I copied only the counters for 1 cpu, as the others are similar. Counters are smaller as before, because performance was less as with 6 vcpus and lock profiling costs about 20% performance with this lock load (the previous measurement had no lock profiling support). calls to shadow_alloc 700 number of shadow pages in use 1986 calls to shadow_free 48 calls to shadow_fault 33508 shadow_fault fast path n/p 3785 shadow_fault fast path error 10 shadow_fault really guest fault 11290 shadow_fault emulates a write 1313 shadow_fault fast emulate 257 shadow_fault fixed fault 16853 calls to shadow_validate_gl2e 604 calls to shadow_validate_gl3e 786 calls to shadow_validate_gl4e 179 calls to shadow_hash_lookup 1079773 shadow hash hit in bucket head 1030265 shadow hash misses 1258 calls to get_shadow_status 1079233 calls to shadow_hash_insert 700 calls to shadow_hash_delete 48 shadow removes write access 650 shadow writeable: linux high 624 shadow writeable: sl1p 7459 shadow writeable brute-force 26 shadow unshadows for fork/exit 1 shadow unshadows a page 1 shadow walks guest tables 393663 shadow checks gwalk 1251 shadow flush tlb by rem wr perm 439 shadow emulates invlpg 6212 shadow OOS fixup adds 7330 shadow OOS unsyncs 7328 shadow OOS evictions 175 shadow OOS resyncs 7459 Juergen -- Juergen Gross Principal Developer Operating Systems TSP ES&S SWE OS6 Telephone: +49 (0) 89 636 47950 Fujitsu Technolgy Solutions e-mail: juergen.gross@xxxxxxxxxxxxxx Otto-Hahn-Ring 6 Internet: ts.fujitsu.com D-81739 Muenchen Company details: ts.fujitsu.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |