Re: [Xen-ia64-devel] kernel build performance data with dom0_VP modeopen

[Isaku Yamahata <yamahata@xxxxxxxxxxxxx>]

On Thu, May 25, 2006 at 10:28:30AM -0700, Magenheimer, Dan (HP Labs Fort 
Collins) wrote:
> > For dom0vp model, VHPT size is reduced to 64KBytes to avoid
> > triggering software lock detection.
> > For P=M model, its size remain 64MB.
> > It would be interesting to measure on P=M model with 64KBtyes 
> > VHPT size.
> > (Edit VHPT_SIZE_LOG2 in xen/include/asm-ia64/vhpt.h)
> 
> Yes, having a VHPT that is 1024x smaller is likely
> to make a big difference in performance.  I'm surprised
> it isn't worse!

64MB was wrong. 16MB is used for VHPT with P==M. Sorry for that.
I also remembered that FAST_ACCESS_REFLECT is disabled hyperprivop.S
for dom0 VP model.
Presumably it isn't so diffucult to enable the option.


> I agree it would be interesting to measure P==M with
> the smaller vhpt (Jingke, is this a measurement you
> could easily make on your system?), but since we are
> definitely moving ahead with VP it would be good to
> look forward rather than back.
> 
> Can you explain in more detail about the software lock
> detection and why a 64KB VHPT avoids it?  Any ideas on
> how to get to a larger VHPT (not necessarily as large
> as 64MB)?

When a page is unmapped or trasnfered by grant table API with dom0
VP model, xen/ia64 flushes all tlb and vhpt per each page.
If two pages are unmapped, VHPT is flushed twice.
(domain_page_flush() calls domain_flush_vtlb_all())
It invalidates all VHPT entry. I.e. it touchs all VHPT
(64KB or 16MB) consuming too much time in xen.

Reducing VHPT size makes VHPT invalidates time shorter enought not 
to trigger software lock detection. 
I tried 32KB for VHPT at first, but xen didn't boot.
So I choosed 64KB VHPT. I didn't tracked it down.

Basic approach is how to avoid flush.
It is discussed.
http://lists.xensource.com/archives/html/xen-ia64-devel/2006-03/msg00445.html
(It was March. The merge effort took long time...
You were very right about the time estimation.
I didn't expect such a long time)

I'd like to try the followings.
- Memory copy
  Given that page size is 16KB, ext3 default fs block is 4KB, packet size
  is about 1.5KB at most or jumbo frame size is about 9KB at most.
  So memory copy is much faster than vhpt flush.

- trust privileged domains
  flush by domain's hint.

- virtual address tracking
  Exploit P2M unused bit to track page state.
  A. untracked 
  B. tracked, no virtual address used
  C. tracked, only one virtual address used
     Maybe virtual address can be maintaned using hash table.
  D. tracked, more than tow virtual addresses used

  When a pageis unmapped or transffered
  for pages of A or D state:    full flush
  for pages of B:               no flush is required
  for pages of C:               only one virtual address
  I expect most granted pages for VBD has state B.

- deffered batched page freeing.

- batched flush
  I think Tristan gave it a try. But it seems that he moved to
  another issue.

-- 
yamahata

_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-ia64-devel