[Xen-ia64-devel] [PATCH] discontig memory support

[Kouya SHIMURA <kouya@xxxxxxxxxxxxxx>]

Hi all,

Sorry for the delay.
I'll post the revised patches to support discontiguous physical memory.

[CHANGES]
* A new boot option 'contig_mem'.
  This is a compatibilty option to keep old behavior. 
  Tables are allocated on physically contiguous memory area.
  It might be helpful for the performance evaluation.

* mpt_table is located on virtual space too. 
  Now Xen can handle entire physical memory even if the machine has
  large memory holes like Alex's machine.

* Using private PGD for virtual address translation.
  swapper_pg_dir and init_mm are now obsolete.

* Cleanup VIRTUAL_MEM_MAP code originated from Linux.
  I might have boldly erased the lines too much. :-P

Thanks,
Kouya

Signed-off-by: Kouya Shimura <kouya@xxxxxxxxxxxxxx>

Attachment: 9685:discontig.patch
Description: Binary data

Attachment: 9686:remove_unused.patch
Description: Binary data

Attachment: 9687:cleanup_vmemmap.patch
Description: Binary data

Kouya SHIMURA writes:
 > Hi xen/ia64 developers.
 > 
 > The attached patch supports discontiguous memory. 
 > It also makes over 4GB memory available.
 > Please comment and review.
 > 
 > Signed-off-by: Kouya Shimura <kouya@xxxxxxxxxxxxxx>
 > 
 > Here is a design memo.
 > --------------------------------------------------------------------
 > VIRTUAL FRAME TABLE DESIGN MEMO
 > (supporting discontig memory)
 > 
 > [PURPOSE] IA64 Xen hypervisor currently limits the maximum size of
 > available memory to 4GB (the half is used for MMIO and actually only
 > 2GB is available).  The purpose of this patch is to remove the
 > restriction.
 > 
 > The reason of this restriction is due to huge linear array of struct
 > page_info named 'frame_table'.  A page_info struct is accessed by a
 > linear index thus frame_table requires a contiguous memory area. If
 > there is a large hole in physical memory space, a block of page_info
 > corresponding to the hole is wasteful.
 > 
 > Ancient IA64 Xen hypervisor allocated the memory for frame_table from
 > xen heap whose size is at most 64MB. Xen heap had various uses and ran
 > out occasionally. I think the above is the reason to restrict the
 > available memory.
 > 
 > FYI, when I tried just to remove that restriction, latest Xen worked
 > well on Tiger4 with 8GB memory. Xen heap seems to have quite enough
 > margin now.
 > 
 > [SOLUTION]
 > Linux has the same problem and there are two solutions.
 > 
 > (1) VIRTUAL_MEM_MAP
 >   mem_map is linux's name of frame_table. places frame_table on
 >   virtual memory space and doesn't allocate physical memory to memory
 >   hole. The advantage is that the impact to existing code is little.
 > 
 > (2) SPARSEMEM
 >   prepares multi level tables and traverses the tree in order to
 >   access page_info. More modification and more influence to the
 >   existing code.  But supporting NUMA and memory hotplug on Linux owe
 >   to SPARSEMEM.
 > 
 > I have no idea which has good performance. Attached patch implements
 > the same way as VIRTUAL_MEM_MAP because of ease. But I think SPARSEMEM
 > should be implemented in future because it would be a mainstream in
 > Linux. In this case, common code might be modified.
 > 
 > [DESIGN]
 > * Location of the virtual address of frame_table
 >   I set the frame_table start at 0xf300000000000000 which region
 >   number is 7. Region number 7 is used for Xen hypervisor. The page
 >   size is 16KB. VHPT is disabled in this region and 16KB/16MB/64MB
 >   page size are mixed. The virtual aliasing might occur but the
 >   performance wouldn't be affected.
 > 
 > * Configuration of address translation table
 >   inherits linux's way and adopts 3-level configuration. 
 > +----------------------------------------------------------------+
 > |6 6             4          3          2          1             0|
 > |3 1             7          6          5          4             0|
 > +----------------------------------------------------------------+
 >  +-+10011         +-PGDoff--++-PMDoff--++-PTEoff--++---offset---+    
 >   region=7          (no PUD)      |          |            |
 >                        |   PGD    |   PMD    |   PTE      |
 >                        |  +---+ +--->+---+ +--->+---+     |
 >                        |  |   | | |  |   | | |  |   |     |
 >                        |  |   | | +->|   |-+ |  |   |     v
 >                        +->|   |-+    |   |   +->|   |--->OR->physical addr
 >                           +---+      +---+      +---+
 >   - Location of PGD.
 >     A page(swapper_pg_dir) pointed from init_mm.pgd seems to be never
 >     used. So I use this page as PGD for virtual frame_table. If
 >     someone uses this page, please tell me.
 > 
 >   - Another tables(PMD,PTE) are dynamically allocated from boot pages.
 > 
 > This structure inherits linux's address translation table. But it is
 > not efficient for small data such as frame_table. I expected that an
 > existing table walker is usable but finally I wrote a new walker. We
 > had better redesign the structure for performance improvement.
 > 
 > * Alternate data TLB miss handler
 > When a TLB miss occurs by accessing the frame_table, alternate data
 > TLB miss handler is called because the VHPT is disabled on region
 > 7. Originally Xen's alt_dtlb_miss handler has TLB mapping mechanism to
 > a granule page (16MB) in order to access physical memory data in
 > virtual mode.
 > 
 > I appended a code that checks the fault address is inside of
 > frame_table and traverses the address translation table and inserts a
 > translation cache. To avoid nested data TLB fault, data address
 > translation is disabled at the time of table walking. On fail of table
 > walking, ia64_fault() is called.
 > 
 > [STATUS]
 >   * Xen and domU and domVTI (cset:9484:ddc279c91502) work well on
 >     Tiger4 with 8MB physical memory.
 > 
 > [TODO]
 >   * mpt_table (defined in xen/arch/ia64/xen/xenmem.c) implies the same
 >     problem. We have to fix it.
 >   * redesign of address translation table
 >   * performance evaluation
 >   * cleanup #ifdefs of CONFIG_VIRTUAL_MEM_MAP
 > --------------------------------------------------------------------
 > _______________________________________________
 > Xen-ia64-devel mailing list
 > Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
 > http://lists.xensource.com/xen-ia64-devel

_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-ia64-devel