Re: [Xen-devel] PCI passthrough for domU allocated with more than 4G memory

[Konrad Rzeszutek Wilk <konrad@xxxxxxxxxx>]

On Wed, Aug 01, 2012 at 05:57:13AM +0000, Fan, Huaxiang wrote:
> Hi,
> 
> I have encountered some strange problems when I was trying to PCI passthrough 
> Broadcom 5709/5716 NICs to domUs allocated with more than 4G memory. Please 
> see below for details.
> 
> My environment is:
> Hardware Platform: DELL R210 with 2 Broadcom 5709 NICs and 2 Broadcom 5716 
> NICs
> Xen: xen 4.2 unstable (64bits for hypervisor and 32bit for tools)
> Kernel for both dom0 and domUs: xenified kernel 2.6.32.57 (32bit)
> OS: CentOS 6.2 (32bit)
> 
> The general info regarding to xen can be get via below command
> 
> # xl info
> 
> host                   : 7.8
> 
> release                : 2.6.32.57
> 
> version                : #1 SMP Fri Jul 6 18:44:16 CST 2012
> 
> machine                : i686
> 
> nr_cpus                : 8
> 
> max_cpu_id             : 31
> 
> nr_nodes               : 1
> 
> cores_per_socket       : 4
> 
> threads_per_core       : 2
> 
> cpu_mhz                : 2660
> 
> hw_caps                : 
> bfebfbff:28100800:00000000:00003b40:0098e3fd:00000000:00000001:00000000
> 
> virt_caps              : hvm hvm_directio
> 
> total_memory           : 8182
> 
> free_memory            : 7046
> 
> sharing_freed_memory   : 0
> 
> sharing_used_memory    : 0
> 
> free_cpus              : 0
> 
> xen_major              : 4
> 
> xen_minor              : 2
> 
> xen_extra              : -unstable
> 
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
> hvm-3.0-x86_32p hvm-3.0-x86_64
> 
> xen_scheduler          : credit
> 
> xen_pagesize           : 4096
> 
> platform_params        : virt_start=0xff400000
> 
> xen_changeset          : unavailable
> 
> xen_commandline        : dom0_mem=1024M dom0_max_vcpus=2 dom0_vcpus_pin
> 
> cc_compiler            : gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)
> 
> cc_compile_by          : root
> 
> cc_compile_domain      :
> 
> cc_compile_date        : Thu Jul 12 11:20:56 CST 2012
> 
> xend_config_format     : 4
> 
> case 1> I specified PV-domU config as below
> 
> memory=3072
> 
> maxmem=6144
> 
> name="bs"
> 
> vif=['ip=169.254.254.1,script=vif-nat',]
> 
> disk=['file:/root/bs.img,xvda1,w']
> 
> kernel='/root/vmlinuz'
> 
> extra="iommu=soft console=hvc0"
> 
> ramdisk='/root/initrd.img'
> 
> root="/dev/xvda1 ro"
> 
> pci=['01:00.0','01:00.1']
> 
> and then start domU. After that I executed below command
> 
> # xl list
> 
> Name                                        ID   Mem VCPUs      State   
> Time(s)
> 
> Domain-0                                     0  1024     2     r-----      
> 88.7
> 
> bs                                           2  3072     1     -b----       
> 1.1
> 
> It seemed normal to me. But when I logon bs domU, and executed below command
> 
> # cat /proc/meminfo | head
> 
> MemTotal:        6158940 kB
> 
> MemFree:         2944776 kB

So that is the right amount. It has around 3GB of free kernel space.

> 
> Buffers:            5108 kB
> 
> Cached:            32292 kB
> 
> SwapCached:            0 kB
> 
> Active:            21456 kB
> 
> Inactive:          22936 kB
> 
> Active(anon):       7000 kB
> 
> Inactive(anon):      108 kB
> 
> Active(file):      14456 kB
> 
> It indicated the total memory was 6G, why?

I wish you also included the DirectMap number. Irregardless of
what /proc/meminfo says, what did your dmesg say in 'Memory' section?

> When I back to dom0, I executed below command
> 
> # xl mem-set bs 6144
> 
> # xl list
> 
> Name                                        ID   Mem VCPUs      State   
> Time(s)
> 
> Domain-0                                     0  1024     2     r-----      
> 93.5
> 
> bs                                           2  6144     1     -b----      
> 10.5
> It seemed normal to me. But when I logon bs domU again and executed below 
> command
> 
> # cat /proc/meminfo | head
> 
> MemTotal:        9304668 kB
> 
> MemFree:         6087540 kB

So that is right. 6GB of free space.
> 
> Buffers:            5168 kB
> 
> Cached:            32464 kB
> 
> SwapCached:            0 kB
> 
> Active:            22300 kB
> 
> Inactive:          22408 kB
> 
> Active(anon):       7080 kB
> 
> Inactive(anon):      108 kB
> 
> Active(file):      15220 kB
> 
> 
> 
> It indicated total memory was 9G (6G + 3G). It was wired. Any idea about this?

Presumarily b/c '3G' of it is the E820_UNUSUABLE or the big gap in the
E820. But irregardless of that - do you have 6GB of ram in your guest?

> Case 2> I specified PV-domU config as below
> 
> memory=6144
> 
> maxmem=6144
> 
> name="bs"
> 
> vif=['ip=169.254.254.1,script=vif-nat',]
> 
> disk=['file:/root/bs.img,xvda1,w']
> 
> kernel='/root/vmlinuz'
> 
> extra="iommu=soft console=hvc0"
> 
> ramdisk='/root/initrd.img'
> 
> root="/dev/xvda1 ro"
> 
> pci=['01:00.0','01:00.1']
> 
> and then start domU. After that I executed below command
> 
> # xl list
> 
> Name                                        ID   Mem VCPUs      State   
> Time(s)
> 
> Domain-0                                     0   648     2     r-----     
> 120.5
> 
> bs                                           3  3360     1     -b----       
> 7.0
> 
> the output was very confusing. Why dom0 memory had been shrank to 648M and 
> only 3360M assigned to bs domU?

 I think you are seeing a bug in the xl with the autoballoon. Smoebody
mentioned this on the xen-devel.
> 
> My own analysis:
> I extracted the bios e820 memory map on bs domU as below
> 
> [    0.000000] BIOS-provided physical RAM map:
> 
> [    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
> 
> [    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
> 
> [    0.000000]  Xen: 0000000000100000 - 00000000bf699000 (usable)
> 
> [    0.000000]  Xen: 00000000bf699000 - 00000000bf6af000 (reserved)
> 
> [    0.000000]  Xen: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
> 
> [    0.000000]  Xen: 00000000bf6ce000 - 00000000c0000000 (reserved)
> 
> [    0.000000]  Xen: 00000000e0000000 - 00000000f0000000 (reserved)
> 
> [    0.000000]  Xen: 00000000fe000000 - 0000000100000000 (reserved)
> 
> [    0.000000]  Xen: 0000000180000000 - 00000001c33ec000 (usable)
> 
> I think the root cause might be related to the holes between c0000000 and 
> e0000000 and between f0000000 and fe000000 and between 100000000 and 
> 180000000. And I think the e820_host option set according to my tracking.

Not sure what you mean by 'my tracking'.

The holes are there to provide the PCI subsystem the space to stick the
MMIO BARs of your PCI device. The memory that was "taken" out of those
areas is then appended to the next E820_RAM region. It should have
gone to 0x1000000, not to 0x18000000 though.

Ohh, you are using the ancient 2.6.32 domU! Well, that is ancient -
and it probably has bugs. Have you considered using something more
recent?
> 
> Thanks in advance
> HUAXIANG FAN
> Software Engineer II
> 
> WEBSENSE NETWORK SECURITY TECHNOLOGY R&D (BEIJING) CO. LTD.
> ph: +8610.5884.4327
> fax: +8610.5884.4727
> www.websense.cn<http://www.websense.cn>
> 
> Websense TRITON(tm)
> For Essential Information Protection(tm)
> Web 
> Security<http://www.websense.com/content/Regional/SCH/WebSecurityOverview.aspx>
>  | Data 
> Security<http://www.websense.com/content/Regional/SCH/DataSecurity.aspx> | 
> Email 
> Security<http://www.websense.com/content/Regional/SCH/MessagingSecurity.aspx>
> 
> 
> 
>  Protected by Websense Hosted Email Security -- www.websense.com 

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel