Re: [Xen-devel] Xen 4.3/AMD: setup ssss:bb:dd.f for d0 failed (-ENODEV)

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 06.08.13 at 16:11, Suravee Suthikulanit <suravee.suthikulpanit@xxxxxxx>
wrote:
> On 8/6/2013 1:55 AM, Jan Beulich wrote:
>>>>> On 06.08.13 at 04:02, Suravee Suthikulanit <suravee.suthikulpanit@xxxxxxx>
>> wrote:
>>> On 8/5/2013 7:58 AM, Jan Beulich wrote:
>>>>>>> On 22.07.13 at 14:55, Stefan Bader <stefan.bader@xxxxxxxxxxxxx> wrote:
>>>>> While testing Xen 4.3 on my AMD testbox I noticed the following output 
>>>>> from
>>>>> the
>>>>> hypervisor (this has no visible effect on at least simple operation):
>>>>>
>>>>> (XEN) setup 0000:00:18.0 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:18.1 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:18.2 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:18.3 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:18.4 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:19.0 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:19.1 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:19.2 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:19.3 for d0 failed (-19)
>>>>> (XEN) setup 0000:00:19.4 for d0 failed (-19)
>>>>>
>>>>> The PCI devices related to the output are all PCI host bridges:
>>>>>
>>>>> 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor HyperTransport Configuration
>>>>> 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor Address Map
>>>>> 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor DRAM Controller
>>>>> 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor Miscellaneous Control
>>>>> 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor Link Control
>>>>> 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor HyperTransport Configuration
>>>>> 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor Address Map
>>>>> 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor DRAM Controller
>>>>> 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor Miscellaneous Control
>>>>> 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD]
>>>>>    Family 10h Processor Link Control
>>>>>
>>>>> This all seems to be related to PCI passthrough setup started from
>>>>> xen/drivers/passthrough/vtd/iommu.c:intel_iommu_dom0_init() or
>>>>> xen/drivers/passthrough/amd/pci_amd_iommu.c:amd_iommu_dom0_init()
>>>>>
>>>>> While the Intel code skips over bridge type entries, the AMD code has no
>>>>> such
>>>>> exception and will fail the handler with -ENODEV when 
>>>>> find_iommu_for_device
>>>>> fails. But I am not sure one can just compare implementations here.
>>>>>
>>>>> Would someone have more insight to decide whether skipping host bridges in
>>>>> amd_iommu_setup_dom0_device would make sense? I do not see any bad effects
>>>>> caused by this but it does not look good and did not happen before.
>>>> Fundamentally this looks like a BIOS bug, and hence doing as you
>>>> suggest unconditionally is not an option imo.
>>> Actually, I don't think this is a BIOS bug.  I have looked through the IVRS
>>> table
>>> on the system that I have and it looks correct to me in the sense that it
>>> does not list
>>> all these processor host bridge devices for IOMMU.  As Stefan mentioned, the
>>> current
>>> dom0 PCI setup code scans through PCI bus does not have filter for the host
>>> bridges which
>>> do not need IOMMU. However, these error message should not be harmful.
>>> Also, in debug mode,
>>> it clearly state that "No iommu for device 0000:00:18.0".
>> So are you saying that you also see those messages in debug mode?
> 
> The default (stock unstable) is showing "(XEN) setup 0000:00:18.0 for d0 
> failed (-19)".
> With the "amd-iommu-debug" boot options, it is also showing "(XEN) AMD-Vi: No 
> iommu for device 0000:00:18.0".

And both messages are a reason for the reader to expect trouble.

>> If so, i.e. if they are expected to always appear, we should indeed
>> try to avoid issuing them, as they may cause unnecessary worries
>> (as is the case for Stefan here).
> Agree
>>
>>>> Adding code to skip host bridges (under the assumption that they won't
>>> initiate
>>>> transactions that might require translation by the IOMMU) may be
>>>> an option if enabled only via command line option.
>>> Currently, pdev_type() in driver/passthrough/pci.c only match the
>>> PCI_CLASS_BRIDGE_PCI (0x0604)
>>> and not the Host/PCI bridge (0x600) which I see what the devices 0:18.x,
>>> 0x19.x are reporting in
>>> the PCI class code structure. This might need to change?
>> We may want to change this, but this needs to be done carefully
>> to not cause regressions on the VT-d side (to which host bridges
>> currently are the same as ordinary PCI devices/PCIe endpoints).
> Which class code are showing on Intel system for host bridges?

0600 of course, just like AMD's. That's why we need to be careful to
not unintentionally change behavior for VT-d.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel