Re: [BUG]Add PCIE devie to SMMUv3 fail

[Rahul Singh <Rahul.Singh@xxxxxxx>]

Hi Sisyphean,

> On 13 Dec 2022, at 12:41 am, sisyphean <sisyphean@zlw.email> wrote:
> 
> Hi,
> 
> 在 2022/12/12 18:05, Julien Grall 写道:
>> 
>> On 12/12/2022 05:49, sisyphean wrote:
>>> Hi,
>> 
>> Hi,
> Sorry for my negligence, I will add a cc to all subsequent emails.
>> When submitting a bug report for Arm, can you please CC the Arm maintainers?
>> 
>>> 在 2022/12/9 17:50, Rahul Singh 写道:
>>> After setting XEN and kernel as above, I tried the following two methods to 
>>> add a PCIE device passthrough:
>>> 
>>> 1. According to your suggestion, use the command xl pci-assignable-add 
>>> 0002:21:00.0 to set in the Dom0. But in function
>>> iommu_do_pci_domctl,  after device_assigned is called, ENODEV error is 
>>> obtained.
>>> 
>>> 2. Add xen-pciback.hide=(0002:21:00.0) to dom0-bootargs in the device tree, 
>>> I encountered the same problem as before
>>> when initializing the kernel. In function pci_add_device, PCIE devices 
>>> cannot be added to SMMUv3.
>>> 
>>> The kernel version I use is 5.10. Does this have an impact?
>>> 
>>> In addition, an error was encountered after XEN enabling ITS:
>>> 
>>> In function gicv3_cpu_init, gicv3_its_setup_collection return -ETIMEDOUT. 
>>> This problem was solved after I made the
>>> following changes:
>>> 
>>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>>> index 9558bad..a12c0d1 100644
>>> --- a/xen/arch/arm/gic-v3-its.c
>>> +++ b/xen/arch/arm/gic-v3-its.c
>>> @@ -118,11 +118,11 @@ static int its_send_command(struct host_its *hw_its, 
>>> const void *its_cmd)
>>>       }
>>> 
>>>       memcpy(hw_its->cmd_buf + writep, its_cmd, ITS_CMD_SIZE);
>>> -    if ( hw_its->flags & HOST_ITS_FLUSH_CMD_QUEUE )
>>> +    // if ( hw_its->flags & HOST_ITS_FLUSH_CMD_QUEUE )
>> 
>> This is suggesting that the logic in its_map_cbaser() doesn't detect the 
>> command queue is uncacheable.
>> 
>> Looking at the code, Xen will write the register with the shareability it 
>> wants and then read back to confirm the ITS "accept it". If it didn't 
>> accept, then we will use uncacheable.
>> 
> Function its_map_cbaser() is called twice, and the reg values read by 
> readq_relaxed(cbasereg)
> are 0xB8000000EBD004FF and 0xB8000000EBC004FF respectively
>> Can you print the value read from the ITS in its_map_cbaser()?

Xen writes "InnerCache, bits [61:59]" to 0x7 (Normal Inner Cacheable 
Read-allocate, Write-allocate, Write-back ) of cbaser to physical ITS and
when we read back we are getting the same value that means command queue memory 
is mapped as cached as per physical ITS.

But what you are observing is opposite, command queue memory is uncached that 
is why you have to clean and invalidate the dcache.
I am not sure why this is happening but maybe HW is not operating correctly.
 
Regards,
Rahul