Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>]

Hi Prem,

Some older versions of Xil_DCacheFlushRange were implemented using
deprecated set/way instructions that are known to cause problems in
virtualized environments.

More recent implementations use the "dc civac" intruction that shouldn't
have any issues.

Can you double-check that you are calling the 64-bit version implemented
by CIVAC?

https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L414



On Fri, 14 May 2021, Prem Harikrishnan wrote:
> Hi Stefano, 
> Thanks for your reply. Yes the axidma application is a baremetal application 
> from the xilinx vitis ide example. 
> Also yes, I have already run hello world as DomU guest, with UART1.
> 
> I am using the simple mode using polling and not interrupts(they are disabled 
> in the c code), which is Why I didn't include in the guest
> config file. This works well when I compile it normally (as EL3). When 
> compiled as hypervisor guest, the execution stops before the
> function xil_dcacheflushrange(). Please find the c code attached
> in 
> https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
>  
> 
> I found the SATA example similar to my use case, also using DMA and SMMU. I 
> just used it as a reference to create my pl.dtb to pass through
> in the guest config.(from pl.dtsi, which I manually created).
> 
> Thanks and Best Regards,
> Prem
> 
> Get Outlook for Android
> 
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> Sent: Friday 14 May 2021, 23:16
> To: Prem Harikrishnan
> Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; 
> xen-users@xxxxxxxxxxxxxxxxxxxx; fnuv@xxxxxxxxxx; edgari@xxxxxxxxxx;
> bwoods288@xxxxxxxxx; Renato Costa Amorim; JOVITAC@xxxxxxxxxx
> Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> masters in ZCU102 dev kit
> 
> I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> a baremetal application? Can you run it successfully, just printing
> "hello world", if you only assign the UART and nothing else?
> 
> I am asking just to verify that you can run the baremetal VM
> successfully, aside from the PL device assignment.
> 
> 
> In regards to the PL device assignment, I'll let Brian answer on the
> Vivado configuration and SMID calculation. But I noticed that you only
> assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> looks like there are two related IRQs: 121 and 122.
> 
> Finally, why did you mention SATA passthrough in your reply? I don't
> follow: I cannot see any SATA controller assignment in your
> configuration.
> 
> 
> On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > Dear Brian, Stefano and Julien,
> >
> > Thank you very much for your replies.
> > Hi Brian and Stefano, Can you please help me with the following questions?
> >
> > @Brian Woods Please find attached my vivado configuration, when i added my 
> > AXI DMA now running on simple polling mode(previously i tried
> it
> > with Scatter Gather mode), Run Automation automatically generated, 1 AXI 
> > Interconnect and 1 smart connect. Since there is only 1 master
> and
> > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
> >  According to this link section 6.5 I have manually tied
> > of AxPROT bits to value of 0x2 to create non secure transactions. Can you 
> > please confirm that my Vivado configuration is correct? I can
> > update you soon with an ILA trace on Vivado.
> >
> > @Stefano Stabellini Please find attached my pl.dtsi generated from 
> > petalinux and system.dtsi also generated after manually adding
> xen.dtsi.
> > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2
> >  Using this SATA
> > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could 
> > really use your help to identify what is missing in my xen
> > configuration.
> >
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Brian Woods <brian.woods@xxxxxxxxxx>
> > Sent: Thursday, May 13, 2021 17:13
> > To: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > Cc: Julien Grall <julien@xxxxxxx>; Prem Harikrishnan 
> > <prem.harikrishnan@xxxxxxxxxxxxxx>; Stefano Stabellini 
> > <sstabellini@xxxxxxxxxx>;
> > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; 
> > brian.woods@xxxxxxxxxx <brian.woods@xxxxxxxxxx>; fnuv@xxxxxxxxxx
> > <fnuv@xxxxxxxxxx>; edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; 
> > bwoods288@xxxxxxxxx <bwoods288@xxxxxxxxx>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > masters in ZCU102 dev kit  
> > Prem,
> >
> > Sorry for the late email, I've been under the weather and have been away
> > from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> > calculated:
> >
> > 14-10: TBU bits
> > 9-6: Master ID bits
> > 5-0: AXI ID bits
> >
> > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > To look up the TBU value, see UG1085 and look for the "System Masters"
> > table which is table 16-4 on page 409.
> >
> > For the master ID, see UG1085 and look for the "Master IDs List" table
> > which is table 16-13 on page 429.
> >
> > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > See: https://www.xilinx.com/support/answers/69447.html
> > With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> > are work arounds.  Here's a couple of examples (with example numbers):
> >
> > AXI master -> PS slave port 0
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> >
> > AXI master -> SmartConnect -> PS slave port 0
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> >
> > AXI master -> SmartConnect -> PS slave port 0
> >                            -> PS slave port 1
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> >
> > Two AXI masters -> SmartConnect -> PS slave port 0
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both 
> > share)
> >
> > Two AXI masters -> SmartConnect -> PS slave port 0
> >                                 -> PS slave port 1
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both 
> > share)
> >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both 
> > share)
> >
> > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master 
> > #1)
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master 
> > #2)
> >
> > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> >                                                         -> PS slave port 1
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master 
> > #1)
> >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master 
> > #1)
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master 
> > #2)
> >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master 
> > #2)
> >
> > The "sideband -> SmartConenct -> sideband" can also be functionally
> > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
> > fairly simple but with some caveats.
> >
> > You can also read in the SMMU faults from Xen and use:
> > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > to decode them. That should tell you the SMMU IDs as well.
> >
> > Let me know if this isn't clear or you have any other questions.
> >
> > Brian
> >
> > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > Hello everyone,
> > > >
> > > > Hi Prem,
> > > >
> > > > > I am using the latest version of the xilinx mpsoc zcu102 development 
> > > > > board
> > > > > and i am trying to create a barematal application that uses the AXI 
> > > > > DMA on
> > > > > the programmable logic, using the scatter gather example from the 
> > > > > xilinx
> > > > > repository. This works perfectly well for the baremetal application. 
> > > > > The
> > > > > diagram below represents the information flow. The AXI DMA and the 
> > > > > AXI data
> > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the 
> > > > > box IP
> > > > > from Vivado and uses a single channel (Does not use AxUSER) which 
> > > > > means the
> > > > > AXI ID cannot manually be set.
> > > > >
> > > > >
> > > > > Now i want to make this as a baremetal hypervisor guest, in future 
> > > > > this will
> > > > > be replaced by a custom IP with cache colouring enabled. I want to 
> > > > > pass
> > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > >
> > > > >
> > > > > I followed this tutorial Baremetal + DomU
> > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) 
> > > > > to
> > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, 
> > > > > xen
> > > > > image and use tftp to boot.
> > > > >
> > > > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > section 3 and MPSOC SMMU
> > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > section 4, stream ids are required by the SMMU to get the correct 
> > > > > address
> > > > > translation.
> > > > >
> > > > > I am still unable to figure out how to get the iommu property for my 
> > > > > AXI DMA
> > > > > and how to set the dtdev config for my hypervisor guest.
> > > > >
> > > > >
> > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 
> > > > > releases
> > > > > which i can be useful to my application
> > > > >
> > > > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > > > >     assignments without SMMU
> > > > >   *
> > > > >       Static assignment of PL blocks, including bus-mastering blocks, 
> > > > >to
> > > > >     Xen virtual machines
> > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > >
> > > > You are usiong a version of Xen that has been customised by Xilinx. So 
> > > > I would
> > > > recommend to contact them directly.
> > > >
> > > > I have CCed Stefano who might be able to help you.
> > >
> > > Hi Julien,
> > >
> > > Many thanks for CC'ing me, I missed the email.
> > >
> > >
> > > Hi Prem,
> > >
> > > Let me CC a few people that might be able to help.
> >
> >
> 
> 
>