Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>]

+Mubin


Hi Prem,

Did you make sure to build FreeRTOS as "hypervisor guest"? I think
FreeRTOS is built to run at EL3 by default so if that is not changed
there are errors when run on any hypervisor. It is possible to build
FreeRTOS to run on Xen but it requires a couple of manual steps:

- source settings64.sh from Vitis builds
- xsct
- set WORKSPACE .
- setws -switch $WORKSPACE
- app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc 
psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
- bsp config stdin psu_uart_1
- bsp config stdout psu_uart_1
- bsp config hypervisor_guest true
- Go to application source path and modify the linker script start address to 
0x40000000,
- app build -name freertos-hello-world
- Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy 
-O binary --gap-fill 0  <elf path> BOOT.BIN"

The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
might be able to provide additional info.



On Thu, 27 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Sorry to bother you again.
> I am currently trying to run my custom IP which sends periodic dma interrupts 
> to the PS with cache coloring enabled. I have now compiled my
> guest running with FreeRTOS and it generates a trap.
> Petalinux and VITIS IDE Version is 2020.2.
> 
> Please find the log attached. After the trap occurs, for every interrupt XEN 
> prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> 
> root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from 
> apu_freertos.cfg
> (XEN) Dom1 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 
> (0 of 64)
> (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> (XEN) sched_null.c:344: 1 <-- d1v0
> (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c 
> gpa=0x000000ff11000c
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, 
> iova=0x41000000, fsynr=0x12, cb=1
> 
> The same application works as expected without FreeRTOS, i.e as a baremetal 
> guest. I look forward for your comment on this.
> 
> Although i see performance degradation, normally i can measure delta time for 
> each interrupt configured to 1ms (without XEN), with XEN for
> the same measurement shows 16 ms for each interval (but the performance is 
> not a priority for now).
> Thanks and Best Regards,
> Prem
> 
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> Sent: Tuesday, May 18, 2021 20:09
> To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; 
> xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; 
> JOVITAC@xxxxxxxxxx
> <JOVITAC@xxxxxxxxxx>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> masters in ZCU102 dev kit  
> Great to hear! Thank you for reporting back.
> 
> 
> On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > Dear Stefano,
> > Thank you very much for your guidance and support. I finally fixed the 
> > issue. I made an error in base memory address like you pointed
> out.
> > I actually modified the base address correctly to 0x40000000 but the 
> > default C code had an additional offset of 0x1100000 , and the DomU
> > memory allocated was only 8MB. I either had to change the offset or 
> > allocate more memory to the DomU guest which i missed until you
> pointed
> > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID 
> > required is 0xE80.
> >
> > Thanks again i was able get my custom IP also working as an AXI master from 
> > PL.
> >
> > Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > Sent: Tuesday, May 18, 2021 02:35
> > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Brian Woods 
> > <brian.woods@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>; Stefano
> > Stabellini <sstabellini@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> > <xen-users@xxxxxxxxxxxxxxxxxxxx>; fnuv@xxxxxxxxxx <fnuv@xxxxxxxxxx>;
> > edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx 
> > <bwoods288@xxxxxxxxx>; Renato Costa Amorim <renato.amorim@xxxxxxxxxxxxxx>;
> > JOVITAC@xxxxxxxxxx <JOVITAC@xxxxxxxxxx>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > masters in ZCU102 dev kit  
> > Your configuration looks correct as far as I can tell.
> >
> > You need dtdev because it triggers the SMMU configuration for your PL
> > block. You don't need device_tree because your baremetal application is
> > not going to read it, but it is not going to cause any issues either.
> >
> > When you say "the execution does not complete the loop at line 244", it
> > means that you see the execution entering the loop and then never
> > leaving? Like an infinite loop?
> >
> > If so, then I imagine the issue could be that the application is
> > crashing when trying to access TxBufferPtr, which is pointing to
> > TX_BUFFER_BASE, which is:
> >
> >
> > #ifndef DDR_BASE_ADDR
> > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> >                  DEFAULT SET TO 0x01000000
> > #define MEM_BASE_ADDR           0x01000000
> > #else
> > #define MEM_BASE_ADDR           (DDR_BASE_ADDR + 0x1000000)
> > #endif
> >
> > #define TX_BUFFER_BASE          (MEM_BASE_ADDR + 0x00100000)
> >
> >
> >
> > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > address? Memory starts at 0x40000000 in a Xen DomU.
> >
> > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> >
> >
> > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano,
> > > Thanks again for your reply and your suggestion.  I did try to replace 
> > > the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> only
> > > in EL3), I added printfs to debug, i can see that the execution does not 
> > > complete the loop at line 244
> > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > >   and
> > > Xil_DCacheFlushRange is not reached. Please find attached my xen 
> > > bootlog.txt.
> > > I am still unable to find out why??
> > >
> > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? 
> > > and do i need dtdev and device_tree in my guest config file when
> > it
> > > is already added to xen.dtsi?
> > >
> > > Best Regards,
> > > Prem
> > >
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > Sent: Saturday, May 15, 2021 00:25
> > > To: Prem Harikrishnan <prem.harikrishnan@xxxxxxxxxxxxxx>
> > > Cc: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>; Brian Woods 
> > > <brian.woods@xxxxxxxxxx>; Julien Grall <julien@xxxxxxx>; Stefano
> > > Stabellini <sstabellini@xxxxxxxxxx>; xen-users@xxxxxxxxxxxxxxxxxxxx 
> > > <xen-users@xxxxxxxxxxxxxxxxxxxx>; fnuv@xxxxxxxxxx
> <fnuv@xxxxxxxxxx>;
> > > edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; bwoods288@xxxxxxxxx 
> > > <bwoods288@xxxxxxxxx>; Renato Costa Amorim <renato.amorim@xxxxxxxxxxxxxx>;
> > > JOVITAC@xxxxxxxxxx <JOVITAC@xxxxxxxxxx>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > > masters in ZCU102 dev kit  
> > > Hi Prem,
> > >
> > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > deprecated set/way instructions that are known to cause problems in
> > > virtualized environments.
> > >
> > > More recent implementations use the "dc civac" intruction that shouldn't
> > > have any issues.
> > >
> > > Can you double-check that you are calling the 64-bit version implemented
> > > by CIVAC?
> > >
> >>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L41
> 4
> >
> > >
> > >
> > >
> > >
> > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano, 
> > > > Thanks for your reply. Yes the axidma application is a baremetal 
> > > > application from the xilinx vitis ide example. 
> > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > >
> > > > I am using the simple mode using polling and not interrupts(they are 
> > > > disabled in the c code), which is Why I didn't include in the
> > guest
> > > > config file. This works well when I compile it normally (as EL3). When 
> > > > compiled as hypervisor guest, the execution stops before the
> > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > in 
> > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > >  
> > > >
> > > > I found the SATA example similar to my use case, also using DMA and 
> > > > SMMU. I just used it as a reference to create my pl.dtb to pass
> > > through
> > > > in the guest config.(from pl.dtsi, which I manually created).
> > > >
> > > > Thanks and Best Regards,
> > > > Prem
> > > >
> > > > Get Outlook for Android
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > Sent: Friday 14 May 2021, 23:16
> > > > To: Prem Harikrishnan
> > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; 
> > > > xen-users@xxxxxxxxxxxxxxxxxxxx; fnuv@xxxxxxxxxx;
> > > edgari@xxxxxxxxxx;
> > > > bwoods288@xxxxxxxxx; Renato Costa Amorim; JOVITAC@xxxxxxxxxx
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL 
> > > > masters in ZCU102 dev kit
> > > >
> > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > a baremetal application? Can you run it successfully, just printing
> > > > "hello world", if you only assign the UART and nothing else?
> > > >
> > > > I am asking just to verify that you can run the baremetal VM
> > > > successfully, aside from the PL device assignment.
> > > >
> > > >
> > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > looks like there are two related IRQs: 121 and 122.
> > > >
> > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > follow: I cannot see any SATA controller assignment in your
> > > > configuration.
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Dear Brian, Stefano and Julien,
> > > > >
> > > > > Thank you very much for your replies.
> > > > > Hi Brian and Stefano, Can you please help me with the following 
> > > > > questions?
> > > > >
> > > > > @Brian Woods Please find attached my vivado configuration, when i 
> > > > > added my AXI DMA now running on simple polling mode(previously i
> > > tried
> > > > it
> > > > > with Scatter Gather mode), Run Automation automatically generated, 1 
> > > > > AXI Interconnect and 1 smart connect. Since there is only 1
> > master
> > > > and
> > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the 
> > > > > stream IDs.
> > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
> > > > >  According to this link section 6.5 I have
> manually
> > > tied
> > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can 
> > > > > you please confirm that my Vivado configuration is correct? I
> > can
> > > > > update you soon with an ILA trace on Vivado.
> > > > >
> > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from 
> > > > > petalinux and system.dtsi also generated after manually adding
> > > > xen.dtsi.
> > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2
> > > > >  Using this
> > > SATA
> > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I 
> > > > > could really use your help to identify what is missing in my
> > xen
> > > > > configuration.
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Brian Woods <brian.woods@xxxxxxxxxx>
> > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > To: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > > > > Cc: Julien Grall <julien@xxxxxxx>; Prem Harikrishnan 
> > > > > <prem.harikrishnan@xxxxxxxxxxxxxx>; Stefano Stabellini
> <sstabellini@xxxxxxxxxx>;
> > > > > xen-users@xxxxxxxxxxxxxxxxxxxx <xen-users@xxxxxxxxxxxxxxxxxxxx>; 
> > > > > brian.woods@xxxxxxxxxx <brian.woods@xxxxxxxxxx>; fnuv@xxxxxxxxxx
> > > > > <fnuv@xxxxxxxxxx>; edgari@xxxxxxxxxx <edgari@xxxxxxxxxx>; 
> > > > > bwoods288@xxxxxxxxx <bwoods288@xxxxxxxxx>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for 
> > > > > PL masters in ZCU102 dev kit  
> > > > > Prem,
> > > > >
> > > > > Sorry for the late email, I've been under the weather and have been 
> > > > > away
> > > > > from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > calculated:
> > > > >
> > > > > 14-10: TBU bits
> > > > > 9-6: Master ID bits
> > > > > 5-0: AXI ID bits
> > > > >
> > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > table which is table 16-4 on page 409.
> > > > >
> > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > which is table 16-13 on page 429.
> > > > >
> > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> > > > > are work arounds.  Here's a couple of examples (with example numbers):
> > > > >
> > > > > AXI master -> PS slave port 0
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > >
> > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > >
> > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > >                            -> PS slave port 1
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > >
> > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 
> > > > > (both share)
> > > > >
> > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > >                                 -> PS slave port 1
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 
> > > > > (both share)
> > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 
> > > > > (both share)
> > > > >
> > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave 
> > > > > port 0
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 
> > > > > (master #1)
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 
> > > > > (master #2)
> > > > >
> > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave 
> > > > > port 0
> > > > >                                                         -> PS slave 
> > > > > port 1
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 
> > > > > (master #1)
> > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 
> > > > > (master #1)
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 
> > > > > (master #2)
> > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 
> > > > > (master #2)
> > > > >
> > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits 
> > > > > are
> > > > > fairly simple but with some caveats.
> > > > >
> > > > > You can also read in the SMMU faults from Xen and use:
> > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > >
> > > > > Let me know if this isn't clear or you have any other questions.
> > > > >
> > > > > Brian
> > > > >
> > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > Hello everyone,
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 
> > > > > > > > development board
> > > > > > > > and i am trying to create a barematal application that uses the 
> > > > > > > > AXI DMA on
> > > > > > > > the programmable logic, using the scatter gather example from 
> > > > > > > > the xilinx
> > > > > > > > repository. This works perfectly well for the baremetal 
> > > > > > > > application. The
> > > > > > > > diagram below represents the information flow. The AXI DMA and 
> > > > > > > > the AXI data
> > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out 
> > > > > > > > of the box IP
> > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) 
> > > > > > > > which means the
> > > > > > > > AXI ID cannot manually be set.
> > > > > > > >
> > > > > > > >
> > > > > > > > Now i want to make this as a baremetal hypervisor guest, in 
> > > > > > > > future this will
> > > > > > > > be replaced by a custom IP with cache colouring enabled. I want 
> > > > > > > > to pass
> > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > >
> > > > > > > >
> > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to 
> > > > > > > > Dom-0) to
> > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my 
> > > > > > > > linux, xen
> > > > > > > > image and use tftp to boot.
> > > > > > > >
> > > > > > > >  From what i understand from the xilinx tutorials 
> > > > > > > >Xen+and+PL+Masters
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > section 4, stream ids are required by the SMMU to get the 
> > > > > > > > correct address
> > > > > > > > translation.
> > > > > > > >
> > > > > > > > I am still unable to figure out how to get the iommu property 
> > > > > > > > for my AXI DMA
> > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > >
> > > > > > > >
> > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 
> > > > > > > > releases
> > > > > > > > which i can be useful to my application
> > > > > > > >
> > > > > > > >   *   1:1 memory mappings for Xen virtual machines, enabling 
> > > > > > > >device
> > > > > > > >     assignments without SMMU
> > > > > > > >   *
> > > > > > > >       Static assignment of PL blocks, including bus-mastering 
> > > > > > > >blocks, to
> > > > > > > >     Xen virtual machines
> > > > > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > > > > >
> > > > > > > You are usiong a version of Xen that has been customised by 
> > > > > > > Xilinx. So I would
> > > > > > > recommend to contact them directly.
> > > > > > >
> > > > > > > I have CCed Stefano who might be able to help you.
> > > > > >
> > > > > > Hi Julien,
> > > > > >
> > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > >
> > > > > >
> > > > > > Hi Prem,
> > > > > >
> > > > > > Let me CC a few people that might be able to help.
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
> 
>