This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DRA80XMEVM:About PCIe Inbound Address Translation

Part Number: DRA80XMEVM
Other Parts Discussed in Thread: TMDX654IDKEVM

Tool/software:

I would like to expose the first few hundred bytes or so of the PCIE1_DAT1 area (address 0x05600000~) on the board side to the Root Complex side via PCIe.
I checked the Technical Reference Manual, and it states that access to BAR0 will see a group of registers in PCIe, no matter what address is set in IATU, as shown below.
Even if I set BAR0 on the Root Complex side, I cannot get the expected value and it is ALL 0xF.

Is there anything else I need to do to set BAR0 or anything else in this case?


  • Hi Masa,

    DRA80XMEVM should be the alternative name for AM65 device. I would recommend looking through the Linux driver for how these registers are set up.

    Specifically, it the driver in use should be the pcie-designware-host.c: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/pci/controller/dwc/pcie-designware-host.c?h=ti-linux-6.6.y#n396

    And pci-keystone.c: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/pci/controller/dwc/pci-keystone.c?h=ti-linux-6.6.y#n298

    Regards,

    Takuma

  • Hi Takuma.
    Thank you for your answer.
    I will change the way I ask the question.

    I want to access the registers on the EndPoint side via PCIe from the RootComplex side.
    When the UART0 address (0x02800000) was specified for BAR0 on the EndPoint side, it was possible to access it from the RootComplex side.
    However, if you specify the address (0x05600000) of PCIE1_DAT, you will not be able to access it correctly.(All values are 0xff.)

    There are some addresses that I can access normally and some that I cannot. What reasons could there be?

    Regards,
    Masa

  • Hi Masa,

    Do you use Linux on the DRA80XMEVM? If so which Processor SDK version do you use? and please share your devicetree patch which configures the PCIe to EP mode.

  • Hi Takuma.

    >Do you use Linux on the DRA80XMEVM? 

    No. OS is T-Kernel.
    When using Linux, have you had any experience of accessing the PCIE1_DAT address (0x05600000) on the EndPoint side from the RootComplex side?
    If you have experience, please tell us which Linux code you should refer to.

  • Hi Masa,

    I didn't have any customer reporting such problem in Linux.

    Just a dumb question, when you see 0xff on the PCIe RC side, have you check the memory region of 0x05600000 on the (EVM) EP side, what is the value there?

  • Hi Liu

    >I didn't have any customer reporting such problem in Linux.

    Can you confirm that you can access PCIE1_DAT (0x05600000) via EP in Linux?

    >Just a dumb question, when you see 0xff on the PCIe RC side, have you check the memory region of 0x05600000 on the (EVM) EP side, what is the value there?
    The area of 0x05600000 is the PCIE_EP_PID Register. 
    The value 6810XX100 should be displayed as read-only.

  • Hi Masa,

    I currently don't have an evm setup to check on this with Linux, but are you accessing PCIE1_DAT on PCIe0 or PCIe1 of AM65x? According to the AM65x TRM, PCIE1_DAT is the MMR region of PCIe1 module, while the MMR region of PCIe0 is 0x0x05500000.

  • Hi Liu

    The same applies when accessing PCIE0_DAT (0x0x05500000).

  • Hi Masa,

    Since you have the evm setup, can you please run Linux on the evm and check if it behaves the same in Linux?

  • Linked below shows how to modify the kernel devicetree to configure the PCIe to EP mode. For AM65x, you would need to modify k3-am654-pcie-usb2.dtso or k3-am654-pcie-usb3.dtso depending on which EVM daughter card you use.

    https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/09_02_01_10/exports/docs/linux/Foundational_Components/Kernel/Kernel_Drivers/PCIe/PCIe_End_Point.html

  • Hi Liu

    >Since you have the evm setup, can you please run Linux on the evm and check if it behaves the same in Linux?
    I don't have the necessary equipment, so I can't set up evm and check it.

    When I checked again, all of the following registers were also 0xff.
    PCIE0_CORE_ECC_AGGR0(0x0002A28000)
    MMC0_CTL_CFG(0x0004F80000)
    ELM0(0x0005380000)
    GPU0(0x00053A0000)
    PCIE0_DAT(0x0005500000)
    PCIE1_DAT(0x0005600000)
    GPU0_KLIOMP1_HYD_MMRS(0x0007004000)

    All registers appear to be correct when accessed from the EndPoint side.
    Is there anything in common with the problematic registers?


    Also, I am currently using the BAR match mode, but is there any possibility that the problem will be solved by using the address match mode?
    If there is a possibility, please provide a sample using the address match mode.

  • Hi Masa,

    I don't have the necessary equipment, so I can't set up evm and check it.

    I am confused, are you not running T-kernel on DRA80XMEVM? what do you mean that you don't have the necessary equipment?

  • I'm sorry for confusing you.
    The current development uses a custom board with an AM65x.

  • I can prepare two EVMs.
    How can I get the PCIE cable to connect the EVMs?
    Is it included as standard?

  • This cable can be obtained from Adex Electronics (https://www.adexelec.com). It is mentioned in the PCIe_End_Point documentation I linked above.

  • I looked for the following URL, but I couldn't find it.
    www.adexelec.com/pcie-single-risers
    Can you tell me the product name?

  • I don't remember the part number.

    You can email them and providing the SDK doc linked above to show the picture of the cable, they will let you know what to order.

    https://www.adexelec.com/contact-us

  • >You can email them and providing the SDK doc linked above to show the picture of the cable, they will let you know what to order.
    Ok, I will check.

    BTW, the following also seems to have reported the same problem as us, that they cannot access 0x05500000 from the BAR.
    e2e.ti.com/.../3283393

    >We assumed that maybe we would be able to access the PCIe EP register space by mapping BAR0 to 0x05500000, but apparently that didn't work

    Is this still a problem that occurs with AM65xx?

  • HI Masa,

    Is this still a problem that occurs with AM65xx?

    It could be. The AM65x processor has not been changed/updated in hardware perspective since 5 years ago when the referred E2E thread conversation happened.

  • Hi Liu

    I am trying to check by running linux on a DRA80XMEVM.
    I am referring to the following procedure.

    software-dl.ti.com/.../Overview_Getting_Started_Guide.html

    I initialized the SD with “1.1.4. Create SD Card” and booted the SD, but Linux does not boot.
    I get the following message.
    What could be the problem?
    --------------------
    k3_system_controller sysctrler: k3_sysctrler_load_response: Firmware certificate authentication failed
    Firmware failed to start on rproc (-22)

    resetting ...
    k3-secure-proxy secproxy@28380000: k3_sec_proxy_send: Thread8 verification failed. ret = -61
    ti_sci dmsc: ti_sci_do_xfer: Message sending failed. ret = -61
    k3-secure-proxy secproxy@28380000: k3_sec_proxy_send: Thread8 verification failed. ret = -61
    ti_sci dmsc: ti_sci_do_xfer: Message sending failed. ret = -61
    k3-secure-proxy secproxy@28380000: k3_sec_proxy_send: Thread8 verification failed. ret = -61
    ti_sci dmsc: ti_sci_do_xfer: Message sending failed. ret = -61
    System reset not supported on this platform
    ### ERROR ### Please RESET the board ###
    --------------------

    The following images are written to the SD card
    dr-download.ti.com/.../tisdk-default-image-am65xx-evm-09.03.05.02.wic.xz

  • Hi Masa,

    First of all, please follow the instruction linked below and ensure to use an older version of Balena tool to write the WIC image to the sdcard.

    (+) [FAQ] AM62A7-Q1: Balena etcher flash error workaround - Processors forum - Processors - TI E2E support forums

    Please attach the full console log when the booting failed. It might provide more information.

  • Hi Liu

    The first terminal displayed the following(The second half is garbled.)

    ----
    Could not get FIT buffer of 267716 bytes
    check CONFIG_SYS_SPL_MALLOC_SIZE
    ??ζ繞討・fユd煇イζb??@ょ田?臚C・徴粢b??ζ繞討・fユd煇イζb??ζ繞討・Dや繚?兎泥いδ繞討庁?・やい舮?つ督F鹹Eう薙綟う粭ユd煇イζb??ζ繞討・Dや繚?兎泥いδ繞討庁?・やい舮?つ督F鹹Eう薙綟う粭ユd煇イζb??ζ繞討・Dや繚?兎泥いδ繞討庁?・やい舮?つ督F稘イb メダ$ いモ
    ----

    The second terminal displayed the following
    ----
    0x420002
    0x820004
    0x4003007
    0x4400A18
    0x42000B
    0x820004
    ----

    The message will change if you replace the file in the SD card
    sysfw-am65x_sr2-hs-evm.itb -> sysfw.itb
    tiboot3-am65x_sr2-hs-evm.bin -> tiboot3.bin
    ----
    Could not get FIT buffer of 276992 bytes
    check CONFIG_SYS_SPL_MALLOC_SIZE
    k3_system_controller sysctrler: k3_sysctrler_load_response: Firmware certificate authentication failed
    Firmware failed to start on rproc (-22)

    resetting ...
    k3-secure-proxy secproxy@28380000: k3_sec_proxy_send: Thread8 verification failed. ret = -61
    ti_sci system-controller@44083000: ti_sci_do_xfer: Message sending failed. ret = -61
    k3-secure-proxy secproxy@28380000: k3_sec_proxy_send: Thread8 verification failed. ret = -61
    ti_sci system-controller@44083000: ti_sci_do_xfer: Message sending failed. ret = -61
    k3-secure-proxy secproxy@28380000: k3_sec_proxy_send: Thread8 verification failed. ret = -61
    ti_sci system-controller@44083000: ti_sci_do_xfer: Message sending failed. ret = -61
    no sysreset
    ### ERROR ### Please RESET the board ###
    ----

  • Hi Takuma,

    Do you know what would cause the problem which Masa saw on the EVM?

    I don't have an DRA80XMEVM to test it myself.

  • Hi Liu, Takuma

    What is the situation?
    I think the procedure itself to use DRA80XMEVM is correct, but please let me know if there are any errors

  • Hi Bin and Masa,

    Apologies, I missed this thread. I also do not have a DRA80XMEVM myself.

    However, looking through past E2E threads, the error message looks similar to the one in this thread: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1042647/dra829v-u-boot-booting-on-dra829-hs-device?tisearch=e2e-sitesearch&keymatch=k3_sysctrler_load_response#

    Is this a HS sample?

    Regards,

    Takuma

  • HI Takuma and Bin

    Sorry . I was using TMDX654IDKEVM instead of DRA80XMEVM.

    In this case, is the procedure for running linux different from the following procedure?
    software-dl.ti.com/.../Overview_Getting_Started_Guide.html Overview - Introduction to

  • Hi Masa,

    I am wondering this is an old EVM which has AM65x SR1.0 device, but not SR2.0/2.1. The SDK doesn't support SR1.0.

  • Hi Bin

    Is there any way to run Linux on TMDX654IDKEVM?

  • Hi Mesa,

    I believe TMDS654IDKEVM and DRA80XMEVM are the same base board, maybe just the daughter cards are different. So the way you run Linux should be correct, but sorry I just cannot tell why the boot process stops at U-Boot, I don't have an EVM to check it on my side.

  • HI Bin and Takuma

    The following binary's “Supported products & hardware” lists “TMDX654IDKEVM.”
    dr-download.ti.com/.../tisdk-default-image-am65xx-evm-09.03.05.02.wic.xz
    Since they say they provide support, it's not acceptable to say they don't know because they don't have the device.
    Can you introduce me to someone who has a TMDX654IDKEVM?

  • Hi, I am out of office this week. Please expect delayed response.

  • HI Bin and Takuma

    When do you think you can reply to this?
    I know you are busy, but I would appreciate your response.

  • Hi Mesa,

    I am currently debugging a customer escalation, but I will try to borrow an EVM from our development team and test it out next week. I will keep you posted.

  • Hi Masa,

    I borrowed an AM65x EVM (TMDX654IDKEVM, PROC062B), and I am able to boot it from sdcard with tisdk-default-image-am65xx-evm-09.03.05.02.wic.xz.

    U-Boot SPL 2023.04-ti-g2a13324ec63c (Dec 04 2024 - 17:27:55 +0000)                                                                                                                                                                                                                                                                                                                                                                        
    SYSFW ABI: 4.0 (firmware rev 0x000a '10.1.8--v10.01.08 (Fiery Fox)')                                                                                                                                                                                                                                                                                                                                                                      
    SPL initial stack usage: 1424 bytes                                                                                                                                                                                                                                                                                                                                                                                                       
    Trying to boot from MMC2                                                                                                                                                                                                                                                                                                                                                                                                                  
    Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted                                                                                                                                                                                                                                                                                    
    Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted                                                                                                                                                                                                                                                                                    
    Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted                                                                                                                                                                                                                                                                                    
    Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted                                                                                                                                                                                                                                                                                    
    Starting ATF on ARM64 core...                                                                                                                                                                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                                                                                                                                                                                              
    NOTICE:  BL31: v2.10.0(release):v2.10.0-367-g00f1ec6b87-dirty                                                                                                                                                                                                                                                                                                                                                                             
    NOTICE:  BL31: Built : 16:09:05, Feb  9 2024                                                                                                                                                                                                                                                                                                                                                                                              
    I/TC:                                                                                                                                                                                                                                                                                                                                                                                                                                     
    I/TC: OP-TEE version: 4.1.0-51-g012cdca49 (gcc version 11.5.0 (GCC)) #1 Tue Jan 30 10:48:03 UTC 2024 aarch64                                                                                                                                                                                                                                                                                                                              
    I/TC: WARNING: This OP-TEE configuration might be insecure!                                                                                                                                                                                                                                                                                                                                                                               
    I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html                                                                                                                                                                                                                                                                                                                                   
    I/TC: Primary CPU initializing                                                                                                                                                                                                                                                                                                                                                                                                            
    I/TC: GIC redistributor base address not provided                                                                                                                                                                                                                                                                                                                                                                                         
    I/TC: Assuming default GIC group status and modifier                                                                                                                                                                                                                                                                                                                                                                                      
    I/TC: SYSFW ABI: 4.0 (firmware rev 0x000a '10.1.8--v10.01.08 (Fiery Fox)')                                                                                                                                                                                                                                                                                                                                                                
    I/TC: HUK Initialized                                                                                                                                                                                                                                                                                                                                                                                                                     
    I/TC: Activated SA2UL device                                                                                                                                                                                                                                                                                                                                                                                                              
    I/TC: Fixing SA2UL firewall owner for GP device                                                                                                                                                                                                                                                                                                                                                                                           
    I/TC: Enabled firewalls for SA2UL TRNG device                                                                                                                                                                                                                                                                                                                                                                                             
    I/TC: SA2UL TRNG initialized                                                                                                                                                                                                                                                                                                                                                                                                              
    I/TC: SA2UL Drivers initialized                                                                                                                                                                                                                                                                                                                                                                                                           
    I/TC: Primary CPU switching to normal world boot                                                                                                                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                                                                                                                                              
    U-Boot SPL 2023.04-ti-g2a13324ec63c (Dec 04 2024 - 17:27:55 +0000)                                                                                                                                                                                                                                                                                                                                                                        
    SYSFW ABI: 4.0 (firmware rev 0x000a '10.1.8--v10.01.08 (Fiery Fox)')                                                                                                                                                                                                                                                                                                                                                                      
    Trying to boot from MMC2                                                                                                                                                                                                                                                                                                                                                                                                                  
    Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted
    Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted
    Warning: Detected image signing certificate on GP device. Skipping certificate to prevent boot failure. This will fail if the image was also encrypted
                                                                                                            
                                                                                                              
    U-Boot 2023.04-ti-g2a13324ec63c (Dec 04 2024 - 17:27:55 +0000)     
                                                                                         
    SoC:   AM65X SR2.0 GP                                                                 
    Model: Texas Instruments AM654 Base Board                                                 
    Board: AM6-COMPROCEVM rev B                                                                      
    DRAM:  2 GiB (effective 4 GiB)                                                                                              
    Core:  124 devices, 32 uclasses, devicetree: separate                                      
    MMC:   mmc@4f80000: 0, mmc@4fa0000: 1                                                      
    Loading Environment from nowhere... OK                                                                                
    In:    serial                                                                              
    Out:   serial                                                   
    Err:   serial                                                                          
    am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA00102 cpsw_ver: 0x6BA80102 ale_ver: 0x00293904 Ports:1 mdio_freq:1000000            
    Detected: AM6-IDKAPPEVM rev A                                                                         
    Detected: SER-PCIE2LEVM rev A                                                                                                                 
    Net:   eth3: icssg2-eth, eth0: ethernet@46000000port@1                                                                                                
    Hit any key to stop autoboot:  0                                                                                                              
    switch to partitions #0, OK                                                         
    mmc1 is current device                                                                                    
    SD/MMC found on device 1                                                 
    Failed to load 'boot.scr'                                                                             
    574 bytes read in 28 ms (19.5 KiB/s)                                                       
    Loaded env from uEnv.txt                                                                                                                    
    Importing environment from mmc1 ...                                                                                                                   
    ## Error: "main_cpsw0_qsgmii_phyinit" not defined                                                                                          
    19507712 bytes read in 228 ms (81.6 MiB/s)                                                   
    66745 bytes read in 30 ms (2.1 MiB/s)                                                                                                            
    Working FDT set to 88000000                                                      
    11685 bytes read in 30 ms (379.9 KiB/s)                                                                                                   
    2333 bytes read in 30 ms (75.2 KiB/s)                                                                     
    ## Flattened Device Tree blob at 88000000                                    
       Booting using the fdt blob at 0x88000000                                      
    Working FDT set to 88000000                                                
       Loading Device Tree to 000000008feec000, end 000000008fffffff ... OK                                   
    Working FDT set to 8feec000                                                                                          
                                                                                                                         
    Starting kernel ...                                                                 
                                                                               
    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]                                     
    [    0.000000] Linux version 6.1.119-ti-gc490f4c0fe51 (oe-user@oe-host) (aarch64-oe-linux-gcc (GCC) 11.5.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SM
    P PREEMPT Wed Dec  4 17:16:52 UTC 2024                                                             
    [    0.000000] Machine model: Texas Instruments AM654 Base Board
    ...

  • HI Bin

    Thank you for your confirmation.

    I checked again with the same image, but the following is displayed in my environment.

    The first terminal displayed the following(The second half is garbled.)
    ----
    Could not get FIT buffer of 267716 bytes
    check CONFIG_SYS_SPL_MALLOC_SIZE
    ??ζ繞討・fユd煇イζb??@ょ田?臚C・徴粢b??ζ繞討・fユd煇イζb??ζ繞討・Dや繚?兎泥いδ繞討庁?・やい舮?つ督F鹹Eう薙綟う粭ユd煇イζb??ζ繞討・Dや繚?兎泥いδ繞討庁?・やい舮?つ督F鹹Eう薙綟う粭ユd煇イζb??ζ繞討・Dや繚?兎泥いδ繞討庁?・やい舮?つ督F稘イb メダ$ いモ
    ----

    The second terminal displayed the following
    ----
    0x420002
    0x820004
    0x4003007
    0x4400A18
    0x42000B
    0x820004
    ----

    The board used is “AM65x EVM (TMDX654IDKEVM, PROC062)”.

    I am attaching a picture of the switch. Are the settings the same?

  • Hi Masa,

    Yes your sysboot switches are correct for sdcard boot. The two terminals printed messages are from SYSFW. Basically PROC062 EVM has AM65x PG1.0 processor, which is no longer supported in the current SDK, so Linux doesn't boot on your board.

  • HI Bin

    Thank you for your response.

    Can you explain in more detail?

    Are there two types of TMDX654IDKEVM, PROC062 and PROC062B, and does Linux not boot on PROC062?
    Is the reason that PROC062 is equipped with PG1.0, while PROC062B is equipped with PG2.0, and Linux only supports PG2.0?

  • Hi Masa,

    There are likely more than 2 versions of the EVM, I walked around the office here and saw PROC062, PROC062A, and PROC062B, and I guess PROC062B is the latest version.

    Is the reason that PROC062 is equipped with PG1.0, while PROC062B is equipped with PG2.0, and Linux only supports PG2.0?

    Yes this is what I figured out today - PROC062 has PG1.0 and PROC062B has PG2.0 processors.

    And I believe the Processor SDK Linux has dropped PG1.0 support for quite a while, since PG1.0 is no longer in production for years.

  • HI Bin

    so if Linux has worked in the past, could you provide that image?

  • Hi Masa,

    All the SDK releases are listed on https://www.ti.com/tool/PROCESSOR-SDK-AM65X (click the read "Download options" right to "PROCESSOR-SDK-LINUX-AM65X", then click link "View all versions" to see the full list of SDK releases). Unfortunately there is no easy way to tell which SDK supports PG1.0 unless test it on the EVM.

  • Hi bin

    Thank you, I will check it out.
    By the way, do you have any information about when PG1.0 was supported?
    Also, did PCIE work with PG1.0?

  • Hi Masa,

    I cannot be certain, but I have a note on my computer that SDK7.0.0.5 stops supporting PG1.0. So if this note is true, SDK6.3.0.106 would be the last SDK which supports PG1.0.

    Once you flashed the SDK (newer than v6.3) Linux image to a SD card, please check the boot partition. If it has tiboot3*.bin and sysfw*.itb files with "sr1.0" or "sr2.0" in the filenames, likely the SDK would support PG1.0.

    I would think PCIe should be supported in old SDKs which support PG1.0, but won't guarantee that it won't have bugs since the kernel is old.