TMS320C6678: TMDSEVM6678 with TMDXEVMPCI do_dsp_dma() hangs on CentOS 7

Hi, Antti,

what kernel version is the CentOS, and what patch does it have from TI distribution?

Rex

Hi Rex,

CentOS kernel is 3.10.0-1062.el7.x86_64. And we we are using desktop-linux-sdk_01_00_03_00.

Br, Antti

Hi,

This is a case using EDMA for transfer. After the transfer finished, the IPR bit corresponding to the DMA channel used is suppose to set. The driver code polls the bit but it is not set, so it stuck.

First thing is you need to make sure if there are simple transfers using CPU, not EDMA, it can work. So we know this driver is functioning, after your copy_from_user & copy_to_user change.

Next, this driver uses EDMA CC0, the base address is 0x0270_0000. The PCIE driver must have change the BAR to access this region, we need to check if BAR is programmed to access this and inbound translation is setup. If you are familiar with PCIE, you can decode 0x2180_1000 region for BAR0-BAR5 setup, and 0x2180_0300 region for IB setup, for IB_BAR0 to IB_BAR3, using PCIE UG http://www.ti.com/lit/ug/sprugs6d/sprugs6d.pdf. If you are not sure, please dump the registers to me, both with length 0x100 are enough for me. 

Assuming above is correct, then we need to check EDMA is correctly programmed or not. EDMA CC0 only has 16 DMA channels with 128 Paraset. The driver starts from channel 0 and paraSet 0 and incremental. Given it is stuck, I assume we should see problem from the first channel and first Paraset. If you are familiar with EDMA, you can check some thing like DMAQNUM, DCHMAP, IESR, ICR, IPR registers. They are starts from 0x270_0000. And paraSet starts from 0x270_4000, it is 0x20 in length for each group, you can decode following 2.3.1 PaRAM Set structure in EDMA UG http://www.ti.com/lit/ug/sprugs5b/sprugs5b.pdf. Or, you can dump the EDMA registers 0x270_0000 for 0x2000 length. And 0x270_4000 for 128* 0x20 length for me to check.

Regards, Eric 

Hi Eric,

Thanks for the help. run_memcpytest.sh works just fine.

Not that familiar with PCIE, Just a basic a question how to dump the registers? lspci -xxx? 

04:00.0 Multimedia controller: Texas Instruments Device b005 (rev 01)
00: 4c 10 05 b0 47 00 10 68 01 00 80 04 00 00 00 00
10: 00 00 90 d7 08 00 40 d1 08 00 00 d1 08 00 00 d0
20: 08 00 49 d1 08 00 48 d1 00 00 00 00 00 00 01 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 00 00
40: 01 50 03 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 80 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 00 02 00 01 87 e8 07 3f 38 0b 00 22 54 03 00
80: 80 00 21 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 1f 00 00 00 00 00 00 00 06 00 00 00
a0: 02 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

We managed to run_dmatest.sh after running setpci -s 04:00.0 COMMAND=0x67 This is trial and error. Is there some reference manual for those commands? We had already before run setpci -s 04:00.0 COMMAND=0x02, because  lspci showed BAR memory regions as disabled.

However, testout.file doesn't match testfiles/test.file. Testout.file always starts with this

# od -x -N 100 testout.file
0000000 4b91 0f35 9dbb f82d c626 9da0 21fa 02b9
0000020 cca5 4c02 7dc2 ba21 75ba 8feb 965c 7846
0000040 6144 6ad9 9470 8f65 3957 585c 00ca 3f29
0000060 3202 81d2 b6a4 8cc7 1087 01c2 fbad c1ac
0000100 484a e292 365e df01 f43b 7f75 e6e5 2cfc
0000120 f3ff fab1 e0bb 949f 1980 ea8e ecdc 06b5
0000140 1c91 f3ee

Br, Antti

Hi,

If you use lspci command, it only dumps the PCIE configuration registers, that starts from 0x2180_1000. This command can't access PCIE application register 0x2180_0000 and EDMA registers 0x0270_0000 and 0x0270_4000. So, please use a JTAG to connect to DSP core without using GEL to capture all the registers required, if you have access to the setup.

>>>.setpci -s 04:00.0 COMMAND=0x67 >>>>> Where did you get this command? I searched the desktop-linux-sdk_01_00_03_00 and didn't see this. It is a Linux command to set PCIE registers: https://www.tutorialspoint.com/unix_commands/setpci.htm Why you need to do this?

From your dump:

00: 4c 10 05 b0 47 00 10 68 01 00 80 04 00 00 00 00
10: 00 00 90 d7 08 00 40 d1 08 00 00 d1 08 00 00 d0
20: 08 00 49 d1 08 00 48 d1 00 00 00 00 00 00 01 00

You have 6 32-bit BAR. In your old post: https://e2e.ti.com/support/processors/f/791/t/897184, you mentioned: memory mapped transfers were not enabled by default. "setpci -s 02:00.0 COMMAND=0x02" solved that issue. What register value changed after that? from ... to ....?

In this post, you showed 

Hi,

After reboot on CentOS 7.7 (kernel 3.10.0-1062.el7.x86_64) lspci -v shows

04:00.0 Multimedia controller: Texas Instruments Device b005 (rev 01)
Flags: fast devsel, IRQ 11
Memory at d7900000 (32-bit, non-prefetchable) [disabled] [size=1M]
Memory at d1400000 (32-bit, prefetchable) [disabled] [size=512K]
Memory at d1000000 (32-bit, prefetchable) [disabled] [size=4M]
Memory at d0000000 (32-bit, prefetchable) [disabled] [size=16M]
Memory at d1490000 (32-bit, prefetchable) [disabled] [size=4K]
Memory at d1480000 (32-bit, prefetchable) [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting

Then after setpci -s 04:00.0 COMMAND=0x2

04:00.0 Multimedia controller: Texas Instruments Device b005 (rev 01)
Flags: fast devsel, IRQ 11
Memory at d7900000 (32-bit, non-prefetchable) [size=1M]
Memory at d1400000 (32-bit, prefetchable) [size=512K]
Memory at d1000000 (32-bit, prefetchable) [size=4M]
Memory at d0000000 (32-bit, prefetchable) [size=16M]
Memory at d1490000 (32-bit, prefetchable) [size=4K]
Memory at d1480000 (32-bit, prefetchable) [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting

In this thread e2e.ti.com/.../311641 there was similar issue. There it was suspected something do with X Display.

JTAG still needs some time. Cable is not connected and I am able to visit the office at the moment.

Br, Antti

Antti,

Thanks for the update. All BARs are 32-bit, that are expected. Then all the PCIE address will be 32-bit also. It is unclear to me the Linux showed 64-bit address earlier. 

I am not sure exactly what [disabled] meant in the lspci log. Some info I found: https://stackoverflow.com/questions/23414288/do-i-need-to-enable-a-pcie-memory-region-in-a-linux-3-12-driver; https://forums.xilinx.com/t5/PCIe-and-CPM/lspci-reports-BAR-0-disabled/m-p/752779. It looks "So either you're not inserting the driver, or something went wrong in the driver insertion." and setpci -s <BUS_ADDR> COMMAND=0x02 can resolve this, as you did. 

Waiting for your JTAG connection some time when feasible. Stay safe!

Regards, Eric

Hi,

Below is the data from registers in the DSP core:

0x21800000    

4E301101    00000007    00000000    00000000   00000000    00010000    00000000    00000000    00000000   00000000   00000000    00000000    00000002   00000000    00000000    00010000    00000000    00000000    00000000

0x02700000    

4001AB00   03313243    00000000    00000000    00000000    00000000    00000000    00000000    00000000   00000000   00000000    00000000    00000000   00000000    00000000    00000000    00000000    00000000    00000000

0x02704000 

85C976B3    EFFDEC79    1B02208C   D6E08C30    AF857502    5A4238C1    56F52383    FABE22F4    45D91D33   C07C8D56   613CF0A2    BF120608   96C91427    1A45B1F9    F1DF9B07    6E0C689D    7ED6792F    AC346CC8    5F6B2C5D

Best regards,
Aleksi

Aleksi,

Thanks for the efforts! This is closer.

If you are familiar with PCIE, you can decode 0x2180_1000 region for BAR0-BAR5 setup, and 0x2180_0300 region for IB setup, for IB_BAR0 to IB_BAR3, using PCIE UG http://www.ti.com/lit/ug/sprugs6d/sprugs6d.pdf. If you are not sure, please dump the registers to me, both with length 0x100 are enough for me. 

If you are familiar with EDMA, you can check some thing like DMAQNUM, DCHMAP, IESR, ICR, IPR registers. They are starts from 0x270_0000. And paraSet starts from 0x270_4000, it is 0x20 in length for each group, you can decode following 2.3.1 PaRAM Set structure in EDMA UG http://www.ti.com/lit/ug/sprugs5b/sprugs5b.pdf. Or, you can dump the EDMA registers 0x270_0000 for 0x2000 length. And 0x270_4000 for 128* 0x20 length for me to check.

So, I didn't get what I need. 

Regards, Eric

So simplicity, just dump 0x2180_0000 for 0x2000 length, 0x0270_0000 for 0x2000 length, 0x0270_4000 for 0x2000 length for me. You can use the CCS memory window to save it into a file and give to me.

Regards, Eric

Hi Eric,

Hopefully this is the right data you wanted. I am not familiar with CCS so please bare with me if I didn't get it right. 

Regards, Aleksi

0x0270_0000.dat

0x0270_4000.dat

0x2180_0000.dat

Hi,

Thanks, that is what I needed, please give me some time to decode. Also, please save it as 32-bit (not 64-bit) next time.

Regards, Eric

Hi,

Decoding of the PCIE registers, 0x2180_0000 dump:

Outbound (irrelevant for now)

OB SIZE: 2 (4MB)
OB: EF800001

INBOUND:
D140000000000001
0270000000000000
D100000000000002
2140000000000000
D000000000000003
8F00000000000000
D149000000000004
0000000000000000

BAR1: D1400000 ====>02700000 EDMA region
BAR2: D1000000 ====>21400000 Hyperlink region, no use
BAR3: D0000000 ====>8F000000 DDR region
BAR4: D1490000 ====>00000000 no use

BAR0-5:
D1400008D3000000
D0000008D1000008
D1480008D1490008

BAR0: D3000000, 32-BIT, MEMORY BAR, NON-PREFETCHABLE
BAR1: D1400008, 32-BIT, MEMORY BAR, PREFETCHABLE
BAR2: D1000008, 32-BIT, MEMORY BAR, PREFETCHABLE
BAR3: D0000008, 32-BIT, MEMORY BAR, PREFETCHABLE
BAR4: D1490008, 32-BIT, MEMORY BAR, PREFETCHABLE
BAR5: D1480008, 32-BIT, MEMORY BAR, PREFETCHABLE

Conclusion: Linux machine enumerated PCIE card properly and assigned BAR0-5 addresses based on BAR mask. The Linux machine also programmed the PCIE inbound translation properly, any incoming access to EDMA region should be mapped properly, the Linux PC should be able to program EDMA registers to start a transfer.

If you do a lspci from the Linux machine, you should be able to see above, like you got before:
Memory at d7900000 (32-bit, non-prefetchable) [disabled] [size=1M] =====> some mismatch here, I guess collected from different runs
Memory at d1400000 (32-bit, prefetchable) [disabled] [size=512K]
Memory at d1000000 (32-bit, prefetchable) [disabled] [size=4M]
Memory at d0000000 (32-bit, prefetchable) [disabled] [size=16M]
Memory at d1490000 (32-bit, prefetchable) [disabled] [size=4K]
Memory at d1480000 (32-bit, prefetchable) [disabled] [size=64K]

Regards, Eric

Hi,

Decode EDMA ParamSet 0x0270_4000 region:

Only the first one is programed, rest are garbage. This is expected given we suspect to stuck at the first transfer.

900000000B10200C
6000000000010064
0000FFFF00800080
0000000100000000

OPT field: A-B SYNC transfer, TCC code is 2 (this is typically DMA channel 2), an interrupt will be generated once the transfer is completed.

Source: 0x9000_0000 this is DDR

ACNT = 100 and BCNT = 1, means each time we transfer 100 bytes.

Destination is: 0x6000_0000, this is a PCIE write into Linux PC memory.

I need to check the code what the DMA test intended to do.

Regards, Eric  

Hi,

Decoding EDMA 0x0270_0000 dump:

0000000000000000 ParamSet 0 used on channel 2
0000000000000004 EVT2 enable
0000000000000004 IER2 enable
0000000000000000 IPR is not set

So the transfer interrupt for channel 2 is enabled, but the transfer didn't happening. The do_dsp_dma() is called multiple times from the code, are you able to add some printk inside the Linxu driver to see which call stuck from the top level do_filetest() function?

Regards, Eric

Hi,

I think the driver doing anything. Is the idea in DMA, that driver just maps the memory to user-mode address. It seems to be in this while loop in sdk\pciedrv\src\pciedrv.c around line 1370. tmp is always 0

Hi,

What you mentioned is a fact, but it is not the root cause. The Linux driver configures the EDMA of DSP through PCIE  interface. We saw the correct data in 0x2704000 region, this is through the inbound translation correctly setup earlier. I am not sure about your user-mode meaning, but it doesn't matter as we can access the EDMA region from Linux.

The Linux has the access to the 0x2700000 region as well, so it can submit a manual EDMA transfer by setting the ESR bit, this bit you can't readback from JTAG or Linux CPU, it is read as 0 only. If the transfer happened, the IPR bit will set. This can be both viewed by Linux driver and JTAG.

The Linux driver code is stuck at polling IPR, we also knew that JTAG showed nothing set. We need the reason why the EDMA transfer didn't happen. 

The test is a file loopback, the file is in the Linux PC, it is sent to DSP via EDMA read (DSP EDMA reads the Linux host machine memory to get the data), then DSP loops it back via EDMA write (from DSP memory into Linux PC memory). Given the dump showed the EDMA ParamSet source address is 0x90000000, destination is 0x60000000. This is EDMA write. It means the Linux sent the data and received by DSP already. So the EDMA code should work in general, but for some reason it stuck from the DSP to PC direction.

The 0x60000000 address translate to 0xEF800000 into Linux memory. I can't tell what it is in Linux. E.g. when Linux writes into DSP, what is the EDMA source address, is it closer to this 0xEF800000? That is why I asked if you can add some printk to trace the call flow.

Given the remote work nature, I don't have this setup. Also, this is a legacy software, it is not maintained. It may have problems when people starting with it. Certainly there are EDMA err stat, event miss registers to check and trace the EDMA ParamSet and channel usage from the beginning (from Linux PC to DSP) to understand this issue, but it is not easy to follow if I asked you repeatedly for dumps.

Regards, Eric

Hi,

Sorry, I had a misunderstanding about driver. I thought it incorrectly as cmem_dev.ko Linux kernel module

So now when looking memory mapping in kernel module:

[ 3036.204767] cmem cmem: Mapping 0x400000 bytes from address 0xef800000
[ 3036.204769] cmem: start 00007fca63c1a000 pgoff 00000000000ef800 prot 39
[ 3036.204781] cmem: % remap_pfn_range 0

If I'm interpreting the logs right, 0xef800000 is mapped to user-mode addresses between 00007fca63c1a000 - 00007fca6401a000, but mem_region_base isn't in that range. I'll continue with debugging where this mem_region_base comes from.

Debug: do_dsp_dma: dsp_id : 0, dma_chan_num 2 dstAddr 0x98000000, srcAddr 0x60000000, size 0x4000, flag 0x2
ib_map_dsp_addr 2700000
Debug: do_dsp_dma: got bar number: 1
Debug: do_dsp_dma: Got semaphore
Debug: do_dsp_dma: mem_region_base: 0x7fca668d6000
Debug: do_dsp_dma: dmaqnum_addr: 0x7fca668d6240 tmp 0x100
Debug: do_dsp_dma: dchmap_param: 0x7fca668d6108
srcAddr to address 0x7fca668da004
dstAddr to address 0x7fca668da00c

Br, Antti

Hi,

If I'm interpreting the logs right, 0xef800000 is mapped to user-mode addresses between 00007fca63c1a000 - 00007fca6401a000 >>> I think so. But you are more familiar with Linux kernel than me.

Debug: do_dsp_dma: dsp_id : 0, dma_chan_num 2 dstAddr 0x98000000, srcAddr 0x60000000, size 0x4000, flag 0x2 >>>> I believe this is the beginning of the transfer, Linux PC writes the file to DSP. From the DSP point of view, it is an EDMA read operation from Linux memory into DSP DDR, so we saw the source address is 0x6000_0000, destination is 0x98000000. The 0x6000_0000 is the DSP PCIE data region (for outbound), it translated into 0xEF80_0000 with 4MB. Then mapped to the user mode 64-bit address in Linux PC as you shown.

Debug: do_dsp_dma: dmaqnum_addr: 0x7fca668d6240 tmp 0x100
Debug: do_dsp_dma: dchmap_param: 0x7fca668d6108 >>>> Those are registers inside EDMA (0x270_0000 region, offset 0x240 and 0x108). Those address (seen from Linux side to program them) are out of the range as you found out.

Regards, Eric

Hi,

I continued debugging. But now I'm constantly having trouble reading memory at 0x2180_0000. What might cause this? Resetting dsp also fails at this point.

./dspallreset.sh 1
Num of devices 1
" ERROR: Reset code is not working : Timedout
Dsp 0: DSP Reset Fail !

Br, Antti

Hi,

Can you confirm you didn't accidentally change the dip switch setting? That is, you still in PCIE boot mode? When in this mode, the IBL code (in EEPROM) powered on the PCIE domain and enabled the PCIE training. If PCIE is powered off, you can connect JTAG but when looking at the 0x2180_0000 region it will hang.

Will power cycle the Linux PC resolve this? Are you able to enumerate the PCIE card? E.g. check with lspci command? The lockup happened after you did some test or just after you boot up the Linux PC?

Regards, Eric

Hi,

I didn't accidentally change the dip switch setting. I'm working remotely with that machine. Power cycle the Linux PC resolves this. 

After power cycle, I can see 0x2180_0000 memory just fine. When running run_dmatest.sh, it goes to "device core is hung" state. That happens already before do_dsp_dma(). Need to continue debugging, but rebooting PC is little slow process remotely...

Next time when visiting the office, I'll also check how it behaves when Linux PC is Ubuntu 12.04.

Br, Antti

Hi,

I tested run_dmatest.sh on Ubuntu 12.04 64-bit and it behaves same way.  do_dsp_dma waits for interrupt and "device core is hung" is shown in CCS. This is on HP Z240 workstation. I tested also on older HP Z420 and Ubuntu 12.04, same thing. I tried some newer machine HP Z2 and prodesk. But on those machines EVM board doesn't show up.

Hi,

Thanks for trying on different computers and OS. The KeyStone I (6657, 6670, and 6678) EVM is an AMC form factor card that provides multiple connectivity options for inter-connection with various systems, primarily for telecommunications. In addition, the device can be connected to a standard PCIe slot through an AMC-to-PCIe adapter card. The manufacturer had tested the adapter card with multiple platforms, however, the adapter card is not designed for complete PCIe compatibility. The most significant compatibility issue is the lack of a PCIe reset control on the adapter card. The PCIe reset is not propagated to the system-on-chip (SoC); this can cause enumeration issues or boot-up issues.

Most likely, the host PC sends a hot reset to the C6678 card after enumeration, putting the C6678 in the PCIE link training disabled state (0x2180_0004 bit 0 changed from 1 to 0), so the PC can't see the card anymore. There is no easy way to recover it.

So please still stick to the setup that enumeration worked. Was the DMA transfer caused the core hang before? Or this is after some driver code  change? 

If the DMA case didn't work, did you debug the CPU file loopback case? In which direction the file corruption happened? E.g, from the host PC you may check how the binary file looked like by a binary viewer, then did the PC write it correctly into DSP's memory? And DSP to PC corrupted the data?

Regards, Eric

Hi,

On the Ubuntu 12.04, there wasn't any driver code changes. I used Desktop Linux SDK 01_00_03_00_64bit without any code changes.

EVM is now on Windows machine. I'll return to Linux DMA debugging after couple of weeks.

Br, Antti