CCS/TMS320C6657: SRIO lane speed issues (TI to TI, TI to Zynq comm)

Josh Cherry

Part Number: TMS320C6657

Tool/software: Code Composer Studio

Hi TI experts,

We are configuring a TI DSP (c6657) board to talk to another of c6657 board, and also a Xilinx Zynq board. The set up includes all three boards & a Vadatech UTC 2002 switch in a MicroTCA Chassis. Upon configuring the TI DSP board that we are using as our master for different numbers of lanes (1 or 4 lanes), we do not see a linear change in speeds (very little change in speeds at all, about 10%). We have also noticed that the RIO_PLM_SP_PATH_CTL register shows does show Configuration 4 mode 0 (when attempting single lane) or Configuration 4 mode 4 (while in four lanes). We have compared our results to the "Throughput Performance Guide for KeyStone II Devices" and see that our speed changes should be changing proportionally when changing from 1 lane to 4 lane communications. Ours are most certainly not, but we have done everything in our ability to figure out why, but to no avail.

We also see very different speeds when performing NREADs from TI(A) to TI(B) and from the Zynq to TI(B), which we believe is to be expected because the "TI has eight LSU register sets, allowing for eight outstanding requests for all transaction types that require a response," while the Zynq needs to wait after each packet for the response. Is this what should be expected? We are seeking assistance in finding the root of this issue. Thanks for your time.

Bandwidths:

4 lane NREAD TI(A) to TI(B): 235MB/s

1 lane NREAD TI(A) to TI(B): 226MB/s

4 lane NREAD TI(A) to Zynq: 235MB/s

4 lane NREAD Zynq to TI(B): 19MB/s

1 lane NREAD Zynq to TI(B): 19MB/s

over 5 years ago

0 Yordan Kovachev over 5 years ago

TI__Guru**** 161600 points

Hi Josh,

Are you using the TI RTOS? If not and this is a custom application code, can you compare your SRIO settings against the ones done in the TI RTOS driver:
PDK_INSTALL_DIR\packages\ti\drv\srio\device\<device>\src\

Best Regards,
Yordan

0 Josh Cherry over 5 years ago in reply to Yordan Kovachev

Prodigy 100 points

I apologize for the delayed response. We are using the TI RTOS. We are using a modified version of the device_srio_loopback.c file found under the pdk_c665x_2_0_7 directory to initialize our SRIO device. Thanks

0 Yordan Kovachev over 5 years ago in reply to Josh Cherry

TI__Guru**** 161600 points

Can you provide a register dump, so we can check the configuration?

Best Regards,
Yordan

0 Josh Cherry over 5 years ago in reply to Yordan Kovachev

Prodigy 100 points

Is there a better method to provide this than copy & pasting all the registers into a text file? Maybe a 'save target register states' command?
Thanks,
Josh

0 Yordan Kovachev over 5 years ago in reply to Josh Cherry

TI__Guru**** 161600 points

Hi Josh,

Yes, there is. From CCS click View -> Registers. Then click on (let's say) Core0 in the debug window, you will see its modules' registers in the right part of the CCS window:

Scroll down to SRIO module and right click on it. Then click Export, you will see the following pop-up:

Select the SRIO related check boxes and navigate to the file, where you want to export it.

Also I will assign this to the design team, as I will be OoO next week and won't be able to provide timely responses.

Best Regards,
Yordan

0 Josh Cherry over 5 years ago in reply to Yordan Kovachev

Prodigy 100 points

Thanks for the help,

I have the register dump available, but cannot post it here (due to company policy). Is there someone I can email directly for further help, or just to provide the register dump?

Thanks,
Josh

0 Garrett Ding over 5 years ago in reply to Josh Cherry

TI__Mastermind 43296 points

Josh,

I received the register dump from Billy, will look into it. Can you please clarify the test case while dumping the registers?
- configuration
- mode
- NREAD path

Regards,
Garrett

0 Josh Cherry over 5 years ago in reply to Garrett Ding

Prodigy 100 points

Garrett,

Config: 4,
Mode: 4
NREAD path: from TI(A) through Vadatech switch (which has a routing table configured) and to TI(B)

Thanks,

Josh

0 Garrett Ding over 5 years ago in reply to Josh Cherry

TI__Mastermind 43296 points

Josh,

The register dump with 'file export' doesn't log the serdes registers (from 0x0260154) contents. Did you check the serdes configurations for each test cases you have run? If you compare the serdes configurations in the 5 test cases, it might help to figure out why the bandwidth is not proportional when changing the lanes.

I have my EVMC6678 EVM set up and dumped the SRIO registers, bit-by-bit comparing with yours....

Regards,
Garrett

0 Josh Cherry over 5 years ago in reply to Garrett Ding

Prodigy 100 points

Garrett,

Thanks for the help. I'm assuming you meant 0x02620154. Checking the value for the current configuration, we have 0x08112245. I will test with more configurations, but based on the Keystone 3-31, it doesn't seem to have an issues.

Thanks again,
Josh

0 Garrett Ding over 5 years ago in reply to Josh Cherry

TI__Mastermind 43296 points

Josh,

The more important one Serders register is SRIO_SERDES_CFGPLL SerDes (0x02620360), see the Serdes register list in SPRUGW1B table 3-2.

SERDES MACRO REGISTERS
0x02620154 SRIO_SERDES_STS SerDes Macro Status Register
0x02620360 SRIO_SERDES_CFGPLL SerDes Macro Configuration Register

SERDES RECEIVE/TRANSMIT CHANNEL CONFIGURATION REGISTERS
0x02620364 SRIO_SERDES_CFGRX0 SerDes Receive Channel Configuration Register
0x02620368 SRIO_SERDES_CFGTX0 SerDes Transmit Channel Configuration Register
0x0262036C SRIO_SERDES_CFGRX1 SerDes Receive Channel Configuration Register
0x02620370 SRIO_SERDES_CFGTX1 SerDes Transmit Channel Configuration Register
0x02620374 SRIO_SERDES_CFGRX2 SerDes Receive Channel Configuration Register
0x02620378 SRIO_SERDES_CFGTX2 SerDes Transmit Channel Configuration Register
0x0262037C SRIO_SERDES_CFGRX3 SerDes Receive Channel Configuration Register
0x02620380 SRIO_SERDES_CFGTX3 SerDes Transmit Channel Configuration Register

Regards,
Garrett

0 Josh Cherry over 5 years ago in reply to Garrett Ding

Prodigy 100 points

Garrett,

The values pulled from the registers (directly after setup & configuring the routing table) are below:
0x02620154: 0x08112245
0x02620360: 0x00000229
0x02620364: 0x00440495
0x02620368: 0x00180795
0x0262036C: 0x00440495
0x02620370: 0x00180795
0x02620374: 0x00440495
0x02620378: 0x00180795
0x0262037C: 0x00440495
0x02620380: 0x00180795

Thanks again for your support,
Josh

0 Garrett Ding over 5 years ago in reply to Josh Cherry

TI__Mastermind 43296 points

Josh,

So from the serdes registers, we know the MPY = 5, and operating rate is half (01b), what is your SRIO reference clock frequency and did you refer to the Table 3-7 Frequency Range versus MPY Value in UG to check if the serders are configured properly in all test cases?

Regards,
Garrett

0 Josh Cherry over 5 years ago in reply to Garrett Ding

Prodigy 100 points

Garrett,

Our ref clk is 250MHz, and we did verify that the Data Rate was 2.5 Gbps, as we expected. This value shouldn't change as we configure for different lane combinations, should it?

Again, we greatly appreciate the help.

Thanks,
Josh

0 Garrett Ding over 5 years ago in reply to Josh Cherry

TI__Mastermind 43296 points

Josh,

Yes, the data rate (2.5Gbps) should be consistent for different lane combinations.

>>We are using a modified version of the device_srio_loopback.c file found under the pdk_c665x_2_0_7 directory to initialize our SRIO device.

What are the modification you have done? Have you tried to measure the throughput with the breakout card setup like described in the thread e2e.ti.com/.../436245
for the test case -
4 lane NREAD TI(A) to TI(B): 235MB/s
1 lane NREAD TI(A) to TI(B): 226MB/s

Did you find out the difference of Serdes and SRIO register values in the two test cases with your setup?

Regards,
Garrett

0 Josh Cherry over 5 years ago in reply to Garrett Ding

Prodigy 100 points

We have continued to have issues, but now think it may be easier to utilize code that TI has rendered. Do you have the code associated with the results from the throughput performance document that we could try to use? I apologize for the delayed response, we have had a busy week.

Thanks again,
Josh

0 Garrett Ding over 5 years ago in reply to Josh Cherry

TI__Mastermind 43296 points

(Edited)

Josh,

The code associated with the throughput performance document should be the old MCSDK - software-dl.ti.com/.../index_FDS.html for keystone 2 device, and for keystone I e.g. C6657 device, which are no longer maintained, and we have moved to Processor SDK as you may know.

Regards,
Garrett

0 Josh Cherry over 5 years ago in reply to Garrett Ding

Prodigy 100 points

Garrett,

We utilized the "Tput" code in SRIO from the BIOS-MCSDK 02 01 02 above. We ran the code in three different configurations, but found the same results. We are still using the C6657 board. What might the issue be?

4x Lane, 5 GBaud
Bytes | Thruput
4 | 83.33
8 | 166.67
16 | 333.33
32 | 666.67

1x Lane, 5 GBaud
Bytes | Thruput
4 | 83.33
8 | 166.67
16 | 333.33
32 | 666.67

1x Lane, 3.125 GBaud
Bytes | Thruput
4 | 83.33
8 | 166.67
16 | 333.33
32 | 666.67

Thanks,
Josh

Processors

Processors forum

CCS/TMS320C6657: SRIO lane speed issues (TI to TI, TI to Zynq comm)