AM62P: BW LIMITERS - Need information on how to enable this and use it to monitor the BW information

Adding this above -

Guide me to get the BUS master frequency for each BUS on AM62P TI SDK Baremetal . I would like to know GPU BW / A53 BW etc. Currently I am exploring BW LIMITERS. 

Hello Shriya,

Thanks for adding more information.

I understand that the scope of this query is:

Understand how to setup  AM62P BW Limiters - enabling the clock if necessary, setup the BW max, etc. for "each of the AM62P BUS-es", however:

Q1: What do you mean under each BUS, if not only GPU and A53 ? Can you provide a full list of the BUS-es you are interested to know about ? Please precise  the scope of the concrete project.

Looking forward to your feedback !

Best Regards

Anastas Yordanov

Hi Anastas ,

Thank you for getting back on this. 

For my project, I need to monitor the Bandwidth for  DDR(here I am using DDR Perf example) , A53 , GPU , Display System ( DSS), CBASS . Currently I am figuring out using BW Limiter to get Bandwidth information. Provide me suggestion how to go about it as I am working AM62P TI SDK with Baremetal. 

Query -

1. BW limiters will be helpful to know BW statistics for A53 / GPU ?

2. Do we have any other examples or registers which provide BW for GPU on Baremetal ?

Thanks,

Shriya

Hi Shriya,

Thank you !

Let me review your inputs.

BR,

Anastas Yordanov

Hi Anastas,

Could you update on this ? I am looking forward to it, as this information shall be critical for my project .

Thanks,

Shriya

Hi Anastas,

I am reading the BW limiters registers but I am getting 0 value. Please help me to understand how to read these values . Sharing code snippet - 

Code ->

bwInfo = HW_RD_REG32(CSL_A53_RS_BW_LIMITER0_REGS_BASE + 0x10CU);
DebugP_log("A53 Read Bandwidth Info: 0x%08X\n", bwInfo);
bwInfo1 = HW_RD_REG32(CSL_A53_RS_BW_LIMITER0_REGS_BASE + 0x138U);
DebugP_log("A53 Read Bandwidth Max bytes: 0x%08X\n", bwInfo1);
bwlim = HW_RD_REG32(CSL_A53_RS_BW_LIMITER0_REGS_BASE + 0x4U);
DebugP_log("A53 Read Limit: 0x%08X\n", bwInfo);

Clock -> 

{ TISCI_DEV_GPU_RS_BW_LIMITER9,TISCI_DEV_GPU_RS_BW_LIMITER9_CLK_CLK, 500000000 },
{ TISCI_DEV_GPU_WS_BW_LIMITER10,TISCI_DEV_GPU_WS_BW_LIMITER10_CLK_CLK, 500000000 },
{ TISCI_DEV_A53_WS_BW_LIMITER1,TISCI_DEV_A53_WS_BW_LIMITER1_CLK_CLK, 500000000 },
{ TISCI_DEV_A53_RS_BW_LIMITER0,TISCI_DEV_A53_RS_BW_LIMITER0_CLK_CLK, 500000000 }

Kindly guide me here as I am reading 0 for all registers mentioned above.

Thanks,

Shriya

Hello Shriya,

Are you sure you have enabled the functional clock of the BW limiter.

I will try to follow-up with more information.

Thanks

Best Regards

Anastas

Hi Anastas , 

I am replying with Delay. Let me clarify the issue I am looking for - 

Problem - Need to know GPU Bandwidth when running GPU workload.

Method - Currently using BW limiters to figure it out. Let me know if there is another way to do it on AM62P + TI SDK + Baremetal.

Analysis - Trying to read BW limiters for GPU when running GPU workload ( GPU triangle example)

Output -

Before Workload : 

GPU RS BW STATS THRESHOLD: 0x00000000
GPU RS BW WINDOW CNT: 0x0000893B
GPU RS BW THRESHOLD CNT: 0x00008ED8
GPU RS BW MAX BYTES: 0x00000000
GPU WR BW STATS THRESHOLD: 0x00000000
GPU WR BW WINDOW CNT: 0x0000A0D9
GPU WR BW THRESHOLD CNT: 0x0000A676
GPU WR BW MAX BYTES: 0x00000000

After workload

GPU RS BW STATS THRESHOLD: 0x00000000
GPU RS BW WINDOW CNT: 0x000083AB
GPU RS BW THRESHOLD CNT: 0x00008948
GPU RS BW MAX BYTES: 0x00000200
GPU WR BW STATS THRESHOLD: 0x00000000
GPU WR BW WINDOW CNT: 0x00009B48
GPU WR BW THRESHOLD CNT: 0x0000A0E6
GPU WR BW MAX BYTES: 0x00000280

Observation - I see there is increase in Window cnt for Read / Write , but the value recevied on MAX BYTES in single window , is this way to calculate BW by multiplying with total windows elapsed? I want some reference here or your inputs .

Regards,

Shriya

Yes I have enabled clock. As mentioned in the previous reply -

Clock -> 

{ TISCI_DEV_GPU_RS_BW_LIMITER9,TISCI_DEV_GPU_RS_BW_LIMITER9_CLK_CLK, 500000000 },
{ TISCI_DEV_GPU_WS_BW_LIMITER10,TISCI_DEV_GPU_WS_BW_LIMITER10_CLK_CLK, 500000000 },
{ TISCI_DEV_A53_WS_BW_LIMITER1,TISCI_DEV_A53_WS_BW_LIMITER1_CLK_CLK, 500000000 },
{ TISCI_DEV_A53_RS_BW_LIMITER0,TISCI_DEV_A53_RS_BW_LIMITER0_CLK_CLK, 500000000 }

Is this the one you have asked ?

Thanks,

Shriya

Hi Shriya,

I'd like to apologize for my delayed feedback. 

shriya kulkarni said:

Do we need to enable clock for BW Limiters for A53/GPU ?

It turns out this  may be even not necessary, because there is no BW limiters local clock management registers as per the AM62Px TRM. From the Sysconfig ClockTree Tool view, the dual core A53  - AM62P52 GPU and A53 BW limiters functional / interface clocks are all sourced from one and the same 500 MHz  MAIN_SYSCLK0 clock. It is supplied and set to 500MHz by default after AM62P power-on reset.

shriya kulkarni said:

{ TISCI_DEV_GPU_RS_BW_LIMITER9,TISCI_DEV_GPU_RS_BW_LIMITER9_CLK_CLK, 500000000 },
{ TISCI_DEV_GPU_WS_BW_LIMITER10,TISCI_DEV_GPU_WS_BW_LIMITER10_CLK_CLK, 500000000 },
{ TISCI_DEV_A53_WS_BW_LIMITER1,TISCI_DEV_A53_WS_BW_LIMITER1_CLK_CLK, 500000000 },
{ TISCI_DEV_A53_RS_BW_LIMITER0,TISCI_DEV_A53_RS_BW_LIMITER0_CLK_CLK, 500000000 }

I am not a TI MCU+SDK software expert. Can you explain to me what do these TISCI APIs actually do ? (I can only guess that 500 MHz setting applied, what else ?)

As per the functional logic of the RS/WS BW Limiters which seem to be explicitly instantiated only per the GPU, A53  and the CODEC subsystems in the SoC  I could not find more info inside the publicly available AM62P TRM documentation, E2E sources or application notes. There should be a certain register programing model to be followed in order to guarantee the appropriate initialization and  accurate behavior of the registers where  you expect to see the values.

Therefore I am re-assigning this query to the pertinent AM62Px MCU+SDK hardware and software experts who can tell you more regarding the availability of the BW Limiters information in the public domain and provide the necessary references if available. Please expect that some delay is possible in their response (early next week).

Thanks for your patience !

Best Regards

Anastas Yordanov

Okay thank you for the update. 

Yes I am using TRM available to understand BW Limiters and the information is limited. Currently , I am measuring only Peak BW as shown below -

WINDOW register default = 0x400 = 1024 cycles

Window duration = 1024 / 500,000,000 = 2.048 µs

Peak BW (MB/s) = RD_BYTES_MAX /(WINDOW_cycles / 500,000,000)

TEST RESULT EXAMPLE

1. Setting threshold to 0x100 , GPU RS BW STATS THRESHOLD: 0x00000100.
2. MAX BYTES: 0x00000000 (before)
3. GPU RS BW MAX BYTES: 0x000001C0 = 448 bytes (after)
4. GPU WS BW MAX BYTES: 0x00000280 = 640 bytes
5. Peak Read BW = 448 bytes / 2.048 µs = 448 / 0.000002048 = 218,750,000 bytes/sec = 208.6 MB/s
6. Peak Write BW = 640 bytes / 2.048 µs = 640 / 0.000002048 = 312,500,000 bytes/sec = 298.8 MB/s
7. Total Peak GPU BW = 208.6 + 298.8 = 507.4 MB/s.
8. Setting threshold = 
9. A53 Read  MAX BYTES = 0x00000840 = 2112 bytes
10. A53 Write MAX BYTES = 0x00000400 = 1024 bytes
11. A53 Read  WINDOW   = 0x0000213F = 8,511 windows = 17.4 ms
12. A53 Write WINDOW   = 0x000053F1 = 21,489 windows = 44.0 ms
13. A53 Peak Read  BW = 2112 / 2.048µs = 1031.25 MB/s
14. A53 Peak Write BW = 1024 / 2.048µs =  500.00 MB/s
15. Total Peak BW : 1531.25 MB/s 

Query - 

1. Need inputs over this data.

2. Is there a way to calculate Average BW for GPU ?

Looking forward for the inputs from the experts.

Regards,

Shriya

Shriya, 

I don't believe the bandwidth limiter is capable of giving you information on the average bandwidth you are looking for. The available statistics are what you have reported, which is max bytes and peak BW. So while you can get the max bytes in a window, and the number of windows elapsed, you do not know how many bytes per window were transferred in each individual window to get an average.

Thanks,

Chris 

Hi Chris,

Thanks for the update and clarifying it.

Could you point me to alternate method to figure out the average bandwidth for GPU ? This is very important information required for our project.

I have one more method -

1. Use limiters for A53 and Codec and limit the no of bytes transferred to DDR. 

2. Run GPU workload and monitor DDR Perf counters, and also GPU BW limiters, does this help to get approximate GPU BW ?

Thanks,

Shriya

Shriya, 

I would be worried about how removing A53 and Codec traffic then measuring may impact the measurement when in real time A53 and Codec would be running. It is possible that bus or memory access by A53/Codec would impact GPU access latency and therefore performance. 

If approximate is desired maybe the BW limiter could be used by clearing the stats frequently and assuming the max throughput to be consistent of the number of elapsed windows? It would depend on the level of variability, but if you could sample the BW limiter frequently enough such that a small number of windows had elapsed it may help to be granular enough to give an approximation. 

I am also going loop in a GPU SW expert to see if there is anything that can be done from the GPU perspective to enable this request. 

Thanks,

Chris 

Hi Chris,

Thanks for the response. Yes I tried the profiling of GPU Bandwidth using BW Limiters. But I see there is huge drop in BW for GPU when compared to A53 . Do I need to enable any clock on AXI bus ? Any other parameter is responsible to increase the GPU BW ?

I am running GPU workload (GPU Triangle) for 50 msec and sampling for every 1000ms. Total 50 samples. 

AVG GPU BW by aggregating all samples -> 

===== GPU BW Final Report =====

Duration:     50167 ms

Samples:      50167

Total read:   23582208 bytes

Total write:  56468480 bytes

Avg read BW:  0.45 MB/s

Avg write BW: 1.07 MB/s

Total BW:     1.52 MB/s

Thanks,

Shriya

Next Query -

Is there any register or counter or clock on Main interconnect which will help to increase the Bandwidth ? basically we are observing very low GPU Bandwidth, I am not finding much information on PERF counter (available for DDR) for GPU / DSS / A53 ? 

Hi Krunal,

Thanks for clarifying it. 

I came across QOS which helps to set priority for GPU or A53, but it did not change GPU BW value. So figuring out similar anything is there that does - 

I need reference to registers if there is any specific to bus masters or CBASS which will help to fulfill Bandwidth requirements.

Apart from these ->

EMIF_SSCFG_PERF_CNT_SEL_REG

EMIF_SSCFG_PERF_CNT1_REG

EMIF_SSCFG_PERF_CNT2_REG

EMIF_SSCFG_PERF_CNT3_REG

EMIF_SSCFG_PERF_CNT4_REG

I am not finding an reference for Performance counters specific to GPU or A53 or other bus masters.

 Thanks,

Shriya

Hi Shriya,

The above registers only come into play if multiple initiators are accessing the DDR simultaneously and you are running into contention. I don't see such a scenario in your usecase. You mentioned you are running a triangle test in your baremetal OS GPU driver and have you analyzed the code to see if it's a bandwidth-intensive test or just a functional test?

Regards,
Krunal

Hi Krunal,

Sorry for the delay in providing response. Its not BW intensive test but functional test . But there should be a away to know the GPU Bandwidth using registers. We can close this ticket , and I shall reopen if I get any inputs. Currently using BW limiters to monitor.

Thanks,

Shriya