[AM335x] Understanding interconnect L3/4

Hi Kawada-san,

Please contact your nearest TI representative for more information. There is no other public information available than what is included in the TRM.

Kawada-san,

as you have seen the L3/L4 Interconnect is very complex and many details are intentionally left out of the TRM.  High level priority control is given to the user with the INIT_PRIORITY registers, but typically the priorities do not need to be altered.  The L3 Interconnect will handle the decoding of the initiator requests, the arbitration of requests to a single target, and proper routing of the data.  You can view the L4 as sort of a sub-interconnect which can handle a subset of initiator requests to many peripherals at a slower speed (half of the L3 speed).

Other than INIT_PRIORITY, there are separate controls for the EMIF initiators using MREQPRIO

Regards,

James

JJD said:

many details are intentionally left out of the TRM

... which is always a great help when trying to understand a complex device.

Fortunately one can get a much clearer picture of the interconnects by reading the TRMs of related devices such as the DM814x (probably closest relative), DM816x, and OMAP4 (most comprehensive docs).  It would be nice however if relevant info could actually be found in the TRM of the device one works with, instead of having to scrape together info from a multitude of sources and having to figure out which parts do or don't apply.

Hi James,

Thanks for your reply. I understood MREQPRIO and INIT_PRIORITY registers were the only input parameters for the interconnects, but it's still unclear for me that we don't have to configure these registers typically.. How these registers can effect to interconnects ?

I seem the interconnects would handle the data traffic appropriately under something interconnect's rule, but as Matthijs said, understanding its rule/scheme should be helpful to debug enigmatic behavior related to heavy data traffic in interconnects. 

Best Regards,
Kawada 

Naoki Kawada said:

debug enigmatic behavior related to heavy data traffic in interconnects. 

I should also mention that in addition to statistics collectors there is a module called "OCP Watchpoint" (OCP-WP) which acts as a kind of packet sniffer on the L3 interconnect and is meant to be of help in such situations.  It is however, unfortunately, completely undocumented (publicly).

Matthijs

Hi Matthijs,

Thanks for your reply. Very interesting. I'll try to get OMAP44xx TRM and understand the details of interconnects registers/behaviors. But it may not still help AM335x users because TI does not mention the details of interconnect register banks in AM335x TRM...  I hope TI would consider about the full description in TRM (or something application report). 

Best Regards,
Kawada

As an addendum, I did a quick scan of the am335x L3 interconnect register blocks and was able to identify most of them with ease:

440000xx host L3F
440001xx -
440002xx target 0x02 SHA
440003xx target 0x03 AES 1
440004xx target 0x10 OCMC RAM
440005xx target 0x04 AES 0
440006xx target 0x06 Expansion Port
440007xx target 0x07 EDMA TC 0
440008xx target 0x08 EDMA TC 1
440009xx target 0x09 EDMA TC 2
44000axx target 0x05 L4-Fast
44000bxx target 0x0b EDMA CC
44000cxx target 0x0e SGX
44000dxx target 0x1f DebugSS
44000exx target 0x0f PCIe
44000fxx target 0x01 EMIF
440010xx flagmux L3F
440011xx flagmux top-level
44002xxx statcoll 0
44003xxx statcoll 1
44004xxx statcoll 2
44005xxx bw regulator

448000xx host L3S
448001xx target 0x11 L4-Per (initiator 0)
448002xx target 0x12 L4-Per (initiator 1)
448003xx target 0x13 L4-Per (initiator 2)
448004xx target 0x14 L4-Per (initiator 3)
448005xx target 0x0a ADC/TSC
448006xx flagmux L3S
448007xx target 0x1e GPMC
448008xx target 0x20 McASP 0
448009xx target 0x21 McASP 1
44800axx target 0x22 McASP 2
44800bxx target 0x27 USB
44800cxx target 0x26 MMC 2
44800dxx target 0x1b L4-Firewall
44800exx target 0x0d L4-Wakeup
44804xxx statcoll 3

I have no idea which bandwidth is being regulated by the bandwidth regulator present in the L3F, nor which statistics can be collected by the statistics collectors, but still, it's a start.

Naoki Kawada said:

But it may not still help AM335x users because TI does not mention the details of interconnect register banks in AM335x TRM...  I hope TI would consider about the full description in TRM (or something application report)

I agree that more public documentation would be good to have, and not just on the interconnect: the debug subsystem is another example where much documentation is missing.

Matthijs

Edit (May 30): fixed two entries in list and identified the two remaining targets.

Naoki Kawada said:

I understood MREQPRIO and INIT_PRIORITY registers were the only input parameters for the interconnects, but it's still unclear for me that we don't have to configure these registers typically.. How these registers can effect to interconnects ?

I've only just learned this myself, so I'll share:

All requests to the L3 interconnect have a 2-bit (1-bit on older devices) "pressure" value which is used for arbitration.  If I understand correctly, this is a hard priority: when two streams are competing for a shared interconnect path which has reached its maximum capacity, if one has higher pressure it can consume all bandwidth and stall the lower-pressure requests, so this mechanism has to be used with care.  Requests with the same pressure are serviced round-robin.

The pressure value used by an initiator can be derived in several ways: for most initiators it is statically configured in the INIT_PRIORITY registers in the control module.  It can also be dynamically determined, and finally a mixed case is possible: one bit dynamic and one bit from INIT_PRIORITY.  Dynamic pressure may come from some kind of signal of urgency from within an initiating subsystem (HDVPSS on DM814x apparently does this) but typically it is provided by a "bandwidth regulator", which is a token bucket filter which can assign higher pressure to "conforming" flow than to "non-conforming" flow.  The bandwidth, max burst, and both pressure levels can be freely configured.

As I mentioned in a previous post, the am335x apparently has one bandwidth regulator.  Although it would be more convenient if someone from TI were kind enough to mention which initiator it is attached to, this can also be determined experimentally:  the pressure of a request is logged in L3 interconnect errors, so all you have to do is change the pressure levels, convince various initiators to make an invalid request, and examine the errors produced.  My guess would be it's either attached to one of the EDMA TCs, or to SGX.

MReqPrio is a very different story:  this is not used by the interconnects but only for EMIF arbitration.  On DM814x only HDVPSS is able to dynamically decide this, while the MReqPrio for other initiators is determined by the "Dynamic Memory Manager" (DMM) "PEG" registers.  AM335x has no DMM, hence there's a control module registers for it instead.  How exactly this value is used by the EMIF is something I'm still studying myself, EMIF can map the priority to "class of service" (PRI_COS_MAP), but can also involve the initiator id (CONNID_COS_x_MAP), and is able to do priority escalation (INT_CONFIG) register.  One thing to note is that apparently EMIF priority levels (unlike pressure levels) are inverted: 0=highest prio, 7=lowest prio.