Which memory map sections to put in L2 with the NDK ?

Hi Clement,

I am in the process of looking into this and will get back to you on it shortly.

Steve

Hi Steve,

Good.

Another thing to consider is my setup:

L1D: 100% cache
L2: 100% data/code, no cache but cacheable
DDR3: Uncacheable

Clement

.const                             DDR3 ?

far                                  DDR3 ?

DDR3.  NDK defines some .far subsections for large buffers that store packets and for its memory manager.  See section 3.1.3 Global Buffer Configuration of the NDK User's Guide for more info.

.resmgr_memregion            L2 ?
.resmgr_pa                             L2 ?
resmgr_handles                    L2 ?

cppi                                            L2 ?
drv_cppi                                    DDR3 ?
drv_pa                                       DDR3?
drv_qmss                                 DDR3 ?

qmss                                         DDR3 ?

platform_lib                             DDR3 ?

I'm not familiar with these sections.

.netctrl                                       L2 ?
nimu_eth_ll2                           L2 ?

stk                                              DDR3 ?
os                                               DDR3 ?
nettool                                       DDR3 ?

--> These are not default sections for NDK.  Does your map file show these as sections?

Steve

Hi Steve,

Thank you for your answer

All these sections where found in my .map file

These sections are from the NDK example albeit slightly renamed

rogram.sectMap[".text"]		= "L2";
Program.sectMap[".switch"]	= "L2";
Program.sectMap[".cio"] 	= "L2";
Program.sectMap[".vecs"] 	= "L2";
Program.sectMap[".far:taskStackSection"] = "L2";
Program.sectMap[".stack"]				= "L2";
Program.sectMap[".nimu_eth_ll2"] 			= "L2";
Program.sectMap[".cppi"] 				= "L2";
Program.sectMap[".neardata"] 				= "L2";
Program.sectMap[".rodata"] 				= "L2";
Program.sectMap[".bss"]					= "L2";
Program.sectMap[".ndk_netctrl_code{ netctrl_min_ipv4.ae66<*.oe66> (.text) }"]	=	"L2";
	
Program.sectMap[".resmgr_memregion"] 		= {loadSegment: "L2", loadAlign:128};
Program.sectMap[".resmgr_handles"] 		= {loadSegment: "L2", loadAlign:16};
Program.sectMap[".resmgr_pa"]			= {loadSegment: "L2", loadAlign:8};
Program.sectMap[".far:IMAGEDATA"] 		= {loadSegment: "L2", loadAlign: 8};
Program.sectMap[".far:NDK_OBJMEM"] 		= {loadSegment: "L2", loadAlign: 8};
	
Program.sectMap["ndk_system_heap"] 	= "DDR3";
Program.sectMap[".far:NDK_PACKETMEM"] 	= {loadSegment: "DDR3", loadAlign: 128}; 
Program.sectMap[".ndk_stk_code{ stk_ppp_pppoe.ae66<*.oe66> (.text) }"]		= "DDR3";
	
Program.sectMap[".ndk_nettool_code{ nettool_ipv4.ae66<*.oe66> (.text) }"]	=	"DDR3";
Program.sectMap[".ndk_drv_pa_code{ ti.drv.pa.ae66<*.oe66> (.text) }"]		=	"DDR3";
Program.sectMap[".ndk_drv_cppi_code{ ti.drv.cppi.ae66<*.oe66> (.text) }"]	=	"DDR3";
Program.sectMap[".ndk_drv_qmss_code{ ti.drv.qmss.ae66<*.oe66> (.text) }"]	=	"DDR3";
Program.sectMap[".ndk_os_code{ os.ae66<*.oe66> (.text) }"]		=	"DDR3";
	
Program.sectMap[".qmss"] 		= 	"DDR3";
Program.sectMap["platform_lib"] 	= 	"DDR3";
	
Program.sectMap[".const"]		=	"DDR3";
Program.sectMap[".far"] 		= 	"DDR3";

Program.sectMap[".fardata"] 		= 	"L2";

Program.sectMap[".cinit"] 		= 	"DDR3";
Program.sectMap[".init_array"] 		= 	"DDR3";

Clement

Hi Clement,

These sections are from the NDK example albeit slightly renamed

Ok, this was a nice hint.  I'm guessing your app is based on the NDK example(s) that shipped in the MCSDK product, then?

I went and looked at the client example in mcsdk_02_01_02_06.  I see that many of these sections are coming out of the *.cfg file:

I still don't see all of them, for example ".ndk_nettool_code".  But, it seems that these section placements are by design.

To come back to your original question, where should you place the NDK sections.  I would say it depends on how networking intensive your app is.  If your app's networking use is minimal - e.g. it will only send a few packets every 30 minutes or something like this, then I would say you should put all NDK sections into DDR, freeing up the faster internal memory for more intensive jobs of your app.

But if your app is highly network intensive, then I would say to place all NDK sections in the faster L2 memory.

Lastly, I saw in the MCSDK User's Guide that they recommend loading all NDK sections into L2:

Steve

Hi Steve,

Thank you for your time on this.

The purpose of this thread was to be more subtle with the NDK than the two options :

- all in DDR3

- all in L2

We want the maximum performance with as little use of L2 as possible because there's our code L2 is already filled at 75%.
that's the difference between our real world scenario vs NDK example where everything is available.

I think someone with a lot of experience with the NDK is needed on this thread.

Regards,
Clement

Hi Clement,

Were you able to resolve this issue?

Steve

Hi Steve,

Well sort of.

We did a bit of trial and error with sections, cache and NIMU parameters.

We managed to get the throughput we wanted mainly by putting NDK data and code in DDR3 cached.

L2 cache didn't have a significant impact on send throughput.

Clement

What's the throughput you're aiming for and what are you actually getting?

Steve

For UDP TX only the throughput we wanted (as a minimal) was 40 MBytes/s.

We acheived ~65 MB/s in a real-world scenario (not the ideal case)
and previously (years ago) in an ideal case one of our internship student got ~120 MB/s

Clement

Hi Clement,

I was out on vacation and now I'm coming back to your post.

Some digging around shows that other users were only able to get around 600-700 Mbits/s (~87MBytes/s).  (see here)... This is still less than 1Gbit but higher than what you're seeing.

Do you know if you are dropping frames?  Have you checked the EMAC registers for errors?  (I believe theres a bunch of stats for that EMAC that you can see in the CCS register view).

Steve

Hi Steve,

With Wireshark we saw that we weren't dropping frames.

I don't think we looked at EMAC errors/stats, it's a good idea.
I'll keep you posted.

Clement