Ethernet printf sample request

Hi Hector,

As per the following wiki link http://processors.wiki.ti.com/index.php/Multicore_System_Analyzer#The_Unified_Instrumentation_Architecture_.28UIA.29,the UIA can be reused across TI platform(single core/multi-core).

We will check and get back for the specific C6748 example .

Regards,

Shankari G

Shankari,

Thank you for the fast turn around on this question.  Do you have a time frame of when you would come up wiht the sample?  I just want to give a time frame to our customer. 

Regards,

Hector Rivera

Hi Hector,

With reference to above discussion, " Hello World" sample application shall be found in the NDK folder. Please find the example source in the below path:

~\ndk_2_20_06_35\packages\ti\ndk\winapps\helloWorld.c

NDK package version used: 2.20.06.35 (single core C6748 LCDK)

The above "hello world" sample just sends the message packet to DSP and try receiving the same packet to the specified destination port and also the data echoed back to the specified udp source.  It is based out of windows application and the printf''s used here are to debug when if it fails to open, close or bind a udp socket. I hope, this sample application shall be used for both single core and multi-core parts of ti dsp platform.

This is our understanding and I hope, this helps the customer in setting up Ethernet printf''s in his/her new project & to validate the same.

Please let me know, if more clarification is needed on the same.

Thanks & Regards,

Sivaraj K

Sivaraj,

Thank you for your post.  I passed this to the customer and he came back wiht the following response. 

      This seems to be something different from what I mean.  From what I can tell, this is a sockets app that echos a string ("Hello World") back to a host machine.

      I'm looking for TCP Transport for system_printf, instead of JTAG transport.  In the UIA, system_printf can be setup as runmode JTAG, stop mode JTAG and a few other variations.  If the NDK is installed, you can also set it up to do system_printf over ethernet.  The higher bandwidth of ethernet versus JTAG cables makes printf logs much more real time.  CCS still displays them the same way.  (i.e. it's transparent to the CCS user whether the system_printf is going over JTAG or over ethernet.) 

      My problem has been that when I try to set it up, the tutorials on ti's website are using the MCSDK and PDK and other software packages that are oriented toward parts like the 6472.  When I asked support about it last year, they said parts like the 6455 could definitely do this.  I was hoping the 6748 could as well.  I believe I was talking to Todd Mulanix at TI about this last year.  Yep.  Here's the e2e thread from when we were using a 6455.

http://e2e.ti.com/support/embedded/bios/f/355/t/187067.aspx?pi239031349=1

      I'm hoping that after a year, the product is more mature and easier to get up and running.  (We never got this working before.  We actually implemented our own, but I would prefer the real thing.)

Can you provide a response?

Regards,

Hector Rivera

I got another e-mail from the customer.  Here is his email. I need an answer soon. 

     I've made some progress.  I've gotten ethernet up and running on the 6748 LCDK board.  I can ping the board, and it appears that everything is good in terms of driver and NDK.  I'm using the NSP in conjunction with the NDK.  Everything seems to be etherneting well.

     I have been working on setting up a LoggerStreamer2, with Ethernet transport so I can get streaming run mode logging of CPU load, task execution and printfs.  Unfortunately, in Code Composer, I'm not seeing any printf statements in the System Analyzer.  I pulled up WireShark and watched that interface.  I can see pings when I send them down and I see the replies, but I never see any other spontaneous activity from the board, like my printfs and CPU logs should produce. 

 4403.app.cfg

     I've tried both System_printf and the LogUC type of printf.  i.e.  Log_iprintUC0(traceLog, "enter taskFxn()\n");

     I'm probably just missing a step somewhere.  Any thoughts?  I attached my config script.

Some info

CCSv5.4.0.00091

IPC 1.23.5.40

NDK 2.22.3.20

NSP 1.10.2.09

SYS/BIOS 6.35.1.29

SYSTEM ANALYZER 1.3.0.02

XDCTools 3.25.0.48

Compiler 7.4.2

jaredj(2024)@fourier:Debug $ vers Blizz_Phy.out

Blizz_Phy.out:

               __ASM__ = /opt/svn/trunk/components/software/Blizz_Phy_configuration/Default/configPkg/package/cfg/app_p674

               __ISA__ = 674

               __PLAT__ = ti.platforms.evm6748

               __TARG__ = ti.targets.C674

               __TRDR__ = ti.targets.omf.cof.Coff

Hi Hector,

    It looks like the customer's project has been archived using svn, which will cause the path information included in the .out file (the __ASM__ string above) to not match the actual location in the file system for the uia.xml and rta.xml metadata files that System Analyzer requries.

Could you ask the customer to create a UIA Config file that has endpoints configured to explicitly point to the required .out file and uia.xml and rta.xml files for their project ?  The steps required are covered in Section 4.5 of the System Analyzer User's guide (spruh43d.pdf) which ships in the docs folder of the UIA package.  For LoggerStreamer, there is no Control and Status port, only an Event Transport port using UDP at port 1235.

After creating the UIA Config file, they will need to launch Tools / System Analyzer / Live, click the [...] button to the right of the Instrumentation (UIA) Config: drop down list

and select the DefaultSession.usmxml  UIA Config file that was created in the previous step.   The cores and transport info should then be displayed in the table underneath this part of the dialog.  Clicking Start should then start System Analyzer. 

Regards,

  Brian

Brian,

Here is the answer from the customer

I did all of that!!  :)  I double checked the endpoint to the core and made sure the .out file and the uia xml files were actually direct paths to my project outputs.  They are, and everything looks correct.  But I still don't get any output in System Analyzer, and I don't see anything on Wireshark other than pings and ping replies that I have running from my host.

When I select "Tools->System Analyzer->Live", I don't get anything in any of the windows.  Just blank.  See two attached screenshots.  Sorry for the size.  I have 4 monitors, so you'll probably have to do a lot of scrolling to see the whole image.  The interesting parts are the lower left of the image where the printf logs should be showing up, and the upper right of the image which is the System Analyzer session window.  CCS is kind of in the middle, and you can see the code which should be generating a ton of printfs.  It's just a task that sleeps for a few milliseconds and tries to send some printfs.

One more screenshot.  Wireshark on the top left.  Ping going in a terminal lower left.  

More info provided by the customer.

One more piece of information.  In my exchange function, I see the buffers filling with data, but it's like it never gets sent over the wire.  Line 78 is my exchange function.1157.eth_logging_hooks.c

Regards,

Hector Rivera

Hi Hector Rivera,

Moved this post to BIOS forum so that the community can benefit with the informations shared here.

Regards,

Shankari.

Thanks for posting the screenshots and eth_logging_hooks.c.

In the customer's code, they've configured 3 instances of LoggerStreamer2, and provided 2 buffers for each instance.  This enables a 'ping-pong' buffer management approach where the LoggerStreamer2 instance will be writing into one buffer while the other is free to be sent out over UDP to  System Analyzer. 

As far as I can tell, it looks like the exchange function in eth_logging_hooks.c does not send out the UIA packets over Ethernet - it simply returns the next packet in the array of buffers to LoggerStreamer2.   What needs to happen is for the packet that was passed in to the exchange function to be handed off to a task that can send it out over a UDP connection to System Analyzer.   This needs to happen outside of the exchange function since, to ensure thread safety, the exchange function is called with interrupts disabled. 

A simple 'next step' to get things working would be to modify the exchange function to post a semaphore that a task is pending on, and to have that task be able to determine which buffer has just been filled and to send that buffer out over UDP.   I'm in the process of writing up a demo of this approach and will post it when I have it working.

Regards,

  Brian

Hi everyone.  I'm Hector's customer.  Thanks for looking into this.  Brian's comment is especially interesting at the moment.  I assumed that the sending of the log data happened "under the hood" somewhere once I had filled a buffer and returned from my Exchange function.  (FYI, I haven't been able to find any documentation on the Exchange and Prime functions.  I've kind of felt my way through, but if there's any documentation, I'd be happy to look through it.)

So I'll setup a semaphore and trigger another task that can do the sending of log data.  Is there an API that I should call?  Something like "Hey_C6748_Please_UDP_My_Log_Data_To_The_Host( Char * buff, Int length );" ??  

Hi ,

   As a first step, I've reworked your eth_logging_hooks.c code to open a UDP connection to the IP address of the PC running System Analyzer and send UDP packets over this.  For flow control, it uses a pretty crude mechanism based on global variables.   It drops a lot of events, but should allow you to start getting the events being received by System Analyzer over UDP. 

To use it, you'll need to change line 162 of eth_logging_hooks.c so that it contains the IP address of your PC.  You'll also need to add the following to your .cfg file:

var taskSendPkt = Task.create('&taskSendPkt');

This will create the task that is monitoring the global variables and sending the packet data when it finds that there is new data to send.

0675.eth_logging_hooks.c

I don't have a C6748 LCDK available at my desk so I've verified that it works on another EVM.  Please let me know if this works for you or not.

Thanks

Brian

Hi again,

   In addition to the eth_logging_hooks.c file in the previous post, please replace your app.cfg file with the attached reworked version: 0284.app.cfg

This comments out a number of lines from your app.cfg in order to avoid pulling in the ServiceMgr and Rta modules as these modules do not support LoggerStreamer or LoggerStreamer2.  The  xdc.useModule('xdc.runtime.LoggerBuf') has also been commented out as this is a legacy logger that is not supported by UIA.

Regards,

  Brian

FANTASTIC!  It worked the very first try.  Thank you so much Brian, Hector, Shankari, Sivaraj, et al.  

I have to say, this is the quickest resolution to an e2e post that I can remember.  Thank you for your help.  

I am a little curious about the LoggerStreamer(2) now.  Is this something that's generally not deployed by most customers?  It seems handy to me to have Ethernet-speed printf rather than JTAG-speed.  I ask because I had a lot of trouble finding in depth information about how to implement it.  Did I just look in the wrong places?

I'm also a bit surprised that it's such a manual process, rather than an under-the-hood CCS/BIOS setup checkbox that just reroutes printf over ethernet to CCS.  Don't get me wrong.  I'm glad it exists.  It just seems like it could be much more easily used (and more commonly used) if some of the underpinnings were automagic with the IDE.

All of that said, I'm still really glad this works.  Thanks everyone!  Dinner's on me next time I'm in TX.  ;)

One last question here.  I got it working and was watching printfs come in really quickly.  I changed my Task_sleep() time and rebuilt and tried to reconnect.  Now I get an error in the Error Log every time I connect.

Error
Tue Jun 04 10:29:25 EDT 2013
Error during DVT action: Unable to open Event Transport 10.5.1.1:1235 : Address already in use: Unable to open Event Transport 10.5.1.1:1235 : Address already in use

If I do a reboot of my host machine, I can get it to run once.  Then subsequent runs throw the same error.  Is there something I'm doing wrong when I stop debugging?  I was hitting the red "stop sign" in System Analyzer and then hitting the red "stop" square in CCS to shut down.  But it seems like something is getting left in a bad state.

Just FYI, I have two NICs on my machine.  One is on our 192.168 network.  The other is my own little 10.5. network where I have a switch and all of my dev boards.  I've tried connecting to localhost, 10.5.1.1 (host machine), 127.0.0.0 and 127.0.0.1 explicitly, and none of those helped.  I'm running out of ideas.

Also, I tried this both auto-detecting analysis activities and using a UIA Config file.  Both performed the same.  Rebooting between runs is sub-optimal.  :)  Is this fixable?

So that seems to work, and it allows me to re-run System Analyzer.  What I've found is that if I halt my 6748 for any reason, I can't just click "run" again.  I have to restart and I have to close down System Analyzer and relaunch it.  The green "play" button in System Analyzer doesn't respond to click.  So I kill System Analyzer, restart it and run again.

However, I'm having some issues with corrupt data.  I get errors like...

eclipse.buildId=M20120914-1540
java.version=1.6.0_13
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=linux, ARCH=x86, WS=gtk, NL=en_US
Framework arguments: -product com.ti.ccstudio.branding.product
Command-line arguments: -os linux -ws gtk -arch x86 -product com.ti.ccstudio.branding.product

Error
Fri Jun 07 10:31:24 EDT 2013
Corrupted data.

java.lang.RuntimeException: Corrupted data.
 at com.ti.uia.host.decoder.BaseLoggerDecoder.decode(Unknown Source)
 at com.ti.uia.host.extensions.loggers.LoggerBuff2Decoder.decode(Unknown Source)
 at com.ti.dvt.uia.core.UIAEngine$EPoint$DecoderStream.decode(UIAEngine.java:796)
 at com.ti.dvt.uia.core.UIAEngine$EPoint.decode(UIAEngine.java:549)
 at com.ti.dvt.uia.core.UIAEngine.decode(UIAEngine.java:942)
 at com.ti.dvt.uia.dp.UIALogSource$UIAThread.run(UIALogSource.java:887)

Is there any way to diagnose what went wrong there?

I just received a C6748 LCDK board to work with and I'll see if I can get a better understanding of why the green 'play' button is not working.

The data corruption typically happens when the buffer that is being transmitted over Ethernet to System Analyzer is handed back to LoggerStreamer2 in the exchange function before it has finished being transmitted.  Then LoggerStreamer2 will start writing in new events, and at some point will write into locations in the buffer that have not yet been transmitted so that the packet received by System Analyzer contains some old events partially overwritten with some new events.

Re: Is there any way to diagnose what went wrong there?

I was a little surprised that the ping pong buffer isn't keeping up.  I could add more buffers.  I just wonder... if two buffers doesn't make real-time, would 4?  My engineering spider sense says "no".  

I was trying to speed this up so it wasn't busy-waiting so much.  I set up a mailbox that my send task would pend on.  But that led to the problem of data corruption as the ping and pong buffers would overflow before it finished sending.

I had HWI, SWI, TASK and Load logging all enabled, as well as printf logs.  The chip isn't doing anything else at the moment as this is a brand new project.  So it was essentially idle, but the CPU Load was registering about 13% just processing all of the logging.

Hi FastFourier,

   The code I posted previously was really simplistic - it was designed to log events to a 'throw away' buffer while waiting for the last packet to be sent out over UDP.   Adding more buffers would definitely help - even adding one more buffer would allow it to work as a real ping-pong buffer arrangement where you would only throw away events by writing into buffer2 if buffer0 and buffer1 were still waiting to be sent out.  If you weren't memory constrained, you could have an array of buffers to start with, and keep track of the 'high water mark' to figure out how many buffers you really needed.

Some of the overhead your seeing is likely due to the NDK.  I'm not an NDK expert, so I've been reading up on it to try to get a better understanding of how to use it more efficiently.   One way to minimize the overhead is to use bare-metal Ethernet programming and have UIA write into pre-formatted UDP packets, with the destination Mac address etc. prepended as required for Ethernet.  This is demonstrated in the Trace Framework example that is mentioned in Tutorial 5C.   I'll see if I can port this to work on the C6748 LCDK, and will post here once I have a better feeling for whether this is doable or not.

Regards,

  Brian

Yeah.  I was making mods, trying to spend a little less time busy waiting, and it looks like that's how I got into this data corruption problem.  I added a mailbox, and I send the context and the buffer via it to the sender task, which then tries to send out the buffer.  I haven't added any type of "did it get sent already?" checking, so if I overrun my buffers, we corrupt the data.

From my app.cfg

/* Mailbox for unblocking UDP task. */
var mailbox0Params = new Mailbox.Params();
mailbox0Params.instance.name = "eth_mbx";
Program.global.eth_mbx = Mailbox.create(8, 64, mailbox0Params);