Steps to add the NDK to an existing SYSBIOS project running on custom hardware

Bump.

Hi Dean,

Dean Hofstetter said:

Step #2:  Here is where the confusion starts.  The NDK Getting Started Guide is way out of date.

My apologies for your confusion on this.  Indeed that doc is outdated (as far as following the steps goes).  However, it still has a lot of good information about how the stack works, and this was the main reason for keeping a reference to it.  This was mentioned in the release notes but obviously needs some more clarification.  I'll update the description to state that the steps themselves are outdated and will be difficult to follow, but the overiew content is still pertinent.  I filed a bug to track this:

SDOCM00114494 update release notes to warn users that Getting Started Guide steps are out of date

Dean Hofstetter said:

Step #7:  Before I even attempt to configure the NDK for use, what exactly have I built so far?  I grabbed a file (emacHooks.c) from a NSP directory that was for the EVM6748.  I'm not running on the EVM6748, I have my own custom hardware.  How does the code know what PHY I have?  How does it know what GPIO pin is controlling the RESET on the PHY?  How does it know if the PHY is connected in MII vs RMII?  This is where I am stuck.  Somewhere I have to configure or modify something specific to my hardware.  What is that?

By using the Emac module, the app will link against the Ethernet driver library for the EVM6748.  The sources for that can be found in the 6748 NSP under packages/ti/drv.  That driver is coded for the PHY on the EVM6748 and this is the hardware specific driver code.  You need to update these files for your hardware.

--------------

Lastly, I've updated our existing "documentation bug report" (which contains all documentation issues that arise) with the following.  This bug report serves as a fix list that will be used in the next documentation iteration (SDOCM00113941 NDK 2.24 documentation items):

• User's Guide, section "1.6.1 Adding NDK Support to an Existing Application"
  • step 7 needs to tell user to also "right click, use module" for Global module, or to drag/drop the Global module into the outline pane.
  • also need to tell user to "right click, use module" for the Emac module
  • After this must also instruct user how to properly configure the Ethernet driver for the application
  • Tell user to add ti/ndk/inc to their application's compiler include path
• User's Guide, section "1.6.2 Troubleshooting ..."
  • ensure user has NSP and refer back to step 1.5.5 for info on NSPs for devices that TIRTOS does not ship Ethernet drivers for.
  • Instruct user to get emacHooks.c from the hello world example
• User's Guide, any part of the doc that references "right click, choose Build Options" --> Build Options is now "Show Build Settings"

Steve

Steve,

I my opinion the useful information that resides in the NDK Getting Started document should be merged in the NDK Users Guide. There are a few nuggets of good info in there, but if I already know "Networking" and I just want to get down to system integration, this document isn't very helpful. Ideally you want to be as clear as possible. Having users skim through the document going: "skip outdated, know that already, good info, skip outdated, skip outdated, good info" is really tedious and not very helpful. TI is a big company, I'm sure there is some intern looking for something to do that can make this right :)

Thanks for logging the steps that are wrong in the NDK Users Guide. Again, clarity should trump everything. When you read instructions in a guide, and then those steps don't match the real world, that gets frustrating fast.

Finally we've arrived at the meat of this discussion. You have pointed me to the NSP download, and told me to modify the files in \packages\ti\drv\omapl138. There are four .c files in this directory: csl_emac.c csl_mdio.c ethdriver.c nimu_eth.c. Assuming I knew which files / routines to update (I don't), if I just modified the files I don't think that would do much for me. I appears I need to follow the "Rebuilding the Driver Library" found in the NSP Release Notes. However that appears to have downsides, as new NSP versions are made available, I would need to make the same custom changes to every new release and rebuild. I'm thinking about bringing these four files into my project and UNCHECKING the NSP from the RTSC tab in my project. That way I make my custom modification once, and then as new NSP versions are released, compare the files, and make any adjustments. Yes, no, maybe? Finally you have told me to make no changes to the emacHooks.c file I have already imported into my project. That one seems pretty important, as it looks like it is setting up the pinmux for MII or RMII, allows the NDK to find our MAC address, and provides a callback for when the physical link status changes (cable plugged / unplugged). I volley the ball back to your side of the court.

- Dean

Dean,

Yes, it should be fine to place the driver sources into your project if that is easier for you.

There is a driver guide that ships with the NDK that should be helpful to you for this, please have a look if you haven't seen this yet:

ndk_2_24_01_18\docs\sprufp2_nspethdrvdesign.pdf

Have a great weekend.

Steve

NIMU layer. According to the NSP driver design guide, this layer is generic and doesn't change between platforms. There is only one file for this layer: Nimu_eth.c. After looking through the routines in this file, I would agree to leave this file alone.

Ethernet Mini-Driver layer. According to the NSP driver design guide, this layer needs to be customized when changing platforms. There is only one file for this layer: Ethdriver.c. Near the top of this file are some configuration options. You can set the HW interrupt vectors (via RXINT and TXINT), so that make sense. CORENUM is not documented anywhere, so I'm not sure if I need to do anything with that. EXTERNAL_MEMORY and EXTMEM are defined in the NSP driver design guide, and imply that the packet buffer memory is located in external memory. Since EXTMEM is set to 0x80000000, that equals the L3(or shared RAM) inside the C6748 DSP. However, when I compile my project and look at the MAP file, it is showing zero bytes used for the L3 RAM. Something is not making sense. PKT_MAX is set to 64, but after reading description for this in the NSP driver design guide, I'm still unsure if I should change it. My system has plenty of memory, so my first thought is to increase it. What are the advantages / disadvantages to increasing PKT_MAX? Finally, there are other configuration variables described in the NSP driver design guide, and they are defined in Nimu_eth.h. They are: PKT_PREPAD, RAM_MCAST, HASH_MCAST, and PKT_MAX_MCAST. I'm leaning towards not changing any of these.

CSL layer. According to the NSP driver design guide, this layer is generic and can be ported across platforms as long as the EMAC HW doesn't vary a lot. Well that is kind of an open ended statement, but since I know this NSP code was written for the OMAPL138, and I'm using the C6748, I know the EMAC HW is exactly the same. This would imply I shouldn't have to make any changes. There are two files for this layer: csl_emac.c and csl_mdio.c. After looking through csl_emac.c, and can't see anything that would need changing. However, I have no idea how EMAC_timerTick( ) gets called every 100mS. Nothing in the NSP code seems to call this routine. Is it needed, not needed??? csl_mdio.c might need some modification. Near the top of the file is the variable macsel. It is currently set to MII (which is fine for my hardware), but what if my hardware was setup to use RMII. Would I change this to RMII? VBUSCLK is also defined at the top of this file, and it is set to 75. According to the comment, it is suppose to represent SYSCLK4. On my hardware SYSCLK4 is running at 91MHz, so would I change VBUSCLK to 91?

Thanks, Dean

Hi Dean,

Yes, the driver code isn't the easiest to understand, I know what you're going through!  Hopefully this will help clarify things.

Dean Hofstetter said:

NIMU layer. According to the NSP driver design guide, this layer is generic and doesn't change between platforms. There is only one file for this layer: Nimu_eth.c. After looking through the routines in this file, I would agree to leave this file alone.

You shouldn't need to change this file.  The NIMU layer is the "glue" between the NDK core stack and the driver layer that you're digging through now.  Nimu_eth.c is tied to ti/ndk/stack/nimu/nimu.c in the core stack.

Dean Hofstetter said:

CORENUM is not documented anywhere, so I'm not sure if I need to do anything with that.

For the OMAPL138, the ARM9 is typically core 0, and the DSP core 1. You shouldn't need to do anything here.  The preprocessor check for "TMS470" is for ARM:

#if defined(__TMS470__)
#define CORENUM 0
...
#else
#define CORENUM 1

...

Since you're on the DSP, you should fall into the #else clause with CORENUM == 1.

Dean Hofstetter said:

EXTERNAL_MEMORY and EXTMEM are defined in the NSP driver design guide, and imply that the packet buffer memory is located in external memory. Since EXTMEM is set to 0x80000000, that equals the L3(or shared RAM) inside the C6748 DSP. However, when I compile my project and look at the MAP file, it is showing zero bytes used for the L3 RAM. Something is not making sense.

These variables are only used to to perform proper cache operations IF the PBM buffers are stored in L3 cache.  The location of the PBM buffers is configured by the application via the *.cfg file.  Since the default for the C6000 is the .far section, I'm guessing that is probably out in DDR in your case (but I couldn't say for sure as you're using custom h/w).  This would also make sense as to why you're not seeing anything in L3 RAM.

Dean Hofstetter said:

PKT_MAX is set to 64, but after reading description for this in the NSP driver design guide, I'm still unsure if I should change it. My system has plenty of memory, so my first thought is to increase it. What are the advantages / disadvantages to increasing PKT_MAX?

PKT_MAX is the number of PBM buffers to allocate for both TX and RX.  Each PBM buffer is used to store an Ethernet frame.

Since you have a lot of memory, you could probably safely increase this if you like.  To start, I would leave it as is, though.  If you find that your app is dropping a lot of Ethernet frames due to the RX and TX queues filling up, then you should increase it.

Have a look at this wiki page for further details on NDK and its memory use.  Note the specific sub section on PBM buffers.

Dean Hofstetter said:

there are other configuration variables described in the NSP driver design guide, and they are defined in Nimu_eth.h. They are: PKT_PREPAD, RAM_MCAST, HASH_MCAST, and PKT_MAX_MCAST. I'm leaning towards not changing any of these.

I think you should leave them as is.  PKT_PREPAD looks to be some extra padding for Ethernet frames for PPPoE.  Without it the size of a frame is the standard 14 byte header plus an additional 4 bytes for the CRC at the end of the frame.  If you're not using PPPoE, you probably could define this to be 0 and save 8 bytes per frame.  But, since you have a lot of memory, I doubt if it would hurt much to leave it in there.

I'm not entirely sure about RAM_MCAST vs. HASH_MCAST, but my guess is it's dependent on the multicast features of the h/w.

Dean Hofstetter said:

I have no idea how EMAC_timerTick( ) gets called every 100mS. Nothing in the NSP code seems to call this routine. Is it needed, not needed???

This function is used for polling mode.  The driver is operating in interrupt mode by default, so you shouldn't need this unless you want to switch to polling mode (which I wouldn't recommend).

As for RMII mode and VBUSCLK, I don't want to make anything up here, but it would seem that you may need to do some different register config for RMII mode, and that VBUSCLK should be consistent with your SYSCLK4 setting.  I can ask some people who may be more familiar with the h/w that I am for insight on that.

Steve

Steve,

Thanks for all your time on this subject. I know this very detailed, but as you said the NSP is complicated and not very straight forward with what needs to be modified, or what are the ramification to some changes.

Nimu_eth.c
We both agree no changes are necessary to this file. However, I believe the documentation is again out of date. Section 1.5 in the Design Driver Guide maps how the calls flow down through the layers. This appears to be outdated. Below I'll list what is in the Design Driver Guide, and then what I see in the NSP code. Even though you say EMAC_timerTick( ) only gets called in polling mode, based on what I see below, I don't see how EMAC_timerTick( ) ever gets called. Not an issue right now, as I'll stay with interrupt mode, but something isn't right.
Guide: EmacInit -> HwPktInit
Code: EmacInit -> HwPktInit -> EMAC_getConfig

Guide: EmacStart-> HwPktOpen -> EMAC_open and MDIO_open
Code: EmacStart -> HwPktOpen -> EMAC_open and MDIO_open and EMAC_initialize

Guide: Emacioctrl -> HwPktSetRx -> EMAC_setReceiverFilter plus five other routines
Code: Emacioctrl -> HwPktSetRx -> EMAC_setReceiverFilter

Guide: EmacSend -> HwPktTxNext -> EMAC_sendPacket and EMAC_TxServiceCheck
Code: EmacSend -> HwPktTxNext -> EMAC_sendPacket

Guide: EmacPktService -> HwRxInt -> EMAC_RxServiceCheck
Code: EmacPktService -> PBMQ_deq -> NIMUReceivePacket

Guide: EmacPoll -> _HwPktPoll -> EMAC_TimerTick and MDIO_timerTick and MDIO_getStatus
Code: EmacPoll -> _HwPktPoll -> MDIO_timerTick


Ethdriver.c
CORENUM. Still don't understand what this really does and why it needs to be 1 for the C6748. I looked briefly at the C6748 Tech Manual, and the EMAC section does define these "cores". There are three of them, and appear to be related to how the EMAC controller interrupts the DSP. Too far down the EMAC rabbit hole for me. I'll just take your word for it that it should be 1.

EXTERNAL_MEMORY and EXTMEM. These two bother me. They are defined at the top of Ethdriver.c, thus the code in these routines are "trying" to do something. I can see on the BUFFERS tab from the *.cfg file where the PBM Buffers are defined. By default frames is 192, and size is 1536. Since this is buffer pool is larger than the L3 internal memory, it can't go there. The data section defined is indeed being placed in DDR2. Furthermore, the way the code is written, it appears that OEMCacheClean( ) will be called if the data section is placed in L3(0x80000000) or DDR2(0xC0000000). I can only speculate that these two variables are only for when the PBM Buffer will fit within L1D or L2, and of course you must force the PBM Buffer to reside in either of these memory sections. Bottom line, these appear to be very important and shouldn't be touched.

PKT_MAX. Our application is going in a mission critical location. Therefore dropped RX or TX Ethernet frames need to be avoided. The links you provided don't really help me understand what are the downsides to increasing this number (aside from more memory used). I want to increase this number, but there must be some relation between this number and the frames + size you define in the PBM Bufffers *.cfg. With frames currently defined to be 192, and PKT_MAX defined to be 64, I'm not seeing the relationship.

PKT_PREPAD. I understand your explanation. I agree I should just leave it alone, as memory space isn't an issue.

RAM_MCST, HASH_MCST, and PKT_MAX_MCST. According to the Driver Design Guide, these appear to very hardware specific. Since this NSP was based on the OMAPL138, and I'm using the C6748, I'll just leave them alone and hope for the best.


csl_emac.c
I would still like some more explanation on EMAC_timerTick( ). Based on what I see from the NSP code, I can't see any way this gets called, even if the NDK is configured for polling mode. The good news, is it appears no code changes are necessary.

csl_mdio.c
Please continue to ask around about the variable macsel, and the VBUSCLK definition. The code that uses macsel appears to only care about GMII and RGMII, which seems odd. The code clearly loads a MDIO register with the VBUSCLK number, so it must be somewhat important. According to the C6748 Tech Manual, it looks like it may control the MDIO CLK frequency. Since the MDIO clock is only rated to run up to 2.5MHz, this divider could make the difference between MDIO working or not.


emacHooks.c
We haven't talked about this file yet, but I believe modifications are necessary. Remember this file was pulled (somewhat randomly) from one of the NSP example projects. EMAC_initialize( ) needs to be modified. I really don't understand what GPIO_CONTROL_REG and TIMER_GLOBAL_REG are doing. The pinmux stuff can go, as that is defined in our project in the GEL file or with the boot ROM (via the AISgen tool). Basically we define all our pinmux stuff up front, and don't change it during run time. Load CFGCHIP[3] is how you tell the EMAC you want MII or RMII (plus the proper pinmux settings). Hey, we found an important change location. This routine would also be a good location to make sure we fired a HW RESET pulse to the Ethernet PHY chip. We control the RESET via a GPIO pin, so we can setup the pulse here. EMAC_getConfig( ) needs to be modified. Basically this routine is poorly named, as all it really does is fetch the MAC address from our application. Where ever we store the MAC address, we should copy it into pMacAddr. Finally EMAC_linkStatus( ) needs to be modified. This tells our application when the Ethernet cable gets plugged / unplugged. Our application can react accordingly to those events.

Thanks, Dean

Sounds like you're making progress, that's great!

Dean Hofstetter said:

PKT_MAX. Our application is going in a mission critical location. Therefore dropped RX or TX Ethernet frames need to be avoided. The links you provided don't really help me understand what are the downsides to increasing this number (aside from more memory used). I want to increase this number, ...

I can't think of any downside to increasing PKT_MAX off the top of my head, unless you change it to be so large that it takes up ALL of your memory.

Dean Hofstetter said:

but there must be some relation between this number and the frames + size you define in the PBM Bufffers *.cfg. With frames currently defined to be 192, and PKT_MAX defined to be 64, I'm not seeing the relationship.

Yes, the frames + size defined for the PBM buffers in the *.cfg file is for the pool.  That is, the memory pool that is drawn upon when PBM_alloc() is called.  Since PKT_MAX is used to set the size of the RX and TX queues, it calls PBM_alloc for a total of PKT_MAX * 2 times.  So, PKT_MAX * 2 must be less than the total number of PBM buffers configured (and would probably be a good idea to leave some room to spare, too).

Dean Hofstetter said:

csl_emac.c
I would still like some more explanation on EMAC_timerTick( ). Based on what I see from the NSP code, I can't see any way this gets called, even if the NDK is configured for polling mode. The good news, is it appears no code changes are necessary.

I'll have to dig into that deeper, but it's possible the driver author didn't follow the design guide verbatim.   Another possibility is that the code was ported from a driver that DID use EMAC_timerTick for polling mode, but the code that called it was removed, and the function left behind.  But these are just my guesses.

Dean Hofstetter said:

csl_mdio.c
Please continue to ask around about the variable macsel, and the VBUSCLK definition. The code that uses macsel appears to only care about GMII and RGMII, which seems odd. The code clearly loads a MDIO register with the VBUSCLK number, so it must be somewhat important. According to the C6748 Tech Manual, it looks like it may control the MDIO CLK frequency. Since the MDIO clock is only rated to run up to 2.5MHz, this divider could make the difference between MDIO working or not.

Ok.  I'll let you know as son as I hear from the driver guys.

Steve

Steve,

I think I'm close to identifying all the things the NEED to be changed when running on custom HW, and the things that COULD be changed. Here is what I got so far.

Ethernet PHY connected to the C6748 EMAC via MII or RMII? The NSP code assumes MII. If you require RMII, then you need to change EMAC_initialize( ) found in emacHooks.c. You'll want to change the pinmuxConfig( ) calls to define the RMII pins and not the MII pins. You'll also need to change CFGCHIP[3] to select RMII mode. Finally you should change the variable "macsel" found in csl_mdio.c to RMII.

Timer code in EMAC_initialize( )? For some reason EMAC_initialize( ) attempts to take control of TIMER0. In my system, TIMER0 is used by SYSBIOS, so having the EMAC come along and start messing with it seems like a bad idea. I need more information from Steve, but for now it appears I need to delete these lines of code. According to the NDK Users Guide, the Timer object required by the NDK is automatically created if you use XGCONF. I can't find any evidence of XGCONF doing this for me, and as a side note I really hate the way XGCONF hides things from you.

What MDIO clock speed do you want? In the file csl_mdio.c there is a #define called VBUSCLK. It appears the NSP code assumed the DSP core would be running at 300MHz, and thus SYSCLK4 would be 75MHz. It has VBUSCLK set to 75, which I think makes the MDIO clock run at 1MHz. The C6748 DSP can run this clock up to 2.5MHz, and the PHY I'm using can go all the way up to 25MHz (2.5MHz is listed as typical). I'm going to change VBUSCLK so the MDIO clock is running closer to the 2.5MHz maximum. For example, if my DSP clock is running at 368.6MHz, that puts my SYSCLK4 at 92.2MHz. So if I set VBUSCLK to 40, that will make my MDIO clock be 2.3MHz. Close enough.

Change the Configuration variables to personalize your system. These are mostly defined in section 3.4 of the NDK Driver Design Guide. CORENUM, RXINT, TXINT, EXTERNAL_MEMORY, EXTMEM, and PKT_MAX are found in ethdriver.c PKT_PREPAD, RAM_MCST, HASH_MCST, and PKT_MAX_MCST are found in nimu_eth.h. As far as I can tell the following should never be changed: CORENUM, PKT_PREPAD, RAM_MCST, HAST_MCST, and PKT_MAX_MCST. EXTERNAL_MEMORY and EXTMEM would only change if you were trying to squeeze the Packet Buffer memory into L2 or L1D. PKT_MAX can be used to "tune" how much memory the Packet Buffer uses. If you have the memory, there seems little downside to increasing this number. Be aware if you do increase this number, you'll also have to increase the Number of Frames field in the XGCONF (NDK -> Buffers -> Number of Frames). Finally if you don't like where the NSP has defined the HW interrupt vectors, you can change RXINT and TXINT.

Do you have a GPIO line controlling the RESET pin on the Ethernet PHY? If so, you'll want to make sure GPIO line has the part out of RESET. Better yet, issue a RESET pulse to put the part in a known state. This code should be put at the top of EMAC_initialize( ) found in emacHooks.c.

Hand the NDK your Ethernet MAC address. This is done via EMAC_getConfig( ). Change the code in this routine to load up your MAC address into *pMacAddr. You must use the binary MAC address, not a string. So for example, your MAC address buffer would look like myMacBuf[6] = 0x00, 0x1B, 0xC4, 0x01, 0x02, 0x03.

What do you want to do when the Ethernet cable is plugged in or unplugged? The NDK notifies you when these two events happen via EMAC_linkStatus( ) found in emacHooks.c. Up to your system how you want to handle these events.


Finally we are down to the changes I'm not sure about.

StatusUpdate( ) and StatisticsUpdate( ), found in ethdriver.c, do nothing. Seems like StatusUpdate( ) is very important, as that tracks fatal EMAC errors. When a fatal EMAC error is found, what should be do? What is the recommendation from TI? Shut down the socket? Reboot the NDK? Reboot the system? Get a beer?

What about the other EMAC mode flags? HwPktOpen( ), which is found in ethdriver.c, defines some EMAC mode flags: EMAC_CONFIG_MODEFLG_RXOFFLENBLOCK and EMAC_CONFIG_MODEFLG_RMII. Why are those two defined and not any of the others (there are 21 total)? What do these do? Why is RMII defined, when the rest of the NSP defaults to MII? Is there any benefit to using some of the unused flags?

Thanks, Dean

Dean,

I just wanted to update you that we have contacted the team who can better help with the driver details, and they should be responding soon.

Steve

Senthil,

Thanks for the confirmation. I now have a clear understanding of how to use VBUSCLK.


Steve,

There are still a few items I'm waiting answers on. They are listed below. Once these are answered, I'm ready to write my doctorate on how to bolt the NDK to custom hardware.

1. EXTERNAL_MEMORY and EXTMEM. You said these variables only come into play if the PBM buffers are located in L3 cache. I disagree, but would like to know if I'm right or wrong. For the C6748 DSP, L3 memory starts at address 0x80000000 and DDR2 starts at address 0xC0000000. The NSP code has EXTERNAL_MEMORY defined to 1, and EXTMEM defined to 0x80000000. EXTERNAL_MEMORY controls a #ifdef statement, so the code is complied. The NSP code that uses EXTMEM logically ands the PBM buffer address with EXTMEM. Since 0x80000000 anded with any address that starts with 0x80000000 or 0xC0000000 will return a non-zero result, the OEMCacheClean( ) function will be called. I'm just trying to understand how to use these two settings, and I think the bottom line is leave them alone. They would only change if you somehow compiled a really small PBM buffer memory allocation that fit in L2 or L1D. Very few people probably do this, so again these are two settings that 90% of system integrators just don't touch.

2. EMAC_timerTick( ). I can't see anyway this function every gets called by the NSP code, or the NDK directly. The NDK is only suppose to interface to the NSP via the NIMU module. During the EmacInit( ) function, the NSP loads up functions for the NDK to call. One of those is the poll function, and the NSP sets that to EmacPoll( ). If you look at the code flow inside EmacPoll( ), it calls _HwPktPoll( ), which in turn calls MDIO_timerTick( ). Nothing calls EMAC_timerTick( ). It is just confusing to have this routine mentioned in the NDK Driver Design Guide, and show up in the NSP code, but appear to be completely unused. Is this a dead function?

3. Code flow from section 1.5 of the NDK Design Driver Guide appears to be wrong or outdated. Please comment.
Guide: EmacInit -> HwPktInit
Code: EmacInit -> HwPktInit -> EMAC_getConfig

Guide: EmacStart-> HwPktOpen -> EMAC_open and MDIO_open
Code: EmacStart -> HwPktOpen -> EMAC_open and MDIO_open and EMAC_initialize

Guide: Emacioctrl -> HwPktSetRx -> EMAC_setReceiverFilter plus five other routines
Code: Emacioctrl -> HwPktSetRx -> EMAC_setReceiverFilter

Guide: EmacSend -> HwPktTxNext -> EMAC_sendPacket and EMAC_TxServiceCheck
Code: EmacSend -> HwPktTxNext -> EMAC_sendPacket

Guide: EmacPktService -> HwRxInt -> EMAC_RxServiceCheck
Code: EmacPktService -> PBMQ_deq -> NIMUReceivePacket

Guide: EmacPoll -> _HwPktPoll -> EMAC_TimerTick and MDIO_timerTick and MDIO_getStatus
Code: EmacPoll -> _HwPktPoll -> MDIO_timerTick

4. GPIO_CONTROL_REG and TIMER_GLOBAL_REG found in the routine EMAC_initialize( ) appear to be not needed and dangerous if left in. For some reason EMAC_initialize( ) attempts to take control of TIMER0 via GPIO_CONTROL_REG and TIMER_GLOBAL_REG. In my system, TIMER0 is used by SYSBIOS, so having the EMAC come along and start messing with it seems like a bad idea. According to the NDK Users Guide, the Timer object required by the NDK is automatically created if you use XGCONF. I can't find any evidence of XGCONF doing this for me, and as a side note I really hate the way XGCONF hides things from you.

5. StatusUpdate( ) and StatisticsUpdate( ), found in ethdriver.c, do nothing. Seems like StatusUpdate( ) is very important, as that tracks fatal EMAC errors. What is a fatal EMAC error? When a fatal EMAC error is found, what should be do? What is the recommendation from TI? Shut down the socket? Reboot the NDK? Reboot the system? Get a beer?

Thanks, Dean

Hi Dean,

Dean Hofstetter said:

1. EXTERNAL_MEMORY and EXTMEM. You said these variables only come into play if the PBM buffers are located in L3 cache. I disagree, but would like to know if I'm right or wrong. For the C6748 DSP, L3 memory starts at address 0x80000000 and DDR2 starts at address 0xC0000000. The NSP code has EXTERNAL_MEMORY defined to 1, and EXTMEM defined to 0x80000000. EXTERNAL_MEMORY controls a #ifdef statement, so the code is complied. The NSP code that uses EXTMEM logically ands the PBM buffer address with EXTMEM. Since 0x80000000 anded with any address that starts with 0x80000000 or 0xC0000000 will return a non-zero result, the OEMCacheClean( ) function will be called. I'm just trying to understand how to use these two settings, and I think the bottom line is leave them alone. They would only change if you somehow compiled a really small PBM buffer memory allocation that fit in L2 or L1D. Very few people probably do this, so again these are two settings that 90% of system integrators just don't touch.

You are correct, what I told you before (that the check was to see if the buffer is in L3) was incorrect - I spoke too soon on my understanding of that.

I talked to our local cache expert.  He told me that it looks like that if statement should be read as "if the buffer is stored in external memory, then perform a cache writeback on it."

Since external memory begins at address 0x80000000 and goes up from there, this if statement would be true for any PBM buffer stored in external memory.  Another thing he pointed out was that it's not doing any check of the MAR bits, to determine if the memory in question is cacheable or not.  So it is blindly calling the OEMCacheClean function.  However, in the event that the MAR bits are not set for the address range of the buffer, it won't actually do any cache operation.  It would result in extra overhead of that function call, though.

Let me get back to you on your other questions you have, above.

Steve

Hi Dean,

I've dug a bit more ... let's see if we can't get that doctorate complete! :)

2. EMAC_timerTick( ). I can't see anyway this function every gets called by the NSP code, or the NDK directly. The NDK is only suppose to interface to the NSP via the NIMU module. During the EmacInit( ) function, the NSP loads up functions for the NDK to call. One of those is the poll function, and the NSP sets that to EmacPoll( ). If you look at the code flow inside EmacPoll( ), it calls _HwPktPoll( ), which in turn calls MDIO_timerTick( ). Nothing calls EMAC_timerTick( ). It is just confusing to have this routine mentioned in the NDK Driver Design Guide, and show up in the NSP code, but appear to be completely unused. Is this a dead function?

As far as I can tell this is indeed a dead function.  I've grep'd around the various NDK drivers from other releases (NDK 1.94, NDK 2.0.0, 2.1.0) and could not find EMAC_timerTick being called or referenced anywhere.  I did, however, find that many of the drivers had the function implementation, just as the OMAPL138 driver does.

I've filed a bug for this issue:

SDOCM00114665 (NSP) EMAC_timerTick function is not referenced anywhere

I've also updated my NDK documentation bug (SDOCM00113941) to make note of this.

3. Code flow from section 1.5 of the NDK Design Driver Guide appears to be wrong or outdated. Please comment.

First, I don't think this table is a mapping of function calls, but rather a a categorization of APIs in the layers defined in the table.  Having said that, here's some comments about all of these ...

Guide: EmacInit -> HwPktInit
Code: EmacInit -> HwPktInit -> EMAC_getConfig

Guide: EmacStart-> HwPktOpen -> EMAC_open and MDIO_open
Code: EmacStart -> HwPktOpen -> EMAC_open and MDIO_open and EMAC_initialize

These appear to be extra functions that are defined in some drivers:

• EMAC_getConfig
• EMAC_initialize

But are not mentioned in the document.  I've updated the doc bug to track this.

Guide: Emacioctrl -> HwPktSetRx -> EMAC_setReceiverFilter plus five other routines
Code: Emacioctrl -> HwPktSetRx -> EMAC_setReceiverFilter

The OMAPL138 driver did not implement the full set of IOCTL functions.  Looks like multicast is being handled directly in the code, but I don't see any call to "EMAC_setMulticast", even though it's implemented.

I also don't see the cases/calls for:

1. EMAC_getReceiveFilter
2. EMAC_getStatus
3. EMAC_getStatistics
4. EMAC_enumerate

Guide: EmacSend -> HwPktTxNext -> EMAC_sendPacket and EMAC_TxServiceCheck
Code: EmacSend -> HwPktTxNext -> EMAC_sendPacket

This looks OK to me.  I don't think that this entry implies that EMAC_sendPacket and EMAC_TxServiceCheck should be called by HwPktSetRx.  Note that EMAC_TxServiceCheck() runs at ISR level.

EMAC_sendPacket enqueues PBM buffers that are to be sent, and then these are dequeued in EMAC_TxServiceCheck at ISR level, and go to layer 1.

Guide: EmacPktService -> HwRxInt -> EMAC_RxServiceCheck
Code: EmacPktService -> PBMQ_deq -> NIMUReceivePacket

Similar to the above, not exactly a call chain model since these are ISRs.

The RX ISR will enqueue frames received from layer 1.  These are dequeued and passed up/into the stack via NIMURecievePacket.  This needs to be mentioned in this table (I've updated the doc bug for this)

Guide: EmacPoll -> _HwPktPoll -> EMAC_TimerTick and MDIO_timerTick and MDIO_getStatus
Code: EmacPoll -> _HwPktPoll -> MDIO_timerTick

Again, I think this isn't a mapping but a categorization of APIs in the layers defined in the table.

4. GPIO_CONTROL_REG and TIMER_GLOBAL_REG found in the routine EMAC_initialize( ) appear to be not needed and dangerous if left in. For some reason EMAC_initialize( ) attempts to take control of TIMER0 via GPIO_CONTROL_REG and TIMER_GLOBAL_REG

Perhaps Senthil can comment on this.

According to the NDK Users Guide, the Timer object required by the NDK is automatically created if you use XGCONF. I can't find any evidence of XGCONF doing this for me

All of the NDK's config settings made in XGCONF are used to generate C code when you build your app.  You can see this in a large *.c file down in the "Debug" or "Release" directory of your app.  For example, I see it here on my project, it should be similar for the OMAPL138:

In that file, you'll find the config settings in C code.  The code will look something like this:

/* Main Thread */
Void ti_ndk_config_Global_stackThread(UArg arg0, UArg arg1)
{
    int rc;
    HANDLE hCfg;

    ti_sysbios_knl_Clock_Params clockParams;

    /* Create the NDK heart beat */
    ti_sysbios_knl_Clock_Params_init(&clockParams);
    clockParams.startFlag = TRUE;
    clockParams.period = 100;
    ti_sysbios_knl_Clock_create(&llTimerTick, clockParams.period, &clockParams, NULL);

5. StatusUpdate( ) and StatisticsUpdate( ), found in ethdriver.c, do nothing. Seems like StatusUpdate( ) is very important, as that tracks fatal EMAC errors. What is a fatal EMAC error? When a fatal EMAC error is found, what should be do? What is the recommendation from TI? Shut down the socket? Reboot the NDK? Reboot the system? Get a beer?

A beer sounds great right about now :)

It seems that a hardware reset is required if this occurs.

I found the following in the TMS320C674x/OMAP-L1x EMAC User's Guide:

"A hardware reset is the only means of recovering from the error interrupts (HOSTPEND), which are
triggered by errors in packet buffer descriptors. Before doing a hardware reset, you should inspect the
error codes in the MAC status register (MACSTATUS) that gives information about the type of software
error that needs to be corrected. For detailed information on error interrupts, see Section 2.16.1.4."

There are further details on error codes in the description of the MACSTATUS register.

Dean,

I also filed a bug report for the missing functionality of the Emacioctrl function:

SDOCM00114671 (NSP) OMAPL138 driver's IOCTL function does not handle all cases

Steve

Steve, I've reached the end of the road in my quest. I will summarize everything below. Please review, and if you have nothing further to add, click the Verify Answer button. For some reason, I can't see a verify answer button.

1. Find the correct NSP. The most recent NDK says it supports: C64P, C66, C674, ARM9, Cortex-A8, Cortex-A9, Cortex-A15, Cortex-M3, and Cortex-M4. Obviously you must be using one of these devices. Furthermore the NDK release notes says that an appropriate NDK Support Package (NSP) can be obtained separately. Finding the correct NSP for your device seems to be one of the biggest challenges. For my device (C6748), the NSP download is located right on the NDK download page. However I have no idea where to look if my device was a C66 for example. Since the NSP is such a critical part for the NDK, TI should consider listing EXACTLY where to go find the NSP for the devices that are supported in the NDK release notes.

2. Download and install all the latest greatest pieces that are required: SYSBIOS, NDK, XDCTOOLS, and Code Gen tools. The NDK release notes does have a Compatibility Information section listing the minimum version required for each piece.

3. Choose how you want to customize the NSP. Since the whole point of this post is to determine what needs to change to run the NDK on custom HW, you'll need to change files in the NSP. Once you make those file changes, you can either rebuild the NSP or bring the NSP files into your project. I tried rebuilding the NSP, but the steps described in the NSP release notes weren't complete. Furthermore, once I built the custom NSP, Code Composer failed to automatically detect this new "package". Finally, for every new NSP release, I will need to make the same changes and rebuild each time. If you are more comfortable with Eclipse, maybe rebuilding is the way to go. For me, taking the files I need out of the NSP are bringing them into my project makes more sense.

4. Bring the NSP files into your project. For my device (C6748) there are five files, I assume this is the same across all devices. From the NSP folder tree (something like \packages\ti\drv\omap138) you'll need: csl_emac.c csl_mdio.c ethdriver.c and nimu_eth.c. You'll also need emacHooks.c from one of the example NSP project folder (something like \packages\ti\ndk\examples\ndk_evm6748_elf_examples\ndk_evm6747_elf_helloWorld). To make them easier to find and stand out in my project, I renamed all the files by adding a prefix and suffix to each file. The prefix is nsp_ and the suffix is the name of my custom HW. For example, csl_emac.c becomes nsp_csl_emac_2400i.c.  There are .h files you'll need from the \drv\omap138\inc directory.  You can either drag all of them into your project (there are a lot), or you can see step #7.  Since none of the .h files need to be modified, I prefer the directions in step #7.
 
5. Open Properties for your SYSBIOS project and go to the RTSC tab. Check the box for the NDK, but not for the NSP. You'll also need to add _INCLUDE_NIMU_CODE to the Predefined Symbols under CCS Build -> C6000 Complier. Save and exit.

6. Open the *.cfg file for you project and add NDK->Global item. Save and exit.

7. Right click on the project and select Show Build Settings. Add the path c:\CCSv5\ndk_2_24_01_18\packages\ti\ndk\inc. Obviously change the beginning of path to where ever you loaded the NDK on your machine. Also make sure the version in the path matches the version you selected in step #5.  Finally if you didn't bring all the .h files into your project from step #4, you'll need to add that path as well.  Add the path c:\CCSv5\nsp_1_10_02_09\packages\ti\drv\omapl138\inc.  Again change the beginning of the path to match were you installed it on your machine, and make sure you are referencing the correct NSP version.
 
8. Verify everything builds with no errors. Now we are finally ready to make the custom HW changes to the NSP files.

9. No matter what custom HW you have, nimu_eth.c shouldn't require any changes.

10. Assuming you started with the correct NSP for your device, csl_emac.c probably won't require any changes.

11. csl_mdio.c will require a few changes. Near the top of the file, VBUSCLK is defined. This controls how fast the MDIO clock runs. MDIO clock = SYSCLK4 / VBUSCLK. The C6748 DSP can run this clock up to 2.5MHz, and the PHY I'm using can go all the way up to 25MHz (2.5MHz is listed as typical). I'm going to change VBUSCLK so the MDIO clock is running closer to the 2.5MHz maximum. For example, if my DSP clock is running at 368.6MHz, that puts my SYSCLK4 at 92.2MHz. So if I set VBUSCLK to 40, that will make my MDIO clock be 2.3MHz. Close enough.

12. csl_mdio.c also requires you to change the macsel variable to match your physical connection to the Ethernet PHY (MII, RMII, GMII, or RGMII).

13. The final change(s) to csl_mdio.c are only to clean up stuff and remove warnings. Remove #if 0 code if you want (bad coding standard). Remove ltmp1 from MDIO_initPHY( ), and also change the i and ack variables to volatile to remove the warnings.

14. ethdriver.c may require some changes. CORENUM probably never has to change. EXTERNAL_MEMORY and EXTMEM would only change if you can fit the PBM into L2 or L1D (very rare). PKT_MAX can be increased to provide more memory to the RX and TX queues. If you do increase this constant, you must also increase Number of Frames field in the XGCONF (NDK -> Buffers -> Number of Frames). RXINT and TXINT can change if you want to move the HW interrupt vectors.

15. ethdriver.c disables global interrupts to service the EMAC. This may not be desirable. You can replace disableInterrupts() with Hwi_disableIER( ) and replace restoreInterrupts( ) with Hwi_enableIER( ) if you only want to disable the EMAC HW interrupts instead of the entire system. WARNING!!!  SEE STEVE'S WARNING THAT SELECTIVELY DISABLING INTERRUPTS MAY CAUSE THE NDK TO LOCKUP.  PROCEED AT YOUR OWN RISK.  Finally you might want to rename HwTxInt( ) and HwRxInt( ) to something else so they better represent what the actually do.

16. The final change to ethdriver.c would be to the StatusUpdate( ) routine. This routine tracks fatal EMAC errors, and if one occurs, you apparently have to reboot the entire machine. Your code will have to decide what to do when this occurs (alert the user, write to a log file, etc.) before you physically reset.  Recommend adding two new function calls.  In RxPacket(), add a function called EMAC_OutofMemory_Error() to the line above memory_squeeze_error++.  In StatusUpdate() add EMAC_Fatal_Error() to the line above emac_fatal_error++.  You'll need to add the prototypes for these function to the top of this file.  These new functions will live in the emacHooks.c file.
 
17. Each routine in emacHooks.c will require modification. EMAC_initialize( ) needs to remove the GPIO_CONTROL_REG and TIMER_GLOBAL_REG code. You'll also have to change CFGCHIP[3] to select MII or RMII mode. Finally the pinmux stuff needs to match your MII or RMII selection. Better yet, remove the pinmux code and define your pinmux settings in your GEL file and AISGEN.

18. emacHooks.c also controls where the Ethernet MAC address comes from. Modify EMAC_getConfig( ) to pass in your custom Ethernet MAC to the NDK.

19. emacHooks.c lets you know when the Ethernet cable get unplugged / plugged in. Modify EMAC_linkStatus( ) to do whatever you system requires when these events happen.

20.  Finally create the new functions EMAC_OutofMemory_Error() and EMAC_Fatal_Error().  Basically you'll need to log any error info you want.  You'll have to reference the C6748 Tech Manual to see what registers you can read to get a reason for why the error occurred.  The NDK must be completely shut down, the EMAC hardware reset (probably the PHY to), and everything restarted.  Since you should do that from these routines, probably just set some global variables you can check in an outside task.  Really depends on how sensitive your system is to fatal errors on what action you should take.

Dean,

This looks great, nice work and thanks for taking the time to write this up.  This is true community support!

The only thing that caught my attention is in regards to step 15:

Dean Hofstetter said:

15. ethdriver.c disables global interrupts to service the EMAC. This may not be desirable. You can replace disableInterrupts() with Hwi_disableIER( ) and replace restoreInterrupts( ) with Hwi_enableIER( ) if you only want to disable the EMAC HW interrupts instead of the entire system.

I think it's worth mentioning that a (very difficult to pin down) issue came about in another NSP driver (for the DM648 hardware platform) that ended up being due to the driver only disabling the EMAC interrupt, instead of globally disabling interrupts.

The issue occurred during a ping flood attack and caused the EMAC to become unresponsive.  The flood attack caused the hardware to generate an interrupt before the previous ISR for that interrupt had run to completion.  The fix was to change the disabling of the single EMAC interrupt to a global disabling of interrupts, to allow the EMAC ISR to run to completion.

Details on the fix for that are here.

The forum thread for the issue is here.

Just think it would be good to consider this possible ramification given how hard it was to debug and track down.

Steve