This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

am243x Freertos jitter

Other Parts Discussed in Thread: SYSCONFIG

Hi

We suffer from unexplained jitter in FreeRTOS (Cortex R, core 0).

Measurement method:

  1. Periodic interrupt, once at 500 uSecs
  2. ISR posted a binary semaphore to task (at highest priority)
  3. Tasks wakes up and samples CPU time stamp counters.

Differences are used to assess system jitter.

Simple example, one task, jitter within 2-5 uSecs. Good.

It looks like latency grows when I enable more parts in software, plus some factors that I cannot explain.

No access to hardware. Low priority tasks wake up periodically, but do nothing.

For example, there is big difference in latency with and without initilization of network code.

Without init of network, latency/jitter is +/- 20 uSec, with init of network, but cable is disconnected (no interrupts from DMA) +/- 80 uSec.

All code run from DDR, DDR is cached.

Cache miss can explain such a behavior?

Any thoughts ?

  • Did you remove HwiP_disable from those 2 functions

    Udma_eventAllocResource
    Udma_eventFreeResource

    Thanks

    Rasty

  • Hello Rasty,

    As far as I know .We did not control global interrupts in our Test Application which means that did not remove HwiP_disable and HwiP_enable in our Test code. As AShwani mentioned that we reduce the work load of TX ISR and RX ISR routines and added these functionalities in Task.

    Regards,

    S.Anil.

  • Hi 

    I identified four (4) sources of latency

    1. ISR itself contrubutes 30-50 uSec.

    2. Udma_eventAllocResource locks interrupts for 8-9 uSec

    3. Udma_eventFreeResource  locks interrupts for 7-8 uSec 

    4. Cache invalidation creates additional uncertanty and I want to use non-cached memory for all DMA buffers, descriptors, frames, etc and get rid of cache invalidation..

    Meanwhile I get a kind of solution proposal  for #1. 

    2,3,4 are still open we need all at once.

    Is it possible to get complete solution after validation as one single package/patch ?

    regards

    Rasty

  • ISR itself contrubutes 30-50 uSec.

    Does it Mean, 

    • You started with enet_lwip_cpsw_tcpserver
    • Open sysconfig
    • Add timer instance with call back function and tick_period =125us

    Added below callback function in main.c file:

    void timerTickIsr(void *args)
    {
        //2 From timer ISR give a semaphore to high priority task
        static uint32_t count =0;
        BaseType_t doTaskSwitch = 0;
        uint32_t current_count =0;
        static uint32_t end_count =0;
    
    
        static uint32_t StartCounter = 0;
    
        uint32_t IsrCurrTime_us = 0;
        uint32_t min_pos[LOOP_COUNT] = {0};
        static uint32_t max_pos[LOOP_COUNT] = {0};
        static uint32_t loop = 0;
    
        static uint32_t min_jitter[LOOP_COUNT] = {0};
        static uint32_t max_jitter[LOOP_COUNT]= {0};
    
        //SemaphoreP_post(&gBinarySem);
        current_count = CycleCounterP_getCount32();
        xSemaphoreGiveFromISR( gSem, &doTaskSwitch); /* wake up task */
        portYIELD_FROM_ISR( doTaskSwitch );
    
    
        IsrCurrTime_us =( (current_count - end_count) / 800U );
    
        if( IsrCurrTime_us < min_jitter[loop])
        {
            min_jitter[loop] = IsrCurrTime_us;
            min_pos[loop] = count;
            //MinJitter[count] = Jitter_CurrTime_us[count];
    
        }
    
        if( IsrCurrTime_us > max_jitter[loop])
        {
            max_jitter[loop] = IsrCurrTime_us;
            max_pos[loop] = count;
            //MaxJitter[count] = Jitter_CurrTime_us[count];
    
        }
    
        count++;
    
        if(count >= SAMPLE_COUNT)
        {
            count = 0;
        }
    
        if(loop > LOOP_COUNT)
        {
            loop = 0;
            count = 0;
        }
    
        end_count = current_count;
    
    }

    My observation is that we are getting this call back hit with 125us interval.

    But, I your case it is around 125us + 30us ?

    Regards

    Ashwani

  • Hi Ashwani

    I'm tired t answering to the same question again and again.

    We found that system suffers from jitter.

    Jitter can be measured by sampling of CPU time-stamp counter by cyclic high pirority task, triggered by semaphore posted from timer interrupt. 

    We search for shortest and longest cycle time (min/max), which reprsents WORST case, NOT STATISTICS.

    Is this measuring approach clear to you (yes/no)?

    do you agree with this approach (yes/no)?

    There  are 4 sources of latencies that I described above (any many times before).

    Do you agree with each 4 (yes/no) ?

    We need task jitter below 5 uSec. Is that archivable with Ethernet traffic (yes/no) ?

    Thanks

    Rasty

  • Understood.

    Please ignore my last reply as I got confuse with freeRTOS ISR jitter = 30-50us.

    Is this measuring approach clear to you (yes/no)?

    We have similar setup at our side and whatever results/observation we are putting here captured on the setup.

    4. Cache invalidation creates additional uncertanty and I want to use non-cached memory for all DMA buffers, descriptors, frames, etc and get rid of cache invalidation..

    We doubt on completely get rid of cache invalidation as it is handled in driver code.

    Is this measuring approach clear to you (yes/no)?

    DMA descriptor and DMA RING MEMORY should always be cached.

    *(*ENET_DMA_PKT_MEMPOOL)

    Packet data can be moved in un-cached memory.

    As we have similar results except point #4.

    Now we are working on optimal memory placement only.

    Will keep you posted.

    Regards

    Ashwani

  • As we have similar results except point #4.

    Now we are working on optimal memory placement only.

    What we will try - to keep source and destination buffers in non-cached mode.

    Add these buffer memories to non-cached memory in the mpu region setting.

    Regards

    Ashwani

  • Do you agree with each 4 (yes/no) ?

    YES AND NO.

    We are still not sure about the cache as our analysis shows that networking use-case is cache friendly. It means, we see cache HITS most of the times.

    Regards

    Ashwani

  • We need task jitter below 5 uSec. Is that archivable with Ethernet traffic (yes/no) ?

    In our current SDK examples, we have not seen this performance.

    But, it does not mean NOT POSSIBLE. Customer need to optimize memory usage as per their use case to get the performance. We have 128K TCM.

    Did you explore to use this memory to optimize performance ?

    If yes, can you provide details ?

    Regards

    Ashwani

  • Hi

    Our use case:

    1. On-chip RAM (~2MB) for realtime, low jitter code, maximum latency is 5 uSec

    2. DDR for everything else, like lwip, networking

    3. TCM is not used, because we do not have anything that can fit into it.

    Bottom line:

    We need under 5 uSec cycle jitter in task that runs from Internal memory and uses internal memory for data, while system is under network load.

    We do not reach it because (explaned many times already)

    a) heavy processing in DMA ISR (50-60 uSec)

    b) two places where interrupts are disabled for 8-9 uSec

    c) cache invalidation, contribution to to jitter is unknown 

    Thanks

    rasty

  • Hello Rasty,

    FYI , we are closely working with development Team  to fix all your issues .

    Here is the current status update : 

    1. ISR itself contributes 30-50 uSec and Application Team (Ashwani has given temporary fix ) and the same patch is reviewing by development Team and after that we will need additional long run to measure performance of network statistics.

    2. We had identified some of the lwip related service/periodic task is adding for a larger variance. We will work on it and try to optimize the example specific to Servtronics requirements.

    3. We will review the entire code and list down all the places of critical section (excluding init seq part of code, rather on the steady state of the code).

    We will review this and decide which one to address first for servtronics .

    4. Not concentrating on on Cache Invalidation.  This is deprioritized as of now.

    We  will come up with the proposals and solutions  for 2nd and 3rd points with in 3 to 4 days

    Regards,

    S.Anil.

  • Hello Rasty,

    We have continuously been doing different implementations and testing for reducing jitter. Below is a summary of what we have done tested at different @10 to 20 Mbps ethernet speeds (in O3 optimized applications in release mode).


    Test case 1: In the current MCU+SDK, the TX and RX isr should not be called in cpsw_enent_example in polling method, but these functions were being still called in the polling implementation. Once this issue was fixed the TX and RX ISR functions are no longer being called, reducing the number of interrupts and context switches.
    The below one is the Patch.So, Tx and Rx processing issues have been fixed.

    Test case 2:We have identified the places where global interrupts are being disabled. Please find the details below:
    In the first set of highlighted functions, we implemented mutex and then verified at different speeds. We noticed that we still had ~10 used jitters.

    Total API's where we disabling global interrupts : 

    First set of  API's where we disable  global interrupts : 


    Test case 3: We have implemented mutex for the second set of functions below as well, but we are seeing Ethernet functionality breaking due to this. We are currently working on fixing this, this should improve jitter.

    Second set of API's where we disable global interrupts : 


    Questions on your implementation: You mentioned that with your patch you are able to get 5-7us of jitter.
    1. What is the ethernet packet rate you have tested for?
    2. For how many hours have you tested this patch? We see some outliers in overnight testing.

    3. Release mode or debug mode ?

    Patch details : 

    https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/81/0001_2D00_remove_2D00_udma_2D00_tx_2D00_ISR_2D00_add_2D00_sem_2D00_protection_2D00_for_2D00_pbufQ.patch

    Regards,

    Anil.

  • Hi

    Thank you for the update

    1. Very slow rate, interactive againts terminal (putty). Printing some data over terminal, few screens of text.

    2. Go/not go test. Few minutes.

    3. All drivers/tcpip/dma in debug mode, just freeRTOS in release mode. because in release we have very strange side offects if we compile with gcc.

    Thanks

    Rasty

  • I briefly checked this patch. It replaced interrupt locking with semaphore. But does not offload work from ISR.

    Is it right ? What do I miss?

  • Hello Rasty,

    I understood your concern.

    Actually, if you do the same test in release mode you may get lesser jitter values as compared to debug mode since release mode is optimized code.

    Currently, we have done testing  in release mode with the TI clang compilers.

    In the patch there are two fixes. One is replacing global interrupts with the semaphore. 

    And another one is running Ethernet  in polling mode. In this case your TX ISR and RX ISR will not hit.

    In the driver, we have already disabled RX and TX ISR but unfortunately TX ISR is hitting currently in the example .

    We have added the below patch to disable the TX ISR and this also need to be fixed in the MCU+SDK as well in  future releases. 

    because in release we have very strange side offects if we compile with gcc

    Is it possible to share what the issues are you have faced in release mode since these issues should be fixed in GCC compilers as well ?

    Regards,

    Anil.

  • Hi,

    Can you re-post all patches?

    Thanks

    Rasty

  • Hello Rasty,

    0001-remove-udma-tx-ISR-add-sem-protection-for-pbufQ.patch

    I have added patch details to the above message.

    Can you please check and confirm if you are facing any issues while accessing it ?

    because in release we have very strange side offects if we compile with gcc.

    I remembered that earlier you have faced problems with the UDMA API, which is supposed to return 64bit value, but it was returned to 32bit. Due to this, you have an  SCI client error. With this error, you are not able to run the Ethernet  example. So, I assume that because of this issue you moved to GCC in debug mode and not in release mode.. 

    Apart from the above issue, did you face any issues with GCC in release mode?

    Regards,

    Anil.

  • Hi

    I see only one (1) pathc that replaces interrupt lock with semaphore, Is there another patch that offloads work from ISR?

    We see strange/unexplanable crashes if we use release compilation. Most probably it is related to out of order access (due to agressive optimization) to peripheral memory. 

    Thansks

    Rasty

  • Hello Rasty,

    We are not offloading ISR works and instead of that, we have  changed to run  the Ethernet example in polling mode.

    To run the ethernet in polling mode, we have added patches in the below commit.

    Please look at the below commit and comments for more details for polling mode .

    I see only one (1) pathc that replaces interrupt lock with semaphore, Is there another patch that offloads work from ISR?

    Currently, we have done testing  in release mode with the TI clang compilers.

    In the patch there are two fixes. One is replacing global interrupts with the semaphore. 

    And another one is running Ethernet  in polling mode. In this case your TX ISR and RX ISR will not hit.

    In the driver, we have already disabled RX and TX ISR but unfortunately TX ISR is hitting currently in the example .

    We have added the below patch to disable the TX ISR and this also need to be fixed in the MCU+SDK as well in  future releases. 

    We see strange/unexplanable crashes if we use release compilation. Most probably it is related to out of order access (due to agressive optimization) to peripheral memory. 

    Yes, this also need to check at our side and we currently given priority to reduce the jitter once we got the expected jitter values then we can finally work in GCC compilers .

    Regards,

    Anil.

  • Rasty,

    FYI, I am on training for the entire this week, and you may get a reply from me after 7PM IST.

    Regards,

    Anil.

  • Hi

    Questtions

    1. Is there also similar treatment for RX path  (like EnetDma_disableTxEvent(pInterface->hTx[txChIdIndex]->hCh)

    2. What is SDK version you work on?

    3. Where is actual polling? Do I need to add polling code/task? 

    Thanks

    Rasty 

  • Hi Rasty,

    Thanks for being with us patiently. We have tried and tested multiple combinations of traffic and polling / Interrupt methods.

    Setup:

    AM243x <=> TI-Network <=> WIndows-PC

    Traffic:

    iperf-2.1.5-win.exe -c 10.24.72.188 -b 10M -i 2 -t 0 -r

    iperf-2.1.5-win.exe -c 10.24.72.188 -b 20M -i 2 -t 0 -r

    Test Case  Saturated Max (5 min Test duration) Global Max (30 min Test duration) TCP Traffic Rate
    Interrupt + Tx, Rx ISR offload function (Similar to Rast Patch) 8us + 200us + 10K
    Interrupt + Tx, Rx ISR offload function (Similar to Rast Patch) 12us + 200us + 10M
    Interrupt + Tx, Rx ISR offload function (Similar to Rast Patch) 14us + 200us + 20M
    Interrupt + Tx, Rx ISR offload function (Similar to Rast Patch) 5-6 us Running No pumping traffic, but network cable is connected
    Polling + Tx, Rx ISR offload function + DMA optimization by Anil 12us+ 200us+ 10M
    Polling + Tx, Rx ISR offload function + DMA workaroud by Anil 15us+ 200us+ 20M
    Outlier removal patch by dev team 11us+ 155us+ 10M

    • To remove outlier, we have tried some changes on our side. Outler not delayed (now it is coming in 1 Hour test), but not removed completely.

    Next Steps:

    • As we have seen task jitter to vary with traffic rate.
      • Can you provide exact quantified value of traffic rate, you are looking for. We will run and optimized the test with that only?
    • We will work on Enet profiling - Anil will update on that.

    Regards

    Ashwani

  • Hello Rasty,

    1. Is there also similar treatment for RX path  (like EnetDma_disableTxEvent(pInterface->hTx[txChIdIndex]->hCh)

    In MCU+SDK, this RX path  already disabled. It was controlled from variable below variable , and again, we don't need to do it separately for RX 

    outArgs->disableEvent = true. So, we have not disabled the Rx path.

    Please look at the patch below.

    2. What is SDK version you work on?

    We are also using the MCU+SDK 09.00 version.

    3. Where is actual polling? Do I need to add polling code/task? 

    No need, it has already added in the example.

    The tasks below are used in polling mode, and you don't need to create a new one.

    LwipifEnetApp_rxPacketTask

    LwipifEnetApp_txPacketTask

    Ashwani mentioned that the similar to Rasty Patch means, we have added changes below.

    1. Offloading TX and RX ISR works in the task based on the flag .

    2. Semaphore lock implemented for 4 UDMA API's

    Udma_queraw, udma_dequeraw, Udma_eventAllocResource and Udma_eventFreeResource 

    Regards,

    Anil

  • Hi,

    There are no special requirements

    our use case

    1. Peridic polling of status via TCP/IP connection - query every 50 uSec. Aproximately 1 kb of data.

    2. Retrieval of file (10 mBytes) via ftp overy few minutes.

    Thanks

    Rasty

  • Hello Rasty,

    Thanks for sharing your requirements above.

    I can check with the internal team about your requirements and let you know about the further steps.

    Regards,

    Anil.

  • Hi Rasty,

    Have few questions on the application design -

    1.

    1. Peridic polling of status via TCP/IP connection - query every 50 uSec. Aproximately 1 kb of data.

    This leads to TCP data throughput to 160+Mbps, with 20000 packets per sec. At this packet rate we would suggest to enable batch processing with much larger polling interval.

    Do you think, you can increase the polling interval in the order of milliseconds from 50us? This will increase packet latency, but if you have a lower thought put (<10mbps), we can suggest alternatives to minimize the latency. Achieving both low latency and high throughput simultaneously is a challenge. Please share your thoughts. Idea is - as we know high Interrupt latency is seen only with ethernet traffic and hence we are trying to explore the options in LWIP that is impacting here. Hence your answer will be helpful to set the right expectations about ethernet traffic pattern.

    2. How are you polling the TCP connection? Is it based on another timer interrupt? Or an external event? What is the relative priority of the task where the connection is polled?

    With regards,

    Pradeep 

  • Hi,

    You ask me question, but I expected to get answer from you actually.

    We can run thread that polls (calls to function that you suggest)  Ethernet RX/TX/DMA/else every 1 msec.

    We exhange files of up to 10 mBytes. Send/recieve time shall take not more than few (2-3) seconds, othewise it will irritate user.

    I have nothing more to add.

    Thanks

    Rasty

  • Hey Rasty,

    Here is the one more update from us .

    We have tried and tested multiple combinations of putting code and data in to different memories and taking jitter values.

    And in our testing we found that when SOC is running on DDR, it creates memory access  latencies which will increase jitter.

    So, we tried to put the entire code in to MSRAM memory and saw jitter almost 5usec at 10mbps data rate and without communication.

    Based on the above results, we need to work on below two tasks .

    1. what are the lwip and enet functions we should keep in MSRAM instead of keeping entire code . We know it , your application should run mostly on DDR .

    2. Next, we have seen that during almost 1Hr long running , we have seen a peak jitter value which is almost 170usec at 10Mbps data rate .

    And, we are trying to work on the above two issues.

    Patch details : 

    1. Enable Polling Mode which you have already checked this patch from the above replies.

    2. Kept code in MSRAM memory. Please look at the Linker file for your reference.

    https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/81/3377.linker.cmd

    Regards,

    Anil.

  • Hi 

    I think that you got us wrong, or quite opposite.

    Real-time application, both code and data, go to  MSRAM (on chip memory). While TCP/IP stack, Ethernet drivers , DMA, buffers go to DDR.

    In this case we expect that Network will not isignificantly nfluence real-time.

    Regards

    Rasty

  • Hi Anil,

       Is it possible TI try using hardware timers instead of Freertos software timers to control the lock of ENET TX RX? 

  • It is OK to do RX/TX polling from low priority task. No need in interrupts and timers.

    Just polling and some sleep in between.

  • Hello Rasty,

    Here is one more update.

    We have seen two issues in our testing.

    Issue 1 : Jitter value is maximum  10usec with the Polling Mode at 10Mbps data rate.

    Issue 2 : Jitter value is maximum 180usec with the Polling Mode at 10Mbps data rate in the long run. (Run > 50 mins)

    For above issues, we have done analysis on each Task and ISR entry and exit timing analysis.

    We found that from task to task switching and memory accessing from DDR are creating the jitter.

    In our analysis,  the latency in between ISR end to T3 (Jitter) Task start time varies from 5usec to 10usec.

    To reduce this latency, we have added the FreeRTOS library to the MSRAM and after we have seen that jitter almost 4.2use is the average value and

    6usec is peak value , with in 50000 * 50 samples, each sample at 125usec with data (10Mbps) and with out data . 

    I have updated the linker command file as below. 

    With the software change below, we have fixed Issue1 

    And almost ~40 Kbytes of MSRAM memory is occupied for the change below.

    For issue 2, we have seen that Task 3 jitter Task was missing, and actually this task should hit approximately 125usec, but it was not hitting at 125usec, and we have not found any root cause for this issue, and we are working on it.

    Please look at the excel sheet document for your reference.

    SECTIONS
    	{
    
    
    	      -l "C:/ti/mcu_plus_sdk_am64x_09_00_00_35/source/kernel/freertos/lib/freertos.am64x.r5f.ti-arm-clang.release.lib" (.text)
    
    	      -l "C:/ti/mcu_plus_sdk_am64x_09_00_00_35/source/kernel/freertos/lib/freertos.am64x.r5f.ti-arm-clang.release.lib" (.data)
    
    	      -l "C:/ti/mcu_plus_sdk_am64x_09_00_00_35/source/kernel/freertos/lib/freertos.am64x.r5f.ti-arm-clang.release.lib" (.bss)
    
    	      -l "C:/ti/mcu_plus_sdk_am64x_09_00_00_35/source/kernel/freertos/lib/freertos.am64x.r5f.ti-arm-clang.release.lib" (.rodata)
    	 } > MSRAM {} palign(8)
    
    

    TestResults_withLoad.xlsx

    Regards,

    Anil.

  • Thank you for update.

    What is the next step?

  • Hello Rasty,

    Thanks for the get back to us .

    1. Is it ok to go with ~40KB MSRAM/TCM  consumption in your use case ?

    2. Is it possible to integrate above changes such as polling mode and moving freertos libraries to MSRAM in your application and see how much jitter getting in your application ?

    3. As I mentioned above during long run we have seen that T3 task was missing due to we are getting peak jitter value and we will work on this issue .

    Regards,

    Anil.

  • Hi Anil

    We can do it (1 and 2).

    Can you summrize all changes and links for # 2 in one single post ?

    Thanks

    Rasty

  • Hello Rasty,

    Please see I have attached all the patches details .

    Patch 1 : To move freeRTOS libraries to MSRAM memory.

    Please integrate only the change below in your linker cmd file and in my test setup I have updated linker file for different Test scenarios.

    Patch details : 

    SECTIONS
    	{
    
    
    	      -l "C:/ti/mcu_plus_sdk_am64x_09_00_00_35/source/kernel/freertos/lib/freertos.am64x.r5f.ti-arm-clang.release.lib" (.text)
    
    	      -l "C:/ti/mcu_plus_sdk_am64x_09_00_00_35/source/kernel/freertos/lib/freertos.am64x.r5f.ti-arm-clang.release.lib" (.data)
    
    	      -l "C:/ti/mcu_plus_sdk_am64x_09_00_00_35/source/kernel/freertos/lib/freertos.am64x.r5f.ti-arm-clang.release.lib" (.bss)
    
    	      -l "C:/ti/mcu_plus_sdk_am64x_09_00_00_35/source/kernel/freertos/lib/freertos.am64x.r5f.ti-arm-clang.release.lib" (.rodata)
    	 } > MSRAM {} palign(8)

    File : 

    https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/81/5468.linker.cmd

    Patch 2 : Replacing disable and enabling global interruptions with Mutex in pbuf and enet_queue.c files .

    Enabled true polling method .

    Files : 

    /*
     *  Copyright (C) 2018-2021 Texas Instruments Incorporated
     *
     *  Redistribution and use in source and binary forms, with or without
     *  modification, are permitted provided that the following conditions
     *  are met:
     *
     *    Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     *    Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the
     *    distribution.
     *
     *    Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
     *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
     *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
     *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
     *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
     *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
     *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
     *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
     *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
     *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
     *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     */
    
    #include <assert.h>
    #include <enet_osal.h>
    #include "lwip/pbuf.h"
    #include "pbufQ.h"
    #include <networking/enet/core/include/core/enet_utils.h>
    #include <kernel/dpl/HwiP.h>
    #include <kernel/dpl/SemaphoreP.h>
    /*
     * Free queue from which buffer pointers are allocated
     */
    pbufQ pfreeQ;
    static bool gPbufQ_initialized = false;
    
    static void*  pbufQ_hMutex = NULL;
    
    static inline void pbufQ_assert(uint8_t cond)
    {
        assert(cond);
    }
    
    /*
     * Initializes the freeQ.
     * MUST BE CALLED BEFORE INITIALIZING THE FIRST BUFFER QUEUE
     * Needs to be called only once as early in the program as possible
     */
    //TODO: Refractor using enQ/deQ
    void pbufQ_init_freeQ(pbufNode *pfree, uint32_t maxSize)
    {
        int fQ_iter;
        pbufQ_assert(NULL != pfree);
    
        if (!gPbufQ_initialized)
        {
            for(fQ_iter=0; fQ_iter<maxSize-1; fQ_iter++)
            {
                pfree[fQ_iter].next = &pfree[fQ_iter+1];
                pfree[fQ_iter].hPbufPkt = NULL;
            }
    
            pfree[fQ_iter].next = NULL;
            pfree[fQ_iter].hPbufPkt = NULL;
    
            pfreeQ.head = &pfree[0];
            pfreeQ.tail = &pfree[maxSize - 1];
            pfreeQ.count = maxSize;
    
            gPbufQ_initialized = true;
            if (pbufQ_hMutex == NULL)
            {
                pbufQ_hMutex = EnetOsal_createMutex();
                Enet_devAssert(pbufQ_hMutex != NULL);
            }
        }
    }
    
    /*
     * Allocates memory from the freeQ to be used by other queues
     */
    pbufNode* mempQ_malloc()
    {
        pbufNode* p = pfreeQ.head;
        pfreeQ.head = pfreeQ.head->next;
        pfreeQ.count--;
        return p;
    }
    
    /*
     * Returns buffer pointers used by other queues back to freeQ
     */
    void mempQ_free(pbufNode* p)
    {
        p->next = NULL;
        p->hPbufPkt = NULL;
        pfreeQ.tail->next = p;
        pfreeQ.tail = p;
        pfreeQ.count++;
    }
    
    /*
     * Initializes a queue. Must be called after declaring a queue
     */
    void pbufQ_init(pbufQ *pQ)
    {
        EnetOsal_lockMutex(pbufQ_hMutex);
        pQ->head = NULL;
        pQ->tail = NULL;
        pQ->count = 0;
        EnetOsal_unlockMutex(pbufQ_hMutex);
    }
    
    /*
     * Enqueues a buffer to the tail of the queue
     */
    void pbufQ_enQ(pbufQ *pQ, struct pbuf *p)
    {
        pbufNode* temp = NULL;
    
        temp = mempQ_malloc();
        pbufQ_assert(NULL != temp);
    
        EnetOsal_lockMutex(pbufQ_hMutex);
    
        temp->hPbufPkt = p;
        temp->next = NULL;
    
        if(pQ->count == 0)
        {
            pQ->head = temp;
            pQ->tail = temp;
        }
        else if(pQ->count == 1)
        {
            pQ->head->next = temp;
            pQ->tail = temp;
        }
        else
        {
            pQ->tail->next = temp;
            pQ->tail = pQ->tail->next;
        }
        pQ->count++;
    
        EnetOsal_unlockMutex(pbufQ_hMutex);
    
    }
    
    /*
     * Enqueues a packet to the head of the queue.
     * NOT USED ANYWHERE. Can be removed
     */
    void pbufQ_enQHead(pbufQ *pQ, struct pbuf *p)
    {
        pbufNode* temp = NULL;
        pbufQ_assert(p != NULL);
    
        temp = mempQ_malloc();
        pbufQ_assert(NULL != temp);
    
        EnetOsal_lockMutex(pbufQ_hMutex);
        temp->hPbufPkt = p;
        temp->next = NULL;
        if(pQ->count == 0)
        {
            pQ->head = temp;
            pQ->tail = temp;
        }
        else
        {
            temp->next = pQ->head;
            pQ->head = temp;
        }
    
        EnetOsal_unlockMutex(pbufQ_hMutex);
    }
    
    /*
     * Dequeues from the queue
     */
    struct pbuf* pbufQ_deQ(pbufQ *pQ)
    {
        struct pbuf *rtnPbuf = NULL;
        pbufNode *temp = NULL;
    
        EnetOsal_lockMutex(pbufQ_hMutex);
    
        if(pQ->count != 0)
        {
            rtnPbuf = pQ->head->hPbufPkt;
            temp = pQ->head;
            pQ->head = pQ->head->next;
            if(pQ->count == 1)
            {
                pQ->tail = NULL;
            }
            pQ->count--;
    
            pbufQ_assert(rtnPbuf != NULL);
            mempQ_free(temp);
        }
    
        EnetOsal_unlockMutex(pbufQ_hMutex);
    
        return rtnPbuf;
    }
    

    enet_udma_priv.c

    /*
     *  Copyright (c) Texas Instruments Incorporated 2020
     *
     *  Redistribution and use in source and binary forms, with or without
     *  modification, are permitted provided that the following conditions
     *  are met:
     *
     *    Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     *    Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the
     *    distribution.
     *
     *    Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
     *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
     *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
     *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
     *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
     *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
     *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
     *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
     *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
     *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
     *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     */
    
    /*!
     * \file  enet_queue.c
     *
     * \brief This file contains the implementation of the Enet Queue.
     */
    
    /* ========================================================================== */
    /*                             Include Files                                  */
    /* ========================================================================== */
    
    #include <stdint.h>
    #include <enet_cfg.h>
    #include <include/core/enet_osal.h>
    #include <include/core/enet_utils.h>
    #include <include/core/enet_queue.h>
    #include <kernel/dpl/SemaphoreP.h>
    
    
    
    
    
    static void* EnetQueue_hMutex = NULL;
    
    /* ========================================================================== */
    /*                           Macros & Typedefs                                */
    /* ========================================================================== */
    
    /*! \brief Magic vlaue used to indicate when a queue has been initialized. */
    #define ENET_Q_INIT_DONE                      (0xABCDABCDU)
    
    /* ========================================================================== */
    /*                         Structure Declarations                             */
    /* ========================================================================== */
    
    /* None */
    
    /* ========================================================================== */
    /*                          Function Declarations                             */
    /* ========================================================================== */
    
    /* None */
    
    /* ========================================================================== */
    /*                            Global Variables                                */
    /* ========================================================================== */
    
    /* None */
    
    /* ========================================================================== */
    /*                          Function Definitions                              */
    /* ========================================================================== */
    
    void EnetQueue_initQ(EnetQ *queue)
    {
    
        queue->count = 0;
        queue->head  = NULL;
        queue->tail  = NULL;
        queue->magic = ENET_Q_INIT_DONE;
    
        if (EnetQueue_hMutex == NULL)
        {
            EnetQueue_hMutex = EnetOsal_createMutex();
            Enet_devAssert(EnetQueue_hMutex != NULL);
        }
    
    
    }
    
    void EnetQueue_copyQ(EnetQ *dstQueue,
                         const EnetQ *srcQueue)
    {
    
        Enet_devAssert(srcQueue->magic == ENET_Q_INIT_DONE,
                       "Source queue is not initialized\n");
    
        EnetOsal_lockMutex(EnetQueue_hMutex);
    
        dstQueue->magic = ENET_Q_INIT_DONE;
        dstQueue->head  = srcQueue->head;
        dstQueue->tail  = srcQueue->tail;
        dstQueue->count = srcQueue->count;
    
        EnetOsal_unlockMutex(EnetQueue_hMutex);
    
    
    }
    
    void EnetQueue_enq(EnetQ *queue,
                       EnetQ_Node *node)
    {
    
    
        Enet_devAssert(queue->magic == ENET_Q_INIT_DONE,
                       "Queue is not initializated\n");
    
        Enet_devAssert(node != NULL,
                       "NULL enq to the queue\n");
    
        EnetOsal_lockMutex(EnetQueue_hMutex);
    
    
        node->next = NULL;
    
        if (0U == queue->count)
        {
            /* Queue is empty - Initialize it with this one node */
            queue->head = node;
        }
        else
        {
            /* Queue is not empty - Push onto tail */
            queue->tail->next = node;
        }
    
        /* Make tail of queue point to new */
        queue->tail = node;
        queue->count++;
    
        EnetOsal_unlockMutex(EnetQueue_hMutex);
    
    
    
    }
    
    void EnetQueue_enqHead(EnetQ *queue,
                           EnetQ_Node *node)
    {
    
    
        Enet_devAssert(queue->magic == ENET_Q_INIT_DONE,
                       "Queue is not initialized\n");
    
        EnetOsal_lockMutex(EnetQueue_hMutex);
    
    
        if (0U == queue->count)
        {
            queue->tail = node;
        }
    
        node->next = queue->head;
        queue->head = node;
        queue->count++;
    
    
    
        EnetOsal_unlockMutex(EnetQueue_hMutex);
    
    
    }
    
    EnetQ_Node *EnetQueue_deq(EnetQ *queue)
    {
        EnetQ_Node *node;
    
    
        Enet_devAssert(queue->magic == ENET_Q_INIT_DONE,
                       "Queue is not initialized\n");
    
        EnetOsal_lockMutex(EnetQueue_hMutex);
    
        node = queue->head;
    
        if (NULL != node)
        {
            queue->head = node->next;
            if (NULL == queue->head)
            {
                queue->tail = NULL;
            }
    
            queue->count--;
            node->next = NULL;
        }
    
    
    
        EnetOsal_unlockMutex(EnetQueue_hMutex);
    
    
        return node;
    }
    
    void EnetQueue_append(EnetQ *dstQueue,
                          EnetQ *srcQueue)
    {
    
        Enet_devAssert(dstQueue->magic == ENET_Q_INIT_DONE,
                       "Destination queue is not initialized\n");
        Enet_devAssert(srcQueue->magic == ENET_Q_INIT_DONE,
                       "Source queue is not initialized\n");
    
        EnetOsal_lockMutex(EnetQueue_hMutex);
    
        if (0U != srcQueue->count)
        {
            if (NULL == dstQueue->head)
            {
                /* queue is empty - Initialize it with the source queue */
                EnetQueue_copyQ(dstQueue, srcQueue);
            }
            else
            {
                /* queue is not empty - Add free queue contents onto end */
                dstQueue->tail->next = srcQueue->head;
                dstQueue->tail   = srcQueue->tail;
                dstQueue->count += srcQueue->count;
            }
        }
    
        EnetOsal_unlockMutex(EnetQueue_hMutex);
    
    
    }
    
    uint32_t EnetQueue_getQCount(EnetQ *queue)
    {
        Enet_devAssert(queue->magic == ENET_Q_INIT_DONE,
                       "Queue is not initialized\n");
    
        return queue->count;
    }
    
    void EnetQueue_verifyQCount(EnetQ *queue)
    {
    #if ENET_CFG_IS_ON(DEV_ERROR)
        EnetQ_Node *node;
        uint32_t count = 0U;
        EnetOsal_lockMutex(EnetQueue_hMutex);
    
        Enet_devAssert(queue->magic == ENET_Q_INIT_DONE);
        node = queue->head;
    
        while (NULL != node)
        {
            count++;
            node = node->next;
        }
    
        Enet_devAssert(queue->count == count);
        EnetOsal_unlockMutex(EnetQueue_hMutex);
    #endif
    }
    

     

    /**
     * Copyright (c) 2001-2004 Swedish Institute of Computer Science.
     * All rights reserved.
     *
     * Redistribution and use in source and binary forms, with or without modification,
     * are permitted provided that the following conditions are met:
     *
     * 1. Redistributions of source code must retain the above copyright notice,
     *    this list of conditions and the following disclaimer.
     * 2. Redistributions in binary form must reproduce the above copyright notice,
     *    this list of conditions and the following disclaimer in the documentation
     *    and/or other materials provided with the distribution.
     * 3. The name of the author may not be used to endorse or promote products
     *    derived from this software without specific prior written permission.
     *
     * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED
     * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
     * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
     * SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
     * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
     * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
     * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
     * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
     * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
     * OF SUCH DAMAGE.
     *
     * This file is part of the lwIP TCP/IP stack.
     *
     * Author: Adam Dunkels <adam@sics.se>
     *
     */
    
    /**
     * Copyright (c) 2018 Texas Instruments Incorporated
     *
     * This file is dervied from the ``ethernetif.c'' skeleton Ethernet network
     * interface driver for lwIP.
     *
     */
    
    /* Standard language headers */
    #include <stdio.h>
    #include <assert.h>
    #include <string.h>
    
    /* xdc header - should be first file included as due to order dependency */
    
    /* OS/Posix headers */
    
    /**
     * lwIP specific header files
     */
    #include "lwip/opt.h"
    #include "lwip/pbuf.h"
    #include "custom_pbuf.h"
    #include "lwip2enet.h"
    #include "lwip/inet_chksum.h"
    #include "lwip/prot/ip.h"
    #include <lwip/netif.h>
    #include <networking/enet/core/lwipif/inc/lwipif2enet_AppIf.h>
    
    /**
     * Enet device specific header files
     */
    #include <enet.h>
    
    #include <enet_appmemutils.h>
    #include <networking/enet/core/include/core/enet_utils.h>
    #include <enet_cfg.h>
    #include "enet_apputils.h"
    #include <networking/enet/core/lwipif/src/lwip2lwipif_priv.h>
    #include <kernel/nortos/dpl/common/printf.h>
    
    
    /*---------------------------------------------------------------------------*\
     |                             Extern Declarations                             |
     \*---------------------------------------------------------------------------*/
    
    
    /*---------------------------------------------------------------------------*\
     |                            Local Macros/Defines                             |
     \*---------------------------------------------------------------------------*/
    
    #define LWIP2ENET_UNKNOWN_MAC_PORT (-1)
    
    /*---------------------------------------------------------------------------*\
     |                         Local Function Declarations                         |
     \*---------------------------------------------------------------------------*/
    
    static void Lwip2Enet_deinitTxObj(Enet_Type enetType, uint32_t instId, Lwip2Enet_TxHandle hTx);
    
    static void Lwip2Enet_deinitRxObj(Enet_Type enetType, uint32_t instId, Lwip2Enet_RxHandle hRx);
    
    static void Lwip2Enet_saveAppIfCfg(Lwip2Enet_Handle hLwip2Enet,
                                       Lwip2Enet_AppInfo *appInfo);
    
    static void Lwip2Enet_initTxObj(Enet_Type enetType, uint32_t instId, uint32_t chEntryIdx, Lwip2Enet_TxObj* pTx);
    
    static void Lwip2Enet_initRxObj(Enet_Type enetType, uint32_t instId, uint32_t chEntryIdx, Lwip2Enet_RxHandle hRx);
    
    static void Lwip2Enet_mapNetif2Tx(struct netif *netif,
                                      bool isDirected,
                                      Lwip2Enet_TxHandle hTxLwip2Enet,
                                      Lwip2Enet_Handle hLwip2Enet);
    
    static void Lwip2Enet_mapNetif2Rx(struct netif *netif,
                                      bool isDirected,
                                      Lwip2Enet_RxHandle hRxLwip2Enet,
                                      Lwip2Enet_Handle hLwip2Enet);
    
    static void Lwip2Enet_notifyRxPackets(void *cbArg);
    
    static void Lwip2Enet_notifyTxPackets(void *cbArg);
    
    static void Lwip2Enet_pbufQ2PktInfoQ(Lwip2Enet_TxObj *tx,
                                         pbufQ *pbufPktQ,
                                         EnetDma_PktQ *pDmaPktInfoQ,
                                         Enet_MacPort macPort);
    
    static void Lwip2Enet_pktInfoQ2PbufQ(EnetDma_PktQ *pDmaPktInfoQ,
                                         pbufQ *pbufPktQ);
    
    static uint32_t Lwip2Enet_prepRxPktQ(Lwip2Enet_RxObj *rx,
                                        EnetDma_PktQ *pPktQ);
    
    static void Lwip2Enet_submitRxPktQ(Lwip2Enet_RxObj *rx);
    
    static uint32_t Lwip2Enet_prepTxPktQ(Lwip2Enet_TxObj *tx,
                                        EnetDma_PktQ *pPktQ);
    
    static void Lwip2Enet_freePbufPackets(EnetDma_PktQ *tempQueue);
    
    static void Lwip2Enet_updateRxNotifyStats(Lwip2Enet_PktTaskStats *pktTaskStats,
                                              uint32_t packetCount,
                                              uint32_t timeDiff);
    
    static void Lwip2Enet_updateTxNotifyStats(Lwip2Enet_PktTaskStats *pktTaskStats,
                                              uint32_t packetCount,
                                              uint32_t timeDiff);
    
    static void Lwip2Enet_print(Lwip2Enet_Handle hLwip2Enet,
                                const char *prnStr,
                                ...);
    
    static int32_t Lwip2Enet_startPollTask(Lwip2Enet_Handle hLwip2Enet);
    
    static void Lwip2Enet_stopRxTx(Lwip2Enet_TxObj* pTx, Lwip2Enet_RxObj* pRx);
    
    static void Lwip2Enet_submitTxPackets(Lwip2Enet_TxObj *tx,
                                          EnetDma_PktQ *pSubmitQ);
    
    static void Lwip2Enet_submitRxPackets(Lwip2Enet_RxObj *rx,
                                         EnetDma_PktQ *pSubmitQ);
    
    void Lwip2Enet_retrieveTxPkts(Lwip2Enet_TxHandle hTx);
    
    static void Lwip2Enet_timerCb(ClockP_Object *hClk, void * arg);
    
    static void Lwip2Enet_createTimer(Lwip2Enet_Handle hLwip2Enet);
    
    static void Lwip2Enet_initReleaseTxHandleInArgs(Lwip2Enet_TxHandle hTx,
            uint32_t chNum,
            LwipifEnetAppIf_ReleaseTxHandleInfo *inArgs);
    
    static void Lwip2Enet_initReleaseRxHandleInArgs(Lwip2Enet_RxHandle hRx,
            uint32_t chNum,
            LwipifEnetAppIf_ReleaseRxHandleInfo *inArgs);
    
    static uint32_t Lwip2Enet_getRxTxObjIdx(Enet_Type enetType, uint32_t instId, Enet_MacPort macPort);
    
    static void Lwip2Enet_setSGList(EnetDma_Pkt *pCurrDmaPacket, struct pbuf *pbuf, bool isRx);
    
    static void Lwip2Enet_freeRxPktCb(void *cbArg,
                                      EnetDma_PktQ *fqPktInfoQ,
                                      EnetDma_PktQ *cqPktInfoQ);
    
    static void Lwip2Enet_freeTxPktCb(void *cbArg,
                                      EnetDma_PktQ *fqPktInfoQ,
                                      EnetDma_PktQ *cqPktInfoQ);
    
    static int32_t Lwip2Enet_setMacAddress(const Enet_Type enetType, uint32_t instId, Enet_Handle hEnet,uint8_t macAddr[ENET_MAC_ADDR_LEN]);
    
    Lwip2Enet_RxMode_t LwipifEnetAppCb_getRxMode(Enet_Type enetType, uint32_t instId);
    
    bool Lwip2Enet_decrementRxRefCount(Lwip2Enet_RxObj* pRx);
    
    bool Lwip2Enet_decrementTxRefCount(Lwip2Enet_TxObj* pTx);
    
    void LwipifEnetApp_getRxChIDs(const Enet_Type enetType, const uint32_t instId, uint32_t* pRxChIdCount, uint32_t rxChIdList[LWIPIF_MAX_RX_CHANNELS_PER_PHERIPHERAL]);
    
    void LwipifEnetApp_getTxChIDs(const Enet_Type enetType, const uint32_t instId, uint32_t* pTxChIdCount, uint32_t txChIdList[LWIPIF_MAX_TX_CHANNELS_PER_PHERIPHERAL]);
    
    uint32_t LwipifEnetAppCb_getMacPort(const Enet_Type enetType, const uint32_t instId);
    
    /*---------------------------------------------------------------------------*\
     |                         Local Variable Declarations                         |
     \*---------------------------------------------------------------------------*/
    
    static Lwip2Enet_Obj  gLwip2EnetObj =  {.isInitDone = false, .isAllocated = false };
    
    /*---------------------------------------------------------------------------*\
     |                         Global Variable Declarations                        |
     \*---------------------------------------------------------------------------*/
    
    static Lwip2Enet_Handle Lwip2Enet_allocateObj(void)
    {
        uintptr_t key = EnetOsal_disableAllIntr();
        Lwip2Enet_Handle hLwip2Enet = &gLwip2EnetObj;
    
        if (!hLwip2Enet->isAllocated)
        {
            hLwip2Enet = &gLwip2EnetObj;
            //phn initialize hLwip2Enet
            hLwip2Enet->isAllocated = true;
            hLwip2Enet->isInitDone = false;
    
            for (uint32_t rxChIdIdx = 0; rxChIdIdx < LWIPIF_MAX_RX_CHANNELS; rxChIdIdx++)
            {
                hLwip2Enet->lwip2EnetRxObj[rxChIdIdx].refCount = 0;
            }
    
            for (uint32_t txChIdIdx = 0; txChIdIdx < LWIPIF_MAX_TX_CHANNELS; txChIdIdx++)
            {
                hLwip2Enet->lwip2EnetTxObj[txChIdIdx].refCount = 0;
            }
        }
    
        EnetOsal_restoreAllIntr(key);
    
        return hLwip2Enet;
    }
    
    void Lwip2Enet_assertPriv(bool cond, const int32_t line, const char* file)
    {
        volatile static bool waitInLoop = TRUE;
    
        if (!(cond))
        {
            void EnetAppUtils_print(const char *pcString,...);
            EnetAppUtils_print("Assertion @ Line: %d in %s : failed !!!\r\n", line, file);
            while (waitInLoop)
            {
    
            }
        }
    }
    static Lwip2Enet_RxHandle Lwip2Enet_allocateRxHandle(Lwip2Enet_Handle hLwip2Enet, Enet_Type enetType, uint32_t instId, const uint32_t objIdx)
    {
        uintptr_t key = EnetOsal_disableAllIntr();
        Lwip2Enet_RxHandle hRx = NULL;
        if (objIdx < LWIPIF_MAX_RX_CHANNELS)
        {
            hRx = &hLwip2Enet->lwip2EnetRxObj[objIdx];
        }
    
        EnetOsal_restoreAllIntr(key);
    
        return hRx;
    }
    
    static Lwip2Enet_TxHandle Lwip2Enet_allocateTxHandle(Lwip2Enet_Handle hLwip2Enet, Enet_Type enetType, uint32_t instId, const uint32_t objIdx)
    {
        uintptr_t key = EnetOsal_disableAllIntr();
        Lwip2Enet_TxHandle hTx = NULL;
        if (objIdx < LWIPIF_MAX_TX_CHANNELS)
        {
            hTx = &hLwip2Enet->lwip2EnetTxObj[objIdx];
        }
    
        EnetOsal_restoreAllIntr(key);
    
        return hTx;
    }
    
    
    Enet_MacPort Lwip2Enet_findMacPortFromEnet(Enet_Type enetType, uint32_t instId) // move to Cb file autogenerated
    {
        Enet_MacPort macPort = ENET_MAC_PORT_INV;
        if (enetType == ENET_ICSSG_DUALMAC)
        {
            macPort = (Enet_MacPort)(instId & 0x01);
        }
        return macPort;
    }
    
    Lwip2Enet_Handle Lwip2Enet_open(Enet_Type enetType, uint32_t instId, struct netif *netif)
    {
        uint32_t txChIdList[LWIPIF_MAX_TX_CHANNELS_PER_PHERIPHERAL];
        uint32_t rxChIdList[LWIPIF_MAX_RX_CHANNELS_PER_PHERIPHERAL];
        EnetApp_GetMacAddrOutArgs          macAddr;
        EnetApp_HandleInfo                 handleInfo;
        Lwip2Enet_Handle    hLwip2Enet = Lwip2Enet_allocateObj();
        Lwip2Enet_netif_t*  pInterface = &(hLwip2Enet->interfaceInfo[hLwip2Enet->numOpenedNetifs++]);
        uint32_t txChIdCount = 0, rxChIdCount = 0;
    
        /* get TxChID & RxChID and the corresponding channel counts */
        LwipifEnetApp_getTxChIDs(enetType, instId, &txChIdCount, &txChIdList[0]);
        LwipifEnetApp_getRxChIDs(enetType, instId, &rxChIdCount, &rxChIdList[0]);
    
    
        EnetApp_acquireHandleInfo(enetType, instId, &handleInfo);
        Lwip2Enet_assert(handleInfo.hEnet != NULL);
        pInterface->hEnet = handleInfo.hEnet;
    
        /* MCast List is EMPTY */
        hLwip2Enet->MCastCnt = 0U;
    
        /* Init internal bookkeeping fields */
        hLwip2Enet->oldMCastCnt = 0U;
    
        /* set the mac address of netif as invalid, as we will set the valid address below */
        netif->hwaddr_len = 0U;
        if (hLwip2Enet->isInitDone == false)
        {
            /* First init tasks, semaphores and clocks. This is required because
             * EnetDma event cb ISR can happen immediately after opening rx flow
             * in LwipifEnetAppCb_getHandle and all semaphores, clocks and tasks should
             * be valid at that point
             */
    
            LwipifEnetAppCb_getEnetLwipIfInstInfo(enetType, instId, &hLwip2Enet->appInfo);
    
            /* Save params received from application interface */
            Lwip2Enet_saveAppIfCfg(hLwip2Enet, &hLwip2Enet->appInfo);
    
            Lwip2Enet_assert(hLwip2Enet->appInfo.isPortLinkedFxn != NULL);
            Lwip2Enet_assert(hLwip2Enet->appInfo.pPbufInfo != NULL);
    
            pbufQ_init_freeQ(hLwip2Enet->appInfo.pPbufInfo, hLwip2Enet->appInfo.pPbufInfoSize);
    
            /* set the print function callback if not null */
            hLwip2Enet->print = (Enet_Print) &EnetUtils_printf;
            hLwip2Enet->isInitDone = true;
            Lwip2Enet_createTimer(hLwip2Enet);
        }
    
        /* Associate with corresponding Tx Channels */
        for (uint32_t txChIdIndex = 0; txChIdIndex < txChIdCount; txChIdIndex++)
        {
            const uint32_t txChId = txChIdList[txChIdIndex];
            pInterface->hTx[txChIdIndex] = Lwip2Enet_allocateTxHandle(hLwip2Enet, enetType, instId, txChId);
            Lwip2Enet_initTxObj(enetType, instId, txChId, pInterface->hTx[txChIdIndex]);
            pInterface->hTx[txChIdIndex]->hLwip2Enet = hLwip2Enet;
            Lwip2Enet_assert(NULL != pInterface->hTx[txChIdIndex]->hCh);
            if (pInterface->hTx[txChIdIndex]->refCount == 1)
            {
                hLwip2Enet->allocPktInfo += pInterface->hTx[txChIdIndex]->numPackets;
                EnetDma_disableTxEvent(pInterface->hTx[txChIdIndex]->hCh);
            }
        }
    
        //EnetDma_disableTxEvent
        pInterface->count_hTx = txChIdCount;
    
        /* Associate with corresponding Rx Channels */
        for (uint32_t rxChIdIdx = 0; rxChIdIdx < rxChIdCount; rxChIdIdx++)
        {
            const uint32_t rxChId = rxChIdList[rxChIdIdx];
            pInterface->hRx[rxChIdIdx] = Lwip2Enet_allocateRxHandle(hLwip2Enet, enetType, instId, rxChId);
            Lwip2Enet_initRxObj(enetType, instId, rxChId, pInterface->hRx[rxChIdIdx]);
    
            /* Process netif related parameters*/
            pInterface->hRx[rxChIdIdx]->mode = LwipifEnetAppCb_getRxMode(enetType, instId);
            if ((pInterface->hRx[rxChIdIdx]->mode == Lwip2Enet_RxMode_SwitchSharedChannel) ||
                (pInterface->hRx[rxChIdIdx]->mode == Lwip2Enet_RxMode_SwitchPort1Channel) ||
                (pInterface->hRx[rxChIdIdx]->mode == Lwip2Enet_RxMode_SwitchPort2Channel))
            {
                for (uint32_t portIdx = 0; portIdx < CPSW_STATS_MACPORT_MAX; portIdx++)
                {
                    pInterface->hRx[rxChIdIdx]->mapPortToNetif[portIdx] = netif;
                }
                pInterface->macPort = ENET_MAC_PORT_INV;
            }
            else if (pInterface->hRx[rxChIdIdx]->mode == Lwip2Enet_RxMode_MacSharedChannel)
            {
                const Enet_MacPort macPort = (Enet_MacPort)((pInterface->hRx[rxChIdIdx]->refCount -1));
                Lwip2Enet_assert(macPort < LWIPIF_MAX_NUM_MAC_PORTS);
                pInterface->hRx[rxChIdIdx]->mapPortToNetif[macPort] = netif;
                pInterface->macPort = macPort;
            }
            else
            {
                /* rxChIdx is treated as MAC PORT in case of dedicated CH per netif.
                 * This is in both ICSSG dual mac and switch usecase */
                const Enet_MacPort macPort = Lwip2Enet_findMacPortFromEnet(enetType, instId);
                Lwip2Enet_assert(macPort < LWIPIF_MAX_NUM_MAC_PORTS);
                pInterface->hRx[rxChIdIdx]->mapPortToNetif[macPort] = netif;
                pInterface->macPort = macPort;
            }
    
            // MAC address allocation for the Netifs
            EnetApp_getMacAddress(rxChId, &macAddr);
            if (macAddr.macAddressCnt > 0)
            {
                /* fall here only for the Rx Channel that has valid mac address */
                Lwip2Enet_assert(netif->hwaddr_len == 0);
                Lwip2Enet_assert(macAddr.macAddressCnt >= (pInterface->hRx[rxChIdIdx]->refCount - 1));
                EnetUtils_copyMacAddr(netif->hwaddr ,&macAddr.macAddr[pInterface->hRx[rxChIdIdx]->refCount - 1][0U]);
                netif->hwaddr_len = ENET_MAC_ADDR_LEN;
                #if ENET_ENABLE_PER_ICSSG
                Lwip2Enet_setMacAddress(enetType, instId, pInterface->hEnet, &macAddr.macAddr[pInterface->hRx[rxChIdIdx]->refCount - 1][0U]);
                #endif // ENET_ENABLE_PER_ICSSG
            }
    
            pInterface->hRx[rxChIdIdx]->hLwip2Enet = hLwip2Enet;
            Lwip2Enet_assert(NULL != pInterface->hRx[rxChIdIdx]->hFlow);
            /* Submit all allocated packets to DMA so it can use for packet RX */
            if (pInterface->hRx[rxChIdIdx]->refCount == 1)
            {
                hLwip2Enet->allocPktInfo += pInterface->hRx[rxChIdIdx]->numPackets;
                Lwip2Enet_submitRxPktQ(pInterface->hRx[rxChIdIdx]);
            }
        }
        pInterface->count_hRx = rxChIdCount;
    
        /* Get initial link/interface status from the driver */
        pInterface->isLinkUp = hLwip2Enet->appInfo.isPortLinkedFxn(pInterface->hEnet);
        pInterface->isPortLinkedFxn = hLwip2Enet->appInfo.isPortLinkedFxn;
        pInterface->pNetif   = netif;
    
        /* Updating the netif params */
        Lwip2Enet_assert(netif->hwaddr_len == ENET_MAC_ADDR_LEN);
        netif->state = (void *)pInterface;
    
        /* assert if clk period is not valid  */
        Lwip2Enet_assert(0U != hLwip2Enet->appInfo.timerPeriodUs);
    
        ClockP_start(&hLwip2Enet->pacingClkObj);
        return hLwip2Enet;
    }
    
    
    Lwip2Enet_netif_t* Lwip2Enet_getInterfaceObj(Lwip2Enet_Handle hLwip2Enet, struct netif *netif)
    {
        Lwip2Enet_netif_t* pInterface = NULL;
        for (uint32_t ifIdx = 0; ifIdx < LWIPIF_MAX_NETIFS_SUPPORTED; ifIdx++)
        {
            if (hLwip2Enet->interfaceInfo[ifIdx].pNetif == netif)
            {
                pInterface = &hLwip2Enet->interfaceInfo[ifIdx];
                break;
            }
            else
            {
                continue;
            }
        }
        return pInterface;
    }
    
    
    /*!
     *  @b Lwip2Enet_close
     *  @n
     *      Closes Ethernet peripheral and disables interrupts.
     *
     *  \param[in]  hLwip2Enet
     *      Lwip2Enet_object structure pointer.
     *
     *  \retval
     *      void
     */
    void Lwip2Enet_close(Lwip2Enet_Handle hLwip2Enet, struct netif *netif)
    {
        Lwip2Enet_assert(NULL != hLwip2Enet);
    
        /* Stop and delete the tick timer */
        ClockP_stop(&hLwip2Enet->pacingClkObj);
        ClockP_destruct(&hLwip2Enet->pacingClkObj);
    
        Lwip2Enet_netif_t* pInterface = Lwip2Enet_getInterfaceObj(hLwip2Enet, netif);
        Lwip2Enet_assert(pInterface != NULL);
        Lwip2Enet_assert(pInterface->count_hTx <= LWIPIF_MAX_TX_CHANNELS_PER_PHERIPHERAL);
        Lwip2Enet_assert(pInterface->count_hRx <= LWIPIF_MAX_RX_CHANNELS_PER_PHERIPHERAL);
    
    //    Lwip2Enet_stopRxTx(pInterface->hTx, pInterface->hRx);
    
        if (hLwip2Enet->allocPktInfo > 0U)
        {
            Lwip2Enet_print(hLwip2Enet, "Lwip2Enet_close() failed to retrieve all PktInfo\n");
        }
    
        /* Deinit TX and RX objects, delete task, semaphore, etc */
        for (uint32_t idx = 0; idx < pInterface->count_hTx; idx++)
        {
            Lwip2Enet_deinitTxObj(pInterface->hEnet->enetPer->enetType, pInterface->hEnet->enetPer->instId, pInterface->hTx[idx]);
            hLwip2Enet->allocPktInfo -= EnetQueue_getQCount(&pInterface->hTx[idx]->freePktInfoQ);
            pInterface->hTx[idx] = NULL;
        }
    
        for (uint32_t idx = 0; idx < pInterface->count_hTx; idx++)
        {
            Lwip2Enet_deinitRxObj(pInterface->hEnet->enetPer->enetType, pInterface->hEnet->enetPer->instId, pInterface->hRx[idx]);
    
            hLwip2Enet->allocPktInfo -= EnetQueue_getQCount(&pInterface->hRx[idx]->readyRxPktQ);
            hLwip2Enet->allocPktInfo -= EnetQueue_getQCount(&pInterface->hRx[idx]->freeRxPktInfoQ);
            pInterface->hRx[idx] = NULL;
        }
        pInterface->isActive = false;
        return;
    }
    
    static void Lwip2Enet_initRxObj(Enet_Type enetType, uint32_t instId, uint32_t chEntryIdx, Lwip2Enet_RxHandle hRx)
    {
    
        if (hRx->refCount > 0)
        {
            hRx->refCount++;
        }
        else
        {
            LwipifEnetAppIf_GetRxHandleInArgs inArgs;
            LwipifEnetAppIf_RxHandleInfo outArgs = {0};
    
            inArgs.enetType        = enetType;
            inArgs.instId          = instId;
            inArgs.cbArg           = hRx;
            inArgs.notifyCb        = &Lwip2Enet_notifyRxPackets;
            inArgs.chId            = chEntryIdx;
            inArgs.pFreePbufInfoQ  = &hRx->freePbufInfoQ;
            inArgs.pReadyRxPktQ    = &hRx->readyRxPktQ;
            inArgs.pFreeRxPktInfoQ = &hRx->freeRxPktInfoQ;
    
            LwipifEnetAppCb_getRxHandleInfo(&inArgs, &outArgs);
    
            hRx->numPackets = outArgs.numPackets;
            hRx->flowIdx     = outArgs.rxFlowIdx;
            hRx->flowStartIdx = outArgs.rxFlowStartIdx;
            hRx->hFlow       = outArgs.hRxFlow;
            Lwip2Enet_assert(hRx->hFlow != NULL);
            hRx->disableEvent = outArgs.disableEvent;
    
            hRx->stats.freeAppPktEnq = outArgs.numPackets;
    
            hRx->refCount = 1U;
            hRx->chEntryIdx = chEntryIdx;
            for (uint32_t portIdx = 0; portIdx < CPSW_STATS_MACPORT_MAX; portIdx++)
            {
                hRx->mapPortToNetif[portIdx] = NULL;
            }
        }
    
        return;
    }
    
    static void Lwip2Enet_saveAppIfCfg(Lwip2Enet_Handle hLwip2Enet,
                                         Lwip2Enet_AppInfo *appInfo)
    {
    
    }
    
    static void Lwip2Enet_deinitRxObj(Enet_Type enetType, uint32_t instId, Lwip2Enet_RxHandle hRx)
    {
        if (Lwip2Enet_decrementRxRefCount(hRx))
        {
            LwipifEnetAppIf_ReleaseRxHandleInfo inArgs;
            inArgs.enetType        = enetType;
            inArgs.instId          = instId;
            inArgs.rxFreePktCb    = &Lwip2Enet_freeRxPktCb;
            inArgs.rxFreePktCbArg = hRx;
            inArgs.rxChNum        = hRx->refCount;
            LwipifEnetAppCb_releaseRxHandle(&inArgs);
            hRx->hFlow = NULL;
        }
    
    }
    
    static void Lwip2Enet_deinitTxObj(Enet_Type enetType, uint32_t instId, Lwip2Enet_TxHandle hTx)
    {
        if (Lwip2Enet_decrementTxRefCount(hTx))
        {
            LwipifEnetAppIf_ReleaseTxHandleInfo inArgs;
            inArgs.enetType       = enetType;
            inArgs.instId         = instId;
            inArgs.txFreePktCb    = &Lwip2Enet_freeTxPktCb;
            inArgs.txFreePktCbArg = hTx;
            inArgs.txChNum        = hTx->refCount;
    
            LwipifEnetAppCb_releaseTxHandle(&inArgs);
            hTx->hCh = NULL;
    
            /* Free pbuf in ready queue */
            while (pbufQ_count(&hTx->readyPbufQ) != 0U)
            {
                struct pbuf *pbuf = pbufQ_deQ(&hTx->readyPbufQ);
                Lwip2Enet_assert(NULL != pbuf);
                pbuf_free(pbuf);
            }
        }
    }
    
    static void Lwip2Enet_initTxObj(Enet_Type enetType, uint32_t instId, uint32_t chEntryIdx, Lwip2Enet_TxObj* pTx)
    {
        if(pTx->refCount > 0)
        {
            pTx->refCount++;
        }
        else
        {
            LwipifEnetAppIf_GetTxHandleInArgs inArgs;
            LwipifEnetAppIf_TxHandleInfo outArgs;
    
            inArgs.chId = chEntryIdx;
            inArgs.cbArg = pTx;
            inArgs.notifyCb = Lwip2Enet_notifyTxPackets;
            inArgs.pktInfoQ = &pTx->freePktInfoQ;
            inArgs.enetType = enetType;
            inArgs.instId = instId;
            LwipifEnetAppCb_getTxHandleInfo(&inArgs, &outArgs);
    
            pTx->numPackets = outArgs.numPackets;
            pTx->hCh = outArgs.hTxChannel;
            Lwip2Enet_assert(pTx->hCh != NULL);
            pTx->disableEvent = outArgs.disableEvent;
    
            pTx->stats.freeAppPktEnq = outArgs.numPackets;
    
            /* Initialize the TX pbuf queues */
            pbufQ_init(&pTx->readyPbufQ);
            pbufQ_init(&pTx->unusedPbufQ);
    
            pTx->refCount = 1U;
            pTx->chEntryIdx = chEntryIdx;
        }
        return;
    }
    
    static void Lwip2Enet_setSGList(EnetDma_Pkt *pCurrDmaPacket, struct pbuf *pbuf, bool isRx)
    {
        struct pbuf *pbufNext = pbuf;
        uint32_t totalPacketFilledLen = 0U;
    
        pCurrDmaPacket->sgList.numScatterSegments = 0;
        while (pbufNext != NULL)
        {
            EnetDma_SGListEntry *list;
    
            Lwip2Enet_assert(pCurrDmaPacket->sgList.numScatterSegments < ENET_ARRAYSIZE(pCurrDmaPacket->sgList.list));
            list = &pCurrDmaPacket->sgList.list[pCurrDmaPacket->sgList.numScatterSegments];
            list->bufPtr = (uint8_t*) pbufNext->payload;
            list->segmentFilledLen = (isRx == true) ? 0U : pbufNext->len;
            list->segmentAllocLen = pbufNext->len;
            if ((pbufNext->type_internal == PBUF_ROM) || (pbufNext->type_internal == PBUF_REF))
            {
                list->disableCacheOps = true;
            }
            else
            {
                list->disableCacheOps = false;
            }
            totalPacketFilledLen += pbufNext->len;
            pCurrDmaPacket->sgList.numScatterSegments++;
            pbufNext = pbufNext->next;
        }
        Lwip2Enet_assert(totalPacketFilledLen == pbuf->tot_len);
    }
    /*!
     *  @b Lwip2Enet_sendTxPackets
     *  @n
     *      Routine to send out queued Tx packets to the hardware driver
     *
     *  \param[in]  hLwip2Enet
     *      Lwip2Enet_object structure pointer.
     *
     *  \retval
     *      void
     */
    void Lwip2Enet_sendTxPackets(Lwip2Enet_netif_t* pInterface, const Enet_MacPort macPort)
    {
        /* If link is not up, simply return */
        if (pInterface->isLinkUp)
        {
            EnetDma_PktQ txSubmitQ;
            Lwip2Enet_TxHandle hTx = pInterface->hTx[0];
            EnetQueue_initQ(&txSubmitQ);
    
            if (pbufQ_count(&hTx->unusedPbufQ))
            {
                /* send any pending TX Q's */
                Lwip2Enet_pbufQ2PktInfoQ(hTx, &hTx->unusedPbufQ, &txSubmitQ, macPort);
            }
    
            /* Check if there is anything to transmit, else simply return */
            while (pbufQ_count(&hTx->readyPbufQ) != 0U)
            {
                /* Dequeue one free TX Eth packet */
                EnetDma_Pkt *pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(&hTx->freePktInfoQ);
    
                if (pCurrDmaPacket == NULL)
                {
                    /* If we run out of packet info Q, retrieve packets from HW
                    * and try to dequeue free packet again */
                    Lwip2Enet_retrieveTxPkts(hTx);
                    pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(&hTx->freePktInfoQ);
                }
    
                if (NULL != pCurrDmaPacket)
                {
                    struct pbuf *hPbufPkt = pbufQ_deQ(&hTx->readyPbufQ);
                    EnetDma_initPktInfo(pCurrDmaPacket);
                    Lwip2Enet_assert(hPbufPkt);
                    Lwip2Enet_setSGList(pCurrDmaPacket, hPbufPkt, false);
                    pCurrDmaPacket->appPriv    = hPbufPkt;
                    pCurrDmaPacket->txPortNum  = macPort;
                    pCurrDmaPacket->node.next  = NULL;
                    pCurrDmaPacket->chkSumInfo = LWIPIF_LWIP_getChkSumInfo(hPbufPkt);
    
                    ENET_UTILS_COMPILETIME_ASSERT(offsetof(EnetDma_Pkt, node) == 0U);
                    EnetQueue_enq(&txSubmitQ, &(pCurrDmaPacket->node));
    
                    LWIP2ENETSTATS_ADDONE(&hTx->stats.freeAppPktDeq);
                    LWIP2ENETSTATS_ADDONE(&hTx->stats.readyPbufPktDeq);
                }
                else
                {
                    break;
                }
            }
    
            /* Submit the accumulated packets to the hardware for transmission */
            Lwip2Enet_submitTxPackets(hTx, &txSubmitQ);
        }
    }
    
    
    /*!
     *  @b Lwip2Enet_ioctl
     *  @n
     *  Low level driver Ioctl interface. This interface can be used for
     *  ALE configuration, control, statistics
     *
     *  \param[in]  hLwip2Enet
     *      Lwip2Enet_object structure pointer.
     *  \param[in]  cmd
     *      Ioctl command.
     *  \param[in]  pBuf
     *      Ioctl buffer with commands and params to set/get
     *      configuration from hardware.
     *  \param[in]  size
     *      Size of Ioctl buffer.
     *
     *  \retval
     *      void
     */
    int32_t Lwip2Enet_ioctl(Lwip2Enet_Handle hLwip2Enet,
                           uint32_t cmd,
                           void *param,
                           uint32_t size)
    {
    
        int32_t retVal = 0U;
    
        // TODO - make IOCTL design consistent with Enet/Enet modules
    
        /* Decode the command and act on it */
        switch (cmd)
        {
            /* Rx Task Load IOCTL */
            case LWIP2ENET_IOCTL_GET_RXTASK_LOAD:
            {
                break;
            }
    
            /* Adding Tx Task Load  */
            case LWIP2ENET_IOCTL_GET_TXTASK_LOAD:
            {
                break;
            }
    
            default:
            {
                Lwip2Enet_assert(FALSE);
            }
        }
    
        return retVal;
    }
    
    /*---------------------------------------------------------------------------*\
     |                           Local Function Definitions                        |
     \*---------------------------------------------------------------------------*/
    
     void Lwip2Enet_periodicFxn(struct netif* netif)
    {
        Lwip2Enet_netif_t* pInterface = (Lwip2Enet_netif_t*)netif->state;
    
        Lwip2Enet_assert(pInterface != NULL);
        Lwip2Enet_assert(pInterface->hRx != NULL);
        Lwip2Enet_assert(pInterface->hTx != NULL);
        uint32_t prevLinkState = pInterface->isLinkUp;
    
    #if (1U == ENET_CFG_DEV_ERROR)
    #if defined (ENET_SOC_HOSTPORT_DMA_TYPE_UDMA)
        int32_t status;
    #endif
    
        for(uint32_t i = 0U; i < pInterface->count_hTx; i++)
        {
            EnetQueue_verifyQCount(&pInterface->hTx[i]->freePktInfoQ);
    
    #if defined (ENET_SOC_HOSTPORT_DMA_TYPE_UDMA)
            status = EnetUdma_checkTxChSanity(pInterface->hTx[i]->hCh, 5U);
            if (status != ENET_SOK)
            {
                Lwip2Enet_print(pInterface->hTx[i]->hLwip2Enet, "EnetUdma_checkTxChSanity Failed\n");
            }
    #endif
        }
    
        for(uint32_t i = 0U; i <  pInterface->count_hRx; i++)
        {
            EnetQueue_verifyQCount(&pInterface->hRx[i]->readyRxPktQ);
    
    #if defined (ENET_SOC_HOSTPORT_DMA_TYPE_UDMA)
            status = EnetUdma_checkRxFlowSanity(pInterface->hRx[i]->hFlow, 5U);
            if (status != ENET_SOK)
            {
                Lwip2Enet_print(pInterface->hRx[i]->hLwip2Enet, "EnetUdma_checkRxFlowSanity Failed\n");
            }
    #endif
        }
    #endif
    
        /*
         * Return the same DMA packets back to the DMA channel (but now
         * associated with a new PBUF Packet and buffer)
         */
    
        for (uint32_t idx = 0; idx < pInterface->count_hRx; idx++)
        {
            if (EnetQueue_getQCount(&pInterface->hRx[idx]->readyRxPktQ) > 0U)
            {
                Lwip2Enet_submitRxPktQ(pInterface->hRx[idx]);
            }
        }
    
        /* Get current link status as reported by the hardware driver */
        pInterface->isLinkUp = pInterface->isPortLinkedFxn(pInterface->hEnet);
        for (uint32_t idx = 0; idx < pInterface->count_hRx; idx++)
        {
            const uint32_t prevLinkInterface = pInterface->hRx[idx]->hLwip2Enet->currLinkedIf;
            // get the linked interface details.
            /* If the interface changed, discard any queue packets (since the MAC would now be wrong) */
            if (prevLinkInterface != pInterface->hRx[idx]->hLwip2Enet->currLinkedIf)
            {
                /* ToDo: Discard all queued packets */
            }
        }
    
        /* If link status changed from down->up, then send any queued packets */
        if ((prevLinkState == 0U) && (pInterface->isLinkUp))
        {
            const Enet_MacPort macPort = pInterface->macPort;
            Lwip2Enet_sendTxPackets(pInterface, macPort);
        }
    
    
    #if 0 //The below CPU load profilling logic has to be re-worked for all OS variants
    #if defined(LWIPIF_INSTRUMENTATION_ENABLED)
        static uint32_t loadCount = 0U;
        TaskP_Load stat;
    
        hLwip2Enet->stats.cpuLoad[loadCount] = TaskP_loadGetTotalCpuLoad();
    
        TaskP_loadGet(&hLwip2Enet->rxPacketTaskObj, &stat);
        for(uint32_t i = 0U; i < hLwip2Enet->numRxChannels; i++)
        {
            hLwip2Enet->rx[i].stats.pktStats.taskLoad[loadCount] = stat.cpuLoad;
        }
    
        TaskP_loadGet(&hLwip2Enet->txPacketTaskObj, &stat);
        for(uint32_t i = 0U; i < hLwip2Enet->numRxChannels; i++)
        {
            hLwip2Enet->tx[i].stats.pktStats.taskLoad[loadCount] = stat.cpuLoad;
        }
    
        loadCount = (loadCount + 1U) & (HISTORY_CNT - 1U);
    #endif
    #endif
    
    
    }
    
    static void Lwip2Enet_processRxUnusedQ(Lwip2Enet_RxObj *rx,
                                           EnetDma_PktQ *unUsedQ)
    {
        EnetDma_Pkt *pCurrDmaPacket;
    
        pCurrDmaPacket = (EnetDma_Pkt*) EnetQueue_deq(unUsedQ);
        while (pCurrDmaPacket != NULL)
        {
            if ((pCurrDmaPacket->sgList.numScatterSegments <= 0)
                    || (pCurrDmaPacket->sgList.numScatterSegments > ENET_ARRAYSIZE(pCurrDmaPacket->sgList.list))
                    || (pCurrDmaPacket->sgList.list[0].bufPtr == NULL))
            {
                Lwip2Enet_assert(FALSE);
            }
            else
            {
                EnetQueue_enq(&rx->readyRxPktQ, &pCurrDmaPacket->node);
            }
    
            pCurrDmaPacket = (EnetDma_Pkt*) EnetQueue_deq(unUsedQ);
        }
    }
    
    static void Lwip2Enet_submitRxPackets(Lwip2Enet_RxObj *rx,
                                          EnetDma_PktQ *pSubmitQ)
    {
        int32_t retVal;
    
        retVal = EnetDma_submitRxPktQ(rx->hFlow, pSubmitQ);
        if (ENET_SOK != retVal)
        {
            Lwip2Enet_print(rx->hLwip2Enet,
                            "EnetDma_submitRxPktQ: failed to submit pkts: %d\n",
                            retVal);
        }
    
        if (EnetQueue_getQCount(pSubmitQ))
        {
            /* Copy unused packets to back to readyQ */
            Lwip2Enet_processRxUnusedQ(rx, pSubmitQ);
        }
    }
    
    /* May lead to infinite loop if no free memory
     * available*/
    static void Lwip2Enet_pbufQ2PktInfoQ(Lwip2Enet_TxObj *tx,
                                         pbufQ *pbufPktQ,
                                         EnetDma_PktQ *pDmaPktInfoQ,
                                         Enet_MacPort macPort)
    {
        EnetDma_Pkt *pCurrDmaPacket;
        struct pbuf *hPbufPkt = NULL;
    
        while (pbufQ_count(pbufPktQ) != 0U)
        {
    
            /* Dequeue one free TX Eth packet */
            pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(&tx->freePktInfoQ);
    
            if (pCurrDmaPacket == NULL)
            {
                /* If we run out of packet info Q, retrieve packets from HW
                * and try to dequeue free packet again */
                Lwip2Enet_retrieveTxPkts(tx);
                pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(&tx->freePktInfoQ);
            }
    
            if (NULL != pCurrDmaPacket)
            {
                hPbufPkt = pbufQ_deQ(pbufPktQ);
                Lwip2Enet_assert(hPbufPkt);
                EnetDma_initPktInfo(pCurrDmaPacket);
    
                Lwip2Enet_setSGList(pCurrDmaPacket, hPbufPkt, false);
                pCurrDmaPacket->appPriv    = hPbufPkt;
    
                pCurrDmaPacket->node.next = NULL;
                pCurrDmaPacket->txPortNum  = macPort;
                pCurrDmaPacket->chkSumInfo = LWIPIF_LWIP_getChkSumInfo(hPbufPkt);
    
                ENET_UTILS_COMPILETIME_ASSERT(offsetof(EnetDma_Pkt, node) == 0U);
                EnetQueue_enq(pDmaPktInfoQ, &(pCurrDmaPacket->node));
    
                LWIP2ENETSTATS_ADDONE(&tx->stats.freeAppPktDeq);
            }
            else
            {
                break;
            }
        }
    }
    
    static void Lwip2Enet_pktInfoQ2PbufQ(EnetDma_PktQ *pDmaPktInfoQ,
                                         pbufQ *pbufPktQ)
    {
        EnetDma_Pkt *pDmaPacket;
        struct pbuf *hPbufPacket;
    
        while (EnetQueue_getQCount(pDmaPktInfoQ) != 0U)
        {
            pDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(pDmaPktInfoQ);
            hPbufPacket = (struct pbuf *)(pDmaPacket->appPriv);
    
            Lwip2Enet_assert(hPbufPacket != NULL);
            /*Don't want to make a copy, since it would cause waste memory*/
            pbufQ_enQ(pbufPktQ, hPbufPacket);
        }
    }
    
    static void Lwip2Enet_submitTxPackets(Lwip2Enet_TxObj *tx,
                                          EnetDma_PktQ *pSubmitQ)
    {
        int32_t retVal;
    
        retVal = EnetDma_submitTxPktQ(tx->hCh, pSubmitQ);
        if (ENET_SOK != retVal)
        {
            Lwip2Enet_print(tx->hLwip2Enet,
                            "EnetDma_submitTxPktQ: failed to submit pkts: %d\n",
                            retVal);
        }
    
        if (EnetQueue_getQCount(pSubmitQ))
        {
            /* TODO: txUnUsedPBMPktQ is needed for packets that were not able to be
             *       submitted to driver.  It can be removed if stack supported any
             *       mechanism to enqueue them to the head of the queue. */
            Lwip2Enet_pktInfoQ2PbufQ(pSubmitQ, &tx->unusedPbufQ);
            EnetQueue_append(&tx->freePktInfoQ, pSubmitQ);
            LWIP2ENETSTATS_ADDNUM(&tx->stats.freeAppPktEnq, EnetQueue_getQCount(pSubmitQ));
        }
    }
    
    static void Lwip2Enet_freePbufPackets(EnetDma_PktQ *tempQueue)
    {
        EnetDma_Pkt *pCurrDmaPacket;
        struct pbuf* hPbufPacket;
    
        while (EnetQueue_getQCount(tempQueue))
        {
            pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(tempQueue);
            hPbufPacket     = (struct pbuf *)pCurrDmaPacket->appPriv;
            pbuf_free(hPbufPacket);
        }
    }
    
    static void Lwip2Enet_notifyRxPackets(void *cbArg)
    {
        Lwip2Enet_RxHandle pRx = (Lwip2Enet_RxHandle)cbArg;
    
        /* do not post events if init not done or shutdown in progress */
        if (pRx->refCount > 0)
        {
            if (pRx->disableEvent)
            {
                EnetDma_disableRxEvent(pRx->hFlow);
            }
            if (pRx->rxPktNotify.cbFxn != NULL)
            {
                pRx->rxPktNotify.cbFxn(pRx->rxPktNotify.cbArg);
            }
        }
    }
    
    static void Lwip2Enet_notifyTxPackets(void *cbArg)
    {
        Lwip2Enet_TxHandle pTx = (Lwip2Enet_TxHandle)cbArg;
    
        /* do not post events if init not done or shutdown in progress */
        if ((pTx->refCount > 0) && (pTx->txPktNotify.cbFxn != NULL))
        {
            /* Notify Callbacks to post event/semephore */
            pTx->txPktNotify.cbFxn(pTx->txPktNotify.cbArg);
        }
    }
    
    void Lwip2Enet_rxPktHandler(Lwip2Enet_RxHandle hRx)
    {
        EnetDma_PktQ tempQueue;
        int32_t retVal;
        uint32_t pktCnt = 0U;
    
        LWIP2ENETSTATS_ADDONE(&hRx->stats.pktStats.rawNotifyCnt);
    
        /* Retrieve the used (filled) packets from the channel */
        {
            EnetQueue_initQ(&tempQueue);
            retVal = EnetDma_retrieveRxPktQ(hRx->hFlow, &tempQueue);
            if (ENET_SOK != retVal)
            {
                Lwip2Enet_print(hRx->hLwip2Enet,
                        "Lwip2Enet_rxPacketTask: failed to retrieve RX pkts: %d\n",
                        retVal);
            }
        }
        if (tempQueue.count == 0U)
        {
            LWIP2ENETSTATS_ADDONE(&hRx->stats.pktStats.zeroNotifyCnt);
        }
    
        /*
         * Call Lwip2Enet_prepRxPktQ() even if no packets were received.
         * This allows new packets to be submitted if PBUF buffers became
         * newly available and there were outstanding free packets.
         */
        {
            /*
             * Get all used Rx DMA packets from the hardware, then send the buffers
             * of those packets on to the LwIP stack to be parsed/processed.
             */
            pktCnt = Lwip2Enet_prepRxPktQ(hRx, &tempQueue);
        }
    
        /*
         * We don't want to time the semaphore post used to notify the LwIP stack as that may cause a
         * task transition. We don't want to time the semaphore pend, since that would time us doing
         * nothing but waiting.
         */
        if (pktCnt != 0U)
        {
            Lwip2Enet_updateRxNotifyStats(&hRx->stats.pktStats, pktCnt, 0U);
        }
    
        // ClockP_start(&hLwip2Enet->pacingClkObj);
    
        if (!hRx->disableEvent)
        {
            EnetDma_enableRxEvent(hRx->hFlow);
        }
    
    }
    
    void Lwip2Enet_txPktHandler(Lwip2Enet_TxHandle hTx)
    {
        Lwip2Enet_retrieveTxPkts(hTx);
    }
    
    /*
     * Enqueue a new packet and make sure that buffer descriptors are properly linked.
     * NOTE: Not thread safe
     */
    static void Lwip2Enet_submitRxPktQ(Lwip2Enet_RxObj *rx)
    {
        EnetDma_PktQ resubmitPktQ;
        EnetDma_Pkt *pCurrDmaPacket = NULL;
    
        EnetQueue_initQ(&resubmitPktQ);
        pCurrDmaPacket = (EnetDma_Pkt*) EnetQueue_deq(&rx->readyRxPktQ);
        while (pCurrDmaPacket != NULL)
        {
            EnetDma_checkPktState(&pCurrDmaPacket->pktState,
                                   ENET_PKTSTATE_MODULE_APP,
                                   ENET_PKTSTATE_APP_WITH_READYQ,
                                   ENET_PKTSTATE_APP_WITH_DRIVER);
            if ((pCurrDmaPacket->sgList.numScatterSegments <= 0)
                    || (pCurrDmaPacket->sgList.numScatterSegments > ENET_ARRAYSIZE(pCurrDmaPacket->sgList.list))
                    || (pCurrDmaPacket->sgList.list[0].bufPtr == NULL))
            {
                Lwip2Enet_assert(FALSE);
            }
            else
            {
                EnetQueue_enq(&resubmitPktQ, &pCurrDmaPacket->node);
            }
    
            pCurrDmaPacket = (EnetDma_Pkt*) EnetQueue_deq(&rx->readyRxPktQ);
        }
        if (EnetQueue_getQCount(&resubmitPktQ))
        {
            Lwip2Enet_submitRxPackets(rx, &resubmitPktQ);
        }
    
    }
    static inline struct pbuf * Lwip2Enet_setCustomPbuf(pbuf_layer l, u16_t length, pbuf_type type, struct pbuf_custom *p,
            void *payload_mem, u16_t payload_mem_len)
    {
        return pbuf_alloced_custom(l, length, type, p, payload_mem, payload_mem_len);
    }
    
    static uint32_t Lwip2Enet_prepRxPktQ(Lwip2Enet_RxObj *rx,
                                         EnetDma_PktQ *pPktQ)
    {
        uint32_t packetCount = 0U;
        EnetDma_Pkt *pCurrDmaPacket;
        bool isChksumError = false;
        uint32_t scatterSegmentIndex = 0U;
        EnetDma_SGListEntry *list = NULL;
        struct pbuf *hPbufPacket = NULL;
        Rx_CustomPbuf *cPbuf = NULL;
        Rx_CustomPbuf *cPbufPrev = NULL;
    
        pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(pPktQ);
        while (pCurrDmaPacket)
        {
            isChksumError = false;
            Lwip2Enet_assert(pCurrDmaPacket->sgList.numScatterSegments <= ENET_ARRAYSIZE(pCurrDmaPacket->sgList.list));
            Lwip2Enet_assert(pCurrDmaPacket->sgList.numScatterSegments != 0);
            if (pbufQ_count(&rx->freePbufInfoQ) >= pCurrDmaPacket->sgList.numScatterSegments)
            {
                EnetDma_checkPktState(&pCurrDmaPacket->pktState,
                                      ENET_PKTSTATE_MODULE_APP,
                                      ENET_PKTSTATE_APP_WITH_DRIVER,
                                      ENET_PKTSTATE_APP_WITH_FREEQ);
                scatterSegmentIndex = 0;
                Enet_MacPort rxPortNum = pCurrDmaPacket->rxPortNum;
                while (scatterSegmentIndex < pCurrDmaPacket->sgList.numScatterSegments)
                {
                    list = &pCurrDmaPacket->sgList.list[scatterSegmentIndex];
                    Lwip2Enet_assert(list->bufPtr != NULL);
    
                    cPbuf  = (Rx_CustomPbuf *)pbufQ_deQ(&rx->freePbufInfoQ);
                    LWIP2ENETSTATS_ADDONE(&rx->stats.freePbufPktDeq);
                    if (scatterSegmentIndex == 0)
                    {
                        /* store the head of the pbuf */
                        hPbufPacket = &(cPbuf->p.pbuf);
                    }
                    Lwip2Enet_assert(hPbufPacket != NULL);
                    /* Fill the pbuf with the sg list data */
                    if (Lwip2Enet_setCustomPbuf(PBUF_RAW, list->segmentFilledLen, PBUF_POOL, &(cPbuf->p), list->bufPtr, list->segmentAllocLen) == NULL)
                    {
                        Lwip2Enet_assert(false);
                    }
                    cPbuf->orgBufLen = list->segmentAllocLen;
                    cPbuf->orgBufPtr = list->bufPtr;
                    cPbuf->alivePbufCount = pCurrDmaPacket->sgList.numScatterSegments;
                    if (scatterSegmentIndex > 0)
                    {
                        cPbufPrev->next = cPbuf;
                        pbuf_cat(hPbufPacket, &(cPbuf->p.pbuf));
                    }
                    if (scatterSegmentIndex == pCurrDmaPacket->sgList.numScatterSegments - 1)
                    {
                        /* Make sure the next pointer of last pbuf is NULL */
                        cPbuf->p.pbuf.next = NULL;
                        /* Point the last to first. If there is only one segment, it points to itself. */
                        cPbuf->next = (Rx_CustomPbuf *) hPbufPacket;
                    }
                    cPbufPrev = cPbuf;
                    scatterSegmentIndex++;
                }
    
    
                {
                    Lwip2Enet_assert(hPbufPacket->payload != NULL);
                    struct ip_hdr* pIpPkt = (struct ip_hdr* ) LWIPIF_LWIP_getIpPktStart((uint8_t*) hPbufPacket->payload);
                    if (IPH_PROTO(pIpPkt) == IP_PROTO_UDPLITE)
                    {
                        /* As HW is can not compute checksum offload for UDP-lite packet, trigger SW checksum validation */
                        isChksumError = LWIPIF_LWIP_UdpLiteValidateChkSum(hPbufPacket);
                    }
                    else
                    {
                        /* We don't check if HW checksum offload is enabled while checking for checksum error
                         * as default value of this field when offload not enabled is false */
                        const uint32_t csumInfo =  pCurrDmaPacket->chkSumInfo;
    
                        if (ENETDMA_RXCSUMINFO_GET_IPV4_FLAG(csumInfo) ||
                            ENETDMA_RXCSUMINFO_GET_IPV6_FLAG(csumInfo))
                        {
                            isChksumError = ENETDMA_RXCSUMINFO_GET_CHKSUM_ERR_FLAG(csumInfo);
                        }
                    }
                }
    
                EnetDma_initPktInfo(pCurrDmaPacket);
                EnetQueue_enq(&rx->freeRxPktInfoQ, &pCurrDmaPacket->node);
                LWIP2ENETSTATS_ADDONE(&rx->stats.freeAppPktEnq);
    
                if (!isChksumError)
                {
                    struct netif* netif = NULL;
                    /* Pass the received packet to the LwIP stack */
                    switch (rx->mode)
                    {
                    case Lwip2Enet_RxMode_SwitchSharedChannel:
                    case Lwip2Enet_RxMode_MacSharedChannel:
                    {
                        Lwip2Enet_assert(rxPortNum < LWIPIF_MAX_NUM_MAC_PORTS);
                        netif = rx->mapPortToNetif[rxPortNum];
                        break;
                    }
                    case Lwip2Enet_RxMode_MacPort1Channel:
                    case Lwip2Enet_RxMode_SwitchPort1Channel:
                    {
                        netif = rx->mapPortToNetif[ENET_MAC_PORT_1];
                        break;
                    }
                    case Lwip2Enet_RxMode_MacPort2Channel:
                    case Lwip2Enet_RxMode_SwitchPort2Channel:
                    {
                        Lwip2Enet_assert(LWIPIF_MAX_NUM_MAC_PORTS == 2);
                        netif = rx->mapPortToNetif[LWIPIF_MAX_NUM_MAC_PORTS - 1];
                        break;
                    }
                    default:
                    {
                        Lwip2Enet_assert(false);
                    }
                    }
    
                    Lwip2Enet_assert(netif != NULL);
                    LWIPIF_LWIP_input(rx, netif, hPbufPacket);
                    packetCount++;
                }
                else
                {
                    /* Free the pbuf as we are not submitting to the stack */
                    pbuf_free(hPbufPacket);
                    LWIP2ENETSTATS_ADDONE(&rx->stats.chkSumErr);
                }
            }
            else
            {
                /*! This should not occur as we have equal number of pbufInfos and bufptrs */
                Lwip2Enet_assert(FALSE);
            }
            pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(pPktQ);
        }
    
        /* return as many packets to driver as we can */
        Lwip2Enet_submitRxPktQ(rx);
    
        return packetCount;
    }
    
    static uint32_t Lwip2Enet_prepTxPktQ(Lwip2Enet_TxObj *tx,
                                         EnetDma_PktQ *pPktQ)
    {
        uint32_t packetCount;
        EnetDma_Pkt *pCurrDmaPacket;
        struct pbuf* hPbufPacket;
    
        packetCount = EnetQueue_getQCount(pPktQ);
    
        pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(pPktQ);
        while (pCurrDmaPacket)
        {
            hPbufPacket = (struct pbuf *)pCurrDmaPacket->appPriv;
    
            Lwip2Enet_assert(hPbufPacket != NULL);
            /* Free PBUF buffer as it is transmitted by DMA now */
            pbuf_free(hPbufPacket);
            /* Return packet info to free pool */
            EnetQueue_enq(&tx->freePktInfoQ, &pCurrDmaPacket->node);
            pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(pPktQ);
        }
    
        LWIP2ENETSTATS_ADDNUM(&tx->stats.freeAppPktEnq, packetCount);
    
        return packetCount;
    }
    
    static void Lwip2Enet_updateTxNotifyStats(Lwip2Enet_PktTaskStats *pktStats,
                                              uint32_t packetCount,
                                              uint32_t timeDiff)
    {
    #if defined(LWIPIF_INSTRUMENTATION_ENABLED)
        uint32_t notificationCount;
        uint32_t timePerPacket = timeDiff / packetCount;
    
        notificationCount = pktStats->dataNotifyCnt & (HISTORY_CNT - 1U);
        pktStats->dataNotifyCnt++;
    
        pktStats->totalPktCnt   += packetCount;
        pktStats->totalCycleCnt += timeDiff;
    
        pktStats->cycleCntPerNotify[notificationCount] = timeDiff;
        if (timeDiff > pktStats->cycleCntPerNotifyMax)
        {
            pktStats->cycleCntPerNotifyMax = timeDiff;
        }
    
        pktStats->pktsPerNotify[notificationCount] = packetCount;
        if (packetCount > pktStats->pktsPerNotifyMax)
        {
            pktStats->pktsPerNotifyMax = packetCount;
        }
    
        pktStats->cycleCntPerPkt[notificationCount] = timePerPacket;
        if (timePerPacket > pktStats->cycleCntPerPktMax)
        {
            pktStats->cycleCntPerPktMax = timePerPacket;
        }
    #endif
    }
    
    static void Lwip2Enet_updateRxNotifyStats(Lwip2Enet_PktTaskStats *pktTaskStats,
                                              uint32_t packetCount,
                                              uint32_t timeDiff)
    {
    #if defined(LWIPIF_INSTRUMENTATION_ENABLED)
        uint32_t notificationCount;
        uint32_t timePerPacket = timeDiff / packetCount;
    
        notificationCount = pktTaskStats->dataNotifyCnt & (HISTORY_CNT - 1U);
        pktTaskStats->dataNotifyCnt++;
    
        pktTaskStats->totalPktCnt   += packetCount;
        pktTaskStats->totalCycleCnt += timeDiff;
    
        pktTaskStats->cycleCntPerNotify[notificationCount] = timeDiff;
        if (timeDiff > pktTaskStats->cycleCntPerNotifyMax)
        {
            pktTaskStats->cycleCntPerNotifyMax = timeDiff;
        }
    
        pktTaskStats->pktsPerNotify[notificationCount] = packetCount;
        if (packetCount > pktTaskStats->pktsPerNotifyMax)
        {
            pktTaskStats->pktsPerNotifyMax = packetCount;
        }
    
        pktTaskStats->cycleCntPerPkt[notificationCount] = timePerPacket;
        if (timePerPacket > pktTaskStats->cycleCntPerPktMax)
        {
            pktTaskStats->cycleCntPerPktMax = timePerPacket;
        }
    #endif
    }
    
    static void Lwip2Enet_print(Lwip2Enet_Handle hLwip2Enet,
                                const char *prnStr,
                                ...)
    {
        // TODO fix me
        if (NULL != hLwip2Enet->print)
        {
            /* Function is non- reentrant */
            va_list vaArgPtr;
            char *buf;
    
            buf = &hLwip2Enet->printBuf[0U];
            va_start(vaArgPtr, prnStr);
            vsnprintf(buf, ENET_CFG_PRINT_BUF_LEN, (const char *)prnStr, vaArgPtr);
            va_end(vaArgPtr);
    
            (*hLwip2Enet->print)("[LWIP2ENET] ");
            (*hLwip2Enet->print)(hLwip2Enet->printBuf);
        }
    }
    
    /* ========================================================================== */
    /*                          Function Definitions                              */
    /* ========================================================================== */
    
    bool Lwip2Enet_decrementRxRefCount(Lwip2Enet_RxObj* pRx)
    {
        if (pRx->refCount > 0)
        {
            pRx->refCount--;
        }
    
        return (pRx->refCount > 0);
    }
    
    bool Lwip2Enet_decrementTxRefCount(Lwip2Enet_TxObj* pTx)
    {
        if (pTx->refCount > 0)
        {
            pTx->refCount--;
        }
    
        return (pTx->refCount > 0);
    
    }
    
    /*
     * Periodically polls for changes in the link status and updates both the abstraction layer
     * as well as the stack
     * arg : netif
     * arg1 : Semaphore
     */
    
    
    static void Lwip2Enet_stopRxTx(Lwip2Enet_TxObj* pTx, Lwip2Enet_RxObj* pRx)
    {
    
        /* Stop TX packet task */
        /* Post to tx packet task/event so that it will terminate (shutDownFlag flag is already set) */
        if (pTx->txPktNotify.cbFxn != NULL)
        {
            pTx->txPktNotify.cbFxn(pTx->txPktNotify.cbArg);
        }
    
        /* Stop RX packet task */
        /* Post to rx packet task/event so that it will terminate (shutDownFlag flag is already set) */
        if (pRx->rxPktNotify.cbFxn != NULL)
        {
            pRx->rxPktNotify.cbFxn(pRx->rxPktNotify.cbArg);
        }
    
    }
    
    static void Lwip2Enet_freeTxPktCb(void *cbArg,
                                      EnetDma_PktQ *fqPktInfoQ,
                                      EnetDma_PktQ *cqPktInfoQ)
    {
        Lwip2Enet_TxObj *tx = (Lwip2Enet_TxObj *)cbArg;
    
        EnetQueue_append(&tx->freePktInfoQ, fqPktInfoQ);
        LWIP2ENETSTATS_ADDNUM(&tx->stats.freeAppPktEnq, EnetQueue_getQCount(fqPktInfoQ));
        Lwip2Enet_freePbufPackets(fqPktInfoQ);
    
        EnetQueue_append(&tx->freePktInfoQ, cqPktInfoQ);
        LWIP2ENETSTATS_ADDNUM(&tx->stats.freeAppPktEnq, EnetQueue_getQCount(cqPktInfoQ));
        Lwip2Enet_freePbufPackets(cqPktInfoQ);
    }
    
    static void Lwip2Enet_freeRxPktCb(void *cbArg,
                                      EnetDma_PktQ *fqPktInfoQ,
                                      EnetDma_PktQ *cqPktInfoQ)
    {
        Lwip2Enet_RxObj *rx = (Lwip2Enet_RxObj *)cbArg;
    
        /* Now that we free PBUF buffers, push all freed pktInfo's into Rx
         * free Q */
        EnetQueue_append(&rx->readyRxPktQ, fqPktInfoQ);
        LWIP2ENETSTATS_ADDNUM(&rx->stats.freeAppPktEnq, EnetQueue_getQCount(fqPktInfoQ));
    
        EnetQueue_append(&rx->readyRxPktQ, cqPktInfoQ);
        LWIP2ENETSTATS_ADDNUM(&rx->stats.freeAppPktEnq, EnetQueue_getQCount(cqPktInfoQ));
    }
    
    void Lwip2Enet_retrieveTxPkts(Lwip2Enet_TxHandle hTx)
    {
        EnetDma_PktQ tempQueue;
        uint32_t packetCount = 0U;
        int32_t retVal;
    
        LWIP2ENETSTATS_ADDONE(&hTx->stats.pktStats.rawNotifyCnt);
        packetCount = 0U;
    
        /* Retrieve the used (sent/empty) packets from the channel */
        {
            EnetQueue_initQ(&tempQueue);
            /* Retrieve all TX packets and keep them locally */
            retVal = EnetDma_retrieveTxPktQ(hTx->hCh, &tempQueue);
            if (ENET_SOK != retVal)
            {
                Lwip2Enet_print(hTx->hLwip2Enet,
                                "Lwip2Enet_retrieveTxPkts: Failed to retrieve TX pkts: %d\n",
                                retVal);
            }
        }
    
        if (tempQueue.count != 0U)
        {
            /*
             * Get all used Tx DMA packets from the hardware, then return those
             * buffers to the txFreePktQ so they can be used later to send with.
             */
            packetCount = Lwip2Enet_prepTxPktQ(hTx, &tempQueue);
        }
        else
        {
            LWIP2ENETSTATS_ADDONE(&hTx->stats.pktStats.zeroNotifyCnt);
        }
    
        if (packetCount != 0U)
        {
            Lwip2Enet_updateTxNotifyStats(&hTx->stats.pktStats, packetCount, 0U);
        }
    }
    
    static void Lwip2Enet_timerCb(ClockP_Object *hClk, void * arg)
    {
    #if defined (ENET_SOC_HOSTPORT_DMA_TYPE_UDMA)
        /* Post semaphore to Rx and Tx Pkt handling task */
        Lwip2Enet_Handle hLwip2Enet = (Lwip2Enet_Handle)arg;
    
        if (hLwip2Enet->isInitDone)
        {
            for (uint32_t txChIdx = 0U; txChIdx < LWIPIF_MAX_TX_CHANNELS; txChIdx++)
            {
                if ((hLwip2Enet->lwip2EnetTxObj[txChIdx].refCount > 0) && (hLwip2Enet->lwip2EnetTxObj[txChIdx].txPktNotify.cbFxn != NULL))
                {
                    hLwip2Enet->lwip2EnetTxObj[txChIdx].txPktNotify.cbFxn(hLwip2Enet->lwip2EnetTxObj[txChIdx].txPktNotify.cbArg);
                }
            }
    
            for (uint32_t rxChIdx = 0U; rxChIdx < LWIPIF_MAX_RX_CHANNELS; rxChIdx++)
            {
                    if ((hLwip2Enet->lwip2EnetRxObj[rxChIdx].refCount > 0) && (hLwip2Enet->lwip2EnetRxObj[rxChIdx].rxPktNotify.cbFxn != NULL))
                    {
                        hLwip2Enet->lwip2EnetRxObj[rxChIdx].rxPktNotify.cbFxn(hLwip2Enet->lwip2EnetRxObj[rxChIdx].rxPktNotify.cbArg);
                    }
            }
    
        }
    #endif
    }
    
    static void Lwip2Enet_createTimer(Lwip2Enet_Handle hLwip2Enet)
    {
        ClockP_Params clkPrms;
        int32_t status;
    
        ClockP_Params_init(&clkPrms);
        clkPrms.start  = false;
        clkPrms.timeout = ClockP_usecToTicks(hLwip2Enet->appInfo.timerPeriodUs);
        //clkPrms.period = ClockP_usecToTicks(hLwip2Enet->appInfo.timerPeriodUs);
    
        clkPrms.period = 0;
        clkPrms.callback = &Lwip2Enet_timerCb;
        clkPrms.args = hLwip2Enet;
    
        status =  ClockP_construct(&hLwip2Enet->pacingClkObj, &clkPrms);
        Lwip2Enet_assert(status == SystemP_SUCCESS);
    }
    
    #if ENET_ENABLE_PER_ICSSG
    static int32_t Lwip2Enet_setMacAddress(const Enet_Type enetType, uint32_t instId, Enet_Handle hEnet, uint8_t macAddr[ENET_MAC_ADDR_LEN])
    {
        int32_t  status = ENET_SOK;
        uint32_t coreId = EnetSoc_getCoreId();
        /* Add port MAC entry in case of ICSSG dual MAC */
        if (ENET_ICSSG_DUALMAC == enetType)
        {
            Enet_IoctlPrms prms;
            IcssgMacPort_SetMacAddressInArgs inArgs;
    
            EnetUtils_copyMacAddr(&inArgs.macAddr[0U], &macAddr[0U]);
            inArgs.macPort = Lwip2Enet_findMacPortFromEnet(enetType, instId);
    
            ENET_IOCTL_SET_IN_ARGS(&prms, &inArgs);
            ENET_IOCTL(hEnet, coreId, ICSSG_MACPORT_IOCTL_SET_MACADDR, &prms, status);
    
            if (status != ENET_SOK)
            {
                EnetAppUtils_print(
                        "Lwip2Enet_setMacAddress() failed ICSSG_MACPORT_IOCTL_ADD_INTERFACE_MACADDR: %d\r\n",
                        status);
            }
            EnetAppUtils_assert(status == ENET_SOK);
        }
        else if (ENET_ICSSG_SWITCH == enetType)
        {
            Enet_IoctlPrms prms;
            Icssg_MacAddr addr;
    
            /* Set host port's MAC address */
            EnetUtils_copyMacAddr(&addr.macAddr[0U], &macAddr[0U]);
            ENET_IOCTL_SET_IN_ARGS(&prms, &addr);
            {
                ENET_IOCTL(hEnet, coreId, ICSSG_HOSTPORT_IOCTL_SET_MACADDR, &prms,
                        status);
                if (status != ENET_SOK)
                {
                    EnetAppUtils_print(
                            "EnetAppUtils_addHostPortEntry() failed ICSSG_HOSTPORT_IOCTL_SET_MACADDR: %d\r\n",
                            status);
                }
                EnetAppUtils_assert(status == ENET_SOK);
            }
        }
        else
        {
            EnetAppUtils_assert(false);
        }
        return status;
    }
    #endif
    
    void Lwip2Enet_setRxNotifyCallback(Lwip2Enet_RxHandle hRx, Enet_notify_t *pRxPktNotify)
    {
        hRx->rxPktNotify = *pRxPktNotify;
    }
    
    void Lwip2Enet_setTxNotifyCallback(Lwip2Enet_TxHandle hTx, Enet_notify_t *pTxPktNotify)
    {
        hTx->txPktNotify = *pTxPktNotify;
    }
    

    //###########################################################################
    //
    // FILE:   lwip2lwipif.c
    //
    // TITLE:  lwIP Interface port file.
    //
    //###########################################################################
    // $TI Release: $
    // $Release Date: $
    // $Copyright: $
    //###########################################################################
    
    /**
     * Copyright (c) 2001-2004 Swedish Institute of Computer Science.
     * All rights reserved.
     *
     * Redistribution and use in source and binary forms, with or without modification,
     * are permitted provided that the following conditions are met:
     *
     * 1. Redistributions of source code must retain the above copyright notice,
     *    this list of conditions and the following disclaimer.
     * 2. Redistributions in binary form must reproduce the above copyright notice,
     *    this list of conditions and the following disclaimer in the documentation
     *    and/or other materials provided with the distribution.
     * 3. The name of the author may not be used to endorse or promote products
     *    derived from this software without specific prior written permission.
     *
     * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED
     * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
     * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
     * SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
     * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
     * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
     * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
     * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
     * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
     * OF SUCH DAMAGE.
     *
     * This file is part of the lwIP TCP/IP stack.
     *
     * Author: Adam Dunkels <adam@sics.se>
     *
     */
    
    /**
     * Copyright (c) 2018 Texas Instruments Incorporated
     *
     * This file is dervied from the ``ethernetif.c'' skeleton Ethernet network
     * interface driver for lwIP.
     *
     */
    
    /* Standard language headers */
    #include <stdio.h>
    
    #include <kernel/dpl/TaskP.h>
    #include <kernel/dpl/SemaphoreP.h>
    #include <kernel/dpl/ClockP.h>
    #include <kernel/dpl/DebugP.h>
    
    /**
     * lwIP specific header files
     */
    #include "lwip/opt.h"
    #include "lwip/def.h"
    #include "lwip/pbuf.h"
    #include <lwip/stats.h>
    #include <lwip/netifapi.h>
    #include <lwip/ip_addr.h>
    #include "netif/etharp.h"
    #include "lwip/inet_chksum.h"
    #include "lwip/prot/ethernet.h"
    #include "lwip/prot/ip.h"
    #include "lwip/prot/udp.h"
    #include "lwip/prot/tcp.h"
    
    /* This module's header */
    #include "lwip2enet.h"
    #include "lwip2lwipif.h"
    #include "lwip2lwipif_priv.h"
    #include "lwipopts.h"
    
    /* Define those to better describe your network interface. */
    #define IFNAME0 't'
    #define IFNAME1 'i'
    
    //TODO this should come from stack
    /* Maximum Ethernet Payload Size. */
    #ifdef _INCLUDE_JUMBOFRAME_SUPPORT
    #define ETH_MAX_PAYLOAD  10236
    #else
    #define ETH_MAX_PAYLOAD  1514
    #endif
    
    #define VLAN_TAG_SIZE         (4U)
    #define ETH_FRAME_SIZE        (ETH_MAX_PAYLOAD + VLAN_TAG_SIZE)
    
    /*---------------- */
    /* Ethernet Header */
    #define ETHHDR_SIZE     14
    
    /*---------------------------------------------------------------------------*\
     |                             Function Definitions                           |
     \*---------------------------------------------------------------------------*/
    
    
    inline uint8_t* LWIPIF_LWIP_getIpPktStart(uint8_t* pEthpkt)
    {
        const u16_t type = ((struct eth_hdr*)pEthpkt)->type;
        const uint32_t ipPacketStartOffset = (type == PP_HTONS(ETHTYPE_VLAN)) ?
                                             (SIZEOF_ETH_HDR + SIZEOF_VLAN_HDR) : (SIZEOF_ETH_HDR);
    
        return &pEthpkt[ipPacketStartOffset];
    }
    
    static inline void LWIPIF_LWIP_getSrcIp(uint8_t *pIpPkt, ip_addr_t* pIpAddr)
    {
        struct ip_hdr* pIpHdr = (struct ip_hdr*)pIpPkt;
        ip_addr_copy_from_ip4(*pIpAddr, pIpHdr->src);
    }
    
    static inline void LWIPIF_LWIP_getDstIp(uint8_t *pIpPkt, ip_addr_t* pIpAddr)
    {
        struct ip_hdr* pIpHdr = (struct ip_hdr*)pIpPkt;
        ip_addr_copy_from_ip4(*pIpAddr, pIpHdr->dest);
    }
    
    static uint32_t LWIPIF_LWIP_getUdpLiteChksum(struct pbuf* p, const uint32_t ipHdrLength,
                                                 const uint32_t udpPktLength, struct udp_hdr* pUdpLiteHdr,
                                                const ip_addr_t *pSrcIp, const ip_addr_t *pDstIp)
    {
        /* For UDP-Lite, checksum length of 0 means checksum
           over the complete packet. (See RFC 3828 chap. 3.1)
           At least the UDP-Lite header must be covered by the
           checksum, therefore, if chksum_len has an illegal
           value, we generate the checksum over the complete
           packet to be safe. */
        const uint32_t chkSumCoverageLength = (lwip_ntohs(pUdpLiteHdr->len) == 0) ? udpPktLength : lwip_ntohs(pUdpLiteHdr->len);
        uint8_t *pEthPkt = (uint8_t*) p->payload;
    
        const uint32_t updlitePktStartOffset = (uint8_t *)pUdpLiteHdr - (uint8_t *)pEthPkt;
    
        // 1. move the packet buff start to UDP-Lite packet start from EthPkt start
        p->payload  = &(pEthPkt[updlitePktStartOffset]);
        p->tot_len -= updlitePktStartOffset;
        p->len     -= updlitePktStartOffset;
    
        // 2. Compute checksum for UDP Lite Packet
        const uint32_t udpLiteChkSum = ip_chksum_pseudo_partial(p, IP_PROTO_UDPLITE, udpPktLength, chkSumCoverageLength, pSrcIp, pDstIp);
    
        // 3. Revert back to EthPkt start point from UDP-Lite Packet start
        p->payload  = pEthPkt;
        p->tot_len += updlitePktStartOffset;
        p->len     += updlitePktStartOffset;
    
        return udpLiteChkSum;
    }
    
    bool LWIPIF_LWIP_UdpLiteValidateChkSum(struct pbuf *p)
    {
        Lwip2Enet_assert(p->len >= (sizeof(struct ip_hdr) + sizeof(struct eth_hdr) + sizeof(struct udp_hdr)));
    
        bool isChksumPass = true;
        struct ip_hdr *pIpPkt   = (struct ip_hdr *)LWIPIF_LWIP_getIpPktStart((uint8_t*) p->payload);
        uint8_t *pIpPayload     = (uint8_t*)pIpPkt + (IPH_HL(pIpPkt) << 2);
        struct udp_hdr* pUdpHdr = (struct udp_hdr*)pIpPayload;
        ip_addr_t srcIp;
        ip_addr_t dstIp;
    
        LWIPIF_LWIP_getSrcIp((uint8_t *)pIpPkt, &srcIp);
        LWIPIF_LWIP_getDstIp((uint8_t *)pIpPkt, &dstIp);
    
        const uint32_t chkSumCovLen = (lwip_ntohs(pUdpHdr->len) == 0) ? (lwip_ntohs(IPH_LEN(pIpPkt)) - (IPH_HL(pIpPkt) << 2)) : lwip_ntohs(pUdpHdr->len);
    
        if (chkSumCovLen < sizeof(struct udp_hdr))
        {
            isChksumPass = false;
            return false;
        }
    
        if (0 == LWIPIF_LWIP_getUdpLiteChksum(p, (IPH_HL(pIpPkt) << 2), (lwip_ntohs(IPH_LEN(pIpPkt)) - (IPH_HL(pIpPkt) << 2)), pUdpHdr, &srcIp, &dstIp))
        {
            isChksumPass = true;
        }
        else
        {
            isChksumPass = false;
        }
    
        /* Return value should indicate true if checksum error found */
        return (!isChksumPass);
    }
    
    uint32_t LWIPIF_LWIP_getChkSumInfo(struct pbuf *p)
    {
        /* Note: This function assumes pbuf 'p' has atleast ETH_HEADER, IP_HEADER and DATAGRAM (TCP/UDP) HEADER in the first node of pbuf */
    
        Lwip2Enet_assert(p != NULL);
    
        uint32_t chkSumInfo     = 0;
        struct eth_hdr *pEthPkt = (struct eth_hdr *) p->payload;
        struct ip_hdr  *pIpPkt  = (struct ip_hdr *) LWIPIF_LWIP_getIpPktStart((uint8_t*)pEthPkt);
    
        Lwip2Enet_assert(p->len >= (sizeof(struct ip_hdr) + sizeof(struct eth_hdr)));
        const uint32_t ipPktHdrLen     = (IPH_HL(pIpPkt) << 2)  /* multiply by 4 */;
        const uint32_t ipPktPayloadLen = lwip_ntohs(IPH_LEN(pIpPkt)) - ipPktHdrLen;
        const uint32_t protocolType    = IPH_PROTO(pIpPkt);
        ip_addr_t srcIp;
        ip_addr_t dstIp;
    
        uint8_t *pIpPayload   = (uint8_t*)pIpPkt + ipPktHdrLen;
    
        LWIPIF_LWIP_getSrcIp((uint8_t *)pIpPkt, &srcIp);
        LWIPIF_LWIP_getDstIp((uint8_t *)pIpPkt, &dstIp);
    
        switch (protocolType)
        {
            case IP_PROTO_UDPLITE:
            {
                Lwip2Enet_assert(p->len >= (sizeof(struct ip_hdr) + sizeof(struct eth_hdr) + sizeof(struct udp_hdr)));
                struct udp_hdr* pUdpHdr = (struct udp_hdr*)pIpPayload;
    
                if (pUdpHdr->chksum == 0U)
                {
                    /* checksum is valid and not computed by stack */
                    /* calculate the checksum in software and fill the corresponding field */
                    pUdpHdr->chksum = LWIPIF_LWIP_getUdpLiteChksum(p, ipPktHdrLen, ipPktPayloadLen, pUdpHdr, &srcIp, &dstIp);
                }
                chkSumInfo = 0;
            }
            break;
            case IP_PROTO_UDP:
            case IP_PROTO_TCP:
            {
                uint8_t csumCoverageStartByte = 0;
                uint8_t csumResultByte = 0;
                uint16_t pseudoIpHdrChkSum = 0;
    
                if (protocolType == IP_PROTO_UDP)
                {
                    Lwip2Enet_assert(p->len >= (sizeof(struct ip_hdr) + sizeof(struct eth_hdr) + sizeof(struct udp_hdr)));
                    struct udp_hdr* pUdpHdr = (struct udp_hdr*)pIpPayload;
    
                    if (pUdpHdr->chksum == 0U)
                    {
                        /* checksum is valid and not computed by stack */
    
                        csumCoverageStartByte = (uint8_t*)pUdpHdr - (uint8_t*)pEthPkt + 1; /* CPSW cksum info indexing starts from 1 */
                        csumResultByte = (uint8_t*)(&(pUdpHdr->chksum)) - (uint8_t*)pEthPkt + 1;
                        pseudoIpHdrChkSum = ~(ip_chksum_pseudo(NULL, IP_PROTO_UDP, ipPktPayloadLen, &srcIp, &dstIp));
                        pUdpHdr->chksum = pseudoIpHdrChkSum;
                        ENETDMA_TXCSUMINFO_SET_CHKSUM_BYTECNT(chkSumInfo, ipPktPayloadLen);
                        ENETDMA_TXCSUMINFO_SET_CHKSUM_STARTBYTE(chkSumInfo, csumCoverageStartByte);
                        ENETDMA_TXCSUMINFO_SET_CHKSUM_RESBYTE(chkSumInfo, csumResultByte);
                    }
                }
                else if (protocolType == IP_PROTO_TCP)
                {
                    Lwip2Enet_assert(p->len >= (sizeof(struct ip_hdr) + sizeof(struct eth_hdr) + sizeof(struct tcp_hdr)));
                    struct tcp_hdr* pTcpHdr = (struct tcp_hdr*)pIpPayload;
    
                    if (pTcpHdr->chksum == 0U)
                    {
                        csumCoverageStartByte = (uint8_t*)pTcpHdr - (uint8_t*)pEthPkt + 1; /* CPSW cksum info indexing starts from 1 */
                        csumResultByte = (uint8_t*)(&(pTcpHdr->chksum)) - (uint8_t*)pEthPkt + 1;
                        pseudoIpHdrChkSum = ~(ip_chksum_pseudo(NULL, IP_PROTO_TCP, ipPktPayloadLen, &srcIp, &dstIp));
                        pTcpHdr->chksum = pseudoIpHdrChkSum;
                        ENETDMA_TXCSUMINFO_SET_CHKSUM_BYTECNT(chkSumInfo, ipPktPayloadLen);
                        ENETDMA_TXCSUMINFO_SET_CHKSUM_STARTBYTE(chkSumInfo, csumCoverageStartByte);
                        ENETDMA_TXCSUMINFO_SET_CHKSUM_RESBYTE(chkSumInfo, csumResultByte);
                    }
                }
                else
                {
                    EnetUtils_printf("[LWIPIF_LWIP] Un-Supported protocol for HW checksum offload\r\n");
                    chkSumInfo = 0;
                }
            }
            break;
            default:
            {
                chkSumInfo = 0;
            }
        }
    
        return chkSumInfo;
    }
    
    /*!
     *  @b LWIPIF_LWIP_input
     *  @n
     *  This is currently a task which consumes the RX packets retrieved from
     *  the driver in RX packet task, and passes them to the LwIP stack via
     *  netif->input().
     *
     *  \param[in]  netif
     *      NETIF_DEVICE structure pointer.
     *
     *  \retval
     *      void
     */
    void LWIPIF_LWIP_input(Lwip2Enet_RxObj *rx,
                           struct netif* netif,
                           struct pbuf *hPbufPacket)
    {
        Lwip2Enet_assert(netif != NULL);
    
        /* Pass the packet to the LwIP stack */
        if (netif->input(hPbufPacket, netif) != ERR_OK)
        {
            LWIP_DEBUGF(NETIF_DEBUG, ("lwipif_input: IP input error\n"));
            if (!ENET_UTILS_IS_ALIGNED(hPbufPacket->payload, ENETDMA_CACHELINE_ALIGNMENT))
            {
                Lwip2Enet_assert(FALSE);
            }
            /* Put the new packet on the free queue */
            pbuf_free(hPbufPacket);
    
            LWIP2ENETSTATS_ADDONE(&rx->stats.freePbufPktEnq);
            LWIP2ENETSTATS_ADDONE(&rx->stats.rxLwipInputFail);
        }
        else
        {
            LWIP2ENETSTATS_ADDONE(&rx->stats.stackNotifyCnt);
        }
    }
    
    void LWIPIF_LWIP_periodic_polling(struct netif *netif)
    {
            Lwip2Enet_netif_t* pInterface = (Lwip2Enet_netif_t*)netif->state;
            Lwip2Enet_assert(pInterface != NULL);
    
            /* Periodic Function to update Link status */
            Lwip2Enet_periodicFxn(netif);
    
            if (!(pInterface->isLinkUp == (netif->flags & 0x04U) >> 2))
            {
                if (pInterface->isLinkUp)
                {
                    netif_set_link_up(netif);
                }
                else
                {
                    netif_set_link_down(netif);
                }
            }
    }
    
    int32_t LWIPIF_LWIP_start(Enet_Type enetType, uint32_t instId, struct netif *netif)
    {
        int32_t retVal = 0U;
        Lwip2Enet_Handle hLwip2Enet;
    
        /* Open the translation layer, which itself opens the hardware driver */
        hLwip2Enet = Lwip2Enet_open(enetType, instId, netif);
    
        if (NULL == hLwip2Enet)
        {
            /* Note - Use System_printf here as we are not sure if hLwip2Enet print
             * is set and not null. */
            EnetUtils_printf("[LWIPIF_LWIP] Failed to start Enet\r\n");
            retVal = -1;
        }
        return retVal;
    }
    
    void LWIPIF_LWIP_setNotifyCallbacks(struct netif *netif, Enet_notify_t *pRxNotify, Enet_notify_t *pTxNotify)
    {
        Lwip2Enet_netif_t* pInterface = (Lwip2Enet_netif_t*)netif->state;
        for (uint32_t idx = 0; idx < pInterface->count_hTx; idx++)
        {
            Lwip2Enet_setTxNotifyCallback(pInterface->hTx[idx], pTxNotify);
        }
    
        for (uint32_t idx = 0; idx < pInterface->count_hRx; idx++)
        {
            Lwip2Enet_setRxNotifyCallback(pInterface->hRx[idx], pRxNotify);
        }
    }
    
    /*!
     *  @b LWIPIF_LWIP_Stop
     *  @n
     *  The function is used to de-initialize and stop the Enet
     *  controller and device.
     *
     *  \param[in] netif
     *      NETIF structure pointer.
     */
    static void LWIPIF_LWIP_stop(struct netif *netif)
    {
        Lwip2Enet_Handle hLwip2Enet;
    
        /* Get the pointer to the private data */
        hLwip2Enet = (Lwip2Enet_Handle)netif->state;
    
        /* Call low-level close function */
        Lwip2Enet_close(hLwip2Enet, netif);
    
        /* Enet controller has been stopped. */
    }
    
    /*!
     *  @b LWIPIF_LWIP_Init
     *  @n
     *  The function is used to initialize and register the peripheral
     *  with the stack.
     *
     *  \param[in]  *netif
     *      NETIF structure pointer
     *
     *  \retval
     *      Success -   ERR_OK
     */
    err_t LWIPIF_LWIP_init(struct netif *netif)
    {
        /* Populate the Network Interface Object */
        netif->name[0] = IFNAME0;
        netif->name[1] = IFNAME1;
    
        /*
         * MTU is i total size of the (IP) packet that can fit into an Ethernet.
         * For Ethernet it is 1500bytes
         */
        netif->mtu = ETH_FRAME_SIZE - ETHHDR_SIZE - VLAN_TAG_SIZE;
    
        /* Populate the Driver Interface Functions. */
        netif->remove_callback      = LWIPIF_LWIP_stop;
        netif->output               = etharp_output; 
        netif->linkoutput           = LWIPIF_LWIP_send;
        netif->flags               |= NETIF_FLAG_ETHARP;
    
    #if !ENET_CFG_IS_ON(CPSW_CSUM_OFFLOAD_SUPPORT)
        EnetUtils_printf("[LWIPIF_LWIP] LwIP Checksum offload support disabled\r\n");
    #endif
        EnetUtils_printf("[LWIPIF_LWIP] NETIF INIT SUCCESS\r\n");
    
        return ERR_OK;
    }
    
    void LWIPIF_LWIP_rxPktHandler(struct netif *netif)
    {
        Lwip2Enet_netif_t* pInterface = (Lwip2Enet_netif_t*)netif->state;
        for (uint32_t idx = 0; idx < pInterface->count_hRx; idx++)
        {
            Lwip2Enet_rxPktHandler(pInterface->hRx[idx]);
        }
    
         ClockP_start(&pInterface->hRx[0]->hLwip2Enet->pacingClkObj);
    }
    
    void LWIPIF_LWIP_txPktHandler(struct netif *netif)
    {
        Lwip2Enet_netif_t* pInterface = (Lwip2Enet_netif_t*)netif->state;
        for (uint32_t idx = 0; idx < pInterface->count_hTx; idx++)
        {
            Lwip2Enet_txPktHandler(pInterface->hTx[idx]);
        }
    }
    
    err_t LWIPIF_LWIP_send(struct netif *netif, struct pbuf *p)
    {
        /* Get the pointer to the private data */
        Lwip2Enet_netif_t* pInterface  = (Lwip2Enet_netif_t*) netif->state;
        const Enet_MacPort macPort = pInterface->macPort;
        Lwip2Enet_TxHandle hTx = pInterface->hTx[0];
    
        Lwip2Enet_assert(pInterface != NULL);
        Lwip2Enet_assert(hTx != NULL);
        /*
         * When transmitting a packet, the buffer may be deleted before transmission by the
         * stack. The stack implements a 'ref' feature within the buffers. The following happens
         * internally:
         *  If p->ref > 1, ref--;
         *  If p->ref == 1, free(p);
         * pbuf_ref(p) increments the ref.
         */
        pbuf_ref(p);
        /* Enqueue the packet */
        pbufQ_enQ(&hTx->readyPbufQ, p);
        LWIP2ENETSTATS_ADDONE(&hTx->stats.readyPbufPktEnq);
        /* Pass the packet to the translation layer */
        Lwip2Enet_sendTxPackets(pInterface, macPort);
        /* Packet has been successfully transmitted or enqueued to be sent when link comes up */
        return ERR_OK;
    }
    

    https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/81/enet_5F00_lwipif.c.xdt

    Patch 3 :  Replacing disable and enabling global interrupts with Mutex in udma_queraw and udma_dequeraw api's 

    https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/81/udma.mutex.patch

    In my test setup, I have verified it in release mode with TI Clang  .. If you do also in release mode which will be good since in real time we use only release build.

    Yes, I knew you were getting some problems with optimization levels in release mode.

    The below details I got from development, and I am not sure which optimization levels are you used in your test set up.

    This enabled LTO, thumbmode, Oz optimization level

     DEFINES_common := \
    -DSOC_AM243X \
    -DENET_ENABLE_PER_CPSW=1 \
    
    CFLAGS_common := \
    -mcpu=cortex-r5 \
    -mfloat-abi=hard \
    -mfpu=vfpv3-d16 \
    -mthumb \
    -Wall \
    -Werror \
    -g \
    -Wno-gnu-variable-sized-type-not-at-end \
    -Wno-unused-function \
    
    CFLAGS_release := \
    -Os \
    -Oz \
    -flto \
    
    
    CFLAGS_cpp_common := \
    -Wno-c99-designator \
    -Wno-extern-c-compat \
    -Wno-c++11-narrowing \
    -Wno-reorder-init-list \
    -Wno-deprecated-register \
    -Wno-writable-strings \
    -Wno-enum-compare \
    -Wno-reserved-user-defined-literal \
    -Wno-unused-const-variable \
    -x c++ \
    
    
    LFLAGS_common = \
    -Wl,--diag_suppress=10063 \
    -Wl,--ram_model \
    -Wl,--reread_libs \
    -Wl,--zero_init=on \
    -Wl,--use_memset=fast \
    -Wl,--use_memcpy=fast \
    
    LNKOPTFLAGS_release = \
    -mcpu=cortex-r5 \
    -mfloat-abi=hard \
    -mfpu=vfpv3-d16 \
    -mthumb \
    -Oz \
    -flto \

    Regards,

    Anil.

  • Thans you very much. We will apply your recommenadation and let know know results.

    It may take some time due to load on team.

    Best regards

    Rasty

  • We will apply your recommenadation and let know know results

    No worry. We will wait for your results.

    Regards

    Ashwani

  • Hi Rasty,

    While you try things on your side, we are aIso working on our side to deep down the issue (why we are getting peak value of task_jitter in long run)

    Summary,

    1. 1st doubt:
      1. The issue may be related to interrupt generation from DMTIMER in long run.
        1. To confirm this, we tried using  "PRU-IEP timer" to provide cyclic trigger at 125us.
        2. But issue persist (we are getting peak value of task jitter using PRU-IEP trigger)
    2. 2nd doubt:
      1. We may be missing 1 timer interrupt and getting value > (2*interrupt_time_cycle) which is ~250us
        1. To confirm this, we tried DMTIMER in 1 shot mode.
        2. After task jitter calculation, we are setting up oneshot timer again
        3. So, if 1 timer trigger is missed then, jitter calculation should also stop after that
        4. But issue persist (we are getting peak value of task jitter using DMTIMER in one shot mode)
    3. Ethernet dev team working to root cause the issue in parallel.

    Regards

    Ashwani

  • Hi Rasty,

    why we are getting peak value of task_jitter in long run

    We root cause the issue of peak / outlier value of task_jitter.

    1. Patches to use:
      1. cpsw_stats_interrupt_disable
      2. sys_arch
    2. Then Build libraries
      1. make -s -j4 makefile.am243x libs
    3. Then build and load the example for long duration
    4. Start TCP traffic 
      1. iperf -c <IP address of EVM> -b 100M -t 2000
    5. Expectation:
      1. task_jitter should not have peak / outlier now.

    Let me know the results from your setup with this patch.

    Regards

    Ashwani

  •  Hi Rasty

    The attached document has the recommendation on the basis of the TI side experiments.

    Let me know if you have any question/ concern/ feedback ?

    AM24 RTOS Jitter Issue Summary

    Regards

    Ashwani