Driver.lib 2.1.1.71 DK versus EK 1294 - EMACDMAIM REG59 and EMACDMARIS REG57 are mirror MCU designs OR !

It would seem so but it appears the compiler is treading the mask bits as an AND. Perhaps the lower 17 bits of HREG are not actually being OR'd with the maskable bits of the DMAIM regardless of the  OR'd TT/PMT/MMC/TS/RS. It would seem the assumption is made if a bit is 0 in the OR operation the compiler should assume the lower 17 bits are a 1 to mask the bit position.

Likewise the EMACIntClear() logic seems to be reversed as when the AIS flag is raised (ui32IntFlags) and DMAIM or DMARIS seems to be missing to assert |= (ui32IntFlags) RW1C cleared by writing 1 Not 0 to the bit position. The condition causes the AIS/RU bits abate the |= take forever to clear, by that time the RU condition has locked the EMACK RX controller. Seems more prudent to |= all abnormal/normal interrupts when ever the AIS/NIS flag is raised since we don't know what bit in the interrupt was set in DMARIS that needs to be cleared. Other than by & the interrupt status via (ui32INtFlags) which really only sets one side of the OR gate equation. Later changed below to &=  so both sides of the AND gate are tested with DMARIS RW1C clearing all masked bits.

Case EMAC0 receiver asserts INT 0x8080 (32896) AIS/RU end up crashing the EMAC. Reading EMACMARIS the RI bit is not set with AIS/RU, guessing since the FIFO has the start of the frame during RU. Then Interrupt status 0x180c0 (98496) NIS,AIS,RU,RI starts posting but only after some time has passed and numerous AIS,RU flags are first reported by EMACDMARIS.

Seems the application was working from the masked status word upper 31:17 bits and generating a SW56INT instead of using HW interrupt status to resolve RU condition. That approach might be ok until the EMAC is being heavily stressed or actually runs out of RX buffers then what?

The interrupt clear seemingly should test for flag bit NIS/AIS and appears was testing for all status bits and only OR the flag bits [15,17] AIS/NIS to (ui32IntFlags). That has been changed below and is clearing all interrupt sources. So far is working but won't know until RU ever occurs again, praying it don't. Making the change above post the EMAC receiver stayed connected to IOT for 50k seconds. The EMAC receiver randomly drops IOT at 10-16k seconds before DMARIS reported AIS (32896) RU. 

Changing the INT priority level of LWIP timer call has helped to negate random exception error 11 INT5 (bus error), debug reveal often NIS/RI/RU on a POR.

EMACIntClear(void)

{
    //
    // Mask in the normal interrupt if one of the sources it relates to is
    // specified.
    //
    if(ui32IntFlags & EMAC_INT_NORMAL_INT) //Read NIS flag
    {
        ui32IntFlags |= EMAC_NORMAL_INTS; // |= all NIS flags
    }

    //
    // Similarly, mask in the abnormal interrupt if one of the sources it
    // relates to is specified.
    //
    if(ui32IntFlags & EMAC_INT_ABNORMAL_INT) //Read AIS flag
    {
        ui32IntFlags |= EMAC_ABNORMAL_INTS; // |= all AIS flags
    }

Hi Amit,

Even more strange is a hardware (HW) interrupt vector is assigned to EMAC0 INT 56 in (startup_ccs.c) yet LWIP timer call to SW INT 56  is being triggered simultaneously or some time later. Text notes to avoid using Mutexes must keep all interrupt sources in the LWIP context avoiding reentrance.

Seems any masked DMAMARIS interrupt flag will vector directly to the HW INT handler blowing through a programmed SW INT trigger and possibly violate non-reentrant LWIP.

Does the HW interrupt vector export  [extern void lwIPEthernetIntHandler(void);] in (startup_ccs.c) negate the SW INT trigger interrupt source 56? How to stop a HW/SW tie to NVIC or repeating the same INT56 into this global export vector?

For a level-sensitive interrupt, when the processor returns from the ISR, the NVIC samples the interrupt signal. If the signal is asserted, the state of the interrupt changes to pending, which might cause the processor to immediately re-enter the ISR. Otherwise, the state of the interrupt changes to inactive

Hi Amit,

Wonder how possible to set NVIC SW triggered INT56 in non-privileged access mode - like all program examples of EMAC/PHY have configured seemingly without privileged access?

HWREG(NVIC_SW_TRIG) |= INT_EMAC0 - 16;

Data sheet states we must unlock REG76 MPUATTR 0xDA0 - AP bit filed = 000, defaulting as No Access to non/privileged SW. Warning Notes REG76 can only be accessed from privileged mode. Writing 0x011 to the AP bit field REG76 gives non privileged SW full RW access in the first place. 
 
What confusion is going on here - might it be Stellaris MCU has default Full RW privilege access AP but field set 0x011 by default?

Accordingly the EMAC INT56 SW interrupt should not be functional, causing a non privileged access error. (Datasheet page 139) AP bit field must be set RW= 011 to gain full access to redirect EMAC0 HW INT 56 source in NVIC upon gaining full privileged access to REG53 SWTRIG 0xF00 page 161.

CCS debug now showing the HW interrupt 56 bit 40 (32-54) as disabled bit position 9 after enabling privileged mode versus setting CFGCTRL, MAINPEND=1.

For some reason CCS Debug NVIC_SW_TRIG bits do not show any changes during refresh but the mask bits of DMAMARIS are all being reset after changing the  |= to &= of all NIS/AIS source INTS.  The strange part is the UARTprintf() messages would print the AIS status bits multiple times in looping not clearing them between IOT connection attempts.  That symptom would seem to infer the DMAMARIS set & clear was failing so try hard to locate the cause.

Masking NIS/AIS all bits had issues that appear in the interrupt Priory group 4:4 NIL rule as LWIP timer HW INT39 was  below SW INT 56. When the priority of LWIP timer priority level was set farther down the 3 bits grouping say 0x10 versus 0xA0, NVIC would raise exception 11 bus error shortly after POR. The bit positions in the grouping priority has something to do with the error cause. Find it odd that EMAC priority 0xC0 and LWIP timer priority 0xA0 the third bit positions are in line to binary 4:4 priority.  

At least learning few things about MPU protected memory mode TEX,S,C, AP bits, adding code below for enabling SW triggers in NVIC per-data sheet.

/* Enable the Cortex-M memory protection unit. PRIV_DEFAULT:
 * Enables the default memory map when in privileged mode and
 * when no other regions are defined. HARDFLT_NMI: Enables
 * the MPU while in a hard fault or NMI exception handler.
 * If this option is not enabled, then the MPU is disabled while
 * in one of these exception handlers and the default
 * memory map is applied. */

 ROM_MPUEnable(MPU_CONFIG_PRIV_DEFAULT | MPU_CONFIG_HARDFLT_NMI);

 /* MPU attributes Software privileged mode, RW full access */
  HWREG(NVIC_MPU_ATTR) = (NVIC_MPU_ATTR_AP_RW_RW);

Hi Amit,

Priority grouping rules state the lower value interrupt takes priority over the higher interrupt value no matter the priority level of the higher interrupt source. Think that is how NVIC now determines NIL sequencing of chained interrupt sources. Explains how NVIC more rapidly services chained interrupts when ever the priority level falls close together in the 3 bit positions. 

Reviewed again post masking of EMACIntStatus(), bear in mind we want the real time interrupt masked source to always pass into the (ui32Val) variable.

So the masked value returned (true) is | with DMAIM static Interrupt mask source bits, &= with (ui32Val). Seemingly the &= combined with | could negate the real time (ui32Val) DMARIS values that were previously taken. Noting the MPU is not flawless - higher speeds can at times have a certain % of random logic error with loosely written code. Otherwise what seemingly works for 13 hours without incident should never bus fault from random unexpected interrupt errors. Should we be forced to run our MPU peripherals at turtle access times when SYSCLK gives us the advantage to warp time and space? Without adding SysCtlDelay() at several turning points the peripheral train derails mostly upon accessing EMAC, EEROM, SRAM.  Case high speed access the interrupts handling has to be near flawless.

The application layer expects real time interrupt status for Integer compares returned from calls made to (emac.c). Adding any static values in the mix could intermittently lead to a bad outcome should the mask | negate real time interrupt status.

Might it be safer to never allow DMAIM into the mix in real time interrupts, that removes question "Could this be an issue" when things go wrong. The &= inside HWREG masks bits if the real time bit positions of TT,PMT,MMC are set. Case logic DMARIS &  1+1=1 / 0+1=0 far better than DMAIM |   0+1=1 / 1+1=1.

    if(bMasked)
    {
        ui32Val &= (EMAC_NON_MASKED_INTS & HWREG(ui32Base + EMAC_O_DMARIS));
        //ui32Val &= (EMAC_NON_MASKED_INTS | HWREG(ui32Base + EMAC_O_DMAIM));
    }

when the two interrupts occur together in time,

That is what the code appears to do, EMAC HW INT56 is set enabled (tiva-tm4c1294.c) and SW INT56 triggering inside LWIP timer call function. Long ago added an unpend statement for SW INT56 into the interrupt handler, later comment it out not understating both HW/SW INT56 were both firing. Today added that NVIC unpend INT56 back into the interrupt handler. Apparently we must handle the HW interrupt clearing when NVIC sets pend and not just clear DMARIS as the code was doing.

Seems when both HW & SW INT56 ties NVIC, he sets a pending status on the (disabled) INT56/40 bit 9 in CCS debug. Sometimes that pending status quickly goes away, assume SW interrupt. Other times pend 56 stops the application in pending status, assume HW 56 is not cleared. Polled level interrupts do not typically auto unpend unless they are pulsed and SW interrupt. How HW/SW INT56 is even working together amazes, not without throwing all caution to the wind as it seems. Believe HW INT56 is asserted when an obscure EMAC DMA condition as it does not raise HW 56 flag very often yet it does. Think it may be safer fewer issues to not use a SW INT56 trigger in the LWIP timer and just poll the HW INT handler as to advert would be NVIC ties for the same interrupt.

Random exception 11 appears to lean more as interrupt, EMAC DMA, timing related issue about the interrupt vector handling. CCS debug reveals the EEROM is still incrementing R/W address and the LWIP timer counting after the phantom pend suddenly stopped the application.

Hi Amit, 

"priority level can only be seen when the two interrupts occur together in time"

Priority grouping changes the way priority ordered IRQ's are serviced.

2.5.6: If multiple pending interrupts have the same group priority, the subpriority field determines the order in which they are processed. If multiple pending interrupts have the same group priority (4:x) and subpriority (x:4), the interrupt with the lowest IRQ number is processed first. This case LWIP timer had a higher priority yet the lower IRQ.

Major surprise:

The datasheet makes no direct connection between IRQ56 and EMAC0 other than NVIC holds a place for IRQ56. There is not an EMAC peripheral interrupt raised when EMACDMARIS flags any number of previously masked interrupts. That appears to have alluded TI programmers for the way NVIC was handling IRQ56. That confused many to believe there were two unique ways IRQ56 would assert in the EMAC when there was no physical IRQ56 hardware assignment EMAC0. The EMAC peripheral makes no claim of a single IRQ56 raised for register EMACDMARIS. Now we poll DMARIS via LWIP timer call and disable IRQ56 remove the priority. Same results AIS at some point in time will not be cleared and start a cascade of un-cleared DMARIS bits 16:0.

One of the reasons to witness at times repeating AIS and crash EMAC0 RX/TX controller. The IOT code made this very wrong assumption we all became a victim of. It's not a matter of [ if(ui32Status) ] rather it is when (ui32Status), other words get the dang IF out that EMACIntClear() and always clear DMARIS or the next time LWIP timer SW triggers or even Polls the interrupt handler the same interrupt will be read many times over.

    /* Read and Clear EMACDMARIS REG-57 interrupt.*/
    ui32Status = MAP_EMACIntStatus(EMAC0_BASE, false);
    
  // If the interrupt really came from the Ethernet and not our
    // timer, clear it.
    //
    //if(ui32Status)
    //{
        MAP_EMACIntClear(EMAC0_BASE, ui32Status);
    //}

Hi Amit,

"The SW trigger as I mentioned earlier is used to Prime an action"

Sorry for beating this horse to death but EMAC0/DMA engine is not behaving in accordance with datasheet disclosure circuit analysis of Ethernet peripheral. We have disabled the EMAC0 IRQ56 and removed the priority being the LWIP timer call is now polling the original interrupt handler assigned to IRQ56.

That being said it appears the EMAC0 believes IRQ56/40 is at first enabled and as noted earlier post magically pends the disabled IRQ56/40 one time and un-pends IRQ56/40 one time only.

Perhaps this is the undocumented EMAC0 interrupt trigger you are referring to get the process started but that would make it HW trigger event in that case.

Please advise why we see IRQ56/40 which is not enabled and not explicitly disabled set the IRQ pending?

Hi Amit,

Don't believe that is even possible. Review datasheet several times no text even implies IRQ56 is raised with or for interrupt sets inside EMACDMARIS. Oddly IRQ56 stays in set & clear pending indefinitely as shown above post.

IRQ 56 at this point is seemingly a hardware IRQ, originating from where and why is the question?

Point is IRQ56 interferes with default SW trigger IRQ56 when SW IRQ56 has been configured.

Far as DMARIS all upper bits 31:17 are completely masked out.

EMACIntStatus(uint32_t ui32Base, bool bMasked)
{
~~~~
    //
    // Get the unmasked (false) interrupt status 16:0 and clear 31:17 status fields.
    //
    ui32Val = HWREG(ui32Base + EMAC_O_DMARIS);
    ui32Val &= ~(EMAC_NON_MASKED_INTS | EMAC_DMARIS_AE_M |
    		                            EMAC_DMARIS_TS_M | EMAC_DMARIS_RS_M);

Hi Amit,

Actually TT, PMT, MMC are all inside EMAC_NON_MASKED_INTS - those 3 were left out of the original EMACIntStatus(). Who knows it they were self setting some how. Really did not see need to have all 31:17 status confusing the basic TX/RX frames, Kissing it at this point.

Still unclear if EMAC peripheral logic asserts IRQ56 since figure 20-1 and EMAC/DMA text do not make that point clear?

All we know for certain is the IRQ assignment map suggest EMAC0 IQ56/40 and NVIC confirms we can configure and use it for SW IRQ56 with proper RW priority RA bits = (0x011) or by other means. Do not like nor trust the results of SW IRQ56 to call the Ethernet interrupt handler. The SW trigger more frequently and randomly causes exception (11). Bus error seems to be cause related to faults that occur anywhere in the instruction decode cycle of peripheral RW access times. Possibly the application code moves faster with a SW trigger reads of EMACDMARIS status than it does by polling is one thought. 

The MAC/PHY peripherals are 1st reset, MAC reset waits for DMA fully reset, then EMACInit() and lastly EmacConfigSet().

// Wait for EMAC0 DMA to signal it has soft reset.
// MACDMABUSMOD SWR[1] toggles 0 prior to initializing DMA.
while(HWREG(EMAC0_BASE + EMAC_O_DMABUSMOD) & EMAC_DMABUSMOD_SWR)
{
}

BTW:

Thanks for assistance and why not try this out with qs_iot to confirm IRQ56 behavior.

The NIS/AIS interrupts were not enable in the vanilla code as they are in post above.

CCS5.4 debug simulator may be part to blame in strange DMARIS -IM registers reading - not the first strange debug post ever captured.

Have to LOL what we see here TU & TI stopped updating these bits in simulator, complete nonsense being EMAC is transmitting just fine at this time.

Ops our bad as we never enabled the TU bit interrupt in EMACDMAIM in above post. Point is we only use the NIS bit to flag any of the OR'd individual enabled interrupts as they are set in the IM. So we don't mask or flag all the OR'd interrupts of NIS, AIS if any single interrupt is enabled as it was programmed in the Tivaware library.

That view point original EMACDMAIM set bits logic may work for awhile but seemingly is backwards at the program level and at heavy load high speed faults (11) like a mad man with hatchet.

Hi Amit,

Agree that might be what was happening at that point in debug time since TU interrupt was not masked in DMAIM.

Early on posts had claimed to set TU, TX_Stopped mask bits DMAIM enable and added TU flag into the interrupt handler (tiva-tm4c129.c) and later removed both during trouble shooting. Both interrupts were not originally configured, after seeing the debug behavior OR's them with existing TI flag.

After re-adding both the TU debug bit stated to cycle more often. More importantly now [tivaif_process_transmit(tivaif);] asserts not only with TI but also TU, TX_Stopped time periods. TU might not seem so important but that also helps to mitigate the raising exception (11) during high speed TX data controller access. Unplugging Ethernet cable is good stress test of TX_Stopped interrupt. The good thing is IOT time since reset count picks up where it left off (reconnected) or until the server dumps the http connection keep alive. Watch dog timer1 starts to complain capturing the Ethernet link event moderating the loop. WDT1 not doing much at this point yet realize he could be used to read the DMAMARIS 31:17 bits status and invoke an Ai like action based on reported EMAC conditions..

Seems EMAC INT56 was being used merely as a SW trigger source and issues in SW interrupt priority queuing appear to raise exception (11).  

Polling the EMACDMARIS register has stopped triggering random exception (11) events at various points in the IOT application. That one random event was most difficult to find cause against all odds.

Hi Amit,
"the polling should not be the cause of a exception" Polling DMARIS helps to prevent exception 11. INT56 when used for a SW trigger into the same interrupt handler may have an errata with any priority level assignment, some levels are worse than others. Confirmed that SW triggers occurring at IRQ priority level 0x10 versus 0x0a causes more random and frequent bus faults.

Polling the DMARIS register appears to be far more stable with a 2.5us(5) or 12.5us LWIP timer interval with only one priory level versus two. Issue seems to be related to access timing on MPU local bus. The tendency for raising exception appears to be linked directly to speed. TI definitely should check this issue out in the LAB.

Hi Amit,

Interesting discovery there all along; DMAIM ETE masked bit is not enabled and RU seems to be reversed with ETI bit in debug. Setting RX descriptors to 4 as to see if RU would toggle more often in CCS debug makes no difference other than posting more frequent code 99525. The AIS UARTprintf() reports code 99525 so the RU bit is actually asserting but not reporting proper debug bit position. Look at above posts the ETI set was assumed to be normal but on review is quite abnormal. : -)

Hi Amit,

Wishful thinking may help the S&P500 but not such hardware artifacts. PHY link ever randomly dropped and returned even while under watchdog interrupt constraints is most unforgiving, 95% of link disconnect tests simply end up locking the EMAC at some point in transmit.

Real world scenarios when the power on the switch suddenly drops and PHY link interrupt then signals linkup to LWIP low level netif. The EMAC runs for a bit then throws up chunks of salsa and stops processing any frames. Huston we got issues in this big MAC. 

Soft resetting the DMA has no effect to recover TX/RX controller when what ever locks up in the EMAC goes into oblivion.

This capture 70k seconds IOT online after the link to switch dropped, returned a few seconds later. AIS seems to be the dream killer causing RU.

WatchdogIntHandler(void)
{

		/* Check if the interrupt belongs to Watchdog-0*/
		if(ROM_WatchdogIntStatus(WATCHDOG0_BASE, true))
		{
			/* Clear watchdog 0 interrupt */
			ROM_WatchdogIntClear(WATCHDOG0_BASE);
			//
			// If the motor is in the running state, fault when this occurs,
			// otherwise, do nothing.
			//
			if(g_ulState & STATE_FLAG_RUN)

			{
				//
				// Indicate an watch dog fault.
				//
				MainSetFault(FAULT_WATCHDOG);

				MainEmergencyStop();

			}
		}
		else /* Check if the interrupt belongs to Watchdog-1*/
			if(ROM_WatchdogIntStatus(WATCHDOG1_BASE, true))
		     {
			  /* Clear watchdog 1 interrupt */
		            ROM_WatchdogIntClear(WATCHDOG1_BASE);

		        UARTprintf("<< Watchdog-1 Timeouts Cleared >> \r\n");

		        /* See if there is an active link. Query the
		         * PHY basic mode status REG73:MR1(EPYBMSR) */
		         bHaveLink = MAP_EMACPHYRead(EMAC0_BASE, 0, EPHY_BMSR) & EPHY_BMSR_LINKSTAT;

		        if(bHaveLink == !EPHY_BMSR_LINKSTAT)
		        {
		            UARTprintf("<< Watchdog-1 No Valid IP Address >>\n\n");

		            bHaveLink = false;

				       /* Set back the timestamp of the last call to sys_check_timeouts()
				        * This is necessary if sys_check_timeouts() hasn't been called for
				        * a long time (While saving energy, waiting for a valid link)
				        * to prevent all timer functions of that period being called.  */
				        sys_restart_timeouts();

			                /* Close the socket reset connection values and buffers */
				        exoHAL_SocketClose(0);

				        /* Temporarilly disable the IOT SW trigger interrupt
				         * vector call to Print All Data */
				         IntDisable(54);

		        }
		        else
		        {
		        	UARTprintf("<< Watchdog-1 Valid IP Address >>\n\n");

                           /* Inform the Application we now have a link */
		            bHaveLink = true;

		           /* Punch 1 times the dog this long delay. */
		            ExoHALPunchDog(1);

                           /* Delay so we don't panic the NVIC or
                            * EMAC as the link becomes stable */
                            SysCtlDelay(g_ui32SysClock * 1);

		           /* Re-enable the IOT SW trigger interrupt
		            * vector call to Print All Data */
		            IntEnable(54);

			   /* Software trigger IOT Print All Data */
			    HWREG(NVIC_SW_TRIG) = 54 - 16;

	        }
      	}
}

Hi Amit,

That is what the PHY at a HW level in conjunction with the EMAC and abstraction layer is supposed to take care of. When the physical link is lost the MAC should drop the packets flushing the FIFO but it seems the TX/RX controller keeps processing the last buffered frames regardless of the AIS status. Hence we added the watchdog timer to disable the Print All Data (tStats) function as not to flood the FIFO when LWIP is not processing PBUF's when the link is down. 
 
Added a power switch to the Ethernet switch today and hard wired the power pack to it and the PCB. So we can now switch it off then see what happens when the link is randomly dropped and returned while the EMAC is working. 95% failure rate are not good results.

Time since reset reaching 70k-225k seconds online often end in fault or just stop processing. CCS debug pause mostly ends up in LWIP TIME_WAIT while loop [if (PBUF !=NULL)]. Reason for adding the LWIP sys_restart_timeouts call in the watchdog,

edit