Linux Interrupt jitter, latency

One far shot since nobody else chimed in. Are you sure debouncing isn't enabled on the GPIO pin you use for the interrupt? (Should be disabled by default AFAIR) In case debouncing is enabled the pin will be debounced with the 32kHz clock which will obviously introduce a 1/32ms= ~30us jitter. For more info about this please have a look in the OMAP3 TRM chapter 25 with respect to the GPIO module...

With respect to jitter getting larger when having Ethernet traffic I'm not sure what can be expected of jitter in Linux on an ARM CONFIG_PREEMPT configured kernel, but I tend to agree that 250us seems to be a (very) long time...

Please let us know if/when you find any further info on this - Best regards and thanks in advance
  Søren

Hi,

This I unfortunately don't have any really great idea about and will let it be up to some of the guys more knowledable in the "upper" Linux layers to answer this. One idea though: You might be able to move you interrupt to the FIQ interrupt line instead ofthe normal IRQ line? I don't know exactly if/how this is supported in Linux, but it might be another valid option for getting you interrupt the highest priority?

Good luck - Glad that you managed to get a bit closer to the problem source :-)
  Søren

Hi Marc,

I think the "threaded interrupt handling" is your problem, as I think the scheduler in 2.6.32 typical run at 100Hz. It has been quite a while since I last looked into this, but try to do something like descrobed at: http://wiki.gumstix.org/index.php?title=GPIO_Event_Driver#Driver_Sources and I'm rether sure you will be able to achieve much better performance than your current 10ms latency...

Good luck
  Søren

Hi Søren

I have tried the gpio lib and test app and I can still only get around 10ms reaction.

Is there anything special about the pinmux and interrupt handler that I need to setup.  From reading the register documentation the control is in mux block 6 and I have configured the input as an input.

Have you previously configured a GPIO input as a source of an interrupt ? and what sort of duty cycle could be applied to get a reliable interrupt ?

I am just a bit confused as I can get 500us reaction time from a little STM32 ARM ucontroller with a fraction of the clock speed of the DM3730.  There has to be a simple explanation to solve my issue.  Hopefully you or someone who has delved into pinmux configuration and the Linux kernel will have an idea.

Thanks in advance

Marc

I am also wondering about the setup of the interrupt controller and if I need to activate the input assigned to INTC instead of GPIO.

One thing I notice is that when I can /proc/interrupts :

root@taodemo:~# cat /proc/interrupts
           CPU0
  7:      33322        INTC  TWL4030-PIH
 11:          0        INTC  prcm
 12:        669        INTC  DMA
 24:          0        INTC  omap-iommu.0, Omap 3 Camera ISP
 25:          2        INTC  OMAP DSS
 37:     130407        INTC  gp timer
 56:    1800910        INTC  i2c_omap
 57:          2        INTC  i2c_omap
 61:         24        INTC  i2c_omap
 72:        735        INTC  serial idle, serial
 73:          3        INTC  serial idle
 74:          3        INTC  serial idle
 77:         73        INTC  ehci_hcd:usb1
 83:          0        INTC  mmc0
 86:       3684        INTC  mmc1
 92:          1        INTC  musb_hdrc
 93:          0        INTC  musb_hdrc
161:      23481        GPIO  eth0
296:          0        GPIO  ads7846
330:         30        GPIO  gpio irq handler
378:      33322     twl4030  twl4030_usb
384:          0     twl4030  mmc0

I have written a simple irq handler and used it to handle the input.  Supplying a 2ms duty cycle signal it sees a couple of interrupts.

How can I check the configuration of the pin to see if there are any additional settings, e.g. debounce filter (as that could cause issues) or anything else that may prevent the signal from being passed through ?

Thanks again

Marc

Hi Marc,

Quick replies to you two previous posts:

1) AFAIR there is nothing special you need to setup, but of cause kernel defaults might have changed since I last checked...

2) Yes - I'm using GPIOs being inputs for interrupts all the time. Haven't tested any minimum duty cycle for a long time though (a I haven't seen problems), but AFAIR it's more in the range of 10-100uS compared to 10ms. The GPIOs can be debounced using the 32kHz clock, which of cause will add something like 31us per 32kHz debouncing clock cycle...

3) It's (unfortunately) a big and unpleasant surprise to many that it's actually possible to get more "fast response/real time" performance out of a 1MHz MSP430 than of a full blown 1GHz Cortex-A8 system running Linux (for small tasks like flipping a GPIO, etc). In general I think unloaded Linux interrupt latency typically is around 10-100us of cause highly dependent on HW.. Going to userspace timing might go up to like 10ms as indicated above as well.
 
4) It's OK to have the interrupt routed to the GPIO. Only internal IP blocks can go directly to INTC => No way of coupling you GPIO pin directly into INTC, as it needs to go through the GPIO IP block on it way to INTC...

5) For further info on complete DM3730 and GPIO module in particular, please check the DM3730 TRM (http://www.ti.com/lit/ug/sprugn4p/sprugn4p.pdf) chapter 25. Maybe in particular "Table 25-47. GPIO_DEBOUNCINGTIME"

I hope this help you forward - Good luck
   Søren

Hi Søren

I have been playing with my simple kernel driver and the only way I can get any kind of speed of reaction out of the irq handler is to have it in level detect rather than edge detect.

With the irq servicing function performing a do_gettimeofday and subtracting last from current to give me a delta then doing a printk I am closer to the 1ms required.  The debug is showing ~1.5ms but I could believe that the printk could easily add 3-400 us.  My next step is to toggle a GPIO output and scope it to see how accurately it resembles the input.

I think my problems have been due to the edge detect mechanism and the fact that the software is still in the irq servicing routine, or at least exiting and not cleared the irq request before the next one appears.

From what I have seen the IRQ is cleard at "some point" once the IRQ_HANDLED is returned by the handler function and I think this may be the latency issue that I am seeing.

I will run some more tests with the level detect rather than edge detect.

I will let you know my findings and hopefully anyone else reading this and who may have a similar issue.

Thanks for you help and being a sounding board.

Marc

Hi Marc,

Please keep us posted with any further results you get. And just as a short encouraging note I agree to your statements above (printk-time, IRQ-handling and edge/level notes), so I definitely think you are on right track... Glad that you found my comments useful and that you are progressing :-)
   Søren

"With the irq servicing function performing a do_gettimeofday and subtracting last from current to give me a delta "

Would you not get a more accurate result if you read a hardware hires counter ? Sorry, do not know whether your system includes any !

Hi Søren

I have been moving accross onto the 2.6.37 kernel.  I now have the system booting via NFS so I will compile the test module to see if there have been any fixes in the irq handling.  There were notes in the new kernel release to state that there have been changes.

I will post my findings.

Marc

Don't know how power-saving mode is implemented in linux but in case of the Windows CE 6.0 100 usec GPIO interrupt latency was caused by reducing CPU clocks. I used simple console application which just enables periodic SPI transfers on GPIO interrupts so WinCE sees no user-mode activity and slows CPU after each interrupt.

Beware of power management!

Sergey

Hi all,

We are facing a similar issue (as said by the original poster of this thread) with our DM3730 based board.
We are using our Vendor supplied TI Arago Linux 2.6.32 Kernel.

Our Processor is acting as an SPI master and an AFE board is acting as SPI slave.

SPI slave is configured to generate periodic interrupts at 125 microseconds.

During this time, some input data to be read at the master side. This is done at the master's tasklet.

A GPIO of DM3730 is configured to generate interrupt when the interrupt is reached from SPI slave.

Using a scope, we have ensured the correctness of interrupts from the SPI slave (AFE board).

But we can see interrupts missing at the master side in every 94 milliseconds.

 We have seen this by toggling another GPIO inside interrupt handler. 

Another observation is if the CPU becomes loaded, interrupt missing happens very frequently.

Because of this, our data acquisition becomes wrong and our output result is incorrect.

We suspect if there is any high priority interrupts being serviced by the Kernel at this 94ms interval.

We have verified the output of cat /proc/interrupts. The other frequent interrupts running simultaneously are 

gp_timer and eth0. We have also tried to increase the priority of our interrupt by writing some registers at our 

IRQ controller. Could anyone please shed some light to this issue? What may be the reason of this interrupt miss / interrupt latency?

The processor is running at 1GHz.

Thanks,

Honey S

HI Honey,

We are facing the same problem

We have ported JB 4.2.p1 with kernel 3.4 onto our custom board having omap4430.  We have connected an interrupt pin of a  device on our board to gpi 80 pin of omap. The device is configured to send out interrupts to the omap gpi 80 at every 10ms i.e 100hz pulse train. We toggled another gpio 3 pin of omap on receiving every interrupt  pulse as mentioned above in the interrupt handler function of our device . We noticed on probing that the gpio 3 did not toggle consistently at 10ms, and it skipped a few pulses, and at times the delay was as high as 200ms.

However on repeating the same experiment n ics with kernel 3.0.4 we noticed that gpio 3 toggeled consistently at 10ms without any issues.

We have to correct this inconsistent behaviour and the interrupt latency. Could you provide suggestions on this?

Thank You

Prakash

Hi all,

    We are facing the same problem , we are connected FPGA to GPMC port and having GPIO pin for interrupt to linux from FPGA to read data.In this system only first inttrupt takes long time to process the ISR. then next on words probability is less . The gitter is approx. 4ms.

    How can i overcome the problem?

Thank You

Nilesh

Not an expert here.  But my understanding is that printk in an interrupt routine is absolutely forbidden, not just for timing (which is bad), but for the possibility of an interrupt causing state problems in the I/O.

Hi honey,

What u did for this problem.

I am also facing same problem. I have interrupt latency . and my isr is big. So I am missing some interrupts.

-bharath