This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM3354: Real-time problems with simple GPIO reading

Part Number: AM3354

Hello,

we are running Linux 4.14.79-rt47 #2 with AM3354 and we are experiencing some strange behaviour related to real-timeness for gpio reading. The kernel is configured with CONFIG_PREEMPT_RT_FULL=y and HZ_1000=y.

Our SW reads gpio by using libgpiod every 1 ms and triggers a print when gpio levels changes, please find below the code.

We inject an input signal which is a square waveform with period equal to 300ms and dudty cycle as 50%.

Typically all edges are detected with a maximum error equal 2-3ms; but sometimes (about 1 over 30) we get an unexpected delay up to 50-150ms.

The application runs in FIFO scheduler with priority 98. The remaining threads run on the default scheduler.

 

// gpiod line initialization...

 

bool lastValue = gpiod_line_get_value(line) != 0;

bool value = lastValue;

uint64_t lastTs = 0;

 

while(!interrupted()) {​​​​

 

    while(value == lastValue) {​​​​

        struct timespec ts = {​​​​ 0 }​​​​;

        ts.tv_nsec = 1000000;

        int ret = -1;

 

        while(ret != 0)

            ret = nanosleep(&ts, &ts);

 

        value = gpiod_line_get_value(line) != 0;

    }​​​​

 

    // Uses clock_gettime(CLOCK_REALTIME)

    uint64_t ts = getMicrosecondsTimestamp();

 
BR cout << "*** Breaker event: " << value << " - ts " << ts << " - diff " << ts - lastTs << " - groupID " << this->groupID << " ***" << endl; lastTs = ts; }

Below a small log from the application where you can see that the difference from the previous edge is much more than expected 150ms.​ In particular, in line no.7 we got about 202ms.

*** Breaker event: 1 - ts 1614014210177509 - diff 147579 - groupID 52707 ***

*** Breaker event: 0 - ts 1614014210330205 - diff 152696 - groupID 52708 ***

*** Breaker event: 1 - ts 1614014210478128 - diff 147923 - groupID 52709 ***

*** Breaker event: 0 - ts 1614014210629910 - diff 151782 - groupID 52710 ***

*** Breaker event: 1 - ts 1614014210777620 - diff 147710 - groupID 52711 ***

*** Breaker event: 0 - ts 1614014210930340 - diff 152720 - groupID 52712 ***

*** Breaker event: 1 - ts 1614014211133018 - diff 202678 - groupID 52713 ***

*** Breaker event: 0 - ts 1614014211230153 - diff 97135 - groupID 52714 ***

*** Breaker event: 1 - ts 1614014211377431 - diff 147278 - groupID 52715 ***

*** Breaker event: 0 - ts 1614014211529728 - diff 152297 - groupID 52716 ***

*** Breaker event: 1 - ts 1614014211677992 - diff 148264 - groupID 52717 ***

*** Breaker event: 0 - ts 1614014211829981 - diff 151989 - groupID 52718 ***

*** Breaker event: 1 - ts 1614014211977247 - diff 147266 - groupID 52719 ***

*** Breaker event: 0 - ts 1614014212130436 - diff 153189 - groupID 52720 ***

*** Breaker event: 1 - ts 1614014212278182 - diff 147746 - groupID 52721 ***

*** Breaker event: 0 - ts 1614014212429678 - diff 151496 - groupID 52722 ***

*** Breaker event: 1 - ts 1614014212577919 - diff 148241 - groupID 52723 ***

*** Breaker event: 0 - ts 1614014212730462 - diff 152543 - groupID 52724 ***

*** Breaker event: 1 - ts 1614014212877560 - diff 147098 - groupID 52725 ***

*** Breaker event: 0 - ts 1614014213030037 - diff 152477 - groupID 52726 ***

*** Breaker event: 1 - ts 1614014213177286 - diff 147249 - groupID 52727 ***

  • Hi,

    Please return the results of this this command:   uname -a 

    Have you used cyclictest to measure overall system latency?  This is an example of how to run cyclictest, this example will run for about 10minutes.

    cyclictest  -m -n -p90  -h400 -q -D10m >output.txt

    Are you using the Kernel config from the TI SDK?

    Have you tried just storing the timestamp data and then printing it out after 100 loops?

    Best Regards,

    Schuyler

  • Hi Schuyler,

    please find the uname output

    uname -a
    Linux ABB-40-bd-32-24-47-e4 4.14.79-rt47 #2 PREEMPT RT Wed Feb 24 18:19:18 CET 2021 armv7l GNU/Linux

    We were using 3 threads with priority RT 98,97 and 95 (the 98 was the gpio detection thread)

    From the original post we changed the priority of our threads to 10,8 and 5 in order to not interfere with some IRQ threads having 51 priority; moreover, we merged the kernel defconfig with the one generated from your SDK, specifically the fragments regarding AM335X and RT(please find below what has changed in defconfig)

    Now the problem seems disappeared; after that cyclic test has latency in order of 30-40µS and the gpio is deteced correctly without any additional delay.

    Do you see some of those changes really fixing the problem?

    Thanks in advance.

    BR.

    Lorenzo

    diff --git a/arch/arm/configs/abb_cmu_defconfig b/arch/arm/configs/abb_cmu_defconfig
    index f7c33d8b40bb..dce2b2ed73b5 100644
    --- a/arch/arm/configs/abb_cmu_defconfig
    +++ b/arch/arm/configs/abb_cmu_defconfig
    @@ -1,5 +1,113 @@
    +CONFIG_ARCH_MULTI_V7=y
    +CONFIG_ARCH_MULTI_V6_V7=y
    +CONFIG_KEYSTONE2_DMA_COHERENT=n
    +CONFIG_ARCH_OMAP=y
    +CONFIG_SOC_AM33XX=y
    +CONFIG_ARCH_OMAP2PLUS=y
    +CONFIG_OMAP_INTERCONNECT_BARRIER=y
    +CONFIG_ARM_PSCI=n
    +CONFIG_ARCH_OMAP2=n
    +CONFIG_ARCH_OMAP3=n
    +CONFIG_ARCH_OMAP4=n
    +CONFIG_SOC_OMAP5=n
    +CONFIG_SOC_AM43XX=n
    +CONFIG_SOC_DRA7XX=n
    +CONFIG_ARCH_KEYSTONE=n
    +CONFIG_SMP=n
    +CONFIG_VFP=y
    +CONFIG_NEON=y
    +CONFIG_KERNEL_MODE_NEON=y
    +CONFIG_CPUSETS=n
     CONFIG_SYSVIPC=y
    +CONFIG_SUSPEND_FREEZER=y
    +CONFIG_SUSPEND_SKIP_SYNC=n
    +CONFIG_HIBERNATION=n
    +CONFIG_PM_SLEEP=y
    +CONFIG_PM_SLEEP_SMP=y
    +CONFIG_PM_AUTOSLEEP=n
    +CONFIG_PM_WAKELOCKS=n
    +CONFIG_PM=y
    +CONFIG_PM_DEBUG=y
    +CONFIG_APM_EMULATION=n
    +CONFIG_PM_OPP=y
    +CONFIG_PM_CLK=y
    +CONFIG_PM_GENERIC_DOMAINS=y
    +CONFIG_WQ_POWER_EFFICIENT_DEFAULT=n
    +CONFIG_PM_GENERIC_DOMAINS_SLEEP=y
    +CONFIG_PM_GENERIC_DOMAINS_OF=y
    +CONFIG_CPU_PM=y
    +CONFIG_ARCH_SUSPEND_POSSIBLE=y
    +CONFIG_ARM_CPU_SUSPEND=y
    +CONFIG_ARCH_HIBERNATION_POSSIBLE=y
    +CONFIG_CLKDEV_LOOKUP=y
    +CONFIG_HAVE_CLK_PREPARE=y
    +CONFIG_COMMON_CLK=y
    +CONFIG_COMMON_CLK_SI5351=n
    +CONFIG_COMMON_CLK_SI514=n
    +CONFIG_COMMON_CLK_SI570=n
    +CONFIG_COMMON_CLK_CDCE925=n
    +CONFIG_CLK_TWL6040=n
    +CONFIG_CLK_QORIQ=n
    +CONFIG_COMMON_CLK_PALMAS=y
    +CONFIG_COMMON_CLK_PWM=n
    +CONFIG_COMMON_CLK_PXA=n
    +CONFIG_COMMON_CLK_CDCE706=n
    +CONFIG_CLKSRC_OF=y
    +CONFIG_CLKSRC_PROBE=y
    +CONFIG_CLKSRC_MMIO=y
    +CONFIG_CLKSRC_TI_32K=y
    +CONFIG_ARM_ARCH_TIMER=y
    +CONFIG_ARM_ARCH_TIMER_EVTSTREAM=y
    +CONFIG_ARM_GLOBAL_TIMER=n
    +CONFIG_ARM_TIMER_SP804=n
    +CONFIG_OMAP_RESET_CLOCKS=y
    +CONFIG_CPU_IDLE_GOV_LADDER=y
    +CONFIG_CPU_IDLE_GOV_MENU=y
    +CONFIG_DT_IDLE_STATES=y
    +CONFIG_ARM_CPUIDLE=y
    +CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED=n
    +CONFIG_CPU_FREQ_GOV_COMMON=y
    +CONFIG_CPU_FREQ_STAT=y
    +CONFIG_CPU_FREQ_STAT_DETAILS=y
    +CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
    +CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
    +CONFIG_CPU_FREQ_GOV_POWERSAVE=y
    +CONFIG_CPU_FREQ_GOV_USERSPACE=y
    +CONFIG_CPU_FREQ_GOV_ONDEMAND=y
    +CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
    +CONFIG_CPUFREQ_DT=y
    +CONFIG_ARM_BIG_LITTLE_CPUFREQ=n
    +CONFIG_ARM_KIRKWOOD_CPUFREQ=n
    +CONFIG_ARM_OMAP2PLUS_CPUFREQ=n
    +CONFIG_ARM_TI_CPUFREQ=y
    +CONFIG_QORIQ_CPUFREQ=n
    +CONFIG_SOC_BRCMSTB=n
    +CONFIG_SUNXI_SRAM=n
    +CONFIG_PM_DEVFREQ=n
    +CONFIG_PINCTRL_TI_IODELAY=y
    +CONFIG_TI_MESSAGE_MANAGER=y
    +CONFIG_TI_SCI_PROTOCOL=y
    +CONFIG_TI_SCI_PM_DOMAINS=y
    +CONFIG_TI_SCI_CLK=y
     CONFIG_POSIX_MQUEUE=y
    +CONFIG_OPP_DOMAIN_TI=y
    +CONFIG_RESET_CONTROLLER=y
    +CONFIG_RESET_TI_SYSCON=y
    +CONFIG_RESET_TI_SCI=y
    +CONFIG_TIMER_STATS=n
    +CONFIG_DEBUG_SPINLOCK=n
    +CONFIG_DEBUG_MUTEXES=n
    +CONFIG_DEBUG_LOCK_ALLOC=n
    +CONFIG_PROVE_LOCKING=n
    +CONFIG_LOCKDEP=n
    +CONFIG_STACKTRACE=n
    +CONFIG_ARM_UNWIND=n
    +CONFIG_DEBUG_PREEMPT=n
    +CONFIG_SLUB_DEBUG=n
    +CONFIG_LOCKUP_DETECTOR=n
    +CONFIG_DETECT_HUNG_TASK=n
    +CONFIG_IRQ_DOMAIN_DEBUG=n
    +CONFIG_DEBUG_BUGVERBOSE=n
     CONFIG_NO_HZ=y
     CONFIG_HIGH_RES_TIMERS=y
     CONFIG_BSD_PROCESS_ACCT=y
    @@ -21,7 +129,6 @@ CONFIG_BLK_DEV_INITRD=y
     CONFIG_KALLSYMS_ALL=y
     CONFIG_EMBEDDED=y
     CONFIG_PERF_EVENTS=y
    -# CONFIG_SLUB_DEBUG is not set
     CONFIG_MODULES=y
     CONFIG_MODULE_FORCE_LOAD=y
     CONFIG_MODULE_UNLOAD=y
    @@ -30,12 +137,12 @@ CONFIG_MODVERSIONS=y
     CONFIG_MODULE_SRCVERSION_ALL=y
     CONFIG_PARTITION_ADVANCED=y
     CONFIG_CMDLINE_PARTITION=y
    -CONFIG_OMAP_RESET_CLOCKS=y
    -CONFIG_SOC_AM33XX=y
     CONFIG_SOC_HAS_OMAP2_SDRC=y
     CONFIG_PCI_ENDPOINT=y
     CONFIG_PCI_ENDPOINT_CONFIGFS=y
     CONFIG_PCI_EPF_TEST=y
    +CONFIG_PREEMPT=y
    +CONFIG_PREEMPT_RT_BASE=y
     CONFIG_PREEMPT_RT_FULL=y
     CONFIG_HZ_1000=y
     CONFIG_CMA=y
    @@ -46,17 +153,15 @@ CONFIG_ZBOOT_ROM_BSS=0x0
     CONFIG_ARM_APPENDED_DTB=y
     CONFIG_ARM_ATAG_DTB_COMPAT=y
     CONFIG_KEXEC=y
    +CONFIG_EDAC=y
    +CONFIG_EDAC_TI=m
     CONFIG_EFI=y
    -CONFIG_CPU_FREQ=y
    -CONFIG_CPU_FREQ_GOV_POWERSAVE=y
    -CONFIG_CPU_FREQ_GOV_USERSPACE=y
    -CONFIG_CPU_FREQ_GOV_ONDEMAND=y
    -CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
    -CONFIG_CPUFREQ_DT=y
    -CONFIG_ARM_TI_CPUFREQ=y
    -CONFIG_CPU_IDLE=y
    -CONFIG_ARM_CPUIDLE=y
    -CONFIG_PM_DEBUG=y
    +CONFIG_CPU_FREQ=n
    +CONFIG_SUSPEND=n
    +CONFIG_HOTPLUG_CPU=y
    +CONFIG_RCU_NOCB_CPU=y
    +CONFIG_RCU_NOCB_CPU_ALL=y
    +CONFIG_CPU_IDLE=n
     CONFIG_NET=y
     CONFIG_PACKET=y
     CONFIG_UNIX=y
    @@ -346,6 +451,7 @@ CONFIG_PINCTRL_AMD=y
     CONFIG_PINCTRL_MCP23S08=y
     CONFIG_PINCTRL_SINGLE=y
     CONFIG_PINCTRL_PALMAS=y
    +CONFIG_GPIOLIB=y
     CONFIG_DEBUG_GPIO=y
     CONFIG_GPIO_SYSFS=y
     CONFIG_GPIO_SYSCON=y
    @@ -565,6 +671,7 @@ CONFIG_RTC_DRV_DS1307=m
     CONFIG_RTC_DRV_PALMAS=m
     CONFIG_RTC_DRV_TPS6586X=m
     CONFIG_RTC_DRV_TPS65910=m
    +CONFIG_RTC_DRV_TWL92330=y
     CONFIG_RTC_DRV_RX8010=y
     CONFIG_RTC_DRV_OMAP=m
     CONFIG_DMADEVICES=y
    @@ -573,16 +680,23 @@ CONFIG_DW_DMAC=y
     CONFIG_TI_CPPI41=y
     CONFIG_UIO=m
     CONFIG_VIRTIO_MMIO=y
    -CONFIG_COMMON_CLK_PALMAS=y
     CONFIG_HWSPINLOCK=y
     CONFIG_HWSPINLOCK_OMAP=y
     CONFIG_MAILBOX=y
     CONFIG_OMAP2PLUS_MBOX=y
    +CONFIG_IOMMU_SUPPORT=y
     CONFIG_OMAP_IOMMU=y
     CONFIG_OMAP_IOMMU_DEBUG=y
     CONFIG_REMOTEPROC=m
    +CONFIG_OMAP_REMOTEPROC=m
    +CONFIG_KEYSTONE_REMOTEPROC=m
    +CONFIG_DA8XX_REMOTEPROC=m
    +CONFIG_PRUSS_REMOTEPROC=m
     CONFIG_WKUP_M3_RPROC=m
     CONFIG_RPMSG_VIRTIO=m
    +CONFIG_RPMSG_PROTO=m
    +CONFIG_RPMSG_RPC=m
    +CONFIG_RPMSG_PRU=m
     CONFIG_SOC_TI=y
     CONFIG_AMX3_PM=m
     CONFIG_WKUP_M3_IPC=m
    @@ -603,8 +717,6 @@ CONFIG_IIO_HRTIMER_TRIGGER=y
     CONFIG_PWM_FSL_FTM=m
     CONFIG_PWM_TIECAP=y
     CONFIG_PWM_TIEHRPWM=y
    -CONFIG_RESET_CONTROLLER=y
    -CONFIG_RESET_TI_SYSCON=y
     CONFIG_OMAP_USB2=y
     CONFIG_TI_PIPE3=y
     CONFIG_PHY_AM654_MMC=y
    @@ -649,8 +761,6 @@ CONFIG_NLS_UTF8=y
     CONFIG_PRINTK_TIME=y
     CONFIG_MAGIC_SYSRQ=y
     CONFIG_SCHEDSTATS=y
    -# CONFIG_DEBUG_PREEMPT is not set
    -# CONFIG_DEBUG_BUGVERBOSE is not set
     CONFIG_IRQSOFF_TRACER=y
     CONFIG_PREEMPT_TRACER=y
     CONFIG_SCHED_TRACER=y

  • Hi,

    If I am following correctly I believe the cout statements inside a high priority thread may have been the issue. As you noted running at the previous priority levels you are above the kernel and IRQs. Therefore calling any print statement might cause the issue that you were experiencing.

    I looked through the config diff and I have a hard following it since some options are removed and then added back. Overall the key advice is that hotplug and cpu freq support are disabled in the TI kernel config. The reason they are disabled is due to these features are latency adders. One other recommendation is to always watch the system latency as you develop your application to make sure it meets your system requirements. So as you add or remove kernel config options always run cylictest to make sure the system latency is what you require.

    Best Regards,

    Schuyler

  • Thank you very much. Now the system latency is ok.

    The change we have made are:

    -Removing printf

    -Making priority lower than kthreads and IRQs. 

    -Disabling  hotplug and cpu freq support in kernel config

    -Replace usleep with nanosleep in our real-time threads.

    BR.

    Lorenzo.

  • Hi,

    That is great news! Thank you for letting us know that you are able to get the performance you need.

    Best Regards,

    Schuyler