This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM4376: I/O memory Error with AM437X SDK 7.03 Linux 5.4

Part Number: AM4376
Other Parts Discussed in Thread: ASH, AM4372

I'm porting a kernel module developed with AM437X SDK 2.00 Linux 4.1.6 that, among other things, simply blinks a LED attached to gpio5 pin 0 every second using a kernel timer. It has worked fine for 5 years and uses what I think is the standard kernel I/O memory access scheme: ioremap to get the gpio5 clear and set data registers virtual addresses and iowrite32 to write 1 in those registers to change the state of the gpio pin. When I print the registers' addresses, I get 0xFA322190 and 0xFA322194 which look reasonable for hardware addresses 0x48322190 and 0x48322194.

When I built this module with SDK 7.03 Linux 5.4, I only changed the timer initialization from the old style init_timer to the new style timer_setup. All the remaining code is identical and with printk's in the timer function, I can see that the timer is working fine with both SDK 2.00 and SDK 7.03 and being called every second as expected. I also see the same virtual addresses for the gpio5 registers with both SDKs. Further, I can successfully blink the LED in a "for" loop in the module initialization function with either SDK.

BUT…when the timer fires with SDK 7.03 the LED does not change states and I get the error:

44000000.ocp:L3 Custom Error: MASTER M2 (64-bit) TARGET L4_PER_0 (Idle): Data Access in Supervisor mode during Functional access

The timer function is:

static void HeartbeatFunc(struct timer_list *arg)

{

                static int LedState=0;     

                if(LedState)

                {

                                printk(KERN_INFO "HeartbeatLed off GPIO5SetDataOutVA=0x%x",GPIO5SetDataOutVA);

                                iowrite32(1,GPIO5SetDataOutVA);

                }else{

                                printk(KERN_INFO "HeartbeatLed off GPIO5ClrDataOutVA=0x%x",GPIO5ClrDataOutVA);

                                iowrite32(1,GPIO5ClrDataOutVA);

                }

                LedState ^= 1;

                If(arg != NULL) mod_timer(&MyTimerList, jiffies + HZ);

}

The module initialization function is (greatly abbreviated)

#define GPIO5_BASE  0x48322000

#define GPIO_CLRDATAOUT_OFFSET 0x190

#define GPIO_SETDATAOUT_OFFSET 0x194

 

static int __init MyDriver_init(void)

{

                 int Result;

                 GPIO5BaseVA = ioremap(GPIO5_BASE, 1024);

                 if(GPIO5BaseVA == NULL)

                  {

                                printj(__FILE__,__LINE__,"Can't map gpio5 io memory in driver %s\n",Driver_Name);

                    }else{

                                printk(KERN_INFO "Mapped gpio5 base to 0x%x\n",GPIO5BaseVA);

                               // calculate the addresses of the gpio5 data control registers for later use

                               GPIO5ClrDataOutVA= (void __iomem *)((unsigned int)GPIO5BaseVA+(GPIO_CLRDATAOUT_OFFSET));

                               GPIO5SetDataOutVA= (void __iomem *)((unsigned int)GPIO5BaseVA+(GPIO_SETDATAOUT_OFFSET));

                               printk(KERN_INFO "Mapped gpio5 GPIO5ClrDataOutVA to 0x%x",GPIO5ClrDataOutVA);

                               printk(KERN_INFO "Mapped gpio5 GPIO5SetDataOutVA to 0x%x",GPIO5SetDataOutVA);

           

                             // toggle the led to show it's working

                    printk(KERN_INFO "Test Heartbeat Start\n");

                    for(i=0;i<10;++i)

                    {

                                HeartbeatFunc(NULL);

                                msleep(500);

                    }

                    printk(KERN_INFO "Test Heartbeat Done\n");

                                    

                    timer_setup(&MyTimerList,HeartbeatFunc,0);

                    mod_timer(&MyTimerList, jiffies + HZ);           

        }

    return Result;

}

 Any thoughts on why this error occurs and how to fix it?

  • UPDATE: I now believe this is a problem with the device tree. I tried building the module as part of the kernel and it fails as described above, but when built as a module and installed after boot up, it works as expected without errors.

  • UPDATE2: I'm now using the am437x-gp-evm.dts slightly modified so that gpio5_0 is pin H22 and the uart3 that previously used H22 is removed. My heartbeat LED is on pin H22 and the heartbeat module described in the op is built into the kernel. The LED flashes as expected at bootup when the module is initialized, and continues working for about 20 seconds after the full bootup and login prompt. At that point I get the dreaded "44000000.ocp:L3 Custom Error: MASTER M2 (64-bit) TARGET L4_PER_0 (Idle): Data Access in Supervisor mode during Functional access" error and the LED stops flashing. The heartbeat timer, however, keeps firing and I get the same error message every second.

    The complete error trace below seems to indicate the problem is with mod_timer in the timer expiration routine. Is there some locking required when using mod_timer, or is there another way to reload the timer so that it fires repetitively? Thanks.

    [ 31.763430] ------------[ cut here ]------------
    [ 31.768089] WARNING: CPU: 0 PID: 0 at drivers/bus/omap_l3_noc.c:141 l3_interrupt_handler+0x28c/0x384
    [ 31.777262] 44000000.ocp:L3 Custom Error: MASTER M2 (64-bit) TARGET L4_PER_0 (Idle): Data Access in Supervisor mode during Functional access
    [ 31.789920] Modules linked in:
    [ 31.792990] CPU: 0 PID: 0 Comm: swapper Tainted: G W 5.4.106-dirty #57
    [ 31.800851] Hardware name: Generic AM43 (Flattened Device Tree)
    [ 31.806807] [<c010eb08>] (unwind_backtrace) from [<c010b2f0>] (show_stack+0x10/0x14)
    [ 31.814593] [<c010b2f0>] (show_stack) from [<c096c420>] (__warn+0xd0/0xe8)
    [ 31.821505] [<c096c420>] (__warn) from [<c096c4d0>] (warn_slowpath_fmt+0x98/0xc8)
    [ 31.829026] [<c096c4d0>] (warn_slowpath_fmt) from [<c047e4c8>] (l3_interrupt_handler+0x28c/0x384)

    [ 31.837943] [<c047e4c8>] (l3_interrupt_handler) from [<c016d7e4>] (__handle_irq_event_percpu+0x50/0x13c)
    [ 31.847466] [<c016d7e4>] (__handle_irq_event_percpu) from [<c016d900>] (handle_irq_event_percpu+0x30/0x8c)
    [ 31.857164] [<c016d900>] (handle_irq_event_percpu) from [<c016d9a8>] (handle_irq_event+0x4c/0x8c)
    [ 31.866079] [<c016d9a8>] (handle_irq_event) from [<c0171828>] (handle_fasteoi_irq+0xb4/0x184)
    [ 31.874649] [<c0171828>] (handle_fasteoi_irq) from [<c016c9bc>] (generic_handle_irq+0x24/0x34)
    [ 31.883301] [<c016c9bc>] (generic_handle_irq) from [<c016d134>] (__handle_domain_irq+0x54/0xa4)
    [ 31.892039] [<c016d134>] (__handle_domain_irq) from [<c047ced8>] (gic_handle_irq+0x3c/0x68)

    [ 31.900431] [<c047ced8>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0xa8)
    [ 31.907944] Exception stack(0xc1201d78 to 0xc1201dc0)
    [ 31.913015] 1d60: c1289238 c1201dd8
    [ 31.921230] 1d80: 00000000 c1201dc8 c1200000 c1289238 c1201dd8 c1201e2c c1201e2c c1289238
    [ 31.929445] 1da0: c07d28d4 c12159d4 c1215a14 c1201dc8 c01859d0 c0184660 600f0113 ffffffff
    [ 31.937662] [<c0101a8c>] (__irq_svc) from [<c0184660>] (lock_timer_base+0x18/0x90)
    [ 31.945267] [<c0184660>] (lock_timer_base) from [<c01859d0>] (mod_timer+0x1e8/0x3b4)
    [ 31.953046] [<c01859d0>] (mod_timer) from [<c0184958>] (call_timer_fn.constprop.0+0x24/0x98)

    [ 31.961524] [<c0184958>] (call_timer_fn.constprop.0) from [<c0184ea0>] (run_timer_softirq+0x4d4/0x554)
    [ 31.970872] [<c0184ea0>] (run_timer_softirq) from [<c01022ac>] (__do_softirq+0xec/0x278)
    [ 31.979003] [<c01022ac>] (__do_softirq) from [<c012e44c>] (irq_exit+0xdc/0xe0)
    [ 31.986259] [<c012e44c>] (irq_exit) from [<c016d138>] (__handle_domain_irq+0x58/0xa4)
    [ 31.994126] [<c016d138>] (__handle_domain_irq) from [<c047ced8>] (gic_handle_irq+0x3c/0x68)
    [ 32.002517] [<c047ced8>] (gic_handle_irq) from [<c0101a8c>] (__irq_svc+0x6c/0xa8)

    [ 32.010029] Exception stack(0xc1201f00 to 0xc1201f48)
    [ 32.015104] 1f00: 00000000 c1277a84 00000001 c1200000 dbea1800 00000001 c1244280 653f06e4
    [ 32.023319] 1f20: 653ef867 00000007 00000007 00000000 00000000 c1201f50 c0196e68 c07649bc
    [ 32.031529] 1f40: 200f0013 ffffffff
    [ 32.035038] [<c0101a8c>] (__irq_svc) from [<c07649bc>] (cpuidle_enter_state+0x80/0x3a0)
    [ 32.043080] [<c07649bc>] (cpuidle_enter_state) from [<c0764d18>] (cpuidle_enter+0x28/0x38)
    [ 32.051382] [<c0764d18>] (cpuidle_enter) from [<c015448c>] (do_idle+0x18c/0x238)
    [ 32.058813] [<c015448c>] (do_idle) from [<c0154820>] (cpu_startup_entry+0xc/0x14)
    [ 32.066332] [<c0154820>] (cpu_startup_entry) from [<c0e00d8c>] (start_kernel+0x43c/0x470)
    [ 32.074543] ---[ end trace 56c88ab7926201c2 ]---

  • Hi 

    At that point I get the dreaded "44000000.ocp:L3 Custom Error: MASTER M2 (64-bit) TARGET L4_PER_0 (Idle): Data Access in Supervisor mode during Functional access" error and the LED stops flashing. The heartbeat timer, however, keeps firing and I get the same error message every second.

    If you modify your driver to not access the GPIO in the timer handler but just add a printk, do you see if the omap_l3_noc error still happens in every second? I want to see if the issue is in the timer or the GPIO.

  • Thanks for your reply and suggestion. I replaced the iowrite32 with printk and the errors stopped and I got the prints every second.

    I subsequently found two problems. First, in my module I had some request_mem_region calls related to gpio registers. I was ignoring their returns as I wasn't doing anything to the memory except writing to the output set or clear register. That was fine with SDK 2.00 Linux 4.1.6 because the the TI driver was being loaded before my module. But in SDK 7.03 Linux 5.4, my module is being loaded before the TI driver and the TI driver checked its own request_mem_region and aborted when it discovered the region was already in use. I fixed that by releasing the mem region after my module initialized. This was a good step but didn't fix everything; I still got the error.

    So then I switched to an almost bare device tree and added everything back in one component at a time and tested each addition. I found that the error stopped when I changed 

    &gpio5 {
    pinctrl-names = "default";
    pinctrl-0 = <&gpio5_pins>;
    status = "okay";
    ti,no-reset-on-init;
    };

    to

    &gpio5 {
    pinctrl-names = "default";
    pinctrl-0 = <&gpio5_pins>;
    status = "okay";
    ti,no-reset-on-init;

    p0 {
       gpio-hog;
       gpios = <0 GPIO_ACTIVE_HIGH>;
       output-high;
       line-name = "HeartbeatLED";
    };
    };

    The "gpio-hog" wasn't necessary with the old SDK and Linux, but seems to be required now. I'm not quite finished with testing so I'm not declaring this resolved just yet. Any comments will be appreciated.

  • Hi Mark,

    I have an theory about why the issue happens, but before I explain it, can you please check your dts that besides this GPIO5 pin for the heartbeat LED, no any other GPIO5 pins are used, correct?

  • Hi. My kernel module only uses 1 pin but we have 20 gpio5 pins defined in the dts. I tried having the module output on pin 4 while still having the "hog pin 0" in the dts, and that lit an led without errors. And I tried changing to "hog pin 7" while using the original pin 0 for output, and that worked, too. I tried commenting out the "hog" structure leaving everything else the same, and the error returned.

    It's been five years since I worked on a device tree, so I really appreciate your help.

  • Mark,

    How are the 20 gpio5 defined in dts? Just defined their pinmux, or any of them is referenced as this gpio5 pin in the p0 gpio-hog?

    The theory of the issue I have is that without this new p0 hog entry in DTS, you don't have any other gpio5 pin is referenced. Then the gpio5 module will be in clock-gated mode after kernel booted (because kernel knows nobody uses this module...), then your timer/LED driver uses ioremap to directly accessing gpio5 registers causes the error.

  • Nice, that sounds plausible. All the gpios are only defined by the pinmux and not used anywhere else. My kernel module enables the clocks by writing to the PRCM_CM_PER_GPIO5_CLKCTRL register, so when my module got loaded last by the SDK 2.00 Linux 4.1.6 everything was fine. With SDK 7.03 Linux 5.4, the TI driver is loaded last and disables the clock.

    Other than the gpio hog hack, is there a preferred way to ensure the clock is enabled? Thanks.

  • I vaguely remember there is an DT flag to tell kernel to not disable the unused module. Let me see if I can find it.

  • I tried another test by removing the gpio hog from the DT and changing the timer interrupt routine so that it wrote to the gpio5 clock control register on every interrupt before it wrote to the set/clear data register. That also worked, so I'm pretty sure your explanation is correct. Thanks a lot for the help. Eventually, I'll rewrite the module to use more modern techniques to control the gpio pins.

    I also had a problem with the USB and found the script you had posted to diagnose USB problems. That was very helpful as well and by using a modified version, I was able to get the USB working. Unfortunately, the original script is for the bash shell and won't work with busybox, which uses the ash shell. Here's the part I extracted from your script to run on a 437x with busybox. Not as elegant as your original, but was enough to help me fix my USB problem.

    Thanks again.

    #!/bin/sh
    
    find -L /proc/device-tree -name 'usb*'
    lsusb
    lst='USB_DWC3 USB_DWC3_DUAL_ROLE USB_OTG USB_DWC3_OMAP USB_XHCI_HCD OMAP_CONTROL_PHY OMAP_USB2'
    for s in $lst ; do
       zcat /proc/config.gz | grep "$s"
    done
    USB0DT="$(find -L /proc/device-tree/ocp@44000000/interconnect@48000000/segment@300000/target-module@80000/omap_dwc3@0 -name 'usb@10000')"
    USB1DT="$(find -L /proc/device-tree/ocp@44000000/interconnect@48000000/segment@300000/target-module@c0000/omap_dwc3@0 -name 'usb@10000')"
    echo USB0DT=$USB0DT
    echo USB1DT=$USB1DT
    
    lst="${USB0DT} ${USB1DT}"
    for _usb_dir in $lst; do
        [ -n "$_usb_dir" ] || continue
        echo Directory $_usb_dir
        [ -f "$_usb_dir/status" ] &&
            _status=`tr -d '\0' <$_usb_dir/status` ||
            _status='(enabled)'
        _dr_mode=`tr -d '\0'  <$_usb_dir/dr_mode`
        echo `basename $_usb_dir`: $_dr_mode, $_status
    
        [ "$_status" = "disabled" -o "$_dr_mode" = "host" ] || gadget_mode=true
    done

  • Hi Mark,

    Somehow I missed your update...

    I am unable to find the DT flag I mentioned to prevent disabling an unused module. Maybe I mixed it up with the DT flag to prevent module reset in kernel init (to preserve the status inherited from U-Boot).

    Anyway, I don't think the way you use 'gpio-hog' is a hack. It is used in many places in kernel. I am sure you know your current GPIO/LED driver is a hack, which mmap GPIO registers and directly manipulate them. Hopefully you will have time to rewrite the driver in a "formal" way, using kenrel gpiod framework to query and control a GPIO pin.

    And thanks for the feedback on the USB diagnosis script. I am glad it is helpful. When I get some time, I will review your script and update mine to make it compatible with busybox sh. Appreciate it.

  • Another update. I replaced my old GPIO driver with a new one, making it a platform driver and using the gpiolib to interact with the TI gpio chip driver. No more directly writing to hardware registers!

    Coincidently, my new driver initially was affected by exactly the same issue: it was being loaded before the TI driver. When I did a "gpiod_get" on my heartbeat led, it failed because the gpio chip driver hadn't yet been loaded. I fixed it by having my probe function return –EPROBE_DEFER if the gpiod_get failed. That allowed the TI driver to load first and when the kernel got around to probing my driver again, the gpiod_get succeeded. BTW, this version works without requiring the gpio hog declaration in the device tree.

    In case anyone else has this problem, here's what I added. To the device tree in am4372.dtsi

    DaqGPIO: DaqGPIO {

                    compatible = "daq,gpio";

    };

    In my dts file

    &DaqGPIO {

                    heartbeat-gpios = <&gpio5 28 GPIO_ACTIVE_HIGH>; /* heartbeat led */

                    FacReset-gpios = <&gpio5 30 GPIO_ACTIVE_HIGH>;       

    };

    In my driver probe function (with some error handling omitted):

    HeartBeatDesc = devm_gpiod_get(&pdev->dev,"heartbeat",GPIOD_OUT_HIGH); if(IS_ERR(HeartBeatDesc))  return -EPROBE_DEFER;

    I am still confused by two things.

    First, if I do a gpiod_put on a gpio_desc, the associated gpio line changes to its idle state rather than retaining the state I last assigned using gpiod_set_value. I assume this is by design, but is not intuitive.

    Second, I'm successfully using the FacReset-gpios in the above device tree as an interrupt input, assigned via devm_request_irq. I'm surprised that I didn't have to make any mention of it as an interrupt in the device tree. I'm guessing this is because all the gpio5 interrupts are handled by one interrupt declared under target-modules@22000 as interrupts = <GIC_SPI 148 IRQ_TYPE_LEVEL_HIGH>  in am437x-l4.dtsi.

    Regards, and thanks again for your help in resolving this.

  • Hi Mark,

    Glad yo see you implemented it so quickly! With this proper implementation, future maintenance work should be much easier.

    HeartBeatDesc = devm_gpiod_get(&pdev->dev,"heartbeat",GPIOD_OUT_HIGH); if(IS_ERR(HeartBeatDesc))  return -EPROBE_DEFER;

    I think your _probe() function should return -EPROBE_DEFER only when the return code of devm_gpiod_get() is -EPROBE_DEFER too. I checked the code of devm_gpiod_get() which could return other error codes.

    if(IS_ERR(HeartBeatDesc) == -EPROBE_DEFER)
            return -EPROBE_DEFER;

    Or maybe better just return the error code which devm_gpiod_get() returned.

    First, if I do a gpiod_put on a gpio_desc, the associated gpio line changes to its idle state rather than retaining the state I last assigned using gpiod_set_value. I assume this is by design, but is not intuitive.

    gpiod_put() is only needed when the gpio_desc is no longer used. So it should only be called in error path or driver _remove().

    Since you use devm_* (devm_gpiod_get()), you don't even need to call gpiod_put() at all in anywhere of the driver, the device driver core will take care of cleaning up the resources allocated during dev_gpiod_get().

    I don't know enough about GPIO and kernel GPIO framework to answer your second question about GPIO interrupt.