This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/TDA2PXEVM: System crash based on visionSDK 03 05

Part Number: TDA2PXEVM

Tool/software: Linux

Hello,

When we power up the board, sometimes, system will crash, our board  based on visionSDK 03 05. We suspect that the temperature of the board is too high, but I'm not sure. Please find the log in the attachment. Could you help to have a look?

Thanks a lot

Terence

5076.crash.txt

  • Hi Terence,

    Can you please indicate the frequency at which you're running the EVM? If you're able to reach kernel prompt can you please run the below command and attach the output?

    #ompaconf show opp

    Also can you try to disable DSP-1 and DSP-2 from the kernel device-tree and see if the issue still exists.

    Regards
    Shravan
  • Hi  Shravan,

    The frequency is about thirty percent.When it reappears, I have run the command "ompaconf show opp".Please find the log in the attachment.

    Thanks a lot

    Terence

    ompaconf_show_opp_log.txt
    [   10.431293] ------------[ cut here ]------------
    [   10.431350]  remoteproc3: remote processor 41000000.dsp is now up
    [   10.442053] WARNING: CPU: 0 PID: 236 at drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x25c/0x368()
    [   10.451569] 44000000.ocp:L3 Custom Error: MASTER DSP1_MDMA TARGET L4_PER3_P3 (Read): Data Access in User mode during Functional access
    [   10.463699] Modules linked in: udc_core virtio_rpmsg_bus(+) bc_example(O) pvrsrvkm(O) ahci_platform libahci_platform libahci libata omap_aes_driver omap_rng extcon_usb_gpio scsi_mod rng_core omap_sham omap_des dwc3_omap extcon omap_remoteproc remoteproc virtio virtio_ring sch_fq_codel
    [   10.489141] CPU: 0 PID: 236 Comm: systemd-udevd Tainted: G           O    4.4.84 #1
    [   10.496825] Hardware name: Generic DRA74X (Flattened Device Tree)
    [   10.502937] Backtrace: 
    [   10.505400] [<c0013b1c>] (dump_backtrace) from [<c0013d18>] (show_stack+0x18/0x1c)
    [   10.512995]  r7:c02e0130 r6:20070193 r5:00000000 r4:c0953590
    [   10.518701] [<c0013d00>] (show_stack) from [<c02b39a8>] (dump_stack+0x8c/0xa0)
    [   10.525951] [<c02b391c>] (dump_stack) from [<c003581c>] (warn_slowpath_common+0x88/0xb8)
    [   10.534070]  r7:c02e0130 r6:00000093 r5:00000009 r4:dbdd3ad8
    [   10.539772] [<c0035794>] (warn_slowpath_common) from [<c0035884>] (warn_slowpath_fmt+0x38/0x40)
    [   10.548502]  r8:00000017 r7:c0848d0c r6:00000002 r5:c0848944 r4:c08489e8
    [   10.555256] [<c0035850>] (warn_slowpath_fmt) from [<c02e0130>] (l3_interrupt_handler+0x25c/0x368)
    [   10.564159]  r3:df20ce40 r2:c08489e8
    [   10.567753]  r4:80080003
    [   10.570299] [<c02dfed4>] (l3_interrupt_handler) from [<c0082048>] (handle_irq_event_percpu+0xb4/0x160)
    [   10.579639]  r10:c098308b r9:df20e240 r8:00000017 r7:00000000 r6:00000000 r5:df20e2a0
    [   10.587525]  r4:df21f3c0
    [   10.590071] [<c0081f94>] (handle_irq_event_percpu) from [<c0082134>] (handle_irq_event+0x40/0x64)
    [   10.598974]  r10:00000000 r9:d14d4000 r8:df008000 r7:00000000 r6:c093bc84 r5:df20e2a0
    [   10.606860]  r4:df20e240
    [   10.609404] [<c00820f4>] (handle_irq_event) from [<c0085478>] (handle_fasteoi_irq+0xc0/0x194)
    [   10.617958]  r7:00000000 r6:c093bc84 r5:df20e2a0 r4:df20e240
    [   10.623661] [<c00853b8>] (handle_fasteoi_irq) from [<c0081674>] (generic_handle_irq+0x2c/0x3c)
    [   10.632303]  r7:00000000 r6:00000000 r5:00000017 r4:c093140c
    [   10.638005] [<c0081648>] (generic_handle_irq) from [<c008194c>] (__handle_domain_irq+0x64/0xbc)
    [   10.646737] [<c00818e8>] (__handle_domain_irq) from [<c00094f0>] (gic_handle_irq+0x40/0x7c)
    [   10.655117]  r9:d14d4000 r8:fa213000 r7:fa212000 r6:dbdd3c98 r5:fa21200c r4:c09368e4
    [   10.662919] [<c00094b0>] (gic_handle_irq) from [<c0014854>] (__irq_svc+0x54/0x90)
    [   10.670428] Exception stack(0xdbdd3c98 to 0xdbdd3ce0)
    [   10.675495] 3c80:                                                       e49af200 d14d3200
    [   10.683703] 3ca0: 00000d80 666e7265 6c655f73 735f6d65 696c6d79 65006b6e 74617272 d14d4000
    [   10.691911] 3cc0: 00000000 dbdd3d3c 615f6d75 dbdd3cec 375f3531 c02b24a8 20070013 ffffffff
    [   10.700116]  r9:d14d4000 r8:74617272 r7:dbdd3ccc r6:ffffffff r5:20070013 r4:c02b24a8
    [   10.707919] [<c02c4f98>] (copy_to_iter) from [<c02c5454>] (copy_page_to_iter+0x1b4/0x3bc)
    [   10.716124]  r10:00000000 r9:dbdd2000 r8:d5d35070 r7:0000009d r6:d14d3000 r5:dbdd3e14
    [   10.724008]  r4:00001000
    [   10.726555] [<c02c52a0>] (copy_page_to_iter) from [<c00d3580>] (generic_file_read_iter+0x270/0x67c)
    [   10.735632]  r10:00001000 r9:dbdd2000 r8:d5d35070 r7:0000009d r6:dbf33480 r5:d5d35154
    [   10.743519]  r4:df8b2dac
    [   10.746063] [<c00d3310>] (generic_file_read_iter) from [<c011ffe4>] (__vfs_read+0xb4/0xdc)
    [   10.754356]  r10:00000000 r9:00000000 r8:000fcb48 r7:00000000 r6:00000000 r5:dbdd3ea0
    [   10.762242]  r4:dbf33480
    [   10.764785] [<c011ff30>] (__vfs_read) from [<c01207a4>] (vfs_read+0x80/0x10c)
    [   10.771945]  r9:dbdd3f44 r8:dbf33480 r7:00000000 r6:dbdd3ea0 r5:dbf33480 r4:e4912000
    [   10.779745] [<c0120724>] (vfs_read) from [<c0125be0>] (kernel_read+0x40/0x54)
    [   10.786905]  r9:dbdd3f44 r8:dbf33480 r7:00000000 r6:00000000 r5:bf000000 r4:ffffe000
    [   10.794708] [<c0125ba0>] (kernel_read) from [<c00aa38c>] (copy_module_from_fd+0xe8/0x114)
    [   10.802914]  r5:dbf33480 r4:00000000
    [   10.806510] [<c00aa2a4>] (copy_module_from_fd) from [<c00adaa4>] (SyS_finit_module+0x70/0x98)
    [   10.815064]  r9:dbdd2000 r8:c000fe24 r7:0000017b r6:b6db2d90 r5:0000000f r4:00000000
    [   10.822868] [<c00ada34>] (SyS_finit_module) from [<c000fc60>] (ret_fast_syscall+0x0/0x3c)
    [   10.831073]  r6:00000000 r5:0051d928 r4:00000000
    [   10.835720] ---[ end trace 600f2174281f2c31 ]---
    [   10.843805] virtio_rpmsg_bus virtio2: rpmsg host is online
    [   10.850580]  remoteproc1: powering up 55020000.ipu
    [   10.864119] net eth1: initializing cpsw version 1.15 (0)
    [   10.869479] net eth0: initialized cpsw ale version 1.4
    [   10.880203] net eth0: ALE Table size 1024
    [  OK  ] Started Network Service.[   10.887142]  remoteproc1: Booting fw image dra7-ipu2-fw.xem4, size 9392416
    
    [   10.892307] libphy: PHY 48485000.mdio:03 not found
    [   10.892312] net eth1: phy "48485000.mdio:03" not found on slave 1, err -19
    [   10.909477] omap-iommu 55082000.mmu: 55082000.mmu: version 2.1
    [   10.915181] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    [   10.933316] net eth0: initializing cpsw version 1.15 (0)
    [   10.946400]  remoteproc1: remote processor 55020000.ipu is now up
    [   10.953807] virtio_rpmsg_bus virtio3: rpmsg host is online
    [   11.006651] virtio_rpmsg_bus virtio3: creating channel rpmsg-proto addr 0x50
    [   11.024279] virtio_rpmsg_bus virtio2: creating channel rpmsg-proto addr 0x50
    [   11.036627] virtio_rpmsg_bus virtio0: creating channel rpmsg-proto addr 0x50
    [   11.050203] virtio_rpmsg_bus virtio1: creating channel rpmsg-proto addr 0x50
    [   11.060931] NET: Registered protocol family 41
    [   11.095263] net eth0: phy found : id is : 0x1ccb00
    [   11.106208] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
    [  OK  ] Reached target Network.
             Starting Network Name Resolution...
    [  OK  ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.
             Starting LSB: Dropbear Secure Shell server...
             Starting weston.service...
    [  OK  ] Started Serial Getty on ttyS0.
    [  OK  ] Started Getty on tty1.
    [  OK  ] Started Network Name Resolution.
    [  OK  ] Started Login Service.
    [  OK  ] Started weston.service.
             Starting telnetd.service...
             Starting tiipclad-daemon.service...
    [  OK  ] Started LSB: Dropbear Secure Shell server.
    [   11.751298] usbcore: registered new interface driver usbfs
    [   11.759573] usbcore: registered new interface driver hub
    [   11.765122] usbcore: registered new device driver usb
    [  OK  ] Started telnetd.service.
    [  OK  ] Started tiipclad-daemon.service.
             Starting ecarxinit.service...
    [   11.965551] xhci-hcd xhci-hcd.2.auto: xHCI Host Controller
    [   11.971326] xhci-hcd xhci-hcd.2.auto: new USB bus registered, assigned bus number 1
    [   11.981569] xhci-hcd xhci-hcd.2.auto: hcc params 0x0220f04c hci version 0x100 quirks 0x00210010
    [   11.990466] xhci-hcd xhci-hcd.2.auto: irq 462, io mem 0x488d0000
    [   11.998326] hub 1-0:1.0: USB hub found
    [   12.002166] hub 1-0:1.0: 1 port detected
    [   12.007078] xhci-hcd xhci-hcd.2.auto: xHCI Host Controller
    [   12.012734] xhci-hcd xhci-hcd.2.auto: new USB bus registered, assigned bus number 2
    [   12.020639] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
    [   12.030611] hub 2-0:1.0: USB hub found
    [   12.034519] hub 2-0:1.0: 1 port detected
    [   12.075815] DMA: Module install successful, device major num = 242 
    [   12.082119] DRV: Module install successful
    [   13.414100] omap_l3_noc 44000000.ocp: L3 application error: target 3 mod:2 (unclearable)
    [   13.422251] omap_l3_noc 44000000.ocp: L3 debug error: target 3 mod:2 (unclearable)
    
     _____                    _____           _         _   
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_ 
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|  
                  |___|                    |___|            
    
    Arago Project http://arago-project.org dra7xx-evm ttyS0
    
    Arago 2016.12 dra7xx-evm ttyS0
    
    dra7xx-evm login: [   22.872590] random: nonblocking pool is initialized
    
     _____                    _____           _         _   
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_ 
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|  
                  |___|                    |___|            
    
    Arago Project http://arago-project.org dra7xx-evm ttyS0
    
    Arago 2016.12 dra7xx-evm ttyS0
    
    dra7xx-evm login: 
     _____                    _____           _         _   
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_ 
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|  
                  |___|                    |___|            
    
    Arago Project http://arago-project.org dra7xx-evm ttyS0
    
    Arago 2016.12 dra7xx-evm ttyS0
    
    dra7xx-evm login: root
    root@dra7xx-evm:~# 
    root@dra7xx-evm:~# 
    root@dra7xx-evm:~# ls
    app_apa          apps.sh          decode_ipumm.sh  init-demo.sh
    root@dra7xx-evm:~# omapconf show opp
    OMAPCONF (rev v1.73-17-g578778b built Mon Sep 17 14:55:36 IST 2018)
    
    HW Platform:
      Generic DRA74X (Flattened Device Tree)
      DRA76X ES1.0 GP Device (STANDARD performance (1.8GHz))
    Error: I2C Read failed
    Error: I2C Read failed
    Error: I2C Read failed
      UNKNOWN POWER IC
    
    SW Build Details:
      Build:
        Version:  _____                    _____           _         _   
      Kernel:
        Version: 4.4.84
        Author: xiachunjiang@ecarx-5288-V3
        Toolchain: gcc version 5.3.1 20160113 (Linaro GCC 5.3-2016.02)
        Type: #1 SMP PREEMPT
        Date: Fri Jan 11 16:24:32 CST 2019
    
    omapconf: warning: no matching rate for VDD_DSPEVE OPP.Please check rates against Data Manual recommendations.
    |-----------------------------------------------------------------------------------|
    |                        | Temperature | Voltage | Frequency      | OPerating Point |
    |-----------------------------------------------------------------------------------|
    | VDD_CORE / VDD_CORE0   | 89C / 192F  | NA      |                | NOM             |
    |   L3                   |             |         |  266  MHz      |                 |
    |   DMM                  |             |         |  266  MHz      |                 |
    |   EMIF1                |             |         |  266  MHz      |                 |
    |   EMIF2                |             |         |  266  MHz      |                 |
    |     LP-DDR2            |             |         |  666  MHz      |                 |
    |   L4                   |             |         |  266  MHz      |                 |
    |   IPU1                 |             |         |  425  MHz      |                 |
    |     Cortex-M4 Cores    |             |         |  212  MHz      |                 |
    |   IPU2                 |             |         |  425  MHz      |                 |
    |     Cortex-M4 Cores    |             |         |  212  MHz      |                 |
    |   DSS                  |             |         |  192  MHz      |                 |
    |   BB2D                 |             |         | (2128 MHz) (1) |                 |
    |                        |             |         |                |                 |
    | VDD_MPU / VDD_CORE1    | 100C / 212F | NA      |                | PLUS            |
    |   MPU (CPU1 ON)        |             |         |  1800 MHz      |                 |
    |                        |             |         |                |                 |
    | VDD_GPU / VDD_CORE2    | 90C / 194F  | NA      |                | HIGH            |
    |   GPU                  |             |         |  532  MHz      |                 |
    |                        |             |         |                |                 |
    | VDD_DSPEVE / VDD_CORE3 | 88C / 190F  | NA      |                | UNKNOWN         |
    |   DSP1                 |             |         |  850  MHz      |                 |
    |   DSP2                 |             |         |  850  MHz      |                 |
    |   EVE1                 |             |         | (20   MHz) (1) |                 |
    |   EVE2                 |             |         | (20   MHz) (1) |                 |
    |                        |             |         |                |                 |
    | VDD_IVA / VDD_CORE4    | 91C / 195F  | NA      |                | HIGH            |
    |   IVA                  |             |         |  532  MHz      |                 |
    |                        |             |         |                |                 |
    |-----------------------------------------------------------------------------------|
    
    Notes:
      (1) Module is disabled, rate may not be relevant.
    
    root@dra7xx-evm:~#

  • Hi shravan,
    According to your suggestion, I had tried to disable DSP-1 and DSP-2 from dra76-evm-infoadas.dts. I tried 30 times, the problem isn't reappear. But we add nothing in DSP core, i don't know why?
    On the other hand, refer to TI wiki,"processors.wiki.ti.com/.../Processor_Training:_Multimedia. for normal boot,we know that linux kernel takes care of loading the ipu and dsp firmware,but i don't know where load the remote core, i can not find in the omap_remoteproc.c

    Thanks a lot
    Terence
  • Hi Terence,

    Thanks for trying this out.
    I have a couple of suggestions/questions:

    1. Can you apply the patches in the below link? This should help avoid some of the thermal issues being observed.
    review.omapzoom.org/
    2. Are you bootstrapping the DSP core from u-boot (early-boot + late-attach), or from kernel?

    Regards
    Shravan
  • Hi Shravan,
    1.Yes, I will try this patch now.
    2.For early boot/late attach, i know that firmware will be loaded to a DDR's address from SD card by MLO,and then uncompress to another address to run.Of course, I had debugged it in our board successfully, and I had consulted you for it before. We will apply the early boot later,but it have not now. So i don't know where to load from kernel. I traced the code in omap_remoteproc.c.

    Thanks a lot
    Terence
  • Hi Shravan,
    Pardon me, for this link "review.omapzoom.org/", there are a lot of patches. which one is for me. I am not sure.

    Thanks a lot
    Terence
  • Hi Terence,

    Please apply all patches in the above link.
    The &<core_id>_cma_pool attribute provided in the DTB indicates where the remote-proc is loaded.
    The attribute is read as part of the of_reserved_mem_device_init function as part of omap_rpoc_probe.

    Regards
    Shravan
  • Hi Shravan,
    Thanks. To disable remote core, does we just changed in .dts? We try to disable in cfg.mk, but it doesn't work. why?

    Terence
  • Hi Terence,

    You need to change ti_config_fragments/auto.cfg if you're disabling remote-proc firmware load from kernel. A DTS change indicates the driver is built, but the device is disabled, thus probe will fail.

    Regards
    Shravan
  • Hi Shravan,
    What need to change in ti_config_fragments/auto.cfg, we need to disabled dsp core. According to your suggestion, we disabled the DSP core from the kernel device-tree, the file of dra76-evm-infoadas.dts. It boots normal. I don't understand this word you said "A DTS change indicates the driver is built, but the device is disabled, thus probe will fail." It may be mean that just change cfg.mk,but the dra76-evm-infoadas.dts don't change,it will probe fail. Is it right?

    Thanks a lot
    Terence
  • Hi Terence,

    If you don't want to load any remote-core firmware (DSP/IPU), you can disable the driver (omap_remoteproc). To do this you will need to modify ti_config_fragments.
    Now if in ti_config_fragments CONFIG_OMAP_REMOTEPROC=y, then the driver is built. Setting dsp1 as disabled in the device-tree implies, the omap_remoteproc driver is probed, but since the DSP device is disabled from DTB it will fail.

    cfg.mk is in Vision-SDK and makes changes to the firmwares being built. If in cfg.mk you set include dsp1=no, then the firmware isn't built and isn't copied to /lib/firmware in the root file-system. Thus the omap_remoteproc probe again fails since the firmware isn't present.

    Regards
    Shravan
  • Hi Shravan,

    Thanks.  I know the driver (omap_remoteproc) and it based on the platform driver, but we set  CONFIG_OMAP_REMOTEPROC =y in  arch/arm/configs/ ti_sdk_dra7x_release_defconfig, not in ti_config_fragments. So, I'm afraid that if we set 'y 'in ti_config_fragments, but set 'n' in ti_sdk_dra7x_release_defconfig, Is there a conflict?

    On the other hand,  I have not find the dsp core setting in  ti_config_fragments/auto.cfg. As the attachment is  the auto.cfg. I find the  CONFIG_ARM_LPAE=y in auto.cfg, but we set 'n' in  ti_sdk_dra7x_release_defconfig.

    At last, we found  If in cfg.mk we set include dsp1=no, it will system crash,like above post,but we set 'status=disabled' in dra76-evm-infoadas.dts,  the crash is gone. That's why I confirmed these configurations with you.

    Thanks a lot for your support

    Terence

    auto.cfg

  • Hi Shravan,
    To disable DSP core, we must set dsp1=no in cfg.mk, and set 'status=disabled' in dra76-evm-infoadas.dts. If we just set only one of the two configuration, it will probe fail,also it will lead to run the app.out failure . Except this, is there another file to set?

    Thanks a lot
    Terence
  • Hi Terence,

    These are the only two files to set.

    Regards
    Shravan
  • Hi Shravan,
    Thanks a lot. To disable DSP core, it is not reappear any more,when we power up the board. But if we put the command 'reboot' in root, it will shut down.I had close this ticket,and post a new ticket. Could you help to have a look. Please move to e2e.ti.com/.../762953.

    Terence