This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Monta Vista Kernel Debug

Expert 1130 points

I'm trying to do a remote debug over ethernet on the kernel for monta vista 4. Has anyone been successful at this?

 

Mike

  • I've used this app note to debug the application, but this does not apply when debugging the kernel. I enabled Kernel hacking,  Compile the kernel with debug info and KGDB: kernel debugging with remote gdb, using menuconfig and build a kernel using this configuration. When I boot this kernel it loads, but then hangs. These are the instructions I was using that I got from the MontaVista Zone. I'm runing MV 4 and these instruction are for MV 5 maybe that's the problem:

     

    MontaVista Linux 5.0.x Tutorials - Using Command-line Tools with a Linux/Solaris Host: Kernel/Kernel Module Debugging

    Part I: Establish a KGDB/GDB Connection - Ethernet

    This part of the tutorial provides a step-by-step example of how to establish a KGDB/GDB connection with target over Ethernet.

    Note: The implementation of KGDB for each architecture is slightly different. This tutorial presents the basic KGDB functionality available on all architectures.

    Begin Part I - Ethernet

    For information about pre-requisites, typographical conventions and variables used in this tutorial, see Kernel/Kernel Module Debugging Tutorial.

    Complete the following steps:

    1. If you have not already done so, set up your environment as specified here. This setup includes: creating work directories, making your work directories available to the target, setting your paths, and installing optional packages. Note: No optional packages are required for this tutorial.
    2. Configure KGDB into the kernel for kernel debugging over Ethernet. To configure the kernel, complete the following steps:

      Note: For kernel configuration, there are several interfaces that can be used. This tutorial assumes you are using make menuconfig. For more information about building and configuring the kernel, see the tutorial Kernel Configuring and Building.

      1. Change to your kernel source directory. Use the command:

        cd $HOME/my_work_directory/mykernel/linux-2.6.18_<kernel_edition>/

      2. Start the kernel configuration interface. Use the command:

        make menuconfig

      3. From the Main Menu, select Kernel Hacking.
      4. Select [ ] Kernel debugging so that a "*" is in [ ].
      5. Select [*] Debug Preemptible kernel to disable it so that there is no “*” in []
      6. Select [ ] Compile the kernel with debug info so that a "*" is in [ ].
      7. Select [ ] KGDB: kernel debugging with remote gdb so that a "*" is in [ ].
      8. Select Method for KGDB communication (KGDB: On ethernet) --->.
      9. Select [ ]KGDB: On ethernet so that a "*"is in [ ].
      10. Select Exit until asked Do you Wish to save your new kernel configuration?, then select Yes.

        Note: The above steps compile KGDB in to the kernel. It is possible to compile and use KGDB as a loadable kernel module instead. For more information about using kernel modules, see the tutorial Kernel Module Compiling and Loading.

         

    3. Rebuild the kernel.

    4. You need to pass a command to the target's kernel so that it has the information it needs to make the connection. Add the following to the kernel's command-line arguments passed to the bootloader:

      kgdboe=<target_port>@<target_ip_address>/dev,<host_port>@<host_ip_address>/<host_macaddr>

      If you statically linked your module or were doing debugging on the kernel, you may want to interrupt the kernel before it executes too much of the boot sequence. You can signal the kernel to stop with kgdbwait. When the kernel checks for this, it will stop its execution and wait for GDB to connect. For this tutorial, you need to set kgdbwait.

      For example, if you are using an x86 target, planning to use Ethernet, and you want to use kgdbwait, you will need to modify the pxelinux.cfg file which will pass the kernel command-line arguments. An example file called /tftpboot/pxelinux.cfg/0A0001E looks like:

      DEFAULT vmlinuz-generic_x86-pc_target root=/dev/nfs rw ip=bootp kgdbwait kgdboe=@10.0.14.3/,@10.0.14.4/
      TIMEOUT 50

      Note: In the kgdboe= command string, only the target and host IP address need to be specified since, by default, the target will receive GDB messages on UDP port 6443 and send GDB messages on UDP port 6442. In addition, the host's MAC address does not need to be specified because it is assumed that the host’s default MAC address will be used.

    5. After the kernel has finished building, download the kernel to the target as described in the Quick Start Guide (Pro, CGE, Mobilinux) for your target.

      Note: If kgdbwait is set, when the kernel hits the check for kgdbwait in the boot sequence, it will stop booting and wait for GDB to connect.

    6. On the host, start GDB from the kernel directory. Use the command:

      <crosscompile_prefix>-gdb vmlinux

      For example:

      sutr4:$HOME/my_work_directory/mykernel/linux-2.6.18_dev # 586-gdb vmlinux
      GNU gdb 6.6.50.20061206 (MontaVista 6.6.50.20061206-11.0.0.0702556 2007-02-06)
      Copyright (C) 2006 Free Software Foundation, Inc.
      GDB is free software, covered by the GNU General Public License, and you are
      welcome to change it and/or distribute copies of it under certain conditions.
      Type "show copying" to see the conditions.
      There is absolutely no warranty for GDB. Type "show warranty" for details.
      This GDB was configured as "--host=i686-pc-linux-gnu
      --target=i586-montavista-linux"...
      (gdb)

       

    7. In the host's GDB, set up the connection to the target. Use the command:

      target remote udp:<target_ip_address>:<target_port>

      For example:

      (gdb) target remote udp:10.0.14.3:6443
      warning: The remote protocol may be unreliable over UDP.
      warning: Some events may be lost, rendering further debugging impossible.
      Remote debugging using udp:10.0.14.3:6443
      0xc0039968 in breakpoint () at kernel/kgdb.c:1619
      1619 wmb();

    You have successfully established a KGDB/GDB connection with with your target.

    You should now continue with Part II: Using KGDB/GDB for Kernel and Kernel Module Debugging of the Kernel/Kernel Module tutorial.

  • Please try the following wiki article,

    http://wiki.davincidsp.com/index.php?title=Debugging_on_DaVinci_using_kgdb

    I believe the instructions in your previous post are missing a couple of key steps.  Let us know if you have any further trouble with this.

  • Juan Gonzales said:

    Please try the following wiki article,

    http://wiki.davincidsp.com/index.php?title=Debugging_on_DaVinci_using_kgdb

    I believe the instructions in your previous post are missing a couple of key steps.  Let us know if you have any further trouble with this.

     

    Thanks for the info and rebuilt the kernel kgdb support. I'm getting the error below when I load the cmem and dsplink modules:

    #sh ./loadmodules.sh

    cmemk: disagrees about version of symbol struct_module
    insmod: error inserting 'cmemk.ko': -1 Invalid module format
    dsplinkk: no version for "struct_module" found: kernel tainted.
    dsplinkk: module license 'DSP/BIOS(TM) LINK' taints kernel.
    mknod: wrong number of arguments
    Try `mknod --help' for more information.

    This is the conents loadmodules script:

    #cat ./loadmodules.sh

    # insert cmemk, tell it to occupy physical 120MB-128MB, create
    # 20 4K buffers, 10 128K buffers  and two 1MB buffers
    insmod cmemk.ko phys_start=0x87800000 phys_end=0x88000000 pools=1x3145728,5x8294
    40,1x61440,1x10240

    # insert dsplinkk
    insmod dsplinkk.ko

    # make /dev/dsplink
    rm -f /dev/dsplink
    mknod /dev/dsplink c `awk "\\$2==\"dsplink\" {print \\$1}" /proc/devices` 0

    Here are my bootargs:

    DaVinci EVM # echo $(bootargs)
    video=davincifb:osd0=720x480x16,1350K:osd1=720x480,1350K:vid0=720x480,2025K:vid1
    =720x480,2025K davinci_enc_mngr.ch0_output=SVIDEO davinci_enc_mngr.ch0_mode=NTSC
     console=ttyS0,115200n8 noinitrd rw ip=dhcp root=/dev/nfs nfsroot=10.8.105.18:/h
    ome/user/workdir/filesys,nolock mem=120M kgdboe=@10.8.105.5/,@10.8.105.18/

    My .config settings are set for kernel hacking:

    ............

    #
    # Wan interfaces
    #
    # CONFIG_WAN is not set
    CONFIG_PPP=m
    # CONFIG_PPP_MULTILINK is not set
    # CONFIG_PPP_FILTER is not set
    CONFIG_PPP_ASYNC=m
    CONFIG_PPP_SYNC_TTY=m
    CONFIG_PPP_DEFLATE=m
    # CONFIG_PPP_BSDCOMP is not set
    # CONFIG_PPPOE is not set
    # CONFIG_SLIP is not set
    # CONFIG_SHAPER is not set
    CONFIG_NETCONSOLE=y

    ...............

    #
    # Kernel hacking
    #
    CONFIG_DEBUG_KERNEL=y
    # CONFIG_MAGIC_SYSRQ is not set
    # CONFIG_SCHEDSTATS is not set
    # CONFIG_DEBUG_SLAB is not set
    # CONFIG_DEBUG_PREEMPT is not set
    # CONFIG_WAKEUP_TIMING is not set
    # CONFIG_CRITICAL_PREEMPT_TIMING is not set
    # CONFIG_CRITICAL_IRQSOFF_TIMING is not set
    # CONFIG_DEBUG_KOBJECT is not set
    # CONFIG_DEBUG_BUGVERBOSE is not set
    # CONFIG_DEBUG_INFO is not set
    CONFIG_FRAME_POINTER=y
    CONFIG_KGDB=y
    # CONFIG_KGDB_CONSOLE is not set
    # CONFIG_KGDB_8250 is not set
    # CONFIG_KGDB_CPM_UART is not set
    # CONFIG_KGDB_SH_SCI is not set
    # CONFIG_KGDB_SIBYTE is not set
    # CONFIG_KGDB_ARM_VERSATILE is not set
    # CONFIG_KGDB_PXA_SERIAL is not set
    # CONFIG_KGDB_TXX9 is not set
    # CONFIG_KGDB_AU1X00 is not set
    # CONFIG_KGDB_IP3106 is not set
    CONFIG_KGDB_ETH=y
    # CONFIG_KGDB_MPSC is not set
    # CONFIG_KGDB_9600BAUD is not set
    # CONFIG_KGDB_19200BAUD is not set
    # CONFIG_KGDB_38400BAUD is not set
    # CONFIG_KGDB_57600BAUD is not set
    # CONFIG_KGDB_115200BAUD is not set
    # CONFIG_DEBUG_STACKOVERFLOW is not set
    # CONFIG_DEBUG_USER is not set
    # CONFIG_DEBUG_WAITQ is not set
    # CONFIG_DEBUG_ERRORS is not set
    # CONFIG_DEBUG_LL is not set

    ..................

    I think I set it up right, but I can't load the modules.

     

    Mike

  • Whenever the kernel is reconfigured (including adding kgdb support), all modules in the system (including CMEM and Link) should be rebuilt.  I added a little commentary to the wiki article to say as much.

    Chris

  • ChrisRing said:

    Whenever the kernel is reconfigured (including adding kgdb support), all modules in the system (including CMEM and Link) should be rebuilt.  I added a little commentary to the wiki article to say as much.

    Chris

    When I run make I get an error. All env variables in Rules.make are overwritten by main Rules.make except for UCTOOL_PREFIX which I don't have:

    # Equivalent path for uClibc compiler tools
    UCTOOL_PREFIX=/db/toolsrc/library/vendors2005/opensource/buildroot/16012006/staging_dir/arm-linux-uclibc/bin

    :

    $ cd ~/dvsdk_1_30_01_41/cmem_2_00_01/packages/ti/sdo/linuxutils/cmem

    $ make

    Making all in subdirectory src...
    make[1]: Entering directory `/home/user/dvsdk_1_30_01_41/cmem_2_00_01/packages/ti/sdo/linuxutils/cmem/src'

    Making all in subdirectory interface...
    make[2]: Entering directory `/home/user/dvsdk_1_30_01_41/cmem_2_00_01/packages/ti/sdo/linuxutils/cmem/src/interface'
    /opt/montavista/pro/devkit/arm/v5t_le/bin/arm_v5t_le-gcc -I. -Wall  -c -g -D__DEBUG -o debug/cmem.o cmem.c
    /opt/montavista/pro/devkit/arm/v5t_le/bin/arm_v5t_le-ar rc ../../lib/cmemd.a debug/cmem.o
    Installing headers...
    /opt/montavista/pro/devkit/arm/v5t_le/bin/arm_v5t_le-gcc -I. -Wall  -c -O2 -o release/cmem.o cmem.c
    /opt/montavista/pro/devkit/arm/v5t_le/bin/arm_v5t_le-ar rc ../../lib/cmem.a release/cmem.o
    /db/toolsrc/library/vendors2005/opensource/buildroot/16012006/staging_dir/arm-linux-uclibc/bin/gcc -c -g -I. -Wall -o debug/cmem.o470uC cmem.c
    make[2]: /db/toolsrc/library/vendors2005/opensource/buildroot/16012006/staging_dir/arm-linux-uclibc/bin/gcc: Command not found
    make[2]: *** [debuguc] Error 127
    make[2]: Leaving directory `/home/user/dvsdk_1_30_01_41/cmem_2_00_01/packages/ti/sdo/linuxutils/cmem/src/interface'
    make[1]: *** [interface] Error 2
    make[1]: Leaving directory `/home/user/dvsdk_1_30_01_41/cmem_2_00_01/packages/ti/sdo/linuxutils/cmem/src'
    make: *** [src] Error 2

  • When I execute loadmodules.sh I get an error in the last line:

    >mknod /dev/dsplink c `awk "\\$2==\"dsplink\" {print \\$1}" /proc/devices` 0

    I see in other posts that this commend should end up being

    > mknod /dev/dsplink c 230 0

    But I'm getting,

    > mknod /dev/dsplink c 0

    The Major number didn't parse out. I've turned on kernel hacking stuff in menuconfig. I can step through the kernel (I added a kgdbwait to the bootargs. You might want to update wiki) when I type it,

    > mknod /dev/dsplink c 230 0

    The encode demo program fails to connect to the dsp. I don't understand why the loadmodules.sh script is having a problem with this last line with kernel debugging turned on and I don't know why using 230 as the Major number didn't work either.

    Mike

  • Chris,

    For some reason when I enable kernel debugging the loadmodules.sh scirpt fails. the problem is with the /proc/devices file. The line 230 dsplink is not listed so the awk script can't find a Major number which should be 230. Why would dsplink not load?

    Mike

  • Chris,

     

    When I diff the /proc/devices files after boot and after loadmodules.sh only 252 cmem is added. /proc/devices is not showing dsplink so somehow it failed to load. Below is the message I get from the insmod dsplink.ko command.

    dsplinkk: module license 'DSP/BIOS(TM) LINK' taints kernel.

     

    Mike

  • MRI said:
    dsplinkk: module license 'DSP/BIOS(TM) LINK' taints kernel.

    Though this message may sound kind of like an error or something bad it is actually a good sign, the kernel prints out these 'tainted kernel' messages when you insert a kernel module, the idea being that the kernel is no longer pure open source, it is tainted with a potentially closed source module. Unfortunately I could not say why link would not load properly when using kgdb, perhaps there are other error/warning messages?

  • MRI said:

    dsplinkk: module license 'DSP/BIOS(TM) LINK' taints kernel.

    Please disregard the error message above; Linus decided that he wants all kernel modules that do not comply with open source GPL license to print out a "taints kernel" message (just a warning as opposed to a serious error).  Since DSPLINK and CMEM have some DSP side dependancies, there were a few barriers that prevented us from making these modules open source; we are working toward this end.

    Do you see /dev/dsplink device?  If so, it means the DSPLINK loaded as I would expect if no other errors were displayed.  This also probrably means that the failure is related to mknod.

  • Juan Gonzales said:

    dsplinkk: module license 'DSP/BIOS(TM) LINK' taints kernel.

    Please disregard the error message above; Linus decided that he wants all kernel modules that do not comply with open source GPL license to print out a "taints kernel" message (just a warning as opposed to a serious error).  Since DSPLINK and CMEM have some DSP side dependancies, there were a few barriers that prevented us from making these modules open source; we are working toward this end.

    Do you see /dev/dsplink device?  If so, it means the DSPLINK loaded as I would expect if no other errors were displayed.  This also probrably means that the failure is related to mknod.

    [/quote]

     

    No, i don't see the dsplink device when in /proc/devices

     

    Mike

  • Do you see dsplink device when you do

    % ls /dev/dsplink

    ?

  • Juan Gonzales said:

    Do you see dsplink device when you do

    % ls /dev/dsplink

    ?

     

    No, I dont'.

    What's interesting is that when I type the command:

    >insmod dsplinkk.ko

    I get an error message:

    insmod: error inserting 'dsplinkk.ko': -1 File exists

    I can remove the module and reinsert it, but I never see it in the /proc/devices file.

     

    Mike

  • This sounds more serious,

    can you try loading dsplink manually

    % rmmod dsplinkk.ko  (optional but should not hurt)

    % insmod dsplink.ko

    Ideally, we want to use the same commands found in the loadmodules file (I believe they are as above, but have not looked at this in a while). 

     

  • Juan Gonzales said:

    This sounds more serious,

    can you try loading dsplink manually

    % rmmod dsplinkk.ko  (optional but should not hurt)

    % insmod dsplink.ko

    Ideally, we want to use the same commands found in the loadmodules file (I believe they are as above, but have not looked at this in a while). 

     

    I can't remove. It never loaded. See below. This does work when kernel debug is off.

    root@10.8.105.5:/opt/dvsdk/dm6446# rmmod dsplink.ko
    ERROR: Module dsplink does not exist in /proc/modules