Remote debugging issue with multithread app using CCSv5.1 and gdbserver

Pierre-Olivier Belzile

Hi,

I have a situation where CCS fails after hitting a breakpoint when remotely debugging a threaded application using gdbserver over tcp. When it happens, it fails to display the current thread function and current line in code. GDB output shows error ("Thread ID 2 has terminated.", see attached gdb trace log).

When paused, CCS has that weird behavior : It scans threads ID starting from 1 to infinite! The error seems to be related to this behavior... Is it normal ?

See the trace log:

Fullscreen 7610.gdbtracelog.txt Download

789,332 1-environment-cd /home/user/VBoxShared/ThreadTest
789,335 1^done
789,340 (gdb) 
789,345 2-gdb-set breakpoint pending on
789,346 2^done
789,346 (gdb) 
789,352 3source /home/user/workspace_v5_1/.gdbinit
789,354 &"source /home/user/workspace_v5_1/.gdbinit\n"
789,356 3^done
789,356 (gdb) 
789,361 4-gdb-set auto-solib-add on
789,362 4^done
789,362 (gdb) 
789,387 5-file-exec-and-symbols /home/user/targetfs/home/root/main
789,399 5^done
789,399 (gdb) 
789,408 6-target-select remote 192.168.0.102:4444
789,454 =thread-created,id="1"
789,514 =library-loaded,id="/lib/ld-linux.so.3",target-name="/lib/ld-linux.so.3",host-name="/home/us\
er/targetfs/lib/ld-linux.so.3",low-address="0x4003c7b0",high-address="0x40054460",symbols-loaded="0"\

789,555 *stopped,frame={addr="0x4003c7b0",func="??",args=[],from="/home/user/targetfs/lib/ld-linux.s\
o.3"},thread-id="1",stopped-threads="all"
789,564 6^connected
789,564 (gdb) 
789,593 7-break-insert -f /home/user/VBoxShared/ThreadTest/main.c:7
789,622 7^done,bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x00008488",func="ma\
in",file="main.c",fullname="/home/user/VBoxShared/ThreadTest/main.c",line="7",times="0",original-loc\
ation="/home/user/VBoxShared/ThreadTest/main.c:7"}
789,622 (gdb) 
789,811 8-break-insert -t -f main
789,818 8^done,bkpt={number="2",type="breakpoint",disp="del",enabled="y",addr="0x0000847c",func="mai\
n",file="main.c",fullname="/home/user/VBoxShared/ThreadTest/main.c",line="5",times="0",original-loca\
tion="main"}
789,818 (gdb) 
789,848 9-exec-continue
789,864 9^running
789,864 *running,thread-id="all"
789,864 (gdb) 
790,013 =library-loaded,id="/lib/libgcc_s.so.1",target-name="/lib/libgcc_s.so.1",host-name="/home/us\
er/targetfs/lib/libgcc_s.so.1",low-address="0x400b0778",high-address="0x400b9190",symbols-loaded="0"\

790,016 =library-loaded,id="/lib/libpthread.so.0",target-name="/lib/libpthread.so.0",host-name="/hom\
e/user/targetfs/lib/libpthread.so.0",low-address="0x400c5ca0",high-address="0x400d4438",symbols-load\
ed="0"
790,022 =library-loaded,id="/lib/libc.so.6",target-name="/lib/libc.so.6",host-name="/home/user/targe\
tfs/lib/libc.so.6",low-address="0x400f6d60",high-address="0x401de100",symbols-loaded="0"
790,254 ~"[Switching to Thread 979]\n"
790,275 *stopped,reason="breakpoint-hit",disp="del",bkptno="2",frame={addr="0x0000847c",func="main",\
args=[{name="argc",value="1"},{name="argv",value="0xbeab7e24"}],file="main.c",fullname="/home/user/V\
BoxShared/ThreadTest/main.c",line="5"},thread-id="1",stopped-threads="all"
790,275 (gdb) 
791,693 10-thread-list-ids
791,696 =thread-exited,id="1"
791,698 =thread-created,id="2"
791,698 ~"[New Thread 979]\n"
791,701 10^done,thread-ids={thread-id="2"},current-thread-id="2",number-of-threads="1"
791,701 (gdb) 
791,711 11-thread-select 2
791,711 12-stack-info-depth 11
791,713 &"Thread ID 2 has terminated.\n"
791,714 11^error,msg="Thread ID 2 has terminated."
791,715 (gdb) 
791,724 12^done,depth="1"
791,724 (gdb) 
791,726 13-stack-list-frames
791,733 13^done,stack=[frame={level="0",addr="0x0000847c",func="main",file="main.c",fullname="/home/\
user/VBoxShared/ThreadTest/main.c",line="5"}]
791,733 (gdb) 
792,178 14info threads
792,180 &"info threads\n"
792,182 =thread-exited,id="2"
792,186 =thread-created,id="3"
792,186 ~"[New Thread 979]\n"
792,201 ~"* 3 Thread 979  main (argc=1, argv=0xbeab7e24) at main.c:5\n"
792,202 14^done
792,202 (gdb) 
792,657 15-thread-list-ids
792,660 =thread-exited,id="3"
792,662 =thread-created,id="4"
792,662 ~"[New Thread 979]\n"
792,665 15^done,thread-ids={thread-id="4"},current-thread-id="4",number-of-threads="1"
792,665 (gdb) 
792,784 16info threads
792,786 &"info threads\n"
792,788 =thread-exited,id="4"
792,790 =thread-created,id="5"
792,792 ~"[New Thread 979]\n"
792,809 ~"* 5 Thread 979  main (argc=1, argv=0xbeab7e24) at main.c:5\n"
792,810 16^done
792,810 (gdb) 
792,915 17-thread-select 4
792,916 18-stack-list-frames 0 0
792,917 &"Thread ID 4 not known.\n"
792,918 17^error,msg="Thread ID 4 not known."
792,918 (gdb) 
792,921 18^done,stack=[frame={level="0",addr="0x0000847c",func="main",file="main.c",fullname="/home/\
user/VBoxShared/ThreadTest/main.c",line="5"}]
792,921 (gdb) 
792,929 19-stack-list-frames
792,934 19^done,stack=[frame={level="0",addr="0x0000847c",func="main",file="main.c",fullname="/home/\
user/VBoxShared/ThreadTest/main.c",line="5"}]
792,934 (gdb) 
792,944 20-stack-info-depth 11
792,945 20^done,depth="1"
792,945 (gdb) 
793,446 21-thread-list-ids
793,451 =thread-exited,id="5"
793,455 =thread-created,id="6"
793,456 ~"[New Thread 979]\n"
793,459 21^done,thread-ids={thread-id="6"},current-thread-id="6",number-of-threads="1"
793,459 (gdb) 
793,463 22-thread-list-ids
793,466 =thread-exited,id="6"
793,469 =thread-created,id="7"
793,470 ~"[New Thread 979]\n"
793,471 22^done,thread-ids={thread-id="7"},current-thread-id="7",number-of-threads="1"
793,471 (gdb) 
793,504 23-thread-select 7
793,504 24-stack-info-depth 11
793,508 &"Thread ID 7 has terminated.\n"
793,508 23^error,msg="Thread ID 7 has terminated."
793,508 (gdb) 
793,521 24^done,depth="1"
793,521 (gdb) 
793,542 25-stack-list-frames
793,546 25^done,stack=[frame={level="0",addr="0x0000847c",func="main",file="main.c",fullname="/home/\
user/VBoxShared/ThreadTest/main.c",line="5"}]
793,547 (gdb) 
793,570 26info threads
793,573 &"info threads\n"
793,574 =thread-exited,id="7"
793,577 =thread-created,id="8"
793,584 ~"[New Thread 979]\n"
793,593 ~"* 8 Thread 979  main (argc=1, argv=0xbeab7e24) at main.c:5\n"
793,594 26^done
793,594 (gdb) 
794,030 27-thread-list-ids
794,033 =thread-exited,id="8"
794,036 =thread-created,id="9"
794,037 ~"[New Thread 979]\n"
794,040 27^done,thread-ids={thread-id="9"},current-thread-id="9",number-of-threads="1"
794,040 (gdb) 
794,075 28-thread-list-ids
794,078 =thread-exited,id="9"
794,084 =thread-created,id="10"
794,084 ~"[New Thread 979]\n"
794,087 28^done,thread-ids={thread-id="10"},current-thread-id="10",number-of-threads="1"
794,087 (gdb) 
794,326 29-thread-list-ids
794,329 =thread-exited,id="10"
794,332 =thread-created,id="11"
794,332 ~"[New Thread 979]\n"
794,334 29^done,thread-ids={thread-id="11"},current-thread-id="11",number-of-threads="1"
794,334 (gdb) 
794,743 30-thread-list-ids
794,746 =thread-exited,id="11"
794,749 =thread-created,id="12"
794,749 ~"[New Thread 979]\n"
794,751 30^done,thread-ids={thread-id="12"},current-thread-id="12",number-of-threads="1"
794,758 (gdb) 
794,863 31-thread-list-ids
794,865 =thread-exited,id="12"
794,868 =thread-created,id="13"
794,869 ~"[New Thread 979]\n"
794,872 31^done,thread-ids={thread-id="13"},current-thread-id="13",number-of-threads="1"
794,872 (gdb) 
795,036 32-thread-list-ids
795,038 =thread-exited,id="13"
795,042 =thread-created,id="14"
795,042 ~"[New Thread 979]\n"
795,044 32^done,thread-ids={thread-id="14"},current-thread-id="14",number-of-threads="1"
795,046 (gdb) 
795,164 33-thread-list-ids
795,166 =thread-exited,id="14"
795,168 =thread-created,id="15"
795,171 ~"[New Thread 979]\n"
795,176 33^done,thread-ids={thread-id="15"},current-thread-id="15",number-of-threads="1"
795,176 (gdb) 
795,297 34-thread-list-ids
795,299 =thread-exited,id="15"
795,303 =thread-created,id="16"
795,304 ~"[New Thread 979]\n"
795,309 34^done,thread-ids={thread-id="16"},current-thread-id="16",number-of-threads="1"
795,309 (gdb) 
795,432 35-thread-list-ids
795,435 =thread-exited,id="16"
795,437 =thread-created,id="17"
795,438 ~"[New Thread 979]\n"
795,442 35^done,thread-ids={thread-id="17"},current-thread-id="17",number-of-threads="1"
795,442 (gdb) 
795,567 36-thread-list-ids
795,569 =thread-exited,id="17"
795,572 =thread-created,id="18"
795,572 ~"[New Thread 979]\n"
795,574 36^done,thread-ids={thread-id="18"},current-thread-id="18",number-of-threads="1"
795,575 (gdb) 
795,828 37-thread-list-ids
795,831 =thread-exited,id="18"
795,833 =thread-created,id="19"
795,836 ~"[New Thread 979]\n"
795,838 37^done,thread-ids={thread-id="19"},current-thread-id="19",number-of-threads="1"
795,838 (gdb) 
795,984 38-exec-continue
795,991 38^running
795,991 *running,thread-id="all"
795,991 (gdb) 
800,764 *stopped,reason="breakpoint-hit",disp="keep",bkptno="1",frame={addr="0x00008488",func="main"\
,args=[{name="argc",value="1"},{name="argv",value="0xbeab7e24"}],file="main.c",fullname="/home/user/\
VBoxShared/ThreadTest/main.c",line="7"},thread-id="19",stopped-threads="all"
800,771 (gdb) 
800,794 39-thread-list-ids
800,796 =thread-exited,id="19"
800,800 =thread-created,id="20"
800,800 ~"[New Thread 979]\n"
800,806 39^done,thread-ids={thread-id="20"},current-thread-id="20",number-of-threads="1"
800,806 (gdb) 
800,976 40-thread-list-ids
800,979 =thread-exited,id="20"
800,986 =thread-created,id="21"
800,986 ~"[New Thread 979]\n"
800,986 40^done,thread-ids={thread-id="21"},current-thread-id="21",number-of-threads="1"
800,986 (gdb) 
801,280 41-thread-list-ids
801,283 =thread-exited,id="21"
801,286 =thread-created,id="22"
801,287 ~"[New Thread 979]\n"
801,290 41^done,thread-ids={thread-id="22"},current-thread-id="22",number-of-threads="1"
801,290 (gdb) 
801,446 42-thread-list-ids
801,449 =thread-exited,id="22"
801,453 =thread-created,id="23"
801,453 ~"[New Thread 979]\n"
801,454 42^done,thread-ids={thread-id="23"},current-thread-id="23",number-of-threads="1"
801,455 (gdb) 
801,557 43-thread-list-ids
801,560 =thread-exited,id="23"
801,562 =thread-created,id="24"
801,562 ~"[New Thread 979]\n"
801,564 43^done,thread-ids={thread-id="24"},current-thread-id="24",number-of-threads="1"
801,564 (gdb) 
801,690 44-thread-list-ids
801,693 =thread-exited,id="24"
801,697 =thread-created,id="25"
801,701 ~"[New Thread 979]\n"
801,703 44^done,thread-ids={thread-id="25"},current-thread-id="25",number-of-threads="1"
801,707 (gdb) 
801,895 45-thread-list-ids
801,899 =thread-exited,id="25"
801,904 =thread-created,id="26"
801,907 ~"[New Thread 979]\n"
801,911 45^done,thread-ids={thread-id="26"},current-thread-id="26",number-of-threads="1"
801,911 (gdb) 
802,171 46-thread-list-ids
802,174 =thread-exited,id="26"
802,177 =thread-created,id="27"
802,178 ~"[New Thread 979]\n"
802,181 46^done,thread-ids={thread-id="27"},current-thread-id="27",number-of-threads="1"
802,181 (gdb) 
802,358 47-thread-list-ids
802,361 =thread-exited,id="27"
802,363 =thread-created,id="28"
802,364 ~"[New Thread 979]\n"
802,366 47^done,thread-ids={thread-id="28"},current-thread-id="28",number-of-threads="1"
802,366 (gdb) 
803,116 48-thread-list-ids
803,119 =thread-exited,id="28"
803,121 =thread-created,id="29"
803,121 ~"[New Thread 979]\n"
803,123 48^done,thread-ids={thread-id="29"},current-thread-id="29",number-of-threads="1"
803,123 (gdb) 
803,490 49-thread-list-ids
803,492 =thread-exited,id="29"
803,496 =thread-created,id="30"
803,497 ~"[New Thread 979]\n"
803,500 49^done,thread-ids={thread-id="30"},current-thread-id="30",number-of-threads="1"
803,500 (gdb) 
803,622 50-thread-list-ids
803,624 =thread-exited,id="30"
803,628 =thread-created,id="31"
803,628 ~"[New Thread 979]\n"
803,630 50^done,thread-ids={thread-id="31"},current-thread-id="31",number-of-threads="1"
803,630 (gdb) 
803,777 51-thread-list-ids
803,780 =thread-exited,id="31"
803,782 =thread-created,id="32"
803,783 ~"[New Thread 979]\n"
803,786 51^done,thread-ids={thread-id="32"},current-thread-id="32",number-of-threads="1"
803,787 (gdb) 
804,023 52-thread-list-ids
804,025 =thread-exited,id="32"
804,029 =thread-created,id="33"
804,030 ~"[New Thread 979]\n"
804,032 52^done,thread-ids={thread-id="33"},current-thread-id="33",number-of-threads="1"
804,032 (gdb) 
804,233 53-thread-list-ids
804,236 =thread-exited,id="33"
804,239 =thread-created,id="34"
804,239 ~"[New Thread 979]\n"
804,241 53^done,thread-ids={thread-id="34"},current-thread-id="34",number-of-threads="1"
804,241 (gdb) 
804,392 54-thread-list-ids
804,394 =thread-exited,id="34"
804,397 =thread-created,id="35"
804,398 ~"[New Thread 979]\n"
804,403 54^done,thread-ids={thread-id="35"},current-thread-id="35",number-of-threads="1"
804,405 (gdb) 
804,543 55-thread-list-ids
804,546 =thread-exited,id="35"
804,552 =thread-created,id="36"
804,553 ~"[New Thread 979]\n"
804,555 55^done,thread-ids={thread-id="36"},current-thread-id="36",number-of-threads="1"
804,556 (gdb) 
804,698 =thread-exited,id="36"
804,698 =thread-created,id="37"
804,698 ~"[New Thread 979]\n"
804,698 56^done,thread-ids={thread-id="37"},current-thread-id="37",number-of-threads="1"
804,698 (gdb) 
804,672 56-thread-list-ids
804,920 57-thread-list-ids
804,923 =thread-exited,id="37"
804,927 =thread-created,id="38"
804,928 ~"[New Thread 979]\n"
804,931 57^done,thread-ids={thread-id="38"},current-thread-id="38",number-of-threads="1"
804,932 (gdb) 
805,136 58-thread-list-ids
805,140 =thread-exited,id="38"
805,144 =thread-created,id="39"
805,146 ~"[New Thread 979]\n"
805,150 58^done,thread-ids={thread-id="39"},current-thread-id="39",number-of-threads="1"
805,150 (gdb) 
805,649 59-thread-list-ids
805,652 =thread-exited,id="39"
805,657 =thread-created,id="40"
805,658 ~"[New Thread 979]\n"
805,662 59^done,thread-ids={thread-id="40"},current-thread-id="40",number-of-threads="1"
805,662 (gdb) 
805,775 60-thread-list-ids
805,777 =thread-exited,id="40"
805,782 =thread-created,id="41"
805,783 ~"[New Thread 979]\n"
805,785 60^done,thread-ids={thread-id="41"},current-thread-id="41",number-of-threads="1"
805,787 (gdb) 
805,906 61-thread-list-ids
805,908 =thread-exited,id="41"
805,911 =thread-created,id="42"
805,911 ~"[New Thread 979]\n"
805,913 61^done,thread-ids={thread-id="42"},current-thread-id="42",number-of-threads="1"
805,913 (gdb) 
806,044 62-thread-list-ids
806,046 =thread-exited,id="42"
806,048 =thread-created,id="43"
806,048 ~"[New Thread 979]\n"
806,050 62^done,thread-ids={thread-id="43"},current-thread-id="43",number-of-threads="1"
806,050 (gdb) 
806,184 63-thread-list-ids
806,186 =thread-exited,id="43"
806,188 =thread-created,id="44"
806,190 ~"[New Thread 979]\n"
806,193 63^done,thread-ids={thread-id="44"},current-thread-id="44",number-of-threads="1"
806,198 (gdb) 
806,324 64-thread-list-ids
806,326 =thread-exited,id="44"
806,330 =thread-created,id="45"
806,331 ~"[New Thread 979]\n"
806,334 64^done,thread-ids={thread-id="45"},current-thread-id="45",number-of-threads="1"
806,335 (gdb) 
806,461 65-thread-list-ids
806,464 =thread-exited,id="45"
806,466 =thread-created,id="46"
806,466 ~"[New Thread 979]\n"
806,468 65^done,thread-ids={thread-id="46"},current-thread-id="46",number-of-threads="1"
806,468 (gdb) 
806,546 66-exec-continue
806,551 66^running
806,551 *running,thread-id="46"
806,551 (gdb) 
806,551 &"warning: Invalid remote reply: \n"

[...]

790,275 *stopped,reason="breakpoint-hit",disp="del",bkptno="2",frame={addr="0x0000847c",func="main",\
args=[{name="argc",value="1"},{name="argv",value="0xbeab7e24"}],file="main.c",fullname="/home/user/V\
BoxShared/ThreadTest/main.c",line="5"},thread-id="1",stopped-threads="all"
790,275 (gdb)
791,693 10-thread-list-ids
791,696 =thread-exited,id="1"

791,698 =thread-created,id="2"
791,698 ~"[New Thread 979]\n"
791,701 10^done,thread-ids={thread-id="2"},current-thread-id="2",number-of-threads="1"
791,701 (gdb)
791,711 11-thread-select 2
791,711 12-stack-info-depth 11
791,713 &"Thread ID 2 has terminated.\n"
791,714 11^error,msg="Thread ID 2 has terminated."
791,715 (gdb)
791,724 12^done,depth="1"
791,724 (gdb)
791,726 13-stack-list-frames
791,733 13^done,stack=[frame={level="0",addr="0x0000847c",func="main",file="main.c",fullname="/home/\
user/VBoxShared/ThreadTest/main.c",line="5"}]
791,733 (gdb)
792,178 14info threads
792,180 &"info threads\n"
792,182 =thread-exited,id="2"
792,186 =thread-created,id="3"
792,186 ~"[New Thread 979]\n"
792,201 ~"* 3 Thread 979 main (argc=1, argv=0xbeab7e24) at main.c:5\n"
792,202 14^done
792,202 (gdb)
792,657 15-thread-list-ids
792,660 =thread-exited,id="3"
792,662 =thread-created,id="4"
792,662 ~"[New Thread 979]\n"
792,665 15^done,thread-ids={thread-id="4"},current-thread-id="4",number-of-threads="1"
792,665 (gdb)
792,784 16info threads
792,786 &"info threads\n"
792,788 =thread-exited,id="4"
792,790 =thread-created,id="5"
792,792 ~"[New Thread 979]\n"
792,809 ~"* 5 Thread 979 main (argc=1, argv=0xbeab7e24) at main.c:5\n"
792,810 16^done
792,810 (gdb)
792,915 17-thread-select 4
792,916 18-stack-list-frames 0 0
792,917 &"Thread ID 4 not known.\n"
792,918 17^error,msg="Thread ID 4 not known."
792,918 (gdb)
792,921 18^done,stack=[frame={level="0",addr="0x0000847c",func="main",file="main.c",fullname="/home/\
user/VBoxShared/ThreadTest/main.c",line="5"}]
792,921 (gdb)
792,929 19-stack-list-frames
792,934 19^done,stack=[frame={level="0",addr="0x0000847c",func="main",file="main.c",fullname="/home/\
user/VBoxShared/ThreadTest/main.c",line="5"}]

[...]

Here is my main.c:

#include <stdio.h>

int main(int argc, char* argv[])
{
   printf("App Start\n");
   getchar();
   printf("App End\n");

   return 0 ;
}

and my makefile:

export EZSDK=/home/user/ti-ezsdk_dm816x-evm_5_02_02_60/
include ${EZSDK}/Rules.make

all:
$(CSTOOL_PREFIX)gcc -pthread -g -o main main.c
cp main /home/user/targetfs/home/root/

over 14 years ago

desouza over 14 years ago

TI__Guru**** 171309 points

Hi,

Is that all in your application? If so, you are not even using multithread, but only the -pthread option... I wonder if GDB is getting confused by that.

My next step would be to test with a multithreaded application before jumping to conclusions... What happens if you run the Pthread creation and termination example shown at the reference below?

https://computing.llnl.gov/tutorials/pthreads/

The component used to debug GDB applications is completely open source and therefore its capabilities and features are not controlled by us. However it is still an interesting exercise.

Regards,

Rafael

Pierre-Olivier Belzile over 14 years ago in reply to desouza

Intellectual 375 points

Hi Rafael,

In the beginning, the application was multithreaded and had some synchronization objects. To get the problem easily reproducible, I started simplifying it to finally figure out that even this simple code is problematic.

I understand that it is opensource, but i really don't know where to start... I have no idea if it is Code Sourcery gdb/gdbserver. All I know, is that when I use CCSv5 with CodeSourcery tool 2009q1-203 and TI EZSDK 5.02, the problem shows.

What do you suggest ? Is there anyone else having a similar problem ?

desouza over 14 years ago in reply to Pierre-Olivier Belzile

TI__Guru**** 171309 points

Pierre,

Oh, I see. Thanks for the explanation. I will try to reproduce this issue here but I will need some time to put together the debug environment. In the meantime, what is exactly the device and the CCS version you are using? Are you running it from Linux or Windows? Are you using any JTAG simultaneously or just the GDB connection?

There are a few developers using multithreaded apps, but I couldn't find specific threads that discuss its use with CCS/GDB.

Regards,

Rafael

desouza over 14 years ago in reply to desouza

TI__Guru**** 171309 points

Pierre,

Unfortunately I could not reproduce the issue you are seeing with orphaned threads in your target. I used a modified version of the code shown in the page I sent before, CodeSourcery 2009q1-203, the EZSDK 5.03 for DM8148 and CCSv5.1.0.09000.

I send attached a tarred file with the project and the GDB logs from within CCS so you could have a baseline to compare and help figure out what may be happening in your system.

Hope this helps,

Rafael

GCC_pthreads_A8.tar.gz

Code Composer Studio™︎

Code Composer Studio forum

Remote debugging issue with multithread app using CCSv5.1 and gdbserver