Print corrupt stacktraces<\/h2>\n\n\n\n
In some circumstances you may get stacktraces that eventually stop because of missing symbols or corruption ( However, you can print the stack in a useful way that gives you leads about what was next in the stack:<\/p>\n\n\n\n You may want to enable asm-demangle: Example output, the 3 last lines give interesting info:<\/p>\n\n\n\n More info: 1<\/a><\/p>\n\n\n\n Sometimes the symbol names aren’t printed in the stack memdump. You can do this trick to iterate the stack and print the symbols found there (take with a grain of salt!):<\/p>\n\n\n\n Many times it’s just a matter of gdb not having loaded the unstripped version of the library. For instance, for this backtrace:<\/p>\n\n\n\n In a shell, we examine smaps and find out that the unknown piece of code comes from libgstomx:<\/p>\n\n\n\n Now we load the unstripped .so in gdb and we’re able to see the new symbol afterwards:<\/p>\n\n\n\n Useful script to prepare the add-symbol-file:<\/p>\n\n\n\n More info: 1<\/a><\/p>\n\n\n\n The “figuring out corrupt ARM stacktraces<\/a>” post has some additional info about how to use addr2line to translate memory addresses to function names on systems with a hostile debugging environment.<\/p>\n\n\n\n There are times when there’s just no way to get debug symbols working, or where we’re simply debugging on a release version of the software. In those cases, we must directly debug the assembly code. The gdb text user interface (TUI) can be used to examine the disassebled code and the CPU registers. It can be enabled with these commands:<\/p>\n\n\n\n Some useful keybindings in this mode:<\/p>\n\n\n\n This screenshot shows how we can infer that an empty RefPtr is causing a crash in some WebKit code.<\/p>\n\n\n\n Sometimes, when you continue (‘c’) execution on ARM there’s no way to stop it again unless a breakpoint is hit. But there’s a trick to retake the control: just send a harmless signal to the process.<\/p>\n\n\n\n Sometimes you need to match threads in the GStreamer logs with threads in a running gdb session. The simplest way is to ask it to GThread for each gdb thread:<\/p>\n\n\n\n This will print a list of gdb threads and If we have a pointer to the pipeline object, we can call the function that dumps the pipeline:<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" This post is a continuation of a series of blog posts about the most interesting debugging tricks I’ve found while working on GStreamer WebKit on embedded devices. These are the other posts of the series published so far: GStreamer WebKit debugging tricks using GDB (1\/2) Print corrupt stacktraces In some circumstances you may get stacktraces … Continue reading GStreamer WebKit debugging tricks using GDB (2\/2)<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[11,13,2,1,7,12],"tags":[20],"_links":{"self":[{"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/posts\/574"}],"collection":[{"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/comments?post=574"}],"version-history":[{"count":3,"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/posts\/574\/revisions"}],"predecessor-version":[{"id":595,"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/posts\/574\/revisions\/595"}],"wp:attachment":[{"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/media?parent=574"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/categories?post=574"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/eocanha.org\/blog\/wp-json\/wp\/v2\/tags?post=574"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}??<\/code> entries).<\/p>\n\n\n\n#3 0x01b8733c in ?? ()\nBacktrace stopped: previous frame identical to this frame (corrupt stack?)<\/pre>\n\n\n\n
x\/256wa $esp<\/code><\/li>x\/256ga $rsp<\/code><\/li>x\/256wa $sp<\/code><\/li><\/ul>\n\n\n\nset print asm-demangle<\/code><\/p>\n\n\n\n0x7ef85550: 0x1b87400 0x2 0x0 0x1b87400\n0x7ef85560: 0x0 0x1b87140 0x1b87140 0x759e88a4\n0x7ef85570: 0x1b87330 0x759c71a9 <gst_base_sink_change_state+956> 0x140c 0x1b87330\n0x7ef85580: 0x759e88a4 0x7ef855b4 0x0 0x7ef855b4\n...\n0x7ef85830: 0x76dbd6c4 <WebCore::AppendPipeline::resetPipeline()::__PRETTY_FUNCTION__> 0x4 0x3 0x1bfeb50\n0x7ef85840: 0x0 0x76d59268 0x75135374 0x75135374\n0x7ef85850: 0x76dbd6c4 <WebCore::AppendPipeline::resetPipeline()::__PRETTY_FUNCTION__> 0x1b7e300 0x1d651d0 0x75151b74<\/pre>\n\n\n\n
(gdb) set $i = 0\n(gdb) p\/a *((void**)($sp + 4*$i++))\n\n[Press ENTER multiple times to repeat the command]\n\n$46 = 0xb6f9fb17 <_dl_lookup_symbol_x+250>\n$58 = 0xb40a9001 <g_log_writer_standard_streams+128>\n$142 = 0xb40a877b <g_return_if_fail_warning+22>\n$154 = 0xb65a93d5 <WebCore::MediaPlayerPrivateGStreamer::changePipelineState(GstState)+180>\n$164 = 0xb65ab4e5 <WebCore::MediaPlayerPrivateGStreamer::playbackPosition() const+420>\n...<\/pre>\n\n\n\n
\/proc\/<PID>\/smaps<\/code> and info proc mappings<\/code> can help to locate the library providing the missing symbol. Then we can load it by hand.<\/p>\n\n\n\n#0 0x740ad3fc in syscall () from \/home\/enrique\/buildroot-wpe\/output\/staging\/lib\/libc.so.6 \n#1 0x74375c44 in g_cond_wait () from \/home\/enrique\/buildroot-wpe\/output\/staging\/usr\/lib\/libglib-2.0.so.0 \n#2 0x6cfd0d60 in ?? ()<\/pre>\n\n\n\n
$ cat \/proc\/715\/smaps\n...\n6cfc1000-6cff8000 r-xp 00000000 b3:02 785380 \/usr\/lib\/gstreamer-1.0\/libgstomx.so\n...<\/pre>\n\n\n\n
(gdb) add-symbol-file \/home\/enrique\/buildroot-wpe\/output\/build\/gst-omx-custom\/omx\/.libs\/libgstomx.so 0x6cfc1000\n(gdb) bt\n#0 0x740ad3fc in syscall () from \/home\/enrique\/buildroot-wpe\/output\/staging\/lib\/libc.so.6\n#1 0x74375c44 in g_cond_wait () from \/home\/enrique\/buildroot-wpe\/output\/staging\/usr\/lib\/libglib-2.0.so.0\n#2 0x6cfd0d60 in gst_omx_video_dec_loop (self=0x6e0c8130) at gstomxvideodec.c:1311\n#3 0x6e0c8130 in ?? ()<\/pre>\n\n\n\n
cat \/proc\/715\/smaps | grep '[.]so' | sed -e 's\/-[0-9a-f]*\/\/' | { while read ADDR _ _ _ _ LIB; do echo \"add-symbol-file $LIB 0x$ADDR\"; done; }<\/pre>\n\n\n\nDebugging a binary without debug symbols<\/h2>\n\n\n\n
layout asm\nlayout regs\nset print asm-demangle<\/pre>\n\n\n\n
![\"\"\/](\"https:\/\/i.imgur.com\/x1LE83l.png\")
<\/figure>\n\n\n\nWake up an unresponsive gdb on ARM<\/h2>\n\n\n\n
kill -SIGCONT 1234<\/pre>\n\n\n\n
Know which GStreamer thread id matches with each gdb thread<\/h2>\n\n\n\n
(gdb) set output-radix 16\n(gdb) thread apply all call g_thread_self()<\/pre>\n\n\n\n
GThread*<\/code>. We only need to find the one we’re looking for.<\/p>\n\n\n\nGenerate a pipeline dump from gdb<\/h2>\n\n\n\n
(gdb) call gst_debug_bin_to_dot_file_with_ts((GstBin*)0x15f0078, GST_DEBUG_GRAPH_SHOW_ALL, \"debug\")<\/pre>\n\n\n\n