Unusable debug info is uploaded

[bobrik]

I have both parca and parca-agent running on the same machine.

I also have dockerd running on the same machine and I get this flamegraph:

I have debug info available for dockerd:

ivan@cube:~$ file /usr/bin/dockerd
/usr/bin/dockerd: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=a52a3c0947657f986c65095f7bbb4e29ab53343f, for GNU/Linux 3.2.0, with debug_info, not stripped

I can also see debug info uploaded:

ivan@cube:~$ cat ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/metadata
{
  "buildId":  "a52a3c0947657f986c65095f7bbb4e29ab53343f",
  "source":  "SOURCE_UPLOAD",
  "upload":  {
    "id":  "a4518131-1c07-4e53-a188-092bf99c80fc",
    "hash":  "41a765552e25adb26da340ed5e39d3e7",
    "state":  "STATE_UPLOADED",
    "startedAt":  "2025-01-27T06:01:30.695240535Z",
    "finishedAt":  "2025-01-27T06:01:30.840336583Z"
  },
  "quality":  {
    "hasDwarf":  true,
    "hasSymtab":  true,
    "hasDynsym":  true
  }

ivan@cube:~$ file~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo
/home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=a52a3c0947657f986c65095f7bbb4e29ab53343f, for GNU/Linux 3.2.0, with debug_info, not stripped

Let's pick one addr (0x22cda8d) and consult debug info with addr2line just to be sure:

ivan@cube:~$ eu-addr2line -f -e ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo 0x22cda8d
??
??:0

No luck! Not even with go tool addr2line:

ivan@cube:~$ eu-addr2line -f -e ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo 0x22cda8d
??
??:0

Now here's the fun thing, running the same thing on the original binary works:

ivan@cube:~$ eu-addr2line -f -e /usr/bin/dockerd 0x22cda8d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1
/_/GOROOT/src/runtime/cgo/gcc_mmap.c:37:3

ivan@cube:~$ echo 0x22cda8d | go tool addr2line /usr/bin/dockerd
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1
/root/build-deb/engine/api/server/server.go:56

It turns out that the uploaded file is quite a bit smaller than the original:

ivan@cube:~$ ls -l ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo /usr/bin/dockerd
-rw-rw-r-- 1 ivan ivan  38452051 Jan 27 06:01 /home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo
-rwxr-xr-x 2 root root 107762880 Sep 20 11:41 /usr/bin/dockerd

Here's the diff in section sizes:

ivan@cube:~$ diff -rup <(size -A /usr/bin/dockerd) <(size -A /home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo)
--- /dev/fd/63	2025-01-27 06:09:05.454218257 +0000
+++ /dev/fd/62	2025-01-27 06:09:05.454218257 +0000
@@ -1,36 +1,36 @@
-/usr/bin/dockerd  :
+/home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo  :
 section                  size       addr
-.interp                    28        848
+.interp                     0        848
 .note.gnu.property         32        880
 .note.gnu.build-id         36        912
 .note.ABI-tag              32        948
 .note.go.buildid          100        980
-.gnu.hash             1254760       1080
+.gnu.hash                   0       1080
 .dynsym               5171328    1255840
 .dynstr               8442653    6427168
-.gnu.version           430944   14869822
-.gnu.version_r            144   15300768
+.gnu.version                0   14869822
+.gnu.version_r              0   15300768
 .rela.dyn             7906632   15300912
 .rela.plt                2088   23207544
-.init                      23   23212032
+.init                       0   23212032
 .plt                     1408   23212064
 .plt.got                    8   23213472
-.text                31684652   23213504
-.fini                       9   54898156
-.rodata               4712716   54898688
-.eh_frame_hdr             724   59611404
-.eh_frame                3636   59612128
-.tbss                       8   59619248
-.init_array                 8   59619248
-.fini_array                 8   59619256
-.data.rel.ro         29827888   59619264
-.dynamic                  512   89447152
-.got                      776   89447664
-.data                  800904   89448448
+.text                       0   23213504
+.fini                       0   54898156
+.rodata                     0   54898688
+.eh_frame_hdr               0   59611404
+.eh_frame                   0   59612128
+.tbss                       0   59619248
+.init_array                 0   59619248
+.fini_array                 0   59619256
+.data.rel.ro                0   59619264
+.dynamic                    0   89447152
+.got                        0   89447664
+.data                       0   89448448
 .go.buildinfo           10832   90249360
-.noptrdata             585943   90260192
-.bss                   470784   90846144
-.noptrbss               85584   91316928
+.noptrdata                  0   90260192
+.bss                        0   90846144
+.noptrbss                   0   91316928
 .comment                   31          0
 .debug_aranges            387          0
 .debug_info             15587          0
@@ -40,6 +40,6 @@ section                  size       addr
 .debug_line_str           688          0
 .debug_loclists          3434          0
 .debug_rnglists           212          0
-Total                91423053
+Total                21563002

I'm not quite sure which section is missing, but something important is definitely being skipped.

This is on latest main (68dcb27).

[brancz]

Thank you for reporting!

The agent attempts to extract just the debuginfo, not the actual executable section. It seems like there are others that we're missing that we shouldn't be zeroing or throwing away.

Could you take the original binary, and perform objdump --only-keep-debug on it, and perform the same diff. That should give us an initial clue as to which sections we differ from other tooling when we extract, then adding those sections not to be skipped/zeroed should be trivial.

If you can't wait to debug this, you can ad the --debuginfo-strip=false flag to your agent and it will upload the original binary as is.

[bobrik]

Copying just debug:

ivan@cube:~$ objcopy --only-keep-debug /usr/bin/dockerd /tmp/dockerd.debug

Produces a smaller file than parca does:

ivan@cube:~$ ls -l ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo /usr/bin/dockerd /tmp/dockerd.debug
-rw-rw-r-- 1 ivan ivan  38452051 Jan 27 06:01 /home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo
-rwxrwxr-x 1 ivan ivan  16917824 Jan 30 04:29 /tmp/dockerd.debug
-rwxr-xr-x 2 root root 107762880 Sep 20 11:41 /usr/bin/dockerd

It's still usable for addr2line:

ivan@cube:~$ eu-addr2line -f -e /tmp/dockerd.debug 0x22cda8d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1
/_/GOROOT/src/runtime/cgo/gcc_mmap.c:37:3

Here's the diff between parca and objcopy:

ivan@cube:~$ diff -rup <(size -A ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo) <(size -A /tmp/dockerd.debug)
--- /dev/fd/63	2025-01-30 04:31:31.450741826 +0000
+++ /dev/fd/62	2025-01-30 04:31:31.449741830 +0000
@@ -1,36 +1,36 @@
-/home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo  :
+/tmp/dockerd.debug  :
 section                  size       addr
-.interp                     0        848
+.interp                    28        848
 .note.gnu.property         32        880
 .note.gnu.build-id         36        912
 .note.ABI-tag              32        948
 .note.go.buildid          100        980
-.gnu.hash                   0       1080
+.gnu.hash             1254760       1080
 .dynsym               5171328    1255840
 .dynstr               8442653    6427168
-.gnu.version                0   14869822
-.gnu.version_r              0   15300768
+.gnu.version           430944   14869822
+.gnu.version_r            144   15300768
 .rela.dyn             7906632   15300912
 .rela.plt                2088   23207544
-.init                       0   23212032
+.init                      23   23212032
 .plt                     1408   23212064
 .plt.got                    8   23213472
-.text                       0   23213504
-.fini                       0   54898156
-.rodata                     0   54898688
-.eh_frame_hdr               0   59611404
-.eh_frame                   0   59612128
-.tbss                       0   59619248
-.init_array                 0   59619248
-.fini_array                 0   59619256
-.data.rel.ro                0   59619264
-.dynamic                    0   89447152
-.got                        0   89447664
-.data                       0   89448448
+.text                31684652   23213504
+.fini                       9   54898156
+.rodata               4712716   54898688
+.eh_frame_hdr             724   59611404
+.eh_frame                3636   59612128
+.tbss                       8   59619248
+.init_array                 8   59619248
+.fini_array                 8   59619256
+.data.rel.ro         29827888   59619264
+.dynamic                  512   89447152
+.got                      776   89447664
+.data                  800904   89448448
 .go.buildinfo           10832   90249360
-.noptrdata                  0   90260192
-.bss                        0   90846144
-.noptrbss                   0   91316928
+.noptrdata             585943   90260192
+.bss                   470784   90846144
+.noptrbss               85584   91316928
 .comment                   31          0
 .debug_aranges            387          0
 .debug_info             15587          0
@@ -40,6 +40,6 @@ section                  size       addr
 .debug_line_str           688          0
 .debug_loclists          3434          0
 .debug_rnglists           212          0
-Total                21563002
+Total                91423053

And here's the diff between the original binary and objdump, which you might find interesting:

ivan@cube:~$ diff -rup <(size -A /usr/bin/dockerd) <(size -A /tmp/dockerd.debug)
--- /dev/fd/63	2025-01-30 04:32:53.087395343 +0000
+++ /dev/fd/62	2025-01-30 04:32:53.088395339 +0000
@@ -1,4 +1,4 @@
-/usr/bin/dockerd  :
+/tmp/dockerd.debug  :
 section                  size       addr
 .interp                    28        848
 .note.gnu.property         32        880

Here's the docker package URL if you want to play around with it locally:

• https://download.docker.com/linux/debian/dists/bookworm/pool/stable/amd64/docker-ce_27.3.1-1%7edebian.12%7ebookworm_amd64.deb

[bobrik]

Non-stripped upload needed some fixing: #3031.

Even with full debug info uploaded, parca is unable to symbolize the stacks. Luckily with parca-dev/parca#5299 that can be fixed: