A Few Quick Notes on Debugging With Delve

My employment depends on me writing a lot of Go, seems like a good idea to become familiar with the tools that make writing Go easier. A few weeks back I spent an afternoon learning how to be more effective with the debugger. Being more comfortable with $\texttt{dlv}$ has already been useful in tracking down a couple tricky bugs. Here are a few notes on using $\texttt{dlv}$.

• Install is simple, see below. On an ARM-based Mac there will be some additional silliness.

go install github.com/go-delve/delve/cmd/dlv@latest && PATH=$PATH:$(go env GOPATH)/bin

You can stop reading here if you’d like. The next few bullets are just a TLDR from the video. The TLDR of the TLDR is that help will do you a lot of good.

• Watch this video with one of the $\texttt{dlv}$ developers twice through, it will get you productive in $\lt 2\text{h}$.

• Start an interactive debugging session with dlv debug main.go.

  • break ${file}:${lineNumber} or break ${ns}.${function} to set breakpoints.
  • cond ${breakpointNumber} ${condition} to break only if ${cond} is true
  • toggle ${breakpointNumber} to enable/disable breakpoints
  • c to continue to the next breakpoint
  • s to step into the next line (up and down to move up and down the stack)
  • n to step over the next line
  • vars to get all variables and their namespace, args to get local context
  • p ${code} to execute code as if we’re in an interactive shell
  • r to rebuild and restart the program
  • watch -w ${object} set break on all writes to $object

• Trace with dlv trace main.go ${regexp} — I’m sure there’s some value here. Could be nice to set the regexp to ${module} and trace the execution of your module’s code. A little lighter touch than using the interactive debugger.

Delve has a server that you can attach to by exposing additional ports in your container image. Not ideal IMO. One can pass the following arguments to $\texttt{CMD}$ to do this. I’m told the following integrates better with IDEs, etc.

  CMD ["/dlv", "--listen=:40000", 
      "--backend=default", "--headless=true", 
      "--api-version=2", "--accept-multiclient", 
      "exec", "/bin/main"
  ]

• Debugging in a container can be made very easy. Most expedient way is just adding $\texttt{dlv}$ to your Dockerfile and then attaching to the running process. There is some downside here. By default $\texttt{dlv}$ produces an unoptimized binary (e.g. $\texttt{gcflags='-N -l'}$) to prevent the compiler from making choices that can make debugging harder. Small price to pay, proceed with caution…

  git diff default.Dockerfile debug.Dockerfile                                                        
  diff --git a/default.Dockerfile b/debug.Dockerfile
  index b836d46..7032666 100644
  --- a/default.Dockerfile
  +++ b/debug.Dockerfile
  @@ -4,9 +4,11 @@ FROM golang:1.21-alpine AS builder
   WORKDIR ./
   COPY ./main.go ./main.go
   RUN go build -o /bin/main ./main.go
  +RUN go install github.com/go-delve/delve/cmd/dlv@latest
  
   # stage 2: final img
   FROM alpine
   COPY --from=builder /bin/main /bin/main
  +COPY --from=builder /go/bin/dlv /
  
   EXPOSE 4221 ## for the application, not for the debugger...
   CMD ["/bin/main"]

  docker build . -f debug.Dockerfile -t svc:debug
  docker run --name pong --cap-add=SYS_PTRACE -p 4221:4221 svc:debug
  docker exec -ti pong sh -c "/dlv attach 1"

I’d need to read up a bit more on system capabilities. One can run docker with --sysctl ${cap}, but granting container code admin access to the host seems like a disaster in the making…

• I haven’t gotten the chance to play with the rr backend yet. The idea is that you can set --backend=rr and then the session is recorded. Getting this to work safely, correctly, and in a container on M1 seems challenging. We’ll see…