Graceful shutdown and health endpoints for Go services in containers
- Why shutdown matters
- The pattern
- Match the grace period
- PID 1
- Two health endpoints
- Background work
- Small images
- Logging during shutdown
- Test it
Go services start fast and use little memory, which makes them pleasant to deploy. Two small things decide whether deploys are actually seamless: handling SIGTERM correctly and exposing honest health endpoints. Here is a minimal pattern.
Why shutdown matters
During a deploy, the orchestrator sends SIGTERM to the old container, waits a grace period, then sends SIGKILL. If your service ignores SIGTERM, in-flight requests are cut off when the kill arrives. Users see errors on every deploy.
The pattern
srv := &http.Server{Addr: ":" + port, Handler: mux}
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatal(err)
}
}()
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()
<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
srv.Shutdown(shutdownCtx)
Shutdown stops accepting new connections and waits for active requests to finish, up to the timeout.
Match the grace period
Set the container stop timeout longer than your shutdown timeout. In Compose:
stop_grace_period: 30s
PID 1
Use the exec form in the Dockerfile so the Go binary receives signals directly:
CMD ["/app/server"]
The shell form wraps your process in /bin/sh, which may not forward SIGTERM.
Two health endpoints
- Liveness (
/livez): returns 200 if the process is running. Keep it trivial. - Readiness (
/readyz): returns 200 only when the service can handle traffic, for example after the database pool is connected. Return 503 once shutdown begins so the proxy stops sending new requests.
Flip readiness to failing at the start of shutdown, wait a couple of seconds for the proxy to notice, then call Shutdown.
Background work
Pass the shutdown context to workers and goroutines that process queues. Let them finish their current item and exit, rather than abandoning work halfway.
Small images
A static Go binary in a scratch or distroless image is a few megabytes. Remember to include CA certificates if the service calls HTTPS APIs, and time zone data if you parse local times.
Logging during shutdown
Log when the signal arrives, when readiness flips and when the server finishes. Those three lines make it obvious in deploy logs whether a slow shutdown was caused by a stuck request or a missing signal.
Test it
Run the container, start a slow request, then docker stop it. The request should complete. If it fails, your signal handling or grace period is wrong.
For the full path from build to domain on your own server, see this guide on deploying a Go app.
Disclosure: I work on Peon.
Write a comment