3
Bugs fixed in lifecycle.go
~30
Dependencies updated
1.20 → 1.22
Go toolchain
1.25 → 1.41
Minimum Docker API
In this case study
The problem
Thousands of homelab operators and small teams rely on Watchtower as a “set it and forget it” way to keep containers updated. It watches running containers, pulls updated images when a registry has a new version, and recreates each container with the options it was started with.
In late 2024 the maintainers posted a deprecation notice and walked away. Without upstream work the codebase drifted: newer Docker SDK releases reorganised their types, modern daemons rejected the ancient API version the tool asked for, and the Dockerfiles still assumed a pre-compiled binary already existed on disk. A tool meant to reduce babysitting had become something you had to babysit.
Goal: turn the abandoned project back into a working, production-ready, drop-in replacement with no configuration changes for existing users, and document every change transparently.
Phase 1 — Code audit and bug fixes
The first pass was a full read of the container lifecycle and registry code. Three distinct defects lived in a single file.
lifecycle.go
Nil pointer dereference
Under certain conditions a pointer was dereferenced without a nil check, crashing the whole updater.
lifecycle.go
Silently swallowed errors
Failures from container stop operations were dropped instead of logged or surfaced, turning production debugging into guesswork.
lifecycle.go
Out-of-order log messages
Events were logged in the wrong sequence, so the timeline read backwards during post-incident review.
registry.go
Log ordering
Update checks appeared to finish before they began. Reordering the log calls restored a chronological story.
go.mod
Stale dependency graph
Roughly 30 packages were years behind. Go moved from 1.20 to 1.22 and every dependency was refreshed.
The major-version trap
The dependency refresh exposed a subtle mismatch: go.mod referenced ginkgo/v2 and robfig/cron/v3 while the source still imported the v1 paths. Go treats major versions as different modules, so the build simply failed. Pinning both back to their v1-compatible releases, ginkgo v1.16.5 and robfig/cron v1.2.0, aligned the manifest with what the code actually imports.
Phase 2 — Building from source inside Docker
The original Dockerfiles copied an already-compiled binary into a scratch image. There was no way to build without a full local Go toolchain. A new multi-stage Dockerfile compiles entirely inside Docker:
# Build stage
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN GOFLAGS="-mod=mod" go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux \
go build \
-ldflags="-X github.com/X4Applegate/watchtower/internal/meta.Version=${VERSION}" \
-o watchtower ./cmd/watchtower/
# Final stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates tzdata
WORKDIR /
COPY --from=builder /app/watchtower .
ENTRYPOINT ["/watchtower"]
GOFLAGS="-mod=mod" is doing real work here. The go.sum checksums were stale after the dependency update and could not be regenerated through GitHub’s web editor, so the flag lets Go resolve module requirements during the build instead of failing on the old checksums.
Phase 3 — Docker SDK v26 migration
Building against a current Docker SDK failed immediately with undefined: types.ContainerListOptions and friends. SDK v26 moved these types out of the top-level types package into types/container under new names.
| Old (SDK ≤ v25) | New (SDK v26+) |
|---|---|
types.ContainerListOptions | container.ListOptions |
types.ContainerRemoveOptions | container.RemoveOptions |
types.ContainerStartOptions | container.StartOptions |
Adding the github.com/docker/docker/api/types/container import and updating the three call sites in pkg/container/client.go got the build green.
Phase 4 — The one-line fix that mattered most
The first test run of the fresh binary died with: “client version 1.25 is too old. Minimum supported API version is 1.40.” A constant in internal/flags/flags.go had not been touched since 2017.
Before
const DockerAPIMinVersion string = "1.25"
After
const DockerAPIMinVersion string = "1.41"
Docker 20.10 and later require API 1.40 at minimum. 1.41 puts the fork in a safe range for any daemon shipped in the last several years.
v1.0.1 — Hardening the release
The first tagged release of the fork focused on the things that silently rot in a forked repository.
- Base images unpinned and staleAll four Dockerfiles pinned
alpine:3.19.0, a December 2023 patch release. They now usealpine:3.21for current CA certificates and time-zone data, and the unpinnedgolang:alpineis fixed togolang:1.22-alpine. - Version string never injected
Dockerfile.dev-self-containedstill pointed its ldflags at the old module path, so builds silently carried no version. Fixed, with agit describefallback for untagged builds. - Building the wrong repository
Dockerfile.self-containedwas cloning the abandoned upstream repo instead of the fork. It now builds this codebase. - Dead flags removed
GO111MODULE=onhas been a no-op since Go 1.17 and is gone; the Compose file and environment variables were simplified. - Transparent historyA
CHANGELOG.mddocuments every change from upstream, and the README carries a clear fork notice in place of the old deprecation warning.
Using the fork
It is a drop-in replacement: point your existing Compose file at the new image and keep your configuration.
services:
watchtower:
image: x4applegate/watchtower:latest
container_name: watchtower
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- WATCHTOWER_CLEANUP=true
- WATCHTOWER_POLL_INTERVAL=3600
- TZ=America/Los_Angeles
Prefer to build it yourself? No local Go toolchain is needed any more:
git clone https://github.com/X4Applegate/watchtower.git
cd watchtower
docker build -t x4applegate/watchtower:latest .
What’s next
GitHub Actions for automated builds and Docker Hub pushes, migrating the test suite from Ginkgo v1 to v2, and finishing the module path rename across every source file so the fork stands fully on its own.
Richard Applegate