The blocker was not ClickHouse#
Docker’s ClickHouse hardened image case study describes a familiar production failure: the container works, the database is ready, and the release is still blocked because the image fails a vulnerability scan.
According to Docker, the example involved a team preparing a Kubernetes production deployment and uploading a ClickHouse image to AWS ECR. The scanner reported three critical vulnerabilities. Docker says the findings were not in ClickHouse itself, but in the base image used to package it.
That difference matters technically, but it may not help during a release gate. Many security programs treat critical CVEs in production-bound images as blockers. The application team then has to prove non-exploitability, request an exception, rebuild the image, or delay deployment.
The useful lesson is broader than Docker’s product pitch: container security failures are often packaging failures. The vulnerable component may be present only because the base distribution brought it along.
Why full base images create noisy but real risk#
Docker says the standard clickhouse/clickhouse-server image is built on Ubuntu 22.04. A full distribution base is convenient because it includes familiar tools, package management, shells, system utilities, and broad compatibility.
It also includes packages the application may not need.
Docker names examples such as Perl, system utilities, apt, and transitive dependencies. ClickHouse may not use many of those components during normal operation, but scanners such as Trivy, Grype, and AWS ECR image scanning inspect what is present in the image. They do not reliably decide whether a vulnerable package is reachable in a specific workload path.
From the application team’s view, the finding can feel irrelevant: the database does not use the affected utility. From the security team’s view, the vulnerable package is present in a production image and the scanner marks it critical. If an attacker gets execution inside the container, unused tools can still become useful tools.
Risk exceptions can be valid, but they are expensive. Someone has to map CVEs to packages, assess reachability, document compensating controls, and get approval. That work repeats when scanners update, base images change, or new CVEs are disclosed. Removing unnecessary packages is often cleaner than repeatedly explaining them.
What the hardened image changes#
Docker says its hardened ClickHouse image starts from a smaller question: what does ClickHouse need to run in production?
That is different from starting with a general-purpose Linux base and accepting everything it includes. The post says Docker Hardened Images reduce the runtime image to required components and leave out tools that are not needed in production.
One important change is the absence of a package manager at runtime. Without apt, an attacker who gains a foothold inside the container has a harder time installing extra tooling from inside the image.
Docker also points to the removal of network utilities such as wget. The post says the standard ClickHouse image carried wget with an unpatched issue because there was no upstream fix. Docker’s hardened approach is not to patch that tool inside the image, but to avoid shipping it when ClickHouse does not need it.
The practical pattern is straightforward:
- remove package managers from production images;
- remove network download tools unless the workload requires them;
- keep debugging utilities out of runtime images;
- separate development convenience from production surface area;
- reduce the number of packages scanners can flag.
Docker says a shell remains available for operational work. That is still a tradeoff, but without a package manager and common download tools, the image gives less help to an attacker after compromise.
Put the dev/runtime split into the pipeline#
Minimal production images can cause friction if they appear too late. Developers test with one environment, security approves another, and production behaves differently.
Docker’s case study describes two image variants: a development image with extra tooling for local testing and debugging, and a runtime image with more of that surface removed for production deployment.
The workflow point is to test the hardened runtime image before the final security gate. If the first serious scan happens only when the image reaches ECR, the team may be forced into a rushed exception, rebuild, or delay. If the smaller runtime base is part of the pipeline earlier, scanner output should be less noisy and the remaining findings are more likely to matter.
This does not make ClickHouse secure by itself. It does not replace ClickHouse configuration, network controls, authentication, encryption, Kubernetes policy, secret handling, backups, or monitoring. It also does not guarantee that future scanners will report zero findings.
It changes the baseline: fewer packages, fewer known vulnerable components, fewer attacker-useful tools, and less exception paperwork for software the database did not need.
Practical checks for production containers#
For teams running ClickHouse, the first question is not only which ClickHouse version is deployed. It is what base image is carrying it.
Review the image bill of materials and scanner output. Separate findings in ClickHouse from findings in the operating system layer. If critical CVEs come from unused base packages, do not stop at writing exceptions. Ask whether those packages can be removed.
A useful review checklist:
- Is the production image based on a full distribution without a clear need?
- Does the runtime image include
apt,yum, or another package manager? - Are
wget,curl, or similar download tools present without a runtime requirement? - Are debugging tools included in the image that faces production traffic?
- Are development and runtime images separated clearly in CI/CD?
- Do scanners show CVEs in packages the application never uses?
The base image is part of what the team ships. Security scanners treat it that way, and production teams should too.