Grafana’s Missed Token Shows the Real CI/CD Risk

A Grafana Labs breach tied to the TanStack npm supply-chain attack appears to have turned on one missed GitHub workflow token.

According to BleepingComputer, Grafana detected malicious activity after compromised TanStack packages were consumed in its CI/CD workflow. Grafana rotated many GitHub workflow tokens as part of its response, but later found that one token had been missed. The attacker used that token to access private GitHub repositories.

What happened#

The incident sits inside the wider Shai-Hulud malware campaign attributed in the report to TeamPCP hackers. In that campaign, dozens of TanStack packages published to npm were infected with credential-stealing code.

Grafana’s environment was one of the affected developer environments. When the malicious npm package was released, Grafana’s CI/CD workflow consumed it. The infostealer component then executed inside Grafana’s GitHub environment and exfiltrated GitHub workflow tokens.

Grafana says it detected malicious activity from the compromised TanStack packages on May 1 and began incident response. That response included rotating GitHub workflow tokens.

The problem was scope.

Grafana rotated a significant number of tokens, but one token was not included. The company later found that a GitHub workflow it had originally judged as not affected had in fact been compromised. That missed token became the attacker’s path into Grafana’s private repositories.

Grafana had previously said the intruders stole source code. Its later review also found that the attacker downloaded operational information and business details used by Grafana. The company described that data as business contact names and email addresses exchanged in a professional relationship context.

Grafana says this was not customer production data and was not data pulled from, or processed through, Grafana Cloud production systems.

What Grafana says was not affected#

Grafana’s current position is that there is no evidence customer production systems or operations were compromised.

The company also says its codebase was not modified during the incident. Based on that finding, Grafana says code downloaded by users during the incident window is considered safe and that users do not need to take action at this time.

That matters. A private repository breach can create several different risks. Stolen source code is one. Modified source code is another. Compromised build or release artifacts are another. They are related, but not the same.

In this case, the public reporting says source code was stolen, but Grafana says the codebase was not changed. It also says production customer systems were not compromised based on the latest evidence and investigation.

That leaves two clear boundaries:

• Grafana has confirmed repository access and data theft.
• Grafana has not reported customer production compromise or malicious code modification.

Those boundaries should hold unless new evidence changes them. Grafana has said it will directly notify affected customers if its assessment changes.

Why the missed token matters#

This breach is a clean example of a common CI/CD failure mode: incident response depends on knowing every credential that matters.

Token rotation sounds simple. In practice, it depends on inventory, workflow mapping, permissions, ownership, and assumptions about which jobs were exposed. If one workflow is misclassified as safe, its token may survive the cleanup. If that token still has meaningful access, the incident continues.

The attacker did not need a second exotic exploit after the npm package compromise. The stolen workflow token was enough.

That is the uncomfortable lesson. Modern software supply chains often fail at the seams between systems: npm, GitHub Actions, repository permissions, build jobs, deployment paths, and secret stores. Each layer has its own logs and access model. During an incident, defenders have to reconstruct the actual trust graph under pressure.

A missed token is not a small clerical error when that token can read private repositories.

What teams should check#

There is no public user action required from Grafana customers based on Grafana’s current statements. But engineering and security teams can use the incident as a practical checklist for their own environments.

Start with CI/CD token inventory. Know which workflows can mint or use tokens, which repositories they can access, and whether their permissions are read-only or broader than needed.

Review package consumption in build environments. If a malicious dependency executes inside CI, assume it may have touched environment variables, workflow tokens, registry credentials, and any reachable secrets.

Revisit rotation playbooks. A good playbook should not only say “rotate tokens.” It should define how to find every token class, every workflow, every app installation, every deploy key, and every machine identity that could have been exposed.

Check workflow permission defaults. GitHub Actions and similar CI systems often allow teams to narrow token permissions. If a workflow only needs read access, it should not hold write access. If it does not need repository contents, it should not have repository contents.

Look for stale trust. Old workflows, disabled jobs, test pipelines, abandoned repositories, and one-off integrations often survive because they are quiet. During a supply-chain incident, quiet does not mean safe.

What not to overclaim#

The current public facts do not support claims that Grafana Cloud production systems were compromised. They also do not support claims that users downloaded backdoored Grafana code during the incident.

The breach is still serious. Private source code and operational business information have value. Attackers can use stolen code for vulnerability research, phishing, impersonation, or future intrusion attempts. Business contact data can support targeted social engineering.

But the distinction matters. Treating every repository breach as a production compromise can create noise. Treating it as harmless because production systems were not affected is also wrong.

The better read is narrower and more useful: a compromised package led to token theft; token rotation missed one credential; that credential allowed access to private repositories and data theft.

That is enough to study.

The deeper control gap#

The Grafana incident shows why supply-chain response cannot stop at removing the malicious dependency.

Once hostile code runs in CI, the question changes. It is no longer only “which package was bad?” It becomes “what could that package see, steal, or trigger from this execution context?”

That includes workflow tokens. It includes secrets. It includes repository access. It includes adjacent jobs. It includes business logic hidden in build scripts.

The hard part is that CI/CD systems are designed to connect things. They pull code, install dependencies, run tests, publish artifacts, authenticate to registries, and sometimes deploy. That connectivity is useful. It is also why a single compromised dependency can become a credential incident.

Grafana’s missed token is the kind of failure many mature teams can still make. The fix is not a slogan about zero trust. It is a boring control set: least privilege, complete inventory, scoped tokens, short lifetimes, reliable rotation, and incident playbooks that assume the first impact estimate may be wrong.

That is where the real lesson sits.