📦 Cache Me If You Can

LabyMod, a popular Minecraft client, allows users to create and publish game addons - small custom modifications adding quality of life features, gameplay tools, and UI elements.

Such modifications can be easily discovered and installed directly from inside LabyMod's addon store. Since addons are executable code, and can potentially contain malware, addon code is closely curated and manually reviewed for every update. As a result, the safety of the downloaded addons is tacitly guaranteed by the staff.

In this post, we will conclude a deep dive into the internal pipeline for user-created addons, identify sources of truth that make the system secure, and then expose a fatal security flaw that made malware distribution possible for months. Let's get started!

Current LabyMod approach attempts to leave zero gaps for malicious actors.

First off, all addons must be open-source and published on either GitHub or GitLab, viewable by anyone. All addons are based on the provided template which dictates the overall structure of the addon, dependencies used to integrate with the Minecraft client, and repositories to fetch dependencies from.

This template also exposes a mandatory GitHub Workflow file that automates the process of building the addon. Every time a code change is committed to the repository, the workflow automatically builds the code into a ready-to-use .jar addon file in the form of an artifact.

Whenever a user wishes to publish their addon, they must provide the link to the corresponding GitHub repository. The staff will then check the complete repository state at the given latest commit, the integrity of the Actions workflow file, and publish the resulting artifact.

This creates an extremely high trust environment. Under the ideal workflow, the malicious actor does not possess the capability to upload arbitrary executable files, and any attempts to modify the dependencies, the code, or the workflow file will be immediately evident to the staff reviewing the repository - should malware be added, it will be present at the precise commit behind the artifact.

The transparency provided by this approach makes addon audits simple and straightforward, and leaves a lot of freedom for independent review by both staff and other community members, effectively eliminating the possibility of malware distribution.

So then, if this is the perfect approach to addon publishing, what could possibly be wrong with it?...

One word: optimization.

The addon template provided by LabyMod implements caching. Instead of fetching Gradle wrapper and dependencies every time the workflow runs, both are downloaded once and then stored on GitHub side in the form of an independent immutable storage.

This allows GitHub CI/CD Actions to stay optimized and complete within seconds, as opposed to taking minutes every time the code is edited.

On the surface, this doesn't tell us much - after all, the cache storage is immutable, and is fetched by the workflow. Logically, editing the workflow file to alter that behaviour, and to modify the cache, would be immediately evident and reflected in the commit behind the workflow, thus defeating the attack.

However, that would miss one important fact - GitHub Actions Cache is persistent across different commits, even across commits that are no longer a part of repository's commit history. No other attacker-controlled "variable" is persistent in the way the cache is. Not only that, the GitHub cache is completely opaque; it cannot possibly be reviewed. There is, essentially, no way to tell what's inside of the GitHub cache.

That persistency singlehandedly enables the devastating CI/CD cache poisoning attack. Let's see how it's done.

Since the attacker has complete control over their own repository, they can first corrupt the cache by creating malicious commits, and then rewrite the commit history completely.

Under normal circumstances, rewriting the malicious commit history to a legitimate one will also undo any hacker damage, as all malicious code will also be removed and will not be considered in new builds. However, because the Workflow allows caching, and does not validate the integrity of cache contents, the attacker can actually infect the build tool in a way that malware persists, even after a complete history rewrite.

In this case, once the history is rewritten, there are zero signs of tampering. In fact, it is impossible to tell that the trusted environment is infected. Meanwhile, all consecutive legitimate builds generated by GitHub Actions will be completely under attacker's control, allowing one to push arbitrary code (including malware), leading to a successful arbitrary code execution.

As you can see there, the master branch only contains just two commits. The first commit corresponds to the precise copy of LabyMod/addon-template repository, commit f4125ec6b08eb1a493085deb3caee388f5e080ee. The very next commit, c670523, only adds a README.md text file.

And indeed, heading over to https://github.com/kolya5544/PoC_CI_Poison/actions we can see two runs: Initial commit, and the latest one - Legitimate README.md commit. However, if we download and run the artifact produced by the second run (aka the final addon jar), we will discover that the .jar does not match the source code, and instead of being just an example addon, it launches calc.exe if run, demonstrating successful delivery of an arbitrary executable through a trusted GitHub Actions workflow.

You can find the missing commit on demonstration branch. Of course, in a real hacker attack, there would be no such branch, and the attack would be nearly completely invisible.

LabyMod has swiftly and efficiently closed the vulnerability by hardening the supply chain. The vulnerability has been completely rectified by changing the caching architecture from restore-keys to gradle/actions/setup-gradle, as well as by introducing a SHA256 hash to verify the Gradle distribution against. This effectively renders the vulnerability impossible.

There are no reasons to believe that any addons have ever been infected through the use of this vulnerability.

This vulnerability demonstrates that a build pipeline is only as trustworthy as its least-audited input, and that trust in a commit history should not extend to trust in produced artifacts. This is not just about LabyMod or Minecraft - any open-source software that utilizes workflows to deliver releases is susceptible to the same issue. Even downloading the .exe from a workflow that built that .exe from current source code, you can audit the code, the commit history, and still miss the fact that .exe is malicious.

I believe this exposes a profound trust boundary issue, and that CI cache poisoning vulnerabilities can be used to covertly deliver malware through legitimate open-source projects. In the majority of cases it means that, unless extra steps have been taken by the repository maintainer to ensure pipeline safety, there is no way to be certain that the produced artifact matches the audited code.

So, next time you use CI/CD workflows, stay cautious of this behaviour. Warn developers that actively deliver updates via GitHub Workflows. Most importantly, stay safe!