The Road to Reproducible Builds on the JVM

The Road to Reproducible Builds on the JVM

Published .
Written by Maarten Mulders , reading time roughly 13 minutes (2627 words).

A reproducible build means anyone can recreate your exact binary output from source code—bit-for-bit identical, every time. Beyond good engineering practice, this capability is increasingly mandatory under certain EU regulations that demand traceable, auditable software lifecycles.

Using Apache Maven and the Reproducible Central registry, you can achieve and demonstrate build reproducibility in three practical steps. This post describes how to do that for one of my personal projects.

Background

First of all, what do we actually mean when we talk about reproducible builds? In the scientific field, reproducibility describes how likely it is that a retry of an experiment would deliver the same outcome as an earlier run, given it uses the same source material. Translated to the software field, a reproducible build is a build that ensures the same binary code will be produced from a given set of source code and build scripts.

Apart from this being a nice definition, why would we bother? What value would this bring that will convince budget holders to allocate time, money and most importantly, backlog priority to this topic? There might not be a direct business need for this, but most business do care about obeying laws and complying with regulations.

There is no single EU‑wide law that explicitly says “you must have an auditable trail from source code to production”. But there are regulations and directives that create obligations in the field of traceability, change management, software lifecycle control and technical documentation and evidence. Together, they more or less require organisations to be able to demonstrate a traceable chain from source code, through build and deployment, into operational software.

Let’s have a quick look over some of these rules and highlight a few things that are relevant. I’m not a lawyer myself, so when in doubt, do consult with a specialist in the field!

  • The Cyber Resilience Act (CRA) requires organisations to provide full lifecycle traceability and SBOMs for their software. Put differently: organisations must be able to trace what code/components ended up in the production system and how.
  • The Digital Operational Resilience Act (DORA) requires organisations in the financial sector to have auditable (tamper‑proof, detailed, and long-retained) change management and logs. This includes all software changes to be deployed in a controlled manner and full auditability of production behaviour linked to change management.
  • The Network and Information Systems Directive (NIS2) does not explicitly state “code-to-production traceability”, but it does require a documented and auditable processes that provides evidence of secure development and deployment. This implies traceable pipelines for producing that software.
  • The all-new EU AI Act puts a direct requirement for end-to-end traceability and logging for so-called “high‑risk AI”.
  • And finally, the Medical Device Regulation (MDR) requires full lifecycle traceability, from design, through development into production and even post‑market. This includes the ability to trace requirements to code and tests, and to track updates and changes over time.

Some of these directives and acts may or may not be relevant to your organisation and/or project. Some probably are. The question is: how capable is your team, your organisation in aligning with them?

The (Bumpy?) Road

So lets see how we can contribute to a positive answer on that question using Apache Maven. Of course, our actions alone will not realise the above goals. On the other hand, not investing in a reproducible build could contribute to ultimately a negative answer, and we would like to avoid that.

The remainder of this post assumes you have a project that you publish to Maven Central. If you only publish internally (inside an organisation), step 2 and 3 are not applicable to you.

  1. Build and rebuild locally
  2. Build and rebuild externally
  3. Publish the results

I will guide you through the process taking my own project Maven Central Search (mcs) as an example.

1. Build and rebuild locally

The first thing we should do is follow the Maven guide Configuring for Reproducible Builds.

Using the artifact:check-buildplan goal, we might for example find this:

[INFO] --- artifact:3.6.1:check-buildplan (default-cli) @ my-project ---
[ERROR] Reproducible Build not activated by project.build.outputTimestamp property: see https://maven.apache.org/guides/mini/guide-reproducible-builds.html
[ERROR] plugin with non-reproducible output: org.apache.maven.plugins:maven-jar-plugin:3.1.0, require minimum 3.2.0
[INFO] No known issue in 7 plugins

This output tells us three things:

  1. There are 7 plugins are up-to-date and fully support reproducible builds.
  2. There is one plugin that is somehow outdated and does not guarantee a reproducible build. The culprit here is the Maven JAR Plugin; its version is roughly 6 year-old version. This plugin uses a shared component that does not reliably set the timestamp on files in a ZIP-archive. The “Reproducible/Verifiable Builds” page on the Apache Maven wiki has a section dedicated to sources of unreproducible bits.
  3. The Project Object Model (POM) does not define an project.build.outputTimestamp property. This property is necessary to ensure build reproducibility. Why so? Because without it, the timestamps of entries in JAR, WAR and other archives would be the current timestamp - making it unreproducible because every build happens at a different point in time. This property, on the other hand, ensures a predictable value, independent from effective build timestamp. Even if you build the project many years later, it will use the value of this property, ensuring the timestamp of each entry in the archive will be stable.

From this output, we have a clear, actionable list:

  1. Update the Maven JAR Plugin to version 3.2.0 or higher.
  2. Define the project.build.outputTimestamp property and assign it a value.

You might wonder what is a sensible value for the project.build.outputTimestamp property? Any date and time value formatted according to ISO-8601 will do! The release:prepare goal (part of the Maven Release Plugin) will automatically update it to the current timestamp once you prepare a new release. As a result, every result will have a “realistic” version.

After these two steps, running mvn artifact:check-buildplan now reports we’re all good.

Before we continue, let’s do a local check.

First, we run mvn clean install. You might’ve seen Andres’ repository of mvn-clean-install memes, but yes, in this case it makes sense to do so:

  • First clean so we really have an empty workbench to start with, just another person would have when they would build your project
  • Then install because we want the artifacts to end up in our local ~/.m2/repository folder for the next step.

Second, we run mvn clean verify artifact:compare.

  • Again, clean so we really have an empty workbench to start with.
  • Then verify to run all unit tests, integration tests and packaging. But there’s no need to install the resulting artifacts this time.
  • Finally, artifact:compare to compare the current build output (from package) against a reference. That reference is either previously install-ed (as we just did) or alternatively, it would be resolved from a remote repository. The comparison results will go to into a file named ${artifact}-${version}.buildcompare inside the target folder.

Inspecting that file, we’re looking for two lines:

ok=2
ko=0

If ko happens to be non-zero, inspect the line with koFiles to see which files differed.

But since we have ko=0, it means it’s time for the next step!

2. Build and rebuild externally

So far, all our tests and verifications where locally. But that does not really proof someone else will be able to produce the exact same artifacts from the code as we do. After all, we might unknowingly be injecting something into the built artifacts. In this next phase, we will completely isolate the build from our own system so it really appears as if someone else is running it. And if that succeeds, we will let someone else build it!

The process for this is again a three-step one:

  1. Create a .buildspec file
  2. Test the .buildspec file
  3. Enable automatic external verification

The first step is to create a fork of the Reproducible Builds repository that hosts a registry of Java projects with their reproducibility details. Next up, clone your fork locally and open it in an editor of choice.

Following their guide, here’s what we need to do:

Create a .buildspec file

The .buildspec file is a short, plaintext file that tells the registry about our project. It’s usually created in the content folder of the reproducible-central repository, which is structured the same way as your local Maven cache is. This file should, at the very least, include project coordinates, where the source code lives and how to build it:

# Project coordinates
groupId=it.mulders
artifactId=mcs
display=${groupId}:${artifactId}
version=0.10.2

# Project source code
gitRepo=https://github.com/mthmulders/${artifactId}.git
gitTag=v${version}

# Project build instructions
tool=mvn-3.9.15
jdk=25
newline=lf
command="mvn -Prelease clean package -DskipTests -Dmaven.javadoc.skip -Dgpg.skip -Dbuildinfo.detect.skip=false"
buildinfo=target/${artifactId}-${version}.buildinfo

Note that I have used some variable substitutions in this example - if they do not work for you, you could always write out the values in full.

The above sample basically says: to build MCS 0.12.0 (published as it.mulders:mcs:0.10.2 on Maven Central):

  • Checkout v0.10.2 from the Git repository found at https://github.com/mthmulders/mcs.git
  • Use Apache Maven 3.9.15 and Java 25
  • Run mvn -P release clean package -DskipTests -Dmaven.javadoc.skip -Dgpg.skip -Dbuildinfo.detect.skip=false.

Also observe that I disabled a few things:

  • All tests, because that is simply a waste of time here: that was already done in the build that I used for publication.
  • Javadoc generation, as that is not part of the binary artifacts that matter for reproducibility checking anyway.
  • GPG signing, as the verification tool doesn’t have my private GPG keys 🙂.
  • Detecting if it makes sense to generate a buildinfo file, as I am very much interested in those; I don’t want that to be automatically skipped because no deployment is taking place.

Test the .buildspec file

Now that we have a buildspec file, we need to double-check if it works as we think. The Reproducible Builds repository that we have just forked and cloned contains a script for this: rebuild.sh. Running it with one argument, the path to the buildspec file we’ve just written, will interpret the instructions and simulate a completely “clean” build. Initially, it will download a few Docker containers, so be prepared that it might take a while to complete.

Assuming all goes well, the last few lines of output will typically be

    ok=3
    okFiles="mcs-0.10.2.pom mcs-0.10.2.jar mcs-0.10.2-sources.jar"

If we’re less lucky, we might see something like this

    ok=1
    okFiles="mcs-0.10.1.pom"
    ko=2
    koFiles="mcs-0.10.1.jar mcs-0.10.1-sources.jar"

This summary says: one file, mcs-0.10.1.pom, could be reproduced exactly - bit-by-bit equality achieved! But two files, mcs-0.10.1.jar and mcs-0.10.1-sources.jar are not bit-by-bit equal to what was published in Maven Central. More on that later.

Enable automatic external verification

With all of this done, the last step is an easy one. Create a GitHub pull request against the base repository with your one .buildspec file in it. Submitting the pull request will trigger a build of the .buildspec, much like we just did. After that, there will be a _second* commit on your branch by “Reproducible Central CI”, adding a few more files: README.md, badge.json, maven-metadata.xml, a .buildcompare and a .buildinfo. From then on, you just have to wait; usually, a green build will be merged pretty quickly. After that, every new release of your project will automatically be checked and verified, and the README.md will be updated as well.

3. Publish the results

Since mcs is a public project, I like to demonstrate the result of this exercise. To me, a reproducible build means that the project has a mature way of building and publishing their software. This applies to end-user tooling like mcs, but even more so to libraries and frameworks that I would consume when building software. So let’s ensure we can add a nice badge to our project README.

For this, have a look in the README.md that was generated during the above pull request. There should be a line inside that starts with ## Project:. The badge typically looks like [![Reproducible Builds](.....); copy that whole piece into your projects README.md. If you have a project README in AsciiDoctor format, you’ll have to convert it first of course.

Congratulations, your project can now prove that its build and release process can be reproduced by an external party. This makes your project less vulnerable to certain (but not all!) supply chain attacks. If an attacker sneaks in some extra bits during the build and manages to get it published on Maven Central, this verification will detect that and turn your badge red!

Non-reproducible builds

Now what if things aren’t that easy?

Investigate

The first step would be to investigate why the verification files differ from the published ones.

A great tool to investigate the differences in binary files is diffoscope. It will run an in-depth comparison of files, archives, and directories. The nice part is that rather than saying “binary files differ”, it will actually try to interpret those files and tell you where these files differ. As an example, it can highlight an empty directory that is present in one ZIP archive, but not in the other. Or it might signal different file sizes for a file that is present in two JAR files.

Diffoscope has limited support for JAR files (or other ZIP files) on macOS due to macOS shipping an ancient version of the zipdetails program. I’ve reported this with the diffoscope team. To circumvent this, you can run diffoscope in a Docker container or try it online.

Causes for non-reproducible builds

Of course, it’s hard to list all possible causes for a non-reproducible build. I had one on the aforementioned mcs project that I’ll list. Feel free to react to this post with your own, and I might add them so others can learn from it, too.

  1. A non-checked in change in my project folder that trickled down in the build. Luckily, it was only an empty directory, but it was copied into the JAR file that got published on Maven Central. That one now belongs to the archives, but you can still see it in the version history for earlier mcs releases.

Another source of inspiration is the Reproducible/Verifiable Builds page on the Apache Maven Confluence space; look for the “Sources of unreproducible bits” section.

Adding an issue

Sometimes, the cause of a non-reproducible build isn’t immediately clear. You can still get published already, signalling that you are aware of the issue and working to resolve it. To do so, you need to add a line to the .buildspec file:

issue=https://github.com/mthmulders/mcs/issues/735

This information will end up in the README file on Reproducible Central. It is a transparent statement to your users: yes, the project is not yet reproducible. But the author(s) are aware of this, and want to address it.

Wrapping up

It’s time to wrap up.

This post started by discussing why a reproducible build is important, today even more than ever. Not only because as engineers, we are proud of being in control. There is an increasing amount of legislation that requires organisations to prove this.

Having a reproducible build is therefore in essence a review of your release and build practices. If all goes well, no “magic sauce” or “secret steps” are necessary for anyone to validate the path from source to binary code.

If you’re using Apache Maven, you have quite some tools available to work on this:

  • the check-buildplan goal of the Maven Artifact Plugin verifies the Project Object Model for obvious mistakes.
  • the compare goal of the Maven Artifact Plugin compares your build with a reference build.
  • the release:prepare and release:perform goals of the Maven Release Plugin embody a lot of good practices to minimise the chances of a non-reproducible build.

As a final remark, special thanks go out to Hervé Boutemy for his indispensable help on figuring out my own non-reproducible builds and his deep knowledge on this material.

And remember: if you found an interesting source of non-reproducibility in your project, leave a comment below (or contact me through one of my social media channels) to have it listed above.

comments powered by Disqus