From Clojure to Java : Wrapping Ziggurat for Kafka Consumers at GoPay
At GoPay, I spend a lot of time working with Kafka. Like many engineering organizations, we have built a set of internal libraries to standardize how services consume events. One of the most widely used is Ziggurat.
According to its README:
“Ziggurat is a framework built to simplify stream processing on Kafka. It can be used to create a full-fledged Clojure app that reads and processes messages from Kafka.”
Ziggurat is a mature framework. It has been battle-tested in production for years and provides many features that simplify Kafka consumer development, including standardized application lifecycle management, error handling, retries, and stream-processing abstractions.
The Problem
I have nothing against Clojure. In fact, I genuinely enjoyed learning and using it. However, when I have a choice between two JVM languages, I naturally gravitate toward Java.
Startup time is another consideration. Clojure applications can take noticeably longer to start, which affects the local development and testing experience. There have been times when I pushed code to GitLab simply to run the test suite in CI because it was faster than waiting for the tests to complete locally.
It turns out I am not the only one who feels this way.
Whenever someone on my team needs to build a new Kafka consumer, there is usually a brief moment of hesitation. We all know how to use Ziggurat, but nobody is particularly excited about writing another Clojure application.
The business problem is usually straightforward.
The language is not.
Exploring JVM Interoperability
I knew that Clojure could use Java libraries, but I started wondering whether the reverse was also possible:
Can a Java application use a Clojure library?
Since both languages run on the JVM, it seemed like this should be possible. What began as a simple question quickly became an exploration of how Clojure applications are structured internally.
Using a Clojure framework from Java is not as straightforward as consuming a typical Java dependency. You need to understand how Clojure namespaces are loaded, how Vars are resolved, and how Clojure functions and data structures can be accessed from Java.
I spent hours reading documentation, browsing Stack Overflow, and experimenting with different approaches. There was no clear guide that explained the entire process end to end, so much of the work involved trial and error.
Looking back, I probably could have saved a lot of time with today’s AI assistants. At the time, however, I simply kept experimenting until everything finally clicked.
Borobudur
Those experiments eventually became Borobudur.
Borobudur is a thin Java wrapper around Ziggurat. It allows developers to build Kafka consumers in Java while continuing to use Ziggurat’s production-tested capabilities.
Rather than reimplementing Ziggurat or introducing another abstraction layer, Borobudur focuses on exposing the existing framework through a Java-friendly API. Developers can continue using proven infrastructure while writing their business logic in Java.
This approach offers several benefits:
- Existing operational knowledge of Ziggurat remains applicable.
- Kafka infrastructure does not need to be duplicated.
- Java developers can work in a language they are already comfortable with.
- The wrapper remains lightweight and easy to maintain.
Current State
Borobudur does not expose every feature available in Ziggurat yet. The current implementation focuses on the functionality required by our production workloads. Based on what I have learned while building it, supporting additional Ziggurat features should be relatively straightforward as new use cases arise.
Most importantly, Borobudur has moved beyond being a personal side project. It is now running in production at GoPay, where several Kafka consumers are built on top of it.
Lessons Learned
This project reminded me that valuable engineering tools often start with simple questions.
In this case, the question was not:
How do we build another Kafka framework?
It was:
Why does writing a Kafka consumer have to mean writing Clojure?
Instead of replacing an existing, well-tested framework, the better solution was to make it more accessible to the developers who already depended on it.
Borobudur is far from finished, and there are still many improvements to make. However, it has already achieved its original goal: creating a new Kafka consumer in Java is no longer something my team tries to avoid.
Sometimes the best engineering projects are not the ones that introduce entirely new systems. They are the ones that remove just enough friction to make everyone’s daily work a little easier.