Here’s what’s missing when developing distributed systems today:
Structure. Consistent, codified structure.
In a monolith, your code is your dependency graph. Everything is codified in the code. You can see every connection between every part of your system, jump to definitions, easily navigate the whole thing. Make a breaking change, and immediately see the red squiggly lines in your IDE.
As we break systems into multiple services, we lose this previously embedded context and spread it into unstructured definitions: Bash scripts, hopefully-up-to-date READMEs, scattered configuration files, CI pipelines, and — worst of all — people’s heads.
But when we hit a certain level of traffic, data volume, or number of developers, we simply _have to _split up our stack. And we don’t do it because it’s in fashion.
It is the only way to scale.
And because we had to do it, we as an industry forged ahead with it. Ever since, we have been trying to claw back at the simplicity, elegance, and rapid feedback we enjoyed when developing monolithic applications.
I am tired of clawing back.
Patching little holes, fixing fractions of the problem at a time, using a tool for this and a tool for that. It creates a new set of problems — you have to stitch so many little tools together, you end up spending more time on tooling than on your actual product. Instead of building products, we’re building plumbing.
What we need is structure.
We need a way to fully codify the path from our git repositories, through development, through testing, to an actual running system. It needs to be composable, portable, and understandable. Human-readable, and machine-readable. You should be able to visualize your system, what depends on what and how, and your tools should be able to do the same.
But how?
To achieve this, we should treat every part of our system as nodes in a graph. Each part should be able to describe itself: here’s how to build me, how to deploy me and test me, and these are the things needed before doing each of those.
Then your tooling finds this metadata and compiles a graph. Specifically, a directed acyclic graph (meaning: no circular dependencies). The graph fully describes your stack, at a certain commit, across all code in all your repositories.
In a monolith, this structure is just there. The compiler works without you worrying about it. The compiler doesn’t ask for manual input along the way — it either compiles or it doesn’t, and if it doesn’t it tells you what went wrong. There’s no confusion between versions or APIs of different parts.
The thing is, your system already is _a graph. We’re just missing the connections. They’re buried in opaque scripts, READMEs, someone’s brain in the other office five time zones away. The graph is there, but it’s not _structured.
The Stack Graph™
Since founding Garden we’ve been working on this concept: the Stack Graph.
As a concept, it’s pretty simple. It’s an opinionated graph structure, with fixed and predictable semantics covering the main workflows of backend development. Such as: building, testing, deploying, and how all those steps depend on each other.
Everything that happens within each of the steps is _pluggable. _Seasons change, platforms change, languages and services come and go. For example, we focus dev time today on supporting Kubernetes, but your core developer tooling shouldn’t be tightly coupled to it — or anything else.
The core primitives — building, testing, deploying — are portable. The tooling shouldn’t care how something is built, deployed, or run. It just needs to know when to do what, and which plugin is responsible.
Turn declarations all across your codebase into a single directed graph
Each module in your stack describes itself by adding a simple configuration file next to its code. This avoids massive, monolithic configuration files, and makes it easy to understand how to develop and operate each part of your stack.
The resulting graph, with each node associated with certain files or repos, determines exactly what needs to happen whenever you make a change. When I change this file, what needs to be rebuilt, what should be redeployed, and which tests need to be run?
The right abstractions
You may notice similarities to other tools and frameworks, notably Terraform and declarative build systems. We indeed draw inspiration from those, but wanted to expand on those ideas with specific semantics for modern development workflows.
Most importantly, we needed to be able to fully codify the path _from source to finish — _it’s not enough to just support infrastructure or builds. The graph needs to cover the whole journey to be truly portable, and to treat applications, tests and tasks as first-class citizens.
Figuring out how opinionated a system like this should be — and in what ways — has been our challenge from the beginning, but what we arrived at is surprisingly simple: Abstract around what developers need to happen, and leave plenty of flexibility in the how.
We need something that can stand the test of time, and work across multiple platforms. I think the Stack Graph strikes that balance. It’s flexible, but not so flexible that you lose the ability to reason about it.
From vision to reality
The Stack Graph provides the foundation, and Garden is the product we’ve created on top of it.
Think of Garden as an iterative compiler for your whole stack.
When you make a change, it works out exactly what needs to be re-built, deployed and tested. It provides a central view of your entire stack, and orchestrates your development workflow. And it makes your workflows understandable and portable.
And with Garden v0.10 onwards, you get all those benefits when working against remote Kubernetes clusters. No more docker builds and minikube burning up your laptop. Keep using any editor you like. Share a build cache and test results with the rest of your team, as well as your CI system. And CI becomes trivial to set up — just run Garden there as if CI were another developer. All possible because your system is fully described with the Stack Graph.
It’s still early, but we’re excited about our progress. We’re on our way to reclaim the developer experience we left behind when moving past the monolith.
We’ve already made a compelling tool for working with Kubernetes, which we hope you’ll find useful today, but we also hope you’re as excited as we are about this concept’s potential. These are just the first steps on our journey toward the development environment of the future.
Check out our project on GitHub. The Garden orchestrator is free and open-source, and we’d love to hear your feedback!
—
p.s. Huge thanks to Ellen Körbes for helping me out with this post! Dunno where we’d be without you ❤
