How would you rate your work-week productivity?
Is it a grind, slow churn, or smooth like butter?
Most of us got into programming because of the rush that comes with building worlds and understanding complex systems with the stroke of a key. However, a lot’s changed over recent years.
Debugging takes longer. Things that used to be simple have become slow, complicated, and error-prone. We spend more time than ever waiting.
In 2021, we commissioned an independent survey of 400 developers where 54% of respondents identify slow feedback loops during the development process as a top 3 frustration. Respondents spend, on average, more than 15 hours every week on tasks outside of writing application code: maintaining internal tooling, setting up dev environments, debugging pipelines, waiting for builds and test results, etc. Seven in ten respondents say the time they spend on specific tasks is time wasted and could be put to more strategic use.
I’d say as a whole we’re less productive and more frustrated. We haven’t lost the touch, but our tools are holding us back.
How did we get here?
Over the past ten years there was a tectonic shift away from monolithic architecture as systems got more complicated.
In large projects that involve dozens of teams and hundreds of developers, it’s easier to make rapid changes to smaller services than to one giant, complex monolith.
The struggles of microservice architecture are, however, very real.
Unlike a monolith, developing and testing a multi-service system requires separate configuration and tooling for each service. More and more processes now flow across service boundaries, increasing reliance on tests that check interactions. Things that were simple to unit-test now require integration tests, which take more time to write and debug.
Most teams resort to carefully curating a CI pipeline that runs the full suite of tests, often against a shared testing environment.
Top tech companies (Netflix, Google, Facebook) can afford to invest massively in in-house developer tooling, but 99% of the industry is left struggling, cobbling together makeshift solutions with sticky-tape and chewing gum.
This limits developers to running only unit tests before CI. Debugging failed integration tests means more round trips. The minutes spent waiting add up to hours and days, and we’re left feeling stymied and impatient at the end of the week.
But what we lose isn’t simply hours of productivity or half a year’s worth of an engineer’s salary. It’s the sense of wonder and flow that we once equated with our craft. There is a crisis of productivity that we’ve ignored for too long.
Bridging the capability cap
The best developer experience comes when working directly with tools native to the programming language and application framework.
That’s what made web frameworks like Django and Rails so successful and productive back in the day.
Core workflows tend to be faster, simpler and richer on a single service. Hot reloading, well-formatted logs, test frameworks, debuggers and REPLs bring immediacy to development and debugging.
The result is that your development environment understands and keeps up with your speed of thought.
That feeling of efficacy and free-ranging creativity is why a lot of us got into software. It’s the positive force that overpowers the thousand little frustrations we face daily.
Unless we reestablish parity between our development and production environments, our day to day will continue to look like that of an eighteenth-century composer who writes all the different parts for orchestra – flute, oboe, harp, etc. – only to hear if it sounds right on rehearsal day.
How to bring back the magic
Twenty years ago, developers used to be able to pick their language and framework and get to work. But as we started breaking things into several services, we had to come up with a different set of tools for each of them, and the CI system ended up being the only place all the pieces are put together. The result is a fragmented development experience with tons of context-switching and waiting.
Our development environments should be able to give us instant feedback on any part of the system as we iterate, even if we're changing logic that flows across service boundaries—and we should be able to run all of our test suites (including integration and end-to-end tests) without waiting for a CI pipeline.
What we need is a production-like development environment with consistent workflows that feels like one system without sacrificing the specialized tooling of each component. At Garden, we have our own spin on how to combine system level understanding with language-specific tools.
As an industry we have to break out of the mentality that we must develop one thing at a time and pray that the big picture fits later. Reestablishing parity between development and production is the key to making things feel cohesive again. We need tooling that understands how the system fits together and keeps up with the speed of thought.
That’s how we can get back into flow and bring back the magic to the craft of writing software.
Feature image by ESA/Hubble.
