An application developer, a member of the security team, and a site reliability engineer (SRE) all walk into a company gathering. Do they get along?
If they are working within an organization leveraging microservices the answer may depend on how well they can avoid talking about work. Each of these teams might have a label for the other that resides in the back of their mind … the cross-cutting concern people. There can be a lot of push and pull between these three teams, with SREs asking for central logging and monitoring, security imposing strict rules and developers left dealing more with these cross-cutting concerns than the actual service they're developing.
I believe we should all learn to get along and appreciate the value that each team brings to the table. All we need to do is make the concerns of each of these teams more manageable.
Let’s take a look at what cross-cutting concerns are, why they need to exist, as well as how to address them in the least painful way possible. We will find the same methods we use to address these cross-cutting concerns will also take the sting out of those less-than-perfect API contracts that may exist between services.
What are Cross-Cutting Concerns?
Cross-cutting concerns are items that depend on, or otherwise must provide for other parts of the system. For instance, the developer of a simple service might be asked to address the concerns of the SRE team, which wants to incorporate the developer’s service into a central logging solution. For the developer, it would mean not spending time building the core functionality of the service but instead working on the infrastructure around that service. This is extremely frustrating to a developer who didn’t sign up for this task.
And it’s not just infrastructure: Even other services that interact or depend on one another could be considered a cross-cutting concern. Expecting perfect contractual relationships between two or more APIs is not always realistic. These concerns have the potential to plague developers and cause them to spend more time working outside their domain.
This has led to a fair amount of cynicism toward the current state of microservice patterns. The initial promise of microservice utopia that a reasonably sized team can focus on a narrow-scoped solution, that they can rely on clean backward compatible contracts (APIs), feels broken. Whatever initial time- and focus-saving gains that microservice architecture provided for the developer has been extinguished through unplanned scope. One thing that seems to have been conveniently overlooked across the many “how to microservice” articles and videos out there is how to address cross-cutting concerns.
How Did We Get Here?
One of the key points that led to adopting microservices was the promise of separation of concerns. As long as each microservice provides a clean contract in the form of a backward- compatible API, the developers should be free to just focus on the service. So what gives?
Many tools claiming to eliminate cross-cutting concerns simply shift those concerns from the developer to someone else — or worse, replace one set of concerns with entirely new ones.
For example, service mesh solutions can move concerns such as security policy, observability, central logging and traffic routing out of the service onto the infrastructure. While that means the developer of the service is no longer responsible for implementing these types of policies themselves, the underlying concerns have not gone away. These concerns are now the responsibility of the team managing the service mesh.
The service developers, who were previously being asked to address these concerns by means they could rationalize and test against, now have two choices: 1) to push their services forward and pray that everything works fine in the new context of the mesh, or 2) figure out a way to incorporate the new mesh policy into their developer workflow. The first option will most likely lead to a lot of failed CI/CD runs, the latter just amounts to another set of cross-cutting concerns.
How to Address Cross-Cutting Concerns
There is no magic pattern that can make cross-cutting concerns disappear. Services need to be observed to ensure they are in a healthy state. Logs need to be accessible to the stakeholders responsible for troubleshooting the services. When the solution consists of many microservices, there needs to be a way of tracing a request’s life cycle, so if things aren’t performing properly, it can be determined which services are at play.
Beyond just the observability and diagnostics, we must also consider the system as a whole. All these microservices are collectively working for a bigger-picture solution. Just as two perfect-pitched singers can still sing out of harmony, so too can two or more perfectly functioning microservices get in the way of the larger goals of the system.
So rather than attempting to eliminate cross-cutting concerns, we should instead strive to reduce their impact on the developer’s productivity.
Expose Cross-Cutting Concerns Early
The best way we can reduce the impact of these cross-cutting concerns is to expose them as early in the development process as possible. To address cross-cutting concerns, or any other form of additional complexity, a developer will have to execute many iterations. These iterations get more and more expensive as far as time and pressure as they move closer to production. Therefore the best, most least expensive time to address these cross-cutting concerns would be during the early development process. This is the time where the developer has the most control and can iterate quickly.
Unfortunately, these cross-cutting concerns are often not addressed until much later in the development life cycle. This amounts to an extremely slow feedback loop for the developer as they try and rationalize how to get their code to coexist within an environment they have limited visibility on and even less control over.
One of the main reasons for this situation is the heavy reliance on a fixed pipeline to bring all these concerns together. The developer is at the mercy of the pipeline to get any feedback from the larger production-like environment. The more complexity, the larger the delta between what the developer is testing against and what needs to pass in order to be promoted into production.
Create Portable Pipelines
What developers need is the ability to address the production complexities within their development environment, allowing them to carry out the majority of the iterations in the fastest, least expensive way possible.
There are tools, such as Garden, that address these challenges by allowing for creation of a portable pipeline that defines basic actions such as build, deploy, run and test. These actions can be expressed with dependency order in mind, and because it's in a graph format, it can isolate and run only the steps required to complete a given task, making it very fast and efficient.
Such a standard portable format not only describes the steps to deploy an application but also accounts for other cross-cutting concerns. That’s because Garden can source other Garden projects, allowing for separation of concerns, such as a separate team who owns and manages service mesh policies.
With the graph and external sources defined, the developer can execute a simple Garden command such as `garden deploy`, or `garden test` to get feedback within seconds. This means that a developer can rationalize and test against concerns that would normally not be exposed until much later in the software development lifecycle, such after a pull request and complete CI/CD run for example.
This creates a fast feedback loop that allows developers to find out pretty quickly if the code they are working on has a conflict with another team's policies or even other services. Collaborating with another team to figure out the best fix will be a lot easier at this early stage than it would be knocking on the door to production.
Having a portable pipeline means that the same graph the developers are using within their environment can also be leveraged within a CI/CD pipeline. (In fact, using Garden can simplify the CI/CD pipeline down to a few commands.) This can open up a new paradigm: Instead of working from local development marching toward production, we start to imagine a world where we define our production state in portable pipelines and simply distribute the graph to the underlying stages, leaving no delta between production, CI/CD and the development environments.
In Conclusion
Despite the pain cross cutting concerns and other complexities cause the developer community, it should be noted that this pain also affects other stakeholders such as security, site reliability engineers, managers and others who can’t afford to have these concerns overlooked because they are too hard or overwhelming to manage. Hopefully when these different teams meet at the next gathering they can all hang out and enjoy each other's company as genuine collaborators and maybe even friends!
Check out Garden to get a deeper look into how we solve some of these challenges and let us embrace the complexities with a newfound confidence.
