Traditionally, companies have a dedicated staging environment to host the application for testing new features. There are a few significant downsides to this dedicated staging environment, like the expense of its upkeep and the rigidity of its developer experience. Rather than having an idle environment that provides nothing while costing something, it’s more efficient to spin up environments when they’re needed, terminate them when they’re not, and provide a lot more flexibility and room for innovation for your developers at the same time.
Let’s take a closer look at how exactly ephemeral environments can improve developer experience for your team. And, as there’s little benefit in adopting a practice that costs an engineering team more time, we’ll discuss how to automate them as well.
What are ephemeral environments?
Ephemeral environments, also known as dynamic environments or on-demand environments, exist for a short period solely to perform certain objectives in the application development lifecycle (e.g., testing new features, load-testing, and so on).
.png)
The benefits of ephemeral environments
Ephemeral environments provide several benefits across an organization, from developers to quality assurance testers to application stakeholders. Let's take a look at three in particular.
1. Early feedback in the development cycle
Traditionally, when developers complete an application feature in their local environment, they have to raise a pull request so key developers can review code changes before merging them into a staging environment for testing.
However, this process can slow down feedback cycles. For instance, if the feature’s code is well written but its functionality doesn't meet spec, the developer needs to fix it, then go through the whole pull request and merging process before it can be tested again.
What if a new pull request triggered an ephemeral environment instead? As key developers review code standards, other stakeholders can review feature functionalities without being blocked by anyone, speeding up the development cycle’s feedback loop.
2. Experimentation and innovation
Since ephemeral environments provide a faster route for early feedback, it’s a lot easier to experiment with new processes and ideas. In the traditional pull request and staging environment process, developers and stakeholders are more likely to be wary of experimentation because of delay in iteration.
3. Increased development velocity
When operations between development and infrastructure are smooth, development velocity tends to increase. Implementing ephemeral environments is one way to smooth out the development lifecycle, as some operations that used to happen in a series can now happen in parallel.
For example, testers can review application functionalities while code reviewers review the application code, and front-end developers can start testing backend APIs more quickly thanks to reduced delay in making APIs public.
4. Cross-team collaboration
With the isolation of an ephemeral environment comes the opportunity to collaborate across teams with lower risk.
Staging environments are traditionally accessible only to a few disciplines for the sake of security, consistency, and simplicity. Working outside of that, in a temporary and much more focused environment, you can afford to bring in more contextual input from tangential teams, like product owners and QA engineers. Even nontechnical departments, like design, marketing, and business development, can get insight into an in-progress project with less complexity and less security risk, eliminating cross-cutting concerns.
Characteristics of an ephemeral environment
Before we discuss setting up the infrastructure and tooling for your new testing approach, keep in mind that there are certain characteristics your ephemeral environment needs to possess. You’ll want to ensure that your EE template hits these standards.
It’s a replica of the production environment
An ephemeral environment must be a replica of your production environment, and it should be pretty obvious why. Both environments have to at least be similar enough that you can catch issues that could arise in production. Otherwise, the tests you run in your EE can’t be depended on for accurate results that you can confidently act on.
It uses replicated data
Ephemeral environments should use replicated data from production, or data that is similar to that in production. This is again to ensure that you’re testing the application with the same conditions that you expect in production.
However, it’s important that an ephemeral environment does not actually share data with other environments, especially production. Ephemeral environments are short-lived; once they’re terminated, all associated data gets deleted. Using replicated data allows you to test the application without fear of corrupting the current database or polluting the database with unnecessary data.
It’s easily shareable
It’s essential that an ephemeral environment be sharable. Application stakeholders such as users, product managers, and quality assurance testers must be able to access the application hosted on the environment to test, provide feedback, and check development progress. Shareable preview environments help foster collaboration.
Its spin-up is automated
Hopefully we’ve stressed this enough so far—the creation of ephemeral environments needs to be automated. If ephemeral environments aren’t automated, all of the benefits we’ve covered probably won’t be realized, or at least they won’t be enough to mitigate the extra time an engineering team needs. Spinning up and tearing down servers at will can be a complex process if left to be manual.
There are several tools available now, such as Terraform, Ansible, CloudFormation, and so on, that simplify the creation and automation of various infrastructure tasks needed for your ephemeral environments, like servers and networking, etc. These tools allow you to create a template of your ephemeral environment in a configuration file. When executed, the template creates the environment automatically, allowing you to create and destroy multiple environments without the hassle of performing complex infrastructure configurations manually.
Combining these tools with your online git repository, you can create triggers that spin up an ephemeral environment when you perform certain actions. Some examples might be raising a new pull request, pushing new code to a feature branch, or approving a pull request.
Your ephemeral environment’s infrastructure and tooling
Now that you understand what an effective ephemeral environment should look like, let’s go over the tools and infrastructure you’ll need to make it happen. Keep in mind that the proper tooling for you will depend on your project scope and the type of infrastructure your organization already uses.
Prepare your infrastructure
It’s important to know which cloud provider (AWS, Google Cloud, etc.) is best suited to handle your infrastructure based on many factors specific to your team, such as pricing, adaptability, and ease of use
Don’t forget to get access keys for your other automated tooling to use when connecting to your cloud provider. They’ll need to be able to create the desired ephemeral environments and destroy them when necessary.
Prepare Your IAC Tool
There are several tools you can use to create your desired infrastructure via a configuration file. The configuration file contains a list of declarative actions that you want the infrastructure-as-code (IAC) tool to perform, like setting up services such as network, storage, and databases.
Once this file is executed, the IAC tool creates and configures the services as declared in the configuration file. At that point, you need to know which IAC tool is right for you.
Notable IAC tools include Terraform, AWS CloudFormation, Ansible, and Pulumi, but again, you’ll want to select your IAC tools based on factors that will be specific to your team, like how its features will directly impact developer experience.
Prepare Your IDE
Preparing your integrated development environment can speed up the process of writing configuration steps for your chosen IAC tool.
By default, most popular IDEs like VS Code, PyCharm, and IntelliJ IDEA provide syntax highlighting and formatting. But to get the most out of your IDE, install the IAC’s associated extension on your code editor for added functionalities like autosuggestion, autocompletion, and error highlighting.
Configure CI/CD
CI/CD tools like GitHub Actions, Jenkins, and CircleCI provide the functionality of triggering the creation of your ephemeral environment when certain git actions are met.
For instance, your CI/CD tool could execute a Terraform file that creates an environment when new code changes occur in a feature branch, and then deploy the code. Afterward, if the branch is deleted or the pull request associated with the branch is closed, your CI/CD tool can trigger the destruction of the ephemeral environment.
Create Templates
The ability to create a template of your infrastructure in a file is what makes IAC shine. Rather than manually creating all the necessary services for your infrastructure by navigating to different parts of your cloud provider, you can define and configure those services in a template file.
These are the template files your CI/CD tools execute to create or destroy ephemeral environments while ensuring that they’re very similar to the production environment.
Ephemeral environments across your SDLC
Ephemeral environments are a great addition to your software development lifecycle. Efficiently automated, they can speed up the development process, improve the feedback loop, and promote innovation in your team.
However, the automation and configuration for each of the tools involved in a proper implementation of ephemeral environments (like IAC and CI/CD tools) can still be a lot of work. While this guide demonstrates an effective process for building ephemeral environments, the steps do take some time. This is especially true with microservices architecture, where each application requires its own ephemeral environment.
That’s where a tool like Garden becomes useful. Garden allows DevOps teams to [spin up preview ephemeral environments with the click of a button. Your engineers can develop in remote, production-like ephemeral environments without spending their time repeating the process shown in this guide.
Since Garden automates every step of the build, the resulting ephemeral environment is going to work as expected every time. Human error is reduced, and configuration drift is pretty much eliminated. Keeping builds within Kubernetes and only testing updated areas of the build means your engineers aren’t running into the bottlenecks common to more protracted processes. Your quality assurance team can also save time, thanks to Garden’s Stack Graph visibility into the build process from testing to deployment. Stakeholders can preview developer work and provide fast feedback, without disrupting your engineering team’s workflow.
Daniel Olaogun is a software engineer with five years of experience building software solutions in consulting, fin-tech, e-commerce, legal, personnel management, and education. Find him on Twitter or check out his personal website.
