Purple white and red flowers.
Our garden is growing. We've raised a Series A funding round.
Read more here

Automated integration testing in Kubernetes using Garden

Adeyinka Adegbenro
Adeyinka Adegbenro
August 24, 2023

Minimizing the possibility of releasing software with bugs is always important. This is especially crucial in the continuous software development process where code changes are introduced iteratively and often. In this article, you'll focus on one particular aspect of quality assurance: automated integration testing in Kubernetes.

Integration testing is a phase in the software development testing cycle that often takes place after unit testing. While unit testing helps to test individual features of an application, integration testing integrates these features and tests them as a combined set to ensure that the application works together as expected. Integration testing is useful for identifying issues between services, enhancing test coverage, and improving reliability.

In this article, you'll learn how to set up integration tests for a Kubernetes workload using Garden, a DevOps automation tool used for rapid testing and development. This test setup will work both locally on your machine as well as in a CI/CD pipeline.

What is integration testing?

Integration tests often test interdependent functionalities of an application or service, such as calling the database from a backend API or connecting to a worker.

Although individually testing individual features may not reveal any issues, combining modules developed by different programmers can expose defects such as version incompatibility, API or network connection problems, insufficient exception handling, or inadequate logging. Integration testing helps to identify these issues early on. 

In addition, integration testing helps guarantee that the system maintains its expected workflows even after introducing new changes. This frees up workers who might have been stuck fixing undetected bugs so they can pursue more productive endeavors.

Integration testing keeps things running smoothly, ensuring that pods are functional and that services and Ingresses respond correctly at the specified hosts, ports, and paths.

Implementing integration tests in Kubernetes

In this tutorial, you're going to learn how to add Integration tests to a Kubernetes project. 


Before you begin, you need to complete the following prerequisites:

Once you've completed the prerequisites, you're ready to begin.

Clone the sample application

First, clone the sample application with the following:

git clone https://github.com/Damaso-DD/food_suggestions.git

Then navigate into the directory with the following:

cd food_suggestions

Deploy the project using Garden:

garden deploy
Garden deploy output

You should be able to visit the project page at http://food.suggestions.app.garden/. If the page doesn't load because the DNS address can't be found, you need to update your host file. 

You can update the host file at <span class="p-color-bg">/etc/hosts</span> by opening it in your preferred editor or via the terminal.

On macOS or Linux, use the following:

sudo vim /etc/hosts

On Windows, open Notepad as an administrator. Then open the <span class="p-color-bg">hosts</span> file in the <span class="p-color-bg">C:\Windows\System32\Drivers\etc</span> directory, add the following, and save it:

``` food.suggestions.app.garden api.suggestions.app.garden

At this point, you should be able to view the page:

Krusty Krab homepage

This is a basic food recommendation app that enables users to suggest new dishes for a restaurant's menu by filling out a form. You can interact with it by filling out the form and then clicking See other's suggestions to see more suggestions:

See other's suggestions page

Behind the scenes, the app is dependent on several small microservices:

  • The folder <span class="p-color-bg">food</span> contains a Node.js/Vue app running the application's user interface.
  • <span class="p-color-bg">api</span> contains the Flask backend API. 
  • <span class="p-color-bg">postgres</span> contains the service running the PostgreSQL database that stores the food suggestions.
  • <span class="p-color-bg">redis</span> contains the service running both a cache and a message broker that sends data from <span class="p-color-bg">api</span> to a Celery worker.
  • <span class="p-color-bg">worker</span> is the microservice that contains a Celery worker that pulls data from the Redis queue and persists the food suggestions to the database.

The application is also configured to be deployed to a Kubernetes cluster with the help of Garden. Notice how you don't have to create separate manifest <span class="p-color-bg">yml</span> files to define pods, deployments, services, or ingresses. Instead, you just have to define project level configurations using <span class="p-color-bg">project.garden.yml</span> and actions to execute for each microservice via <span class="p-color-bg">*.garden.yml</span> files.

For example, consider this <span class="p-color-bg">api/garden.yml</span> file:

kind: Build
name: api
description: Builds the backend container for the Restaurant Food Suggestion App
type: container
kind: Deploy
name: api
description: Deploys the backend service for the Restaurant Food Suggestion App
type: container
build: api
dependencies: [deploy.worker]
  args: [python, app.py]
	- name: http
  	protocol: TCP
  	containerPort: 8080
  	servicePort: 80
  	path: /health
  	port: http
	- path: /
  	hostname: "api.${var.base-hostname}"
  	port: http

In this file, two action kinds are defined for the API backend: <span class="p-color-bg">Build</span> and <span class="p-color-bg">Deploy</span>. The first builds the backend container using the corresponding <span class="p-color-bg">Dockerfile</span> and the second deploys the container on Kubernetes.

Adding automated integration tests

Although unit tests can help catch errors within isolated parts of an application, they're not sufficient for detecting all possible errors that can occur between services and the system as a whole. It's crucial to detect these errors before the services are deployed to production. To accomplish this, let’s create integration tests for the backend <span class="p-color-bg">api</span>. 

In the <span class="p-color-bg">api/tests/integration_tests</span> directory, create a file called <span class="p-color-bg">test_app_int.py</span> with the following contents:

# api/tests/integration_tests/test_app_int.py
import psycopg2

from app import app
from time import sleep
from unittest import TestCase

class TestIntegrations(TestCase):

    def setUp(self) -> None:
        self.db = self.connect_to_db()
        self.cursor = self.db.cursor()

    def connect_to_db(self):
        db = psycopg2.connect(
        return db

    def test_suggestions(self):
        # check that the redis cache "suggestions" is initially empty
        response = app.test_client().get('/food')
        self.assertEqual(response.status_code, 200)
        suggestions = response.json['suggestions']
        assert suggestions == []
        # send POST requests to the /food api
        foods = ['Rice', 'Yam', 'Beans']
        for food in foods:
            response = app.test_client().post('/food',
                        json={'food': food})
            self.assertEqual(response.status_code, 200)

        # check that the data is now saved in the postgresql database

        self.cursor.execute("SELECT food FROM food_suggestions;")
        db_suggestions = self.cursor.fetchall()
        # check that all the foods exist in database
        for food in foods:
            self.assertIn((food,), db_suggestions)
        # check that the data is now in the redis cache key "suggestions"
        response = app.test_client().get('/food')
        self.assertEqual(response.status_code, 200)
        redis_suggestions = response.json['suggestions']
        # check that all the foods exist in redis cache
        for food in foods:
            self.assertIn({'name': food}, redis_suggestions)


This integration test has been set up in the <span class="p-color-bg">test_suggestions</span> method to test all four layers of the backend (<span class="p-color-bg">redis</span>, <span class="p-color-bg">worker</span>, <span class="p-color-bg">postgres</span>, and <span class="p-color-bg">api</span>). It tests the creation of food suggestions by the API and makes sure that the food suggestions are stored in the database. This test also indirectly verifies that the worker functionality is working since the worker's job is to store suggestions in the database. 

The <span class="p-color-bg">test_suggestions</span> method not only verifies the created food suggestions but also confirms the presence of all the food suggestions in the Redis cache key <span class="p-color-bg">suggestions</span>. This approach indirectly tests the functionality of the Celery scheduled cron job <span class="p-color-bg">add-every-5-second</span> that updates the cache in the worker microservice. Since the integration test in the <span class="p-color-bg">api</span> backend already tests the interactions with other microservices such as <span class="p-color-bg">redis</span>, <span class="p-color-bg">postgres</span>, and <span class="p-color-bg">worker</span>, adding separate integration tests in each of them is unnecessary.

Once ready, you can run each test separately using the commands <span class="p-color-bg">garden test api-unit</span> and <span class="p-color-bg">garden test api-integ</span> which should return output like this:

Unit test output

And this:

Integration test output

Alternatively, you can run all tests using simply the command: <span class="p-color-bg">garden test</span>

With the tests out of the way, it's time to review how to take advantage of this functionality to solve some bugs.

Simulate a bug

Now that you've added some automated integration tests, let's introduce a bug in the code of one of the services to see if the integration tests work. 

Let's create a scenario where something goes wrong when you start up the Celery worker in the <span class="p-color-bg">worker</span> service. In <span class="p-color-bg">worker/Dockerfile</span>, remove the <span class="p-color-bg">arg "--beat"</span> on line 14 (*ie* change <span class="p-color-bg">CMD ["celery", "-A", "tasks", "worker", "--beat", "--loglevel=INFO"]</span> to <span class="p-color-bg">CMD ["celery", "-A", "tasks", "worker", "--loglevel=INFO"]</span>).

Remove –beat from dockerfile cmd

This modification will stop the Celery scheduled cron jobs from executing, which will consequently prevent the Redis cache key <span class="p-color-bg">suggestions</span> from updating with all the created food suggestions present in the database.

Now, see if you can catch the bug by running the following command:

garden test

As you can see, the unit tests run without any errors. However, as expected, integration tests fail:

Tests output

This occurs because the code checks whether the Redis cache was updated with all the food suggestions present in the database.

 After setting up your project, you can deploy it effortlessly with <span class="p-color-bg">garden deploy</span> and when finished, all you have to do is remove it using <span class="p-color-bg">garden cleanup deploy</span>. This streamlined process highlights the indispensability of Garden for developers conducting integration tests on Kubernetes. It provides a much-needed solution by creating production-like environments on demand. This valuable feature allows developers to test their services in environments mirroring the actual production settings, leading to more accurate and reliable results.

 Additionally, Garden can rebuild services as required. This feature is made possible through its Stack Graph, which identifies and quickly rebuilds or re-tests the necessary microservices. 

Moreover, Garden significantly expedites the testing process by providing a remarkable 80 percent acceleration in end-to-end testing. This substantial improvement minimizes wait times, enables faster feedback loops, and facilitates more rapid iteration within your work processes. For more information on how to add tests using Garden, check out the official documentation.


This article provided you with an understanding of automated integration testing and its significance in the software development process. You learned how to incorporate integration testing in a Kubernetes application using Garden, a tool that can be utilized to improve developer productivity.

Garden combines rapid development, testing, and DevOps automation into a single tool. With Garden, you get access to production-like development environments with minimal configuration required, and you’ll have more time to focus on software development. Visit Garden's official documentation to learn more.

Adeyinka Adegbenro is a software engineer based in Lagos, Nigeria, currently at BriteCore. She loves researching and writing in-depth technical content.

previous arrow
newt arrow