Automated integration testing in Kubernetes using Garden

Adeyinka Adegbenro

August 24, 2023

Minimizing the possibility of releasing software with bugs is always important. This is especially crucial in the continuous software development process where code changes are introduced iteratively and often. In this article, you'll focus on one particular aspect of quality assurance: automated integration testing in Kubernetes.

Integration testing is a phase in the software development testing cycle that often takes place after unit testing. While unit testing helps to test individual features of an application, integration testing integrates these features and tests them as a combined set to ensure that the application works together as expected. Integration testing is useful for identifying issues between services, enhancing test coverage, and improving reliability.

In this article, you'll learn how to set up integration tests for a Kubernetes workload using Garden, a DevOps automation tool used for rapid testing and development. This test setup will work both locally on your machine as well as in a CI/CD pipeline.

What is integration testing?

Integration tests often test interdependent functionalities of an application or service, such as calling the database from a backend API or connecting to a worker.

Although individually testing individual features may not reveal any issues, combining modules developed by different programmers can expose defects such as version incompatibility, API or network connection problems, insufficient exception handling, or inadequate logging. Integration testing helps to identify these issues early on.

In addition, integration testing helps guarantee that the system maintains its expected workflows even after introducing new changes. This frees up workers who might have been stuck fixing undetected bugs so they can pursue more productive endeavors.

Integration testing keeps things running smoothly, ensuring that pods are functional and that services and Ingresses respond correctly at the specified hosts, ports, and paths.

Implementing integration tests in Kubernetes

In this tutorial, you're going to learn how to add Integration tests to a Kubernetes project.

Prerequisites

Before you begin, you need to complete the following prerequisites:

Install Garden.
Install Docker Desktop. You'll need this to run Kubernetes. Once Docker Desktop has been installed, enable Kubernetes on Docker Desktop using this short guide.
Install Helm, Kubernetes package provider
Garden requires Kubernetes ingress controller to be properly configured. If in doubt, use a clean installation of Docker Desktop and Garden will take care of deploying and configuring ingress for you.

Once you've completed the prerequisites, you're ready to begin.

Clone the sample application

First, clone the sample application with the following:


```bash
git clone https://github.com/Damaso-DD/food_suggestions.git
```

Then navigate into the directory with the following:


```
cd food_suggestions
```

Deploy the project using Garden:


```
garden deploy
```

You should be able to visit the project page at http://food.suggestions.app.garden/. If the page doesn't load because the DNS address can't be found, you need to update your host file.

You can update the host file at /etc/hosts by opening it in your preferred editor or via the terminal.

On macOS or Linux, use the following:


```bash
sudo vim /etc/hosts
```

On Windows, open Notepad as an administrator. Then open the hosts file in the C:\Windows\System32\Drivers\etc directory, add the following, and save it:


```
127.0.0.1 food.suggestions.app.garden
127.0.0.1 api.suggestions.app.garden
```

At this point, you should be able to view the page:

This is a basic food recommendation app that enables users to suggest new dishes for a restaurant's menu by filling out a form. You can interact with it by filling out the form and then clicking See other's suggestions to see more suggestions:

Behind the scenes, the app is dependent on several small microservices:

The folder food contains a Node.js/Vue app running the application's user interface.
api contains the Flask backend API.
postgres contains the service running the PostgreSQL database that stores the food suggestions.
redis contains the service running both a cache and a message broker that sends data from api to a Celery worker.
worker is the microservice that contains a Celery worker that pulls data from the Redis queue and persists the food suggestions to the database.

The application is also configured to be deployed to a Kubernetes cluster with the help of Garden. Notice how you don't have to create separate manifest yml files to define pods, deployments, services, or ingresses. Instead, you just have to define project level configurations using project.garden.yml and actions to execute for each microservice via *.garden.yml files.

For example, consider this api/garden.yml file:


```yml
kind: Build
name: api
description: Builds the backend container for the Restaurant Food Suggestion App
type: container
---
kind: Deploy
name: api
description: Deploys the backend service for the Restaurant Food Suggestion App
type: container
build: api
dependencies: [deploy.worker]
spec:
  args: [python, app.py]
  ports:
	- name: http
  	protocol: TCP
  	containerPort: 8080
  	servicePort: 80
  healthCheck:
	httpGet:
  	path: /health
  	port: http
  ingresses:
	- path: /
  	hostname: "api.${var.base-hostname}"
  	port: http
```

In this file, two action kinds are defined for the API backend: Build and Deploy. The first builds the backend container using the corresponding Dockerfile and the second deploys the container on Kubernetes.

Adding automated integration tests

Although unit tests can help catch errors within isolated parts of an application, they're not sufficient for detecting all possible errors that can occur between services and the system as a whole. It's crucial to detect these errors before the services are deployed to production. To accomplish this, let’s create integration tests for the backend api.

In the api/tests/integration_tests directory, create a file called test_app_int.py with the following contents:


```python
# api/tests/integration_tests/test_app_int.py
import psycopg2

from app import app
from time import sleep
from unittest import TestCase

class TestIntegrations(TestCase):

    def setUp(self) -> None:
        self.db = self.connect_to_db()
        self.cursor = self.db.cursor()

    def connect_to_db(self):
        db = psycopg2.connect(
            host="postgres",
            database="postgres",
            user="postgres",
            password="postgres"
        )
        return db

    def test_suggestions(self):
        # check that the redis cache "suggestions" is initially empty
        response = app.test_client().get('/food')
        self.assertEqual(response.status_code, 200)
        suggestions = response.json['suggestions']
        assert suggestions == []
        # send POST requests to the /food api
        foods = ['Rice', 'Yam', 'Beans']
        for food in foods:
            response = app.test_client().post('/food',
                        json={'food': food})
            
            self.assertEqual(response.status_code, 200)

        # check that the data is now saved in the postgresql database
        sleep(15)

        self.cursor.execute("SELECT food FROM food_suggestions;")
        db_suggestions = self.cursor.fetchall()
        # check that all the foods exist in database
        for food in foods:
            self.assertIn((food,), db_suggestions)
        
        # check that the data is now in the redis cache key "suggestions"
        response = app.test_client().get('/food')
        self.assertEqual(response.status_code, 200)
        redis_suggestions = response.json['suggestions']
        # check that all the foods exist in redis cache
        for food in foods:
            self.assertIn({'name': food}, redis_suggestions)

```

This integration test has been set up in the test_suggestions method to test all four layers of the backend (redis, worker, postgres, and api). It tests the creation of food suggestions by the API and makes sure that the food suggestions are stored in the database. This test also indirectly verifies that the worker functionality is working since the worker's job is to store suggestions in the database.

The test_suggestions method not only verifies the created food suggestions but also confirms the presence of all the food suggestions in the Redis cache key suggestions. This approach indirectly tests the functionality of the Celery scheduled cron job add-every-5-second that updates the cache in the worker microservice. Since the integration test in the api backend already tests the interactions with other microservices such as redis, postgres, and worker, adding separate integration tests in each of them is unnecessary.

Once ready, you can run each test separately using the commands garden test api-unit and garden test api-integ which should return output like this:

And this:

Alternatively, you can run all tests using simply the command: garden test

With the tests out of the way, it's time to review how to take advantage of this functionality to solve some bugs.

Simulate a bug

Now that you've added some automated integration tests, let's introduce a bug in the code of one of the services to see if the integration tests work.

Let's create a scenario where something goes wrong when you start up the Celery worker in the worker service. In worker/Dockerfile, remove the arg "--beat" on line 14 (*ie* change CMD ["celery", "-A", "tasks", "worker", "--beat", "--loglevel=INFO"] to CMD ["celery", "-A", "tasks", "worker", "--loglevel=INFO"]).

This modification will stop the Celery scheduled cron jobs from executing, which will consequently prevent the Redis cache key suggestions from updating with all the created food suggestions present in the database.

Now, see if you can catch the bug by running the following command:


```bash
garden test
```

As you can see, the unit tests run without any errors. However, as expected, integration tests fail:

This occurs because the code checks whether the Redis cache was updated with all the food suggestions present in the database.

After setting up your project, you can deploy it effortlessly with garden deploy and when finished, all you have to do is remove it using garden cleanup deploy. This streamlined process highlights the indispensability of Garden for developers conducting integration tests on Kubernetes. It provides a much-needed solution by creating production-like environments on demand. This valuable feature allows developers to test their services in environments mirroring the actual production settings, leading to more accurate and reliable results.

Additionally, Garden can rebuild services as required. This feature is made possible through its Stack Graph, which identifies and quickly rebuilds or re-tests the necessary microservices.

Moreover, Garden significantly expedites the testing process by providing a remarkable 80 percent acceleration in end-to-end testing. This substantial improvement minimizes wait times, enables faster feedback loops, and facilitates more rapid iteration within your work processes. For more information on how to add tests using Garden, check out the official documentation.

Conclusion

This article provided you with an understanding of automated integration testing and its significance in the software development process. You learned how to incorporate integration testing in a Kubernetes application using Garden, a tool that can be utilized to improve developer productivity.

Garden combines rapid development, testing, and DevOps automation into a single tool. With Garden, you get access to production-like development environments with minimal configuration required, and you’ll have more time to focus on software development. Visit Garden's official documentation to learn more.

‍

Adeyinka Adegbenro is a software engineer based in Lagos, Nigeria, currently at BriteCore. She loves researching and writing in-depth technical content.