A Kubernetes operator is a software extension to Kubernetes that enables you to manage any application, including Kubernetes itself, with custom resources. Operators apply the rules of controllers, which are the control loop mechanism on Kubernetes.
Testing Kubernetes operators tends to be difficult. Even if you can test the components by mocking them, it’s not like testing in a real Kubernetes environment. Setting up a Kubernetes environment for a test process isn’t easy, though. You need to automate the Kubernetes start-stop actions and be able to properly configure it for each test.
However, Testcontainers can simplify this testing process considerably. It’s a library that helps you run specific technologies in Docker containers to be used in integration tests. You can use it to quickly spin up a containerized application instance in a Docker container before your tests start. When your tests finish, your container is stopped.
Testcontainers libraries are available with different clients for different languages and frameworks. Its K3s module uses lightweight Kubernetes K3s to help developers write integration tests for anything that runs on Kubernetes, including operators.
In this article, you’ll learn how to use Testcontainers to test Kubernetes operators. You’ll configure a Quarkus-based Kubernetes operator to use Testcontainers’ K3s module and add some tests to confirm the operator code works as expected.
You can find the code for this tutorial in this repository.
Prerequisites
- JDK 17+
- Any of the supported container runtimes
Tutorial Scenario: Creating a Game Application for GameContainers Inc.
For this tutorial, imagine you’ve just started working for a game development company called GameContainers Inc. The company needs a new game system that runs on Kubernetes and can be managed as a Kubernetes resource.
Luckily, an open source project named KubeGame can provide you the scaffolding for a Kubernetes operator for gamification. However, you soon realize this project has no tests written for it, so you decide to add some, starting with integration tests that run the operator on a real Kubernetes cluster. For this, you decide to use Testcontainers K3s module.
The rough architecture diagram for the operator and the requested state is illustrated in the following diagram:
The following sections provide a hands-on tutorial on how to implement the requested architecture.
Examining the KubeGame Kubernetes Operator
Before you write your tests, let’s first examine the operator code.
Start by cloning the source code onto your local machine:
git clone https://github.com/testcontainers-community/testcontainers-kubernetes-operator-demo
Open the testcontainers-kubernetes-operator-demo folder with the editor of your choice. You can use any IDE that supports Java or Quarkus, such as IntelliJ IDEA or VSCode.
The application is an operator that was developed by using the Quarkus Operator SDK, which uses the Java Operator SDK.
If you navigate to the src/main/java folder, you will see different folders under the com.systemcraftsman.kubegame package:
| Package | Content |
| com.systemcraftsman.kubegame.customresource | The custom resources of the operator |
| com.systemcraftsman.kubegame.reconciler | The reconcilers of the operator, including the controller logic |
| com.systemcraftsman.kubegame.service | Quarkus services, which are application-scoped Java beans. They separate the business logic from the operator controllers. |
| com.systemcraftsman.kubegame.spec | The spec classes used in the custom resource |
| com.systemcraftsman.kubegame.status | The status classes used in the custom resource |
The current state of the operator simply needs two custom resources.
The Game custom resource, which deploys a Postgres database and creates its Kubernetes service, is the main resource for a game.
The second one is the World custom resource. When an instance of this custom resource is created in Kubernetes, it accesses the Postgres database, creates a world table if it doesn’t exist, and then inserts a record of it with its name.
Here is a diagram of how the operator works and manages the components:
Setting Up Testcontainers
Quarkus already uses Testcontainers behind the curtains in its developer helper service called Dev Services. For example, if you need a Kubernetes or an Apache Kafka instance, it spins it up before running the tests or your application and makes those available for your test or development environment.
However, because Quarkus doesn’t yet support Testcontainers’ official K3sContainer module, in this tutorial, you will not use Dev Services. Instead, you’ll use Testcontainers directly because it provides more control of the Testcontainers configuration and let’s you use the official containers.
Deactivate the Kubernetes Client Dev Services by configuring the following property in the src/main/resources/application.properties file.
%test.quarkus.kubernetes-client.devservices.enabled=false
Next, you must create your own Testcontainers container instance to use in your test.
Add the following Testcontainers dependencies in the pom.xml file.
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>junit-jupiter</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>k3s</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
These dependencies enable you to use Testcontainers and its K3s module.
Next, create a Java class file called K3sResource.java under the com.systemcraftsman.kubegame.test package and add the following content to it:
package com.systemcraftsman.kubegame.test;
import io.quarkus.test.common.QuarkusTestResourceLifecycleManager;
import org.testcontainers.k3s.K3sContainer;
import org.testcontainers.utility.DockerImageName;
import java.util.Collections;
import java.util.Map;
public class K3sResource implements QuarkusTestResourceLifecycleManager {
//Initializes the K3sContainer instance.
//It uses the Docker image "rancher/k3s:v1.24.12-k3s1"
static K3sContainer k3sContainer = new K3sContainer(DockerImageName.parse("rancher/k3s:v1.24.12-k3s1"));
//Start method is one of the methods in the interface QuarkusTestResourceLifecycleManager
//This method runs when a test lifecycle is started
//In this case, it is used for starting the container and setting the kubeConfigYaml value into a property.
@Override
public Map<String, String> start() {
k3sContainer.start();
return Collections.singletonMap("kubeConfigYaml", k3sContainer.getKubeConfigYaml());
}
//Stop method is one of the methods in the interface QuarkusTestResourceLifecycleManager
//This method runs when a test lifecycle is stopped
//In this case, it is used for stopping the container
@Override
public void stop() {
k3sContainer.stop();
}
}
As you can see from the commands in the code, the K3sResource class is responsible for a test lifecycle by implementing the QuarkusTestResourceLifecycleManager interface and its start() and stop() methods. Using this class as a test resource gives you a ready-to-use K3s cluster until the test stops. The start() method not only starts the container but also assigns the Kubernetes configuration of the K3s cluster to a property value called kubeConfigYaml.
In the same directory, you have a previously created class called OperatorFunctionalTest, which is your test class. To enable this test to use the K3sResource you’ve created, open the OperatorFunctionalTest.java class and add the following annotation above the class definition:
K3sContainer
@QuarkusTestResource(K3sResource.class)
...code omitted...
public class OperatorFunctionalTest {
...
...
}
Your test is now able to spin up a K3s cluster before the test starts.
However, the Quarkus context, where your tests must run, should know about your K3s cluster or the tests will fail.
Quarkus uses a KubernetesConfigProducer Java bean for setting up the Kubernetes configuration on Dev Services. By overriding this bean, you can use the K3s cluster that you’ve set up.
To do so, create a Java class called K3sConfigProducer in the com.systemcraftsman.kubegame.test package with the following content:
package com.systemcraftsman.kubegame.test;
import io.fabric8.kubernetes.client.Config;
import io.quarkus.arc.Priority;
import io.quarkus.kubernetes.client.runtime.KubernetesClientBuildConfig;
import io.quarkus.kubernetes.client.runtime.KubernetesConfigProducer;
import io.quarkus.runtime.TlsConfig;
import org.eclipse.microprofile.config.inject.ConfigProperty;
import javax.enterprise.inject.Alternative;
import javax.enterprise.inject.Produces;
import javax.inject.Singleton;
@Alternative
@Priority(1)
@Singleton
public class K3sConfigProducer extends KubernetesConfigProducer {
//Injects the kubeConfigYaml that you've set in the K3sResource
@ConfigProperty(name = "kubeConfigYaml")
String kubeConfigYaml;
//Returns the kubeConfigYaml as the config
@Singleton
@Produces
public Config config(KubernetesClientBuildConfig buildConfig, TlsConfig tlsConfig) {
return Config.fromKubeconfig(kubeConfigYaml);
}
}
When you run your tests in the OperatorFunctionalTest class, Quarkus uses the bean class above to override its own so that you can use the configuration of the K3s cluster you’ve set up for running your tests.
Adding Tests for the Game Resource
To verify whether your test setup works, you must add some tests.
Let’s start by adding some tests in your test class to see your setup in action and test the operator.
Open your OperatorFunctionalTest class. Notice that you have three methods already created. The @TestInstance(TestInstance.Lifecycle.PER_CLASS) and @TestMethodOrder(MethodOrderer.OrderAnnotation.class) annotations on the OperatorFunctionalTest class ensure that the test lifecycle is per class and the tests are ordered by methods. The @Order annotations on the test methods define their order.
Navigate to the testGame() method, which is the first test method to be run in this test class. Add the following content to it:
//Apply the oasis.yaml resource, which is a Game resource.
client.resources(Game.class).inNamespace(NAMESPACE)
.load(getClass().getResource("/examples/oasis.yaml").getFile()).create();
//Get the Game instance for "oasis"
Game game = gameService.getGame("oasis", NAMESPACE);
//Assert the "oasis" game is not null.
Assert.assertNotNull(game);
//Postgres deployment and its being ready takes time.
//Wait for at most 60 seconds for the assertions.
await().atMost(60, TimeUnit.SECONDS).untilAsserted(() -> {
//Get the Postgres deployment instance of the related game.
Deployment postgresDeployment = gameService.getPostgresDeployment(game);
//Assert the deployment object's being not null
Assert.assertNotNull(postgresDeployment);
//Assert if the deployment is ready
Assert.assertEquals(Integer.valueOf(1), postgresDeployment.getStatus().getReadyReplicas());
//Assert if the "oasis" game status is ready
Assert.assertTrue(gameService.getGame(game.getMetadata().getName(), NAMESPACE).getStatus().isReady());
});
The comments on the code above allow you to examine what the method does. The method applies a game resource from a YAML file to create a game called oasis in the system, and it then verifies the oasis game’s existence on Kubernetes. If it does exist, it verifies if the dependent resources—such as the Postgres deployment and its service—exist and their state is ready. Because the game instance cannot be ready before its dependencies, the test finally verifies the game instance’s readiness.
Next, you’ll run your test to see it in action. In your project directory, run the command: ./mvnw test
The test might take a few minutes to complete and the successful test output should be similar to the following:
...output omitted... 2023-04-19 15:04:07,017 INFO [org.tes.DockerClientFactory] (pool-4-thread-1) Connected to docker: Server Version: 20.10.21 API Version: 1.41 Operating System: Docker Desktop Total Memory: 7859 MB 2023-04-19 15:04:07,086 INFO [🐳 .3.4]] (pool-4-thread-1) Creating container for image: testcontainers/ryuk:0.3.4 ...output ommited... 2023-04-19 15:04:33,390 INFO [io.jav.ope.Operator] (main) Operator SDK 4.2.7 (commit: 5d8c567) built on Fri Feb 10 20:07:04 TRT 2023 starting... 2023-04-19 15:04:33,391 INFO [io.jav.ope.Operator] (main) Client version: 6.3.1 2023-04-19 15:04:33,393 INFO [io.jav.ope.pro.Controller] (Controller Starter for: gamereconciler) Starting 'gamereconciler' controller for reconciler: com.systemcraftsman.kubegame.reconciler.GameReconciler, resource: com.systemcraftsman.kubegame.customresource.Game ...output ommited... Apr 19, 2023 3:05:26 PM io.quarkus.bootstrap.runner.Timing printStopTime INFO: kubegame stopped in 0.045s [INFO] [INFO] Results: [INFO] [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:36 min [INFO] Finished at: 2023-04-19T15:05:29+03:00 [INFO] ------------------------------------------------------------------------
As you can see, the K3s container starts successfully, and then the operator starts running on it. In the end, the tests are run, and depending on the assertions, they become successful. The [INFO] BUILD SUCCESS message means that all three tests—your game test and the rest, which are empty—have run successfully.
Note: You did not skip the empty test methods—testWorld() and testDeletion()—in this turn, which is why you see the test results as [INFO] Tests run: 3.
Adding Tests for the World Resource
Now that you know your Game resource works as expected and creates the related dependencies without a problem, you can add a test for the World resource.
Navigate to the testWorld() method in the test class and add the following content:
//Apply the YAML resources for the worlds "archaide", "incipio", "chthonia"
client.resources(World.class).inNamespace(NAMESPACE)
.load(getClass().getResource("/examples/archaide.yaml").getFile()).create();
client.resources(World.class).inNamespace(NAMESPACE)
.load(getClass().getResource("/examples/incipio.yaml").getFile()).create();
client.resources(World.class).inNamespace(NAMESPACE)
.load(getClass().getResource("/examples/chthonia.yaml").getFile()).create();
//Get the world instances "archaide", "incipio", "chthonia"
World worldArchaide = worldService.getWorld("archaide", NAMESPACE);
World worldIncipio = worldService.getWorld("incipio", NAMESPACE);
World worldChthonia = worldService.getWorld("chthonia", NAMESPACE);
//Assert the world instances checking they are not null
Assert.assertNotNull(worldArchaide);
Assert.assertNotNull(worldIncipio);
Assert.assertNotNull(worldChthonia);
//Assert the world instances expecting their status is "Ready"
//Wait for at most 60 seconds for the assertions.
await().atMost(60, TimeUnit.SECONDS).untilAsserted(() -> {
Assert.assertTrue(worldService.getWorld(worldArchaide.getMetadata().getName(), NAMESPACE).getStatus().isReady());
Assert.assertTrue(worldService.getWorld(worldIncipio.getMetadata().getName(), NAMESPACE).getStatus().isReady());
Assert.assertTrue(worldService.getWorld(worldChthonia.getMetadata().getName(), NAMESPACE).getStatus().isReady());
});
//Get the game from one of the worlds
Game game = gameService.getGame(worldArchaide.getSpec().getGame(), worldArchaide.getMetadata().getNamespace());
//Run a select query against the postgres instance for the World table
ResultSet resultSet = postgresService.executeQuery(
gameService.getPostgresServiceName(game) + ":" + GameService.POSTGRES_DB_PORT,
"postgres", game.getSpec().getDatabase().getUsername(), game.getSpec().getDatabase().getPassword(),
"SELECT * FROM World WHERE game=?",
game.getMetadata().getName());
//Iterate over the result set and assert if the records are in the database
int resultCount = 0;
while(resultSet.next()){
resultCount++;
}
Assert.assertEquals(3, resultCount);
This test adds three different world objects—archaide, incipio, and chthonia—for the game oasis in the Kubernetes instance and verifies their existence. Then it runs a query against the Postgres database, which the operator automatically creates for the oasis game.
The query selects the world table’s records from the database. Because three World resource objects have been created, it verifies whether there are three records too.
To see this test in action, run the test with the same test command as before: ./mvnw test
After a few minutes, the output should be as follows:
...output omitted... 2023-04-19 15:55:02,003 WARN [io.fab.kub.cli.dsl.int.VersionUsageUtils] (InformerWrapper [worlds.kubegame.systemcraftsman.com/v1alpha1] 142) The client is using resource type 'worlds' with unstable version 'v1alpha1' 2023-04-19 15:55:02,083 INFO [io.jav.ope.pro.Controller] (Controller Starter for: worldreconciler) 'worldreconciler' controller started ...output omitted... [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:40 min [INFO] Finished at: 2023-04-19T15:56:04+03:00 [INFO] ------------------------------------------------------------------------
Note: You may see java.net.SocketException: Socket closed in the test output. This error can be expected because there is a port forwarding mechanism that continues to run even after the operator stops.
Adding Tests for Resource Deletion
Now that you’ve successfully tested the operator and verified how it manages the game and the world objects when they are created, let’s see if the operator acts the way you expect it to when you delete the resources.
To test resource deletion, you will implement the testDeletion() method. Navigate to this method and add the following content in it:
//Delete the worlds "archaide", "incipio", "chthonia"
client.resources(World.class).inNamespace(NAMESPACE)
.load(getClass().getResource("/examples/archaide.yaml").getFile()).delete();
client.resources(World.class).inNamespace(NAMESPACE)
.load(getClass().getResource("/examples/incipio.yaml").getFile()).delete();
client.resources(World.class).inNamespace(NAMESPACE)
.load(getClass().getResource("/examples/chthonia.yaml").getFile()).delete();
//Assert the world instances are deleted
//Wait for at most 60 seconds for the assertions.
await().atMost(60, TimeUnit.SECONDS).untilAsserted(() -> {
Assert.assertNull(worldService.getWorld("archaide", NAMESPACE));
Assert.assertNull(worldService.getWorld("incipio", NAMESPACE));
Assert.assertNull(worldService.getWorld("chthonia", NAMESPACE));
});
//Delete the "oasis" game
client.resources(Game.class).inNamespace(NAMESPACE)
.load(getClass().getResource("/examples/oasis.yaml").getFile()).delete();
//Assert the game instance, its postgres instance and the service of it is deleted
//Wait for at most 60 seconds for the assertions.
await().atMost(60, TimeUnit.SECONDS).untilAsserted(() -> {
Assert.assertNull(gameService.getGame("oasis", NAMESPACE));
Assert.assertNull(client.apps().deployments().inNamespace(NAMESPACE).withName("oasis-postgres").get());
Assert.assertNull(client.services().inNamespace(NAMESPACE).withName("oasis-postgres").get());
});
The test method deletes the world instances first and verifies whether they are deleted. It then deletes the game instance `oasis` and verifies both the instance and its Postgres-related dependencies are deleted.
Now run all three tests in order with the same command as before: ./mvnw test
The test results should be successful:
[INFO] Results: [INFO] [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:17 min [INFO] Finished at: 2023-04-19T21:33:52+03:00 [INFO] ------------------------------------------------------------------------
And congratulations! You’ve spun up a K3s cluster by using Testcontainers and run all your tests on a real Kubernetes cluster. You now have a test lifecycle that tests how your operator creates custom resources, verifies their dependencies, and deletes the resource instances when the tests are done.
Conclusion
In this article, you’ve learned how you can use Testcontainers to test Kubernetes operators. You examined a Quarkus-based Kubernetes operator and configured it to use Testcontainers’ K3s module. You’ve also added some tests to confirm that the operator code works as expected and have gotten the test results successfully.
You can find the full solution for this tutorial in this repository’s `solution` branch.
Testcontainers-java is the easiest way to reliably test your Kubernetes Operators even if you’re writing your operators in Go or Quarkus and testing them in Java. Be sure to check out Testcontainers!
by Aykut Bulgu
Aykut Bulgu has 15+ years of experience working as a software engineer, consultant, trainer, and technical content architect. He has worked on many enterprise projects—mainly Java— and has expertise in JBoss middleware. He’s an expert on Apache Kafka, Apache Camel, Kubernetes/OpenShift, and Strimzi (Kafka on Kubernetes). He helps organizations with technical content development and training via System Craftsman.
