Infrastructure Code Testing Concepts

Internal

• Infrastructure as Code Concepts
• Continuous Delivery
• Software Testing Concepts

Overview

A practice recommended when developing infrastructure as code is to continuously test and deliver, same as for application software development.

Continuously testing small pieces encourages a modular, loosely coupled design and it helps finding problems sooner, then quickly iterating, fixing and rebuilding the problematic code. This process yields better infrastructure. The fact that test suite such developed remains with the code base and is continuously exercises as part of CD runs is referred to as "building quality in" rather than "testing quality in". Finding and fixing problems continuously avoids the accumulation of technical debt.

Even if that the word "infrastructure" may suggest that it is built once, and then forgotten, this is far from the truth. Infrastructure needs contestant change: patching, upgrading, fixing and improving. Every time the infrastructure is modified, automated tests decrease the likeliness that something will break. This is why building the delivery and testing systems within the primary system is a good idea. If that is done properly, "going live" is almost an arbitrary event, a change in who is using the system, but not how the system is managed.

The Infrastructure Test Diamond

The test pyramid is a good model for application software testing, but does not apply very well to infrastructure code testing. Low level offline unit tests are not very valuable for declarative code so we don't need so many of them. The testing model for infrastructure code looks like a diamond:

The pyramid may make sense for imperative infrastructure code.

Challenges with Testing Infrastructure Code

Declarative Code Tests Often Have Low Value

Many infrastructure tools use declarative languages, which express desired state. Testing that all details of the desired state have been changed correctly can become soon very tedious, and in fact represents testing of the infrastructure tool. One valid testing scenario is to ensure that the change has been in fact applied, but for that, the test of a single detail of the end state should be sufficient. In the context of "Given, When, Then" tests, "When" can be missing for declarative code tests, which suggests that the code does not create variable outcomes. Many tools and practices for testing dynamic code are not appropriate for declarative code.

When variables or conditionals are used with declarative code, it makes sense to test the code with more complex tests (there is a "When" now). However, if the declarative code is complex enough that it needs complex testing, that is a sign that the logic should be pulled out of the declarative section and consolidated into a library written into a procedural language, and tested independently.

Another useful tests for declarative code is to ensure the complex infrastructure created or modified by a complex piece of declarative code works as intended, as opposite to checking its state and ensuring the desired state has been transferred correctly to the infrastructure resources.

In any case, the declarative code tests are not useful without the actual infrastructure, they are not meaningful when run with test doubles.

Infrastructure Tests are Slow

Speeding up the test execution involves a combination of strategies. Some of these strategies are not particular to infrastructure testing, they apply to software testing in general:

• Divide the system into small, tractable components. Small, loosely coupled infrastructure components are usually faster and cheaper to provision and test.
• Use test doubles.
• Use progressive testing.

The following strategies are specific to infrastructure testing:

Decide between Ephemeral or Persistent Instances

An infrastructure resource instance may be created and then destroyed every time it is used (an ephemeral instance) or it can be left running between tests and reused (persistent instance). Persistent instances can make the test significantly faster, but it could make the tests inconsistent, as its state can be changed in unpredictable ways by the previous tests. On the other hand, ephemeral instances may slow down the tests, as they need to be created every time, but are cleaner and give more consistent results. The right choice depends on the particular risks involved on a case-to-case basis.

Decide between Online and Offline Tests

Some types of tests must run online, requiring infrastructure on the real cloud platform. Others can run offline, on the build agent, without need for connection to the cloud platform. Offline tests are usually much faster, but they have limited use when it comes to infrastructure software. They are limited to syntax checking, static security analysis or tests that run inside a container instance. An online test can be turned into an offline test it it is possible to use a test double.

Dependencies Complicate Testing Infrastructure

Infrastructure code is particular in that it needs to infrastructure platform and its APIs to work, so the infrastructure platform (or subsets of its APIs) are a required dependency. This may be worked around by using test doubles. Also, there is a growing number of tools that allow mocking the API of the cloud vendors. However, it's more useful to use test doubles for other infrastructure components that for the infrastructure platform itself.

Cloud Mocking Tools

• LocalStack
• moto
• Azurite