Bats Operations

From NovaOrdis Knowledge Base
Jump to navigation Jump to search

Internal

Best Practices

Script to Run all .bats Tests from a Directory or Multiple Directories

Single Directory

#!/usr/bin/env bash

for i in $(dirname $0)/*.bats; do
    echo "running $(basename ${i}) tests ..."
    bats $i
done

Multiple Directories

#!/usr/bin/env bash

cd ${dir}

for d in $(find . -type d -not -name '.'); do
    (
        d=${d#./}
        echo "running all tests from directory '${d}' ..."
        cd ${d};
        for i in *.bats; do
            echo "running $(basename ${i}) tests ..."
            bats ${i}
        done
    )
done

Exit after the First Test Failure

The above loops do not look if an individual bats command failed, and go through all tests reporting failures. This is variant that fails the overall loop if at least one test fails. 

local test_failed=0
    ...
    bats ${i}
    if [ $? -ne 0 ]; then
        test_failed=1
    fi
...
if [ $test_failed -eq 1 ]; then
    exit 1
fi

Recommended Layout

 .
 └─ src
     ├─ main
     │   └─ bash
     │       └─lib
     │          ├─ mylib1.shlib
     │          └─ mylib2.shlib
     └─ test
         └─ bash
             │ 
             ├─ mylib1
             │    ├─ libraries.bash
             │    ├─ data
             │    │   └─ .gitkeep
             │    ├─ myfunction1.bats
             │    └─ myfunction2.bats
             │ 
             └─ mylib2
                  ├─ libraries.bash 
                  ├─ data
                  │   └─ .gitkeep
                  ├─ myfunction3.bats
                  └─ myfunction4.bats

  • Each library to be tested has its own directory.
  • Each library-specific directory contains a libraries.bash that lists dependencies, including the library to be tested. For more details on how to compose that file, see Loading a Library to be Tested into a .bats Test below.
  • Each function to be tested has its own .bats file, where various scenarios to test the function are implemented as individual @test functions.
  • The library's directory has a data subdirectory in which a data/tmp will be created dynamically as needed.

Loading a Library to be Tested into a .bats Test

Assuming that the library you want to test is called blue.shlib, create a blue-library.bash file in the directory that contain the .bats tests for the library. If the library is located at a fixed relative location to the .bats test files, use this arrangement: 

source "${BATS_TEST_DIRNAME}/[...]/blue.shlib"

Otherwise, you can use an absolute path, but that is more fragile.

From each .bats test that wants to test functionality from that library: 

load blue-library

@test "something" {
...
}

If more than one library needs to be loaded, create a libraries.bash file that sources all libraries, including the one that needs to be tested: 

source "${BATS_TEST_DIRNAME}/[...]/lib1.shlib"
source "${BATS_TEST_DIRNAME}/[...]/lib2.shlib"
source "${BATS_TEST_DIRNAME}/[...]/lib3.shlib"
source "${BATS_TEST_DIRNAME}/[...]/blue.shlib"

and from each BATS test, load it with: 

load libraries
@test "something" {
...
}

For more details, see:

Libraries being Tested and Helpers

Loading an Executable Script as Library to be Tested into a .bats Test

It is possible to test functions from an executable script by loading the executable script as library, even if the script loads external libraries itself, and invokes a main function. We use the same idea described above in the Loading a Library to be Tested into a .bats Test section, with a few additions. Let's assume there is a script called blue that is executable (contains a main() invocation), sources the bash.shlib library, and contains functions that we would like to unit test: 

#!/usr/bin/env bash

#
# this is blue
#
[ -f $(dirname $0)/lib/bash.shlib ] && source $(dirname $0)/lib/bash.shlib || { echo "[error]: $(dirname $0)/lib/bash.shlib not found" 1>&2; exit 1; }

function main() {
...
}

function auxiliary() {
...
}

main "$@"

We would like to test auxiliary() by loading blue as a library.

For that, the following changes need to be introduced in blue:

1. Guard library loading with a BATS_TESTING variable: 

...
[ -z "${BATS_TESTING}" ]  && { [ -f $(dirname $0)/lib/bash.shlib ] && source $(dirname $0)/lib/bash.shlib || { echo "[error]: $(dirname $0)/lib/bash.shlib not found" 1>&2; exit 1; } }
...


2. Guard main() invocation with the same BATS_TESTING variable: 

...
if [ -z "${BATS_TESTING}" ]; then
    main "$@"
fi
...

Interestingly enough, this does not work, it breaks BATS: 

...
[ -z "${BATS_TESTING}" ] && main "$@"
...


3. Create the all-required-libraries.bash library loader in the directory that contains the .bats tests. The library loader is similar to the one described in the section Loading a Library to be Tested into a .bats Test above. The difference is that it must set the BATS_TESTING to true, load the executable script as library, and also load all libraries required by the executable script: 

export BATS_TESTING=true
source ${BATS_TEST_DIRNAME}/../../lib/bash.shlib
source ${BATS_TEST_DIRNAME}/../../blue

4. For each .bats test that wants to test functionality from the executable script, load "all-required-libraries", similarly to how is described in Loading a Library to be Tested into a .bats Test: 

load all-required-libraries

@test "something" {
...
}

Handling stdout and stderr

function teardown() {

    rm -f ${BATS_TEST_DIRNAME}/tmp/stdout
    rm -f ${BATS_TEST_DIRNAME}/tmp/stderr
}

@test "stderr" {

    run $(fail "blah" 1>${BATS_TEST_DIRNAME}/tmp/stdout 2>${BATS_TEST_DIRNAME}/tmp/stderr )

    [[ -z $(cat ${BATS_TEST_DIRNAME}/tmp/stdout) ]]
    [[ $(cat ${BATS_TEST_DIRNAME}/tmp/stderr) = "[failure]: blah" ]]
}


This is a proposed solution, I need to do more research: 

@test "test all outputl" {
  local stdoutPath="${BATS_TMPDIR}/${BATS_TEST_NAME}.stdout"
  local stderrPath="${BATS_TMPDIR}/${BATS_TEST_NAME}.stderr"

  myCommandOrFunction 1>${stdoutPath} 2>${stderrPath}

  grep "What Im looking for in stdout"  ${stdoutPath}
  ! grep "something it cannot appear in stdout"  ${stdoutPath}

  grep "What I'm looking for in stderr" ${stderrPath}
  ! grep "Something it cannot appear in stderr" ${stderrPath}
}

Handling data

A useful pattern is to create a "data" subdirectory in the directory that holds the .bats tests, and then access it from the tests with ${BATS_TEST_DIRNAME}/data/...

"Override" Functions

A useful technique when testing a layer is to "override" (declare) the functions called from that layer right before the test, so we can control input and output in the layer. 

function setup() {

    function some-function() {
        echo "some-function($@)"
    }
}

...

@test "test invocation through layer to some-function()" {

    run caller-function A B C

    [[ ${status} -eq 0 ]]
    [[ "${output}" =~ some-function\(.*\) ]]

    args=${output#some-function(}
    args=${args%)}

    arg0=${args%% *}
    [[ ${arg0} = "A" ]]

    args=${args#${arg0} }
    arg1=${args%% *}
    [[ ${arg1} = "B" ]]

    ...
}

The function does not need to be "overridden" in setup(), they can be overridden in the test itself: 

@test "test invocation through layer to some-function()" {

    function some-function-that-is-invoked-inside-caller-function() {
       exit 1
    }

    run caller-function A B C

    [[ ${status} -eq 1 ]]

    # ...
}

A default, innocuous override can be specified in setup() - so the majority of the tests that rely on the correct behavior get that - while tests that are involving with specific error conditions can "override" the function locally and test a specific error condition.

Moreover, the "overridden" function can test its arguments and cause the test to fail on invalid arguments: 

@test "test invocation through layer to some-function()" {

    function some-function-that-is-invoked-inside-caller-function() {
       local arg1=$1
       [[ ${arg1} = "some expected value" ]] # this will fail the test if the arg1 value is not the one that is expected
    }

    run caller-function A B C

    [[ ${status} -eq 0 ]]

    # ...
}

If tests specified in more than one .bats files need to share the same function behavior, the function can be declared in its own .shlib file in the same directory as the bats files, and then it can be sourced in the setup() or individual tests:

mock-curl.shlib: 

function curl() {
 ...
}

In the test's setup() function: 

function() setup() {
   ...
   source ${BATS_TEST_DIRNAME}/mock-curl.shlib
}

Testing Code that Reads stdin

A tested pattern that works when stdin is pulled from with read is to "override" read as such: 

@test "some test" {

    # bash read "override"
    function read() {

        debug "read($@)"
        local variable_name=$3
        eval "${variable_name}=\"black\""
    }

    run some_function

    ...
}

This behaves as "black" was provided at stdin.

Data and Temporary Directories

function setup() {
  [[ -d ${BATS_TEST_DIRNAME}/data/tmp ]] || mkdir "${BATS_TEST_DIRNAME}"/data/tmp
}

function teardown() {
  rm -rf "${BATS_TEST_DIRNAME}"/data/tmp/*
}

Debugging Failures

BATS does not handle error logging gracefully. A useful technique is to turn verbosity in bash.shlib debug()/error()/warn()/info() functions and to configure them to send output into an alternate file, for problematic tests: 

VERBOSE=true; DEBUG_OUTPUT=~/bats.out

Run the tests as follows: 

rm ~/tmp/bats.out ; clear; bats ...test/some-test.bats; cat ~/tmp/bats.out
Retrieved from "https://kb.novaordis.com/index.php?title=Bats_Operations&oldid=89801"