Bats Operations

From NovaOrdis Knowledge Base
Jump to navigation Jump to search

Internal

Best Practices

Script to Run all .bats Tests from a Directory or Multiple Directories

Single Directory

#!/usr/bin/env bash

for i in $(dirname $0)/*.bats; do
    echo "running $(basename ${i}) tests ..."
    bats $i
done

Multiple Directories

#!/usr/bin/env bash

cd ${dir}

for d in $(find . -type d -not -name '.'); do
    (
        d=${d#./}
        echo "running all tests from directory '${d}' ..."
        cd ${d};
        for i in *.bats; do
            echo "running $(basename ${i}) tests ..."
            bats ${i}
        done
    )
done

Exit after the First Test Failure

The above loops do not look if an individual bats command failed, and go through all tests reporting failures. This is variant that fails the overall loop if at least one test fails.

local test_failed=0
    ...
    bats ${i}
    if [ $? -ne 0 ]; then
        test_failed=1
    fi
...
if [ $test_failed -eq 1 ]; then
    exit 1
fi

Loading a Library to be Tested into a .bats Test

Assuming that the library you want to test is called 'blue.shlib', create a 'blue-library.bash' file in the directory that contain the .bats tests. If the library is located at a fixed relative location to the .bats test files, use this arrangement:

source ${BATS_TEST_DIRNAME}/[...]/blue.shlib

Otherwise, you can use an absolute path, but that is more fragile.

From each .bats test that wants to test functionality from that library:

load blue-library

@test "something" {
...
}

For more details, see:

Libraries being Tested and Helpers

Loading an Executable Script as Library to be Tested into a .bats Test

It is possible to test functions from an executable script by loading the executable script as library, even if the script loads external libraries itself, and invokes a main function. We use the same idea described above in the Loading a Library to be Tested into a .bats Test section, with a few additions. Let's assume there is a script called blue that is executable (contains a main() invocation), sources the bash.shlib library, and contains functions that we would like to unit test:

#!/usr/bin/env bash

#
# this is blue
#
[ -f $(dirname $0)/lib/bash.shlib ] && source $(dirname $0)/lib/bash.shlib || { echo "[error]: $(dirname $0)/lib/bash.shlib not found" 1>&2; exit 1; }

function main() {
...
}

function auxiliary() {
...
}

main "$@"

We would like to test auxiliary() by loading blue as a library.

For that, the following changes need to be introduced in blue:

1. Guard library loading with a BATS_TESTING variable:

...
[ -z "${BATS_TESTING}" ]  && { [ -f $(dirname $0)/lib/bash.shlib ] && source $(dirname $0)/lib/bash.shlib || { echo "[error]: $(dirname $0)/lib/bash.shlib not found" 1>&2; exit 1; } }
...


2. Guard main() invocation with the same BATS_TESTING variable:

...
if [ -z "${BATS_TESTING}" ]; then
    main "$@"
fi
...

Interestingly enough, this does not work, it breaks BATS:

...
[ -z "${BATS_TESTING}" ] && main "$@"
...


3. Create the all-required-libraries.bash library loader in the directory that contains the .bats tests. The library loader is similar to the one described in the section Loading a Library to be Tested into a .bats Test above. The difference is that it must set the BATS_TESTING to true, load the executable script as library, and also load all libraries required by the executable script:

export BATS_TESTING=true
source ${BATS_TEST_DIRNAME}/../../lib/bash.shlib
source ${BATS_TEST_DIRNAME}/../../blue

4. For each .bats test that wants to test functionality from the executable script, load "all-required-libraries", similarly to how is described in Loading a Library to be Tested into a .bats Test:

load all-required-libraries

@test "something" {
...
}

Handling stdout and stderr

function teardown() {

    rm -f ${BATS_TEST_DIRNAME}/tmp/stdout
    rm -f ${BATS_TEST_DIRNAME}/tmp/stderr
}

@test "stderr" {

    run $(fail "blah" 1>${BATS_TEST_DIRNAME}/tmp/stdout 2>${BATS_TEST_DIRNAME}/tmp/stderr )

    [[ -z $(cat ${BATS_TEST_DIRNAME}/tmp/stdout) ]]
    [[ $(cat ${BATS_TEST_DIRNAME}/tmp/stderr) = "[failure]: blah" ]]
}


This is a proposed solution, I need to do more research:

@test "test all outputl" {
  local stdoutPath="${BATS_TMPDIR}/${BATS_TEST_NAME}.stdout"
  local stderrPath="${BATS_TMPDIR}/${BATS_TEST_NAME}.stderr"

  myCommandOrFunction 1>${stdoutPath} 2>${stderrPath}

  grep "What Im looking for in stdout"  ${stdoutPath}
  ! grep "something it cannot appear in stdout"  ${stdoutPath}

  grep "What I'm looking for in stderr" ${stderrPath}
  ! grep "Something it cannot appear in stderr" ${stderrPath}
}

Handling data

A useful pattern is to create a "data" subdirectory in the directory that holds the .bats tests, and then access it from the tests with ${BATS_TEST_DIRNAME}/data/...

"Override" Functions

A useful technique when testing a layer is to "override" (declare) the functions called from that layer right before the test, so we can control input and output in the layer.

function setup() {

    function some-function() {
        echo "some-function($@)"
    }
}

...

@test "test invocation through layer to some-function()" {

    run caller-function A B C

    [[ ${status} -eq 0 ]]
    [[ "${output}" =~ some-function\(.*\) ]]

    args=${output#some-function(}
    args=${args%)}

    arg0=${args%% *}
    [[ ${arg0} = "A" ]]

    args=${args#${arg0} }
    arg1=${args%% *}
    [[ ${arg1} = "B" ]]

    ...
}

The function do not need to be "overridden" in setup(), they can be overridden in the test itself:

@test "test invocation through layer to some-function()" {

    function some-function-that-is-invoked-inside-caller-function() {
       exit 1
    }

    run caller-function A B C

    [[ ${status} -eq 1 ]]

    # ...
}

Moreover, the "overridden" function can test its arguments and cause the test to fail on invalid arguments:

@test "test invocation through layer to some-function()" {

    function some-function-that-is-invoked-inside-caller-function() {
       local arg1=$1
       [[ ${arg1} = "some expected value" ]] # this will fail the test if the arg1 value is not the one that is expected
    }

    run caller-function A B C

    [[ ${status} -eq 0 ]]

    # ...
}

Debugging Failures

BATS does not handle error logging gracefully. A useful technique is to turn verbosity in bash.shlib debug()/error()/warn()/info() functions and to configure them to send output into an alternate file, for problematic tests:

VERBOSE=true; DEBUG_OUTPUT=~/bats.out

Run the tests as follows:

rm ~/tmp/bats.out ; clear; bats ...test/some-test.bats; cat ~/tmp/bats.out