Bats Operations

From NovaOrdis Knowledge Base
Jump to navigation Jump to search

Internal

Best Practices

Script to Recursively Run all .bats Tests in Multiple Directories

The following code will run all tests, even some of the tests fail. At the end of the full run, the failed_files variable will contain the list of files for which there were test failures. If the variable is empty, it means all tests passed. 

function do-test() {
  bats --version >/dev/null 2>/dev/null || fail "bats not installed, cannot run tests"
  local bats_tests_root_dir failed_test_files
  bats_tests_root_dir="$(dirname $0)/src/test/bash"
  cd "${bats_tests_root_dir}" || fail "failed to cd to ${bats_tests_root_dir}"
  # shellcheck disable=SC2044
  for d in $(find . -type d -not -name '.'); do
    d=${d#./}
    pushd ${d} >/dev/null || fail "failed to cd to ${d}"
    if ls -- *.bats >/dev/null 2>/dev/null; then
      # .bats files exist in the directory
      echo "running all BATS tests from ${bats_tests_root_dir}/${d} ..."
      for i in *.bats; do
        echo "running $(basename ${i}) tests ..."
        bats ${i} || failed_test_files="${failed_test_files} ${bats_tests_root_dir}/${d}/${i}"
      done
    fi
    popd >/dev/null || fail "failed to popd"
  done
  [[ -n ${failed_test_files} ]] && fail "test failures in${failed_test_files}"
}

Exit after the First Test Failure

The above loops do not look if an individual bats command failed, and go through all tests reporting failures. This is variant that fails the overall loop if at least one test fails. 

local test_failed=0
    ...
    bats ${i}
    if [ $? -ne 0 ]; then
        test_failed=1
    fi
...
if [ $test_failed -eq 1 ]; then
    exit 1
fi

Recommended Layout

 .
 └─ src
     ├─ main
     │   └─ bash
     │       └─lib
     │          ├─ mylib1.shlib
     │          └─ mylib2.shlib
     └─ test
         └─ bash
             │ 
             ├─ mylib1
             │    ├─ libraries.bash
             │    ├─ data
             │    │   └─ .gitkeep
             │    ├─ myfunction1.bats
             │    └─ myfunction2.bats
             │ 
             └─ mylib2
                  ├─ libraries.bash 
                  ├─ data
                  │   └─ .gitkeep
                  ├─ myfunction3.bats
                  └─ myfunction4.bats

  • Each library to be tested has its own directory.
  • Each library-specific directory contains a libraries.bash that lists dependencies, including the library to be tested. For more details on how to compose that file, see Loading a Library to be Tested into a .bats Test below.
  • Each function to be tested has its own .bats file, where various scenarios to test the function are implemented as individual @test functions.
  • The library's directory has a data subdirectory in which a data/tmp will be created dynamically as needed. For more details see Handling data section below.

Loading a Library to be Tested into a .bats Test

Assuming that the library you want to test is called blue.shlib, create a blue-library.bash file in the directory that contain the .bats tests for the library. If the library is located at a fixed relative location to the .bats test files, use this arrangement: 

source "${BATS_TEST_DIRNAME}/[...]/blue.shlib"

Otherwise, you can use an absolute path, but that is more fragile.

From each .bats test that wants to test functionality from that library: 

load blue-library 

@test "something" {
...
}

If more than one library needs to be loaded, create a libraries.bash file that sources all libraries, including the one that needs to be tested: 

source "${BATS_TEST_DIRNAME}/[...]/lib1.shlib"
source "${BATS_TEST_DIRNAME}/[...]/lib2.shlib"
source "${BATS_TEST_DIRNAME}/[...]/lib3.shlib"
source "${BATS_TEST_DIRNAME}/[...]/blue.shlib"

and from each BATS test, load it with: 

load libraries
@test "something" {
...
}

For more details, see:

Libraries being Tested and Helpers

Loading an Executable Script as Library to be Tested into a .bats Test

It is possible to test functions from an executable script by loading the executable script as library, even if the script loads external libraries itself, and invokes a main function. We use the same idea described above in the Loading a Library to be Tested into a .bats Test section, with a few additions. Let's assume there is a script called blue that is executable (contains a main() invocation), sources the bash.shlib library, and contains functions that we would like to unit test: 

#!/usr/bin/env bash

#
# this is blue
#
[ -f $(dirname $0)/lib/bash.shlib ] && source $(dirname $0)/lib/bash.shlib || { echo "[error]: $(dirname $0)/lib/bash.shlib not found" 1>&2; exit 1; }

function main() {
...
}

function auxiliary() {
...
}

main "$@"

We would like to test auxiliary() by loading blue as a library.

For that, the following changes need to be introduced in blue:

1. Guard library loading with a BATS_TESTING variable: 

...
[ -z "${BATS_TESTING}" ]  && { [ -f $(dirname $0)/lib/bash.shlib ] && source $(dirname $0)/lib/bash.shlib || { echo "[error]: $(dirname $0)/lib/bash.shlib not found" 1>&2; exit 1; } }
...


2. Guard main() invocation with the same BATS_TESTING variable: 

...
if [ -z "${BATS_TESTING}" ]; then
    main "$@"
fi
...

Interestingly enough, this does not work, it breaks BATS: 

...
[ -z "${BATS_TESTING}" ] && main "$@"
...

3. Create the libraries.bash library loader in the directory that contains the .bats tests. The library loader is similar to the one described in the section Loading a Library to be Tested into a .bats Test above. The difference is that it must set the BATS_TESTING to true, load the executable script as library, and also load all libraries required by the executable script: 

export BATS_TESTING=true
source ${BATS_TEST_DIRNAME}/../../lib/bash.shlib
source ${BATS_TEST_DIRNAME}/../../blue

4. For each .bats test that wants to test functionality from the executable script, load libraries, similarly to how is described in Loading a Library to be Tested into a .bats Test: 

load libraries

@test "something" {
...
}

Handling stdout and stderr

function teardown() {

    rm -f ${BATS_TEST_DIRNAME}/tmp/stdout
    rm -f ${BATS_TEST_DIRNAME}/tmp/stderr
}

@test "stderr" {

    run $(fail "blah" 1>${BATS_TEST_DIRNAME}/tmp/stdout 2>${BATS_TEST_DIRNAME}/tmp/stderr )

    [[ -z $(cat ${BATS_TEST_DIRNAME}/tmp/stdout) ]]
    [[ $(cat ${BATS_TEST_DIRNAME}/tmp/stderr) = "[failure]: blah" ]]
}


This is a proposed solution, I need to do more research: 

@test "test all outputl" {
  local stdoutPath="${BATS_TMPDIR}/${BATS_TEST_NAME}.stdout"
  local stderrPath="${BATS_TMPDIR}/${BATS_TEST_NAME}.stderr"

  myCommandOrFunction 1>${stdoutPath} 2>${stderrPath}

  grep "What Im looking for in stdout"  ${stdoutPath}
  ! grep "something it cannot appear in stdout"  ${stdoutPath}

  grep "What I'm looking for in stderr" ${stderrPath}
  ! grep "Something it cannot appear in stderr" ${stderrPath}
}

Handling stdin (Testing Code that Reads stdin)

A tested pattern that works when stdin is pulled from with read is to "override" read as such: 

@test "some test" {

    # bash read "override"
    function read() {

        debug "read($@)"
        local variable_name=$3 # this is the index of the argument that specifies 
                               # the variable name. It is valid for "read -p 'something' some_var".
        eval "${variable_name}=\"black\""
    }

    run some_function

    ...
}

This behaves as "black" was provided at stdin.

Handling data

A useful pattern is to create a data subdirectory in the directory that holds the .bats tests, and then access it from the tests with ${BATS_TEST_DIRNAME}/data/....

Data and Temporary Directories

function setup() {
  [[ -d "${BATS_TEST_DIRNAME}/data/tmp" ]] || mkdir -p "${BATS_TEST_DIRNAME}/data/tmp"
}

function teardown() {
  rm -rf "${BATS_TEST_DIRNAME}/data/tmp"
}

"Override" Functions

A useful technique when testing a layer is to "override" (declare) the functions called from that layer right before the test, so we can control input and output in the layer. 

function setup() {

    function some-function() {
        echo "some-function($@)"
    }
}

...

@test "test invocation through layer to some-function()" {

    run caller-function A B C

    [[ ${status} -eq 0 ]]
    [[ "${output}" =~ some-function\(.*\) ]]

    args=${output#some-function(}
    args=${args%)}

    arg0=${args%% *}
    [[ ${arg0} = "A" ]]

    args=${args#${arg0} }
    arg1=${args%% *}
    [[ ${arg1} = "B" ]]

    ...
}

The function does not need to be "overridden" in setup(), they can be overridden in the test itself: 

@test "test invocation through layer to some-function()" {

    function some-function-that-is-invoked-inside-caller-function() {
       exit 1
    }

    run caller-function A B C

    [[ ${status} -eq 1 ]]

    # ...
}

A default, innocuous override can be specified in setup() - so the majority of the tests that rely on the correct behavior get that - while tests that are involving with specific error conditions can "override" the function locally and test a specific error condition.

Moreover, the "overridden" function can test its arguments and cause the test to fail on invalid arguments: 

@test "test invocation through layer to some-function()" {

    function some-function-that-is-invoked-inside-caller-function() {
       local arg1=$1
       [[ ${arg1} = "some expected value" ]] # this will fail the test if the arg1 value is not the one that is expected
    }

    run caller-function A B C

    [[ ${status} -eq 0 ]]

    # ...
}

If tests specified in more than one .bats files need to share the same function behavior, the function can be declared in its own .shlib file in the same directory as the bats files, and then it can be sourced in the setup() or individual tests:

mock-curl.shlib: 

function curl() {
 ...
}

In the test's setup() function: 

function() setup() {
   ...
   source ${BATS_TEST_DIRNAME}/mock-curl.shlib
}

Debugging Function Invocations

function setup() {
    [[ -d ${BATS_TEST_DIRNAME}/data/tmp ]] || mkdir "${BATS_TEST_DIRNAME}"/data/tmp
    INVOCATION_FILE=${BATS_TEST_DIRNAME}/data/tmp/invocations.txt
    function some-function() {
      echo "$@" >> "${INVOCATION_FILE}"
      ...
    }
}

function teardown() {
  rm -rf "${BATS_TEST_DIRNAME}"/data/tmp
}

@test "some test" {
  run ...
  ...
  # shellcheck disable=SC2002
  invocation_count=$(cat "${INVOCATION_FILE}" | wc -l | awk '{print $1}')
  [[ ${invocation_count} == 2 ]]
  first_invocation=$(head -1 "${INVOCATION_FILE}")
  [[ ${first_invocation} == "something" ]]
  second_invocation=$(head -2 "${INVOCATION_FILE}" | tail -1)
  [[ ${second_invocation} == "something else" ]]
}

Running Individual Tests

Individual tests from a .bats test file can be executed by using the -f, --filter option, with a regular expression that matches the test(s) names. 

bats ./some-file.bats -f "blue test"

An empty filter acts as no filter, all tests will be executed: 

bats ./some-file.bats -f ""

Debugging Failures

BATS does not handle error logging gracefully. A useful technique is to turn verbosity in bash.shlib debug()/error()/warn()/info() functions and to configure them to send output into an alternate file, for problematic tests: 

VERBOSE=true; DEBUG_OUTPUT=~/bats.out

Run the tests as follows: 

rm ~/tmp/bats.out ; clear; bats ...test/some-test.bats; cat ~/tmp/bats.out
Retrieved from "https://kb.novaordis.com/index.php?title=Bats_Operations&oldid=97349"