Testing BitSet
P.J. Drongowski
18 September 2004 (Updated 28 January 2005)

Doing a conditional build for testing

  * Surround the function main and any auxilliary functions and testing
    data tables with:
        #ifdef TESTING
          ...
        #endif
  * The source module can then be compiled in two ways: testing and production
    + If the symbol TESTING is defined, the testing code is compiled and
      will be linked in; This is good for unit (module level) testing
    + If the symbol TESTING is not defined, the testing code is not compiled;
      The resulting object module can be linked into the production system
  * There are two ways to define TESTING:
    + Add a #define to the top of the module; This requires changing the
      source module.
        #define TESTING 1
    + Add a "-DTESTING" option to the compile command:
        g++ -o bitset -DTESTING bitset.cpp
  * This scheme keeps the unit test driver (main) with the module
    implementation (may not be workable in large system build)


General considerations

  * You almost always will need to postprocess test output
    + Be sure to generate output that is easy to parse ("read and dissect")
    + Generate standard keywords to easily determine test outcome
      - PASS TEST #57
      - FAIL TEST #89
      - ATTEMPTED TEST #23
    + It may be necessary to generate a test summary:
         34 PASS / 3 FAIL out of 37 ATTEMPTED
    + Try to identify the specific cause of the failure in the output -- you
      most certainly need to find and fix the cause of the failure
  * Generate test results to a "log" file instead of standard output (cout)
    or standard error (cerr)
    + Keep program output separate from test output
    + Allows separate postprocessing of test and program output


Testing workflow

          Test data ("test vector")
                     |
                     V
           System under test (SUT)
                     |
                     V
             Actual test results        Known good (KG) results
                     |                             |
                     V                             |
                  Compare <------------------------
                     |
                     V
       Same?  ---------------  Different?
             |               |
             V               V
           PASS             FAIL


Writing test code

  * Need test data to be applied to the system under test ("the module")
  * Need known good (KG) results
    + Can be actual, hand-checked, golden output
    + Can be hand-crafted expected results
  * Need a test driver (main) to apply test data, acquire actual results
    and compare with actual results
  * Main drawback: You cannot acquire and compare program output, i.e.,
    stuff written to standard output or standard error streams


Test scripts

  * Test scripts can be written to:
    + Apply data to the program to be tested (data may be in files)
    + Acquire output from the program
    + Compare actual results with known good results (in files)
    + Write PASS/FAIL to testing log
  * Advantage can collect and test program output
  * You now have even more artifacts (source files, scripts, test data
    files, known good result files, etc.) to manage
  * Test scripts are often written in PERL, the everything, but the
    kitchen sink scripting language
  * Can accommodate tests that induce error conditions forcing program exit


Testing is hard due to "state space explosion"

  * BitSet has only two data members (variables): currentMembers, setSize
  * Consider the problem of testing UNIX processes
  * UNIX data structure has 63+ data members, some of which are
    complex data structures themselves!