Regression Testing

Running Tests

If you are making changes to µCsim you should both run tests to ensure nothing breaks and write new tests to ensure your changes work and do not get broken in the future!

In all cases the tests are run using the locally built simulators. i.e. you need to have completed a successful build before you can test it! Tests produce output when run which is captured to a file and compared with baseline output held in the source tree. If there are any differences they are output and the test is considered failed. In that case either the code needs to be changed to correct the output or the baseline needs to be updated to reflect the new expected output. Only when there are no differences to the baseline can tests be said to have passed.

All Tests

To run all tests run make test in the top-level ucsim directory.
E.g.: 

.../ucsim$ make test

For a Single MCU

To run tests for a single MCU type run make test in the directory for that simulator.
E.g.: 

.../ucsim/s51.src$ make test
or run make in the test subdirectory.
E.g.: 
.../ucsim/s51.src/test$ make

Note that tests for functionality generic to all simulators and that does not depend on a particular MCU type lives in sim.src/test and should be run there. These tests require at least one simulator to have been built but unless there is only one it is indeterminate which simulator will be used.

Running a Single Test

To run a single test run make test_name in the test subdirectory of the relevant simulator. E.g.: 

.../ucsim/s51.src/test$ make config
In this case the test is run verbosely and the commands executed by make are displayed. In all other cases only the final differences to baseline (if any) are displayed.

Updating the Baselines

If changes to the simulator code base lead to changes in test outputs and these changes are as expected then the baselines should be updated to match the new output by copying the contents of the out directories to the corresponding baseline directories.

To update all baselines run make baseline in the top-level ucsim directory.

To update baselines for a single MCU type run make baseline in the directory for that simulator.

If you need to update the baseline for a single test you should copy the relevant files manually.

Remember that the updated baselines form part of your changes and need to be committed or submitted as part of your patch!

Writing Tests

  1. Create a directory named after your test under the relevant simulator's test directory:
  2. Place the data needed for your test, .cmd files, .asm or C files etc, in your test directory:
  3. Create a baseline directory in your test directory:
  4. Create an empty file in the baseline directory for each output that your test generates:
  5. Create a Makefile that lists the output(s) to be generated, the recipes to create them and which includes test-lib.mk:
  6. Run your test, verify the contents of the out directory and then copy it to the baseline directory: 
.../ucsim/s51.src/test$ mkdir name
.../ucsim/s51.src/test$ cd name
.../ucsim/s51.src/test/name$ vim test.{cmd,asm}
.../ucsim/s51.src/test/name$ mkdir baseline
.../ucsim/s51.src/test/name$ touch baseline/stdout
.../ucsim/s51.src/test/name$ cat > Makefile <<EOF
OUTPUTS = stdout

stdout: mytest.ihx mytest.cmd
        $(call run-sim)

include test-lib.mk
EOF
.../ucsim/s51.src/test/name$ cd ..
.../ucsim/s51.src/test$ make name
.../ucsim/s51.src/test$ cp -r name/out name/baseline/.

test-lib.mk takes care of building any .ihx files needed from corresponding .asm or .c files. The call to run-sim will invoke the simulator with the .ihx and .cmd prerequisites, capturing the output.

If there are no .cmd prerequisites and there is no -e in the arguments to run-sim it will add a -g argument to start the simulator running. It is assumed there is at least one .ihx file in the prerequisites in this case and that it places code at the reset address.

If you wish to pass arguments of your own to the simulator these can be given as arguments to run-sim. This can be used to pass commands with the -e option rather than using a .cmd file for instance, as is done with the config tests.

If you create any other output from within the simulator you should specify a pathname starting with out/. Anything written to the out directory will be compared to the baseline when the test is complete.