|
TH3: SQLite Test Harness #3
1.0 Overview
SQLite Test Harness #3 (hereafter "TH3") is one of three test harnesses used for testing SQLite. TH3 is designed to meet the following objectives:
TH3 will work on embedded platforms that lack the support infrastructure of workstations.
TH3 will test SQLite in an as-deployed configuration, without the need to enable special test hooks in the core SQLite library.
TH3 will test SQLite's response to out-of-memory errors, disk I/O errors, and power loss during transaction commit.
TH3 will test SQLite in a variety of configurations (UTF8 vs UTF16, different pages sizes, varying journal modes, etc.)
TH3 will achieve 100% branch test coverage.
TH3 is intended for validation testing. Though it could be used for testing during development and debugging, TH3 is not designed for that purpose. The original TCL-based test suite for SQLite will continue to serve as the primary platform for development and debug testing. The reason that TH3 exists is that it meets other objectives that are difficult to achieve with the TCL-based tests.
2.0 Operation
TH3 is a test program generator. The output of TH3 is a program written in ANSI-C and intended to be linked against the SQLite library under test. The generated test program is compiled and run on the target platform in order to verify correct operation of SQLite on that platform.
The inputs to TH3 are test modules written in C or SQL and small configuration files that determine how to initialize SQLite. The TH3 package includes hundreds of test modules and dozens of configuration files. New modules and configurations can be added to customize TH3 for specialized applications. Each time TH3 is run, it reads a subset of the available test modules and configuration files to generate a custom C program that performs all of the specified tests under all configurations. A complete test of SQLite normally involves running TH3 multiple times to generate multiple test programs covering different aspects of SQLite's operation, then linking all test programs against a common SQLite library and running them separately on the target platform. SQLite will be found to work if all test programs pass.
There are no arbitrary limits in TH3. One could generate a single test program that contained all test modules and configuration files. However, such a test program might be too large to deploy on embedded platforms. Hence, TH3 provides the ability to break the library of test modules up into smaller, more easily digested pieces.
Each individual test module might contain dozens, hundreds, or thousands of separate tests. The test modules can be written in C or as scripts of SQL. Test modules done in SQL are very easy to write and the test modules written in C are not as nearly as cumbersome to write as one might suppose. The TH3 system provides high-level interfaces that simplify test writing. A typical C-language test case in TH3 might contains slightly more syntax that the corresponding TCL-script test, but the difference is not that great. The test modules written as SQL contain special comments that define the boundaries and operation of each test case and the expected results.
Each test module file contains a header which describes the circumstances under which the test is valid. For a particular configuration, only those modules that are compatible with the configuration are run.
3.0 Generating A Test Program
The TH3 program generator is a TCL script named "mkth3.tcl". To generate a test program, one has merely to run this script and supply the names of files containing test modules and configurations on the command line. TH3 does not force any specific names for test modules or configuration files, but it is customary to use suffixes ".test" and ".cfg". With these conventions, one might generate a test program as follows:
tclsh mkth3.tcl *.test *.cfg >testprog1.c
The output from the mkth3.tcl script is a C program that contains everything needed to run the tests - everything that is except for the SQLite library itself. The generated test program contains implementations for all of the support interfaces used by the test modules and it contains the main() routine that drives the tests. To convert the test program into a working executable, simply compile it against SQLite:
cc -o testprog1 testprog1.c sqlite3.c
The compilation step shown immediately above is merely representative. In a working installation, one would normally want to specify optimization parameters and compile-time switches on the compiler command line.
Once the test program is generated, it is run with no arguments to perform the tests. Progress information as well as error diagnostics appear on standard output. The program returns zero if there are no errors and non-zero if any problems were detected.
4.0 Test Coverage
Using one particular subset of the available TH3 test modules (the "cov1" tests) SQLite obtained 100% branch test coverage and 100% MC/DC as measured by gcov on SuSE Linux 10.1 on x86 hardware on 2009-07-25. The SQLite developers are committed to maintaining 100% branch coverage and MC/DC for all future releases of SQLite.
The cov1 test set used to obtain 100% branch test coverage are only a subset of the tests currently implemented using TH3. New test modules are added on a regular basis.
5.0 TH3 License
SQLite itself is in the public domain and can be used for any purpose. But TH3 is proprietary and requires a license. Members of the SQLite Consortium get free and unlimited access to TH3. Others can contact the SQLite developers for information on how to obtain a license to access and use TH3.
Licensees of TH3 are given read access to the software configuration management system used to manage TH3 and so can download the latest version of TH3 (or any historical version) whenever they like.
Even though open-source users do not have direct access to TH3, all users of SQLite benefit from TH3 indirectly since each version of SQLite is validated by TH3 prior to release. So anyone using an official release of SQLite can deploy their application with the confidence of knowing that it has been tested using TH3. They simply cannot rerun those tests themselves without purchasing a TH3 license.
*** DRAFT ***