We're in the process of converting the existing testsuite machinery to use the new style DejaGnu framework. Eventually, we'll abandon ../mkcheck.in in favor of this new testsuite framework. // 1: Thoughts on naming test cases, and structuring them. The testsuite directory has been divided into 11 directories, directly correlated to the relevant chapters in the standard. For example, the directory testsuite/21_strings contains tests related to "Chapter 21, Strings library" in the C++ standard. So, the first step in making a new test case is to choose the correct directory. The second item is seeing if a test file exists that tests the item in question. Generally, within chapters test files are named after the section headings in ISO 14882, the C++ standard. For instance, 21.3.7.9 Inserters and Extractors Has a related test case: 21_strings/inserters_extractors.cc Not so hard. Some time, the words "ctor" and "dtor" are used instead of "construct", "constructor", "cons", "destructor", etc. Other than that, the naming seems mostly consistent. If the file exists, add a test to it. If it does not, then create a new file. All files are copyright the FSF, and GPL'd: this is very important. Inside a test file, the plan is to test the relevant parts of the standard, and then add specific regressions as additional test functions, ie test04() can represent a specific regression noted in GNATS. Once test files get unwieldy or too big, then they should be broken up into multiple sub-categories, hopefully intelligently named after the relevant (and more specific) part of the standard. // 2: How to write a test case, from a dejagnu perspective As per the dejagnu instructions, always return 0 from main to indicate success. Basically, a test case contains dg-keywords (see dg.exp) indicating what to do and what kinds of behaviour are to be expected. New testcases should be written with the new style DejaGnu framework in mind. To ease transition, here is the list of dg-keyword documentation lifted from dg.exp -- eventually we should improve DejaGnu documentation, but getting checkin account currently demands Pyrrhic effort. # The currently supported options are: # # dg-prms-id N # set prms_id to N # # dg-options "options ..." [{ target selector }] # specify special options to pass to the tool (eg: compiler) # # dg-do do-what-keyword [{ target/xfail selector }] # `do-what-keyword' is tool specific and is passed unchanged to # ${tool}-dg-test. An example is gcc where `keyword' can be any of: # preprocess|compile|assemble|link|run # and will do one of: produce a .i, produce a .s, produce a .o, # produce an a.out, or produce an a.out and run it (the default is # compile). # # dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]] # indicate an error message is expected on this line # (the test fails if it doesn't occur) # Linenum=0 for general tool messages (eg: -V arg missing). # "." means the current line. # # dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]] # indicate a warning message is expected on this line # (the test fails if it doesn't occur) # # dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]] # indicate a bogus error message use to occur here # (the test fails if it does occur) # # dg-build regexp comment [{ target/xfail selector }] # indicate the build use to fail for some reason # (errors covered here include bad assembler generated, tool crashes, # and link failures) # (the test fails if it does occur) # # dg-excess-errors comment [{ target/xfail selector }] # indicate excess errors are expected (any line) # (this should only be used sparingly and temporarily) # # dg-output regexp [{ target selector }] # indicate the expected output of the program is # (there may be multiple occurrences of this, they are concatenated) # # dg-final { tcl code } # add some tcl code to be run at the end # (there may be multiple occurrences of this, they are concatenated) # (unbalanced braces must be \-escaped) # # "{ target selector }" is a list of expressions that determine whether the # test succeeds or fails for a particular target, or in some cases whether the # option applies for a particular target. If the case of `dg-do' it specifies # whether the testcase is even attempted on the specified target. # # The target selector is always optional. The format is one of: # # { xfail *-*-* ... } - the test is expected to fail for the given targets # { target *-*-* ... } - the option only applies to the given targets # # At least one target must be specified, use *-*-* for "all targets". # At present it is not possible to specify both `xfail' and `target'. # "native" may be used in place of "*-*-*". Example 1: Testing compilation only (to just have a testcase do compile testing, without linking and executing) // { dg-do compile } Example 2: Testing for expected warings on line 36 // { dg-warning "string literals" "" { xfail *-*-* } 36 Example 3: Testing for compilation errors on line 41 // { dg-do compile } // { dg-error "no match for" "" { xfail *-*-* } 41 } More examples can be found in the libstdc++-v3/testsuite/*/*.cc files. // 3: Test harness notes, invocation, and debugging. Configuring the dejagnu harness to work with libstdc++-v3 in a cross compilation environment has been maddening. However, it does work now, and on a variety of platforms. Including solaris, linux, and cygwin. To debug the test harness during runs, try invoking with make check-target-libstdc++-v3 RUNTESTFLAGS="-v" or make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v" There are two ways to run on a simulator: set up DEJAGNU to point to a specially crafted site.exp, or pass down --target_board flags. Example flags to pass down for various embedded builds are as follows: --target=powerpc-eabism (libgloss/sim) make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim" --target=calmrisc32 (libgloss/sid) make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid" --target=xscale-elf (newlib/sim) make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim" // 4: Future plans, to be done Shared runs need to be implemented, for targets that support shared libraries. Diffing of expected output to standard streams needs to be finished off. The V3 testing framework supports, or will eventually support, additional keywords for the purpose of easing the job of writing testcases. All V3-keywords are of the form @xxx@. Currently plans for supported keywords include: @require@ The existence of is essential for the test to complete successfully. For example, a testcase foo.C using bar.baz as input file could say // @require@ bar.baz The special variable % stands for the rootname, e.g. the file-name without its `.C' extension. Example of use (taken verbatim from 27_io/filebuf.cc) // @require@ %-*.tst %-*.txt @diff@ After the testcase compiles and ran successfully, diff against , these lists should have the same length. The test fails if diff returns non-zero a pair of files. Current testing problems with cygwin-hosted tools: There are two known problems which I have not addressed. The first is that when testing cygwin hosted tools from the unix build dir, it does the wrong thing building the wrapper program (testglue.c) because host and target are the same in site.exp (host and target are the same from the perspective of the target libraries) Problem number two is a little more annoying. In order for me to make v3 testing work on Windows, I had to tell dejagnu to copy over the debug_assert.h file to the remote host and then set the includes to -I./. This is how all the other tests like this are done so I didn't think much of it. However, this had some unfortunate results due to gcc having a testcase called "limits" and C++ having an include file called "limits". The gcc "limits" binary was in the temporary dir when the v3 tests were being built. As a result, the gcc "limits" binary was being #included rather than the intended one. The only way to fix this is to go through the testsuites and make sure binaries are deleted on the remote host when testing is done with them. That is a lot more work than I want to do so I worked around it by cleaning out D:\kermit on compsognathus and rerunning tests.