1 <?xml version="1.0" encoding="UTF-8" standalone="no"?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Test</title><meta name="generator" content="DocBook XSL Stylesheets V1.74.3" /><meta name="keywords" content=" ISO C++ , test , testsuite , performance , conformance , ABI , exception safety " /><meta name="keywords" content=" ISO C++ , library " /><link rel="home" href="../spine.html" title="The GNU C++ Library Documentation" /><link rel="up" href="setup.html" title="Chapter 2. Setup" /><link rel="prev" href="make.html" title="Make" /><link rel="next" href="using.html" title="Chapter 3. Using" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Test</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="make.html">Prev</a> </td><th width="60%" align="center">Chapter 2. Setup</th><td width="20%" align="right"> <a accesskey="n" href="using.html">Next</a></td></tr></table><hr /></div><div class="sect1" lang="en" xml:lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="manual.intro.setup.test"></a>Test</h2></div></div></div><p>
4 The libstdc++ testsuite includes testing for standard conformance,
5 regressions, ABI, and performance.
6 </p><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.organization"></a>Organization</h3></div></div></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.organization.layout"></a>Directory Layout</h4></div></div></div><p>
7 The directory <span class="emphasis"><em>libsrcdir/testsuite</em></span> contains the
8 individual test cases organized in sub-directories corresponding to
9 chapters of the C++ standard (detailed below), the dejagnu test
10 harness support files, and sources to various testsuite utilities
11 that are packaged in a separate testing library.
13 All test cases for functionality required by the runtime components
14 of the C++ standard (ISO 14882) are files within the following
16 </p><pre class="programlisting">
28 In addition, the following directories include test files:
29 </p><pre class="programlisting">
30 tr1 Tests for components as described by the Technical Report on Standard Library Extensions (TR1).
31 backward Tests for backwards compatibility and deprecated features.
32 demangle Tests for __cxa_demangle, the IA 64 C++ ABI demangler
33 ext Tests for extensions.
34 performance Tests for performance analysis, and performance regressions.
35 thread Tests for threads.
37 Some directories don't have test files, but instead contain
38 auxiliary information:
39 </p><pre class="programlisting">
40 config Files for the dejagnu test harness.
41 lib Files for the dejagnu test harness.
42 libstdc++* Files for the dejagnu test harness.
43 data Sample text files for testing input and output.
44 util Files for libtestc++, utilities and testing routines.
46 Within a directory that includes test files, there may be
47 additional subdirectories, or files. Originally, test cases
48 were appended to one file that represented a particular section
49 of the chapter under test, and was named accordingly. For
50 instance, to test items related to <code class="code"> 21.3.6.1 -
51 basic_string::find [lib.string::find]</code> in the standard,
52 the following was used:
53 </p><pre class="programlisting">
56 However, that practice soon became a liability as the test cases
57 became huge and unwieldy, and testing new or extended
58 functionality (like wide characters or named locales) became
59 frustrating, leading to aggressive pruning of test cases on some
60 platforms that covered up implementation errors. Now, the test
61 suite has a policy of one file, one test case, which solves the
62 above issues and gives finer grained results and more manageable
63 error debugging. As an example, the test case quoted above
65 </p><pre class="programlisting">
66 21_strings/basic_string/find/char/1.cc
67 21_strings/basic_string/find/char/2.cc
68 21_strings/basic_string/find/char/3.cc
69 21_strings/basic_string/find/wchar_t/1.cc
70 21_strings/basic_string/find/wchar_t/2.cc
71 21_strings/basic_string/find/wchar_t/3.cc
73 All new tests should be written with the policy of one test
74 case, one file in mind.
75 </p></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.organization.naming"></a>Naming Conventions</h4></div></div></div><p>
76 In addition, there are some special names and suffixes that are
77 used within the testsuite to designate particular kinds of
79 </p><div class="itemizedlist"><ul type="disc"><li><p>
80 <span class="emphasis"><em>_xin.cc</em></span>
82 This test case expects some kind of interactive input in order
83 to finish or pass. At the moment, the interactive tests are not
84 run by default. Instead, they are run by hand, like:
85 </p><pre class="programlisting">
86 g++ 27_io/objects/char/3_xin.cc
87 cat 27_io/objects/char/3_xin.in | a.out
89 <span class="emphasis"><em>.in</em></span>
91 This file contains the expected input for the corresponding <span class="emphasis"><em>
92 _xin.cc</em></span> test case.
94 <span class="emphasis"><em>_neg.cc</em></span>
96 This test case is expected to fail: it's a negative test. At the
97 moment, these are almost always compile time errors.
99 <span class="emphasis"><em>char</em></span>
101 This can either be a directory name or part of a longer file
102 name, and indicates that this file, or the files within this
103 directory are testing the <code class="code">char</code> instantiation of a
106 <span class="emphasis"><em>wchar_t</em></span>
108 This can either be a directory name or part of a longer file
109 name, and indicates that this file, or the files within this
110 directory are testing the <code class="code">wchar_t</code> instantiation of
111 a template. Some hosts do not support <code class="code">wchar_t</code>
112 functionality, so for these targets, all of these tests will not
115 <span class="emphasis"><em>thread</em></span>
117 This can either be a directory name or part of a longer file
118 name, and indicates that this file, or the files within this
119 directory are testing situations where multiple threads are
122 <span class="emphasis"><em>performance</em></span>
124 This can either be an enclosing directory name or part of a
125 specific file name. This indicates a test that is used to
126 analyze runtime performance, for performance regression testing,
127 or for other optimization related analysis. At the moment, these
128 test cases are not run by default.
129 </p></li></ul></div></div></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.run"></a>Running the Testsuite</h3></div></div></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.basic"></a>Basic</h4></div></div></div><p>
130 You can check the status of the build without installing it
131 using the dejagnu harness, much like the rest of the gcc
132 tools.</p><pre class="programlisting"> make check</pre><p>in the <span class="emphasis"><em>libbuilddir</em></span> directory.</p><p>or</p><pre class="programlisting"> make check-target-libstdc++-v3</pre><p>in the <span class="emphasis"><em>gccbuilddir</em></span> directory.
134 These commands are functionally equivalent and will create a
135 'testsuite' directory underneath
136 <span class="emphasis"><em>libbuilddir</em></span> containing the results of the
137 tests. Two results files will be generated: <span class="emphasis"><em>
138 libstdc++.sum</em></span>, which is a PASS/FAIL summary for each
139 test, and <span class="emphasis"><em>libstdc++.log</em></span> which is a log of
140 the exact command line passed to the compiler, the compiler
141 output, and the executable output (if any).
143 Archives of test results for various versions and platforms are
144 available on the GCC website in the <a class="ulink" href="http://gcc.gnu.org/gcc-4.3/buildstat.html" target="_top">build
145 status</a> section of each individual release, and are also
146 archived on a daily basis on the <a class="ulink" href="http://gcc.gnu.org/ml/gcc-testresults/current" target="_top">gcc-testresults</a>
147 mailing list. Please check either of these places for a similar
148 combination of source version, operating system, and host CPU.
149 </p></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.variations"></a>Variations</h4></div></div></div><p>
150 There are several options for running tests, including testing
151 the regression tests, testing a subset of the regression tests,
152 testing the performance tests, testing just compilation, testing
153 installed tools, etc. In addition, there is a special rule for
154 checking the exported symbols of the shared library.
156 To debug the dejagnu test harness during runs, try invoking with a
157 specific argument to the variable RUNTESTFLAGS, as below.
158 </p><pre class="programlisting">
159 make check-target-libstdc++-v3 RUNTESTFLAGS="-v"
162 </p><pre class="programlisting">
163 make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"
165 To run a subset of the library tests, you will need to generate
166 the <span class="emphasis"><em>testsuite_files</em></span> file by running
167 <span class="command"><strong>make testsuite_files</strong></span> in the
168 <span class="emphasis"><em>libbuilddir/testsuite</em></span> directory, described
169 below. Edit the file to remove the tests you don't want and
170 then run the testsuite as normal.
172 There are two ways to run on a simulator: set up DEJAGNU to point to a
173 specially crafted site.exp, or pass down --target_board flags.
175 Example flags to pass down for various embedded builds are as follows:
176 </p><pre class="programlisting">
177 --target=powerpc-eabism (libgloss/sim)
178 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"
180 --target=calmrisc32 (libgloss/sid)
181 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"
183 --target=xscale-elf (newlib/sim)
184 make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"
186 Also, here is an example of how to run the libstdc++ testsuite
187 for a multilibed build directory with different ABI settings:
188 </p><pre class="programlisting">
189 make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'
191 You can run the tests with a compiler and library that have
192 already been installed. Make sure that the compiler (e.g.,
193 <code class="code">g++</code>) is in your <code class="code">PATH</code>. If you are
194 using shared libraries, then you must also ensure that the
195 directory containing the shared version of libstdc++ is in your
196 <code class="code">LD_LIBRARY_PATH</code>, or equivalent. If your GCC source
197 tree is at <code class="code">/path/to/gcc</code>, then you can run the tests
199 </p><pre class="programlisting">
200 runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite
202 The testsuite will create a number of files in the directory in
203 which you run this command,. Some of those files might use the
204 same name as files created by other testsuites (like the ones
205 for GCC and G++), so you should not try to run all the
206 testsuites in parallel from the same directory.
208 In addition, there are some testing options that are mostly of
209 interest to library maintainers and system integrators. As such,
210 these tests may not work on all cpu and host combinations, and
211 may need to be executed in the
212 <span class="emphasis"><em>libbuilddir/testsuite</em></span> directory. These
213 options include, but are not necessarily limited to, the
215 </p><pre class="programlisting">
218 Five files are generated that determine what test files
219 are run. These files are:
220 </p><div class="itemizedlist"><ul type="disc"><li><p>
221 <span class="emphasis"><em>testsuite_files</em></span>
223 This is a list of all the test cases that will be run. Each
224 test case is on a separate line, given with an absolute path
225 from the <span class="emphasis"><em>libsrcdir/testsuite</em></span> directory.
227 <span class="emphasis"><em>testsuite_files_interactive</em></span>
229 This is a list of all the interactive test cases, using the
230 same format as the file list above. These tests are not run
233 <span class="emphasis"><em>testsuite_files_performance</em></span>
235 This is a list of all the performance test cases, using the
236 same format as the file list above. These tests are not run
239 <span class="emphasis"><em>testsuite_thread</em></span>
241 This file indicates that the host system can run tests which
242 involved multiple threads.
244 <span class="emphasis"><em>testsuite_wchar_t</em></span>
246 This file indicates that the host system can run the wchar_t
247 tests, and corresponds to the macro definition <code class="code">
248 _GLIBCXX_USE_WCHAR_T</code> in the file c++config.h.
249 </p></li></ul></div><pre class="programlisting">
252 The library ABI can be tested. This involves testing the shared
253 library against an ABI-defining previous version of symbol
255 </p><pre class="programlisting">
258 This rule compiles, but does not link or execute, the
259 <span class="emphasis"><em>testsuite_files</em></span> test cases and displays the
261 </p><pre class="programlisting">
262 make check-performance
264 This rule runs through the
265 <span class="emphasis"><em>testsuite_files_performance</em></span> test cases and
266 collects information for performance analysis and can be used to
267 spot performance regressions. Various timing information is
268 collected, as well as number of hard page faults, and memory
269 used. This is not run by default, and the implementation is in
272 We are interested in any strange failures of the testsuite;
273 please email the main libstdc++ mailing list if you see
274 something odd or have questions.
275 </p></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.run.permutations"></a>Permutations</h4></div></div></div><p>
276 To run the libstdc++ test suite under the <a class="link" href="debug_mode.html" title="Chapter 30. Debug Mode">debug mode</a>, edit
277 <code class="filename">libstdc++-v3/scripts/testsuite_flags</code> to add the
278 compile-time flag <code class="constant">-D_GLIBCXX_DEBUG</code> to the
279 result printed by the <code class="literal">--build-cxx</code>
280 option. Additionally, add the
281 <code class="constant">-D_GLIBCXX_DEBUG_PEDANTIC</code> flag to turn on
282 pedantic checking. The libstdc++ test suite should produce
283 precisely the same results under debug mode that it does under
284 release mode: any deviation indicates an error in either the
285 library or the test suite.
287 The <a class="link" href="parallel_mode.html" title="Chapter 31. Parallel Mode">parallel
288 mode</a> can be tested in much the same manner, substituting
289 <code class="constant">-D_GLIBCXX_PARALLEL</code> for
290 <code class="constant">-D_GLIBCXX_DEBUG</code> in the previous paragraph.
292 Or, just run the testsuites with <code class="constant">CXXFLAGS</code>
293 set to <code class="constant">-D_GLIBCXX_DEBUG</code> or
294 <code class="constant">-D_GLIBCXX_PARALLEL</code>.
295 </p></div></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.new_tests"></a>Writing a new test case</h3></div></div></div><p>
296 The first step in making a new test case is to choose the correct
297 directory and file name, given the organization as previously
300 All files are copyright the FSF, and GPL'd: this is very
301 important. The first copyright year should correspond to the date
302 the file was checked in to SVN.
304 As per the dejagnu instructions, always return 0 from main to
307 A bunch of utility functions and classes have already been
308 abstracted out into the testsuite utility library, <code class="code">
309 libtestc++</code>. To use this functionality, just include the
310 appropriate header file: the library or specific object files will
311 automatically be linked in as part of the testsuite run.
313 For a test that needs to take advantage of the dejagnu test
314 harness, what follows below is a list of special keyword that
315 harness uses. Basically, a test case contains dg-keywords (see
316 dg.exp) indicating what to do and what kinds of behavior are to be
317 expected. New test cases should be written with the new style
318 DejaGnu framework in mind.
320 To ease transition, here is the list of dg-keyword documentation
322 </p><pre class="programlisting">
323 # The currently supported options are:
328 # dg-options "options ..." [{ target selector }]
329 # specify special options to pass to the tool (eg: compiler)
331 # dg-do do-what-keyword [{ target/xfail selector }]
332 # `do-what-keyword' is tool specific and is passed unchanged to
333 # ${tool}-dg-test. An example is gcc where `keyword' can be any of:
334 # preprocess|compile|assemble|link|run
335 # and will do one of: produce a .i, produce a .s, produce a .o,
336 # produce an a.out, or produce an a.out and run it (the default is
339 # dg-error regexp comment [{ target/xfail selector } [{.|0|linenum}]]
340 # indicate an error message <regexp> is expected on this line
341 # (the test fails if it doesn't occur)
342 # Linenum=0 for general tool messages (eg: -V arg missing).
343 # "." means the current line.
345 # dg-warning regexp comment [{ target/xfail selector } [{.|0|linenum}]]
346 # indicate a warning message <regexp> is expected on this line
347 # (the test fails if it doesn't occur)
349 # dg-bogus regexp comment [{ target/xfail selector } [{.|0|linenum}]]
350 # indicate a bogus error message <regexp> use to occur here
351 # (the test fails if it does occur)
353 # dg-build regexp comment [{ target/xfail selector }]
354 # indicate the build use to fail for some reason
355 # (errors covered here include bad assembler generated, tool crashes,
357 # (the test fails if it does occur)
359 # dg-excess-errors comment [{ target/xfail selector }]
360 # indicate excess errors are expected (any line)
361 # (this should only be used sparingly and temporarily)
363 # dg-output regexp [{ target selector }]
364 # indicate the expected output of the program is <regexp>
365 # (there may be multiple occurrences of this, they are concatenated)
367 # dg-final { tcl code }
368 # add some tcl code to be run at the end
369 # (there may be multiple occurrences of this, they are concatenated)
370 # (unbalanced braces must be \-escaped)
372 # "{ target selector }" is a list of expressions that determine whether the
373 # test succeeds or fails for a particular target, or in some cases whether the
374 # option applies for a particular target. If the case of `dg-do' it specifies
375 # whether the test case is even attempted on the specified target.
377 # The target selector is always optional. The format is one of:
379 # { xfail *-*-* ... } - the test is expected to fail for the given targets
380 # { target *-*-* ... } - the option only applies to the given targets
382 # At least one target must be specified, use *-*-* for "all targets".
383 # At present it is not possible to specify both `xfail' and `target'.
384 # "native" may be used in place of "*-*-*".
386 Example 1: Testing compilation only
389 Example 2: Testing for expected warnings on line 36, which all targets fail
390 // { dg-warning "string literals" "" { xfail *-*-* } 36
392 Example 3: Testing for expected warnings on line 36
393 // { dg-warning "string literals" "" { target *-*-* } 36
395 Example 4: Testing for compilation errors on line 41
397 // { dg-error "no match for" "" { target *-*-* } 41 }
399 Example 5: Testing with special command line settings, or without the
400 use of pre-compiled headers, in particular the stdc++.h.gch file. Any
401 options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set
402 up in the normal.exp file.
403 // { dg-options "-O0" { target *-*-* } }
405 More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.
406 </p></div><div class="sect2" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="test.harness"></a>Test Harness and Utilities</h3></div></div></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.harness.dejagnu"></a>Dejagnu Harness Details</h4></div></div></div><p>
407 Underlying details of testing for conformance and regressions are
408 abstracted via the GNU Dejagnu package. This is similar to the
410 </p><p>This is information for those looking at making changes to the testsuite
411 structure, and/or needing to trace dejagnu's actions with --verbose. This
412 will not be useful to people who are "merely" adding new tests to the existing
414 </p><p>The first key point when working with dejagnu is the idea of a "tool".
415 Files, directories, and functions are all implicitly used when they are
416 named after the tool in use. Here, the tool will always be "libstdc++".
417 </p><p>The <code class="code">lib</code> subdir contains support routines. The
418 <code class="code">lib/libstdc++.exp</code> file ("support library") is loaded
419 automagically, and must explicitly load the others. For example, files can
420 be copied from the core compiler's support directory into <code class="code">lib</code>.
421 </p><p>Some routines in <code class="code">lib/libstdc++.exp</code> are callbacks, some are
422 our own. Callbacks must be prefixed with the name of the tool. To easily
423 distinguish the others, by convention our own routines are named "v3-*".
424 </p><p>The next key point when working with dejagnu is "test files". Any
425 directory whose name starts with the tool name will be searched for test files.
426 (We have only one.) In those directories, any <code class="code">.exp</code> file is
427 considered a test file, and will be run in turn. Our main test file is called
428 <code class="code">normal.exp</code>; it runs all the tests in testsuite_files using the
429 callbacks loaded from the support library.
430 </p><p>The <code class="code">config</code> directory is searched for any particular "target
431 board" information unique to this library. This is currently unused and sets
432 only default variables.
433 </p></div><div class="sect3" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="test.harness.utils"></a>Utilities</h4></div></div></div><p>
435 The testsuite directory also contains some files that implement
436 functionality that is intended to make writing test cases easier,
437 or to avoid duplication, or to provide error checking in a way that
438 is consistent across platforms and test harnesses. A stand-alone
439 executable, called <span class="emphasis"><em>abi_check</em></span>, and a static
440 library called <span class="emphasis"><em>libtestc++</em></span> are
441 constructed. Both of these items are not installed, and only used
444 These files include the following functionality:
445 </p><div class="itemizedlist"><ul type="disc"><li><p>
446 <span class="emphasis"><em>testsuite_abi.h</em></span>,
447 <span class="emphasis"><em>testsuite_abi.cc</em></span>,
448 <span class="emphasis"><em>testsuite_abi_check.cc</em></span>
450 Creates the executable <span class="emphasis"><em>abi_check</em></span>.
451 Used to check correctness of symbol versioning, visibility of
452 exported symbols, and compatibility on symbols in the shared
453 library, for hosts that support this feature. More information
454 can be found in the ABI documentation <a class="link" href="abi.html" title="ABI Policy and Guidelines">here</a>
456 <span class="emphasis"><em>testsuite_allocator.h</em></span>,
457 <span class="emphasis"><em>testsuite_allocator.cc</em></span>
459 Contains specialized allocators that keep track of construction
460 and destruction. Also, support for overriding global new and
461 delete operators, including verification that new and delete
462 are called during execution, and that allocation over max_size
465 <span class="emphasis"><em>testsuite_character.h</em></span>
467 Contains <code class="code">std::char_traits</code> and
468 <code class="code">std::codecvt</code> specializations for a user-defined
471 <span class="emphasis"><em>testsuite_hooks.h</em></span>,
472 <span class="emphasis"><em>testsuite_hooks.cc</em></span>
474 A large number of utilities, including:
475 </p><div class="itemizedlist"><ul type="circle"><li><p>VERIFY</p></li><li><p>set_memory_limits</p></li><li><p>verify_demangle</p></li><li><p>run_tests_wrapped_locale</p></li><li><p>run_tests_wrapped_env</p></li><li><p>try_named_locale</p></li><li><p>try_mkfifo</p></li><li><p>func_callback</p></li><li><p>counter</p></li><li><p>copy_tracker</p></li><li><p>copy_constructor</p></li><li><p>assignment_operator</p></li><li><p>destructor</p></li><li><p>pod_char, pod_int and associated char_traits specializations</p></li></ul></div></li><li><p>
476 <span class="emphasis"><em>testsuite_io.h</em></span>
478 Error, exception, and constraint checking for
479 <code class="code">std::streambuf, std::basic_stringbuf, std::basic_filebuf</code>.
481 <span class="emphasis"><em>testsuite_iterators.h</em></span>
483 Wrappers for various iterators.
485 <span class="emphasis"><em>testsuite_performance.h</em></span>
487 A number of class abstractions for performance counters, and
488 reporting functions including:
489 </p><div class="itemizedlist"><ul type="circle"><li><p>time_counter</p></li><li><p>resource_counter</p></li><li><p>report_performance</p></li></ul></div></li></ul></div></div></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="make.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="setup.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="using.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Make </td><td width="20%" align="center"><a accesskey="h" href="../spine.html">Home</a></td><td width="40%" align="right" valign="top"> Chapter 3. Using</td></tr></table></div></body></html>