diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst index 44f3904f4e426..e7645cb5885f1 100644 --- a/libcxx/docs/TestingLibcxx.rst +++ b/libcxx/docs/TestingLibcxx.rst @@ -325,18 +325,98 @@ This macro is in a different header as ``assert_macros.h`` since it pulls in additional headers. .. note: This macro can only be used in test using C++20 or newer. The macro - was added at a time where most of lib++'s C++17 support was complete. + was added at a time where most of libc++'s C++17 support was complete. Since it is not expected to add this to existing tests no effort was taken to make it work in earlier language versions. -Additional reading ------------------- - -The function ``CxxStandardLibraryTest`` in the file -``libcxx/utils/libcxx/test/format.py`` has documentation about writing test. It -explains the difference between the test named ``foo.pass.cpp`` and named -``foo.verify.cpp`` are. +Test names +---------- + +The names of test files have meaning for the libc++-specific configuration of +Lit. Based on the pattern that matches the name of a test file, Lit will test +the code contained therein in different ways. Refer to the `Lit Meaning of libc++ +Test Filenames`_ when determining the names for new test files. + +.. _Lit Meaning of libc++ Test Filenames: +.. list-table:: Lit Meaning of libc++ Test Filenames + :widths: 25 75 + :header-rows: 1 + + * - Name Pattern + - Meaning + * - ``FOO.pass.cpp`` + - Checks whether the C++ code in the file compiles, links and runs successfully. + * - ``FOO.pass.mm`` + - Same as ``FOO.pass.cpp``, but for Objective-C++. + + * - ``FOO.compile.pass.cpp`` + - Checks whether the C++ code in the file compiles successfully. In general, prefer ``compile`` tests over ``verify`` tests, + subject to the specific recommendations, below, for when to write ``verify`` tests. + * - ``FOO.compile.pass.mm`` + - Same as ``FOO.compile.pass.cpp``, but for Objective-C++. + * - ``FOO.compile.fail.cpp`` + - Checks that the code in the file does *not* compile successfully. + + * - ``FOO.verify.cpp`` + - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify. + For additional information about how to write ``verify`` tests, see the `Internals Manual `_. + Prefer `verify` tests over ``compile`` tests to test that compilation fails for a particular reason. For example, use a ``verify`` test + to ensure that + + * an expected ``static_assert`` is triggered; + * the use of deprecated functions generates the proper warning; + * removed functions are no longer usable; or + * return values from functions marked ``[[nodiscard]]`` are stored. + + * - ``FOO.link.pass.cpp`` + - Checks that the C++ code in the file compiles and links successfully -- no run attempted. + * - ``FOO.link.pass.mm`` + - Same as ``FOO.link.pass.cpp``, but for Objective-C++. + * - ``FOO.link.fail.cpp`` + - Checks whether the C++ code in the file fails to link after successful compilation. + * - ``FOO.link.fail.mm`` + - Same as ``FOO.link.fail.cpp``, but for Objective-C++. + + * - ``FOO.sh.`` + - A *builtin Lit Shell* test. + * - ``FOO.gen.`` + - A variant of a *Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected + by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type + of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties + in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques. + + +libc++-Specific Lit Features +---------------------------- + +Custom Directives +~~~~~~~~~~~~~~~~~ + +Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libc++ adds two additional libc++-specific directives that makes +writing tests easier. See `libc++-specific Lit Directives`_ for more information about the ``FILE_DEPENDENCIES`` and ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific directives. + +.. _libc++-specific Lit Directives: +.. list-table:: libc++-specific Lit Directives + :widths: 20 35 45 + :header-rows: 1 + + * - Directive + - Parameters + - Usage + * - ``FILE_DEPENDENCIES`` + - ``// FILE_DEPENDENCIES: file, directory, /path/to/file, ...`` + - The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. For example, a test that requires some test + input stored in a data file would use this libc++-specific Lit directive. When a test file contains the ``FILE_DEPENDENCIES`` directive, Lit will collect the named files and copy + them to the directory represented by the ``%T`` substitution before the test executes. The copy is performed from the directory represented by the ``%S`` substitution + (i.e. the source directory of the test being executed) which makes it possible to use relative paths to specify the location of dependency files. After Lit copies + all the dependent files to the directory specified by the ``%T`` substitution, that directory should contain *all* the necessary inputs to run. In other words, + it should be possible to copy the contents of the directory specified by the ``%T`` substitution to a remote host where the execution of the test will actually occur. + * - ``ADDITIONAL_COMPILE_FLAGS`` + - ``// ADDITIONAL_COMPILE_FLAGS: flag1 flag2 ...`` + - The additional compiler flags specified by a space-separated list to the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the end of the ``%{compile_flags}`` + substitution for the test that contains it. This libc++-specific Lit directive makes it possible to add special compilation flags without having to resort to writing a ``.sh.cpp`` test (see + `Lit Meaning of libc++ Test Filenames`_), more powerful but perhaps overkill. Benchmarks ========== diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py index e58e404bfcd2a..5d84711bf5d28 100644 --- a/libcxx/utils/libcxx/test/format.py +++ b/libcxx/utils/libcxx/test/format.py @@ -151,38 +151,11 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest): """ Lit test format for the C++ Standard Library conformance test suite. - This test format is based on top of the ShTest format -- it basically - creates a shell script performing the right operations (compile/link/run) - based on the extension of the test file it encounters. It supports files - with the following extensions: - - FOO.pass.cpp - Compiles, links and runs successfully - FOO.pass.mm - Same as .pass.cpp, but for Objective-C++ - - FOO.compile.pass.cpp - Compiles successfully, link and run not attempted - FOO.compile.pass.mm - Same as .compile.pass.cpp, but for Objective-C++ - FOO.compile.fail.cpp - Does not compile successfully - - FOO.link.pass.cpp - Compiles and links successfully, run not attempted - FOO.link.pass.mm - Same as .link.pass.cpp, but for Objective-C++ - FOO.link.fail.cpp - Compiles successfully, but fails to link - - FOO.sh. - A builtin Lit Shell test - - FOO.gen. - A .sh test that generates one or more Lit tests on the - fly. Executing this test must generate one or more files - as expected by LLVM split-file, and each generated file - leads to a separate Lit test that runs that file as - defined by the test format. This can be used to generate - multiple Lit tests from a single source file, which is - useful for testing repetitive properties in the library. - Be careful not to abuse this since this is not a replacement - for usual code reuse techniques. - - FOO.verify.cpp - Compiles with clang-verify. This type of test is - automatically marked as UNSUPPORTED if the compiler - does not support Clang-verify. - + Lit tests are contained in files that follow a certain pattern, which determines the semantics of the test. + Under the hood, we basically generate a builtin Lit shell test that follows the ShTest format, and perform + the appropriate operations (compile/link/run). See + https://libcxx.llvm.org/TestingLibcxx.html#test-names + for a complete description of those semantics. Substitution requirements =============================== @@ -200,30 +173,6 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest): in the same command line. In other words, the test format doesn't perform separate compilation and linking steps in this case. - - Additional supported directives - =============================== - In addition to everything that's supported in Lit ShTests, this test format - also understands the following directives inside test files: - - // FILE_DEPENDENCIES: file, directory, /path/to/file - - This directive expresses that the test requires the provided files - or directories in order to run. An example is a test that requires - some test input stored in a data file. When a test file contains - such a directive, this test format will collect them and copy them - to the directory represented by %T. The intent is that %T contains - all the inputs necessary to run the test, such that e.g. execution - on a remote host can be done by simply copying %T to the host. - - // ADDITIONAL_COMPILE_FLAGS: flag1 flag2 flag3 - - This directive will cause the provided flags to be added to the - %{compile_flags} substitution for the test that contains it. This - allows adding special compilation flags without having to use a - .sh.cpp test, which would be more powerful but perhaps overkill. - - Additional provided substitutions and features ============================================== The test format will define the following substitutions for use inside tests: