Updating branches/google/stable to r169803

git-svn-id: https://llvm.org/svn/llvm-project/llvm/branches/google/stable@170212 91177308-0d34-0410-b5e6-96231b3b80d8

1 files changed, 460 insertions, 0 deletions

diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst
new file mode 100644
index 0000000000..f66cae1d14
--- /dev/null
+++ b/docs/TestingGuide.rst
@@ -0,0 +1,460 @@
+=================================
+LLVM Testing Infrastructure Guide
+=================================
+
+Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya
+Lattner
+
+.. contents::
+   :local:
+
+.. toctree::
+   :hidden:
+
+   TestSuiteMakefileGuide
+
+Overview
+========
+
+This document is the reference manual for the LLVM testing
+infrastructure. It documents the structure of the LLVM testing
+infrastructure, the tools needed to use it, and how to add and run
+tests.
+
+Requirements
+============
+
+In order to use the LLVM testing infrastructure, you will need all of
+the software required to build LLVM, as well as
+`Python <http://python.org>`_ 2.4 or later.
+
+LLVM testing infrastructure organization
+========================================
+
+The LLVM testing infrastructure contains two major categories of tests:
+regression tests and whole programs. The regression tests are contained
+inside the LLVM repository itself under ``llvm/test`` and are expected
+to always pass -- they should be run before every commit.
+
+The whole programs tests are referred to as the "LLVM test suite" (or
+"test-suite") and are in the ``test-suite`` module in subversion. For
+historical reasons, these tests are also referred to as the "nightly
+tests" in places, which is less ambiguous than "test-suite" and remains
+in use although we run them much more often than nightly.
+
+Regression tests
+----------------
+
+The regression tests are small pieces of code that test a specific
+feature of LLVM or trigger a specific bug in LLVM. The language they are
+written in depends on the part of LLVM being tested. These tests are driven by
+the :doc:`Lit <CommandGuide/lit>` testing tool (which is part of LLVM), and
+are located in the ``llvm/test`` directory.
+
+Typically when a bug is found in LLVM, a regression test containing just
+enough code to reproduce the problem should be written and placed
+somewhere underneath this directory. For example, it can be a small
+piece of LLVM IR distilled from an actual application or benchmark.
+
+``test-suite``
+--------------
+
+The test suite contains whole programs, which are pieces of code which
+can be compiled and linked into a stand-alone program that can be
+executed. These programs are generally written in high level languages
+such as C or C++.
+
+These programs are compiled using a user specified compiler and set of
+flags, and then executed to capture the program output and timing
+information. The output of these programs is compared to a reference
+output to ensure that the program is being compiled correctly.
+
+In addition to compiling and executing programs, whole program tests
+serve as a way of benchmarking LLVM performance, both in terms of the
+efficiency of the programs generated as well as the speed with which
+LLVM compiles, optimizes, and generates code.
+
+The test-suite is located in the ``test-suite`` Subversion module.
+
+Debugging Information tests
+---------------------------
+
+The test suite contains tests to check quality of debugging information.
+The test are written in C based languages or in LLVM assembly language.
+
+These tests are compiled and run under a debugger. The debugger output
+is checked to validate of debugging information. See README.txt in the
+test suite for more information . This test suite is located in the
+``debuginfo-tests`` Subversion module.
+
+Quick start
+===========
+
+The tests are located in two separate Subversion modules. The
+regressions tests are in the main "llvm" module under the directory
+``llvm/test`` (so you get these tests for free with the main LLVM tree).
+Use ``make check-all`` to run the regression tests after building LLVM.
+
+The more comprehensive test suite that includes whole programs in C and C++
+is in the ``test-suite`` module. See :ref:`test-suite Quickstart
+<test-suite-quickstart>` for more information on running these tests.
+
+Regression tests
+----------------
+
+To run all of the LLVM regression tests, use the master Makefile in the
+``llvm/test`` directory. LLVM Makefiles require GNU Make (read the :doc:`LLVM
+Makefile Guide <MakefileGuide>` for more details):
+
+.. code-block:: bash
+
+    % make -C llvm/test
+
+or:
+
+.. code-block:: bash
+
+    % make check
+
+If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
+can run the LLVM and Clang tests simultaneously using:
+
+.. code-block:: bash
+
+    % make check-all
+
+To run the tests with Valgrind (Memcheck by default), just append
+``VG=1`` to the commands above, e.g.:
+
+.. code-block:: bash
+
+    % make check VG=1
+
+To run individual tests or subsets of tests, you can use the ``llvm-lit``
+script which is built as part of LLVM. For example, to run the
+``Integer/BitPacked.ll`` test by itself you can run:
+
+.. code-block:: bash
+
+    % llvm-lit ~/llvm/test/Integer/BitPacked.ll 
+
+or to run all of the ARM CodeGen tests:
+
+.. code-block:: bash
+
+    % llvm-lit ~/llvm/test/CodeGen/ARM
+
+For more information on using the :program:`lit` tool, see ``llvm-lit --help``
+or the :doc:`lit man page <CommandGuide/lit>`.
+
+Debugging Information tests
+---------------------------
+
+To run debugging information tests simply checkout the tests inside
+clang/test directory.
+
+.. code-block:: bash
+
+    % cd clang/test
+    % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
+
+These tests are already set up to run as part of clang regression tests.
+
+Regression test structure
+=========================
+
+The LLVM regression tests are driven by :program:`lit` and are located in the
+``llvm/test`` directory.
+
+This directory contains a large array of small tests that exercise
+various features of LLVM and to ensure that regressions do not occur.
+The directory is broken into several sub-directories, each focused on a
+particular area of LLVM.
+
+Writing new regression tests
+----------------------------
+
+The regression test structure is very simple, but does require some
+information to be set. This information is gathered via ``configure``
+and is written to a file, ``test/lit.site.cfg`` in the build directory.
+The ``llvm/test`` Makefile does this work for you.
+
+In order for the regression tests to work, each directory of tests must
+have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine
+how to run the tests. This file is just Python code and thus is very
+flexible, but we've standardized it for the LLVM regression tests. If
+you're adding a directory of tests, just copy ``lit.local.cfg`` from
+another directory to get running. The standard ``lit.local.cfg`` simply
+specifies which files to look in for tests. Any directory that contains
+only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
+documentation <CommandGuide/lit>` for more information.
+
+Each test file must contain lines starting with "RUN:" that tell :program:`lit`
+how to run it. If there are no RUN lines, :program:`lit` will issue an error
+while running a test.
+
+RUN lines are specified in the comments of the test program using the
+keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
+to execute. Together, these lines form the "script" that :program:`lit`
+executes to run the test case. The syntax of the RUN lines is similar to a
+shell's syntax for pipelines including I/O redirection and variable
+substitution. However, even though these lines may *look* like a shell
+script, they are not. RUN lines are interpreted by :program:`lit`.
+Consequently, the syntax differs from shell in a few ways. You can specify
+as many RUN lines as needed.
+
+:program:`lit` performs substitution on each RUN line to replace LLVM tool names
+with the full paths to the executable built for each tool (in
+``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does
+not invoke any stray LLVM tools in the user's path during testing.
+
+Each RUN line is executed on its own, distinct from other lines unless
+its last character is ``\``. This continuation character causes the RUN
+line to be concatenated with the next one. In this way you can build up
+long pipelines of commands without making huge line lengths. The lines
+ending in ``\`` are concatenated until a RUN line that doesn't end in
+``\`` is found. This concatenated set of RUN lines then constitutes one
+execution. :program:`lit` will substitute variables and arrange for the pipeline
+to be executed. If any process in the pipeline fails, the entire line (and
+test case) fails too.
+
+Below is an example of legal RUN lines in a ``.ll`` file:
+
+.. code-block:: llvm
+
+    ; RUN: llvm-as < %s | llvm-dis > %t1
+    ; RUN: llvm-dis < %s.bc-13 > %t2
+    ; RUN: diff %t1 %t2
+
+As with a Unix shell, the RUN lines permit pipelines and I/O
+redirection to be used. However, the usage is slightly different than
+for Bash. In general, it's useful to read the code of other tests to figure out
+what you can use in yours. The major differences are:
+
+-  You can't do ``2>&1``. That will cause :program:`lit` to write to a file
+   named ``&1``. Usually this is done to get stderr to go through a pipe. You
+   can do that with ``|&`` so replace this idiom:
+   ``... 2>&1 | grep`` with ``... |& grep``
+-  You can only redirect to a file, not to another descriptor and not
+   from a here document.
+
+There are some quoting rules that you must pay attention to when writing
+your RUN lines. In general nothing needs to be quoted. :program:`lit` won't
+strip off any quote characters so they will get passed to the invoked program.
+For example:
+
+.. code-block:: bash
+
+    ... | grep 'find this string'
+
+This will fail because the ``'`` characters are passed to ``grep``. This would
+make ``grep`` to look for ``'find`` in the files ``this`` and
+``string'``. To avoid this use curly braces to tell :program:`lit` that it
+should treat everything enclosed as one value. So our example would become:
+
+.. code-block:: bash
+
+    ... | grep {find this string}
+
+In general, you should strive to keep your RUN lines as simple as possible,
+using them only to run tools that generate the output you can then examine. The
+recommended way to examine output to figure out if the test passes it using the
+:doc:`FileCheck tool <CommandGuide/FileCheck>`. The usage of ``grep`` in RUN
+lines is discouraged.
+
+The FileCheck utility
+---------------------
+
+A powerful feature of the RUN lines is that it allows any arbitrary
+commands to be executed as part of the test harness. While standard
+(portable) unix tools like ``grep`` work fine on run lines, as you see
+above, there are a lot of caveats due to interaction with shell syntax,
+and we want to make sure the run lines are portable to a wide range of
+systems. Another major problem is that ``grep`` is not very good at checking
+to verify that the output of a tools contains a series of different
+output in a specific order. The :program:`FileCheck` tool was designed to
+help with these problems.
+
+:program:`FileCheck` is designed to read a file to check from standard input,
+and the set of things to verify from a file specified as a command line
+argument. :program:`FileCheck` is described in :doc:`the FileCheck man page
+<CommandGuide/FileCheck>`.
+
+Variables and substitutions
+---------------------------
+
+With a RUN line there are a number of substitutions that are permitted.
+To make a substitution just write the variable's name preceded by a ``$``.
+Additionally, for compatibility reasons with previous versions of the
+test library, certain names can be accessed with an alternate syntax: a
+% prefix. These alternates are deprecated and may go away in a future
+version.
+
+Here are the available variable names. The alternate syntax is listed in
+parentheses.
+
+``$test`` (``%s``)
+   The full path to the test case's source. This is suitable for passing on
+   the command line as the input to an LLVM tool.
+
+``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
+   The number of the line where this variable is used, with an optional
+   integer offset. This can be used in tests with multiple RUN lines,
+   which reference test file's line numbers.
+
+``$srcdir``
+   The source directory from where the ``make check`` was run.
+
+``objdir``
+   The object directory that corresponds to the ``$srcdir``.
+
+``subdir``
+   A partial path from the ``test`` directory that contains the
+   sub-directory that contains the test source being executed.
+
+``srcroot``
+   The root directory of the LLVM src tree.
+
+``objroot``
+   The root directory of the LLVM object tree. This could be the same as
+   the srcroot.
+
+``path``
+   The path to the directory that contains the test case source. This is
+   for locating any supporting files that are not generated by the test,
+   but used by the test.
+
+``tmp``
+   The path to a temporary file name that could be used for this test case.
+   The file name won't conflict with other test cases. You can append to it
+   if you need multiple temporaries. This is useful as the destination of
+   some redirected output.
+
+``target_triplet`` (``%target_triplet``)
+   The target triplet that corresponds to the current host machine (the one
+   running the test cases). This should probably be called "host".
+
+``link`` (``%link``)
+   This full link command used to link LLVM executables. This has all the
+   configured ``-I``, ``-L`` and ``-l`` options.
+
+``shlibext`` (``%shlibext``)
+   The suffix for the host platforms shared library (DLL) files. This
+   includes the period as the first character.
+
+To add more variables, look at ``test/lit.cfg``.
+
+Other Features
+--------------
+
+To make RUN line writing easier, there are several helper scripts and programs
+in the ``llvm/test/Scripts`` directory. This directory is in the PATH
+when running tests, so you can just call these scripts using their name.
+For example:
+
+``ignore``
+   This script runs its arguments and then always returns 0. This is useful
+   in cases where the test needs to cause a tool to generate an error (e.g.
+   to check the error output). However, any program in a pipeline that
+   returns a non-zero result will cause the test to fail.  This script
+   overcomes that issue and nicely documents that the test case is
+   purposefully ignoring the result code of the tool
+``not``
+   This script runs its arguments and then inverts the result code from it.
+   Zero result codes become 1. Non-zero result codes become 0.
+
+Sometimes it is necessary to mark a test case as "expected fail" or
+XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
+on a line near the top of the file. This signals that the test case
+should succeed if the test fails. Such test cases are counted separately
+by the testing tool. To specify an expected fail, use the XFAIL keyword
+in the comments of the test program followed by a colon and one or more
+failure patterns. Each failure pattern can be either ``*`` (to specify
+fail everywhere), or a part of a target triple (indicating the test
+should fail on that platform), or the name of a configurable feature
+(for example, ``loadable_module``). If there is a match, the test is
+expected to fail. If not, the test is expected to succeed. To XFAIL
+everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
+line:
+
+.. code-block:: llvm
+
+    ; XFAIL: darwin,sun
+
+To make the output more useful, :program:`lit` will scan
+the lines of the test case for ones that contain a pattern that matches
+``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
+that is related to the test case. The number after "PR" specifies the
+LLVM bugzilla number. When a PR number is specified, it will be used in
+the pass/fail reporting. This is useful to quickly get some context when
+a test fails.
+
+Finally, any line that contains "END." will cause the special
+interpretation of lines to terminate. This is generally done right after
+the last RUN: line. This has two side effects:
+
+(a) it prevents special interpretation of lines that are part of the test
+    program, not the instructions to the test case, and
+
+(b) it speeds things up for really big test cases by avoiding
+    interpretation of the remainder of the file.
+
+``test-suite`` Overview
+=======================
+
+The ``test-suite`` module contains a number of programs that can be
+compiled and executed. The ``test-suite`` includes reference outputs for
+all of the programs, so that the output of the executed program can be
+checked for correctness.
+
+``test-suite`` tests are divided into three types of tests: MultiSource,
+SingleSource, and External.
+
+-  ``test-suite/SingleSource``
+
+   The SingleSource directory contains test programs that are only a
+   single source file in size. These are usually small benchmark
+   programs or small programs that calculate a particular value. Several
+   such programs are grouped together in each directory.
+
+-  ``test-suite/MultiSource``
+
+   The MultiSource directory contains subdirectories which contain
+   entire programs with multiple source files. Large benchmarks and
+   whole applications go here.
+
+-  ``test-suite/External``
+
+   The External directory contains Makefiles for building code that is
+   external to (i.e., not distributed with) LLVM. The most prominent
+   members of this directory are the SPEC 95 and SPEC 2000 benchmark
+   suites. The ``External`` directory does not contain these actual
+   tests, but only the Makefiles that know how to properly compile these
+   programs from somewhere else. When using ``LNT``, use the
+   ``--test-externals`` option to include these tests in the results.
+
+.. _test-suite-quickstart:
+
+``test-suite`` Quickstart
+-------------------------
+
+The modern way of running the ``test-suite`` is focused on testing and
+benchmarking complete compilers using the
+`LNT <http://llvm.org/docs/lnt>`_ testing infrastructure.
+
+For more information on using LNT to execute the ``test-suite``, please
+see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_
+documentation.
+
+``test-suite`` Makefiles
+------------------------
+
+Historically, the ``test-suite`` was executed using a complicated setup
+of Makefiles. The LNT based approach above is recommended for most
+users, but there are some testing scenarios which are not supported by
+the LNT approach. In addition, LNT currently uses the Makefile setup
+under the covers and so developers who are interested in how LNT works
+under the hood may want to understand the Makefile based setup.
+
+For more information on the ``test-suite`` Makefile setup, please see
+the :doc:`Test Suite Makefile Guide <TestSuiteMakefileGuide>`.