From 0d80c9c8c78279124bb0d7e275c2b634ca640fef Mon Sep 17 00:00:00 2001 From: Dmitri Gribenko Date: Sun, 30 Dec 2012 14:51:03 +0000 Subject: Documentation: add a section to prevent spurious test failures like the one fixed in r171243. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171258 91177308-0d34-0410-b5e6-96231b3b80d8 --- docs/TestingGuide.rst | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) (limited to 'docs/TestingGuide.rst') diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst index f66cae1d14..f26d1bf60a 100644 --- a/docs/TestingGuide.rst +++ b/docs/TestingGuide.rst @@ -262,6 +262,43 @@ recommended way to examine output to figure out if the test passes it using the :doc:`FileCheck tool `. The usage of ``grep`` in RUN lines is discouraged. +Fragile tests +------------- + +It is easy to write a fragile test that would fail spuriously if the tool being +tested outputs a full path to the input file. For example, :program:`opt` by +default outputs a ``ModuleID``: + +.. code-block:: console + + $ cat example.ll + define i32 @main() nounwind { + ret i32 0 + } + + $ opt -S /path/to/example.ll + ; ModuleID = '/path/to/example.ll' + + define i32 @main() nounwind { + ret i32 0 + } + +``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example: + +.. code-block:: llvm + + ; RUN: opt -S %s | FileCheck + + define i32 @main() nounwind { + ; CHECK-NOT: load + ret i32 0 + } + +This test will fail if placed into a ``download`` directory. + +To make your tests robust, always use ``opt ... < %s`` in the RUN line. +:program:`opt` does not output a ``ModuleID`` when input comes from stdin. + The FileCheck utility --------------------- -- cgit v1.2.3-18-g5258 From 2ba4bd97d1aedf50c424c45ee327abddbf34dd12 Mon Sep 17 00:00:00 2001 From: Chris Lattner Date: Thu, 10 Jan 2013 21:24:04 +0000 Subject: remove the rest of the "written by" lines in the documentation. It is against the developer policy to include this sort of thing as SVN blame already captures this in a far more fine-grained way. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172109 91177308-0d34-0410-b5e6-96231b3b80d8 --- docs/TestingGuide.rst | 3 --- 1 file changed, 3 deletions(-) (limited to 'docs/TestingGuide.rst') diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst index f26d1bf60a..143c323154 100644 --- a/docs/TestingGuide.rst +++ b/docs/TestingGuide.rst @@ -2,9 +2,6 @@ LLVM Testing Infrastructure Guide ================================= -Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya -Lattner - .. contents:: :local: -- cgit v1.2.3-18-g5258 From 317794939cda6fe9bff2ba45c9040185cfa39802 Mon Sep 17 00:00:00 2001 From: Eli Bendersky Date: Fri, 18 Jan 2013 19:01:34 +0000 Subject: Streamline the mentions of grep and FileCheck in TestingGuide.rst grep is now only mentioned once in a sentence that explicitly says it's deprecated. For FileCheck, there's no reason to repeat part of the documentation that exists in CommandGuide/FileCheck. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172835 91177308-0d34-0410-b5e6-96231b3b80d8 --- docs/TestingGuide.rst | 44 +++++++------------------------------------- 1 file changed, 7 insertions(+), 37 deletions(-) (limited to 'docs/TestingGuide.rst') diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst index 143c323154..4d8c8ce307 100644 --- a/docs/TestingGuide.rst +++ b/docs/TestingGuide.rst @@ -231,33 +231,21 @@ what you can use in yours. The major differences are: - You can't do ``2>&1``. That will cause :program:`lit` to write to a file named ``&1``. Usually this is done to get stderr to go through a pipe. You can do that with ``|&`` so replace this idiom: - ``... 2>&1 | grep`` with ``... |& grep`` + ``... 2>&1 | FileCheck`` with ``... |& FileCheck`` - You can only redirect to a file, not to another descriptor and not from a here document. There are some quoting rules that you must pay attention to when writing your RUN lines. In general nothing needs to be quoted. :program:`lit` won't strip off any quote characters so they will get passed to the invoked program. -For example: - -.. code-block:: bash - - ... | grep 'find this string' - -This will fail because the ``'`` characters are passed to ``grep``. This would -make ``grep`` to look for ``'find`` in the files ``this`` and -``string'``. To avoid this use curly braces to tell :program:`lit` that it -should treat everything enclosed as one value. So our example would become: - -.. code-block:: bash - - ... | grep {find this string} +To avoid this use curly braces to tell :program:`lit` that it should treat +everything enclosed as one value. In general, you should strive to keep your RUN lines as simple as possible, -using them only to run tools that generate the output you can then examine. The -recommended way to examine output to figure out if the test passes it using the -:doc:`FileCheck tool `. The usage of ``grep`` in RUN -lines is discouraged. +using them only to run tools that generate textual output you can then examine. +The recommended way to examine output to figure out if the test passes it using +the :doc:`FileCheck tool `. *[The usage of grep in RUN +lines is deprecated - please do not send or commit patches that use it.]* Fragile tests ------------- @@ -296,24 +284,6 @@ This test will fail if placed into a ``download`` directory. To make your tests robust, always use ``opt ... < %s`` in the RUN line. :program:`opt` does not output a ``ModuleID`` when input comes from stdin. -The FileCheck utility ---------------------- - -A powerful feature of the RUN lines is that it allows any arbitrary -commands to be executed as part of the test harness. While standard -(portable) unix tools like ``grep`` work fine on run lines, as you see -above, there are a lot of caveats due to interaction with shell syntax, -and we want to make sure the run lines are portable to a wide range of -systems. Another major problem is that ``grep`` is not very good at checking -to verify that the output of a tools contains a series of different -output in a specific order. The :program:`FileCheck` tool was designed to -help with these problems. - -:program:`FileCheck` is designed to read a file to check from standard input, -and the set of things to verify from a file specified as a command line -argument. :program:`FileCheck` is described in :doc:`the FileCheck man page -`. - Variables and substitutions --------------------------- -- cgit v1.2.3-18-g5258