diff options
| author | mike-m <mikem.llvm@gmail.com> | 2010-05-07 00:28:04 +0000 |
|---|---|---|
| committer | mike-m <mikem.llvm@gmail.com> | 2010-05-07 00:28:04 +0000 |
| commit | e2c3a49c8029ebd9ef530101cc24c66562e3dff5 (patch) | |
| tree | 91bf9600cc8df90cf99751a8f8bafc317cffc91e /docs/TestingGuide.html | |
| parent | c10b5afbe8138b0fdf3af4ed3e1ddf96cf3cb4cb (diff) | |
Revert r103213. It broke several sections of live website.
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@103219 91177308-0d34-0410-b5e6-96231b3b80d8
Diffstat (limited to 'docs/TestingGuide.html')
| -rw-r--r-- | docs/TestingGuide.html | 1212 |
1 files changed, 1212 insertions, 0 deletions
diff --git a/docs/TestingGuide.html b/docs/TestingGuide.html new file mode 100644 index 0000000000..c39065a282 --- /dev/null +++ b/docs/TestingGuide.html @@ -0,0 +1,1212 @@ +<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" + "http://www.w3.org/TR/html4/strict.dtd"> +<html> +<head> + <title>LLVM Testing Infrastructure Guide</title> + <link rel="stylesheet" href="llvm.css" type="text/css"> +</head> +<body> + +<div class="doc_title"> + LLVM Testing Infrastructure Guide +</div> + +<ol> + <li><a href="#overview">Overview</a></li> + <li><a href="#requirements">Requirements</a></li> + <li><a href="#org">LLVM testing infrastructure organization</a> + <ul> + <li><a href="#dejagnu">DejaGNU tests</a></li> + <li><a href="#testsuite">Test suite</a></li> + </ul> + </li> + <li><a href="#quick">Quick start</a> + <ul> + <li><a href="#quickdejagnu">DejaGNU tests</a></li> + <li><a href="#quicktestsuite">Test suite</a></li> + </ul> + </li> + <li><a href="#dgstructure">DejaGNU structure</a> + <ul> + <li><a href="#dgcustom">Writing new DejaGNU tests</a></li> + <li><a href="#FileCheck">The FileCheck utility</a></li> + <li><a href="#dgvars">Variables and substitutions</a></li> + <li><a href="#dgfeatures">Other features</a></li> + </ul> + </li> + <li><a href="#testsuitestructure">Test suite structure</a></li> + <li><a href="#testsuiterun">Running the test suite</a> + <ul> + <li><a href="#testsuiteexternal">Configuring External Tests</a></li> + <li><a href="#testsuitetests">Running different tests</a></li> + <li><a href="#testsuiteoutput">Generating test output</a></li> + <li><a href="#testsuitecustom">Writing custom tests for llvm-test</a></li> + </ul> + </li> + <li><a href="#nightly">Running the nightly tester</a></li> +</ol> + +<div class="doc_author"> + <p>Written by John T. Criswell, <a + href="http://llvm.x10sys.com/rspencer">Reid Spencer</a>, and Tanya Lattner</p> +</div> + +<!--=========================================================================--> +<div class="doc_section"><a name="overview">Overview</a></div> +<!--=========================================================================--> + +<div class="doc_text"> + +<p>This document is the reference manual for the LLVM testing infrastructure. It documents +the structure of the LLVM testing infrastructure, the tools needed to use it, +and how to add and run tests.</p> + +</div> + +<!--=========================================================================--> +<div class="doc_section"><a name="requirements">Requirements</a></div> +<!--=========================================================================--> + +<div class="doc_text"> + +<p>In order to use the LLVM testing infrastructure, you will need all of the software +required to build LLVM, plus the following:</p> + +<dl> +<dt><a href="http://www.gnu.org/software/dejagnu/">DejaGNU</a></dt> +<dd>The Feature and Regressions tests are organized and run by DejaGNU.</dd> +<dt><a href="http://expect.nist.gov/">Expect</a></dt> +<dd>Expect is required by DejaGNU.</dd> +<dt><a href="http://www.tcl.tk/software/tcltk/">tcl</a></dt> +<dd>Tcl is required by DejaGNU. </dd> +</dl> + +</div> + +<!--=========================================================================--> +<div class="doc_section"><a name="org">LLVM testing infrastructure organization</a></div> +<!--=========================================================================--> + +<div class="doc_text"> + +<p>The LLVM testing infrastructure contains two major categories of tests: code +fragments and whole programs. Code fragments are referred to as the "DejaGNU +tests" and are in the <tt>llvm</tt> module in subversion under the +<tt>llvm/test</tt> directory. The whole programs tests are referred to as the +"Test suite" and are in the <tt>test-suite</tt> module in subversion. +</p> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="dejagnu">DejaGNU tests</a></div> +<!-- _______________________________________________________________________ --> + +<div class="doc_text"> + +<p>Code fragments are small pieces of code that test a specific +feature of LLVM or trigger a specific bug in LLVM. They are usually +written in LLVM assembly language, but can be written in other +languages if the test targets a particular language front end (and the +appropriate <tt>--with-llvmgcc</tt> options were used +at <tt>configure</tt> time of the <tt>llvm</tt> module). These tests +are driven by the DejaGNU testing framework, which is hidden behind a +few simple makefiles.</p> + +<p>These code fragments are not complete programs. The code generated +from them is never executed to determine correct behavior.</p> + +<p>These code fragment tests are located in the <tt>llvm/test</tt> +directory.</p> + +<p>Typically when a bug is found in LLVM, a regression test containing +just enough code to reproduce the problem should be written and placed +somewhere underneath this directory. In most cases, this will be a small +piece of LLVM assembly language code, often distilled from an actual +application or benchmark.</p> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="testsuite">Test suite</a></div> +<!-- _______________________________________________________________________ --> + +<div class="doc_text"> + +<p>The test suite contains whole programs, which are pieces of +code which can be compiled and linked into a stand-alone program that can be +executed. These programs are generally written in high level languages such as +C or C++, but sometimes they are written straight in LLVM assembly.</p> + +<p>These programs are compiled and then executed using several different +methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation, +etc). The output of these programs is compared to ensure that LLVM is compiling +the program correctly.</p> + +<p>In addition to compiling and executing programs, whole program tests serve as +a way of benchmarking LLVM performance, both in terms of the efficiency of the +programs generated as well as the speed with which LLVM compiles, optimizes, and +generates code.</p> + +<p>The test-suite is located in the <tt>test-suite</tt> Subversion module.</p> + +</div> + +<!--=========================================================================--> +<div class="doc_section"><a name="quick">Quick start</a></div> +<!--=========================================================================--> + +<div class="doc_text"> + + <p>The tests are located in two separate Subversion modules. The + DejaGNU tests are in the main "llvm" module under the directory + <tt>llvm/test</tt> (so you get these tests for free with the main llvm tree). + The more comprehensive test suite that includes whole +programs in C and C++ is in the <tt>test-suite</tt> module. This module should +be checked out to the <tt>llvm/projects</tt> directory (don't use another name +then the default "test-suite", for then the test suite will be run every time +you run <tt>make</tt> in the main <tt>llvm</tt> directory). +When you <tt>configure</tt> the <tt>llvm</tt> module, +the <tt>test-suite</tt> directory will be automatically configured. +Alternatively, you can configure the <tt>test-suite</tt> module manually.</p> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="quickdejagnu">DejaGNU tests</a></div> +<!-- _______________________________________________________________________ --> +<p>To run all of the simple tests in LLVM using DejaGNU, use the master Makefile + in the <tt>llvm/test</tt> directory:</p> + +<div class="doc_code"> +<pre> +% gmake -C llvm/test +</pre> +</div> + +<p>or</p> + +<div class="doc_code"> +<pre> +% gmake check +</pre> +</div> + +<p>To run only a subdirectory of tests in <tt>llvm/test</tt> using DejaGNU (ie. +Transforms), just set the TESTSUITE variable to the path of the +subdirectory (relative to <tt>llvm/test</tt>):</p> + +<div class="doc_code"> +<pre> +% gmake TESTSUITE=Transforms check +</pre> +</div> + +<p><b>Note: If you are running the tests with <tt>objdir != subdir</tt>, you +must have run the complete testsuite before you can specify a +subdirectory.</b></p> + +<p>To run only a single test, set <tt>TESTONE</tt> to its path (relative to +<tt>llvm/test</tt>) and make the <tt>check-one</tt> target:</p> + +<div class="doc_code"> +<pre> +% gmake TESTONE=Feature/basictest.ll check-one +</pre> +</div> + +<p>To run the tests with Valgrind (Memcheck by default), just append +<tt>VG=1</tt> to the commands above, e.g.:</p> + +<div class="doc_code"> +<pre> +% gmake check VG=1 +</pre> +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="quicktestsuite">Test suite</a></div> +<!-- _______________________________________________________________________ --> + +<p>To run the comprehensive test suite (tests that compile and execute whole +programs), first checkout and setup the <tt>test-suite</tt> module:</p> + +<div class="doc_code"> +<pre> +% cd llvm/projects +% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite +% cd .. +% ./configure --with-llvmgccdir=$LLVM_GCC_DIR +</pre> +</div> + +<p>where <tt>$LLVM_GCC_DIR</tt> is the directory where +you <em>installed</em> llvm-gcc, not it's src or obj +dir. The <tt>--with-llvmgccdir</tt> option assumes that +the <tt>llvm-gcc-4.2</tt> module was configured with +<tt>--program-prefix=llvm-</tt>, and therefore that the C and C++ +compiler drivers are called <tt>llvm-gcc</tt> and <tt>llvm-g++</tt> +respectively. If this is not the case, +use <tt>--with-llvmgcc</tt>/<tt>--with-llvmgxx</tt> to specify each +executable's location.</p> + +<p>Then, run the entire test suite by running make in the <tt>test-suite</tt> +directory:</p> + +<div class="doc_code"> +<pre> +% cd projects/test-suite +% gmake +</pre> +</div> + +<p>Usually, running the "nightly" set of tests is a good idea, and you can also +let it generate a report by running:</p> + +<div class="doc_code"> +<pre> +% cd projects/test-suite +% gmake TEST=nightly report report.html +</pre> +</div> + +<p>Any of the above commands can also be run in a subdirectory of +<tt>projects/test-suite</tt> to run the specified test only on the programs in +that subdirectory.</p> + +</div> + +<!--=========================================================================--> +<div class="doc_section"><a name="dgstructure">DejaGNU structure</a></div> +<!--=========================================================================--> +<div class="doc_text"> + <p>The LLVM DejaGNU tests are driven by DejaGNU together with GNU Make and are + located in the <tt>llvm/test</tt> directory. + + <p>This directory contains a large array of small tests + that exercise various features of LLVM and to ensure that regressions do not + occur. The directory is broken into several sub-directories, each focused on + a particular area of LLVM. A few of the important ones are:</p> + + <ul> + <li><tt>Analysis</tt>: checks Analysis passes.</li> + <li><tt>Archive</tt>: checks the Archive library.</li> + <li><tt>Assembler</tt>: checks Assembly reader/writer functionality.</li> + <li><tt>Bitcode</tt>: checks Bitcode reader/writer functionality.</li> + <li><tt>CodeGen</tt>: checks code generation and each target.</li> + <li><tt>Features</tt>: checks various features of the LLVM language.</li> + <li><tt>Linker</tt>: tests bitcode linking.</li> + <li><tt>Transforms</tt>: tests each of the scalar, IPO, and utility + transforms to ensure they make the right transformations.</li> + <li><tt>Verifier</tt>: tests the IR verifier.</li> + </ul> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="dgcustom">Writing new DejaGNU tests</a></div> +<!-- _______________________________________________________________________ --> +<div class="doc_text"> + <p>The DejaGNU structure is very simple, but does require some information to + be set. This information is gathered via <tt>configure</tt> and is written + to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt> + Makefile does this work for you.</p> + + <p>In order for DejaGNU to work, each directory of tests must have a + <tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the + tests. This file is just a Tcl script and it can do anything you want, but + we've standardized it for the LLVM regression tests. If you're adding a + directory of tests, just copy <tt>dg.exp</tt> from another directory to get + running. The standard <tt>dg.exp</tt> simply loads a Tcl + library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt> + function defined in that library with a list of file names to run. The names + are obtained by using Tcl's glob command. Any directory that contains only + directories does not need the <tt>dg.exp</tt> file.</p> + + <p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to + it and gathers any lines together that match "RUN:". This are the "RUN" lines + that specify how the test is to be run. So, each test script must contain + RUN lines if it is to do anything. If there are no RUN lines, the + <tt>llvm-runtests</tt> function will issue an error and the test will + fail.</p> + + <p>RUN lines are specified in the comments of the test program using the + keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline) + to execute. Together, these lines form the "script" that + <tt>llvm-runtests</tt> executes to run the test case. The syntax of the + RUN lines is similar to a shell's syntax for pipelines including I/O + redirection and variable substitution. However, even though these lines + may <i>look</i> like a shell script, they are not. RUN lines are interpreted + directly by the Tcl <tt>exec</tt> command. They are never executed by a + shell. Consequently the syntax differs from normal shell script syntax in a + few ways. You can specify as many RUN lines as needed.</p> + + <p>Each RUN line is executed on its own, distinct from other lines unless + its last character is <tt>\</tt>. This continuation character causes the RUN + line to be concatenated with the next one. In this way you can build up long + pipelines of commands without making huge line lengths. The lines ending in + <tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is + found. This concatenated set of RUN lines then constitutes one execution. + Tcl will substitute variables and arrange for the pipeline to be executed. If + any process in the pipeline fails, the entire line (and test case) fails too. + </p> + + <p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p> + +<div class="doc_code"> +<pre> +; RUN: llvm-as < %s | llvm-dis > %t1 +; RUN: llvm-dis < %s.bc-13 > %t2 +; RUN: diff %t1 %t2 +</pre> +</div> + + <p>As with a Unix shell, the RUN: lines permit pipelines and I/O redirection + to be used. However, the usage is slightly different than for Bash. To check + what's legal, see the documentation for the + <a href="http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2">Tcl exec</a> + command and the + <a href="http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html">tutorial</a>. + The major differences are:</p> + <ul> + <li>You can't do <tt>2>&1</tt>. That will cause Tcl to write to a + file named <tt>&1</tt>. Usually this is done to get stderr to go through + a pipe. You can do that in tcl with <tt>|&</tt> so replace this idiom: + <tt>... 2>&1 | grep</tt> with <tt>... |& grep</tt></li> + <li>You can only redirect to a file, not to another descriptor and not from + a here document.</li> + <li>tcl supports redirecting to open files with the @ syntax but you + shouldn't use that here.</li> + </ul> + + <p>There are some quoting rules that you must pay attention to when writing + your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any + ' or " so they will get passed to the invoked program. For example:</p> + +<div class="doc_code"> +<pre> +... | grep 'find this string' +</pre> +</div> + + <p>This will fail because the ' characters are passed to grep. This would + instruction grep to look for <tt>'find</tt> in the files <tt>this</tt> and + <tt>string'</tt>. To avoid this use curly braces to tell Tcl that it should + treat everything enclosed as one value. So our example would become:</p> + +<div class="doc_code"> +<pre> +... | grep {find this string} +</pre> +</div> + + <p>Additionally, the characters <tt>[</tt> and <tt>]</tt> are treated + specially by Tcl. They tell Tcl to interpret the content as a command to + execute. Since these characters are often used in regular expressions this can + have disastrous results and cause the entire test run in a directory to fail. + For example, a common idiom is to look for some basicblock number:</p> + +<div class="doc_code"> +<pre> +... | grep bb[2-8] +</pre> +</div> + + <p>This, however, will cause Tcl to fail because its going to try to execute + a program named "2-8". Instead, what you want is this:</p> + +<div class="doc_code"> +<pre> +... | grep {bb\[2-8\]} +</pre> +</div> + + <p>Finally, if you need to pass the <tt>\</tt> character down to a program, + then it must be doubled. This is another Tcl special character. So, suppose + you had: + +<div class="doc_code"> +<pre> +... | grep 'i32\*' +</pre> +</div> + + <p>This will fail to match what you want (a pointer to i32). First, the + <tt>'</tt> do not get stripped off. Second, the <tt>\</tt> gets stripped off + by Tcl so what grep sees is: <tt>'i32*'</tt>. That's not likely to match + anything. To resolve this you must use <tt>\\</tt> and the <tt>{}</tt>, like + this:</p> + +<div class="doc_code"> +<pre> +... | grep {i32\\*} +</pre> +</div> + +<p>If your system includes GNU <tt>grep</tt>, make sure +that <tt>GREP_OPTIONS</tt> is not set in your environment. Otherwise, +you may get invalid results (both false positives and false +negatives).</p> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="FileCheck">The FileCheck utility</a></div> +<!-- _______________________________________________________________________ --> + +<div class="doc_text"> + +<p>A powerful feature of the RUN: lines is that it allows any arbitrary commands + to be executed as part of the test harness. While standard (portable) unix + tools like 'grep' work fine on run lines, as you see above, there are a lot + of caveats due to interaction with Tcl syntax, and we want to make sure the + run lines are portable to a wide range of systems. Another major problem is + that grep is not very good at checking to verify that the output of a tools + contains a series of different output in a specific order. The FileCheck + tool was designed to help with these problems.</p> + +<p>FileCheck (whose basic command line arguments are described in <a + href="http://llvm.org/cmds/FileCheck.html">the FileCheck man page</a> is + designed to read a file to check from standard input, and the set of things + to verify from a file specified as a command line argument. A simple example + of using FileCheck from a RUN line looks like this:</p> + +<div class="doc_code"> +<pre> +; RUN: llvm-as < %s | llc -march=x86-64 | <b>FileCheck %s</b> +</pre> +</div> + +<p>This syntax says to pipe the current file ("%s") into llvm-as, pipe that into +llc, then pipe the output of llc into FileCheck. This means that FileCheck will +be verifying its standard input (the llc output) against the filename argument +specified (the original .ll file specified by "%s"). To see how this works, +lets look at the rest of the .ll file (after the RUN line):</p> + +<div class="doc_code"> +<pre> +define void @sub1(i32* %p, i32 %v) { +entry: +; <b>CHECK: sub1:</b> +; <b>CHECK: subl</b> + %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v) + ret void +} + +define void @inc4(i64* %p) { +entry: +; <b>CHECK: inc4:</b> +; <b>CHECK: incq</b> + %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1) + ret void +} +</pre> +</div> + +<p>Here you can see some "CHECK:" lines specified in comments. Now you can see +how the file is piped into llvm-as, then llc, and the machine code output is +what we are verifying. FileCheck checks the machine code output to verify that +it matches what the "CHECK:" lines specify.</p> + +<p>The syntax of the CHECK: lines is very simple: they are fixed strings that +must occur in order. FileCheck defaults to ignoring horizontal whitespace +differences (e.g. a space is allowed to match a tab) but otherwise, the contents +of the CHECK: line is required to match some thing in the test file exactly.</p> + +<p>One nice thing about FileCheck (compared to grep) is that it allows merging +test cases together into logical groups. For example, because the test above +is checking for the "sub1:" and "inc4:" labels, it will not match unless there +is a "subl" in between those labels. If it existed somewhere else in the file, +that would not count: "grep subl" matches if subl exists anywhere in the +file.</p> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsubsection"><a +name="FileCheck-check-prefix">The FileCheck -check-prefix option</a></div> + +<div class="doc_text"> + +<p>The FileCheck -check-prefix option allows multiple test configurations to be +driven from one .ll file. This is useful in many circumstances, for example, +testing different architectural variants with llc. Here's a simple example:</p> + +<div class="doc_code"> +<pre> +; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \ +; RUN: | <b>FileCheck %s -check-prefix=X32</b> +; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \ +; RUN: | <b>FileCheck %s -check-prefix=X64</b> + +define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind { + %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1 + ret <4 x i32> %tmp1 +; <b>X32:</b> pinsrd_1: +; <b>X32:</b> pinsrd $1, 4(%esp), %xmm0 + +; <b>X64:</b> pinsrd_1: +; <b>X64:</b> pinsrd $1, %edi, %xmm0 +} +</pre> +</div> + +<p>In this case, we're testing that we get the expected code generation with +both 32-bit and 64-bit code generation.</p> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsubsection"><a +name="FileCheck-CHECK-NEXT">The "CHECK-NEXT:" directive</a></div> + +<div class="doc_text"> + +<p>Sometimes you want to match lines and would like to verify that matches +happen on exactly consequtive lines with no other lines in between them. In +this case, you can use CHECK: and CHECK-NEXT: directives to specify this. If +you specified a custom check prefix, just use "<PREFIX>-NEXT:". For +example, something like this works as you'd expect:</p> + +<div class="doc_code"> +<pre> +define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) { + %tmp3 = load <2 x double>* %A, align 16 + %tmp7 = insertelement <2 x double> undef, double %B, i32 0 + %tmp9 = shufflevector <2 x double> %tmp3, + <2 x double> %tmp7, + <2 x i32> < i32 0, i32 2 > + store <2 x double> %tmp9, <2 x double>* %r, align 16 + ret void + +; <b>CHECK:</b> t2: +; <b>CHECK:</b> movl 8(%esp), %eax +; <b>CHECK-NEXT:</b> movapd (%eax), %xmm0 +; <b>CHECK-NEXT:</b> movhpd 12(%esp), %xmm0 +; <b>CHECK-NEXT:</b> movl 4(%esp), %eax +; <b>CHECK-NEXT:</b> movapd %xmm0, (%eax) +; <b>CHECK-NEXT:</b> ret +} +</pre> +</div> + +<p>CHECK-NEXT: directives reject the input unless there is exactly one newline +between it an the previous directive. A CHECK-NEXT cannot be the first +directive in a file.</p> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsubsection"><a +name="FileCheck-CHECK-NOT">The "CHECK-NOT:" directive</a></div> + +<div class="doc_text"> + +<p>The CHECK-NOT: directive is used to verify that a string doesn't occur +between two matches (or the first match and the beginning of the file). For +example, to verify that a load is removed by a transformation, a test like this +can be used:</p> + +<div class="doc_code"> +<pre> +define i8 @coerce_offset0(i32 %V, i32* %P) { + store i32 %V, i32* %P + + %P2 = bitcast i32* %P to i8* + %P3 = getelementptr i8* %P2, i32 2 + + %A = load i8* %P3 + ret i8 %A +; <b>CHECK:</b> @coerce_offset0 +; <b>CHECK-NOT:</b> load +; <b>CHECK:</b> ret i8 +} +</pre> +</div> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsubsection"><a +name="FileCheck-Matching">FileCheck Pattern Matching Syntax</a></div> + +<div class="doc_text"> + +<p>The CHECK: and CHECK-NOT: directives both take a pattern to match. For most +uses of FileCheck, fixed string matching is perfectly sufficient. For some +things, a more flexible form of matching is desired. To support this, FileCheck +allows you to specify regular expressions in matching strings, surrounded by +double braces: <b>{{yourregex}}</b>. Because we want to use fixed string +matching for a majority of what we do, FileCheck has been designed to support +mixing and matching fixed string matching with regular expressions. This allows +you to write things like this:</p> + +<div class="doc_code"> +<pre> +; CHECK: movhpd <b>{{[0-9]+}}</b>(%esp), <b>{{%xmm[0-7]}}</b> +</pre> +</div> + +<p>In this case, any offset from the ESP register will be allowed, and any xmm +register will be allowed.</p> + +<p>Because regular expressions are enclosed with double braces, they are +visually distinct, and you don't need to use escape characters within the double +braces like you would in C. In the rare case that you want to match double +braces explicitly from the input, you can use something ugly like +<b>{{[{][{]}}</b> as your pattern.</p> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsubsection"><a +name="FileCheck-Variables">FileCheck Variables</a></div> + +<div class="doc_text"> + +<p>It is often useful to match a pattern and then verify that it occurs again +later in the file. For codegen tests, this can be useful to allow any register, +but verify that that register is used consistently later. To do this, FileCheck +allows named variables to be defined and substituted into patterns. Here is a +simple example:</p> + +<div class="doc_code"> +<pre> +; CHECK: test5: +; CHECK: notw <b>[[REGISTER:%[a-z]+]]</b> +; CHECK: andw {{.*}}<b>[[REGISTER]]</b> +</pre> +</div> + +<p>The first check line matches a regex (<tt>%[a-z]+</tt>) and captures it into +the variables "REGISTER". The second line verifies that whatever is in REGISTER +occurs later in the file after an "andw". FileCheck variable references are +always contained in <tt>[[ ]]</tt> pairs, are named, and their names can be +formed with the regex "<tt>[a-zA-Z][a-zA-Z0-9]*</tt>". If a colon follows the +name, then it is a definition of the variable, if not, it is a use.</p> + +<p>FileCheck variables can be defined multiple times, and uses always get the +latest value. Note that variables are all read at the start of a "CHECK" line +and are all defined at the end. This means that if you have something like +"<tt>CHECK: [[XYZ:.*]]x[[XYZ]]</tt>" that the check line will read the previous +value of the XYZ variable and define a new one after the match is performed. If +you need to do something like this you can probably take advantage of the fact +that FileCheck is not actually line-oriented when it matches, this allows you to +define two separate CHECK lines that match on the same line. +</p> + +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="dgvars">Variables and +substitutions</a></div> +<!-- _______________________________________________________________________ --> +<div class="doc_text"> + <p>With a RUN line there are a number of substitutions that are permitted. In + general, any Tcl variable that is available in the <tt>substitute</tt> + function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line. + To make a substitution just write the variable's name preceded by a $. + Additionally, for compatibility reasons with previous versions of the test + library, certain names can be accessed with an alternate syntax: a % prefix. + These alternates are deprecated and may go away in a future version. + </p> + <p>Here are the available variable names. The alternate syntax is listed in + parentheses.</p> + + <dl style="margin-left: 25px"> + <dt><b>$test</b> (%s)</dt> + <dd>The full path to the test case's source. This is suitable for passing + on the command line as the input to an llvm tool.</dd> + + <dt><b>$srcdir</b></dt> + <dd>The source directory from where the "<tt>make check</tt>" was run.</dd> + + <dt><b>objdir</b></dt> + <dd>The object directory that corresponds to the <tt>$srcdir</tt>.</dd> + + <dt><b>subdir</b></dt> + <dd>A partial path from the <tt>test</tt> directory that contains the + sub-directory that contains the test source being executed.</dd> + + <dt><b>srcroot</b></dt> + <dd>The root directory of the LLVM src tree.</dd> + + <dt><b>objroot</b></dt> + <dd>The root directory of the LLVM object tree. This could be the same + as the srcroot.</dd> + + <dt><b>path</b><dt> + <dd>The path to the directory that contains the test case source. This is + for locating any supporting files that are not generated by the test, but + used by the test.</dd> + + <dt><b>tmp</b></dt> + <dd>The path to a temporary file name that could be used for this test case. + The file name won't conflict with other test cases. You can append to it if + you need multiple temporaries. This is useful as the destination of some + redirected output.</dd> + + <dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt> + <dd>The directory where the LLVM libraries are located.</dd> + + <dt><b>target_triplet</b> (%target_triplet)</dt> + <dd>The target triplet that corresponds to the current host machine (the one + running the test cases). This should probably be called "host".<dd> + + <dt><b>llvmgcc</b> (%llvmgcc)</dt> + <dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the + configured LLVM environment</dd> + + <dt><b>llvmgxx</b> (%llvmgxx)</dt> + <dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the + configured LLVM environment</dd> + + <dt><b>gccpath</b></dt> + <dd>The full path to the C compiler used to <i>build </i> LLVM. Note that + this might not be gcc.</dd> + + <dt><b>gxxpath</b></dt> + <dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that + this might not be g++.</dd> + + <dt><b>compile_c</b> (%compile_c)</dt> + <dd>The full command line used to compile LLVM C source code. This has all + the configured -I, -D and optimization options.</dd> + + <dt><b>compile_cxx</b> (%compile_cxx)</dt> + <dd>The full command used to compile LLVM C++ source code. This has + all the configured -I, -D and optimization options.</dd> + + <dt><b>link</b> (%link)</dt> + <dd>This full link command used to link LLVM executables. This has all the + configured -I, -L and -l options.</dd> + + <dt><b>shlibext</b> (%shlibext)</dt> + <dd>The suffix for the host platforms share library (dll) files. This + includes the period as the first character.</dd> + </dl> + <p>To add more variables, two things need to be changed. First, add a line in + the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will + "set" the variable as a global in the site.exp file. Second, in the + <tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name + to the list of "global" declarations at the beginning of the proc. That's it, + the variable can then be used in test scripts.</p> +</div> + +<!-- _______________________________________________________________________ --> +<div class="doc_subsection"><a name="dgfeatures">Other Features</a></div> +<!-- _______________________________________________________________________ --> +<div class="doc_text"> + <p>To make RUN line writing easier, there are several shell scripts located + in the <tt>llvm/test/Scripts</tt> directory. This directory is in the PATH + when running tests, so you can just call these scripts using their name. For + example:</p> + <dl> + <dt><b>ignore</b></dt> + <dd>This script runs its arguments and then always returns 0. This is useful + in cases where the test needs to cause a tool to generate an error (e.g. to + check the error output). However, any program in a pipeline that returns a + non-zero result will cause the test to fail. This script overcomes that + issue and nicely documents that the test case is purposefully ignoring the + result code of the tool</dd> + + <dt><b>not</b></dt> + <dd>This script runs its arguments and then inverts the result code from + it. Zero result codes become 1. Non-zero result codes become 0. This is + useful to invert the result of a grep. For example "not grep X" means + succeed only if you don't find X in the input.</dd |