diff options
| author | Dmitri Gribenko <gribozavr@gmail.com> | 2012-12-15 20:41:17 +0000 |
|---|---|---|
| committer | Dmitri Gribenko <gribozavr@gmail.com> | 2012-12-15 20:41:17 +0000 |
| commit | 5cc0580c6c6e4f05bdff1a53c5ca9f088393f072 (patch) | |
| tree | 0e9f07003c2a12e1c5a1882fac8f0a375b5ecde1 /docs/InternalsManual.html | |
| parent | 1228d66b5e866c88f96a3d23866f84389dd47e6c (diff) | |
Documentation: convert InternalsManual.html to reST
Patch by Anastasi Voitova with with small fixes by me.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@170275 91177308-0d34-0410-b5e6-96231b3b80d8
Diffstat (limited to 'docs/InternalsManual.html')
| -rw-r--r-- | docs/InternalsManual.html | 2019 |
1 files changed, 0 insertions, 2019 deletions
diff --git a/docs/InternalsManual.html b/docs/InternalsManual.html deleted file mode 100644 index 57f06316b1..0000000000 --- a/docs/InternalsManual.html +++ /dev/null @@ -1,2019 +0,0 @@ -<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" - "http://www.w3.org/TR/html4/strict.dtd"> -<html> -<head> -<title>"Clang" CFE Internals Manual</title> -<link type="text/css" rel="stylesheet" href="../menu.css"> -<link type="text/css" rel="stylesheet" href="../content.css"> -<style type="text/css"> -td { - vertical-align: top; -} -</style> -</head> -<body> - -<!--#include virtual="../menu.html.incl"--> - -<div id="content"> - -<h1>"Clang" CFE Internals Manual</h1> - -<ul> -<li><a href="#intro">Introduction</a></li> -<li><a href="#libsupport">LLVM Support Library</a></li> -<li><a href="#libbasic">The Clang 'Basic' Library</a> - <ul> - <li><a href="#Diagnostics">The Diagnostics Subsystem</a></li> - <li><a href="#SourceLocation">The SourceLocation and SourceManager - classes</a></li> - <li><a href="#SourceRange">SourceRange and CharSourceRange</a></li> - </ul> -</li> -<li><a href="#libdriver">The Driver Library</a> -</li> -<li><a href="#pch">Precompiled Headers</a> -<li><a href="#libfrontend">The Frontend Library</a> -</li> -<li><a href="#liblex">The Lexer and Preprocessor Library</a> - <ul> - <li><a href="#Token">The Token class</a></li> - <li><a href="#Lexer">The Lexer class</a></li> - <li><a href="#AnnotationToken">Annotation Tokens</a></li> - <li><a href="#TokenLexer">The TokenLexer class</a></li> - <li><a href="#MultipleIncludeOpt">The MultipleIncludeOpt class</a></li> - </ul> -</li> -<li><a href="#libparse">The Parser Library</a> -</li> -<li><a href="#libast">The AST Library</a> - <ul> - <li><a href="#Type">The Type class and its subclasses</a></li> - <li><a href="#QualType">The QualType class</a></li> - <li><a href="#DeclarationName">Declaration names</a></li> - <li><a href="#DeclContext">Declaration contexts</a> - <ul> - <li><a href="#Redeclarations">Redeclarations and Overloads</a></li> - <li><a href="#LexicalAndSemanticContexts">Lexical and Semantic - Contexts</a></li> - <li><a href="#TransparentContexts">Transparent Declaration Contexts</a></li> - <li><a href="#MultiDeclContext">Multiply-Defined Declaration Contexts</a></li> - </ul> - </li> - <li><a href="#CFG">The CFG class</a></li> - <li><a href="#Constants">Constant Folding in the Clang AST</a></li> - </ul> -</li> -<li><a href="#Howtos">Howto guides</a> - <ul> - <li><a href="#AddingAttributes">How to add an attribute</a></li> - <li><a href="#AddingExprStmt">How to add a new expression or statement</a></li> - </ul> -</li> -</ul> - - -<!-- ======================================================================= --> -<h2 id="intro">Introduction</h2> -<!-- ======================================================================= --> - -<p>This document describes some of the more important APIs and internal design -decisions made in the Clang C front-end. The purpose of this document is to -both capture some of this high level information and also describe some of the -design decisions behind it. This is meant for people interested in hacking on -Clang, not for end-users. The description below is categorized by -libraries, and does not describe any of the clients of the libraries.</p> - -<!-- ======================================================================= --> -<h2 id="libsupport">LLVM Support Library</h2> -<!-- ======================================================================= --> - -<p>The LLVM libsupport library provides many underlying libraries and -<a href="http://llvm.org/docs/ProgrammersManual.html">data-structures</a>, -including command line option processing, various containers and a system -abstraction layer, which is used for file system access.</p> - -<!-- ======================================================================= --> -<h2 id="libbasic">The Clang 'Basic' Library</h2> -<!-- ======================================================================= --> - -<p>This library certainly needs a better name. The 'basic' library contains a -number of low-level utilities for tracking and manipulating source buffers, -locations within the source buffers, diagnostics, tokens, target abstraction, -and information about the subset of the language being compiled for.</p> - -<p>Part of this infrastructure is specific to C (such as the TargetInfo class), -other parts could be reused for other non-C-based languages (SourceLocation, -SourceManager, Diagnostics, FileManager). When and if there is future demand -we can figure out if it makes sense to introduce a new library, move the general -classes somewhere else, or introduce some other solution.</p> - -<p>We describe the roles of these classes in order of their dependencies.</p> - - -<!-- ======================================================================= --> -<h3 id="Diagnostics">The Diagnostics Subsystem</h3> -<!-- ======================================================================= --> - -<p>The Clang Diagnostics subsystem is an important part of how the compiler -communicates with the human. Diagnostics are the warnings and errors produced -when the code is incorrect or dubious. In Clang, each diagnostic produced has -(at the minimum) a unique ID, an English translation associated with it, a <a -href="#SourceLocation">SourceLocation</a> to "put the caret", and a severity (e.g. -<tt>WARNING</tt> or <tt>ERROR</tt>). They can also optionally include a number -of arguments to the dianostic (which fill in "%0"'s in the string) as well as a -number of source ranges that related to the diagnostic.</p> - -<p>In this section, we'll be giving examples produced by the Clang command line -driver, but diagnostics can be <a href="#DiagnosticClient">rendered in many -different ways</a> depending on how the DiagnosticClient interface is -implemented. A representative example of a diagnostic is:</p> - -<pre> -t.c:38:15: error: invalid operands to binary expression ('int *' and '_Complex float') - <span style="color:darkgreen">P = (P-42) + Gamma*4;</span> - <span style="color:blue">~~~~~~ ^ ~~~~~~~</span> -</pre> - -<p>In this example, you can see the English translation, the severity (error), -you can see the source location (the caret ("^") and file/line/column info), -the source ranges "~~~~", arguments to the diagnostic ("int*" and "_Complex -float"). You'll have to believe me that there is a unique ID backing the -diagnostic :).</p> - -<p>Getting all of this to happen has several steps and involves many moving -pieces, this section describes them and talks about best practices when adding -a new diagnostic.</p> - -<!-- ============================= --> -<h4>The Diagnostic*Kinds.td files</h4> -<!-- ============================= --> - -<p>Diagnostics are created by adding an entry to one of the <tt> -clang/Basic/Diagnostic*Kinds.td</tt> files, depending on what library will -be using it. From this file, tblgen generates the unique ID of the diagnostic, -the severity of the diagnostic and the English translation + format string.</p> - -<p>There is little sanity with the naming of the unique ID's right now. Some -start with err_, warn_, ext_ to encode the severity into the name. Since the -enum is referenced in the C++ code that produces the diagnostic, it is somewhat -useful for it to be reasonably short.</p> - -<p>The severity of the diagnostic comes from the set {<tt>NOTE</tt>, -<tt>WARNING</tt>, <tt>EXTENSION</tt>, <tt>EXTWARN</tt>, <tt>ERROR</tt>}. The -<tt>ERROR</tt> severity is used for diagnostics indicating the program is never -acceptable under any circumstances. When an error is emitted, the AST for the -input code may not be fully built. The <tt>EXTENSION</tt> and <tt>EXTWARN</tt> -severities are used for extensions to the language that Clang accepts. This -means that Clang fully understands and can represent them in the AST, but we -produce diagnostics to tell the user their code is non-portable. The difference -is that the former are ignored by default, and the later warn by default. The -<tt>WARNING</tt> severity is used for constructs that are valid in the currently -selected source language but that are dubious in some way. The <tt>NOTE</tt> -level is used to staple more information onto previous diagnostics.</p> - -<p>These <em>severities</em> are mapped into a smaller set (the -Diagnostic::Level enum, {<tt>Ignored</tt>, <tt>Note</tt>, <tt>Warning</tt>, -<tt>Error</tt>, <tt>Fatal</tt> }) of output <em>levels</em> by the diagnostics -subsystem based on various configuration options. Clang internally supports a -fully fine grained mapping mechanism that allows you to map almost any -diagnostic to the output level that you want. The only diagnostics that cannot -be mapped are <tt>NOTE</tt>s, which always follow the severity of the previously -emitted diagnostic and <tt>ERROR</tt>s, which can only be mapped to -<tt>Fatal</tt> (it is not possible to turn an error into a warning, -for example).</p> - -<p>Diagnostic mappings are used in many ways. For example, if the user -specifies <tt>-pedantic</tt>, <tt>EXTENSION</tt> maps to <tt>Warning</tt>, if -they specify <tt>-pedantic-errors</tt>, it turns into <tt>Error</tt>. This is -used to implement options like <tt>-Wunused_macros</tt>, <tt>-Wundef</tt> etc. -</p> - -<p> -Mapping to <tt>Fatal</tt> should only be used for diagnostics that are -considered so severe that error recovery won't be able to recover sensibly from -them (thus spewing a ton of bogus errors). One example of this class of error -are failure to #include a file. -</p> - -<!-- ================= --> -<h4>The Format String</h4> -<!-- ================= --> - -<p>The format string for the diagnostic is very simple, but it has some power. -It takes the form of a string in English with markers that indicate where and -how arguments to the diagnostic are inserted and formatted. For example, here -are some simple format strings:</p> - -<pre> - "binary integer literals are an extension" - "format string contains '\\0' within the string body" - "more '<b>%%</b>' conversions than data arguments" - "invalid operands to binary expression (<b>%0</b> and <b>%1</b>)" - "overloaded '<b>%0</b>' must be a <b>%select{unary|binary|unary or binary}2</b> operator" - " (has <b>%1</b> parameter<b>%s1</b>)" -</pre> - -<p>These examples show some important points of format strings. You can use any - plain ASCII character in the diagnostic string except "%" without a problem, - but these are C strings, so you have to use and be aware of all the C escape - sequences (as in the second example). If you want to produce a "%" in the - output, use the "%%" escape sequence, like the third diagnostic. Finally, - Clang uses the "%...[digit]" sequences to specify where and how arguments to - the diagnostic are formatted.</p> - -<p>Arguments to the diagnostic are numbered according to how they are specified - by the C++ code that <a href="#producingdiag">produces them</a>, and are - referenced by <tt>%0</tt> .. <tt>%9</tt>. If you have more than 10 arguments - to your diagnostic, you are doing something wrong :). Unlike printf, there - is no requirement that arguments to the diagnostic end up in the output in - the same order as they are specified, you could have a format string with - <tt>"%1 %0"</tt> that swaps them, for example. The text in between the - percent and digit are formatting instructions. If there are no instructions, - the argument is just turned into a string and substituted in.</p> - -<p>Here are some "best practices" for writing the English format string:</p> - -<ul> -<li>Keep the string short. It should ideally fit in the 80 column limit of the - <tt>DiagnosticKinds.td</tt> file. This avoids the diagnostic wrapping when - printed, and forces you to think about the important point you are conveying - with the diagnostic.</li> -<li>Take advantage of location information. The user will be able to see the - line and location of the caret, so you don't need to tell them that the - problem is with the 4th argument to the function: just point to it.</li> -<li>Do not capitalize the diagnostic string, and do not end it with a - period.</li> -<li>If you need to quote something in the diagnostic string, use single - quotes.</li> -</ul> - -<p>Diagnostics should never take random English strings as arguments: you -shouldn't use <tt>"you have a problem with %0"</tt> and pass in things like -<tt>"your argument"</tt> or <tt>"your return value"</tt> as arguments. Doing -this prevents <a href="#translation">translating</a> the Clang diagnostics to -other languages (because they'll get random English words in their otherwise -localized diagnostic). The exceptions to this are C/C++ language keywords -(e.g. auto, const, mutable, etc) and C/C++ operators (<tt>/=</tt>). Note -that things like "pointer" and "reference" are not keywords. On the other -hand, you <em>can</em> include anything that comes from the user's source code, -including variable names, types, labels, etc. The 'select' format can be -used to achieve this sort of thing in a localizable way, see below.</p> - -<!-- ==================================== --> -<h4>Formatting a Diagnostic Argument</h4> -<!-- ==================================== --> - -<p>Arguments to diagnostics are fully typed internally, and come from a couple -different classes: integers, types, names, and random strings. Depending on -the class of the argument, it can be optionally formatted in different ways. -This gives the DiagnosticClient information about what the argument means -without requiring it to use a specific presentation (consider this MVC for -Clang :).</p> - -<p>Here are the different diagnostic argument formats currently supported by -Clang:</p> - -<table> -<tr><td colspan="2"><b>"s" format</b></td></tr> -<tr><td>Example:</td><td><tt>"requires %1 parameter%s1"</tt></td></tr> -<tr><td>Class:</td><td>Integers</td></tr> -<tr><td>Description:</td><td>This is a simple formatter for integers that is - useful when producing English diagnostics. When the integer is 1, it prints - as nothing. When the integer is not 1, it prints as "s". This allows some - simple grammatical forms to be to be handled correctly, and eliminates the - need to use gross things like <tt>"requires %1 parameter(s)"</tt>.</td></tr> - -<tr><td colspan="2"><b>"select" format</b></td></tr> -<tr><td>Example:</td><td><tt>"must be a %select{unary|binary|unary or binary}2 - operator"</tt></td></tr> -<tr><td>Class:</td><td>Integers</td></tr> -<tr><td>Description:</td><td><p>This format specifier is used to merge multiple - related diagnostics together into one common one, without requiring the - difference to be specified as an English string argument. Instead of - specifying the string, the diagnostic gets an integer argument and the - format string selects the numbered option. In this case, the "%2" value - must be an integer in the range [0..2]. If it is 0, it prints 'unary', if - it is 1 it prints 'binary' if it is 2, it prints 'unary or binary'. This - allows other language translations to substitute reasonable words (or entire - phrases) based on the semantics of the diagnostic instead of having to do - things textually.</p> - <p>The selected string does undergo formatting.</p></td></tr> - -<tr><td colspan="2"><b>"plural" format</b></td></tr> -<tr><td>Example:</td><td><tt>"you have %1 %plural{1:mouse|:mice}1 connected to - your computer"</tt></td></tr> -<tr><td>Class:</td><td>Integers</td></tr> -<tr><td>Description:</td><td><p>This is a formatter for complex plural forms. - It is designed to handle even the requirements of languages with very - complex plural forms, as many Baltic languages have. The argument consists - of a series of expression/form pairs, separated by ':', where the first form - whose expression evaluates to true is the result of the modifier.</p> - <p>An expression can be empty, in which case it is always true. See the - example at the top. Otherwise, it is a series of one or more numeric - conditions, separated by ','. If any condition matches, the expression - matches. Each numeric condition can take one of three forms.</p> - <ul> - <li>number: A simple decimal number matches if the argument is the same - as the number. Example: <tt>"%plural{1:mouse|:mice}4"</tt></li> - <li>range: A range in square brackets matches if the argument is within - the range. Then range is inclusive on both ends. Example: - <tt>"%plural{0:none|1:one|[2,5]:some|:many}2"</tt></li> - <li>modulo: A modulo operator is followed by a number, and - equals sign and either a number or a range. The tests are the - same as for plain - numbers and ranges, but the argument is taken modulo the number first. - Example: <tt>"%plural{%100=0:even hundred|%100=[1,50]:lower half|:everything - else}1"</tt></li> - </ul> - <p>The parser is very unforgiving. A syntax error, even whitespace, will - abort, as will a failure to match the argument against any - expression.</p></td></tr> - -<tr><td colspan="2"><b>"ordinal" format</b></td></tr> -<tr><td>Example:</td><td><tt>"ambiguity in %ordinal0 argument"</tt></td></tr> -<tr><td>Class:</td><td>Integers</td></tr> -<tr><td>Description:</td><td><p>This is a formatter which represents the - argument number as an ordinal: the value <tt>1</tt> becomes <tt>1st</tt>, - <tt>3</tt> becomes <tt>3rd</tt>, and so on. Values less than <tt>1</tt> - are not supported.</p> - <p>This formatter is currently hard-coded to use English ordinals.</p></td></tr> - -<tr><td colspan="2"><b>"objcclass" format</b></td></tr> -<tr><td>Example:</td><td><tt>"method %objcclass0 not found"</tt></td></tr> -<tr><td>Class:</td><td>DeclarationName</td></tr> -<tr><td>Description:</td><td><p>This is a simple formatter that indicates the - DeclarationName corresponds to an Objective-C class method selector. As - such, it prints the selector with a leading '+'.</p></td></tr> - -<tr><td colspan="2"><b>"objcinstance" format</b></td></tr> -<tr><td>Example:</td><td><tt>"method %objcinstance0 not found"</tt></td></tr> -<tr><td>Class:</td><td>DeclarationName</td></tr> -<tr><td>Description:</td><td><p>This is a simple formatter that indicates the - DeclarationName corresponds to an Objective-C instance method selector. As - such, it prints the selector with a leading '-'.</p></td></tr> - -<tr><td colspan="2"><b>"q" format</b></td></tr> -<tr><td>Example:</td><td><tt>"candidate found by name lookup is %q0"</tt></td></tr> -<tr><td>Class:</td><td>NamedDecl*</td></tr> -<tr><td>Description</td><td><p>This formatter indicates that the fully-qualified name of the declaration should be printed, e.g., "std::vector" rather than "vector".</p></td></tr> - -<tr><td colspan="2"><b>"diff" format</b></td></tr> -<tr><td>Example:</td><td><tt>"no known conversion %diff{from | to | }1,2"</tt></td></tr> -<tr><td>Class:</td><td>QualType</td></tr> -<tr><td>Description</td><td><p>This formatter takes two QualTypes and attempts to print a template difference between the two. If tree printing is off, the text inside the braces before the pipe is printed, with the formatted text replacing the $. If tree printing is on, the text after the pipe is printed and a type tree is printed after the diagnostic message. -</p></td></tr> - -</table> - -<p>It is really easy to add format specifiers to the Clang diagnostics system, -but they should be discussed before they are added. If you are creating a lot -of repetitive diagnostics and/or have an idea for a useful formatter, please -bring it up on the cfe-dev mailing list.</p> - -<!-- ===================================================== --> -<h4 id="producingdiag">Producing the Diagnostic</h4> -<!-- ===================================================== --> - -<p>Now that you've created the diagnostic in the DiagnosticKinds.td file, you -need to write the code that detects the condition in question and emits the -new diagnostic. Various components of Clang (e.g. the preprocessor, Sema, -etc) provide a helper function named "Diag". It creates a diagnostic and -accepts the arguments, ranges, and other information that goes along with -it.</p> - -<p>For example, the binary expression error comes from code like this:</p> - -<pre> - if (various things that are bad) - Diag(Loc, diag::err_typecheck_invalid_operands) - << lex->getType() << rex->getType() - << lex->getSourceRange() << rex->getSourceRange(); -</pre> - -<p>This shows that use of the Diag method: they take a location (a <a -href="#SourceLocation">SourceLocation</a> object) and a diagnostic enum value -(which matches the name from DiagnosticKinds.td). If the diagnostic takes -arguments, they are specified with the << operator: the first argument -becomes %0, the second becomes %1, etc. The diagnostic interface allows you to -specify arguments of many different types, including <tt>int</tt> and -<tt>unsigned</tt> for integer arguments, <tt>const char*</tt> and -<tt>std::string</tt> for string arguments, <tt>DeclarationName</tt> and -<tt>const IdentifierInfo*</tt> for names, <tt>QualType</tt> for types, etc. -SourceRanges are also specified with the << operator, but do not have a -specific ordering requirement.</p> - -<p>As you can see, adding and producing a diagnostic is pretty straightforward. -The hard part is deciding exactly what you need to say to help the user, picking -a suitable wording, and providing the information needed to format it correctly. -The good news is that the call site that issues a diagnostic should be -completely independent of how the diagnostic is formatted and in what language -it is rendered. -</p> - -<!-- ==================================================== --> -<h4 id="fix-it-hints">Fix-It Hints</h4> -<!-- ==================================================== --> - -<p>In some cases, the front end emits diagnostics when it is clear -that some small change to the source code would fix the problem. For -example, a missing semicolon at the end of a statement or a use of -deprecated syntax that is easily rewritten into a more modern form. -Clang tries very hard to emit the diagnostic and recover gracefully -in these and other cases.</p> - -<p>However, for these cases where the fix is obvious, the diagnostic -can be annotated with a hint (referred to as a "fix-it hint") that -describes how to change the code referenced by the diagnostic to fix -the problem. For example, it might add the missing semicolon at the -end of the statement or rewrite the use of a deprecated construct -into something more palatable. Here is one such example from the C++ -front end, where we warn about the right-shift operator changing -meaning from C++98 to C++11:</p> - -<pre> -test.cpp:3:7: warning: use of right-shift operator ('>>') in template argument will require parentheses in C++11 -A<100 >> 2> *a; - ^ - ( ) -</pre> - -<p>Here, the fix-it hint is suggesting that parentheses be added, -and showing exactly where those parentheses would be inserted into the -source code. The fix-it hints themselves describe what changes to make -to the source code in an abstract manner, which the text diagnostic -printer renders as a line of "insertions" below the caret line. <a -href="#DiagnosticClient">Other diagnostic clients</a> might choose -to render the code differently (e.g., as markup inline) or even give -the user the ability to automatically fix the problem.</p> - -<p>Fix-it hints on errors and warnings need to obey these rules:</p> - -<ul> -<li>Since they are automatically applied if <code>-Xclang -fixit</code> -is passed to the driver, they should only be used when it's very likely they -match the user's intent.</li> -<li>Clang must recover from errors as if the fix-it had been applied.</li> -</ul> - -<p>If a fix-it can't obey these rules, put the fix-it on a note. Fix-its on -notes are not applied automatically.</p> - -<p>All fix-it hints are described by the <code>FixItHint</code> class, -instances of which should be attached to the diagnostic using the -<< operator in the same way that highlighted source ranges and -arguments are passed to the diagnostic. Fix-it hints can be created -with one of three constructors:</p> - -<dl> - <dt><code>FixItHint::CreateInsertion(Loc, Code)</code></dt> - <dd>Specifies that the given <code>Code</code> (a string) should be inserted - before the source location <code>Loc</code>.</dd> - - <dt><code>FixItHint::CreateRemoval(Range)</code></dt> - <dd>Specifies that the code in the given source <code>Range</code> - should be removed.</dd> - - <dt><code>FixItHint::CreateReplacement(Range, Code)</code></dt> - <dd>Specifies that the code in the given source <code>Range</code> - should be removed, and replaced with the given <code>Code</code> string.</dd> -</dl> - -<!-- ============================================================= --> -<h4><a name="DiagnosticClient">The DiagnosticClient Interface</a></h4> -<!-- ============================================================= --> - -<p>Once code generates a diagnostic with all of the arguments and the rest of -the relevant information, Clang needs to know what to do with it. As previously -mentioned, the diagnostic machinery goes through some filtering to map a -severity onto a diagnostic level, then (assuming the diagnostic is not mapped to -"<tt>Ignore</tt>") it invokes an object that implements the DiagnosticClient -interface with the information.</p> - -<p>It is possible to implement this interface in many different ways. For -example, the normal Clang DiagnosticClient (named 'TextDiagnosticPrinter') turns -the arguments into strings (according to the various formatting rules), prints -out the file/line/column information and the string, then prints out the line of -code, the source ranges, and the caret. However, this behavior isn't required. -</p> - -<p>Another implementation of the DiagnosticClient interface is the -'TextDiagnosticBuffer' class, which is used when Clang is in -verify mode. -Instead of formatting and printing out the diagnostics, this implementation just -captures and remembers the diagnostics as they fly by. Then -verify compares -the list of produced diagnostics to the list of expected ones. If they disagree, -it prints out its own output. Full documentation for the -verify mode can be -found in the Clang API documentation for VerifyDiagnosticConsumer, <a -href="/doxygen/classclang_1_1VerifyDiagnosticConsumer.html#details">here</a>. -</p> - -<p>There are many other possible implementations of this interface, and this is -why we prefer diagnostics to pass down rich structured information in arguments. -For example, an HTML output might want declaration names be linkified to where -they come from in the source. Another example is that a GUI might let you click -on typedefs to expand them. This application would want to pass significantly -more information about types through to the GUI than a simple flat string. The -interface allows this to happen.</p> - -<!-- ====================================================== --> -<h4><a name="translation">Adding Translations to Clang</a></h4> -<!-- ====================================================== --> - -<p>Not possible yet! Diagnostic strings should be written in UTF-8, the client -can translate to the relevant code page if needed. Each translation completely -replaces the format string for the diagnostic.</p> - - -<!-- ======================================================================= --> -<h3 id="SourceLocation">The SourceLocation and SourceManager classes</h3> -<!-- ======================================================================= --> - -<p>Strangely enough, the SourceLocation class represents a location within the -source code of the program. Important design points include:</p> - -<ol> -<li>sizeof(SourceLocation) must be extremely small, as these are embedded into - many AST nodes and are passed around often. Currently it is 32 bits.</li> -<li>SourceLocation must be a simple value object that can be efficiently - copied.</li> -<li>We should be able to represent a source location for any byte of any input - file. This includes in the middle of tokens, in whitespace, in trigraphs, - etc.</li> -<li>A SourceLocation must encode the current #include stack that was active when - the location was processed. For example, if the location corresponds to a - token, it should contain the set of #includes active when the token was - lexed. This allows us to print the #include stack for a diagnostic.</li> -<li>SourceLocation must be able to describe macro expansions, capturing both - the ultimate instantiation point and the source of the original character - data.</li> -</ol> - -<p>In practice, the SourceLocation works together with the SourceManager class -to encode two pieces of information about a location: its spelling location -and its instantiation location. For most tokens, these will be the same. -However, for a macro expansion (or tokens that came from a _Pragma directive) -these will describe the location of the characters corresponding to the token -and the location where the token was used (i.e. the macro instantiation point -or the location of the _Pragma itself).</p> - -<p>The Clang front-end inherently depends on the location of a token being -tracked correctly. If it is ever incorrect, the front-end may get confused and -die. The reason for this is that the notion of the 'spelling' of a Token in -Clang depends on being able to find the original input characters for the token. -This concept maps directly to the "spelling location" for the token.</p> - - -<!-- ======================================================================= --> -<h3 id="SourceRange">SourceRange and CharSourceRange</h3> -<!-- ======================================================================= --> -<!-- mostly taken from - http://lists.cs.uiuc.edu/pipermail/cfe-dev/2010-August/010595.html --> - -<p>Clang represents most source ranges by [first, last], where first and last -each point to the beginning of their respective tokens. For example -consider the SourceRange of the following statement:</p> -<pre> -x = foo + bar; -^first ^last -</pre> - -<p>To map from this representation to a character-based -representation, the 'last' location needs to be adjusted to point to -(or past) the end of that token with either -<code>Lexer::MeasureTokenLength()</code> or -<code>Lexer::getLocForEndOfToken()</code>. For the rare cases -where character-level source ranges information is needed we use -the <code>CharSourceRange</code> class.</p> - - -<!-- ======================================================================= --> -<h2 id="libdriver">The Driver Library</h2> -<!-- ======================================================================= --> - -<p>The clang Driver and library are documented <a -href="DriverInternals.html">here</a>.<p> - -<!-- ======================================================================= --> -<h2 id="pch">Precompiled Headers</h2> -<!-- ======================================================================= --> - -<p>Clang supports two implementations of precompiled headers. The - default implementation, precompiled headers (<a - href="PCHInternals.html">PCH</a>) uses a serialized representation - of Clang's internal data structures, encoded with the <a - href="http://llvm.org/docs/BitCodeFormat.html">LLVM bitstream - format</a>. Pretokenized headers (<a - href="PTHInternals.html">PTH</a>), on the other hand, contain a - serialized representation of the tokens encountered when - preprocessing a header (and anything that header includes).</p> - - -<!-- ======================================================================= --> -<h2 id="libfrontend">The Frontend Library</h2> -<!-- ======================================================================= --> - -<p>The Frontend library contains functionality useful for building -tools on top of the clang libraries, for example several methods for -outputting diagnostics.</p> - -<!-- ======================================================================= --> -<h2 id="liblex">The Lexer and Preprocessor Library</h2> -<!-- ======================================================================= --> - -<p>The Lexer library contains several tightly-connected classes that are involved -with the nasty process of lexing and preprocessing C source code. The main -interface to this library for outside clients is the large <a -href="#Preprocessor">Preprocessor</a> class. -It contains the various pieces of state that are required to coherently read -tokens out of a translation unit.</p> - -<p>The core interface to the Preprocessor object (once it is set up) is the -Preprocessor::Lex method, which returns the next <a href="#Token">Token</a> from -the preprocessor stream. There are two types of token providers that the -preprocessor is capable of reading from: a buffer lexer (provided by the <a -href="#Lexer">Lexer</a> class) and a buffered token stream (provided by the <a -href="#TokenLexer">TokenLexer</a> class). - - -<!-- ======================================================================= --> -<h3 id="Token">The Token class</h3> -<!-- ======================================================================= --> - -<p>The Token class is used to represent a single lexed token. Tokens are -intended to be used by the lexer/preprocess and parser libraries, but are not -intended to live beyond them (for example, they should not live in the ASTs).<p> - -<p>Tokens most often live on the stack (or some other location that is efficient -to access) as the parser is running, but occasionally do get buffered up. For -example, macro definitions are stored as a series of tokens, and the C++ -front-end periodically needs to buffer tokens up for tentative parsing and -various pieces of look-ahead. As such, the size of a Token matter. On a 32-bit -system, sizeof(Token) is currently 16 bytes.</p> - -<p>Tokens occur in two forms: "<a href="#AnnotationToken">Annotation -Tokens</a>" and normal tokens. Normal tokens are those returned by the lexer, -annotation tokens represent semantic information and are produced by the parser, -replacing normal tokens in the token stream. Normal tokens contain the -following information:</p> - -<ul> -<li><b>A SourceLocation</b> - This indicates the location of the start of the -token.</li> - -<li><b>A length</b> - This stores the length of the token as stored in the -SourceBuffer. For tokens that include them, this length includes trigraphs and -escaped newlines which are ignored by later phases of the compiler. By pointing -into the original source buffer, it is always possible to get the original -spelling of a token completely accurately.</li> - -<li><b>IdentifierInfo</b> - If a token takes the form of an identifier, and if -identifier lookup was enabled when the token was lexed (e.g. the lexer was not -reading in 'raw' mode) this contains a pointer to the unique hash value for the -identifier. Because the lookup happens before keyword identification, this -field is set even for language keywords like 'for'.</li> - -<li><b>TokenKind</b> - This indicates the kind of token as classified by the -lexer. This includes things like <tt>tok::starequal</tt> (for the "*=" -operator), <tt>tok::ampamp</tt> for the "&&" token, and keyword values -(e.g. <tt>tok::kw_for</tt>) for identifiers that correspond to keywords. Note -that some tokens can be spelled multiple ways. For example, C++ supports -"operator keywords", where things like "and" are treated exactly like the -"&&" operator. In these cases, the kind value is set to -<tt>tok::ampamp</tt>, which is good for the parser, which doesn't have to -consider both forms. For something that cares about which form is used (e.g. -the preprocessor 'stringize' operator) the spelling indicates the original -form.</li> - -<li><b>Flags</b> - There are currently four flags tracked by the -lexer/preprocessor system on a per-token basis: - - <ol> - <li><b>StartOfLine</b> - This was the first token that occurred on its input - source line.</li> - <li><b>LeadingSpace</b> - There was a space character either immediately - before the token or transitively before the token as it was expanded - through a macro. The definition of this flag is very closely defined by - the stringizing requirements of the preprocessor.</li> - <li><b>DisableExpand</b> - This flag is used internally to the preprocessor to - represent identifier tokens which have macro expansion disabled. This - prevents them from being considered as candidates for macro expansion ever - in the future.</li> - <li><b>NeedsCleaning</b> - This flag is set if the original spelling for the - token includes a trigraph or escaped newline. Since this is uncommon, - many pieces of code can fast-path on tokens that did not need cleaning. - </ol> -</li> -</ul> - -<p>One interesting (and somewhat unusual) aspect of normal tokens is that they -don't contain any semantic information about the lexed value. For example, if -the token was a pp-number token, we do not represent the value of the number -that was lexed (this is left for later pieces of code to decide). Additionally, |