aboutsummaryrefslogtreecommitdiff
path: root/include/clang/Lex/Lexer.h
AgeCommit message (Collapse)Author
2013-03-25Correct indentation. No functionality change.Nick Lewycky
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177916 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-21Preprocessor: preserve whitespace in -traditional-cpp mode.Jordan Rose
Note that unlike GNU cpp we currently do not preserve whitespace in macros (even in -traditional-cpp mode). <rdar://problem/12897179> git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@175778 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-24Handle universal character names and Unicode characters outside of literals.Jordan Rose
This is a missing piece for C99 conformance. This patch handles UCNs by adding a '\\' case to LexTokenInternal and LexIdentifier -- if we see a backslash, we tentatively try to read in a UCN. If the UCN is not syntactically well-formed, we fall back to the old treatment: a backslash followed by an identifier beginning with 'u' (or 'U'). Because the spelling of an identifier with UCNs still has the UCN in it, we need to convert that to UTF-8 in Preprocessor::LookUpIdentifierInfo. Of course, valid code that does *not* use UCNs will see only a very minimal performance hit (checks after each identifier for non-ASCII characters, checks when converting raw_identifiers to identifiers that they do not contain UCNs, and checks when getting the spelling of an identifier that it does not contain a UCN). This patch also adds basic support for actual UTF-8 in the source. This is treated almost exactly the same as UCNs except that we consider stray Unicode characters to be mistakes and offer a fixit to remove them. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@173369 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-07Pull the bulk of Lexer::MeasureTokenLength() out into a new function,Argyrios Kyrtzidis
Lexer::getRawToken(). No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@171771 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04Sort #include lines for all files under include/...Chandler Carruth
This is a simpler sort, entirely automatic with the help of llvm/utils/sort_includes.py -- no manual edits here. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@169238 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-11s/BCPLComment/LineComment/Nico Weber
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167690 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-06Delete an outdated comment.Nico Weber
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167441 91177308-0d34-0410-b5e6-96231b3b80d8
2012-09-15Use LLVM_DELETED_FUNCTION in place of 'DO NOT IMPLEMENT' comments.Dmitri Gribenko
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163983 91177308-0d34-0410-b5e6-96231b3b80d8
2012-09-07Moved back getCharAndSizeNoWarn to public area.Abramo Bagnara
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@163408 91177308-0d34-0410-b5e6-96231b3b80d8
2012-08-31Make a bunch of methods on Lexer private.Eli Friedman
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162970 91177308-0d34-0410-b5e6-96231b3b80d8
2012-08-28Fix a -Wdocumentation warning.Dmitri Gribenko
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162721 91177308-0d34-0410-b5e6-96231b3b80d8
2012-08-23Fix a bunch of -Wdocumentation warnings.Dmitri Gribenko
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162452 91177308-0d34-0410-b5e6-96231b3b80d8
2012-06-17Documentation cleanup: escaping #define in Doxygen commentsJames Dennett
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158629 91177308-0d34-0410-b5e6-96231b3b80d8
2012-06-08Correct method name in comment: from LexRawToken to LexFromRawLexer, accordingDmitri Gribenko
to a change done long ago in r57393. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158243 91177308-0d34-0410-b5e6-96231b3b80d8
2012-06-07Insert a space if necessary when suggesting CFBridgingRetain/Release.Jordan Rose
This was a problem for people who write 'return(result);' Also fix ARCMT's corresponding code, though there's no test case for this because implicit casts like this are rejected by the migrator for being ambiguous, and explicit casts have no problem. <rdar://problem/11577346> git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158130 91177308-0d34-0410-b5e6-96231b3b80d8
2012-05-18Lexer::ReadToEndOfLine: Only build the string if it's actually used and do ↵Benjamin Kramer
so in a less malloc-intensive way. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@157064 91177308-0d34-0410-b5e6-96231b3b80d8
2012-05-04Add more comments for Lexer::makeFileCharRange.Argyrios Kyrtzidis
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@156188 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-11Unify naming of LangOptions variable/get function across the Clang stack ↵David Blaikie
(Lex to AST). The member variable is always "LangOpts" and the member function is always "getLangOpts". Reviewed by Chris Lattner git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152536 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-05Lexing support for user-defined literals. Currently these lex as the same tokenRichard Smith
kinds as the underlying string literals, and we silently drop the ud-suffix; those issues will be fixed by subsequent patches. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152012 91177308-0d34-0410-b5e6-96231b3b80d8
2012-02-03Change Lexer::makeFileCharRange() to have it accept a CharSourceRangeArgyrios Kyrtzidis
instead of a SourceRange, and handle the case where the range is a char (not token) range. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149677 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-19Introduce Lexer::getSourceText() that returns a string for the sourceArgyrios Kyrtzidis
that the given source range encompasses. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148481 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-19Introduce Lexer::makeFileCharRange() that accepts a token source rangeArgyrios Kyrtzidis
and returns a character range with file locations. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148480 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-19For Lexer's isAt[Start/End]OfMacroExpansion add an out parameter for the macroArgyrios Kyrtzidis
start/end location. It is commonly needed after calling the function; with this way we avoid recalculating it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148479 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-18Refactor: Pull getImmediateMacroName() out of DiagnosticRenderer andAnna Zaks
into Lexer and Preprocessor; making it widely available. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148410 91177308-0d34-0410-b5e6-96231b3b80d8
2011-12-20Unweaken vtables as per ↵David Blaikie
http://llvm.org/docs/CodingStandards.html#ll_virtual_anch git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@146959 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-12Handle Perforce-style conflict markers like normal conflict markers. PerforceRichard Smith
swaps over the <<<< and >>>> markers, and uses shorter markers than traditional tools. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141751 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-25Rename Diagnostic to DiagnosticsEngine as per issue 5397David Blaikie
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140478 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-04Support code-completion for C++ inline methods and ObjC buffering methods.Argyrios Kyrtzidis
Previously we would cut off the source file buffer at the code-completion point; this impeded code-completion inside C++ inline methods and, recently, with buffering ObjC methods. Have the code-completion inserted into the source buffer so that it can be buffered along with a method body. When we actually hit the code-completion point the cut-off lexing or parsing. Fixes rdar://10056932&8319466 git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139086 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-25Make Lexer::ComputePreamble accept a LangOptions parameter, otherwise it may beArgyrios Kyrtzidis
out-of-sync how a file is compiled. Patch by Matthias Kleine! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138580 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-11Add support for C++0x raw string literals.Craig Topper
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137298 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-27Add a utility function to the Lexer, which makes it easier to find a token ↵Anna Zaks
after the given location. (It is a generalized version of trans::findLocationAfterSemi from ArcMigrate, which will be changed to use the Lexer utility). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136268 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-27Add support for C++0x unicode string and character literals, from Craig Topper!Douglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136210 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-20now that we have a centralized place to do so, add some using declarations forChris Lattner
some common llvm types: stringref and smallvector. This cleans up the codebase quite a bit. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135576 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-14Convert terminology in the Lexer from 'instantiate' and variants toChandler Carruth
'expand'. Also update the public API it provides to the new term, and propagate that update to the various clients. No functionality changed. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135138 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07Move SourceManager::isAt[Start/End]OfMacroInstantiation functions to the ↵Argyrios Kyrtzidis
Lexer, since they depend on it now. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134644 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-23Remove unused STL header includes.Jay Foad
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130068 91177308-0d34-0410-b5e6-96231b3b80d8
2011-03-08Fix my earlier commit to work with escaped newlines and leave breadcrumbsJohn McCall
in case we want to make a world where we can check intermediate instantiations for this kind of breadcrumb. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127221 91177308-0d34-0410-b5e6-96231b3b80d8
2010-11-17move getSpelling from Preprocessor to Lexer, which it is more conceptually ↵Chris Lattner
related to. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119479 91177308-0d34-0410-b5e6-96231b3b80d8
2010-11-17move AdvanceToTokenCharacter and getLocForEndOfToken fromChris Lattner
Preprocessor to Lexer where they make more sense. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119474 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30Revert my user-defined literal commits - r1124{58,60,67} pendingSean Hunt
some issues being sorted out. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112493 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-29Implement C++0x user-defined string literals.Sean Hunt
The extra data stored on user-defined literal Tokens is stored in extra allocated memory, which is managed by the PreprocessorLexer because there isn't a better place to put it that makes sure it gets deallocated, but only after it's used up. My testing has shown no significant slowdown as a result, but independent testing would be appreciated. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112458 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-09Use precompiled preambles for in-process code completion.Douglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110596 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-26Introduce basic support for loading a precompiled preamble whileDouglas Gregor
reparsing an ASTUnit. When saving a preamble, create a buffer larger than the actual file we're working with but fill everything from the end of the preamble to the end of the file with spaces (so the lexer will quickly skip them). When we load the file, create a buffer of the same size, filling it with the file and then spaces. Then, instruct the lexer to start lexing after the preamble, therefore continuing the parse from the spot where the preamble left off. It's now possible to perform a simple preamble build + parse (+ reparse) with ASTUnit. However, one has to disable a bunch of checking in the PCH reader to do so. That part isn't committed; it will likely be handled with some other kind of flag (e.g., -fno-validate-pch). As part of this, fix some issues with null termination of the memory buffers created for the preamble; we were trying to explicitly NULL-terminate them, even though they were also getting implicitly NULL terminated, leading to excess warnings about NULL characters in source files. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109445 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-22Improve performance during cursor traversal when a region of interestDouglas Gregor
is present. Rather than using clang_getCursorExtent(), which requires us to lex the token at the ending position to determine its length. Then, we'd be comparing [a, b) source ranges that cover the characters in the range rather than the normal behavior for Clang's source ranges, which covers the tokens in the range. However, relexing causes us to read the source file (which may come from a precompiled header), which is rather unfortunate and affects performance. In the new scheme, we only use Clang-style source ranges that cover the tokens in the range. At the entry points where this matters (clang_annotateTokens, clang_getCursor), we make sure to move source locations to the start of the token. Addresses most of <rdar://problem/8049381>. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109134 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-20Introduce a new lexer function to compute the "preamble" of a file,Douglas Gregor
which is the part of the file that contains all of the initial comments, includes, and preprocessor directives that occur before any of the actual code. Added a new -print-preamble cc1 action that is only used for testing. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108913 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-26Introduce a CIndex API for lexing the raw tokens within a given sourceDouglas Gregor
range. The token-annotation function does nothing, yet. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@94551 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-23update commentsChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92022 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-14teach clang to recover gracefully from conflict markers left in sourceChris Lattner
files: PR5238. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@91270 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-02Extend the source manager with the ability to override the contents ofDouglas Gregor
files with the contents of an arbitrary memory buffer. Use this new functionality to drastically clean up the way in which we handle file truncation for code-completion: all of the truncation/completion logic is now encapsulated in the preprocessor where it belongs (<rdar://problem/7434737>). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90300 91177308-0d34-0410-b5e6-96231b3b80d8
2009-11-30Fix PR5633 by making the preprocessor handle the case where we canChris Lattner
stat a file but where mmaping it fails. In this case, we emit an error like: t.c:1:10: fatal error: error opening file '../../foo.h' instead of "cannot find file". git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90110 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-07-31 21:20:59 +0000