aboutsummaryrefslogtreecommitdiff
path: root/lib/Lex/Lexer.cpp
AgeCommit message (Collapse)Author
2011-04-14Implement C++0x [lex.pptoken]p3's handling of <::.Richard Smith
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129525 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-09Eat the UTF-8 BOM at the beginning of a file since it's ignored anyhow.Eric Christopher
Nom Nom Nom. Patch by Anton Korobeynikov! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129174 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-06Fix getLocForEndOfToken to not double-count spurious internal characters John McCall
within a token, like trigraphs and escaped newlines. Patch by Marcin Kowalczyk! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@128978 91177308-0d34-0410-b5e6-96231b3b80d8
2011-03-18Lexer: Add extremely limited support for -traditional-cpp, ignoring BCPLDaniel Dunbar
comments. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127910 91177308-0d34-0410-b5e6-96231b3b80d8
2011-03-08Fix my earlier commit to work with escaped newlines and leave breadcrumbsJohn McCall
in case we want to make a world where we can check intermediate instantiations for this kind of breadcrumb. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127221 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-28Rename tok::eom to tok::eod.Peter Collingbourne
The previous name was inaccurate as this token in fact appears at the end of every preprocessing directive, not just macro definitions. No functionality change, except for a diagnostic tweak. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126631 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-15Warn for missing terminating " or ' instead of error for gcc compatibility. ↵Argyrios Kyrtzidis
Fixed rdar://8914293. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@125616 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-09Lexer: add CUDA kernel call tokensPeter Collingbourne
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@125218 91177308-0d34-0410-b5e6-96231b3b80d8
2011-01-31Harden Lexer::GetBeginningOfToken() against bogus source locations andDouglas Gregor
the disappearance/alteration of files. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@124616 91177308-0d34-0410-b5e6-96231b3b80d8
2010-12-22Introduced raw_identifier token kind.Abramo Bagnara
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@122394 91177308-0d34-0410-b5e6-96231b3b80d8
2010-11-17move getSpelling from Preprocessor to Lexer, which it is more conceptually ↵Chris Lattner
related to. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119479 91177308-0d34-0410-b5e6-96231b3b80d8
2010-11-17move AdvanceToTokenCharacter and getLocForEndOfToken fromChris Lattner
Preprocessor to Lexer where they make more sense. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119474 91177308-0d34-0410-b5e6-96231b3b80d8
2010-10-23Update remaining attribute macros to new style.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@117204 91177308-0d34-0410-b5e6-96231b3b80d8
2010-09-30In MeasureTokenLength, the FileLoc supplied to the lexer must point to the ↵Sebastian Redl
start of the buffer, or we risk overflow. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@115117 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-31improve isHexaLiteral to work with escaped newlines and trigraphs,Chris Lattner
patch by Francois Pichet! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112602 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30silence a warningChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112549 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30Revert my user-defined literal commits - r1124{58,60,67} pendingSean Hunt
some issues being sorted out. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112493 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30add a fixme.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112491 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30use 'features' instead of 'PP->getLangOptions'.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112490 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30In Microsoft compatibility mode, don't parse the exponent as part ofDouglas Gregor
the pp-number in a hexadecimal floating point literal, from Francois Pichet! Fixes PR7968. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112481 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-29Implement C++0x user-defined string literals.Sean Hunt
The extra data stored on user-defined literal Tokens is stored in extra allocated memory, which is managed by the PreprocessorLexer because there isn't a better place to put it that makes sure it gets deallocated, but only after it's used up. My testing has shown no significant slowdown as a result, but independent testing would be appreciated. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112458 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-25Introduce a preprocessor code-completion hook for contexts where weDouglas Gregor
expect "natural" language and should not provide any completions, e.g., comments, string literals, #error. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112054 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-24Introduce basic code-completion support for preprocessor directives,Douglas Gregor
e.g., after a "#" we'll suggest #if, #ifdef, etc. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111943 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-12Don't emit end-of-file diagnostics like "unterminated conditional" orDouglas Gregor
"unterminated string" when we're performing code completion. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110933 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-11Random temporary string cleanup.Benjamin Kramer
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110807 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-09Use precompiled preambles for in-process code completion.Douglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110596 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-26Introduce basic support for loading a precompiled preamble whileDouglas Gregor
reparsing an ASTUnit. When saving a preamble, create a buffer larger than the actual file we're working with but fill everything from the end of the preamble to the end of the file with spaces (so the lexer will quickly skip them). When we load the file, create a buffer of the same size, filling it with the file and then spaces. Then, instruct the lexer to start lexing after the preamble, therefore continuing the parse from the spot where the preamble left off. It's now possible to perform a simple preamble build + parse (+ reparse) with ASTUnit. However, one has to disable a bunch of checking in the PCH reader to do so. That part isn't committed; it will likely be handled with some other kind of flag (e.g., -fno-validate-pch). As part of this, fix some issues with null termination of the memory buffers created for the preamble; we were trying to explicitly NULL-terminate them, even though they were also getting implicitly NULL terminated, leading to excess warnings about NULL characters in source files. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109445 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-22Improve performance during cursor traversal when a region of interestDouglas Gregor
is present. Rather than using clang_getCursorExtent(), which requires us to lex the token at the ending position to determine its length. Then, we'd be comparing [a, b) source ranges that cover the characters in the range rather than the normal behavior for Clang's source ranges, which covers the tokens in the range. However, relexing causes us to read the source file (which may come from a precompiled header), which is rather unfortunate and affects performance. In the new scheme, we only use Clang-style source ranges that cover the tokens in the range. At the entry points where this matters (clang_annotateTokens, clang_getCursor), we make sure to move source locations to the start of the token. Addresses most of <rdar://problem/8049381>. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109134 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-20Introduce a new lexer function to compute the "preamble" of a file,Douglas Gregor
which is the part of the file that contains all of the initial comments, includes, and preprocessor directives that occur before any of the actual code. Added a new -print-preamble cc1 action that is only used for testing. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108913 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-07fix PR4499, patch by Kyle Dean!Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@107836 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-30simpler fix for rdar://8044135 - escaped newlines have alreadyChris Lattner
been processed, so they don't have to be tip-toed around. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@105182 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-30Improve our handling of NULL after an escaping '\' in a stringDouglas Gregor
literal. Fixes <rdar://problem/8044135>. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@105181 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-25Improve code completion in failure cases in two ways:Douglas Gregor
1) Suppress diagnostics as soon as we form the code-completion token, so we don't get any error/warning spew from the early end-of-file. 2) If we consume a code-completion token when we weren't expecting one, go into a code-completion recovery path that produces the best results it can based on the context that the parser is in. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@104585 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-17robustify the conflict marker stuff. Don't add 7 twice, which wouldChris Lattner
make it miss (invalid) things like: <<<<<<< >>>>>>> and crash if <<<<<<< was at the end of the line. When we find a >>>>>>> that is not at the end of the line, make sure to reset Pos so we don't crash on something like: <<<<<<< >>>>>>> This isn't worth making testcases for, since each would require a new file. rdar://7987078 - signal 11 compiling "<<<<<<<<<<" git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@103968 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-16when code completing inside a C-style block comment, don't emit errors aboutChris Lattner
a missing */ since we truncated the file. This fixes rdar://7948776 git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@103913 91177308-0d34-0410-b5e6-96231b3b80d8
2010-04-12fix a minor bug I noticed while work with Jordy's patch for PR6101,Chris Lattner
in an input file like this: # 42 int x; we were emitting: # <something> int x; (with a space before the int) because we weren't clearing the leading whitespace flag properly after the \n from the directive was handled. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@101084 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-31Reinstate my CodeModificationHint -> FixItHint renaming patch, withoutDouglas Gregor
the C-only "optimization". git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@100022 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-31Revert r100008, which inexplicably breaks the clang-i686-darwin10 builderDouglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@100018 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-31Rename CodeModificationHint to FixItHint, since we've been using theDouglas Gregor
term "fix-it" everywhere and even *I* get tired of long names sometimes. No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@100008 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Remove unused variableDouglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98691 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Audit all Preprocessor::getSpelling() callers, improving failureDouglas Gregor
recovery for those that need it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98689 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Audit all callers of SourceManager::getCharacterData(); update some ofDouglas Gregor
them to recover more gracefully on failure. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98672 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Let SourceManager::getBufferData return StringRef instead of a pair of two ↵Benjamin Kramer
const char*. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98630 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Give SourceManager a Diagnostic object with which to report errors,Douglas Gregor
and start simplifying the interfaces in SourceManager that can fail. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98594 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-15Introduce a new BufferResult class to act as the return type ofDouglas Gregor
SourceManager's getBuffer() (and similar) operations. This abstract can be used to force callers to cope with errors in getBuffer(), such as missing files and changed files. Fix a bunch of callers to use the new interface. Add some very basic checks for file consistency (file size, modification time) into ContentCache::getBuffer(), although these checks don't help much until we've updated the main callers (e.g., SourceManager::getSpelling()). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98585 91177308-0d34-0410-b5e6-96231b3b80d8
2010-02-03don't inform comment handlers about comments in #if 0 blocks,Chris Lattner
doing so invalidates the file guard optimization and is not in the spirit of "#if 0" because it is supposed to completely skip everything, even if it isn't lexically valid. Patch by Abramo Bagnara! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@95253 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-22Teach CIndex's cursor visitor to restrict its traversal to a specificDouglas Gregor
region of interest (if provided). Implement clang_getCursor() in terms of this traversal rather than using the Index library; the unified cursor visitor is more complete, and will be The Way Forward. Minor other tweaks needed to make this work: - Extend Preprocessor::getLocForEndOfToken() to accept an offset from the end, making it easy to move to the last character in the token (rather than just past the end of the token). - In Lexer::MeasureTokenLength(), the length of whitespace is zero. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@94200 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-18allow the HandlerComment callback to push tokens into theChris Lattner
preprocessor. This could be used by an OpenMP implementation or something. Patch by Abramo Bagnara! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93795 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-11add a TODO for a perf improvement in LexIdentifier.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93141 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-10Do not parse hexadecimal floating point literals in C++0x mode because they areSean Hunt
incompatible with user-defined literals, specifically with the following form: 0x1p+1 The preprocessing-number token extends only as far as the 'p'; the '+' is not included. Previously we could get away with this extension as p was an invalid suffix, but now with user-defined literals, 'p' might well be a valid suffix and we are forced to consider it as such. This patch also adds a warning in non-0x C++ modes telling the user that this extension is incompatible with C++0x that is enabled by default (previously and with other languages, we warn only with a compliance option such as -pedantic). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93135 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-05-05 09:58:50 +0000