aboutsummaryrefslogtreecommitdiff
path: root/lib/Lex/Lexer.cpp
AgeCommit message (Collapse)Author
2010-10-23Update remaining attribute macros to new style.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@117204 91177308-0d34-0410-b5e6-96231b3b80d8
2010-09-30In MeasureTokenLength, the FileLoc supplied to the lexer must point to the ↵Sebastian Redl
start of the buffer, or we risk overflow. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@115117 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-31improve isHexaLiteral to work with escaped newlines and trigraphs,Chris Lattner
patch by Francois Pichet! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112602 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30silence a warningChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112549 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30Revert my user-defined literal commits - r1124{58,60,67} pendingSean Hunt
some issues being sorted out. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112493 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30add a fixme.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112491 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30use 'features' instead of 'PP->getLangOptions'.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112490 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-30In Microsoft compatibility mode, don't parse the exponent as part ofDouglas Gregor
the pp-number in a hexadecimal floating point literal, from Francois Pichet! Fixes PR7968. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112481 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-29Implement C++0x user-defined string literals.Sean Hunt
The extra data stored on user-defined literal Tokens is stored in extra allocated memory, which is managed by the PreprocessorLexer because there isn't a better place to put it that makes sure it gets deallocated, but only after it's used up. My testing has shown no significant slowdown as a result, but independent testing would be appreciated. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112458 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-25Introduce a preprocessor code-completion hook for contexts where weDouglas Gregor
expect "natural" language and should not provide any completions, e.g., comments, string literals, #error. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@112054 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-24Introduce basic code-completion support for preprocessor directives,Douglas Gregor
e.g., after a "#" we'll suggest #if, #ifdef, etc. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111943 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-12Don't emit end-of-file diagnostics like "unterminated conditional" orDouglas Gregor
"unterminated string" when we're performing code completion. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110933 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-11Random temporary string cleanup.Benjamin Kramer
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110807 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-09Use precompiled preambles for in-process code completion.Douglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110596 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-26Introduce basic support for loading a precompiled preamble whileDouglas Gregor
reparsing an ASTUnit. When saving a preamble, create a buffer larger than the actual file we're working with but fill everything from the end of the preamble to the end of the file with spaces (so the lexer will quickly skip them). When we load the file, create a buffer of the same size, filling it with the file and then spaces. Then, instruct the lexer to start lexing after the preamble, therefore continuing the parse from the spot where the preamble left off. It's now possible to perform a simple preamble build + parse (+ reparse) with ASTUnit. However, one has to disable a bunch of checking in the PCH reader to do so. That part isn't committed; it will likely be handled with some other kind of flag (e.g., -fno-validate-pch). As part of this, fix some issues with null termination of the memory buffers created for the preamble; we were trying to explicitly NULL-terminate them, even though they were also getting implicitly NULL terminated, leading to excess warnings about NULL characters in source files. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109445 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-22Improve performance during cursor traversal when a region of interestDouglas Gregor
is present. Rather than using clang_getCursorExtent(), which requires us to lex the token at the ending position to determine its length. Then, we'd be comparing [a, b) source ranges that cover the characters in the range rather than the normal behavior for Clang's source ranges, which covers the tokens in the range. However, relexing causes us to read the source file (which may come from a precompiled header), which is rather unfortunate and affects performance. In the new scheme, we only use Clang-style source ranges that cover the tokens in the range. At the entry points where this matters (clang_annotateTokens, clang_getCursor), we make sure to move source locations to the start of the token. Addresses most of <rdar://problem/8049381>. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@109134 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-20Introduce a new lexer function to compute the "preamble" of a file,Douglas Gregor
which is the part of the file that contains all of the initial comments, includes, and preprocessor directives that occur before any of the actual code. Added a new -print-preamble cc1 action that is only used for testing. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108913 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-07fix PR4499, patch by Kyle Dean!Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@107836 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-30simpler fix for rdar://8044135 - escaped newlines have alreadyChris Lattner
been processed, so they don't have to be tip-toed around. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@105182 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-30Improve our handling of NULL after an escaping '\' in a stringDouglas Gregor
literal. Fixes <rdar://problem/8044135>. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@105181 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-25Improve code completion in failure cases in two ways:Douglas Gregor
1) Suppress diagnostics as soon as we form the code-completion token, so we don't get any error/warning spew from the early end-of-file. 2) If we consume a code-completion token when we weren't expecting one, go into a code-completion recovery path that produces the best results it can based on the context that the parser is in. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@104585 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-17robustify the conflict marker stuff. Don't add 7 twice, which wouldChris Lattner
make it miss (invalid) things like: <<<<<<< >>>>>>> and crash if <<<<<<< was at the end of the line. When we find a >>>>>>> that is not at the end of the line, make sure to reset Pos so we don't crash on something like: <<<<<<< >>>>>>> This isn't worth making testcases for, since each would require a new file. rdar://7987078 - signal 11 compiling "<<<<<<<<<<" git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@103968 91177308-0d34-0410-b5e6-96231b3b80d8
2010-05-16when code completing inside a C-style block comment, don't emit errors aboutChris Lattner
a missing */ since we truncated the file. This fixes rdar://7948776 git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@103913 91177308-0d34-0410-b5e6-96231b3b80d8
2010-04-12fix a minor bug I noticed while work with Jordy's patch for PR6101,Chris Lattner
in an input file like this: # 42 int x; we were emitting: # <something> int x; (with a space before the int) because we weren't clearing the leading whitespace flag properly after the \n from the directive was handled. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@101084 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-31Reinstate my CodeModificationHint -> FixItHint renaming patch, withoutDouglas Gregor
the C-only "optimization". git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@100022 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-31Revert r100008, which inexplicably breaks the clang-i686-darwin10 builderDouglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@100018 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-31Rename CodeModificationHint to FixItHint, since we've been using theDouglas Gregor
term "fix-it" everywhere and even *I* get tired of long names sometimes. No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@100008 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Remove unused variableDouglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98691 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Audit all Preprocessor::getSpelling() callers, improving failureDouglas Gregor
recovery for those that need it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98689 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Audit all callers of SourceManager::getCharacterData(); update some ofDouglas Gregor
them to recover more gracefully on failure. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98672 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Let SourceManager::getBufferData return StringRef instead of a pair of two ↵Benjamin Kramer
const char*. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98630 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Give SourceManager a Diagnostic object with which to report errors,Douglas Gregor
and start simplifying the interfaces in SourceManager that can fail. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98594 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-15Introduce a new BufferResult class to act as the return type ofDouglas Gregor
SourceManager's getBuffer() (and similar) operations. This abstract can be used to force callers to cope with errors in getBuffer(), such as missing files and changed files. Fix a bunch of callers to use the new interface. Add some very basic checks for file consistency (file size, modification time) into ContentCache::getBuffer(), although these checks don't help much until we've updated the main callers (e.g., SourceManager::getSpelling()). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98585 91177308-0d34-0410-b5e6-96231b3b80d8
2010-02-03don't inform comment handlers about comments in #if 0 blocks,Chris Lattner
doing so invalidates the file guard optimization and is not in the spirit of "#if 0" because it is supposed to completely skip everything, even if it isn't lexically valid. Patch by Abramo Bagnara! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@95253 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-22Teach CIndex's cursor visitor to restrict its traversal to a specificDouglas Gregor
region of interest (if provided). Implement clang_getCursor() in terms of this traversal rather than using the Index library; the unified cursor visitor is more complete, and will be The Way Forward. Minor other tweaks needed to make this work: - Extend Preprocessor::getLocForEndOfToken() to accept an offset from the end, making it easy to move to the last character in the token (rather than just past the end of the token). - In Lexer::MeasureTokenLength(), the length of whitespace is zero. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@94200 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-18allow the HandlerComment callback to push tokens into theChris Lattner
preprocessor. This could be used by an OpenMP implementation or something. Patch by Abramo Bagnara! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93795 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-11add a TODO for a perf improvement in LexIdentifier.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93141 91177308-0d34-0410-b5e6-96231b3b80d8
2010-01-10Do not parse hexadecimal floating point literals in C++0x mode because they areSean Hunt
incompatible with user-defined literals, specifically with the following form: 0x1p+1 The preprocessing-number token extends only as far as the 'p'; the '+' is not included. Previously we could get away with this extension as p was an invalid suffix, but now with user-defined literals, 'p' might well be a valid suffix and we are forced to consider it as such. This patch also adds a warning in non-0x C++ modes telling the user that this extension is incompatible with C++0x that is enabled by default (previously and with other languages, we warn only with a compliance option such as -pedantic). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@93135 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-17reimplement r90860, fixing a couple of problems:Chris Lattner
1. Don't make a copy of LangOptions every time a lexer is created. 2. Don't make CharInfo global mutable state. 3. Fix the implementation to properly treat ^Z as EOF instead of as horizontal whitespace, which matches the semantic implemented by VC++. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@91586 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-14teach clang to recover gracefully from conflict markers left in sourceChris Lattner
files: PR5238. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@91270 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-08Integrate the following from the 'objective-rewrite' branch:Steve Naroff
http://llvm.org/viewvc/llvm-project?view=rev&revision=80043 git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90860 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-02Extend the source manager with the ability to override the contents ofDouglas Gregor
files with the contents of an arbitrary memory buffer. Use this new functionality to drastically clean up the way in which we handle file truncation for code-completion: all of the truncation/completion logic is now encapsulated in the preprocessor where it belongs (<rdar://problem/7434737>). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90300 91177308-0d34-0410-b5e6-96231b3b80d8
2009-11-30Fix PR5633 by making the preprocessor handle the case where we canChris Lattner
stat a file but where mmaping it fails. In this case, we emit an error like: t.c:1:10: fatal error: error opening file '../../foo.h' instead of "cannot find file". git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90110 91177308-0d34-0410-b5e6-96231b3b80d8
2009-11-14Move DISABLE_INLINE to the front of the decl so MSVC can parse it. Patch by ↵Benjamin Kramer
Amine Khaldi! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@88797 91177308-0d34-0410-b5e6-96231b3b80d8
2009-10-14Teach Lexer::MeasureTokenLength to be able to measure theChris Lattner
length of comment tokens. Patch by Abramo Bagnara! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@84100 91177308-0d34-0410-b5e6-96231b3b80d8
2009-09-22Replace the -code-completion-dump option with Douglas Gregor
-code-completion-at=filename:line:column which performs code completion at the specified location by truncating the file at that position and enabling code completion. This approach makes it possible to run multiple tests from a single test file, and gives a more natural command-line interface. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@82571 91177308-0d34-0410-b5e6-96231b3b80d8
2009-09-21Refactor and simplify the CodeCompleteConsumer, so that all of theDouglas Gregor
real work is performed within Sema. Addresses Chris's comments, but still retains the heavyweight list-of-multimaps data structure. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@82459 91177308-0d34-0410-b5e6-96231b3b80d8
2009-09-17Initial implementation of a code-completion interface in Clang. InDouglas Gregor
essence, code completion is triggered by a magic "code completion" token produced by the lexer [*], which the parser recognizes at certain points in the grammar. The parser then calls into the Action object with the appropriate CodeCompletionXXX action. Sema implements the CodeCompletionXXX callbacks by performing minimal translation, then forwarding them to a CodeCompletionConsumer subclass, which uses the results of semantic analysis to provide code-completion results. At present, only a single, "printing" code completion consumer is available, for regression testing and debugging. However, the design is meant to permit other code-completion consumers. This initial commit contains two code-completion actions: one for member access, e.g., "x." or "p->", and one for nested-name-specifiers, e.g., "std::". More code-completion actions will follow, along with improved gathering of code-completion results for the various contexts. [*] In the current -code-completion-dump testing/debugging mode, the file is truncated at the completion point and EOF is translated into "code completion". git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@82166 91177308-0d34-0410-b5e6-96231b3b80d8
2009-09-09Remove tabs, and whitespace cleanups.Mike Stump
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81346 91177308-0d34-0410-b5e6-96231b3b80d8
2009-07-07Convert the CharInfo table to be statically initialized, instead of ↵Chris Lattner
dynamically initialized. Patch by Ryan Flynn! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@74919 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-05-10 18:14:42 +0000