aboutsummaryrefslogtreecommitdiff
path: root/include/clang/Lex/Lexer.h
AgeCommit message (Collapse)Author
2009-09-18allow clearing this value.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@82271 91177308-0d34-0410-b5e6-96231b3b80d8
2009-09-17Initial implementation of a code-completion interface in Clang. InDouglas Gregor
essence, code completion is triggered by a magic "code completion" token produced by the lexer [*], which the parser recognizes at certain points in the grammar. The parser then calls into the Action object with the appropriate CodeCompletionXXX action. Sema implements the CodeCompletionXXX callbacks by performing minimal translation, then forwarding them to a CodeCompletionConsumer subclass, which uses the results of semantic analysis to provide code-completion results. At present, only a single, "printing" code completion consumer is available, for regression testing and debugging. However, the design is meant to permit other code-completion consumers. This initial commit contains two code-completion actions: one for member access, e.g., "x." or "p->", and one for nested-name-specifiers, e.g., "std::". More code-completion actions will follow, along with improved gathering of code-completion results for the various contexts. [*] In the current -code-completion-dump testing/debugging mode, the file is truncated at the completion point and EOF is translated into "code completion". git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@82166 91177308-0d34-0410-b5e6-96231b3b80d8
2009-09-09Remove tabs, and whitespace cleanups.Mike Stump
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81346 91177308-0d34-0410-b5e6-96231b3b80d8
2009-04-18add a new Lexer::SkipEscapedNewLines method.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69483 91177308-0d34-0410-b5e6-96231b3b80d8
2009-04-18factor escape newline measuring out into its own helper function.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69482 91177308-0d34-0410-b5e6-96231b3b80d8
2009-04-14Change Lexer::MeasureTokenLength to take a LangOptions reference.Chris Lattner
This allows it to accurately measure tokens, so that we get: t.cpp:8:13: error: unknown type name 'X' static foo::X P; ~~~~~^ instead of the woefully inferior: t.cpp:8:13: error: unknown type name 'X' static foo::X P; ~~~~ ^ Most of this is just plumbing to push the reference around. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69099 91177308-0d34-0410-b5e6-96231b3b80d8
2009-03-08simplify some logic by making ScratchBuffer handle the application of trailingChris Lattner
\0's to created tokens instead of making all clients do it. No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@66373 91177308-0d34-0410-b5e6-96231b3b80d8
2009-02-15track "just a little more" location information for macro instantiations.Chris Lattner
Now instead of just tracking the expansion history, also track the full range of the macro that got replaced. For object-like macros, this doesn't change anything. For _Pragma and function-like macros, this means we track the locations of the ')'. This is required for PR3579 because apparently GCC uses the line of the ')' of a function-like macro as the location to expand __LINE__ to. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64601 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-26Check in the long promised SourceLocation rewrite. This lays theChris Lattner
ground work for implementing #line, and fixes the "out of macro ID's" problem. There is nothing particularly tricky about the code, other than the very performance sensitive SourceManager::getFileID() method. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62978 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-19Make SourceLocation::getFileLoc private to reduce the API exposure of Chris Lattner
SourceLocation. This requires making some cleanups to token pasting and _Pragma expansion. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62490 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17Change the Lexer ctor used to lex _Pragma directives into a static factoryChris Lattner
method. This lets us clean up the interface and make it more obvious that this method is *really really* _Pragma specific. Note that _Pragma handling uglifies the Lexer in the critical path. It would be very interesting to consider making _Pragma remapping be a new special lexer class of its own. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62425 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17Change the Lexer ctor used in the non _Pragma case to take a FileID insteadChris Lattner
of a SourceLocation. This should speed it up and definitely simplifies it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62422 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17More simplifications to the lexer ctors.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62419 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17make the verbose raw-lexer ctor fully explicit instead of havingChris Lattner
embedded magic. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62417 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17add a simplified lexer ctor that sets up the lexer to raw-lex anChris Lattner
entire file. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62414 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17refactor some common initialization code out of the two lexer ctors.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62411 91177308-0d34-0410-b5e6-96231b3b80d8
2008-12-10PreprocessorLexer (and subclasses):Ted Kremenek
- Added virtual method 'getSourceLocation()' (no arguments) that gets the location of the next "observable" location (e.g., next character, next token). PPLexerChange.cpp: - Implemented FIXME by using PreprocessorLexer::getSourceLocation() to get the location in the file we are returning to after lexing a #included file. This appears to be slightly faster than having the branch (i.e., 'if(CurLexer)'). It's also not a really hot part of the Preprocessor. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60860 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-22actually, this version isn't really needed.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59859 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-22remove a sneaky version of Diag hiding in PreprocessorLexer.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59858 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-22Split the DiagnosticInfo class into two disjoint classes:Chris Lattner
one for building up the diagnostic that is in flight (DiagnosticBuilder) and one for pulling structured information out of the diagnostic when formatting and presenting it. There is no functionality change with this patch. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59849 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-18Convert the lexer and start converting the PP over to using canonical Diag ↵Chris Lattner
methods. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59511 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-18- Add Lexer::isPragma() accessor for clients of Lexer that aren't friends.Ted Kremenek
- Add static method to test if the current lexer is a non-macro/non-pragma lexer. - Refactor some code in PPLexerChange to use this static method. - No performance change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59486 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-12Move some diagnostic handling to PreprocessorLexer.Ted Kremenek
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59191 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-12Move LexIncludeFilename from Lexer to PreprocessorLexer.Ted Kremenek
PreprocessorLexer now has a virtual method "IndirectLex" which allows it to call the lex method of its subclasses. This is not for performance intensive operations. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59185 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-12Move pieces of Lexer that the Preprocessor mutates to a new base class ↵Ted Kremenek
'PreprocessorLexer'. This will also be the base class of the new Preprocessed-Token-Header (PTH) lexer. No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59168 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Change FormTokenWithChars to take the token kind to form, since all clientsChris Lattner
were setting a kind and then forming it. This is just a minor API cleanup, no functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57404 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Add a new mode to the lexer which enables it to return all characters,Chris Lattner
even whitespace, as tokens from the file. This is enabled with L->SetKeepWhitespaceMode(true) on a raw lexer. In this mode, you too can use clang as a really complex version of 'cat' with code like this: Lexer RawLex(SourceLocation::getFileLoc(SM.getMainFileID(), 0), PP.getLangOptions(), File.first, File.second); RawLex.SetKeepWhitespaceMode(true); Token RawTok; RawLex.LexFromRawLexer(RawTok); while (RawTok.isNot(tok::eof)) { std::cout << PP.getSpelling(RawTok); RawLex.LexFromRawLexer(RawTok); } This will emit exactly the input file, with no canonicalization or other translation. Realistic clients actually do something with the tokens of course :) git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57401 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12add a new inKeepCommentMode() accessor to abstract the KeepCommentModeChris Lattner
ivar. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57397 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Simplify raw mode lexing by treating an unterminate /**/ comment theChris Lattner
same we we do an unterminated string or character literal. This makes it so we can guarantee that the lexer never calls into the preprocessor (which would be suicide for a raw lexer). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57395 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Change how raw lexers are handled: instead of creating them and thenChris Lattner
using LexRawToken, create one and use LexFromRawLexer. This avoids twiddling the RawLexer flag around and simplifies some code (even speeding raw lexing up a tiny bit). This change also improves the token paster to use a Lexer on the stack instead of new/deleting it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57393 91177308-0d34-0410-b5e6-96231b3b80d8
2008-04-16reenable highlighting of (the first line of) commentsChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@49816 91177308-0d34-0410-b5e6-96231b3b80d8
2007-12-29Don't attribute in file headers anymore. See llvmdev for theChris Lattner
discussion of this change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@45410 91177308-0d34-0410-b5e6-96231b3b80d8
2007-10-17Fix assertion for raw lexer.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43091 91177308-0d34-0410-b5e6-96231b3b80d8
2007-10-17Move token length calculation out of the diagnostics machinery into Chris Lattner
the lexer, where it can be shared. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43090 91177308-0d34-0410-b5e6-96231b3b80d8
2007-10-17Add a new type of lexer: a raw lexer, which does not require a preprocessorChris Lattner
object in order to do its thing. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43084 91177308-0d34-0410-b5e6-96231b3b80d8
2007-10-17add some comments.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@43079 91177308-0d34-0410-b5e6-96231b3b80d8
2007-10-09Make a significant change to invert the control flow handlingChris Lattner
predefined macros. Previously, these were handled by the driver, now they are handled by the preprocessor. Some fallout of this: 1. Instead of preprocessing two buffers (the predefines, then the main source file) we now start preprocessing the main source file and inject the predefines as a "psuedo #include" from the main source file. 2. #1 allows us to nuke the Lexer::IsMainFile flag and simplify Preprocessor::isInPrimaryFile. 3. The driver doesn't have to know about standard #defines, the preprocessor knows, which is nice for people wanting to define their own drivers. 4. This allows us to put normal tokens in the predefine buffer, for example a definition for __builtin_va_list that is target-specific, and a typedef for id in objc. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@42818 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-24Use a smallstring instead of an std::string in FileChanged to avoid some ↵Chris Lattner
malloc traffic. This speeds up -E on xalancbmk by 2.4% git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40461 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-22Change hte lexer to start a start pointer to the underlying Chris Lattner
memorybuffer instead of a pointer to the memorybuffer itself. This reduces coupling and eliminates a pointer dereference on a hot path. This speeds up -Eonly on 483.xalancbmk by 2.1% git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40394 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-20At one point there were going to be lexer and parser tokens.Chris Lattner
Since that point is now long gone, we should rename LexerToken to Token, as it is the only kind of token we have. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40105 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-20simplify the lexer ctor to take a SLoc instead of a sloc and a redundant ↵Chris Lattner
buffer*. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40104 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-20Reimplement SourceLocation. Instead of having a Chris Lattner
fileid/offset pair, it now contains a bit discriminating between mapped locations and file locations. This separates the tables for macros and files in SourceManager, and allows better separation of concepts in the rest of the compiler. This allows us to have *many* macro instantiations before running out of 'addressing space'. This is also more efficient, because testing whether something is a macro expansion is now a bit test instead of a table lookup (which also used to require having a srcmgr around, now it doesn't). This is fully functional, but there are several refinements and optimizations left. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@40103 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-16Add a new Preprocessor::AdvanceToTokenCharacter method which, given a slocChris Lattner
specifying the start of a token and a logical (phase 3) character number, returns a sloc representing the input character corresponding to it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39905 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-16factor a common predicate into a static method.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39903 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-12Improve portability to compilers where <cassert> is not implicitlyChris Lattner
included. Patch contributed by Benoit Boissinot! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39779 91177308-0d34-0410-b5e6-96231b3b80d8
2007-07-11Stage two of getting CFE top correct.Reid Spencer
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@39734 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2025-09-25 23:31:55 +0000