aboutsummaryrefslogtreecommitdiff
path: root/lib/Lex/TokenLexer.cpp
AgeCommit message (Collapse)Author
2010-03-15Introduce a new BufferResult class to act as the return type ofDouglas Gregor
SourceManager's getBuffer() (and similar) operations. This abstract can be used to force callers to cope with errors in getBuffer(), such as missing files and changed files. Fix a bunch of callers to use the new interface. Add some very basic checks for file consistency (file size, modification time) into ContentCache::getBuffer(), although these checks don't help much until we've updated the main callers (e.g., SourceManager::getSpelling()). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98585 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-13Use SmallString instead of SmallVectorKovarththanan Rajaratnam
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98436 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-13No need to call setIdentifierInfo() after LookUpIdentifierInfo() which ↵Kovarththanan Rajaratnam
LookUpIdentifierInfo() will automatically do git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98435 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-28The PreExpArgTokens array is indexed with an argument #,Chris Lattner
not a token number. Fix the reserve logic to get the right amount of space. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92202 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-23comment tweakChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92055 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-14move the VarargsElided member of MacrosArgs to shrink the MacroArgs structChris Lattner
on 64-bit targets. Pass Preprocessor into create/destroy methods of MacroArgs even though it isn't used yet. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@91345 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-04minor tidy.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90543 91177308-0d34-0410-b5e6-96231b3b80d8
2009-09-09Remove tabs, and whitespace cleanups.Mike Stump
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81346 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-28fix the "pasting formed 'a]', an invalid preprocessing token"Chris Lattner
diagnostic to include the full instantiation location for the invalid paste. For: #define foo(a, b) a ## b #define bar(x) foo(x, ]) bar(a) bar(zdy) Instead of: t.c:3:22: error: pasting formed 'a]', an invalid preprocessing token #define foo(a, b) a ## b ^ t.c:3:22: error: pasting formed 'zdy]', an invalid preprocessing token we now produce: t.c:7:1: error: pasting formed 'a]', an invalid preprocessing token bar(a) ^ t.c:4:16: note: instantiated from: #define bar(x) foo(x, ]) ^ t.c:3:22: note: instantiated from: #define foo(a, b) a ## b ^ t.c:8:1: error: pasting formed 'zdy]', an invalid preprocessing token bar(zdy) ^ t.c:4:16: note: instantiated from: #define bar(x) foo(x, ]) ^ t.c:3:22: note: instantiated from: #define foo(a, b) a ## b ^ git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72519 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-27Make the bad paste diagnostic print the entire pasted token.Eli Friedman
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72497 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-25improve comment, no functionality change.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72386 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-24Make sure an invalid concatentaion doesn't insert whitespace before Eli Friedman
the RHS. Fixes assembler-with-cpp issue reported on cfe-dev. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72370 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-22The TokenLexer may encounter annotations if the parser enters them using ↵Argyrios Kyrtzidis
Preprocessor::EnterTokenStream. So check for annotation before using the Token's IdentifierInfo. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72278 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-03Fix for PR4132: make sure to insert whitespace consistently before a Eli Friedman
pasted token. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@70793 91177308-0d34-0410-b5e6-96231b3b80d8
2009-04-25Silence gcc warnings.Eli Friedman
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@70086 91177308-0d34-0410-b5e6-96231b3b80d8
2009-04-19move token paste poisoning diagnostics to after the instantiation locChris Lattner
for a token is set, this makes the diagnostic "expanded from stack" work for this diagnostic. Add a testcase for PR3918. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69544 91177308-0d34-0410-b5e6-96231b3b80d8
2009-04-19Fix PR3918: Invalid use of __VA_ARGS__ not diagnosed,Chris Lattner
by rejecting invalid poisoned tokens in the token pasting path. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@69536 91177308-0d34-0410-b5e6-96231b3b80d8
2009-03-23The individual pieces of an invalid paste as still candidates for expansion.Chris Lattner
This matters in assembler mode, where this is silently allowed. This fixes rdar://6709206. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67539 91177308-0d34-0410-b5e6-96231b3b80d8
2009-03-18constructs like:Chris Lattner
#define Y X ## . Y are ok in .S files. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@67231 91177308-0d34-0410-b5e6-96231b3b80d8
2009-03-08simplify some logic by making ScratchBuffer handle the application of trailingChris Lattner
\0's to created tokens instead of making all clients do it. No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@66373 91177308-0d34-0410-b5e6-96231b3b80d8
2009-03-04make the token lexer allocate its temporary token buffers forChris Lattner
preexpanded macro arguments from the preprocessor's bump pointer. This reduces # mallocs from 12444 to 11792. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@66025 91177308-0d34-0410-b5e6-96231b3b80d8
2009-02-15track "just a little more" location information for macro instantiations.Chris Lattner
Now instead of just tracking the expansion history, also track the full range of the macro that got replaced. For object-like macros, this doesn't change anything. For _Pragma and function-like macros, this means we track the locations of the ')'. This is required for PR3579 because apparently GCC uses the line of the ')' of a function-like macro as the location to expand __LINE__ to. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@64601 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-29move library-specific diagnostic headers into library private dirs. ReduceChris Lattner
redundant #includes. Patch by Anders Johnsen! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63271 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-27Split the single monolithic DiagnosticKinds.def file into oneChris Lattner
.def file for each library. This means that adding a diagnostic to sema doesn't require all the other libraries to be rebuilt. Patch by Anders Johnsen! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63111 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-26remove my hacks that aggressively threw away multiple Chris Lattner
instantiation history in an effort to speed up c99-intconst-1.c. Now that multiple nested instantiations are allowed, we just make them and don't pay the cost of lookups. With the other changes that went in before this, reverting this is actually a speedup for c99-intconst-1.c, speeding it up from 1.96s to 1.80s, and preserves much better loc info. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63036 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-26This change refactors some of the low-level lexer interfaces a bit.Chris Lattner
Token now has a class of kinds for "literals", which include numeric constants, strings, etc. These tokens can optionally have a pointer to the start of the token in the lexer buffer. This makes it faster to get spelling and do other gymnastics, because we don't have to go through source locations. This change is performance neutral, but will make other changes more feasible down the road. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@63028 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-26Only resolve a macro's instantiation loc once per macro, instead of onceChris Lattner
per token lexed from it. This speeds up clang on c99-intconst-1.c from the GCC testsuite from 3.64s to 2.66s (36%). This reduces the number of binary search FileID lookups from 251570522 to 114175649 on this testcase. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62992 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-26Check in the long promised SourceLocation rewrite. This lays theChris Lattner
ground work for implementing #line, and fixes the "out of macro ID's" problem. There is nothing particularly tricky about the code, other than the very performance sensitive SourceManager::getFileID() method. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62978 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-23This is a follow-up to r62675:Chris Lattner
Refactor how the preprocessor changes a token from being an tok::identifier to a keyword (e.g. tok::kw_for). Instead of doing this in HandleIdentifier, hoist this common case out into the caller, so that every keyword doesn't have to go through HandleIdentifier. This drops time in HandleIdentifier from 1.25ms to .62ms, and speeds up clang -Eonly with PTH by about 1%. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62855 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-21Add a bit to IdentifierInfo that acts as a simple predicate whichChris Lattner
tells us whether Preprocessor::HandleIdentifier needs to be called. Because this method is only rarely needed, this saves a call and a bunch of random checks. This drops the time in HandleIdentifier from 3.52ms to .98ms on cocoa.h on my machine. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62675 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-19Make SourceLocation::getFileLoc private to reduce the API exposure of Chris Lattner
SourceLocation. This requires making some cleanups to token pasting and _Pragma expansion. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62490 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17make the verbose raw-lexer ctor fully explicit instead of havingChris Lattner
embedded magic. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62417 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17this massive patch introduces a simple new abstraction: it makesChris Lattner
"FileID" a concept that is now enforced by the compiler's type checker instead of yet-another-random-unsigned floating around. This is an important distinction from the "FileID" currently tracked by SourceLocation. *That* FileID may refer to the start of a file or to a chunk within it. The new FileID *only* refers to the file (and its #include stack and eventually #line data), it cannot refer to a chunk. FileID is a completely opaque datatype to all clients, only SourceManager is allowed to poke and prod it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62407 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-18Remove the last of the old-style Preprocessor::Diag methods.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59554 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Simplify raw mode lexing by treating an unterminate /**/ comment theChris Lattner
same we we do an unterminated string or character literal. This makes it so we can guarantee that the lexer never calls into the preprocessor (which would be suicide for a raw lexer). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57395 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Change how raw lexers are handled: instead of creating them and thenChris Lattner
using LexRawToken, create one and use LexFromRawLexer. This avoids twiddling the RawLexer flag around and simplifies some code (even speeding raw lexing up a tiny bit). This change also improves the token paster to use a Lexer on the stack instead of new/deleting it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57393 91177308-0d34-0410-b5e6-96231b3b80d8
2008-09-29whitespace and comment changes, to fix grammar and 80 col violationsNico Weber
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@56776 91177308-0d34-0410-b5e6-96231b3b80d8
2008-03-15Make a major restructuring of the clang tree: introduce a top-levelChris Lattner
lib dir and move all the libraries into it. This follows the main llvm tree, and allows the libraries to be built in parallel. The top level now enforces that all the libs are built before Driver, but we don't care what order the libs are built in. This speeds up parallel builds, particularly incremental ones. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48402 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2024-11-25 13:00:37 +0000