aboutsummaryrefslogtreecommitdiff
path: root/lib/Lex/TokenLexer.cpp
AgeCommit message (Collapse)Author
2012-02-05Basic: import SmallString<> into clang namespaceDylan Noblesmith
(I was going to fix the TODO about DenseMap too, but that would break self-host right now. See PR11922.) git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149799 91177308-0d34-0410-b5e6-96231b3b80d8
2012-02-04Move a method from IdentifierTable.h out of line and remove the SmallString ↵Benjamin Kramer
include. Fix all the transitive include users. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149783 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-03Fixed exapnsion range for # and ##.Abramo Bagnara
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141012 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-19Rename SourceLocation::getFileLocWithOffset -> getLocWithOffset.Argyrios Kyrtzidis
It already works (and is useful with) macro locs as well. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140057 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-17Rename LangOptions::Microsoft to LangOptions::MicrosoftExt to make it clear ↵Francois Pichet
that this flag must be used only for Microsoft extensions and not emulation; to avoid confusion with the new LangOptions::MicrosoftMode flag. Many of the code now under LangOptions::MicrosoftExt will eventually be moved under the LangOptions::MicrosoftMode flag. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139987 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-24Silence 'may be used uninitialized' warnings.Argyrios Kyrtzidis
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138475 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-23Amend r138129 (reduction of SLocEntries) which introduced performance ↵Argyrios Kyrtzidis
regression due to increased calls to SourceManager::getFileID. (rdar://9992664) Use a slightly different approach that is more efficient both in terms of speed (no extra getFileID calls) and in SLocEntries reduction. Comparing pre-r138129 and this patch we get: For compiling SemaExpr.cpp reduction of SLocEntries by 26%. For the boost enum library: -SLocEntries -34% (note that this was -5% for r138129) -Memory consumption -50% -PCH size -31% Reduced SLocEntries also benefit the hot function SourceManager::getFileID, evident by the reduced "FileID scans". git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138380 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-23Introduce SourceManager::isInSLocAddrSpace and use it in TokenLexer instead ↵Argyrios Kyrtzidis
of isInFileID since it is a bit more efficient. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138379 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-23Rename SourceManager::isBeforeInSourceLocationOffset -> isBeforeInSLocAddrSpace.Argyrios Kyrtzidis
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138378 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-23TokenLexer::getExpansionLocForMacroDefLoc doesn't need to return an invalid ↵Argyrios Kyrtzidis
SourceLocation. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138377 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-19For assigning SourceLocations to macro arg tokens, reserve a single SLocEntryArgyrios Kyrtzidis
for tokens that are lexed consecutively from the same FileID, instead of creating a SLocEntry for each token. e.g for assert(foo == bar); there will be a single SLocEntry for the "foo == bar" chunk and locations for the 'foo', '==', 'bar' tokens will point inside that chunk. For parsing SemaExpr.cpp, this reduced the number of SLocEntries by 25%. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138129 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-19Rename TokenLexer::getMacroExpansionLocation -> ↵Argyrios Kyrtzidis
getExpansionLocForMacroDefLoc, no functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138128 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-26Rename create(MacroArg)InstantiationLoc to create(MacroArg)ExpansionLoc.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136054 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-25Mechanically rename SourceManager::getInstantiationLoc andChandler Carruth
FullSourceLoc::getInstantiationLoc to ...::getExpansionLoc. This is part of the API and documentation update from 'instantiation' as the term for macros to 'expansion'. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135914 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-23remove unneeded llvm:: namespace qualifiers on some core types now that ↵Chris Lattner
LLVM.h imports them into the clang namespace. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135852 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-19Revamp the SourceManager to separate the representation of parsedDouglas Gregor
source locations from source locations loaded from an AST/PCH file. Previously, loading an AST/PCH file involved carefully pre-allocating space at the beginning of the source manager for the source locations and FileIDs that correspond to the prefix, and then appending the source locations/FileIDs used for parsing the remaining translation unit. This design forced us into loading PCH files early, as a prefix, whic has become a rather significant limitation. This patch splits the SourceManager space into two parts: for source location "addresses", the lower values (growing upward) are used to describe parsed code, while upper values (growing downward) are used for source locations loaded from AST/PCH files. Similarly, positive FileIDs are used to describe parsed code while negative FileIDs are used to file/macro locations loaded from AST/PCH files. As a result, we can load PCH/AST files even during parsing, making various improvemnts in the future possible, e.g., teaching #include <foo.h> to look for and load <foo.h.gch> if it happens to be already available. This patch was originally written by Sebastian Redl, then brought forward to the modern age by Jonathan Turner, and finally polished/finished by me to be committed. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135484 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-14Switch the TokenLexer's terminology from various forms of 'instantiate'Chandler Carruth
to 'expand' for macros. Only comments and uses local to the TokenLexer are updated. No functionality changed. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135137 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-08Tweak formatting.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134675 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-08Switch the token-paste source locations inside of function style macroChandler Carruth
argument expansion to use the macro argument source locations as well. Add a few tests to exercise this. There is still a bit more work needed here though. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134674 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07Keep track of which source locations are part of a macro argumentChandler Carruth
instantiation and improve diagnostics which are stem from macro arguments to trace the argument itself back through the layers of macro expansion. This requires some tricky handling of the source locations, as the argument appears to be expanded in the opposite direction from the surrounding macro. This patch provides helper routines that encapsulate the logic and explain the reasoning behind how we step through macros during diagnostic printing. This fixes the rest of the test cases originially in PR9279, and later split out into PR10214 and PR10215. There is still some more work we can do here to improve the macro backtrace, but those will follow as separate patches. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134660 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07Turn hashhash into tok::unkwown when it comes from expanding an argument, ↵Argyrios Kyrtzidis
per Chris' suggestion. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134621 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07When expanding macro arguments, treat '##' coming from an argument as a ↵Argyrios Kyrtzidis
normal token. e.g. #define M(x) A x B M(##) // should expand to 'A ## B', not 'AB' git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134588 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07Make the Preprocessor more memory efficient and improve macro instantiation ↵Argyrios Kyrtzidis
diagnostics. When a macro instantiation occurs, reserve a SLocEntry chunk with length the full length of the macro definition source. Set the spelling location of this chunk to point to the start of the macro definition and any tokens that are lexed directly from the macro definition will get a location from this chunk with the appropriate offset. For any tokens that come from argument expansion, '##' paste operator, etc. have their instantiation location point at the appropriate place in the instantiated macro definition (the argument identifier and the '##' token respectively). This improves macro instantiation diagnostics: Before: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:5:11: note: instantiated from: int y = M(/); ^ After: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:3:20: note: instantiated from: \#define M(op) (foo op 3); ~~~ ^ ~ t.c:5:11: note: instantiated from: int y = M(/); ^ The memory savings for a candidate boost library that abuses the preprocessor are: - 32% less SLocEntries (37M -> 25M) - 30% reduction in PCH file size (900M -> 635M) - 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M) git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134587 91177308-0d34-0410-b5e6-96231b3b80d8
2011-06-29Introduce a caching mechanism for macro expanded tokens.Argyrios Kyrtzidis
Previously macro expanded tokens were added to Preprocessor's bump allocator and never released, even after the TokenLexer that were lexing them was finished, thus they were wasting memory. A very "useful" boost library was causing clang to eat 1 GB just for the expanded macro tokens. Introduce a special cache that works like a stack; a TokenLexer can add the macro expanded tokens in the cache, and when it finishes, the tokens are removed from the end of the cache. Now consumed memory by expanded tokens for that library is ~ 1.5 MB. Part of rdar://9327049. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134105 91177308-0d34-0410-b5e6-96231b3b80d8
2011-06-14revert r133003 and fix the bug properly: the issue was that ## in a tokenChris Lattner
lexer is not a paste operator, it is a normal token. This fixes a conformance issue shown here: http://p99.gforge.inria.fr/c99-conformance/c99-conformance-clang-2.9.html and it defines away the crash from before. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@133005 91177308-0d34-0410-b5e6-96231b3b80d8
2011-06-14Fix a crash on the testcase in PR9981 / rdar://9486765.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@133003 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-28Parsing/AST support for Structured Exception HandlingJohn Wiegley
Patch authored by Sohail Somani. Provide parsing and AST support for Windows structured exception handling. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@130366 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-28Rename tok::eom to tok::eod.Peter Collingbourne
The previous name was inaccurate as this token in fact appears at the end of every preprocessing directive, not just macro definitions. No functionality change, except for a diagnostic tweak. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126631 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-22Make TokenLexer capable of storing preprocessor directive tokensPeter Collingbourne
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126220 91177308-0d34-0410-b5e6-96231b3b80d8
2010-12-22Introduced raw_identifier token kind.Abramo Bagnara
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@122394 91177308-0d34-0410-b5e6-96231b3b80d8
2010-10-30Rename alignof -> alignOf to avoid irritating C++'0x compilers,Chris Lattner
PR8423 git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@117775 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-21fix PR7943, a corner case with the GNU __VA_ARGS__ comma Chris Lattner
swallowing extension. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@111701 91177308-0d34-0410-b5e6-96231b3b80d8
2010-08-11Random temporary string cleanup.Benjamin Kramer
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@110807 91177308-0d34-0410-b5e6-96231b3b80d8
2010-07-17Add another terrible VC++ compatibility hack: allow users toChris Lattner
allow invalid token pastes (when in -fms-extensions mode) with -Wno-invalid-token-paste git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@108624 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Audit all Preprocessor::getSpelling() callers, improving failureDouglas Gregor
recovery for those that need it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98689 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Let SourceManager::getBufferData return StringRef instead of a pair of two ↵Benjamin Kramer
const char*. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98630 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-16Give SourceManager a Diagnostic object with which to report errors,Douglas Gregor
and start simplifying the interfaces in SourceManager that can fail. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98594 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-15Introduce a new BufferResult class to act as the return type ofDouglas Gregor
SourceManager's getBuffer() (and similar) operations. This abstract can be used to force callers to cope with errors in getBuffer(), such as missing files and changed files. Fix a bunch of callers to use the new interface. Add some very basic checks for file consistency (file size, modification time) into ContentCache::getBuffer(), although these checks don't help much until we've updated the main callers (e.g., SourceManager::getSpelling()). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98585 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-13Use SmallString instead of SmallVectorKovarththanan Rajaratnam
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98436 91177308-0d34-0410-b5e6-96231b3b80d8
2010-03-13No need to call setIdentifierInfo() after LookUpIdentifierInfo() which ↵Kovarththanan Rajaratnam
LookUpIdentifierInfo() will automatically do git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@98435 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-28The PreExpArgTokens array is indexed with an argument #,Chris Lattner
not a token number. Fix the reserve logic to get the right amount of space. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92202 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-23comment tweakChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@92055 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-14move the VarargsElided member of MacrosArgs to shrink the MacroArgs structChris Lattner
on 64-bit targets. Pass Preprocessor into create/destroy methods of MacroArgs even though it isn't used yet. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@91345 91177308-0d34-0410-b5e6-96231b3b80d8
2009-12-04minor tidy.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@90543 91177308-0d34-0410-b5e6-96231b3b80d8
2009-09-09Remove tabs, and whitespace cleanups.Mike Stump
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@81346 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-28fix the "pasting formed 'a]', an invalid preprocessing token"Chris Lattner
diagnostic to include the full instantiation location for the invalid paste. For: #define foo(a, b) a ## b #define bar(x) foo(x, ]) bar(a) bar(zdy) Instead of: t.c:3:22: error: pasting formed 'a]', an invalid preprocessing token #define foo(a, b) a ## b ^ t.c:3:22: error: pasting formed 'zdy]', an invalid preprocessing token we now produce: t.c:7:1: error: pasting formed 'a]', an invalid preprocessing token bar(a) ^ t.c:4:16: note: instantiated from: #define bar(x) foo(x, ]) ^ t.c:3:22: note: instantiated from: #define foo(a, b) a ## b ^ t.c:8:1: error: pasting formed 'zdy]', an invalid preprocessing token bar(zdy) ^ t.c:4:16: note: instantiated from: #define bar(x) foo(x, ]) ^ t.c:3:22: note: instantiated from: #define foo(a, b) a ## b ^ git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72519 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-27Make the bad paste diagnostic print the entire pasted token.Eli Friedman
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72497 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-25improve comment, no functionality change.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72386 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-24Make sure an invalid concatentaion doesn't insert whitespace before Eli Friedman
the RHS. Fixes assembler-with-cpp issue reported on cfe-dev. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72370 91177308-0d34-0410-b5e6-96231b3b80d8
2009-05-22The TokenLexer may encounter annotations if the parser enters them using ↵Argyrios Kyrtzidis
Preprocessor::EnterTokenStream. So check for annotation before using the Token's IdentifierInfo. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@72278 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-05-11 12:21:07 +0000