aboutsummaryrefslogtreecommitdiff
path: root/lib/Lex/Lexer.cpp
AgeCommit message (Collapse)Author
2011-12-20Unweaken vtables as per ↵David Blaikie
http://llvm.org/docs/CodingStandards.html#ll_virtual_anch git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@146959 91177308-0d34-0410-b5e6-96231b3b80d8
2011-11-22Remove assert from hot code path and add a clarifying comment.Benjamin Kramer
The assert wasn't adding much value but slowed down Release+Asserts builds. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145082 91177308-0d34-0410-b5e6-96231b3b80d8
2011-11-22Lexer: Don't throw away the hard work SSE did to find a slash.Benjamin Kramer
We can reuse the information and avoid looping over all the bytes again. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145070 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-17Move about 20 random diagnostics under -W flags. Patch by Ahmed Charles!Ted Kremenek
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@142284 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-15-Wc++98-compat warnings for the lexer.Richard Smith
This also adds a -Wc++98-compat-pedantic for warning on constructs which would be diagnosed by -std=c++98 -pedantic (that is, it warns even on C++11 features which we enable by default, with no warning, in C++98 mode). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@142034 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-12We do parse hexfloats in C++11; make it actually work.Douglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141798 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-12Handle Perforce-style conflict markers like normal conflict markers. PerforceRichard Smith
swaps over the <<<< and >>>> markers, and uses shorter markers than traditional tools. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141751 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-03Fixed exapnsion range for # and ##.Abramo Bagnara
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141012 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-19Rename SourceLocation::getFileLocWithOffset -> getLocWithOffset.Argyrios Kyrtzidis
It already works (and is useful with) macro locs as well. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140057 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-17Rename LangOptions::Microsoft to LangOptions::MicrosoftExt to make it clear ↵Francois Pichet
that this flag must be used only for Microsoft extensions and not emulation; to avoid confusion with the new LangOptions::MicrosoftMode flag. Many of the code now under LangOptions::MicrosoftExt will eventually be moved under the LangOptions::MicrosoftMode flag. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139987 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-05Speed up BCPL comment lexing by looking aggressively for newlines and then ↵Benjamin Kramer
scannig backwards to see if the newline is escaped. 3% speedup in preprocessing all of clang with -Eonly. Also includes a small testcase for coverage. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139116 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-05Use the Lexer's definition of whitespace here.Benjamin Kramer
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139115 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-04Support code-completion for C++ inline methods and ObjC buffering methods.Argyrios Kyrtzidis
Previously we would cut off the source file buffer at the code-completion point; this impeded code-completion inside C++ inline methods and, recently, with buffering ObjC methods. Have the code-completion inserted into the source buffer so that it can be buffered along with a method body. When we actually hit the code-completion point the cut-off lexing or parsing. Fixes rdar://10056932&8319466 git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139086 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-04Fix Lexer::ComputePreamble when MaxLines parameter is non-zero.Argyrios Kyrtzidis
The function was only counting lines that included tokens and not empty lines, but MaxLines (mainly initiated to the line where the code-completion point resides) is a count of overall lines (even empty ones). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139085 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-26Introduce support for a simple module import declaration, whichDouglas Gregor
loads the named module. The syntax itself is intentionally hideous and will be replaced at some later point with something more palatable. For now, we're focusing on the semantics: - Module imports are handled first by the preprocessor (to get macro definitions) and then the same tokens are also handled by the parser (to get declarations). If both happen (as in normal compilation), the second one is redundant, because we currently have no way to hide macros or declarations when loading a module. Chris gets credit for this mad-but-workable scheme. - The Preprocessor now holds on to a reference to a module loader, which is responsible for loading named modules. CompilerInstance is the only important module loader: it now knows how to create and wire up an AST reader on demand to actually perform the module load. - We search for modules in the include path, using the module name with the suffix ".pcm" (precompiled module) for the file name. This is a temporary hack; we hope to improve the situation in the future. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138679 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-25Make Lexer::ComputePreamble accept a LangOptions parameter, otherwise it may beArgyrios Kyrtzidis
out-of-sync how a file is compiled. Patch by Matthias Kleine! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138580 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-23In Lexer::isAtEndOfMacroExpansion use SourceManager::isInFileID and avoidArgyrios Kyrtzidis
the extra SourceManager::getFileID call. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138376 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-17Make Lexer::GetBeginningOfToken able to handle macro arg expansion locations.Argyrios Kyrtzidis
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137795 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-11Add support for C++0x raw string literals.Craig Topper
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137298 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-27Add a utility function to the Lexer, which makes it easier to find a token ↵Anna Zaks
after the given location. (It is a generalized version of trans::findLocationAfterSemi from ArcMigrate, which will be changed to use the Lexer utility). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136268 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-27Add support for C++0x unicode string and character literals, from Craig Topper!Douglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136210 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-26Migrate 'Instantiation' data and API bits of SLocEntry to 'Expansion'Chandler Carruth
etc. With this I think essentially all of the SourceManager APIs are converted. Comments and random other bits of cleanup should be all thats left. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136057 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-26Convert InstantiationInfo and much of the related code to ExpansionInfoChandler Carruth
and various other 'expansion' based terms. I've tried to reformat where appropriate and catch as many references in comments but I'm going to do several more passes. Also I've tried to expand parameter names to be more clear where appropriate. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136056 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-26Rename create(MacroArg)InstantiationLoc to create(MacroArg)ExpansionLoc.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136054 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-25Rename SourceManager::getImmediateInstantiationRange toChandler Carruth
getImmediateExpansionRange. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135960 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-25Rename SourceManager::getInstantiationRange to getExpansionRange.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135915 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-25Mechanically rename SourceManager::getInstantiationLoc andChandler Carruth
FullSourceLoc::getInstantiationLoc to ...::getExpansionLoc. This is part of the API and documentation update from 'instantiation' as the term for macros to 'expansion'. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135914 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-23remove unneeded llvm:: namespace qualifiers on some core types now that ↵Chris Lattner
LLVM.h imports them into the clang namespace. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135852 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-20SpellingJoerg Sonnenberger
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135545 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-19Revamp the SourceManager to separate the representation of parsedDouglas Gregor
source locations from source locations loaded from an AST/PCH file. Previously, loading an AST/PCH file involved carefully pre-allocating space at the beginning of the source manager for the source locations and FileIDs that correspond to the prefix, and then appending the source locations/FileIDs used for parsing the remaining translation unit. This design forced us into loading PCH files early, as a prefix, whic has become a rather significant limitation. This patch splits the SourceManager space into two parts: for source location "addresses", the lower values (growing upward) are used to describe parsed code, while upper values (growing downward) are used for source locations loaded from AST/PCH files. Similarly, positive FileIDs are used to describe parsed code while negative FileIDs are used to file/macro locations loaded from AST/PCH files. As a result, we can load PCH/AST files even during parsing, making various improvemnts in the future possible, e.g., teaching #include <foo.h> to look for and load <foo.h.gch> if it happens to be already available. This patch was originally written by Sebastian Redl, then brought forward to the modern age by Jonathan Turner, and finally polished/finished by me to be committed. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135484 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-14Convert terminology in the Lexer from 'instantiate' and variants toChandler Carruth
'expand'. Also update the public API it provides to the new term, and propagate that update to the various clients. No functionality changed. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135138 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07Move SourceManager::isAt[Start/End]OfMacroInstantiation functions to the ↵Argyrios Kyrtzidis
Lexer, since they depend on it now. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134644 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07Make the Preprocessor more memory efficient and improve macro instantiation ↵Argyrios Kyrtzidis
diagnostics. When a macro instantiation occurs, reserve a SLocEntry chunk with length the full length of the macro definition source. Set the spelling location of this chunk to point to the start of the macro definition and any tokens that are lexed directly from the macro definition will get a location from this chunk with the appropriate offset. For any tokens that come from argument expansion, '##' paste operator, etc. have their instantiation location point at the appropriate place in the instantiated macro definition (the argument identifier and the '##' token respectively). This improves macro instantiation diagnostics: Before: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:5:11: note: instantiated from: int y = M(/); ^ After: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:3:20: note: instantiated from: \#define M(op) (foo op 3); ~~~ ^ ~ t.c:5:11: note: instantiated from: int y = M(/); ^ The memory savings for a candidate boost library that abuses the preprocessor are: - 32% less SLocEntries (37M -> 25M) - 30% reduction in PCH file size (900M -> 635M) - 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M) git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134587 91177308-0d34-0410-b5e6-96231b3b80d8
2011-06-24Allow Lexer::getLocForEndOfToken to return the location just passed the ↵Argyrios Kyrtzidis
macro instantiation if the location given points at the last token of the macro instantiation. Fixes rdar://9045701. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@133804 91177308-0d34-0410-b5e6-96231b3b80d8
2011-05-10Don't strlen() every file before parsing it.Eli Friedman
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@131132 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-15fix a bunch of comment typos found by codespell. Patch byChris Lattner
Luis Felipe Strano Moraes! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129559 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-14Implement C++0x [lex.pptoken]p3's handling of <::.Richard Smith
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129525 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-09Eat the UTF-8 BOM at the beginning of a file since it's ignored anyhow.Eric Christopher
Nom Nom Nom. Patch by Anton Korobeynikov! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129174 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-06Fix getLocForEndOfToken to not double-count spurious internal characters John McCall
within a token, like trigraphs and escaped newlines. Patch by Marcin Kowalczyk! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@128978 91177308-0d34-0410-b5e6-96231b3b80d8
2011-03-18Lexer: Add extremely limited support for -traditional-cpp, ignoring BCPLDaniel Dunbar
comments. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127910 91177308-0d34-0410-b5e6-96231b3b80d8
2011-03-08Fix my earlier commit to work with escaped newlines and leave breadcrumbsJohn McCall
in case we want to make a world where we can check intermediate instantiations for this kind of breadcrumb. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@127221 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-28Rename tok::eom to tok::eod.Peter Collingbourne
The previous name was inaccurate as this token in fact appears at the end of every preprocessing directive, not just macro definitions. No functionality change, except for a diagnostic tweak. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@126631 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-15Warn for missing terminating " or ' instead of error for gcc compatibility. ↵Argyrios Kyrtzidis
Fixed rdar://8914293. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@125616 91177308-0d34-0410-b5e6-96231b3b80d8
2011-02-09Lexer: add CUDA kernel call tokensPeter Collingbourne
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@125218 91177308-0d34-0410-b5e6-96231b3b80d8
2011-01-31Harden Lexer::GetBeginningOfToken() against bogus source locations andDouglas Gregor
the disappearance/alteration of files. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@124616 91177308-0d34-0410-b5e6-96231b3b80d8
2010-12-22Introduced raw_identifier token kind.Abramo Bagnara
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@122394 91177308-0d34-0410-b5e6-96231b3b80d8
2010-11-17move getSpelling from Preprocessor to Lexer, which it is more conceptually ↵Chris Lattner
related to. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119479 91177308-0d34-0410-b5e6-96231b3b80d8
2010-11-17move AdvanceToTokenCharacter and getLocForEndOfToken fromChris Lattner
Preprocessor to Lexer where they make more sense. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@119474 91177308-0d34-0410-b5e6-96231b3b80d8
2010-10-23Update remaining attribute macros to new style.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@117204 91177308-0d34-0410-b5e6-96231b3b80d8
2010-09-30In MeasureTokenLength, the FileLoc supplied to the lexer must point to the ↵Sebastian Redl
start of the buffer, or we risk overflow. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@115117 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-07-19 14:58:22 +0000