aboutsummaryrefslogtreecommitdiff
path: root/lib/Lex/Lexer.cpp
AgeCommit message (Collapse)Author
2012-03-08Implement C++11 [lex.ext]p10 for string and character literals: a ud-suffix notRichard Smith
starting with an underscore is ill-formed. Since this rule rejects programs that were using <inttypes.h>'s macros, recover from this error by treating the ud-suffix as a separate preprocessing-token, with a DefaultError ExtWarn. The approach of treating such cases as two tokens is under discussion for standardization, but is in any case a conforming extension and allows existing codebases to keep building while the committee makes up its mind. Reword the warning on the definition of literal operators not starting with underscores (which are, strangely, legal) to more explicitly state that such operators can't be called by literals. Remove the special-case diagnostic for hexfloats, since it was both triggering in the wrong cases and incorrect. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152287 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-07Add -Wc++11-compat warning for string and character literals followed byRichard Smith
identifiers, in cases where those identifiers would be treated as user-defined literal suffixes in C++11. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152198 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-06User-defined literals: reject string and character UDLs in all places where theRichard Smith
grammar requires a string-literal and not a user-defined-string-literal. The two constructs are still represented by the same TokenKind, in order to prevent a combinatorial explosion of different kinds of token. A flag on Token tracks whether a ud-suffix is present, in order to prevent clients from needing to look at the token's spelling. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152098 91177308-0d34-0410-b5e6-96231b3b80d8
2012-03-05Lexing support for user-defined literals. Currently these lex as the same tokenRichard Smith
kinds as the underlying string literals, and we silently drop the ud-suffix; those issues will be fixed by subsequent patches. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@152012 91177308-0d34-0410-b5e6-96231b3b80d8
2012-02-03Change Lexer::makeFileCharRange() to have it accept a CharSourceRangeArgyrios Kyrtzidis
instead of a SourceRange, and handle the case where the range is a char (not token) range. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@149677 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-23Improve Lexer::getImmediateMacroName to take into account inner macrosArgyrios Kyrtzidis
of macro arguments. For "MAC1( MAC2(foo) )" and location of 'foo' token it would return "MAC1" instead of "MAC2". git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148704 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-20Enhance Lexer::makeFileCharRange to check for ranges inside a macro argumentArgyrios Kyrtzidis
expansion, in which case it returns a file range in the location where the argument was spelled. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148551 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-19Introduce Lexer::getSourceText() that returns a string for the sourceArgyrios Kyrtzidis
that the given source range encompasses. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148481 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-19Introduce Lexer::makeFileCharRange() that accepts a token source rangeArgyrios Kyrtzidis
and returns a character range with file locations. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148480 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-19For Lexer's isAt[Start/End]OfMacroExpansion add an out parameter for the macroArgyrios Kyrtzidis
start/end location. It is commonly needed after calling the function; with this way we avoid recalculating it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148479 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-18Refactor: Pull getImmediateMacroName() out of DiagnosticRenderer andAnna Zaks
into Lexer and Preprocessor; making it widely available. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148410 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-15Two variables had been added for an assert, but their values wereChandler Carruth
re-computed rather than the variables be re-used just after the assert. Just use the variables since we have them already. Fixes an unused variable warning. Also fix an 80-column violation. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148212 91177308-0d34-0410-b5e6-96231b3b80d8
2011-12-22In Lexer::getCharAndSizeSlow[NoWarn] if we come up againstArgyrios Kyrtzidis
\<newline><newline> don't consume the second newline. Thanks to David Blaikie for pointing out the crash! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147138 91177308-0d34-0410-b5e6-96231b3b80d8
2011-12-21In Lexer::getCharAndSizeSlow[NoWarn] make sure we don't go over the end of ↵Argyrios Kyrtzidis
the buffer when the end of the buffer is immediately after an escaped newline. Fixes http://llvm.org/PR10153. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@147091 91177308-0d34-0410-b5e6-96231b3b80d8
2011-12-20Unweaken vtables as per ↵David Blaikie
http://llvm.org/docs/CodingStandards.html#ll_virtual_anch git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@146959 91177308-0d34-0410-b5e6-96231b3b80d8
2011-11-22Remove assert from hot code path and add a clarifying comment.Benjamin Kramer
The assert wasn't adding much value but slowed down Release+Asserts builds. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145082 91177308-0d34-0410-b5e6-96231b3b80d8
2011-11-22Lexer: Don't throw away the hard work SSE did to find a slash.Benjamin Kramer
We can reuse the information and avoid looping over all the bytes again. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@145070 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-17Move about 20 random diagnostics under -W flags. Patch by Ahmed Charles!Ted Kremenek
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@142284 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-15-Wc++98-compat warnings for the lexer.Richard Smith
This also adds a -Wc++98-compat-pedantic for warning on constructs which would be diagnosed by -std=c++98 -pedantic (that is, it warns even on C++11 features which we enable by default, with no warning, in C++98 mode). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@142034 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-12We do parse hexfloats in C++11; make it actually work.Douglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141798 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-12Handle Perforce-style conflict markers like normal conflict markers. PerforceRichard Smith
swaps over the <<<< and >>>> markers, and uses shorter markers than traditional tools. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141751 91177308-0d34-0410-b5e6-96231b3b80d8
2011-10-03Fixed exapnsion range for # and ##.Abramo Bagnara
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@141012 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-19Rename SourceLocation::getFileLocWithOffset -> getLocWithOffset.Argyrios Kyrtzidis
It already works (and is useful with) macro locs as well. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@140057 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-17Rename LangOptions::Microsoft to LangOptions::MicrosoftExt to make it clear ↵Francois Pichet
that this flag must be used only for Microsoft extensions and not emulation; to avoid confusion with the new LangOptions::MicrosoftMode flag. Many of the code now under LangOptions::MicrosoftExt will eventually be moved under the LangOptions::MicrosoftMode flag. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139987 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-05Speed up BCPL comment lexing by looking aggressively for newlines and then ↵Benjamin Kramer
scannig backwards to see if the newline is escaped. 3% speedup in preprocessing all of clang with -Eonly. Also includes a small testcase for coverage. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139116 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-05Use the Lexer's definition of whitespace here.Benjamin Kramer
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139115 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-04Support code-completion for C++ inline methods and ObjC buffering methods.Argyrios Kyrtzidis
Previously we would cut off the source file buffer at the code-completion point; this impeded code-completion inside C++ inline methods and, recently, with buffering ObjC methods. Have the code-completion inserted into the source buffer so that it can be buffered along with a method body. When we actually hit the code-completion point the cut-off lexing or parsing. Fixes rdar://10056932&8319466 git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139086 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-04Fix Lexer::ComputePreamble when MaxLines parameter is non-zero.Argyrios Kyrtzidis
The function was only counting lines that included tokens and not empty lines, but MaxLines (mainly initiated to the line where the code-completion point resides) is a count of overall lines (even empty ones). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139085 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-26Introduce support for a simple module import declaration, whichDouglas Gregor
loads the named module. The syntax itself is intentionally hideous and will be replaced at some later point with something more palatable. For now, we're focusing on the semantics: - Module imports are handled first by the preprocessor (to get macro definitions) and then the same tokens are also handled by the parser (to get declarations). If both happen (as in normal compilation), the second one is redundant, because we currently have no way to hide macros or declarations when loading a module. Chris gets credit for this mad-but-workable scheme. - The Preprocessor now holds on to a reference to a module loader, which is responsible for loading named modules. CompilerInstance is the only important module loader: it now knows how to create and wire up an AST reader on demand to actually perform the module load. - We search for modules in the include path, using the module name with the suffix ".pcm" (precompiled module) for the file name. This is a temporary hack; we hope to improve the situation in the future. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138679 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-25Make Lexer::ComputePreamble accept a LangOptions parameter, otherwise it may beArgyrios Kyrtzidis
out-of-sync how a file is compiled. Patch by Matthias Kleine! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138580 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-23In Lexer::isAtEndOfMacroExpansion use SourceManager::isInFileID and avoidArgyrios Kyrtzidis
the extra SourceManager::getFileID call. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@138376 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-17Make Lexer::GetBeginningOfToken able to handle macro arg expansion locations.Argyrios Kyrtzidis
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137795 91177308-0d34-0410-b5e6-96231b3b80d8
2011-08-11Add support for C++0x raw string literals.Craig Topper
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@137298 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-27Add a utility function to the Lexer, which makes it easier to find a token ↵Anna Zaks
after the given location. (It is a generalized version of trans::findLocationAfterSemi from ArcMigrate, which will be changed to use the Lexer utility). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136268 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-27Add support for C++0x unicode string and character literals, from Craig Topper!Douglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136210 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-26Migrate 'Instantiation' data and API bits of SLocEntry to 'Expansion'Chandler Carruth
etc. With this I think essentially all of the SourceManager APIs are converted. Comments and random other bits of cleanup should be all thats left. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136057 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-26Convert InstantiationInfo and much of the related code to ExpansionInfoChandler Carruth
and various other 'expansion' based terms. I've tried to reformat where appropriate and catch as many references in comments but I'm going to do several more passes. Also I've tried to expand parameter names to be more clear where appropriate. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136056 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-26Rename create(MacroArg)InstantiationLoc to create(MacroArg)ExpansionLoc.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@136054 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-25Rename SourceManager::getImmediateInstantiationRange toChandler Carruth
getImmediateExpansionRange. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135960 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-25Rename SourceManager::getInstantiationRange to getExpansionRange.Chandler Carruth
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135915 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-25Mechanically rename SourceManager::getInstantiationLoc andChandler Carruth
FullSourceLoc::getInstantiationLoc to ...::getExpansionLoc. This is part of the API and documentation update from 'instantiation' as the term for macros to 'expansion'. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135914 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-23remove unneeded llvm:: namespace qualifiers on some core types now that ↵Chris Lattner
LLVM.h imports them into the clang namespace. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135852 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-20SpellingJoerg Sonnenberger
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135545 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-19Revamp the SourceManager to separate the representation of parsedDouglas Gregor
source locations from source locations loaded from an AST/PCH file. Previously, loading an AST/PCH file involved carefully pre-allocating space at the beginning of the source manager for the source locations and FileIDs that correspond to the prefix, and then appending the source locations/FileIDs used for parsing the remaining translation unit. This design forced us into loading PCH files early, as a prefix, whic has become a rather significant limitation. This patch splits the SourceManager space into two parts: for source location "addresses", the lower values (growing upward) are used to describe parsed code, while upper values (growing downward) are used for source locations loaded from AST/PCH files. Similarly, positive FileIDs are used to describe parsed code while negative FileIDs are used to file/macro locations loaded from AST/PCH files. As a result, we can load PCH/AST files even during parsing, making various improvemnts in the future possible, e.g., teaching #include <foo.h> to look for and load <foo.h.gch> if it happens to be already available. This patch was originally written by Sebastian Redl, then brought forward to the modern age by Jonathan Turner, and finally polished/finished by me to be committed. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135484 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-14Convert terminology in the Lexer from 'instantiate' and variants toChandler Carruth
'expand'. Also update the public API it provides to the new term, and propagate that update to the various clients. No functionality changed. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@135138 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07Move SourceManager::isAt[Start/End]OfMacroInstantiation functions to the ↵Argyrios Kyrtzidis
Lexer, since they depend on it now. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134644 91177308-0d34-0410-b5e6-96231b3b80d8
2011-07-07Make the Preprocessor more memory efficient and improve macro instantiation ↵Argyrios Kyrtzidis
diagnostics. When a macro instantiation occurs, reserve a SLocEntry chunk with length the full length of the macro definition source. Set the spelling location of this chunk to point to the start of the macro definition and any tokens that are lexed directly from the macro definition will get a location from this chunk with the appropriate offset. For any tokens that come from argument expansion, '##' paste operator, etc. have their instantiation location point at the appropriate place in the instantiated macro definition (the argument identifier and the '##' token respectively). This improves macro instantiation diagnostics: Before: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:5:11: note: instantiated from: int y = M(/); ^ After: t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int') int y = M(/); ^~~~ t.c:3:20: note: instantiated from: \#define M(op) (foo op 3); ~~~ ^ ~ t.c:5:11: note: instantiated from: int y = M(/); ^ The memory savings for a candidate boost library that abuses the preprocessor are: - 32% less SLocEntries (37M -> 25M) - 30% reduction in PCH file size (900M -> 635M) - 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M) git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134587 91177308-0d34-0410-b5e6-96231b3b80d8
2011-06-24Allow Lexer::getLocForEndOfToken to return the location just passed the ↵Argyrios Kyrtzidis
macro instantiation if the location given points at the last token of the macro instantiation. Fixes rdar://9045701. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@133804 91177308-0d34-0410-b5e6-96231b3b80d8
2011-05-10Don't strlen() every file before parsing it.Eli Friedman
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@131132 91177308-0d34-0410-b5e6-96231b3b80d8
2011-04-15fix a bunch of comment typos found by codespell. Patch byChris Lattner
Luis Felipe Strano Moraes! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@129559 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-05-05 04:20:07 +0000