aboutsummaryrefslogtreecommitdiff
path: root/lib/Lex/Lexer.cpp
AgeCommit message (Collapse)Author
2009-01-26Check in the long promised SourceLocation rewrite. This lays theChris Lattner
ground work for implementing #line, and fixes the "out of macro ID's" problem. There is nothing particularly tricky about the code, other than the very performance sensitive SourceManager::getFileID() method. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62978 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-23This is a follow-up to r62675:Chris Lattner
Refactor how the preprocessor changes a token from being an tok::identifier to a keyword (e.g. tok::kw_for). Instead of doing this in HandleIdentifier, hoist this common case out into the caller, so that every keyword doesn't have to go through HandleIdentifier. This drops time in HandleIdentifier from 1.25ms to .62ms, and speeds up clang -Eonly with PTH by about 1%. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62855 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-21a trivial micro optimization to save a load.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62676 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-21Add a bit to IdentifierInfo that acts as a simple predicate whichChris Lattner
tells us whether Preprocessor::HandleIdentifier needs to be called. Because this method is only rarely needed, this saves a call and a bunch of random checks. This drops the time in HandleIdentifier from 3.52ms to .98ms on cocoa.h on my machine. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62675 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-19Rename SourceManager::getCanonicalFileID -> getFileID. There isChris Lattner
no longer such thing as a non-canonical FileID. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62499 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-19Make SourceLocation::getFileLoc private to reduce the API exposure of Chris Lattner
SourceLocation. This requires making some cleanups to token pasting and _Pragma expansion. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62490 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17Rename SourceLocation::getFileID to getChunkID, because it returnsChris Lattner
the chunk ID not the file ID. This exposes problems in TextDiagnosticPrinter where it should have been using the canonical file ID but wasn't. Fix these along the way. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62427 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17simplify some lookups.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62426 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17Change the Lexer ctor used to lex _Pragma directives into a static factoryChris Lattner
method. This lets us clean up the interface and make it more obvious that this method is *really really* _Pragma specific. Note that _Pragma handling uglifies the Lexer in the critical path. It would be very interesting to consider making _Pragma remapping be a new special lexer class of its own. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62425 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17Change the Lexer ctor used in the non _Pragma case to take a FileID insteadChris Lattner
of a SourceLocation. This should speed it up and definitely simplifies it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62422 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17More simplifications to the lexer ctors.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62419 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17make the verbose raw-lexer ctor fully explicit instead of havingChris Lattner
embedded magic. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62417 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17add a simplified lexer ctor that sets up the lexer to raw-lex anChris Lattner
entire file. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62414 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17refactor some common initialization code out of the two lexer ctors.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62411 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17this massive patch introduces a simple new abstraction: it makesChris Lattner
"FileID" a concept that is now enforced by the compiler's type checker instead of yet-another-random-unsigned floating around. This is an important distinction from the "FileID" currently tracked by SourceLocation. *That* FileID may refer to the start of a file or to a chunk within it. The new FileID *only* refers to the file (and its #include stack and eventually #line data), it cannot refer to a chunk. FileID is a completely opaque datatype to all clients, only SourceManager is allowed to poke and prod it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62407 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-17Instead of iterating over FileID's, have PTH generation iterate over theChris Lattner
content cache directly. Content cache has a 1-1 mapping with fileentries, whereas multiple FileIDs can be the same FileEntry. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62401 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-16Fix PR2477 - clang misparses "//*" in C89 modeChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62368 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-16more SourceLocation lexicon change: instead of referring to theChris Lattner
"logical" location, refer to the "instantiation" location. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62316 91177308-0d34-0410-b5e6-96231b3b80d8
2009-01-16Change some terminology in SourceLocation: instead of referring to Chris Lattner
the "physical" location of tokens, refer to the "spelling" location. This is more concrete and useful, tokens aren't really physical objects! git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@62309 91177308-0d34-0410-b5e6-96231b3b80d8
2008-12-12rdar://6060752 - don't warn about trigraphs in bcpl-style commentsChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60942 91177308-0d34-0410-b5e6-96231b3b80d8
2008-12-12fix thought-oChris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60937 91177308-0d34-0410-b5e6-96231b3b80d8
2008-12-01Objective-C keywords are not always identifiers. Some are also C++ keywordsDouglas Gregor
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@60373 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-25Comment fix.Daniel Dunbar
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59997 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-22Fix a weird inconsistency with hex floats. Previously the lexer Chris Lattner
would not eat the "-1" in "0x0p-1", but LiteralSupport would accept it when extensions are on. This caused strangeness and failures when hexfloats were properly treated as an extension (not error) in LiteralSupport. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59865 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-22actually, this version isn't really needed.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59859 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-22remove a sneaky version of Diag hiding in PreprocessorLexer.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59858 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-22Change the Lexer::Diag method to not magically silence warnings,Chris Lattner
force the caller to check instead. This eliminates the need (and the risk!) of weird null DiagnosticBuilder's floating around. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59856 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-22Split the DiagnosticInfo class into two disjoint classes:Chris Lattner
one for building up the diagnostic that is in flight (DiagnosticBuilder) and one for pulling structured information out of the diagnostic when formatting and presenting it. There is no functionality change with this patch. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59849 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-19- Move static function IsNonPragmaNonMacroLexer into Preprocessor.h.Ted Kremenek
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry (simplifies some uses). - Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile. - Add 'FileID' to PreprocessorLexer, and have Preprocessor query this fileid when looking up the FileEntry for a file Performance testing of -Eonly on Cocoa.h shows no performance regression because of this patch. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59666 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-18Convert the lexer and start converting the PP over to using canonical Diag ↵Chris Lattner
methods. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59511 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-12Move some diagnostic handling to PreprocessorLexer.Ted Kremenek
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59191 91177308-0d34-0410-b5e6-96231b3b80d8
2008-11-12Remove Lexer::LexIncludeFilename.Ted Kremenek
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@59186 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Change FormTokenWithChars to take the token kind to form, since all clientsChris Lattner
were setting a kind and then forming it. This is just a minor API cleanup, no functionality change. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57404 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12When in keep whitespace mode, make sure to return block comments that areChris Lattner
unterminated. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57403 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Change SkipBlockComment and SkipBCPLComment to return true when inChris Lattner
keep comment mode, instead of returning false. This matches SkipWhitespace. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57402 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Add a new mode to the lexer which enables it to return all characters,Chris Lattner
even whitespace, as tokens from the file. This is enabled with L->SetKeepWhitespaceMode(true) on a raw lexer. In this mode, you too can use clang as a really complex version of 'cat' with code like this: Lexer RawLex(SourceLocation::getFileLoc(SM.getMainFileID(), 0), PP.getLangOptions(), File.first, File.second); RawLex.SetKeepWhitespaceMode(true); Token RawTok; RawLex.LexFromRawLexer(RawTok); while (RawTok.isNot(tok::eof)) { std::cout << PP.getSpelling(RawTok); RawLex.LexFromRawLexer(RawTok); } This will emit exactly the input file, with no canonicalization or other translation. Realistic clients actually do something with the tokens of course :) git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57401 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Fix a couple more places that poke KeepCommentMode unnecesarily. Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57398 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12add a new inKeepCommentMode() accessor to abstract the KeepCommentModeChris Lattner
ivar. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57397 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12fix misleading comment.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57396 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Simplify raw mode lexing by treating an unterminate /**/ comment theChris Lattner
same we we do an unterminated string or character literal. This makes it so we can guarantee that the lexer never calls into the preprocessor (which would be suicide for a raw lexer). git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57395 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12add a comment.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57394 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12Change how raw lexers are handled: instead of creating them and thenChris Lattner
using LexRawToken, create one and use LexFromRawLexer. This avoids twiddling the RawLexer flag around and simplifies some code (even speeding raw lexing up a tiny bit). This change also improves the token paster to use a Lexer on the stack instead of new/deleting it. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57393 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12silence some release-assert warnings.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57391 91177308-0d34-0410-b5e6-96231b3b80d8
2008-10-12improve a comment.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@57389 91177308-0d34-0410-b5e6-96231b3b80d8
2008-08-14Change Parser & Sema to use interned "super" for comparions.Daniel Dunbar
- Added as private members for each because it is not clear where to put the common definition. Perhaps the IdentifierInfos all of these "pseudo-keywords" should be collected into one place (this would KnownFunctionIDs and Objective-C property IDs, for example). Remove Token::isNamedIdentifier. - There isn't a good reason to use strcmp when we have interned strings, and there isn't a good reason to encourage clients to do so. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@54794 91177308-0d34-0410-b5e6-96231b3b80d8
2008-04-14Fix typoNate Begeman
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@49632 91177308-0d34-0410-b5e6-96231b3b80d8
2008-04-12don't diagnose empty source files, thanks Neil!Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@49575 91177308-0d34-0410-b5e6-96231b3b80d8
2008-04-11don't read off the front of the buffer. Thanks to Sam for pointing this out.Chris Lattner
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@49535 91177308-0d34-0410-b5e6-96231b3b80d8
2008-03-15Make a major restructuring of the clang tree: introduce a top-levelChris Lattner
lib dir and move all the libraries into it. This follows the main llvm tree, and allows the libraries to be built in parallel. The top level now enforces that all the libs are built before Driver, but we don't care what order the libs are built in. This speeds up parallel builds, particularly incremental ones. git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@48402 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-05-13 04:53:34 +0000