Introduce a caching mechanism for macro expanded tokens.

Previously macro expanded tokens were added to Preprocessor's bump allocator and never released,
even after the TokenLexer that were lexing them was finished, thus they were wasting memory.
A very "useful" boost library was causing clang to eat 1 GB just for the expanded macro tokens.

Introduce a special cache that works like a stack; a TokenLexer can add the macro expanded tokens
in the cache, and when it finishes, the tokens are removed from the end of the cache.

Now consumed memory by expanded tokens for that library is ~ 1.5 MB.

Part of rdar://9327049.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134105 91177308-0d34-0410-b5e6-96231b3b80d8

1 files changed, 19 insertions, 0 deletions

diff --git a/include/clang/Lex/Preprocessor.h b/include/clang/Lex/Preprocessor.h
index abba959dee..56fc9d1727 100644
--- a/include/clang/Lex/Preprocessor.h
+++ b/include/clang/Lex/Preprocessor.h
@@ -29,6 +29,7 @@
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/OwningPtr.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/ArrayRef.h"
 #include "llvm/Support/Allocator.h"
 #include <vector>
 
@@ -240,6 +241,14 @@ class Preprocessor : public llvm::RefCountedBase<Preprocessor> {
   unsigned NumCachedTokenLexers;
   TokenLexer *TokenLexerCache[TokenLexerCacheSize];
 
+  /// \brief Keeps macro expanded tokens for TokenLexers.
+  //
+  /// Works like a stack; a TokenLexer adds the macro expanded tokens that is
+  /// going to lex in the cache and when it finishes the tokens are removed
+  /// from the end of the cache.
+  llvm::SmallVector<Token, 16> MacroExpandedTokens;
+  std::vector<std::pair<TokenLexer *, size_t> > MacroExpandingLexersStack;
+
   /// \brief A record of the macro definitions and instantiations that
   /// occurred during preprocessing. 
   ///
@@ -979,6 +988,16 @@ private:
   /// the macro should not be expanded return true, otherwise return false.
   bool HandleMacroExpandedIdentifier(Token &Tok, MacroInfo *MI);
 
+  /// \brief Cache macro expanded tokens for TokenLexers.
+  //
+  /// Works like a stack; a TokenLexer adds the macro expanded tokens that is
+  /// going to lex in the cache and when it finishes the tokens are removed
+  /// from the end of the cache.
+  Token *cacheMacroExpandedTokens(TokenLexer *tokLexer,
+                                  llvm::ArrayRef<Token> tokens);
+  void removeCachedMacroExpandedTokensOfLastLexer();
+  friend void TokenLexer::ExpandFunctionArguments();
+
   /// isNextPPTokenLParen - Determine whether the next preprocessor token to be
   /// lexed is a '('.  If so, consume the token and return true, if not, this
   /// method should have no observable side-effect on the lexed tokens.