Introduce a caching mechanism for macro expanded tokens.

Previously macro expanded tokens were added to Preprocessor's bump allocator and never released,
even after the TokenLexer that were lexing them was finished, thus they were wasting memory.
A very "useful" boost library was causing clang to eat 1 GB just for the expanded macro tokens.

Introduce a special cache that works like a stack; a TokenLexer can add the macro expanded tokens
in the cache, and when it finishes, the tokens are removed from the end of the cache.

Now consumed memory by expanded tokens for that library is ~ 1.5 MB.

Part of rdar://9327049.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@134105 91177308-0d34-0410-b5e6-96231b3b80d8

1 files changed, 3 insertions, 1 deletions

diff --git a/lib/Lex/Preprocessor.cpp b/lib/Lex/Preprocessor.cpp
index 48a23880f3..2f43c8e30b 100644
--- a/lib/Lex/Preprocessor.cpp
+++ b/lib/Lex/Preprocessor.cpp
@@ -118,6 +118,8 @@ Preprocessor::Preprocessor(Diagnostic &diags, const LangOptions &opts,
 
 Preprocessor::~Preprocessor() {
   assert(BacktrackPositions.empty() && "EnableBacktrack/Backtrack imbalance!");
+  assert(MacroExpandingLexersStack.empty() && MacroExpandedTokens.empty() &&
+         "Preprocessor::HandleEndOfTokenLexer should have cleared those");
 
   while (!IncludeMacroStack.empty()) {
     delete IncludeMacroStack.back().TheLexer;
@@ -226,7 +228,7 @@ Preprocessor::macro_begin(bool IncludeExternalMacros) const {
 }
 
 size_t Preprocessor::getTotalMemory() const {
-  return BP.getTotalMemory();
+  return BP.getTotalMemory() + MacroExpandedTokens.capacity()*sizeof(Token);
 }
 
 Preprocessor::macro_iterator