aboutsummaryrefslogtreecommitdiff
path: root/lib/Transforms/Vectorize/LoopVectorize.cpp
AgeCommit message (Collapse)Author
2013-05-14LoopVectorize: Handle loops with multiple forward inductionsArnold Schwaighofer
We used to give up if we saw two integer inductions. After this patch, we base further induction variables on the chosen one like we do in the reverse induction and pointer induction case. Fixes PR15720. radar://13851975 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181746 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-11LoopVectorize: Use the widest induction variable typeArnold Schwaighofer
Use the widest induction type encountered for the cannonical induction variable. We used to turn the following loop into an empty loop because we used i8 as induction variable type and truncated 1024 to 0 as trip count. int a[1024]; void fail() { int reverse_induction = 1023; unsigned char forward_induction = 0; while ((reverse_induction) >= 0) { forward_induction++; a[reverse_induction] = forward_induction; --reverse_induction; } } radar://13862901 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181667 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-11LoopVectorize: Use variable instead of repeated function callArnold Schwaighofer
No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181666 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-11LoopVectorize: Use IRBuilder interface in more placesArnold Schwaighofer
No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181665 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-09LoopVectorizer: Don't assert on the absence of induction variablesArnold Schwaighofer
A computable loop exit count does not imply the presence of an induction variable. Scalar evolution can return a value for an infinite loop. Fixes PR15926. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181495 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-07LoopVectorizer: Improve reduction variable identificationArnold Schwaighofer
The two nested loops were confusing and also conservative in identifying reduction variables. This patch replaces them by a worklist based approach. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181369 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-07LoopVectorize: getConsecutiveVector must respect signed arithmeticArnold Schwaighofer
We were passing an i32 to ConstantInt::get where an i64 was needed and we must also pass the sign if we pass negatives numbers. The start index passed to getConsecutiveVector must also be signed. Should fix PR15882. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181286 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-06Update the comment to mention that we use TTI.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181178 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-05LoopVectorize: Print values instead of pointers in debug output.Benjamin Kramer
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181157 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-05LoopVectorize: Add support for floating point min/max reductionsArnold Schwaighofer
Add support for min/max reductions when "no-nans-float-math" is enabled. This allows us to assume we have ordered floating point math and treat ordered and unordered predicates equally. radar://13723044 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181144 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-05LoopVectorizer: Cleanup of miminimum/maximum pattern match codeArnold Schwaighofer
No need for setting the operands. The pointers are going to be bound by the matcher. radar://13723044 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181142 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-05LoopVectorize: We don't need an identity element for min/max reductionsArnold Schwaighofer
We can just use the initial element that feeds the reduction. max(max(x, y), z) == max(max(x,y), max(x,z)) radar://13723044 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181141 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-05Add ArrayRef constructor from None, and do the cleanups that this ↵Dmitri Gribenko
constructor enables Patch by Robert Wilhelm. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181138 91177308-0d34-0410-b5e6-96231b3b80d8
2013-05-03LoopVectorizer: Add support for if-conversion of PHINodes with 3+ incoming ↵Nadav Rotem
values. By supporting the vectorization of PHINodes with more than two incoming values we can increase the complexity of nested if statements. We can now vectorize this loop: int foo(int *A, int *B, int n) { for (int i=0; i < n; i++) { int x = 9; if (A[i] > B[i]) { if (A[i] > 19) { x = 3; } else if (B[i] < 4 ) { x = 4; } else { x = 5; } } A[i] = x; } } git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@181037 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-26LoopVectorizer: Calculate the number of pointers to disambiguate at runtime ↵Nadav Rotem
based on the numbers of reads and writes. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180593 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-25LoopVectorizer: No need to generate pointer disambiguation checks between ↵Nadav Rotem
readonly pointers. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180570 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-24LoopVectorizer: Change variable name Stride to ConsecutiveStrideArnold Schwaighofer
This makes it easier to read the code. No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180197 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-24LoopVectorize: Scalarize padded typesArnold Schwaighofer
This patch disables memory-instruction vectorization for types that need padding bytes, e.g., x86_fp80 has 10 bytes store size with 6 bytes padding in darwin on x86_64. Because the load/store vectorization is performed by the bit casting to a packed vector, which has incompatible memory layout due to the lack of padding bytes, the present vectorizer produces inconsistent result for memory instructions of those types. This patch checks an equality of the AllocSize of a scalar type and allocated size for each vector element, to ensure that there is no padding bytes and the array can be read/written using vector operations. Patch by Daisuke Takahashi! Fixes PR15758. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180196 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-24LoopVectorizer: Bail out if we don't have datalayout we need itArnold Schwaighofer
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180195 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-23LoopVectorizer: Fix 15830. When scalarizing and unrolling stores make sure ↵Nadav Rotem
that the order in which the elements are scalarized is the same as the original order. This fixes a miscompilation in FreeBSD's regex library. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180121 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-23Call the potentially costly isAnnotatedParallel() only once. Pekka Jaaskelainen
Made the uniform write test's checks a bit stricter. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180119 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-23Refuse to (even try to) vectorize loops which have uniform writes,Pekka Jaaskelainen
even if erroneously annotated with the parallel loop metadata. Fixes Bug 15794: "Loop Vectorizer: Crashes with the use of llvm.loop.parallel metadata" git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180081 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-19LoopVectorizer: Use matcher from PatternMatch.h for the min/max patternsArnold Schwaighofer
Also make some static function class functions to avoid having to mention the class namespace for enums all the time. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179886 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-18Fix a -Wdocumentation warningDmitri Gribenko
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179789 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-18LoopVectorizer: Recognize min/max reductionsArnold Schwaighofer
A min/max operation is represented by a select(cmp(lt/le/gt/ge, X, Y), X, Y) sequence in LLVM. If we see such a sequence we can treat it just as any other commutative binary instruction and reduce it. This appears to help bzip2 by about 1.5% on an imac12,2. radar://12960601 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179773 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-18LoopVectorize: Use a set to avoid longer cycles in the reduction chain too.Benjamin Kramer
Fixes PR15748. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179757 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-12LoopVectorizer: integer division is not a reduction operationArnold Schwaighofer
Don't classify idiv/udiv as a reduction operation. Integer division is lossy. For example : (1 / 2) * 4 != 4/2. Example: int a[] = { 2, 5, 2, 2} int x = 80; for() x /= a[i]; Scalar: x /= 2 // = 40 x /= 5 // = 8 x /= 2 // = 4 x /= 2 // = 2 Vectorized: <80, 1> / <2,5> //= <40,0> <40, 0> / <2,2> //= <20,0> 20*0 = 0 radar://13640654 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179381 91177308-0d34-0410-b5e6-96231b3b80d8
2013-04-04LoopVectorizer: Pass OperandValueKind information to the cost modelArnold Schwaighofer
Pass down the fact that an operand is going to be a vector of constants. This should bring the performance of MultiSource/Benchmarks/PAQ8p/paq8p on x86 back. It had degraded to scalar performance due to my pervious shift cost change that made all shifts expensive on x86. radar://13576547 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@178809 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-14LoopVectorize: Invert case when we use a vector cmp value to query select costArnold Schwaighofer
We generate a select with a vectorized condition argument when the condition is NOT loop invariant. Not the other way around. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@177098 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-09Remove a source of nondeterminism from the LoopVectorizer.Benjamin Kramer
This made us emit runtime checks in a random order. Hopefully bootstrap miscompares will go away now. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176775 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-09LoopVectorizer: Ignore all dbg intrinisicArnold Schwaighofer
Ignore all DbgIntriniscInfo instructions instead of just DbgValueInst. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176769 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-09LoopVectorizer: Ignore dbg.value instructionsArnold Schwaighofer
We want vectorization to happen at -g. Ignore calls to the dbg.value intrinsic and don't transfer them to the vectorized code. radar://13378964 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176768 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-08Insert the reduction start value into the first bypass block to preserve ↵Benjamin Kramer
domination. Fixes PR15344. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176701 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-02PR14448 - prevent the loop vectorizer from vectorizing the same loop twice.Nadav Rotem
The LoopVectorizer often runs multiple times on the same function due to inlining. When this happens the loop vectorizer often vectorizes the same loops multiple times, increasing code size and adding unneeded branches. With this patch, the vectorizer during vectorization puts metadata on scalar loops and marks them as 'already vectorized' so that it knows to ignore them when it sees them a second time. PR14448. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176399 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-01LoopVectorize: Don't hang forever if a PHI only has skipped PHI uses.Benjamin Kramer
Fixes PR15384. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176366 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-27LoopVectorize: Vectorize math builtin calls.Benjamin Kramer
This properly asks TargetLibraryInfo if a call is available and if it is, it can be translated into the corresponding LLVM builtin. We don't vectorize sqrt() yet because I'm not sure about the semantics for negative numbers. The other intrinsic should be exact equivalents to the libm functions. Differential Revision: http://llvm-reviews.chandlerc.com/D465 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176188 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-21Allow GlobalValues to vectorize with AliasAnalysisRenato Golin
Storing the load/store instructions with the values and inspect them using Alias Analysis to make sure they don't alias, since the GEP pointer operand doesn't take the offset into account. Trying hard to not add any extra cost to loads and stores that don't overlap on global values, AA is *only* calculated if all of the previous attempts failed. Using biggest vector register size as the stride for the vectorization access, as we're being conservative and the cost model (which calculates the real vectorization factor) is only run after the legalization phase. We might re-think this relationship in the future, but for now, I'd rather be safe than sorry. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175818 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-13LoopVectorize: Simplify code for clarity.Benjamin Kramer
No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175076 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-13Metadata for annotating loops as parallel. The first consumer for this Pekka Jaaskelainen
metadata is the loop vectorizer. See the documentation update for more info. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175060 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-08Typos.Jakob Stoklund Olesen
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174723 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-08ARM cost model: Address computation in vector mem ops not freeArnold Schwaighofer
Adds a function to target transform info to query for the cost of address computation. The cost model analysis pass now also queries this interface. The code in LoopVectorize adds the cost of address computation as part of the memory instruction cost calculation. Only there, we know whether the instruction will be scalarized or not. Increase the penality for inserting in to D registers on swift. This becomes necessary because we now always assume that address computation has a cost and three is a closer value to the architecture. radar://13097204 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174713 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-08Test CommitMichael Kuperstein
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174709 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-07fix 80-col violation and fix the docs.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174671 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-07Loop Vectorizer: Refactor Memory Cost ComputationArnold Schwaighofer
We don't want too many classes in a pass and the classes obscure the details. I was going a little overboard with object modeling here. Replace classes by generic code that handles both loads and stores. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174646 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-05Loop Vectorizer: Refactor code to compute vectorized memory instruction costArnold Schwaighofer
Introduce a helper class that computes the cost of memory access instructions. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174422 91177308-0d34-0410-b5e6-96231b3b80d8
2013-02-05Loop Vectorizer: Handle pointer stores/loads in getWidestType()Arnold Schwaighofer
In the loop vectorizer cost model, we used to ignore stores/loads of a pointer type when computing the widest type within a loop. This meant that if we had only stores/loads of pointers in a loop we would return a widest type of 8bits (instead of 32 or 64 bit) and therefore a vector factor that was too big. Now, if we see a consecutive store/load of pointers we use the size of a pointer (from data layout). This problem occured in SingleSource/Benchmarks/Shootout-C++/hash.cpp (reduced test case is the first test in vector_ptr_load_store.ll). radar://13139343 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174377 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-29LoopVectorize: convert TinyTripCountVectorThreshold constantPekka Jaaskelainen
to a command line switch. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173837 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-29LoopVectorize: Clean up ValueMap a bit and avoid double lookups.Benjamin Kramer
No intended functionality change. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173809 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-28Vectorization Factor clarificationRenato Golin
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173691 91177308-0d34-0410-b5e6-96231b3b80d8
2013-01-25LoopVectorize: Refactor the code that vectorizes loads/stores to remove ↵Nadav Rotem
duplication. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173500 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-04-30 12:17:40 +0000