aboutsummaryrefslogtreecommitdiff
path: root/lib/Transforms/Vectorize
AgeCommit message (Collapse)Author
2012-12-18LoopVectorize: Emit reductions as log2(vectorsize) shuffles + vector ops ↵Benjamin Kramer
instead of scalar operations. For example on x86 with SSE4.2 a <8 x i8> add reduction becomes movdqa %xmm0, %xmm1 movhlps %xmm1, %xmm1 ## xmm1 = xmm1[1,1] paddw %xmm0, %xmm1 pshufd $1, %xmm1, %xmm0 ## xmm0 = xmm1[1,0,0,0] paddw %xmm1, %xmm0 phaddw %xmm0, %xmm0 pextrb $0, %xmm0, %edx instead of pextrb $2, %xmm0, %esi pextrb $0, %xmm0, %edx addb %sil, %dl pextrb $4, %xmm0, %esi addb %dl, %sil pextrb $6, %xmm0, %edx addb %sil, %dl pextrb $8, %xmm0, %esi addb %dl, %sil pextrb $10, %xmm0, %edi pextrb $14, %xmm0, %edx addb %sil, %dil pextrb $12, %xmm0, %esi addb %dil, %sil addb %sil, %dl git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@170439 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-13Enable the Loop Vectorizer by default for O2 and O3. Disable if-conversion ↵Nadav Rotem
by default. I plan to revert this patch later today. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@170157 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-13Teach the cost model about the optimization in r169904: Truncation of ↵Nadav Rotem
induction variables costs the same as scalar trunc. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@170051 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-12Fix indentation.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@170005 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-12LoopVectorizer: Use the "optsize" attribute to decide if we are allowed to ↵Nadav Rotem
increase the function size. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@170004 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-12Fix the ascii drawing that was ruined when I split the H and CPPNadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169955 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-12fix a typo.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169953 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-12LoopVectorizer: When -Os is used, vectorize only loops that dont require a ↵Nadav Rotem
tail loop. There is no testcase because I dont know of a way to initialize the loop vectorizer pass without adding an additional hidden flag. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169950 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-11PR14574. Fix a bug in the code that calculates the mask the converted PHIs ↵Nadav Rotem
in if-conversion. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169916 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-11Loop Vectorize: optimize the vectorization of trunc(induction_var). The ↵Nadav Rotem
truncation is now done on scalars. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169904 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-11Fix PR14565. Don't if-convert loops that have switch statements in them.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169813 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-10Split the LoopVectorizer into H and CPP.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169771 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-10Add support for reverse induction variables. For example:Nadav Rotem
while (i--) sum+=A[i]; git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169752 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-09LoopVectorize: support vectorizing intrinsic callsPaul Redmond
- added function to VectorTargetTransformInfo to query cost of intrinsics - vectorize trivially vectorizable intrinsic calls such as sin, cos, log, etc. Reviewed by: Nadav git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169711 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-09test commit.Paul Redmond
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169709 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04LoopVectorizer: Increase the number of pointers that can be tested at ↵Nadav Rotem
runtime. If we cant prove statically that the pointers are disjoint then we add the runtime check. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169334 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04Enable if-conversion during vectorization.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169331 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04Fix a bug in vectorization of if-converted reduction variables. If theNadav Rotem
reduction variable is not used outside the loop then we ran into an endless loop. This change checks if we found the original PHI. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169324 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04Add support for reduction variables when IF-conversion is enabled. Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169288 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04Give scalar if-converted blocks half the score because they are not always ↵Nadav Rotem
executed due to CF. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169223 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04Add the last part that is needed for vectorization of if-converted code.Nadav Rotem
Added the code that actually performs the if-conversion during vectorization. We can now vectorize this code: for (int i=0; i<n; ++i) { unsigned k = 0; if (a[i] > b[i]) <------ IF inside the loop. k = k * 5 + 3; a[i] = k; <---- K is a phi node that becomes vector-select. } git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169217 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04LoopVectorize.cpp: Suppress a warning. [-Wunused-variable]NAKAMURA Takumi
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169195 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-04Fix whitespace.NAKAMURA Takumi
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169194 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-03minor renaming, documentation and cleanups.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169175 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-03IF-conversion: teach the cost-model how to grade if-converted loops.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169171 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-03Now that we have a basic if-conversion infrastructure we can rename theNadav Rotem
"single basic block loop vectorizer" to "innermost loop vectorizer". git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169158 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-03Add initial support for IF-conversion. This patch implements the first 1/3,Nadav Rotem
which is the legality of the if-conversion transformation. The next step is to implement the cost-model for the if-converted code as well as the vectorization itself. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169152 91177308-0d34-0410-b5e6-96231b3b80d8
2012-12-03Use the new script to sort the includes of every file under lib.Chandler Carruth
Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169131 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-30minor cleanupsNadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169048 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-30Remove the use of LPPassManager. We can remove LPM because we dont need to ↵Nadav Rotem
run any additional loop passes on the new vector loop. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@169016 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-29When broadcasting invariant scalars into vectors, place the broadcast code ↵Nadav Rotem
in the preheader. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168927 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-28BBVectorize: Correctly merge SubclassOptionalDataHal Finkel
When two instructions are combined into a vector instruction, the resulting instruction must have the most-conservative flags. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168765 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-26Move the code that uses SCEVs prior to creating the new loops. Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168601 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-25Move the max vector width to a constant parameter. No functionality change.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168570 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-25Fix the document style.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168569 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-25Refactor the ptr runtime check generation code. No functionality change.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168568 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-25Rename method. No functionality change.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168560 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-25The induction-pointer work is inspired by a research paper. This commit adds ↵Nadav Rotem
a reference. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168559 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-25Add support for pointer induction variables even when there is no integer ↵Nadav Rotem
induction variable. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168558 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-17LoopVectorizer: Add initial support for pointer induction variables (for ↵Nadav Rotem
example: *dst++ = *src++). At the moment we still require to have an integer induction variable (for example: i++). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168231 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-16LoopVectorize: Division reductions generate incorrect code. Remove the part ↵Nadav Rotem
of the code that deals with divs. Thanks to Paul Redmond for catching this while reviewing the code. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168142 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-14Replace std::vector -> SmallVector in BBVectorizeHal Finkel
For now, this uses 8 on-stack elements. I'll need to do some profiling to see if this is the best number. Pointed out by Jakob in post-commit review. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167966 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-14Fix the largest offender of determinism in BBVectorizeHal Finkel
Iterating over the children of each node in the potential vectorization plan must happen in a deterministic order (because it affects which children are erased when two children conflict). There was no need for this data structure to be a map in the first place, so replacing it with a vector is a small change. I believe that this was the last remaining instance if iterating over the elements of a Dense* container where the iteration order could matter. There are some remaining iterations over std::*map containers where the order might matter, but so long as the Value* for instructions in a block increase with the order of the instructions in the block (or decrease) monotonically, then this will appear to be deterministic. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167942 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-14use the getSplat API. Patch by Paul Redmond.Nadav Rotem
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167892 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-13BBVectorize: Remove temporary assert used for debuggingHal Finkel
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167817 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-13BBVectorize: Don't vectorize vector-manipulation chainsHal Finkel
Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167811 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-12BBVectorize: Only some insert element operand pairs are free.Hal Finkel
This fixes another infinite recursion case when using target costs. We can only replace insert element input chains that are pure (end with inserting into an undef). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167784 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-12BBVectorize: Use a more sophisticated check for input costHal Finkel
The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167750 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-12BBVectorize: Check the types of compare instructionsHal Finkel
The pass would previously assert when trying to compute the cost of compare instructions with illegal vector types (like struct pointers). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167743 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-12BBVectorize: Check the input types of shuffles for legalityHal Finkel
This fixes a bug where shuffles were being fused such that the resulting input types were not legal on the target. This would occur only when both inputs and dependencies were also foldable operations (such as other shuffles) and there were other connected pairs in the same block. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167731 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.2.3-18-g5258 (git 2.32.0) at 2025-05-16 12:01:36 +0000