diff options
Diffstat (limited to 'Documentation/atomic_ops.txt')
| -rw-r--r-- | Documentation/atomic_ops.txt | 202 |
1 files changed, 174 insertions, 28 deletions
diff --git a/Documentation/atomic_ops.txt b/Documentation/atomic_ops.txt index 2a63d5662a9..68542fe13b8 100644 --- a/Documentation/atomic_ops.txt +++ b/Documentation/atomic_ops.txt @@ -12,10 +12,17 @@ Also, it should be made opaque such that any kind of cast to a normal C integer type will fail. Something like the following should suffice: - typedef struct { volatile int counter; } atomic_t; + typedef struct { int counter; } atomic_t; - The first operations to implement for atomic_t's are the -initializers and plain reads. +Historically, counter has been declared volatile. This is now discouraged. +See Documentation/volatile-considered-harmful.txt for the complete rationale. + +local_t is very similar to atomic_t. If the counter is per CPU and only +updated by one CPU, local_t is probably more appropriate. Please see +Documentation/local_ops.txt for the semantics of local_t. + +The first operations to implement for atomic_t's are the initializers and +plain reads. #define ATOMIC_INIT(i) { (i) } #define atomic_set(v, i) ((v)->counter = (i)) @@ -24,6 +31,12 @@ The first macro is used in definitions, such as: static atomic_t my_counter = ATOMIC_INIT(1); +The initializer is atomic in that the return values of the atomic operations +are guaranteed to be correct reflecting the initialized value if the +initializer is used before runtime. If the initializer is used at runtime, a +proper implicit or explicit read memory barrier is needed before reading the +value with atomic_read from another thread. + The second interface can be used at runtime, as in: struct foo { atomic_t counter; }; @@ -36,13 +49,130 @@ The second interface can be used at runtime, as in: return -ENOMEM; atomic_set(&k->counter, 0); +The setting is atomic in that the return values of the atomic operations by +all threads are guaranteed to be correct reflecting either the value that has +been set with this operation or set with another operation. A proper implicit +or explicit memory barrier is needed before the value set with the operation +is guaranteed to be readable with atomic_read from another thread. + Next, we have: #define atomic_read(v) ((v)->counter) -which simply reads the current value of the counter. +which simply reads the counter value currently visible to the calling thread. +The read is atomic in that the return value is guaranteed to be one of the +values initialized or modified with the interface operations if a proper +implicit or explicit memory barrier is used after possible runtime +initialization by any other thread and the value is modified only with the +interface operations. atomic_read does not guarantee that the runtime +initialization by any other thread is visible yet, so the user of the +interface must take care of that with a proper implicit or explicit memory +barrier. + +*** WARNING: atomic_read() and atomic_set() DO NOT IMPLY BARRIERS! *** + +Some architectures may choose to use the volatile keyword, barriers, or inline +assembly to guarantee some degree of immediacy for atomic_read() and +atomic_set(). This is not uniformly guaranteed, and may change in the future, +so all users of atomic_t should treat atomic_read() and atomic_set() as simple +C statements that may be reordered or optimized away entirely by the compiler +or processor, and explicitly invoke the appropriate compiler and/or memory +barrier for each use case. Failure to do so will result in code that may +suddenly break when used with different architectures or compiler +optimizations, or even changes in unrelated code which changes how the +compiler optimizes the section accessing atomic_t variables. + +*** YOU HAVE BEEN WARNED! *** + +Properly aligned pointers, longs, ints, and chars (and unsigned +equivalents) may be atomically loaded from and stored to in the same +sense as described for atomic_read() and atomic_set(). The ACCESS_ONCE() +macro should be used to prevent the compiler from using optimizations +that might otherwise optimize accesses out of existence on the one hand, +or that might create unsolicited accesses on the other. + +For example consider the following code: + + while (a > 0) + do_something(); + +If the compiler can prove that do_something() does not store to the +variable a, then the compiler is within its rights transforming this to +the following: + + tmp = a; + if (a > 0) + for (;;) + do_something(); + +If you don't want the compiler to do this (and you probably don't), then +you should use something like the following: + + while (ACCESS_ONCE(a) < 0) + do_something(); + +Alternatively, you could place a barrier() call in the loop. + +For another example, consider the following code: + + tmp_a = a; + do_something_with(tmp_a); + do_something_else_with(tmp_a); + +If the compiler can prove that do_something_with() does not store to the +variable a, then the compiler is within its rights to manufacture an +additional load as follows: + + tmp_a = a; + do_something_with(tmp_a); + tmp_a = a; + do_something_else_with(tmp_a); -Now, we move onto the actual atomic operation interfaces. +This could fatally confuse your code if it expected the same value +to be passed to do_something_with() and do_something_else_with(). + +The compiler would be likely to manufacture this additional load if +do_something_with() was an inline function that made very heavy use +of registers: reloading from variable a could save a flush to the +stack and later reload. To prevent the compiler from attacking your +code in this manner, write the following: + + tmp_a = ACCESS_ONCE(a); + do_something_with(tmp_a); + do_something_else_with(tmp_a); + +For a final example, consider the following code, assuming that the +variable a is set at boot time before the second CPU is brought online +and never changed later, so that memory barriers are not needed: + + if (a) + b = 9; + else + b = 42; + +The compiler is within its rights to manufacture an additional store +by transforming the above code into the following: + + b = 42; + if (a) + b = 9; + +This could come as a fatal surprise to other code running concurrently +that expected b to never have the value 42 if a was zero. To prevent +the compiler from doing this, write something like: + + if (a) + ACCESS_ONCE(b) = 9; + else + ACCESS_ONCE(b) = 42; + +Don't even -think- about doing this without proper use of memory barriers, +locks, or atomic operations if variable a can change at runtime! + +*** WARNING: ACCESS_ONCE() DOES NOT IMPLY A BARRIER! *** + +Now, we move onto the atomic operation interfaces typically implemented with +the help of assembly code. void atomic_add(int i, atomic_t *v); void atomic_sub(int i, atomic_t *v); @@ -117,6 +247,14 @@ operation. Then: + int atomic_xchg(atomic_t *v, int new); + +This performs an atomic exchange operation on the atomic variable v, setting +the given new value. It returns the old value that the atomic variable v had +just before the operation. + +atomic_xchg requires explicit memory barriers around the operation. + int atomic_cmpxchg(atomic_t *v, int old, int new); This performs an atomic compare exchange operation on the atomic value v, @@ -137,7 +275,8 @@ If the atomic value v is not equal to u, this function adds a to v, and returns non zero. If v is equal to u then it returns zero. This is done as an atomic operation. -atomic_add_unless requires explicit memory barriers around the operation. +atomic_add_unless requires explicit memory barriers around the operation +unless it fails (returns 0). atomic_inc_not_zero, equivalent to atomic_add_unless(v, 1, 0) @@ -146,15 +285,13 @@ If a caller requires memory barrier semantics around an atomic_t operation which does not return a value, a set of interfaces are defined which accomplish this: - void smp_mb__before_atomic_dec(void); - void smp_mb__after_atomic_dec(void); - void smp_mb__before_atomic_inc(void); - void smp_mb__after_atomic_dec(void); + void smp_mb__before_atomic(void); + void smp_mb__after_atomic(void); -For example, smp_mb__before_atomic_dec() can be used like so: +For example, smp_mb__before_atomic() can be used like so: obj->dead = 1; - smp_mb__before_atomic_dec(); + smp_mb__before_atomic(); atomic_dec(&obj->ref_count); It makes sure that all memory operations preceding the atomic_dec() @@ -163,15 +300,10 @@ operation. In the above example, it guarantees that the assignment of "1" to obj->dead will be globally visible to other cpus before the atomic counter decrement. -Without the explicit smp_mb__before_atomic_dec() call, the +Without the explicit smp_mb__before_atomic() call, the implementation could legally allow the atomic counter update visible to other cpus before the "obj->dead = 1;" assignment. -The other three interfaces listed are used to provide explicit -ordering with respect to memory operations after an atomic_dec() call -(smp_mb__after_atomic_dec()) and around atomic_inc() calls -(smp_mb__{before,after}_atomic_inc()). - A missing memory barrier in the cases where they are required by the atomic_t implementation above can have disastrous results. Here is an example, which follows a pattern occurring frequently in the Linux @@ -179,10 +311,10 @@ kernel. It is the use of atomic counters to implement reference counting, and it works such that once the counter falls to zero it can be guaranteed that no other entity can be accessing the object: -static void obj_list_add(struct obj *obj) +static void obj_list_add(struct obj *obj, struct list_head *head) { obj->active = 1; - list_add(&obj->list); + list_add(&obj->list, head); } static void obj_list_del(struct obj *obj) @@ -270,7 +402,7 @@ counter decrement would not become globally visible until the obj->active update does. As a historical note, 32-bit Sparc used to only allow usage of -24-bits of it's atomic_t type. This was because it used 8 bits +24-bits of its atomic_t type. This was because it used 8 bits as a spinlock for SMP safety. Sparc32 lacked a "compare and swap" type instruction. However, 32-bit Sparc has since been moved over to a "hash table of spinlocks" scheme, that allows the full 32-bit @@ -348,12 +480,12 @@ Finally there is the basic operation: Which returns a boolean indicating if bit "nr" is set in the bitmask pointed to by "addr". -If explicit memory barriers are required around clear_bit() (which -does not return a value, and thus does not need to provide memory -barrier semantics), two interfaces are provided: +If explicit memory barriers are required around {set,clear}_bit() (which do +not return a value, and thus does not need to provide memory barrier +semantics), two interfaces are provided: - void smp_mb__before_clear_bit(void); - void smp_mb__after_clear_bit(void); + void smp_mb__before_atomic(void); + void smp_mb__after_atomic(void); They are used as follows, and are akin to their atomic_t operation brothers: @@ -361,13 +493,27 @@ brothers: /* All memory operations before this call will * be globally visible before the clear_bit(). */ - smp_mb__before_clear_bit(); + smp_mb__before_atomic(); clear_bit( ... ); /* The clear_bit() will be visible before all * subsequent memory operations. */ - smp_mb__after_clear_bit(); + smp_mb__after_atomic(); + +There are two special bitops with lock barrier semantics (acquire/release, +same as spinlocks). These operate in the same way as their non-_lock/unlock +postfixed variants, except that they are to provide acquire/release semantics, +respectively. This means they can be used for bit_spin_trylock and +bit_spin_unlock type operations without specifying any more barriers. + + int test_and_set_bit_lock(unsigned long nr, unsigned long *addr); + void clear_bit_unlock(unsigned long nr, unsigned long *addr); + void __clear_bit_unlock(unsigned long nr, unsigned long *addr); + +The __clear_bit_unlock version is non-atomic, however it still implements +unlock barrier semantics. This can be useful if the lock itself is protecting +the other bits in the word. Finally, there are non-atomic versions of the bitmask operations provided. They are used in contexts where some other higher-level SMP |