diff options
Diffstat (limited to 'Documentation/RCU/torture.txt')
| -rw-r--r-- | Documentation/RCU/torture.txt | 212 |
1 files changed, 183 insertions, 29 deletions
diff --git a/Documentation/RCU/torture.txt b/Documentation/RCU/torture.txt index a342b6e1cc1..dac02a6219b 100644 --- a/Documentation/RCU/torture.txt +++ b/Documentation/RCU/torture.txt @@ -30,11 +30,43 @@ MODULE PARAMETERS This module has the following parameters: -irqreaders Says to invoke RCU readers from irq level. This is currently +fqs_duration Duration (in microseconds) of artificially induced bursts + of force_quiescent_state() invocations. In RCU + implementations having force_quiescent_state(), these + bursts help force races between forcing a given grace + period and that grace period ending on its own. + +fqs_holdoff Holdoff time (in microseconds) between consecutive calls + to force_quiescent_state() within a burst. + +fqs_stutter Wait time (in seconds) between consecutive bursts + of calls to force_quiescent_state(). + +gp_normal Make the fake writers use normal synchronous grace-period + primitives. + +gp_exp Make the fake writers use expedited synchronous grace-period + primitives. If both gp_normal and gp_exp are set, or + if neither gp_normal nor gp_exp are set, then randomly + choose the primitive so that about 50% are normal and + 50% expedited. By default, neither are set, which + gives best overall test coverage. + +irqreader Says to invoke RCU readers from irq level. This is currently done via timers. Defaults to "1" for variants of RCU that permit this. (Or, more accurately, variants of RCU that do -not- permit this know to ignore this variable.) +n_barrier_cbs If this is nonzero, RCU barrier testing will be conducted, + in which case n_barrier_cbs specifies the number of + RCU callbacks (and corresponding kthreads) to use for + this testing. The value cannot be negative. If you + specify this to be non-zero when torture_type indicates a + synchronous RCU implementation (one for which a member of + the synchronize_rcu() rather than the call_rcu() family is + used -- see the documentation for torture_type below), an + error will be reported and no testing will be carried out. + nfakewriters This is the number of RCU fake writer threads to run. Fake writer threads repeatedly use the synchronous "wait for current readers" function of the interface selected by @@ -49,11 +81,49 @@ nreaders This is the number of RCU reading threads supported. To properly exercise RCU implementations with preemptible read-side critical sections. +onoff_interval + The number of seconds between each attempt to execute a + randomly selected CPU-hotplug operation. Defaults to + zero, which disables CPU hotplugging. In HOTPLUG_CPU=n + kernels, rcutorture will silently refuse to do any + CPU-hotplug operations regardless of what value is + specified for onoff_interval. + +onoff_holdoff The number of seconds to wait until starting CPU-hotplug + operations. This would normally only be used when + rcutorture was built into the kernel and started + automatically at boot time, in which case it is useful + in order to avoid confusing boot-time code with CPUs + coming and going. + shuffle_interval The number of seconds to keep the test threads affinitied to a particular subset of the CPUs, defaults to 3 seconds. Used in conjunction with test_no_idle_hz. +shutdown_secs The number of seconds to run the test before terminating + the test and powering off the system. The default is + zero, which disables test termination and system shutdown. + This capability is useful for automated testing. + +stall_cpu The number of seconds that a CPU should be stalled while + within both an rcu_read_lock() and a preempt_disable(). + This stall happens only once per rcutorture run. + If you need multiple stalls, use modprobe and rmmod to + repeatedly run rcutorture. The default for stall_cpu + is zero, which prevents rcutorture from stalling a CPU. + + Note that attempts to rmmod rcutorture while the stall + is ongoing will hang, so be careful what value you + choose for this module parameter! In addition, too-large + values for stall_cpu might well induce failures and + warnings in other parts of the kernel. You have been + warned! + +stall_cpu_holdoff + The number of seconds to wait after rcutorture starts + before stalling a CPU. Defaults to 10 seconds. + stat_interval The number of seconds between output of torture statistics (via printk()). Regardless of the interval, statistics are printed when the module is unloaded. @@ -67,17 +137,71 @@ stutter The length of time to run the test before pausing for this Specifying "stutter=0" causes the test to run continuously without pausing, which is the old default behavior. +test_boost Whether or not to test the ability of RCU to do priority + boosting. Defaults to "test_boost=1", which performs + RCU priority-inversion testing only if the selected + RCU implementation supports priority boosting. Specifying + "test_boost=0" never performs RCU priority-inversion + testing. Specifying "test_boost=2" performs RCU + priority-inversion testing even if the selected RCU + implementation does not support RCU priority boosting, + which can be used to test rcutorture's ability to + carry out RCU priority-inversion testing. + +test_boost_interval + The number of seconds in an RCU priority-inversion test + cycle. Defaults to "test_boost_interval=7". It is + usually wise for this value to be relatively prime to + the value selected for "stutter". + +test_boost_duration + The number of seconds to do RCU priority-inversion testing + within any given "test_boost_interval". Defaults to + "test_boost_duration=4". + test_no_idle_hz Whether or not to test the ability of RCU to operate in a kernel that disables the scheduling-clock interrupt to idle CPUs. Boolean parameter, "1" to test, "0" otherwise. Defaults to omitting this test. -torture_type The type of RCU to test: "rcu" for the rcu_read_lock() API, - "rcu_sync" for rcu_read_lock() with synchronous reclamation, - "rcu_bh" for the rcu_read_lock_bh() API, "rcu_bh_sync" for - rcu_read_lock_bh() with synchronous reclamation, "srcu" for - the "srcu_read_lock()" API, and "sched" for the use of - preempt_disable() together with synchronize_sched(). +torture_type The type of RCU to test, with string values as follows: + + "rcu": rcu_read_lock(), rcu_read_unlock() and call_rcu(). + + "rcu_sync": rcu_read_lock(), rcu_read_unlock(), and + synchronize_rcu(). + + "rcu_expedited": rcu_read_lock(), rcu_read_unlock(), and + synchronize_rcu_expedited(). + + "rcu_bh": rcu_read_lock_bh(), rcu_read_unlock_bh(), and + call_rcu_bh(). + + "rcu_bh_sync": rcu_read_lock_bh(), rcu_read_unlock_bh(), + and synchronize_rcu_bh(). + + "rcu_bh_expedited": rcu_read_lock_bh(), rcu_read_unlock_bh(), + and synchronize_rcu_bh_expedited(). + + "srcu": srcu_read_lock(), srcu_read_unlock() and + call_srcu(). + + "srcu_sync": srcu_read_lock(), srcu_read_unlock() and + synchronize_srcu(). + + "srcu_expedited": srcu_read_lock(), srcu_read_unlock() and + synchronize_srcu_expedited(). + + "sched": preempt_disable(), preempt_enable(), and + call_rcu_sched(). + + "sched_sync": preempt_disable(), preempt_enable(), and + synchronize_sched(). + + "sched_expedited": preempt_disable(), preempt_enable(), and + synchronize_sched_expedited(). + + Defaults to "rcu". verbose Enable debug printk()s. Default is disabled. @@ -86,12 +210,12 @@ OUTPUT The statistics output is as follows: - rcu-torture: --- Start of test: nreaders=16 stat_interval=0 verbose=0 - rcu-torture: rtc: 0000000000000000 ver: 1916 tfle: 0 rta: 1916 rtaf: 0 rtf: 1915 - rcu-torture: Reader Pipe: 1466408 9747 0 0 0 0 0 0 0 0 0 - rcu-torture: Reader Batch: 1464477 11678 0 0 0 0 0 0 0 0 - rcu-torture: Free-Block Circulation: 1915 1915 1915 1915 1915 1915 1915 1915 1915 1915 0 - rcu-torture: --- End of test + rcu-torture:--- Start of test: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 + rcu-torture: rtc: (null) ver: 155441 tfle: 0 rta: 155441 rtaf: 8884 rtf: 155440 rtmbe: 0 rtbe: 0 rtbke: 0 rtbre: 0 rtbf: 0 rtb: 0 nt: 3055767 + rcu-torture: Reader Pipe: 727860534 34213 0 0 0 0 0 0 0 0 0 + rcu-torture: Reader Batch: 727877838 17003 0 0 0 0 0 0 0 0 0 + rcu-torture: Free-Block Circulation: 155440 155440 155440 155440 155440 155440 155440 155440 155440 155440 0 + rcu-torture:--- End of test: SUCCESS: nreaders=16 nfakewriters=4 stat_interval=30 verbose=0 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/0 test_boost_interval=7 test_boost_duration=4 The command "dmesg | grep torture:" will extract this information on most systems. On more esoteric configurations, it may be necessary to @@ -99,26 +223,58 @@ use other commands to access the output of the printk()s used by the RCU torture test. The printk()s use KERN_ALERT, so they should be evident. ;-) +The first and last lines show the rcutorture module parameters, and the +last line shows either "SUCCESS" or "FAILURE", based on rcutorture's +automatic determination as to whether RCU operated correctly. + The entries are as follows: o "rtc": The hexadecimal address of the structure currently visible to readers. -o "ver": The number of times since boot that the rcutw writer task +o "ver": The number of times since boot that the RCU writer task has changed the structure visible to readers. o "tfle": If non-zero, indicates that the "torture freelist" - containing structure to be placed into the "rtc" area is empty. + containing structures to be placed into the "rtc" area is empty. This condition is important, since it can fool you into thinking that RCU is working when it is not. :-/ o "rta": Number of structures allocated from the torture freelist. o "rtaf": Number of allocations from the torture freelist that have - failed due to the list being empty. + failed due to the list being empty. It is not unusual for this + to be non-zero, but it is bad for it to be a large fraction of + the value indicated by "rta". o "rtf": Number of frees into the torture freelist. +o "rtmbe": A non-zero value indicates that rcutorture believes that + rcu_assign_pointer() and rcu_dereference() are not working + correctly. This value should be zero. + +o "rtbe": A non-zero value indicates that one of the rcu_barrier() + family of functions is not working correctly. + +o "rtbke": rcutorture was unable to create the real-time kthreads + used to force RCU priority inversion. This value should be zero. + +o "rtbre": Although rcutorture successfully created the kthreads + used to force RCU priority inversion, it was unable to set them + to the real-time priority level of 1. This value should be zero. + +o "rtbf": The number of times that RCU priority boosting failed + to resolve RCU priority inversion. + +o "rtb": The number of times that rcutorture attempted to force + an RCU priority inversion condition. If you are testing RCU + priority boosting via the "test_boost" module parameter, this + value should be non-zero. + +o "nt": The number of times rcutorture ran RCU read-side code from + within a timer handler. This value should be non-zero only + if you specified the "irqreader" module parameter. + o "Reader Pipe": Histogram of "ages" of structures seen by readers. If any entries past the first two are non-zero, RCU is broken. And rcutorture prints the error flag string "!!!" to make sure @@ -148,19 +304,15 @@ o "Free-Block Circulation": Shows the number of torture structures somehow gets incremented farther than it should. Different implementations of RCU can provide implementation-specific -additional information. For example, SRCU provides the following: +additional information. For example, SRCU provides the following +additional line: - srcu-torture: rtc: f8cf46a8 ver: 355 tfle: 0 rta: 356 rtaf: 0 rtf: 346 rtmbe: 0 - srcu-torture: Reader Pipe: 559738 939 0 0 0 0 0 0 0 0 0 - srcu-torture: Reader Batch: 560434 243 0 0 0 0 0 0 0 0 - srcu-torture: Free-Block Circulation: 355 354 353 352 351 350 349 348 347 346 0 srcu-torture: per-CPU(idx=1): 0(0,1) 1(0,1) 2(0,0) 3(0,1) -The first four lines are similar to those for RCU. The last line shows -the per-CPU counter state. The numbers in parentheses are the values -of the "old" and "current" counters for the corresponding CPU. The -"idx" value maps the "old" and "current" values to the underlying array, -and is useful for debugging. +This line shows the per-CPU counter state. The numbers in parentheses are +the values of the "old" and "current" counters for the corresponding CPU. +The "idx" value maps the "old" and "current" values to the underlying +array, and is useful for debugging. USAGE @@ -170,11 +322,13 @@ The following script may be used to torture RCU: #!/bin/sh modprobe rcutorture - sleep 100 + sleep 3600 rmmod rcutorture dmesg | grep torture: The output can be manually inspected for the error flag of "!!!". One could of course create a more elaborate script that automatically -checked for such errors. The "rmmod" command forces a "SUCCESS" or -"FAILURE" indication to be printk()ed. +checked for such errors. The "rmmod" command forces a "SUCCESS", +"FAILURE", or "RCU_HOTPLUG" indication to be printk()ed. The first +two are self-explanatory, while the last indicates that while there +were no RCU failures, CPU-hotplug problems were detected. |