diff options
Diffstat (limited to 'Documentation/fault-injection/fault-injection.txt')
| -rw-r--r-- | Documentation/fault-injection/fault-injection.txt | 178 |
1 files changed, 111 insertions, 67 deletions
diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt index b7ca560b934..4cf1a2a6bd7 100644 --- a/Documentation/fault-injection/fault-injection.txt +++ b/Documentation/fault-injection/fault-injection.txt @@ -21,6 +21,11 @@ o fail_make_request /sys/block/<device>/make-it-fail or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) +o fail_mmc_request + + injects MMC data errors on devices permitted by setting + debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request + Configure fault-injection capabilities behavior ----------------------------------------------- @@ -29,16 +34,16 @@ o debugfs entries fault-inject-debugfs kernel module provides some debugfs entries for runtime configuration of fault-injection capabilities. -- /debug/fail*/probability: +- /sys/kernel/debug/fail*/probability: likelihood of failure injection, in percent. Format: <percent> Note that one-failure-per-hundred is a very high error rate for some testcases. Consider setting probability=100 and configure - /debug/fail*/interval for such testcases. + /sys/kernel/debug/fail*/interval for such testcases. -- /debug/fail*/interval: +- /sys/kernel/debug/fail*/interval: specifies the interval between failures, for calls to should_fail() that pass all the other tests. @@ -46,18 +51,18 @@ configuration of fault-injection capabilities. Note that if you enable this, by setting interval>1, you will probably want to set probability=100. -- /debug/fail*/times: +- /sys/kernel/debug/fail*/times: specifies how many times failures may happen at most. A value of -1 means "no limit". -- /debug/fail*/space: +- /sys/kernel/debug/fail*/space: specifies an initial resource "budget", decremented by "size" on each call to should_fail(,size). Failure injection is suppressed until "space" reaches zero. -- /debug/fail*/verbose +- /sys/kernel/debug/fail*/verbose Format: { 0 | 1 | 2 } specifies the verbosity of the messages when failure is @@ -65,17 +70,17 @@ configuration of fault-injection capabilities. log line per failure; '2' will print a call trace too -- useful to debug the problems revealed by fault injection. -- /debug/fail*/task-filter: +- /sys/kernel/debug/fail*/task-filter: Format: { 'Y' | 'N' } A value of 'N' disables filtering by process (default). Any positive value limits failures to only processes indicated by /proc/<pid>/make-it-fail==1. -- /debug/fail*/require-start: -- /debug/fail*/require-end: -- /debug/fail*/reject-start: -- /debug/fail*/reject-end: +- /sys/kernel/debug/fail*/require-start: +- /sys/kernel/debug/fail*/require-end: +- /sys/kernel/debug/fail*/reject-start: +- /sys/kernel/debug/fail*/reject-end: specifies the range of virtual addresses tested during stacktrace walking. Failure is injected only if some caller @@ -84,25 +89,30 @@ configuration of fault-injection capabilities. Default required range is [0,ULONG_MAX) (whole of virtual address space). Default rejected range is [0,0). -- /debug/fail*/stacktrace-depth: +- /sys/kernel/debug/fail*/stacktrace-depth: specifies the maximum stacktrace depth walked during search for a caller within [require-start,require-end) OR [reject-start,reject-end). -- /debug/fail_page_alloc/ignore-gfp-highmem: +- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: Format: { 'Y' | 'N' } default is 'N', setting it to 'Y' won't inject failures into highmem/user allocations. -- /debug/failslab/ignore-gfp-wait: -- /debug/fail_page_alloc/ignore-gfp-wait: +- /sys/kernel/debug/failslab/ignore-gfp-wait: +- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: Format: { 'Y' | 'N' } default is 'N', setting it to 'Y' will inject failures only into non-sleep allocations (GFP_ATOMIC allocations). +- /sys/kernel/debug/fail_page_alloc/min-order: + + specifies the minimum page allocation order to be injected + failures. + o Boot option In order to inject faults while debugfs is not available (early boot time), @@ -110,7 +120,8 @@ use the boot option: failslab= fail_page_alloc= - fail_make_request=<interval>,<probability>,<space>,<times> + fail_make_request= + mmc_core.fail_request=<interval>,<probability>,<space>,<times> How to add new fault injection capability ----------------------------------------- @@ -138,8 +149,7 @@ o provide a way to configure fault attributes failslab, fail_page_alloc, and fail_make_request use this way. Helper functions: - init_fault_attr_entries(entries, attr, name); - void cleanup_fault_attr_entries(entries); + fault_create_debugfs_attr(name, parent, attr); - module parameters @@ -156,70 +166,104 @@ o add a hook to insert failures Application Examples -------------------- -o inject slab allocation failures into module init/cleanup code +o Inject slab allocation failures into module init/exit code ------------------------------------------------------------------------------- #!/bin/bash -FAILCMD=Documentation/fault-injection/failcmd.sh -BLACKLIST="root_plug evbug" - -FAILNAME=failslab -echo Y > /debug/$FAILNAME/task-filter -echo 10 > /debug/$FAILNAME/probability -echo 100 > /debug/$FAILNAME/interval -echo -1 > /debug/$FAILNAME/times -echo 2 > /debug/$FAILNAME/verbose -echo 1 > /debug/$FAILNAME/ignore-gfp-wait +FAILTYPE=failslab +echo Y > /sys/kernel/debug/$FAILTYPE/task-filter +echo 10 > /sys/kernel/debug/$FAILTYPE/probability +echo 100 > /sys/kernel/debug/$FAILTYPE/interval +echo -1 > /sys/kernel/debug/$FAILTYPE/times +echo 0 > /sys/kernel/debug/$FAILTYPE/space +echo 2 > /sys/kernel/debug/$FAILTYPE/verbose +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait -blacklist() +faulty_system() { - echo $BLACKLIST | grep $1 > /dev/null 2>&1 + bash -c "echo 1 > /proc/self/make-it-fail && exec $*" } -oops() -{ - dmesg | grep BUG > /dev/null 2>&1 -} +if [ $# -eq 0 ] +then + echo "Usage: $0 modulename [ modulename ... ]" + exit 1 +fi + +for m in $* +do + echo inserting $m... + faulty_system modprobe $m -find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; | - while read i - do - oops && exit 1 - - if ! blacklist $i - then - echo inserting $i... - bash $FAILCMD modprobe $i - fi - done - -lsmod | awk '{ if ($3 == 0) { print $1 } }' | - while read i - do - oops && exit 1 - - if ! blacklist $i - then - echo removing $i... - bash $FAILCMD modprobe -r $i - fi - done + echo removing $m... + faulty_system modprobe -r $m +done ------------------------------------------------------------------------------ -o inject slab allocation failures only for a specific module +o Inject page allocation failures only for a specific module ------------------------------------------------------------------------------- #!/bin/bash -FAILMOD=Documentation/fault-injection/failmodule.sh +FAILTYPE=fail_page_alloc +module=$1 -echo injecting errors into the module $1... +if [ -z $module ] +then + echo "Usage: $0 <modulename>" + exit 1 +fi -modprobe $1 -bash $FAILMOD failslab $1 10 -echo 25 > /debug/failslab/probability +modprobe $module ------------------------------------------------------------------------------- +if [ ! -d /sys/module/$module/sections ] +then + echo Module $module is not loaded + exit 1 +fi + +cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start +cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end + +echo N > /sys/kernel/debug/$FAILTYPE/task-filter +echo 10 > /sys/kernel/debug/$FAILTYPE/probability +echo 100 > /sys/kernel/debug/$FAILTYPE/interval +echo -1 > /sys/kernel/debug/$FAILTYPE/times +echo 0 > /sys/kernel/debug/$FAILTYPE/space +echo 2 > /sys/kernel/debug/$FAILTYPE/verbose +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem +echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth + +trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT + +echo "Injecting errors into the module $module... (interrupt to stop)" +sleep 1000000 + +Tool to run command with failslab or fail_page_alloc +---------------------------------------------------- +In order to make it easier to accomplish the tasks mentioned above, we can use +tools/testing/fault-injection/failcmd.sh. Please run a command +"./tools/testing/fault-injection/failcmd.sh --help" for more information and +see the following examples. + +Examples: + +Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab +allocation failure. + + # ./tools/testing/fault-injection/failcmd.sh \ + -- make -C tools/testing/selftests/ run_tests + +Same as above except to specify 100 times failures at most instead of one time +at most by default. + + # ./tools/testing/fault-injection/failcmd.sh --times=100 \ + -- make -C tools/testing/selftests/ run_tests + +Same as above except to inject page allocation failure instead of slab +allocation failure. + # env FAILCMD_TYPE=fail_page_alloc \ + ./tools/testing/fault-injection/failcmd.sh --times=100 \ + -- make -C tools/testing/selftests/ run_tests |