diff options
Diffstat (limited to 'Documentation/fault-injection')
| -rw-r--r-- | Documentation/fault-injection/failcmd.sh | 4 | ||||
| -rw-r--r-- | Documentation/fault-injection/failmodule.sh | 31 | ||||
| -rw-r--r-- | Documentation/fault-injection/fault-injection.txt | 178 | ||||
| -rw-r--r-- | Documentation/fault-injection/notifier-error-inject.txt | 99 | ||||
| -rw-r--r-- | Documentation/fault-injection/provoke-crashes.txt | 38 |
5 files changed, 248 insertions, 102 deletions
diff --git a/Documentation/fault-injection/failcmd.sh b/Documentation/fault-injection/failcmd.sh deleted file mode 100644 index 63177aba810..00000000000 --- a/Documentation/fault-injection/failcmd.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/bash - -echo 1 > /proc/self/make-it-fail -exec $* diff --git a/Documentation/fault-injection/failmodule.sh b/Documentation/fault-injection/failmodule.sh deleted file mode 100644 index 474a8b971f9..00000000000 --- a/Documentation/fault-injection/failmodule.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/bin/bash -# -# Usage: failmodule <failname> <modulename> [stacktrace-depth] -# -# <failname>: "failslab", "fail_alloc_page", or "fail_make_request" -# -# <modulename>: module name that you want to inject faults. -# -# [stacktrace-depth]: the maximum number of stacktrace walking allowed -# - -STACKTRACE_DEPTH=5 -if [ $# -gt 2 ]; then - STACKTRACE_DEPTH=$3 -fi - -if [ ! -d /debug/$1 ]; then - echo "Fault-injection $1 does not exist" >&2 - exit 1 -fi -if [ ! -d /sys/module/$2 ]; then - echo "Module $2 does not exist" >&2 - exit 1 -fi - -# Disable any fault injection -echo 0 > /debug/$1/stacktrace-depth - -echo `cat /sys/module/$2/sections/.text` > /debug/$1/require-start -echo `cat /sys/module/$2/sections/.exit.text` > /debug/$1/require-end -echo $STACKTRACE_DEPTH > /debug/$1/stacktrace-depth diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt index b7ca560b934..4cf1a2a6bd7 100644 --- a/Documentation/fault-injection/fault-injection.txt +++ b/Documentation/fault-injection/fault-injection.txt @@ -21,6 +21,11 @@ o fail_make_request /sys/block/<device>/make-it-fail or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) +o fail_mmc_request + + injects MMC data errors on devices permitted by setting + debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request + Configure fault-injection capabilities behavior ----------------------------------------------- @@ -29,16 +34,16 @@ o debugfs entries fault-inject-debugfs kernel module provides some debugfs entries for runtime configuration of fault-injection capabilities. -- /debug/fail*/probability: +- /sys/kernel/debug/fail*/probability: likelihood of failure injection, in percent. Format: <percent> Note that one-failure-per-hundred is a very high error rate for some testcases. Consider setting probability=100 and configure - /debug/fail*/interval for such testcases. + /sys/kernel/debug/fail*/interval for such testcases. -- /debug/fail*/interval: +- /sys/kernel/debug/fail*/interval: specifies the interval between failures, for calls to should_fail() that pass all the other tests. @@ -46,18 +51,18 @@ configuration of fault-injection capabilities. Note that if you enable this, by setting interval>1, you will probably want to set probability=100. -- /debug/fail*/times: +- /sys/kernel/debug/fail*/times: specifies how many times failures may happen at most. A value of -1 means "no limit". -- /debug/fail*/space: +- /sys/kernel/debug/fail*/space: specifies an initial resource "budget", decremented by "size" on each call to should_fail(,size). Failure injection is suppressed until "space" reaches zero. -- /debug/fail*/verbose +- /sys/kernel/debug/fail*/verbose Format: { 0 | 1 | 2 } specifies the verbosity of the messages when failure is @@ -65,17 +70,17 @@ configuration of fault-injection capabilities. log line per failure; '2' will print a call trace too -- useful to debug the problems revealed by fault injection. -- /debug/fail*/task-filter: +- /sys/kernel/debug/fail*/task-filter: Format: { 'Y' | 'N' } A value of 'N' disables filtering by process (default). Any positive value limits failures to only processes indicated by /proc/<pid>/make-it-fail==1. -- /debug/fail*/require-start: -- /debug/fail*/require-end: -- /debug/fail*/reject-start: -- /debug/fail*/reject-end: +- /sys/kernel/debug/fail*/require-start: +- /sys/kernel/debug/fail*/require-end: +- /sys/kernel/debug/fail*/reject-start: +- /sys/kernel/debug/fail*/reject-end: specifies the range of virtual addresses tested during stacktrace walking. Failure is injected only if some caller @@ -84,25 +89,30 @@ configuration of fault-injection capabilities. Default required range is [0,ULONG_MAX) (whole of virtual address space). Default rejected range is [0,0). -- /debug/fail*/stacktrace-depth: +- /sys/kernel/debug/fail*/stacktrace-depth: specifies the maximum stacktrace depth walked during search for a caller within [require-start,require-end) OR [reject-start,reject-end). -- /debug/fail_page_alloc/ignore-gfp-highmem: +- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: Format: { 'Y' | 'N' } default is 'N', setting it to 'Y' won't inject failures into highmem/user allocations. -- /debug/failslab/ignore-gfp-wait: -- /debug/fail_page_alloc/ignore-gfp-wait: +- /sys/kernel/debug/failslab/ignore-gfp-wait: +- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: Format: { 'Y' | 'N' } default is 'N', setting it to 'Y' will inject failures only into non-sleep allocations (GFP_ATOMIC allocations). +- /sys/kernel/debug/fail_page_alloc/min-order: + + specifies the minimum page allocation order to be injected + failures. + o Boot option In order to inject faults while debugfs is not available (early boot time), @@ -110,7 +120,8 @@ use the boot option: failslab= fail_page_alloc= - fail_make_request=<interval>,<probability>,<space>,<times> + fail_make_request= + mmc_core.fail_request=<interval>,<probability>,<space>,<times> How to add new fault injection capability ----------------------------------------- @@ -138,8 +149,7 @@ o provide a way to configure fault attributes failslab, fail_page_alloc, and fail_make_request use this way. Helper functions: - init_fault_attr_entries(entries, attr, name); - void cleanup_fault_attr_entries(entries); + fault_create_debugfs_attr(name, parent, attr); - module parameters @@ -156,70 +166,104 @@ o add a hook to insert failures Application Examples -------------------- -o inject slab allocation failures into module init/cleanup code +o Inject slab allocation failures into module init/exit code ------------------------------------------------------------------------------- #!/bin/bash -FAILCMD=Documentation/fault-injection/failcmd.sh -BLACKLIST="root_plug evbug" - -FAILNAME=failslab -echo Y > /debug/$FAILNAME/task-filter -echo 10 > /debug/$FAILNAME/probability -echo 100 > /debug/$FAILNAME/interval -echo -1 > /debug/$FAILNAME/times -echo 2 > /debug/$FAILNAME/verbose -echo 1 > /debug/$FAILNAME/ignore-gfp-wait +FAILTYPE=failslab +echo Y > /sys/kernel/debug/$FAILTYPE/task-filter +echo 10 > /sys/kernel/debug/$FAILTYPE/probability +echo 100 > /sys/kernel/debug/$FAILTYPE/interval +echo -1 > /sys/kernel/debug/$FAILTYPE/times +echo 0 > /sys/kernel/debug/$FAILTYPE/space +echo 2 > /sys/kernel/debug/$FAILTYPE/verbose +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait -blacklist() +faulty_system() { - echo $BLACKLIST | grep $1 > /dev/null 2>&1 + bash -c "echo 1 > /proc/self/make-it-fail && exec $*" } -oops() -{ - dmesg | grep BUG > /dev/null 2>&1 -} +if [ $# -eq 0 ] +then + echo "Usage: $0 modulename [ modulename ... ]" + exit 1 +fi + +for m in $* +do + echo inserting $m... + faulty_system modprobe $m -find /lib/modules/`uname -r` -name '*.ko' -exec basename {} .ko \; | - while read i - do - oops && exit 1 - - if ! blacklist $i - then - echo inserting $i... - bash $FAILCMD modprobe $i - fi - done - -lsmod | awk '{ if ($3 == 0) { print $1 } }' | - while read i - do - oops && exit 1 - - if ! blacklist $i - then - echo removing $i... - bash $FAILCMD modprobe -r $i - fi - done + echo removing $m... + faulty_system modprobe -r $m +done ------------------------------------------------------------------------------ -o inject slab allocation failures only for a specific module +o Inject page allocation failures only for a specific module ------------------------------------------------------------------------------- #!/bin/bash -FAILMOD=Documentation/fault-injection/failmodule.sh +FAILTYPE=fail_page_alloc +module=$1 -echo injecting errors into the module $1... +if [ -z $module ] +then + echo "Usage: $0 <modulename>" + exit 1 +fi -modprobe $1 -bash $FAILMOD failslab $1 10 -echo 25 > /debug/failslab/probability +modprobe $module ------------------------------------------------------------------------------- +if [ ! -d /sys/module/$module/sections ] +then + echo Module $module is not loaded + exit 1 +fi + +cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start +cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end + +echo N > /sys/kernel/debug/$FAILTYPE/task-filter +echo 10 > /sys/kernel/debug/$FAILTYPE/probability +echo 100 > /sys/kernel/debug/$FAILTYPE/interval +echo -1 > /sys/kernel/debug/$FAILTYPE/times +echo 0 > /sys/kernel/debug/$FAILTYPE/space +echo 2 > /sys/kernel/debug/$FAILTYPE/verbose +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem +echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth + +trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT + +echo "Injecting errors into the module $module... (interrupt to stop)" +sleep 1000000 + +Tool to run command with failslab or fail_page_alloc +---------------------------------------------------- +In order to make it easier to accomplish the tasks mentioned above, we can use +tools/testing/fault-injection/failcmd.sh. Please run a command +"./tools/testing/fault-injection/failcmd.sh --help" for more information and +see the following examples. + +Examples: + +Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab +allocation failure. + + # ./tools/testing/fault-injection/failcmd.sh \ + -- make -C tools/testing/selftests/ run_tests + +Same as above except to specify 100 times failures at most instead of one time +at most by default. + + # ./tools/testing/fault-injection/failcmd.sh --times=100 \ + -- make -C tools/testing/selftests/ run_tests + +Same as above except to inject page allocation failure instead of slab +allocation failure. + # env FAILCMD_TYPE=fail_page_alloc \ + ./tools/testing/fault-injection/failcmd.sh --times=100 \ + -- make -C tools/testing/selftests/ run_tests diff --git a/Documentation/fault-injection/notifier-error-inject.txt b/Documentation/fault-injection/notifier-error-inject.txt new file mode 100644 index 00000000000..09adabef513 --- /dev/null +++ b/Documentation/fault-injection/notifier-error-inject.txt @@ -0,0 +1,99 @@ +Notifier error injection +======================== + +Notifier error injection provides the ability to inject artificial errors to +specified notifier chain callbacks. It is useful to test the error handling of +notifier call chain failures which is rarely executed. There are kernel +modules that can be used to test the following notifiers. + + * CPU notifier + * PM notifier + * Memory hotplug notifier + * powerpc pSeries reconfig notifier + +CPU notifier error injection module +----------------------------------- +This feature can be used to test the error handling of the CPU notifiers by +injecting artificial errors to CPU notifier chain callbacks. + +If the notifier call chain should be failed with some events notified, write +the error code to debugfs interface +/sys/kernel/debug/notifier-error-inject/cpu/actions/<notifier event>/error + +Possible CPU notifier events to be failed are: + + * CPU_UP_PREPARE + * CPU_UP_PREPARE_FROZEN + * CPU_DOWN_PREPARE + * CPU_DOWN_PREPARE_FROZEN + +Example1: Inject CPU offline error (-1 == -EPERM) + + # cd /sys/kernel/debug/notifier-error-inject/cpu + # echo -1 > actions/CPU_DOWN_PREPARE/error + # echo 0 > /sys/devices/system/cpu/cpu1/online + bash: echo: write error: Operation not permitted + +Example2: inject CPU online error (-2 == -ENOENT) + + # echo -2 > actions/CPU_UP_PREPARE/error + # echo 1 > /sys/devices/system/cpu/cpu1/online + bash: echo: write error: No such file or directory + +PM notifier error injection module +---------------------------------- +This feature is controlled through debugfs interface +/sys/kernel/debug/notifier-error-inject/pm/actions/<notifier event>/error + +Possible PM notifier events to be failed are: + + * PM_HIBERNATION_PREPARE + * PM_SUSPEND_PREPARE + * PM_RESTORE_PREPARE + +Example: Inject PM suspend error (-12 = -ENOMEM) + + # cd /sys/kernel/debug/notifier-error-inject/pm/ + # echo -12 > actions/PM_SUSPEND_PREPARE/error + # echo mem > /sys/power/state + bash: echo: write error: Cannot allocate memory + +Memory hotplug notifier error injection module +---------------------------------------------- +This feature is controlled through debugfs interface +/sys/kernel/debug/notifier-error-inject/memory/actions/<notifier event>/error + +Possible memory notifier events to be failed are: + + * MEM_GOING_ONLINE + * MEM_GOING_OFFLINE + +Example: Inject memory hotplug offline error (-12 == -ENOMEM) + + # cd /sys/kernel/debug/notifier-error-inject/memory + # echo -12 > actions/MEM_GOING_OFFLINE/error + # echo offline > /sys/devices/system/memory/memoryXXX/state + bash: echo: write error: Cannot allocate memory + +powerpc pSeries reconfig notifier error injection module +-------------------------------------------------------- +This feature is controlled through debugfs interface +/sys/kernel/debug/notifier-error-inject/pSeries-reconfig/actions/<notifier event>/error + +Possible pSeries reconfig notifier events to be failed are: + + * PSERIES_RECONFIG_ADD + * PSERIES_RECONFIG_REMOVE + * PSERIES_DRCONF_MEM_ADD + * PSERIES_DRCONF_MEM_REMOVE + +For more usage examples +----------------------- +There are tools/testing/selftests using the notifier error injection features +for CPU and memory notifiers. + + * tools/testing/selftests/cpu-hotplug/on-off-test.sh + * tools/testing/selftests/memory-hotplug/on-off-test.sh + +These scripts first do simple online and offline tests and then do fault +injection tests if notifier error injection module is available. diff --git a/Documentation/fault-injection/provoke-crashes.txt b/Documentation/fault-injection/provoke-crashes.txt new file mode 100644 index 00000000000..7a9d3d81525 --- /dev/null +++ b/Documentation/fault-injection/provoke-crashes.txt @@ -0,0 +1,38 @@ +The lkdtm module provides an interface to crash or injure the kernel at +predefined crashpoints to evaluate the reliability of crash dumps obtained +using different dumping solutions. The module uses KPROBEs to instrument +crashing points, but can also crash the kernel directly without KRPOBE +support. + + +You can provide the way either through module arguments when inserting +the module, or through a debugfs interface. + +Usage: insmod lkdtm.ko [recur_count={>0}] cpoint_name=<> cpoint_type=<> + [cpoint_count={>0}] + + recur_count : Recursion level for the stack overflow test. Default is 10. + + cpoint_name : Crash point where the kernel is to be crashed. It can be + one of INT_HARDWARE_ENTRY, INT_HW_IRQ_EN, INT_TASKLET_ENTRY, + FS_DEVRW, MEM_SWAPOUT, TIMERADD, SCSI_DISPATCH_CMD, + IDE_CORE_CP, DIRECT + + cpoint_type : Indicates the action to be taken on hitting the crash point. + It can be one of PANIC, BUG, EXCEPTION, LOOP, OVERFLOW, + CORRUPT_STACK, UNALIGNED_LOAD_STORE_WRITE, OVERWRITE_ALLOCATION, + WRITE_AFTER_FREE, + + cpoint_count : Indicates the number of times the crash point is to be hit + to trigger an action. The default is 10. + +You can also induce failures by mounting debugfs and writing the type to +<mountpoint>/provoke-crash/<crashpoint>. E.g., + + mount -t debugfs debugfs /mnt + echo EXCEPTION > /mnt/provoke-crash/INT_HARDWARE_ENTRY + + +A special file is `DIRECT' which will induce the crash directly without +KPROBE instrumentation. This mode is the only one available when the module +is built on a kernel without KPROBEs support. |