diff options
Diffstat (limited to 'Documentation/fault-injection')
| -rw-r--r-- | Documentation/fault-injection/fault-injection.txt | 108 | ||||
| -rw-r--r-- | Documentation/fault-injection/notifier-error-inject.txt | 99 | ||||
| -rw-r--r-- | Documentation/fault-injection/provoke-crashes.txt | 38 |
3 files changed, 207 insertions, 38 deletions
diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt index 4bc374a1434..4cf1a2a6bd7 100644 --- a/Documentation/fault-injection/fault-injection.txt +++ b/Documentation/fault-injection/fault-injection.txt @@ -21,6 +21,11 @@ o fail_make_request /sys/block/<device>/make-it-fail or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request()) +o fail_mmc_request + + injects MMC data errors on devices permitted by setting + debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request + Configure fault-injection capabilities behavior ----------------------------------------------- @@ -29,16 +34,16 @@ o debugfs entries fault-inject-debugfs kernel module provides some debugfs entries for runtime configuration of fault-injection capabilities. -- /debug/fail*/probability: +- /sys/kernel/debug/fail*/probability: likelihood of failure injection, in percent. Format: <percent> Note that one-failure-per-hundred is a very high error rate for some testcases. Consider setting probability=100 and configure - /debug/fail*/interval for such testcases. + /sys/kernel/debug/fail*/interval for such testcases. -- /debug/fail*/interval: +- /sys/kernel/debug/fail*/interval: specifies the interval between failures, for calls to should_fail() that pass all the other tests. @@ -46,18 +51,18 @@ configuration of fault-injection capabilities. Note that if you enable this, by setting interval>1, you will probably want to set probability=100. -- /debug/fail*/times: +- /sys/kernel/debug/fail*/times: specifies how many times failures may happen at most. A value of -1 means "no limit". -- /debug/fail*/space: +- /sys/kernel/debug/fail*/space: specifies an initial resource "budget", decremented by "size" on each call to should_fail(,size). Failure injection is suppressed until "space" reaches zero. -- /debug/fail*/verbose +- /sys/kernel/debug/fail*/verbose Format: { 0 | 1 | 2 } specifies the verbosity of the messages when failure is @@ -65,17 +70,17 @@ configuration of fault-injection capabilities. log line per failure; '2' will print a call trace too -- useful to debug the problems revealed by fault injection. -- /debug/fail*/task-filter: +- /sys/kernel/debug/fail*/task-filter: Format: { 'Y' | 'N' } A value of 'N' disables filtering by process (default). Any positive value limits failures to only processes indicated by /proc/<pid>/make-it-fail==1. -- /debug/fail*/require-start: -- /debug/fail*/require-end: -- /debug/fail*/reject-start: -- /debug/fail*/reject-end: +- /sys/kernel/debug/fail*/require-start: +- /sys/kernel/debug/fail*/require-end: +- /sys/kernel/debug/fail*/reject-start: +- /sys/kernel/debug/fail*/reject-end: specifies the range of virtual addresses tested during stacktrace walking. Failure is injected only if some caller @@ -84,26 +89,26 @@ configuration of fault-injection capabilities. Default required range is [0,ULONG_MAX) (whole of virtual address space). Default rejected range is [0,0). -- /debug/fail*/stacktrace-depth: +- /sys/kernel/debug/fail*/stacktrace-depth: specifies the maximum stacktrace depth walked during search for a caller within [require-start,require-end) OR [reject-start,reject-end). -- /debug/fail_page_alloc/ignore-gfp-highmem: +- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem: Format: { 'Y' | 'N' } default is 'N', setting it to 'Y' won't inject failures into highmem/user allocations. -- /debug/failslab/ignore-gfp-wait: -- /debug/fail_page_alloc/ignore-gfp-wait: +- /sys/kernel/debug/failslab/ignore-gfp-wait: +- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait: Format: { 'Y' | 'N' } default is 'N', setting it to 'Y' will inject failures only into non-sleep allocations (GFP_ATOMIC allocations). -- /debug/fail_page_alloc/min-order: +- /sys/kernel/debug/fail_page_alloc/min-order: specifies the minimum page allocation order to be injected failures. @@ -115,7 +120,8 @@ use the boot option: failslab= fail_page_alloc= - fail_make_request=<interval>,<probability>,<space>,<times> + fail_make_request= + mmc_core.fail_request=<interval>,<probability>,<space>,<times> How to add new fault injection capability ----------------------------------------- @@ -143,8 +149,7 @@ o provide a way to configure fault attributes failslab, fail_page_alloc, and fail_make_request use this way. Helper functions: - init_fault_attr_entries(entries, attr, name); - void cleanup_fault_attr_entries(entries); + fault_create_debugfs_attr(name, parent, attr); - module parameters @@ -166,13 +171,13 @@ o Inject slab allocation failures into module init/exit code #!/bin/bash FAILTYPE=failslab -echo Y > /debug/$FAILTYPE/task-filter -echo 10 > /debug/$FAILTYPE/probability -echo 100 > /debug/$FAILTYPE/interval -echo -1 > /debug/$FAILTYPE/times -echo 0 > /debug/$FAILTYPE/space -echo 2 > /debug/$FAILTYPE/verbose -echo 1 > /debug/$FAILTYPE/ignore-gfp-wait +echo Y > /sys/kernel/debug/$FAILTYPE/task-filter +echo 10 > /sys/kernel/debug/$FAILTYPE/probability +echo 100 > /sys/kernel/debug/$FAILTYPE/interval +echo -1 > /sys/kernel/debug/$FAILTYPE/times +echo 0 > /sys/kernel/debug/$FAILTYPE/space +echo 2 > /sys/kernel/debug/$FAILTYPE/verbose +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait faulty_system() { @@ -217,21 +222,48 @@ then exit 1 fi -cat /sys/module/$module/sections/.text > /debug/$FAILTYPE/require-start -cat /sys/module/$module/sections/.data > /debug/$FAILTYPE/require-end +cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start +cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end -echo N > /debug/$FAILTYPE/task-filter -echo 10 > /debug/$FAILTYPE/probability -echo 100 > /debug/$FAILTYPE/interval -echo -1 > /debug/$FAILTYPE/times -echo 0 > /debug/$FAILTYPE/space -echo 2 > /debug/$FAILTYPE/verbose -echo 1 > /debug/$FAILTYPE/ignore-gfp-wait -echo 1 > /debug/$FAILTYPE/ignore-gfp-highmem -echo 10 > /debug/$FAILTYPE/stacktrace-depth +echo N > /sys/kernel/debug/$FAILTYPE/task-filter +echo 10 > /sys/kernel/debug/$FAILTYPE/probability +echo 100 > /sys/kernel/debug/$FAILTYPE/interval +echo -1 > /sys/kernel/debug/$FAILTYPE/times +echo 0 > /sys/kernel/debug/$FAILTYPE/space +echo 2 > /sys/kernel/debug/$FAILTYPE/verbose +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait +echo 1 > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem +echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth -trap "echo 0 > /debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT +trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT echo "Injecting errors into the module $module... (interrupt to stop)" sleep 1000000 +Tool to run command with failslab or fail_page_alloc +---------------------------------------------------- +In order to make it easier to accomplish the tasks mentioned above, we can use +tools/testing/fault-injection/failcmd.sh. Please run a command +"./tools/testing/fault-injection/failcmd.sh --help" for more information and +see the following examples. + +Examples: + +Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab +allocation failure. + + # ./tools/testing/fault-injection/failcmd.sh \ + -- make -C tools/testing/selftests/ run_tests + +Same as above except to specify 100 times failures at most instead of one time +at most by default. + + # ./tools/testing/fault-injection/failcmd.sh --times=100 \ + -- make -C tools/testing/selftests/ run_tests + +Same as above except to inject page allocation failure instead of slab +allocation failure. + + # env FAILCMD_TYPE=fail_page_alloc \ + ./tools/testing/fault-injection/failcmd.sh --times=100 \ + -- make -C tools/testing/selftests/ run_tests diff --git a/Documentation/fault-injection/notifier-error-inject.txt b/Documentation/fault-injection/notifier-error-inject.txt new file mode 100644 index 00000000000..09adabef513 --- /dev/null +++ b/Documentation/fault-injection/notifier-error-inject.txt @@ -0,0 +1,99 @@ +Notifier error injection +======================== + +Notifier error injection provides the ability to inject artificial errors to +specified notifier chain callbacks. It is useful to test the error handling of +notifier call chain failures which is rarely executed. There are kernel +modules that can be used to test the following notifiers. + + * CPU notifier + * PM notifier + * Memory hotplug notifier + * powerpc pSeries reconfig notifier + +CPU notifier error injection module +----------------------------------- +This feature can be used to test the error handling of the CPU notifiers by +injecting artificial errors to CPU notifier chain callbacks. + +If the notifier call chain should be failed with some events notified, write +the error code to debugfs interface +/sys/kernel/debug/notifier-error-inject/cpu/actions/<notifier event>/error + +Possible CPU notifier events to be failed are: + + * CPU_UP_PREPARE + * CPU_UP_PREPARE_FROZEN + * CPU_DOWN_PREPARE + * CPU_DOWN_PREPARE_FROZEN + +Example1: Inject CPU offline error (-1 == -EPERM) + + # cd /sys/kernel/debug/notifier-error-inject/cpu + # echo -1 > actions/CPU_DOWN_PREPARE/error + # echo 0 > /sys/devices/system/cpu/cpu1/online + bash: echo: write error: Operation not permitted + +Example2: inject CPU online error (-2 == -ENOENT) + + # echo -2 > actions/CPU_UP_PREPARE/error + # echo 1 > /sys/devices/system/cpu/cpu1/online + bash: echo: write error: No such file or directory + +PM notifier error injection module +---------------------------------- +This feature is controlled through debugfs interface +/sys/kernel/debug/notifier-error-inject/pm/actions/<notifier event>/error + +Possible PM notifier events to be failed are: + + * PM_HIBERNATION_PREPARE + * PM_SUSPEND_PREPARE + * PM_RESTORE_PREPARE + +Example: Inject PM suspend error (-12 = -ENOMEM) + + # cd /sys/kernel/debug/notifier-error-inject/pm/ + # echo -12 > actions/PM_SUSPEND_PREPARE/error + # echo mem > /sys/power/state + bash: echo: write error: Cannot allocate memory + +Memory hotplug notifier error injection module +---------------------------------------------- +This feature is controlled through debugfs interface +/sys/kernel/debug/notifier-error-inject/memory/actions/<notifier event>/error + +Possible memory notifier events to be failed are: + + * MEM_GOING_ONLINE + * MEM_GOING_OFFLINE + +Example: Inject memory hotplug offline error (-12 == -ENOMEM) + + # cd /sys/kernel/debug/notifier-error-inject/memory + # echo -12 > actions/MEM_GOING_OFFLINE/error + # echo offline > /sys/devices/system/memory/memoryXXX/state + bash: echo: write error: Cannot allocate memory + +powerpc pSeries reconfig notifier error injection module +-------------------------------------------------------- +This feature is controlled through debugfs interface +/sys/kernel/debug/notifier-error-inject/pSeries-reconfig/actions/<notifier event>/error + +Possible pSeries reconfig notifier events to be failed are: + + * PSERIES_RECONFIG_ADD + * PSERIES_RECONFIG_REMOVE + * PSERIES_DRCONF_MEM_ADD + * PSERIES_DRCONF_MEM_REMOVE + +For more usage examples +----------------------- +There are tools/testing/selftests using the notifier error injection features +for CPU and memory notifiers. + + * tools/testing/selftests/cpu-hotplug/on-off-test.sh + * tools/testing/selftests/memory-hotplug/on-off-test.sh + +These scripts first do simple online and offline tests and then do fault +injection tests if notifier error injection module is available. diff --git a/Documentation/fault-injection/provoke-crashes.txt b/Documentation/fault-injection/provoke-crashes.txt new file mode 100644 index 00000000000..7a9d3d81525 --- /dev/null +++ b/Documentation/fault-injection/provoke-crashes.txt @@ -0,0 +1,38 @@ +The lkdtm module provides an interface to crash or injure the kernel at +predefined crashpoints to evaluate the reliability of crash dumps obtained +using different dumping solutions. The module uses KPROBEs to instrument +crashing points, but can also crash the kernel directly without KRPOBE +support. + + +You can provide the way either through module arguments when inserting +the module, or through a debugfs interface. + +Usage: insmod lkdtm.ko [recur_count={>0}] cpoint_name=<> cpoint_type=<> + [cpoint_count={>0}] + + recur_count : Recursion level for the stack overflow test. Default is 10. + + cpoint_name : Crash point where the kernel is to be crashed. It can be + one of INT_HARDWARE_ENTRY, INT_HW_IRQ_EN, INT_TASKLET_ENTRY, + FS_DEVRW, MEM_SWAPOUT, TIMERADD, SCSI_DISPATCH_CMD, + IDE_CORE_CP, DIRECT + + cpoint_type : Indicates the action to be taken on hitting the crash point. + It can be one of PANIC, BUG, EXCEPTION, LOOP, OVERFLOW, + CORRUPT_STACK, UNALIGNED_LOAD_STORE_WRITE, OVERWRITE_ALLOCATION, + WRITE_AFTER_FREE, + + cpoint_count : Indicates the number of times the crash point is to be hit + to trigger an action. The default is 10. + +You can also induce failures by mounting debugfs and writing the type to +<mountpoint>/provoke-crash/<crashpoint>. E.g., + + mount -t debugfs debugfs /mnt + echo EXCEPTION > /mnt/provoke-crash/INT_HARDWARE_ENTRY + + +A special file is `DIRECT' which will induce the crash directly without +KPROBE instrumentation. This mode is the only one available when the module +is built on a kernel without KPROBEs support. |