diff options
Diffstat (limited to 'tools/testing/ktest')
| -rw-r--r-- | tools/testing/ktest/examples/README | 32 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/crosstests.conf | 254 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/include/bisect.conf | 90 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/include/defaults.conf | 157 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/include/min-config.conf | 60 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/include/patchcheck.conf | 111 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/include/tests.conf | 74 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/kvm.conf | 92 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/snowball.conf | 53 | ||||
| -rw-r--r-- | tools/testing/ktest/examples/test.conf | 62 | ||||
| -rwxr-xr-x | tools/testing/ktest/ktest.pl | 764 | ||||
| -rw-r--r-- | tools/testing/ktest/sample.conf | 195 |
12 files changed, 1857 insertions, 87 deletions
diff --git a/tools/testing/ktest/examples/README b/tools/testing/ktest/examples/README new file mode 100644 index 00000000000..a12d295a09d --- /dev/null +++ b/tools/testing/ktest/examples/README @@ -0,0 +1,32 @@ +This directory contains example configs to use ktest for various tasks. +The configs still need to be customized for your environment, but it +is broken up by task which makes it easier to understand how to set up +ktest. + +The configs are based off of real working configs but have been modified +and commented to show more generic use cases that are more helpful for +developers. + +crosstests.conf - this config shows an example of testing a git repo against + lots of different architectures. It only does build tests, but makes + it easy to compile test different archs. You can download the arch + cross compilers from: + http://kernel.org/pub/tools/crosstool/files/bin/x86_64/ + +test.conf - A generic example of a config. This is based on an actual config + used to perform real testing. + +kvm.conf - A example of a config that is used to test a virtual guest running + on a host. + +snowball.conf - An example config that was used to demo ktest.pl against + a snowball ARM board. + +include/ - The include directory holds default configs that can be + included into other configs. This is a real use example that shows how + to reuse configs for various machines or set ups. The files here + are included by other config files, where the other config files define + options and variables that will make the included config work for the + given environment. + + diff --git a/tools/testing/ktest/examples/crosstests.conf b/tools/testing/ktest/examples/crosstests.conf new file mode 100644 index 00000000000..a1203148dfa --- /dev/null +++ b/tools/testing/ktest/examples/crosstests.conf @@ -0,0 +1,254 @@ +# +# Example config for cross compiling +# +# In this config, it is expected that the tool chains from: +# +# http://kernel.org/pub/tools/crosstool/files/bin/x86_64/ +# +# running on a x86_64 system have been downloaded and installed into: +# +# /usr/local/ +# +# such that the compiler binaries are something like: +# +# /usr/local/gcc-4.5.2-nolibc/mips-linux/bin/mips-linux-gcc +# +# Some of the archs will use gcc-4.5.1 instead of gcc-4.5.2 +# this config uses variables to differentiate them. +# +# Comments describe some of the options, but full descriptions of +# options are described in the samples.conf file. + +# ${PWD} is defined by ktest.pl to be the directory that the user +# was in when they executed ktest.pl. It may be better to hardcode the +# path name here. THIS_DIR is the variable used through out the config file +# in case you want to change it. + +THIS_DIR := ${PWD} + +# Update the BUILD_DIR option to the location of your git repo you want to test. +BUILD_DIR = ${THIS_DIR}/linux.git + +# The build will go into this directory. It will be created when you run the test. +OUTPUT_DIR = ${THIS_DIR}/cross-compile + +# The build will be compiled with -j8 +BUILD_OPTIONS = -j8 + +# The test will not stop when it hits a failure. +DIE_ON_FAILURE = 0 + +# If you want to have ktest.pl store the failure somewhere, uncomment this option +# and change the directory where ktest should store the failures. +#STORE_FAILURES = ${THIS_DIR}/failures + +# The log file is stored in the OUTPUT_DIR called cross.log +# If you enable this, you need to create the OUTPUT_DIR. It wont be created for you. +LOG_FILE = ${OUTPUT_DIR}/cross.log + +# The log file will be cleared each time you run ktest. +CLEAR_LOG = 1 + +# As some archs do not build with the defconfig, they have been marked +# to be ignored. If you want to test them anyway, change DO_FAILED to 1. +# If a test that has been marked as DO_FAILED passes, then you should change +# that test to be DO_DEFAULT + +DO_FAILED := 0 +DO_DEFAULT := 1 + +# By setting both DO_FAILED and DO_DEFAULT to zero, you can pick a single +# arch that you want to test. (uncomment RUN and chose your arch) +#RUN := m32r + +# At the bottom of the config file exists a bisect test. You can update that +# test and set DO_FAILED and DO_DEFAULT to zero, and uncomment this variable +# to run the bisect on the arch. +#RUN := bisect + +# By default all tests will be running gcc 4.5.2. Some tests are using 4.5.1 +# and they select that in the test. +# Note: GCC_VER is declared as on option and not a variable ('=' instead of ':=') +# This is important. A variable is used only in the config file and if it is set +# it stays that way for the rest of the config file until it is change again. +# Here we want GCC_VER to remain persistent and change for each test, as it is used in +# the MAKE_CMD. By using '=' instead of ':=' we achieve our goal. + +GCC_VER = 4.5.2 +MAKE_CMD = PATH=/usr/local/gcc-${GCC_VER}-nolibc/${CROSS}/bin:$PATH CROSS_COMPILE=${CROSS}- make ARCH=${ARCH} + +# all tests are only doing builds. +TEST_TYPE = build + +# If you want to add configs on top of the defconfig, you can add those configs into +# the add-config file and uncomment this option. This is useful if you want to test +# all cross compiles with PREEMPT set, or TRACING on, etc. +#ADD_CONFIG = ${THIS_DIR}/add-config + +# All tests are using defconfig +BUILD_TYPE = defconfig + +# The test names will have the arch and cross compiler used. This will be shown in +# the results. +TEST_NAME = ${ARCH} ${CROSS} + +# alpha +TEST_START IF ${RUN} == alpha || ${DO_DEFAULT} +# Notice that CROSS and ARCH are also options and not variables (again '=' instead +# of ':='). This is because TEST_NAME and MAKE_CMD wil use them for each test. +# Only options are available during runs. Variables are only present in parsing the +# config file. +CROSS = alpha-linux +ARCH = alpha + +# arm +TEST_START IF ${RUN} == arm || ${DO_DEFAULT} +CROSS = arm-unknown-linux-gnueabi +ARCH = arm + +# black fin +TEST_START IF ${RUN} == bfin || ${DO_DEFAULT} +CROSS = bfin-uclinux +ARCH = blackfin +BUILD_OPTIONS = -j8 vmlinux + +# cris - FAILS? +TEST_START IF ${RUN} == cris || ${RUN} == cris64 || ${DO_FAILED} +CROSS = cris-linux +ARCH = cris + +# cris32 - not right arch? +TEST_START IF ${RUN} == cris || ${RUN} == cris32 || ${DO_FAILED} +CROSS = crisv32-linux +ARCH = cris + +# ia64 +TEST_START IF ${RUN} == ia64 || ${DO_DEFAULT} +CROSS = ia64-linux +ARCH = ia64 + +# frv +TEST_START IF ${RUN} == frv || ${DO_FAILED} +CROSS = frv-linux +ARCH = frv +GCC_VER = 4.5.1 + +# m68k fails with error? +TEST_START IF ${RUN} == m68k || ${DO_DEFAULT} +CROSS = m68k-linux +ARCH = m68k + +# mips64 +TEST_START IF ${RUN} == mips || ${RUN} == mips64 || ${DO_DEFAULT} +CROSS = mips64-linux +ARCH = mips + +# mips32 +TEST_START IF ${RUN} == mips || ${RUN} == mips32 || ${DO_DEFAULT} +CROSS = mips-linux +ARCH = mips + +# m32r +TEST_START IF ${RUN} == m32r || ${DO_FAILED} +CROSS = m32r-linux +ARCH = m32r +GCC_VER = 4.5.1 +BUILD_OPTIONS = -j8 vmlinux + +# parisc64 failed? +TEST_START IF ${RUN} == hppa || ${RUN} == hppa64 || ${DO_FAILED} +CROSS = hppa64-linux +ARCH = parisc + +# parisc +TEST_START IF ${RUN} == hppa || ${RUN} == hppa32 || ${DO_FAILED} +CROSS = hppa-linux +ARCH = parisc + +# ppc +TEST_START IF ${RUN} == ppc || ${RUN} == ppc32 || ${DO_DEFAULT} +CROSS = powerpc-linux +ARCH = powerpc + +# ppc64 +TEST_START IF ${RUN} == ppc || ${RUN} == ppc64 || ${DO_DEFAULT} +CROSS = powerpc64-linux +ARCH = powerpc + +# s390 +TEST_START IF ${RUN} == s390 || ${DO_DEFAULT} +CROSS = s390x-linux +ARCH = s390 + +# sh +TEST_START IF ${RUN} == sh || ${DO_DEFAULT} +CROSS = sh4-linux +ARCH = sh + +# sparc64 +TEST_START IF ${RUN} == sparc || ${RUN} == sparc64 || ${DO_DEFAULT} +CROSS = sparc64-linux +ARCH = sparc64 + +# sparc +TEST_START IF ${RUN} == sparc || ${RUN} == sparc32 || ${DO_DEFAULT} +CROSS = sparc-linux +ARCH = sparc + +# xtensa failed +TEST_START IF ${RUN} == xtensa || ${DO_FAILED} +CROSS = xtensa-linux +ARCH = xtensa + +# UML +TEST_START IF ${RUN} == uml || ${DO_DEFAULT} +MAKE_CMD = make ARCH=um SUBARCH=x86_64 +ARCH = uml +CROSS = + +TEST_START IF ${RUN} == x86 || ${RUN} == i386 || ${DO_DEFAULT} +MAKE_CMD = make ARCH=i386 +ARCH = i386 +CROSS = + +TEST_START IF ${RUN} == x86 || ${RUN} == x86_64 || ${DO_DEFAULT} +MAKE_CMD = make ARCH=x86_64 +ARCH = x86_64 +CROSS = + +################################# + +# This is a bisect if needed. You need to give it a MIN_CONFIG that +# will be the config file it uses. Basically, just copy the created defconfig +# for the arch someplace and point MIN_CONFIG to it. +TEST_START IF ${RUN} == bisect +MIN_CONFIG = ${THIS_DIR}/min-config +CROSS = s390x-linux +ARCH = s390 +TEST_TYPE = bisect +BISECT_TYPE = build +BISECT_GOOD = v3.1 +BISECT_BAD = v3.2 +CHECKOUT = v3.2 + +################################# + +# These defaults are needed to keep ktest.pl from complaining. They are +# ignored because the test does not go pass the build. No install or +# booting of the target images. + +DEFAULTS +MACHINE = crosstest +SSH_USER = root +BUILD_TARGET = cross +TARGET_IMAGE = image +POWER_CYCLE = cycle +CONSOLE = console +LOCALVERSION = version +GRUB_MENU = grub + +REBOOT_ON_ERROR = 0 +POWEROFF_ON_ERROR = 0 +POWEROFF_ON_SUCCESS = 0 +REBOOT_ON_SUCCESS = 0 + diff --git a/tools/testing/ktest/examples/include/bisect.conf b/tools/testing/ktest/examples/include/bisect.conf new file mode 100644 index 00000000000..009bea65bfb --- /dev/null +++ b/tools/testing/ktest/examples/include/bisect.conf @@ -0,0 +1,90 @@ +# +# This example shows the bisect tests (git bisect and config bisect) +# + + +# The config that includes this file may define a RUN_TEST +# variable that will tell this config what test to run. +# (what to set the TEST option to). +# +DEFAULTS IF NOT DEFINED RUN_TEST +# Requires that hackbench is in the PATH +RUN_TEST := ${SSH} hackbench 50 + + +# Set TEST to 'bisect' to do a normal git bisect. You need +# to modify the options below to make it bisect the exact +# commits you are interested in. +# +TEST_START IF ${TEST} == bisect +TEST_TYPE = bisect +# You must set the commit that was considered good (git bisect good) +BISECT_GOOD = v3.3 +# You must set the commit that was considered bad (git bisect bad) +BISECT_BAD = HEAD +# It's best to specify the branch to checkout before starting the bisect. +CHECKOUT = origin/master +# This can be build, boot, or test. Here we are doing a bisect +# that requires to run a test to know if the bisect was good or bad. +# The test should exit with 0 on good, non-zero for bad. But see +# the BISECT_RET_* options in samples.conf to override this. +BISECT_TYPE = test +TEST = ${RUN_TEST} +# It is usually a good idea to confirm that the GOOD and the BAD +# commits are truly good and bad respectively. Having BISECT_CHECK +# set to 1 will check both that the good commit works and the bad +# commit fails. If you only want to check one or the other, +# set BISECT_CHECK to 'good' or to 'bad'. +BISECT_CHECK = 1 +#BISECT_CHECK = good +#BISECT_CHECK = bad + +# Usually it's a good idea to specify the exact config you +# want to use throughout the entire bisect. Here we placed +# it in the directory we called ktest.pl from and named it +# 'config-bisect'. +MIN_CONFIG = ${THIS_DIR}/config-bisect +# By default, if we are doing a BISECT_TYPE = test run but the +# build or boot fails, ktest.pl will do a 'git bisect skip'. +# Uncomment the below option to make ktest stop testing on such +# an error. +#BISECT_SKIP = 0 +# Now if you had BISECT_SKIP = 0 and the test fails, you can +# examine what happened and then do 'git bisect log > /tmp/replay' +# Set BISECT_REPLAY to /tmp/replay and ktest.pl will run the +# 'git bisect replay /tmp/replay' before continuing the bisect test. +#BISECT_REPLAY = /tmp/replay +# If you used BISECT_REPLAY after the bisect test failed, you may +# not want to continue the bisect on that commit that failed. +# By setting BISECT_START to a new commit. ktest.pl will checkout +# that commit after it has performed the 'git bisect replay' but +# before it continues running the bisect test. +#BISECT_START = 2545eb6198e7e1ec50daa0cfc64a4cdfecf24ec9 + +# Now if you don't trust ktest.pl to make the decisions for you, then +# set BISECT_MANUAL to 1. This will cause ktest.pl not to decide +# if the commit was good or bad. Instead, it will ask you to tell +# it if the current commit was good. In the mean time, you could +# take the result, load it on any machine you want. Run several tests, +# or whatever you feel like. Then, when you are happy, you can tell +# ktest if you think it was good or not and ktest.pl will continue +# the git bisect. You can even change what commit it is currently at. +#BISECT_MANUAL = 1 + + +# One of the unique tests that ktest does is the config bisect. +# Currently (which hopefully will be fixed soon), the bad config +# must be a superset of the good config. This is because it only +# searches for a config that causes the target to fail. If the +# good config is not a subset of the bad config, or if the target +# fails because of a lack of a config, then it will not find +# the config for you. +TEST_START IF ${TEST} == config-bisect +TEST_TYPE = config_bisect +# set to build, boot, test +CONFIG_BISECT_TYPE = boot +# Set the config that is considered bad. +CONFIG_BISECT = ${THIS_DIR}/config-bad +# This config is optional. By default it uses the +# MIN_CONFIG as the good config. +CONFIG_BISECT_GOOD = ${THIS_DIR}/config-good diff --git a/tools/testing/ktest/examples/include/defaults.conf b/tools/testing/ktest/examples/include/defaults.conf new file mode 100644 index 00000000000..63a1a83f4f0 --- /dev/null +++ b/tools/testing/ktest/examples/include/defaults.conf @@ -0,0 +1,157 @@ +# This file holds defaults for most the tests. It defines the options that +# are most common to tests that are likely to be shared. +# +# Note, after including this file, a config file may override any option +# with a DEFAULTS OVERRIDE section. +# + +# For those cases that use the same machine to boot a 64 bit +# and a 32 bit version. The MACHINE is the DNS name to get to the +# box (usually different if it was 64 bit or 32 bit) but the +# BOX here is defined as a variable that will be the name of the box +# itself. It is useful for calling scripts that will power cycle +# the box, as only one script needs to be created to power cycle +# even though the box itself has multiple operating systems on it. +# By default, BOX and MACHINE are the same. + +DEFAULTS IF NOT DEFINED BOX +BOX := ${MACHINE} + + +# Consider each box as 64 bit box, unless the config including this file +# has defined BITS = 32 + +DEFAULTS IF NOT DEFINED BITS +BITS := 64 + + +DEFAULTS + +# THIS_DIR is used through out the configs and defaults to ${PWD} which +# is the directory that ktest.pl was called from. + +THIS_DIR := ${PWD} + + +# to organize your configs, having each machine save their configs +# into a separate directly is useful. +CONFIG_DIR := ${THIS_DIR}/configs/${MACHINE} + +# Reset the log before running each test. +CLEAR_LOG = 1 + +# As installing kernels usually requires root privilege, default the +# user on the target as root. It is also required that the target +# allows ssh to root from the host without asking for a password. + +SSH_USER = root + +# For accesing the machine, we will ssh to root@machine. +SSH := ssh ${SSH_USER}@${MACHINE} + +# Update this. The default here is ktest will ssh to the target box +# and run a script called 'run-test' located on that box. +TEST = ${SSH} run-test + +# Point build dir to the git repo you use +BUILD_DIR = ${THIS_DIR}/linux.git + +# Each machine will have its own output build directory. +OUTPUT_DIR = ${THIS_DIR}/build/${MACHINE} + +# Yes this config is focused on x86 (but ktest works for other archs too) +BUILD_TARGET = arch/x86/boot/bzImage +TARGET_IMAGE = /boot/vmlinuz-test + +# have directory for the scripts to reboot and power cycle the boxes +SCRIPTS_DIR := ${THIS_DIR}/scripts + +# You can have each box/machine have a script to power cycle it. +# Name your script <box>-cycle. +POWER_CYCLE = ${SCRIPTS_DIR}/${BOX}-cycle + +# This script is used to power off the box. +POWER_OFF = ${SCRIPTS_DIR}/${BOX}-poweroff + +# Keep your test kernels separate from your other kernels. +LOCALVERSION = -test + +# The /boot/grub/menu.lst is searched for the line: +# title Test Kernel +# and ktest will use that kernel to reboot into. +# For grub2 or other boot loaders, you need to set BOOT_TYPE +# to 'script' and define other ways to load the kernel. +# See snowball.conf example. +# +GRUB_MENU = Test Kernel + +# The kernel build will use this option. +BUILD_OPTIONS = -j8 + +# Keeping the log file with the output dir is convenient. +LOG_FILE = ${OUTPUT_DIR}/${MACHINE}.log + +# Each box should have their own minum configuration +# See min-config.conf +MIN_CONFIG = ${CONFIG_DIR}/config-min + +# For things like randconfigs, there may be configs you find that +# are already broken, or there may be some configs that you always +# want set. Uncomment ADD_CONFIG and point it to the make config files +# that set the configs you want to keep on (or off) in your build. +# ADD_CONFIG is usually something to add configs to all machines, +# where as, MIN_CONFIG is specific per machine. +#ADD_CONFIG = ${THIS_DIR}/config-broken ${THIS_DIR}/config-general + +# To speed up reboots for bisects and patchcheck, instead of +# waiting 60 seconds for the console to be idle, if this line is +# seen in the console output, ktest will know the good kernel has +# finished rebooting and it will be able to continue the tests. +REBOOT_SUCCESS_LINE = ${MACHINE} login: + +# The following is different ways to end the test. +# by setting the variable REBOOT to: none, error, fail or +# something else, ktest will power cycle or reboot the target box +# at the end of the tests. +# +# REBOOT := none +# Don't do anything at the end of the test. +# +# REBOOT := error +# Reboot the box if ktest detects an error +# +# REBOOT := fail +# Do not stop on failure, and after all tests are complete +# power off the box (for both success and error) +# This is good to run over a weekend and you don't want to waste +# electricity. +# + +DEFAULTS IF ${REBOOT} == none +REBOOT_ON_SUCCESS = 0 +REBOOT_ON_ERROR = 0 +POWEROFF_ON_ERROR = 0 +POWEROFF_ON_SUCCESS = 0 + +DEFAULTS ELSE IF ${REBOOT} == error +REBOOT_ON_SUCCESS = 0 +REBOOT_ON_ERROR = 1 +POWEROFF_ON_ERROR = 0 +POWEROFF_ON_SUCCESS = 0 + +DEFAULTS ELSE IF ${REBOOT} == fail +REBOOT_ON_SUCCESS = 0 +POWEROFF_ON_ERROR = 1 +POWEROFF_ON_SUCCESS = 1 +POWEROFF_AFTER_HALT = 120 +DIE_ON_FAILURE = 0 + +# Store the failure information into this directory +# such as the .config, dmesg, and build log. +STORE_FAILURES = ${THIS_DIR}/failures + +DEFAULTS ELSE +REBOOT_ON_SUCCESS = 1 +REBOOT_ON_ERROR = 1 +POWEROFF_ON_ERROR = 0 +POWEROFF_ON_SUCCESS = 0 diff --git a/tools/testing/ktest/examples/include/min-config.conf b/tools/testing/ktest/examples/include/min-config.conf new file mode 100644 index 00000000000..c703cc46d15 --- /dev/null +++ b/tools/testing/ktest/examples/include/min-config.conf @@ -0,0 +1,60 @@ +# +# This file has some examples for creating a MIN_CONFIG. +# (A .config file that is the minimum for a machine to boot, or +# to boot and make a network connection.) +# +# A MIN_CONFIG is very useful as it is the minimum configuration +# needed to boot a given machine. You can debug someone else's +# .config by only setting the configs in your MIN_CONFIG. The closer +# your MIN_CONFIG is to the true minimum set of configs needed to +# boot your machine, the closer the config you test with will be +# to the users config that had the failure. +# +# The make_min_config test allows you to create a MIN_CONFIG that +# is truly the minimum set of configs needed to boot a box. +# +# In this example, the final config will reside in +# ${CONFIG_DIR}/config-new-min and ${CONFIG_DIR}/config-new-min-net. +# Just move one to the location you have set for MIN_CONFIG. +# +# The first test creates a MIN_CONFIG that will be the minimum +# configuration to boot ${MACHINE} and be able to ssh to it. +# +# The second test creates a MIN_CONFIG that will only boot +# the target and most likely will not let you ssh to it. (Notice +# how the second test uses the first test's result to continue with. +# This is because the second test config is a subset of the first). +# +# The ${CONFIG_DIR}/config-skip (and -net) will hold the configs +# that ktest.pl found would not boot the target without them set. +# The config-new-min holds configs that ktest.pl could not test +# directly because another config that was needed to boot the box +# selected them. Sometimes it is possible that this file will hold +# the true minimum configuration. You can test to see if this is +# the case by running the boot test with BOOT_TYPE = allnoconfig and +# setting setting the MIN_CONFIG to ${CONFIG_DIR}/config-skip. If the +# machine still boots, then you can use the config-skip as your MIN_CONFIG. +# +# These tests can run for several hours (and perhaps days). +# It's OK to kill the test with a Ctrl^C. By restarting without +# modifying this config, ktest.pl will notice that the config-new-min(-net) +# exists, and will use that instead as the starting point. +# The USE_OUTPUT_MIN_CONFIG is set to 1 to keep ktest.pl from asking +# you if you want to use the OUTPUT_MIN_CONFIG as the starting point. +# By using the OUTPUT_MIN_CONFIG as the starting point will allow ktest.pl to +# start almost where it left off. +# +TEST_START IF ${TEST} == min-config +TEST_TYPE = make_min_config +OUTPUT_MIN_CONFIG = ${CONFIG_DIR}/config-new-min-net +IGNORE_CONFIG = ${CONFIG_DIR}/config-skip-net +MIN_CONFIG_TYPE = test +TEST = ${SSH} echo hi +USE_OUTPUT_MIN_CONFIG = 1 + +TEST_START IF ${TEST} == min-config && ${MULTI} +TEST_TYPE = make_min_config +OUTPUT_MIN_CONFIG = ${CONFIG_DIR}/config-new-min +IGNORE_CONFIG = ${CONFIG_DIR}/config-skip +MIN_CONFIG = ${CONFIG_DIR}/config-new-min-net +USE_OUTPUT_MIN_CONFIG = 1 diff --git a/tools/testing/ktest/examples/include/patchcheck.conf b/tools/testing/ktest/examples/include/patchcheck.conf new file mode 100644 index 00000000000..0eb0a5ac77d --- /dev/null +++ b/tools/testing/ktest/examples/include/patchcheck.conf @@ -0,0 +1,111 @@ +# patchcheck.conf +# +# This contains a test that takes two git commits and will test each +# commit between the two. The build test will look at what files the +# commit has touched, and if any of those files produce a warning, then +# the build will fail. + + +# PATCH_START is the commit to begin with and PATCH_END is the commit +# to end with (inclusive). This is similar to doing a git rebase -i PATCH_START~1 +# and then testing each commit and doing a git rebase --continue. +# You can use a SHA1, a git tag, or anything that git will accept for a checkout + +PATCH_START := HEAD~3 +PATCH_END := HEAD + +# Use the oldconfig if build_type wasn't defined +DEFAULTS IF NOT DEFINED BUILD_TYPE +DO_BUILD_TYPE := oldconfig + +DEFAULTS ELSE +DO_BUILD_TYPE := ${BUILD_TYPE} + +DEFAULTS + + +# Change PATCH_CHECKOUT to be the branch you want to test. The test will +# do a git checkout of this branch before starting. Obviously both +# PATCH_START and PATCH_END must be in this branch (and PATCH_START must +# be contained by PATCH_END). + +PATCH_CHECKOUT := test/branch + +# Usually it's a good idea to have a set config to use for testing individual +# patches. +PATCH_CONFIG := ${CONFIG_DIR}/config-patchcheck + +# Change PATCH_TEST to run some test for each patch. Each commit that is +# tested, after it is built and installed on the test machine, this command +# will be executed. Usually what is done is to ssh to the target box and +# run some test scripts. If you just want to boot test your patches +# comment PATCH_TEST out. +PATCH_TEST := ${SSH} "/usr/local/bin/ktest-test-script" + +DEFAULTS IF DEFINED PATCH_TEST +PATCH_TEST_TYPE := test + +DEFAULTS ELSE +PATCH_TEST_TYPE := boot + +# If for some reason a file has a warning that one of your patches touch +# but you do not care about it, set IGNORE_WARNINGS to that commit(s) +# (space delimited) +#IGNORE_WARNINGS = 39eaf7ef884dcc44f7ff1bac803ca2a1dcf43544 6edb2a8a385f0cdef51dae37ff23e74d76d8a6ce + +# Instead of just checking for warnings to files that are changed +# it can be advantageous to check for any new warnings. If a +# header file is changed, it could cause a warning in a file not +# touched by the commit. To detect these kinds of warnings, you +# can use the WARNINGS_FILE option. +# +# If the variable CREATE_WARNINGS_FILE is set, this config will +# enable the WARNINGS_FILE during the patchcheck test. Also, +# before running the patchcheck test, it will create the +# warnings file. +# +DEFAULTS IF DEFINED CREATE_WARNINGS_FILE +WARNINGS_FILE = ${OUTPUT_DIR}/warnings_file + +TEST_START IF DEFINED CREATE_WARNINGS_FILE +# WARNINGS_FILE is already set by the DEFAULTS above +TEST_TYPE = make_warnings_file +# Checkout the commit before the patches to test, +# and record all the warnings that exist before the patches +# to test are added +CHECKOUT = ${PATCHCHECK_START}~1 +# Force a full build +BUILD_NOCLEAN = 0 +BUILD_TYPE = ${DO_BUILD_TYPE} + +# If you are running a multi test, and the test failed on the first +# test but on, say the 5th patch. If you want to restart on the +# fifth patch, set PATCH_START1. This will make the first test start +# from this commit instead of the PATCH_START commit. +# Note, do not change this option. Just define PATCH_START1 in the +# top config (the one you pass to ktest.pl), and this will use it, +# otherwise it will just use PATCH_START if PATCH_START1 is not defined. +DEFAULTS IF NOT DEFINED PATCH_START1 +PATCH_START1 := ${PATCH_START} + +TEST_START IF ${TEST} == patchcheck +TEST_TYPE = patchcheck +MIN_CONFIG = ${PATCH_CONFIG} +TEST = ${PATCH_TEST} +PATCHCHECK_TYPE = ${PATCH_TEST_TYPE} +PATCHCHECK_START = ${PATCH_START1} +PATCHCHECK_END = ${PATCH_END} +CHECKOUT = ${PATCH_CHECKOUT} +BUILD_TYPE = ${DO_BUILD_TYPE} + +TEST_START IF ${TEST} == patchcheck && ${MULTI} +TEST_TYPE = patchcheck +MIN_CONFIG = ${PATCH_CONFIG} +TEST = ${PATCH_TEST} +PATCHCHECK_TYPE = ${PATCH_TEST_TYPE} +PATCHCHECK_START = ${PATCH_START} +PATCHCHECK_END = ${PATCH_END} +CHECKOUT = ${PATCH_CHECKOUT} +# Use multi to test different compilers? +MAKE_CMD = CC=gcc-4.5.1 make +BUILD_TYPE = ${DO_BUILD_TYPE} diff --git a/tools/testing/ktest/examples/include/tests.conf b/tools/testing/ktest/examples/include/tests.conf new file mode 100644 index 00000000000..60cedb1a115 --- /dev/null +++ b/tools/testing/ktest/examples/include/tests.conf @@ -0,0 +1,74 @@ +# +# This is an example of various tests that you can run +# +# The variable TEST can be of boot, build, randconfig, or test. +# +# Note that TEST is a variable created with ':=' and only exists +# throughout the config processing (not during the tests itself). +# +# The TEST option (defined with '=') is used to tell ktest.pl +# what test to run after a successful boot. The TEST option is +# persistent into the test runs. +# + +# The config that includes this file may define a BOOT_TYPE +# variable that tells this config what type of boot test to run. +# If it's not defined, the below DEFAULTS will set the default +# to 'oldconfig'. +# +DEFAULTS IF NOT DEFINED BOOT_TYPE +BOOT_TYPE := oldconfig + +# The config that includes this file may define a RUN_TEST +# variable that will tell this config what test to run. +# (what to set the TEST option to). +# +DEFAULTS IF NOT DEFINED RUN_TEST +# Requires that hackbench is in the PATH +RUN_TEST := ${SSH} hackbench 50 + + +# If TEST is set to 'boot' then just build a kernel and boot +# the target. +TEST_START IF ${TEST} == boot +TEST_TYPE = boot +# Notice how we set the BUILD_TYPE option to the BOOT_TYPE variable. +BUILD_TYPE = ${BOOT_TYPE} +# Do not do a make mrproper. +BUILD_NOCLEAN = 1 + +# If you only want to build the kernel, and perhaps install +# and test it yourself, then just set TEST to build. +TEST_START IF ${TEST} == build +TEST_TYPE = build +BUILD_TYPE = ${BOOT_TYPE} +BUILD_NOCLEAN = 1 + +# Build, install, boot and test with a randconfg 10 times. +# It is important that you have set MIN_CONFIG in the config +# that includes this file otherwise it is likely that the +# randconfig will not have the necessary configs needed to +# boot your box. This version of the test requires a min +# config that has enough to make sure the target has network +# working. +TEST_START ITERATE 10 IF ${TEST} == randconfig +MIN_CONFIG = ${CONFIG_DIR}/config-min-net +TEST_TYPE = test +BUILD_TYPE = randconfig +TEST = ${RUN_TEST} + +# This is the same as above, but only tests to a boot prompt. +# The MIN_CONFIG used here does not need to have networking +# working. +TEST_START ITERATE 10 IF ${TEST} == randconfig && ${MULTI} +TEST_TYPE = boot +BUILD_TYPE = randconfig +MIN_CONFIG = ${CONFIG_DIR}/config-min +MAKE_CMD = make + +# This builds, installs, boots and tests the target. +TEST_START IF ${TEST} == test +TEST_TYPE = test +BUILD_TYPE = ${BOOT_TYPE} +TEST = ${RUN_TEST} +BUILD_NOCLEAN = 1 diff --git a/tools/testing/ktest/examples/kvm.conf b/tools/testing/ktest/examples/kvm.conf new file mode 100644 index 00000000000..fbc134f9ac6 --- /dev/null +++ b/tools/testing/ktest/examples/kvm.conf @@ -0,0 +1,92 @@ +# +# This config is an example usage of ktest.pl with a kvm guest +# +# The guest is called 'Guest' and this would be something that +# could be run on the host to test a virtual machine target. + +MACHINE = Guest + + +# Use virsh to read the serial console of the guest +CONSOLE = virsh console ${MACHINE} + +# Use SIGKILL to terminate virsh console. We can't kill virsh console +# by the default signal, SIGINT. +CLOSE_CONSOLE_SIGNAL = KILL + +#*************************************# +# This part is the same as test.conf # +#*************************************# + +# The include files will set up the type of test to run. Just set TEST to +# which test you want to run. +# +# TESTS = patchcheck, randconfig, boot, test, config-bisect, bisect, min-config +# +# See the include/*.conf files that define these tests +# +TEST := patchcheck + +# Some tests may have more than one test to run. Define MULTI := 1 to run +# the extra tests. +MULTI := 0 + +# In case you want to differentiate which type of system you are testing +BITS := 64 + +# REBOOT = none, error, fail, empty +# See include/defaults.conf +REBOOT := empty + + +# The defaults file will set up various settings that can be used by all +# machine configs. +INCLUDE include/defaults.conf + + +#*************************************# +# Now we are different from test.conf # +#*************************************# + + +# The example here assumes that Guest is running a Fedora release +# that uses dracut for its initfs. The POST_INSTALL will be executed +# after the install of the kernel and modules are complete. +# +POST_INSTALL = ${SSH} /sbin/dracut -f /boot/initramfs-test.img $KERNEL_VERSION + +# Guests sometimes get stuck on reboot. We wait 3 seconds after running +# the reboot command and then do a full power-cycle of the guest. +# This forces the guest to restart. +# +POWERCYCLE_AFTER_REBOOT = 3 + +# We do the same after the halt command, but this time we wait 20 seconds. +POWEROFF_AFTER_HALT = 20 + + +# As the defaults.conf file has a POWER_CYCLE option already defined, +# and options can not be defined in the same section more than once +# (all DEFAULTS sections are considered the same). We use the +# DEFAULTS OVERRIDE to tell ktest.pl to ignore the previous defined +# options, for the options set in the OVERRIDE section. +# +DEFAULTS OVERRIDE + +# Instead of using the default POWER_CYCLE option defined in +# defaults.conf, we use virsh to cycle it. To do so, we destroy +# the guest, wait 5 seconds, and then start it up again. +# Crude, but effective. +# +POWER_CYCLE = virsh destroy ${MACHINE}; sleep 5; virsh start ${MACHINE} + + +DEFAULTS + +# The following files each handle a different test case. +# Having them included allows you to set up more than one machine and share +# the same tests. +INCLUDE include/patchcheck.conf +INCLUDE include/tests.conf +INCLUDE include/bisect.conf +INCLUDE include/min-config.conf diff --git a/tools/testing/ktest/examples/snowball.conf b/tools/testing/ktest/examples/snowball.conf new file mode 100644 index 00000000000..a82a3c5bc2b --- /dev/null +++ b/tools/testing/ktest/examples/snowball.conf @@ -0,0 +1,53 @@ +# This example was used to boot the snowball ARM board. +# See http://people.redhat.com/srostedt/ktest-embedded-2012/ + +# PWD is a ktest.pl variable that will result in the process working +# directory that ktest.pl is executed in. + +# THIS_DIR is automatically assigned the PWD of the path that generated +# the config file. It is best to use this variable when assigning other +# directory paths within this directory. This allows you to easily +# move the test cases to other locations or to other machines. +# +THIS_DIR := /home/rostedt/work/demo/ktest-embed +LOG_FILE = ${OUTPUT_DIR}/snowball.log +CLEAR_LOG = 1 +MAKE_CMD = PATH=/usr/local/gcc-4.5.2-nolibc/arm-unknown-linux-gnueabi/bin:$PATH CROSS_COMPILE=arm-unknown-linux-gnueabi- make ARCH=arm +ADD_CONFIG = ${THIS_DIR}/addconfig + +SCP_TO_TARGET = echo "don't do scp" + +TFTPBOOT := /var/lib/tftpboot +TFTPDEF := ${TFTPBOOT}/snowball-default +TFTPTEST := ${OUTPUT_DIR}/${BUILD_TARGET} + +SWITCH_TO_GOOD = cp ${TFTPDEF} ${TARGET_IMAGE} +SWITCH_TO_TEST = cp ${TFTPTEST} ${TARGET_IMAGE} + +# Define each test with TEST_START +# The config options below it will override the defaults +TEST_START SKIP +TEST_TYPE = boot +BUILD_TYPE = u8500_defconfig +BUILD_NOCLEAN = 1 + +TEST_START +TEST_TYPE = make_min_config +OUTPUT_MIN_CONFIG = ${THIS_DIR}/config.newmin +START_MIN_CONFIG = ${THIS_DIR}/config.orig +IGNORE_CONFIG = ${THIS_DIR}/config.ignore +BUILD_NOCLEAN = 1 + + +DEFAULTS +LOCALVERSION = -test +POWER_CYCLE = echo use the thumb luke; read a +CONSOLE = cat ${THIS_DIR}/snowball-cat +REBOOT_TYPE = script +SSH_USER = root +BUILD_OPTIONS = -j8 uImage +BUILD_DIR = ${THIS_DIR}/linux.git +OUTPUT_DIR = ${THIS_DIR}/snowball-build +MACHINE = snowball +TARGET_IMAGE = /var/lib/tftpboot/snowball-image +BUILD_TARGET = arch/arm/boot/uImage diff --git a/tools/testing/ktest/examples/test.conf b/tools/testing/ktest/examples/test.conf new file mode 100644 index 00000000000..b725210efb7 --- /dev/null +++ b/tools/testing/ktest/examples/test.conf @@ -0,0 +1,62 @@ +# +# Generic config for a machine +# + +# Name your machine (the DNS name, what you ssh to) +MACHINE = foo + +# BOX can be different than foo, if the machine BOX has +# multiple partitions with different systems installed. For example, +# you may have a i386 and x86_64 installation on a test box. +# If this is the case, MACHINE defines the way to connect to the +# machine, which may be different between which system the machine +# is booting into. BOX is used for the scripts to reboot and power cycle +# the machine, where it does not matter which system the machine boots into. +# +#BOX := bar + +# Define a way to read the console +CONSOLE = stty -F /dev/ttyS0 115200 parodd; cat /dev/ttyS0 + +# The include files will set up the type of test to run. Just set TEST to +# which test you want to run. +# +# TESTS = patchcheck, randconfig, boot, test, config-bisect, bisect, min-config +# +# See the include/*.conf files that define these tests +# +TEST := patchcheck + +# Some tests may have more than one test to run. Define MULTI := 1 to run +# the extra tests. +MULTI := 0 + +# In case you want to differentiate which type of system you are testing +BITS := 64 + +# REBOOT = none, error, fail, empty +# See include/defaults.conf +REBOOT := empty + +# The defaults file will set up various settings that can be used by all +# machine configs. +INCLUDE include/defaults.conf + +# In case you need to add a patch for a bisect or something +#PRE_BUILD = patch -p1 < ${THIS_DIR}/fix.patch + +# Reset the repo after the build and remove all 'test' modules from the target +# Notice that DO_POST_BUILD is a variable (defined by ':=') and POST_BUILD +# is the option (defined by '=') + +DO_POST_BUILD := git reset --hard +POST_BUILD = ${SSH} 'rm -rf /lib/modules/*-test*'; ${DO_POST_BUILD} + +# The following files each handle a different test case. +# Having them included allows you to set up more than one machine and share +# the same tests. +INCLUDE include/patchcheck.conf +INCLUDE include/tests.conf +INCLUDE include/bisect.conf +INCLUDE include/min-config.conf + diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl index 9507c4b251a..40631569a0f 100755 --- a/tools/testing/ktest/ktest.pl +++ b/tools/testing/ktest/ktest.pl @@ -18,6 +18,7 @@ $| = 1; my %opt; my %repeat_tests; my %repeats; +my %evals; #default opts my %default = ( @@ -25,6 +26,7 @@ my %default = ( "TEST_TYPE" => "build", "BUILD_TYPE" => "randconfig", "MAKE_CMD" => "make", + "CLOSE_CONSOLE_SIGNAL" => "INT", "TIMEOUT" => 120, "TMP_DIR" => "/tmp/ktest/\${MACHINE}", "SLEEP_TIME" => 60, # sleep time between tests @@ -39,6 +41,8 @@ my %default = ( "CLEAR_LOG" => 0, "BISECT_MANUAL" => 0, "BISECT_SKIP" => 1, + "BISECT_TRIES" => 1, + "MIN_CONFIG_TYPE" => "boot", "SUCCESS_LINE" => "login:", "DETECT_TRIPLE_FAULT" => 1, "NO_INSTALL" => 0, @@ -46,10 +50,15 @@ my %default = ( "DIE_ON_FAILURE" => 1, "SSH_EXEC" => "ssh \$SSH_USER\@\$MACHINE \$SSH_COMMAND", "SCP_TO_TARGET" => "scp \$SRC_FILE \$SSH_USER\@\$MACHINE:\$DST_FILE", + "SCP_TO_TARGET_INSTALL" => "\${SCP_TO_TARGET}", "REBOOT" => "ssh \$SSH_USER\@\$MACHINE reboot", "STOP_AFTER_SUCCESS" => 10, "STOP_AFTER_FAILURE" => 60, "STOP_TEST_AFTER" => 600, + "MAX_MONITOR_WAIT" => 1800, + "GRUB_REBOOT" => "grub2-reboot", + "SYSLINUX" => "extlinux", + "SYSLINUX_PATH" => "/boot/extlinux", # required, and we will ask users if they don't have them but we keep the default # value something that is common. @@ -65,7 +74,9 @@ my %default = ( my $ktest_config; my $version; +my $have_version = 0; my $machine; +my $last_machine; my $ssh_user; my $tmpdir; my $builddir; @@ -74,6 +85,11 @@ my $output_config; my $test_type; my $build_type; my $build_options; +my $final_post_ktest; +my $pre_ktest; +my $post_ktest; +my $pre_test; +my $post_test; my $pre_build; my $post_build; my $pre_build_die; @@ -86,16 +102,26 @@ my $reboot_on_error; my $switch_to_good; my $switch_to_test; my $poweroff_on_error; +my $reboot_on_success; my $die_on_failure; my $powercycle_after_reboot; my $poweroff_after_halt; +my $max_monitor_wait; my $ssh_exec; my $scp_to_target; +my $scp_to_target_install; my $power_off; my $grub_menu; +my $last_grub_menu; +my $grub_file; my $grub_number; +my $grub_reboot; +my $syslinux; +my $syslinux_path; +my $syslinux_label; my $target; my $make; +my $pre_install; my $post_install; my $no_install; my $noclean; @@ -103,6 +129,9 @@ my $minconfig; my $start_minconfig; my $start_minconfig_defined; my $output_minconfig; +my $minconfig_type; +my $use_output_minconfig; +my $warnings_file; my $ignore_config; my $ignore_errors; my $addconfig; @@ -111,6 +140,7 @@ my $bisect_bad_commit = ""; my $reverse_bisect; my $bisect_manual; my $bisect_skip; +my $bisect_tries; my $config_bisect_good; my $bisect_ret_good; my $bisect_ret_bad; @@ -137,6 +167,7 @@ my $timeout; my $booted_timeout; my $detect_triplefault; my $console; +my $close_console_signal; my $reboot_success_line; my $success_line; my $stop_after_success; @@ -160,6 +191,7 @@ my $bisect_check; my $config_bisect; my $config_bisect_type; +my $config_bisect_check; my $patchcheck_type; my $patchcheck_start; @@ -169,17 +201,26 @@ my $patchcheck_end; # which would require more options. my $buildonly = 1; +# tell build not to worry about warnings, even when WARNINGS_FILE is set +my $warnings_ok = 0; + # set when creating a new config my $newconfig = 0; my %entered_configs; my %config_help; my %variable; + +# force_config is the list of configs that we force enabled (or disabled) +# in a .config file. The MIN_CONFIG and ADD_CONFIG configs. my %force_config; # do not force reboots on config problems my $no_reboot = 1; +# reboot on success +my $reboot_success = 0; + my %option_map = ( "MACHINE" => \$machine, "SSH_USER" => \$ssh_user, @@ -187,6 +228,10 @@ my %option_map = ( "OUTPUT_DIR" => \$outputdir, "BUILD_DIR" => \$builddir, "TEST_TYPE" => \$test_type, + "PRE_KTEST" => \$pre_ktest, + "POST_KTEST" => \$post_ktest, + "PRE_TEST" => \$pre_test, + "POST_TEST" => \$post_test, "BUILD_TYPE" => \$build_type, "BUILD_OPTIONS" => \$build_options, "PRE_BUILD" => \$pre_build, @@ -199,11 +244,20 @@ my %option_map = ( "MIN_CONFIG" => \$minconfig, "OUTPUT_MIN_CONFIG" => \$output_minconfig, "START_MIN_CONFIG" => \$start_minconfig, + "MIN_CONFIG_TYPE" => \$minconfig_type, + "USE_OUTPUT_MIN_CONFIG" => \$use_output_minconfig, + "WARNINGS_FILE" => \$warnings_file, "IGNORE_CONFIG" => \$ignore_config, "TEST" => \$run_test, "ADD_CONFIG" => \$addconfig, "REBOOT_TYPE" => \$reboot_type, "GRUB_MENU" => \$grub_menu, + "GRUB_FILE" => \$grub_file, + "GRUB_REBOOT" => \$grub_reboot, + "SYSLINUX" => \$syslinux, + "SYSLINUX_PATH" => \$syslinux_path, + "SYSLINUX_LABEL" => \$syslinux_label, + "PRE_INSTALL" => \$pre_install, "POST_INSTALL" => \$post_install, "NO_INSTALL" => \$no_install, "REBOOT_SCRIPT" => \$reboot_script, @@ -211,10 +265,12 @@ my %option_map = ( "SWITCH_TO_GOOD" => \$switch_to_good, "SWITCH_TO_TEST" => \$switch_to_test, "POWEROFF_ON_ERROR" => \$poweroff_on_error, + "REBOOT_ON_SUCCESS" => \$reboot_on_success, "DIE_ON_FAILURE" => \$die_on_failure, "POWER_OFF" => \$power_off, "POWERCYCLE_AFTER_REBOOT" => \$powercycle_after_reboot, "POWEROFF_AFTER_HALT" => \$poweroff_after_halt, + "MAX_MONITOR_WAIT" => \$max_monitor_wait, "SLEEP_TIME" => \$sleep_time, "BISECT_SLEEP_TIME" => \$bisect_sleep_time, "PATCHCHECK_SLEEP_TIME" => \$patchcheck_sleep_time, @@ -222,6 +278,7 @@ my %option_map = ( "IGNORE_ERRORS" => \$ignore_errors, "BISECT_MANUAL" => \$bisect_manual, "BISECT_SKIP" => \$bisect_skip, + "BISECT_TRIES" => \$bisect_tries, "CONFIG_BISECT_GOOD" => \$config_bisect_good, "BISECT_RET_GOOD" => \$bisect_ret_good, "BISECT_RET_BAD" => \$bisect_ret_bad, @@ -234,6 +291,7 @@ my %option_map = ( "TIMEOUT" => \$timeout, "BOOTED_TIMEOUT" => \$booted_timeout, "CONSOLE" => \$console, + "CLOSE_CONSOLE_SIGNAL" => \$close_console_signal, "DETECT_TRIPLE_FAULT" => \$detect_triplefault, "SUCCESS_LINE" => \$success_line, "REBOOT_SUCCESS_LINE" => \$reboot_success_line, @@ -243,6 +301,7 @@ my %option_map = ( "BUILD_TARGET" => \$build_target, "SSH_EXEC" => \$ssh_exec, "SCP_TO_TARGET" => \$scp_to_target, + "SCP_TO_TARGET_INSTALL" => \$scp_to_target_install, "CHECKOUT" => \$checkout, "TARGET_IMAGE" => \$target_image, "LOCALVERSION" => \$localversion, @@ -258,6 +317,7 @@ my %option_map = ( "CONFIG_BISECT" => \$config_bisect, "CONFIG_BISECT_TYPE" => \$config_bisect_type, + "CONFIG_BISECT_CHECK" => \$config_bisect_check, "PATCHCHECK_TYPE" => \$patchcheck_type, "PATCHCHECK_START" => \$patchcheck_start, @@ -335,7 +395,7 @@ EOF ; $config_help{"REBOOT_TYPE"} = << "EOF" Way to reboot the box to the test kernel. - Only valid options so far are "grub" and "script". + Only valid options so far are "grub", "grub2", "syslinux", and "script". If you specify grub, it will assume grub version 1 and will search in /boot/grub/menu.lst for the title \$GRUB_MENU @@ -345,11 +405,19 @@ $config_help{"REBOOT_TYPE"} = << "EOF" The entry in /boot/grub/menu.lst must be entered in manually. The test will not modify that file. + + If you specify grub2, then you also need to specify both \$GRUB_MENU + and \$GRUB_FILE. + + If you specify syslinux, then you may use SYSLINUX to define the syslinux + command (defaults to extlinux), and SYSLINUX_PATH to specify the path to + the syslinux install (defaults to /boot/extlinux). But you have to specify + SYSLINUX_LABEL to define the label to boot to for the test kernel. EOF ; $config_help{"GRUB_MENU"} = << "EOF" The grub title name for the test kernel to boot - (Only mandatory if REBOOT_TYPE = grub) + (Only mandatory if REBOOT_TYPE = grub or grub2) Note, ktest.pl will not update the grub menu.lst, you need to manually add an option for the test. ktest.pl will search @@ -360,6 +428,22 @@ $config_help{"GRUB_MENU"} = << "EOF" title Test Kernel kernel vmlinuz-test GRUB_MENU = Test Kernel + + For grub2, a search of \$GRUB_FILE is performed for the lines + that begin with "menuentry". It will not detect submenus. The + menu must be a non-nested menu. Add the quotes used in the menu + to guarantee your selection, as the first menuentry with the content + of \$GRUB_MENU that is found will be used. +EOF + ; +$config_help{"GRUB_FILE"} = << "EOF" + If grub2 is used, the full path for the grub.cfg file is placed + here. Use something like /boot/grub2/grub.cfg to search. +EOF + ; +$config_help{"SYSLINUX_LABEL"} = << "EOF" + If syslinux is used, the label that boots the target kernel must + be specified with SYSLINUX_LABEL. EOF ; $config_help{"REBOOT_SCRIPT"} = << "EOF" @@ -368,6 +452,27 @@ $config_help{"REBOOT_SCRIPT"} = << "EOF" EOF ; +sub _logit { + if (defined($opt{"LOG_FILE"})) { + open(OUT, ">> $opt{LOG_FILE}") or die "Can't write to $opt{LOG_FILE}"; + print OUT @_; + close(OUT); + } +} + +sub logit { + if (defined($opt{"LOG_FILE"})) { + _logit @_; + } else { + print @_; + } +} + +sub doprint { + print @_; + _logit @_; +} + sub read_prompt { my ($cancel, $prompt) = @_; @@ -488,6 +593,15 @@ sub get_ktest_configs { if ($rtype eq "grub") { get_ktest_config("GRUB_MENU"); } + + if ($rtype eq "grub2") { + get_ktest_config("GRUB_MENU"); + get_ktest_config("GRUB_FILE"); + } + + if ($rtype eq "syslinux") { + get_ktest_config("SYSLINUX_LABEL"); + } } sub process_variables { @@ -540,6 +654,18 @@ sub set_value { # Note if a test is something other than build, then we # will need other manditory options. if ($prvalue ne "install") { + # for bisect, we need to check BISECT_TYPE + if ($prvalue ne "bisect") { + $buildonly = 0; + } + } else { + # install still limits some manditory options. + $buildonly = 2; + } + } + + if ($buildonly && $lvalue =~ /^BISECT_TYPE(\[.*\])?$/ && $prvalue ne "build") { + if ($prvalue ne "install") { $buildonly = 0; } else { # install still limits some manditory options. @@ -564,6 +690,22 @@ sub set_value { } } +sub set_eval { + my ($lvalue, $rvalue, $name) = @_; + + my $prvalue = process_variables($rvalue); + my $arr; + + if (defined($evals{$lvalue})) { + $arr = $evals{$lvalue}; + } else { + $arr = []; + $evals{$lvalue} = $arr; + } + + push @{$arr}, $rvalue; +} + sub set_variable { my ($lvalue, $rvalue) = @_; @@ -590,6 +732,10 @@ sub process_compare { return $lval eq $rval; } elsif ($cmp eq "!=") { return $lval ne $rval; + } elsif ($cmp eq "=~") { + return $lval =~ m/$rval/; + } elsif ($cmp eq "!~") { + return $lval !~ m/$rval/; } my $statement = "$lval $cmp $rval"; @@ -645,7 +791,7 @@ sub process_expression { } } - if ($val =~ /(.*)(==|\!=|>=|<=|>|<)(.*)/) { + if ($val =~ /(.*)(==|\!=|>=|<=|>|<|=~|\!~)(.*)/) { my $ret = process_compare($1, $2, $3); if ($ret < 0) { die "$name: $.: Unable to process comparison\n"; @@ -803,7 +949,9 @@ sub __read_config { if ($rest =~ /\sIF\s+(.*)/) { # May be a ELSE IF section. - if (!process_if($name, $1)) { + if (process_if($name, $1)) { + $if_set = 1; + } else { $skip = 1; } $rest = ""; @@ -843,6 +991,20 @@ sub __read_config { $test_case = 1; } + } elsif (/^\s*([A-Z_\[\]\d]+)\s*=~\s*(.*?)\s*$/) { + + next if ($skip); + + my $lvalue = $1; + my $rvalue = $2; + + if ($default || $lvalue =~ /\[\d+\]$/) { + set_eval($lvalue, $rvalue, $name); + } else { + my $val = "$lvalue\[$test_num\]"; + set_eval($val, $rvalue, $name); + } + } elsif (/^\s*([A-Z_\[\]\d]+)\s*=\s*(.*?)\s*$/) { next if ($skip); @@ -977,7 +1139,7 @@ sub read_config { } sub __eval_option { - my ($option, $i) = @_; + my ($name, $option, $i) = @_; # Add space to evaluate the character before $ $option = " $option"; @@ -1009,7 +1171,11 @@ sub __eval_option { my $o = "$var\[$i\]"; my $parento = "$var\[$parent\]"; - if (defined($opt{$o})) { + # If a variable contains itself, use the default var + if (($var eq $name) && defined($opt{$var})) { + $o = $opt{$var}; + $retval = "$retval$o"; + } elsif (defined($opt{$o})) { $o = $opt{$o}; $retval = "$retval$o"; } elsif ($repeated && defined($opt{$parento})) { @@ -1018,6 +1184,10 @@ sub __eval_option { } elsif (defined($opt{$var})) { $o = $opt{$var}; $retval = "$retval$o"; + } elsif ($var eq "KERNEL_VERSION" && defined($make)) { + # special option KERNEL_VERSION uses kernel version + get_version(); + $retval = "$retval$version"; } else { $retval = "$retval\$\{$var\}"; } @@ -1032,8 +1202,35 @@ sub __eval_option { return $retval; } +sub process_evals { + my ($name, $option, $i) = @_; + + my $option_name = "$name\[$i\]"; + my $ev; + + my $old_option = $option; + + if (defined($evals{$option_name})) { + $ev = $evals{$option_name}; + } elsif (defined($evals{$name})) { + $ev = $evals{$name}; + } else { + return $option; + } + + for my $e (@{$ev}) { + eval "\$option =~ $e"; + } + + if ($option ne $old_option) { + doprint("$name changed from '$old_option' to '$option'\n"); + } + + return $option; +} + sub eval_option { - my ($option, $i) = @_; + my ($name, $option, $i) = @_; my $prev = ""; @@ -1049,31 +1246,12 @@ sub eval_option { "Check for recursive variables\n"; } $prev = $option; - $option = __eval_option($option, $i); - } - - return $option; -} - -sub _logit { - if (defined($opt{"LOG_FILE"})) { - open(OUT, ">> $opt{LOG_FILE}") or die "Can't write to $opt{LOG_FILE}"; - print OUT @_; - close(OUT); + $option = __eval_option($name, $option, $i); } -} -sub logit { - if (defined($opt{"LOG_FILE"})) { - _logit @_; - } else { - print @_; - } -} + $option = process_evals($name, $option, $i); -sub doprint { - print @_; - _logit @_; + return $option; } sub run_command; @@ -1084,6 +1262,9 @@ sub wait_for_monitor; sub reboot { my ($time) = @_; + # Make sure everything has been written to disk + run_ssh("sync"); + if (defined($time)) { start_monitor; # flush out current monitor @@ -1103,7 +1284,24 @@ sub reboot { } if (defined($time)) { + + # We only want to get to the new kernel, don't fail + # if we stumble over a call trace. + my $save_ignore_errors = $ignore_errors; + $ignore_errors = 1; + + # Look for the good kernel to boot + if (wait_for_monitor($time, "Linux version")) { + # reboot got stuck? + doprint "Reboot did not finish. Forcing power cycle\n"; + run_command "$power_cycle"; + } + + $ignore_errors = $save_ignore_errors; + + # Still need to wait for the reboot to finish wait_for_monitor($time, $reboot_success_line); + end_monitor; } } @@ -1113,7 +1311,6 @@ sub reboot_to_good { if (defined($switch_to_good)) { run_command $switch_to_good; - return; } reboot $time; @@ -1169,7 +1366,7 @@ sub close_console { my ($fp, $pid) = @_; doprint "kill child process $pid\n"; - kill 2, $pid; + kill $close_console_signal, $pid; print "closing!\n"; close($fp); @@ -1188,6 +1385,7 @@ sub start_monitor { } sub end_monitor { + return if (!defined $console); if (--$monitor_cnt) { return; } @@ -1199,6 +1397,11 @@ sub wait_for_monitor { my $full_line = ""; my $line; my $booted = 0; + my $start_time = time; + my $skip_call_trace = 0; + my $bug = 0; + my $bug_ignored = 0; + my $now; doprint "** Wait for monitor to settle down **\n"; @@ -1214,11 +1417,39 @@ sub wait_for_monitor { $booted = 1; } + if ($full_line =~ /\[ backtrace testing \]/) { + $skip_call_trace = 1; + } + + if ($full_line =~ /call trace:/i) { + if (!$bug && !$skip_call_trace) { + if ($ignore_errors) { + $bug_ignored = 1; + } else { + $bug = 1; + } + } + } + + if ($full_line =~ /\[ end of backtrace testing \]/) { + $skip_call_trace = 0; + } + + if ($full_line =~ /Kernel panic -/) { + $bug = 1; + } + if ($line =~ /\n/) { $full_line = ""; } + $now = time; + if ($now - $start_time >= $max_monitor_wait) { + doprint "Exiting monitor flush due to hitting MAX_MONITOR_WAIT\n"; + return 1; + } } print "** Monitor flushed **\n"; + return $bug; } sub save_logs { @@ -1260,6 +1491,10 @@ sub save_logs { sub fail { + if (defined($post_test)) { + run_command $post_test; + } + if ($die_on_failure) { dodie @_; } @@ -1349,8 +1584,7 @@ sub run_ssh { } sub run_scp { - my ($src, $dst) = @_; - my $cp_scp = $scp_to_target; + my ($src, $dst, $cp_scp) = @_; $cp_scp =~ s/\$SRC_FILE/$src/g; $cp_scp =~ s/\$DST_FILE/$dst/g; @@ -1358,12 +1592,70 @@ sub run_scp { return run_command "$cp_scp"; } +sub run_scp_install { + my ($src, $dst) = @_; + + my $cp_scp = $scp_to_target_install; + + return run_scp($src, $dst, $cp_scp); +} + +sub run_scp_mod { + my ($src, $dst) = @_; + + my $cp_scp = $scp_to_target; + + return run_scp($src, $dst, $cp_scp); +} + +sub get_grub2_index { + + return if (defined($grub_number) && defined($last_grub_menu) && + $last_grub_menu eq $grub_menu && defined($last_machine) && + $last_machine eq $machine); + + doprint "Find grub2 menu ... "; + $grub_number = -1; + + my $ssh_grub = $ssh_exec; + $ssh_grub =~ s,\$SSH_COMMAND,cat $grub_file,g; + + open(IN, "$ssh_grub |") + or die "unable to get $grub_file"; + + my $found = 0; + + while (<IN>) { + if (/^menuentry.*$grub_menu/) { + $grub_number++; + $found = 1; + last; + } elsif (/^menuentry\s/) { + $grub_number++; + } + } + close(IN); + + die "Could not find '$grub_menu' in $grub_file on $machine" + if (!$found); + doprint "$grub_number\n"; + $last_grub_menu = $grub_menu; + $last_machine = $machine; +} + sub get_grub_index { + if ($reboot_type eq "grub2") { + get_grub2_index; + return; + } + if ($reboot_type ne "grub") { return; } - return if (defined($grub_number)); + return if (defined($grub_number) && defined($last_grub_menu) && + $last_grub_menu eq $grub_menu && defined($last_machine) && + $last_machine eq $machine); doprint "Find grub menu ... "; $grub_number = -1; @@ -1390,6 +1682,8 @@ sub get_grub_index { die "Could not find '$grub_menu' in /boot/grub/menu on $machine" if (!$found); doprint "$grub_number\n"; + $last_grub_menu = $grub_menu; + $last_machine = $machine; } sub wait_for_input @@ -1406,7 +1700,7 @@ sub wait_for_input $rin = ''; vec($rin, fileno($fp), 1) = 1; - $ready = select($rin, undef, undef, $time); + ($ready, $time) = select($rin, undef, undef, $time); $line = ""; @@ -1430,6 +1724,10 @@ sub reboot_to { if ($reboot_type eq "grub") { run_ssh "'(echo \"savedefault --default=$grub_number --once\" | grub --batch)'"; + } elsif ($reboot_type eq "grub2") { + run_ssh "$grub_reboot $grub_number"; + } elsif ($reboot_type eq "syslinux") { + run_ssh "$syslinux --once \\\"$syslinux_label\\\" $syslinux_path"; } elsif (defined $reboot_script) { run_command "$reboot_script"; } @@ -1460,6 +1758,7 @@ sub get_sha1 { sub monitor { my $booted = 0; my $bug = 0; + my $bug_ignored = 0; my $skip_call_trace = 0; my $loops; @@ -1531,9 +1830,13 @@ sub monitor { } if ($full_line =~ /call trace:/i) { - if (!$ignore_errors && !$bug && !$skip_call_trace) { - $bug = 1; - $failure_start = time; + if (!$bug && !$skip_call_trace) { + if ($ignore_errors) { + $bug_ignored = 1; + } else { + $bug = 1; + $failure_start = time; + } } } @@ -1563,7 +1866,7 @@ sub monitor { # We already booted into the kernel we are testing, # but now we booted into another kernel? # Consider this a triple fault. - doprint "Aleady booted in Linux kernel $version, but now\n"; + doprint "Already booted in Linux kernel $version, but now\n"; doprint "we booted into Linux kernel $1.\n"; doprint "Assuming that this is a triple fault.\n"; doprint "To disable this: set DETECT_TRIPLE_FAULT to 0\n"; @@ -1595,6 +1898,10 @@ sub monitor { fail "failed - never got a boot prompt." and return 0; } + if ($bug_ignored) { + doprint "WARNING: Call Trace detected but ignored due to IGNORE_ERRORS=1\n"; + } + return 1; } @@ -1615,13 +1922,29 @@ sub do_post_install { dodie "Failed to run post install"; } +# Sometimes the reboot fails, and will hang. We try to ssh to the box +# and if we fail, we force another reboot, that should powercycle it. +sub test_booted { + if (!run_ssh "echo testing connection") { + reboot $sleep_time; + } +} + sub install { return if ($no_install); + if (defined($pre_install)) { + my $cp_pre_install = eval_kernel_version $pre_install; + run_command "$cp_pre_install" or + dodie "Failed to run pre install"; + } + my $cp_target = eval_kernel_version $target_image; - run_scp "$outputdir/$build_target", "$cp_target" or + test_booted; + + run_scp_install "$outputdir/$build_target", "$cp_target" or dodie "failed to copy image"; my $install_mods = 0; @@ -1631,8 +1954,10 @@ sub install { open(IN, "$output_config") or dodie("Can't read config file"); while (<IN>) { if (/CONFIG_MODULES(=y)?/) { - $install_mods = 1 if (defined($1)); - last; + if (defined($1)) { + $install_mods = 1; + last; + } } } close(IN); @@ -1643,7 +1968,7 @@ sub install { return; } - run_command "$make INSTALL_MOD_PATH=$tmpdir modules_install" or + run_command "$make INSTALL_MOD_STRIP=1 INSTALL_MOD_PATH=$tmpdir modules_install" or dodie "Failed to install modules"; my $modlib = "/lib/modules/$version"; @@ -1656,7 +1981,7 @@ sub install { run_command "cd $tmpdir && tar -cjf $modtar lib/modules/$version" or dodie "making tarball"; - run_scp "$tmpdir/$modtar", "/tmp" or + run_scp_mod "$tmpdir/$modtar", "/tmp" or dodie "failed to copy modules"; unlink "$tmpdir/$modtar"; @@ -1671,31 +1996,112 @@ sub install { sub get_version { # get the release name + return if ($have_version); doprint "$make kernelrelease ... "; $version = `$make kernelrelease | tail -1`; chomp($version); doprint "$version\n"; + $have_version = 1; } sub start_monitor_and_boot { # Make sure the stable kernel has finished booting - start_monitor; - wait_for_monitor 5; - end_monitor; + + # Install bisects, don't need console + if (defined $console) { + start_monitor; + wait_for_monitor 5; + end_monitor; + } get_grub_index; get_version; install; - start_monitor; + start_monitor if (defined $console); return monitor; } +my $check_build_re = ".*:.*(warning|error|Error):.*"; +my $utf8_quote = "\\x{e2}\\x{80}(\\x{98}|\\x{99})"; + +sub process_warning_line { + my ($line) = @_; + + chomp $line; + + # for distcc heterogeneous systems, some compilers + # do things differently causing warning lines + # to be slightly different. This makes an attempt + # to fixe those issues. + + # chop off the index into the line + # using distcc, some compilers give different indexes + # depending on white space + $line =~ s/^(\s*\S+:\d+:)\d+/$1/; + + # Some compilers use UTF-8 extended for quotes and some don't. + $line =~ s/$utf8_quote/'/g; + + return $line; +} + +# Read buildlog and check against warnings file for any +# new warnings. +# +# Returns 1 if OK +# 0 otherwise sub check_buildlog { + return 1 if (!defined $warnings_file); + + my %warnings_list; + + # Failed builds should not reboot the target + my $save_no_reboot = $no_reboot; + $no_reboot = 1; + + if (-f $warnings_file) { + open(IN, $warnings_file) or + dodie "Error opening $warnings_file"; + + while (<IN>) { + if (/$check_build_re/) { + my $warning = process_warning_line $_; + + $warnings_list{$warning} = 1; + } + } + close(IN); + } + + # If warnings file didn't exist, and WARNINGS_FILE exist, + # then we fail on any warning! + + open(IN, $buildlog) or dodie "Can't open $buildlog"; + while (<IN>) { + if (/$check_build_re/) { + my $warning = process_warning_line $_; + + if (!defined $warnings_list{$warning}) { + fail "New warning found (not in $warnings_file)\n$_\n"; + $no_reboot = $save_no_reboot; + return 0; + } + } + } + $no_reboot = $save_no_reboot; + close(IN); +} + +sub check_patch_buildlog { my ($patch) = @_; my @files = `git show $patch | diffstat -l`; + foreach my $file (@files) { + chomp $file; + } + open(IN, "git show $patch |") or dodie "failed to show $patch"; while (<IN>) { @@ -1762,12 +2168,16 @@ sub make_oldconfig { apply_min_config; } - if (!run_command "$make oldnoconfig") { - # Perhaps oldnoconfig doesn't exist in this version of the kernel - # try a yes '' | oldconfig - doprint "oldnoconfig failed, trying yes '' | make oldconfig\n"; - run_command "yes '' | $make oldconfig" or - dodie "failed make config oldconfig"; + if (!run_command "$make olddefconfig") { + # Perhaps olddefconfig doesn't exist in this version of the kernel + # try oldnoconfig + doprint "olddefconfig failed, trying make oldnoconfig\n"; + if (!run_command "$make oldnoconfig") { + doprint "oldnoconfig failed, trying yes '' | make oldconfig\n"; + # try a yes '' | oldconfig + run_command "yes '' | $make oldconfig" or + dodie "failed make config oldconfig"; + } } } @@ -1775,6 +2185,7 @@ sub make_oldconfig { sub load_force_config { my ($config) = @_; + doprint "Loading force configs from $config\n"; open(IN, $config) or dodie "failed to read $config"; while (<IN>) { @@ -1797,6 +2208,9 @@ sub build { my $save_no_reboot = $no_reboot; $no_reboot = 1; + # Calculate a new version from here. + $have_version = 0; + if (defined($pre_build)) { my $ret = run_command $pre_build; if (!$ret && defined($pre_build_die) && @@ -1814,7 +2228,7 @@ sub build { # old config can ask questions if ($type eq "oldconfig") { - $type = "oldnoconfig"; + $type = "olddefconfig"; # allow for empty configs run_command "touch $output_config"; @@ -1844,7 +2258,7 @@ sub build { load_force_config($minconfig); } - if ($type ne "oldnoconfig") { + if ($type ne "olddefconfig") { run_command "$make $type" or dodie "failed make config"; } @@ -1856,6 +2270,9 @@ sub build { undef $redirect; if (defined($post_build)) { + # Because a post build may change the kernel version + # do it now. + get_version; my $ret = run_command $post_build; if (!$ret && defined($post_build_die) && $post_build_die) { @@ -1892,6 +2309,10 @@ sub halt { sub success { my ($i) = @_; + if (defined($post_test)) { + run_command $post_test; + } + $successes++; my $name = ""; @@ -1958,6 +2379,7 @@ sub do_run_test { my $line; my $full_line; my $bug = 0; + my $bug_ignored = 0; wait_for_monitor 1; @@ -1982,7 +2404,11 @@ sub do_run_test { doprint $line; if ($full_line =~ /call trace:/i) { - $bug = 1; + if ($ignore_errors) { + $bug_ignored = 1; + } else { + $bug = 1; + } } if ($full_line =~ /Kernel panic -/) { @@ -1995,6 +2421,10 @@ sub do_run_test { } } while (!$child_done && !$bug); + if (!$bug && $bug_ignored) { + doprint "WARNING: Call Trace detected but ignored due to IGNORE_ERRORS=1\n"; + } + if ($bug) { my $failure_start = time; my $now; @@ -2157,14 +2587,31 @@ sub run_bisect { $buildtype = "useconfig:$minconfig"; } - my $ret = run_bisect_test $type, $buildtype; + # If the user sets bisect_tries to less than 1, then no tries + # is a success. + my $ret = 1; - if ($bisect_manual) { + # Still let the user manually decide that though. + if ($bisect_tries < 1 && $bisect_manual) { $ret = answer_bisect; } + for (my $i = 0; $i < $bisect_tries; $i++) { + if ($bisect_tries > 1) { + my $t = $i + 1; + doprint("Running bisect trial $t of $bisect_tries:\n"); + } + $ret = run_bisect_test $type, $buildtype; + + if ($bisect_manual) { + $ret = answer_bisect; + } + + last if (!$ret); + } + # Are we looking for where it worked, not failed? - if ($reverse_bisect) { + if ($reverse_bisect && $ret >= 0) { $ret = !$ret; } @@ -2317,9 +2764,23 @@ sub bisect { success $i; } +# config_ignore holds the configs that were set (or unset) for +# a good config and we will ignore these configs for the rest +# of a config bisect. These configs stay as they were. my %config_ignore; + +# config_set holds what all configs were set as. my %config_set; +# config_off holds the set of configs that the bad config had disabled. +# We need to record them and set them in the .config when running +# olddefconfig, because olddefconfig keeps the defaults. +my %config_off; + +# config_off_tmp holds a set of configs to turn off for now +my @config_off_tmp; + +# config_list is the set of configs that are being tested my %config_list; my %null_config; @@ -2398,12 +2859,21 @@ sub create_config { } } + # turn off configs to keep off + foreach my $config (keys %config_off) { + print OUT "# $config is not set\n"; + } + + # turn off configs that should be off for now + foreach my $config (@config_off_tmp) { + print OUT "# $config is not set\n"; + } + foreach my $config (keys %config_ignore) { print OUT "$config_ignore{$config}\n"; } close(OUT); -# exit; make_oldconfig; } @@ -2480,6 +2950,13 @@ sub run_config_bisect { do { my @tophalf = @start_list[0 .. $half]; + # keep the bottom half off + if ($half < $#start_list) { + @config_off_tmp = @start_list[$half + 1 .. $#start_list]; + } else { + @config_off_tmp = (); + } + create_config @tophalf; read_current_config \%current_config; @@ -2496,7 +2973,11 @@ sub run_config_bisect { if (!$found) { # try the other half doprint "Top half produced no set configs, trying bottom half\n"; + + # keep the top half off + @config_off_tmp = @tophalf; @tophalf = @start_list[$half + 1 .. $#start_list]; + create_config @tophalf; read_current_config \%current_config; foreach my $config (@tophalf) { @@ -2601,7 +3082,7 @@ sub config_bisect { # read directly what we want to check my %config_check; open (IN, $output_config) - or dodie "faied to open $output_config"; + or dodie "failed to open $output_config"; while (<IN>) { if (/^((CONFIG\S*)=.*)/) { @@ -2634,6 +3115,10 @@ sub config_bisect { $added_configs{$2} = $1; $config_list{$2} = $1; } + } elsif (/^# ((CONFIG\S*).*)/) { + # Keep these configs disabled + $config_set{$2} = $1; + $config_off{$2} = $1; } } close(IN); @@ -2656,6 +3141,8 @@ sub config_bisect { my %config_test; my $once = 0; + @config_off_tmp = (); + # Sometimes kconfig does weird things. We must make sure # that the config we autocreate has everything we need # to test, otherwise we may miss testing configs, or @@ -2674,6 +3161,18 @@ sub config_bisect { } } my $ret; + + if (defined($config_bisect_check) && $config_bisect_check) { + doprint " Checking to make sure bad config with min config fails\n"; + create_config keys %config_list; + $ret = run_config_bisect_test $config_bisect_type; + if ($ret) { + doprint " FAILED! Bad config with min config boots fine\n"; + return -1; + } + doprint " Bad config with min config fails as expected\n"; + } + do { $ret = run_config_bisect; } while (!$ret); @@ -2767,11 +3266,13 @@ sub patchcheck { build "oldconfig" or return 0; } - - if (!defined($ignored_warnings{$sha1})) { - check_buildlog $sha1 or return 0; + # No need to do per patch checking if warnings file exists + if (!defined($warnings_file) && !defined($ignored_warnings{$sha1})) { + check_patch_buildlog $sha1 or return 0; } + check_buildlog or return 0; + next if ($type eq "build"); my $failed = 0; @@ -3066,7 +3567,7 @@ sub test_this_config { } # Remove this config from the list of configs - # do a make oldnoconfig and then read the resulting + # do a make olddefconfig and then read the resulting # .config to make sure it is missing the config that # we had before my %configs = %min_configs; @@ -3088,6 +3589,12 @@ sub test_this_config { sub make_min_config { my ($i) = @_; + my $type = $minconfig_type; + if ($type ne "boot" && $type ne "test") { + fail "Invalid MIN_CONFIG_TYPE '$minconfig_type'\n" . + " make_min_config works only with 'boot' and 'test'\n" and return; + } + if (!defined($output_minconfig)) { fail "OUTPUT_MIN_CONFIG not defined" and return; } @@ -3097,8 +3604,15 @@ sub make_min_config { # that instead. if (-f $output_minconfig && !$start_minconfig_defined) { print "$output_minconfig exists\n"; - if (read_yn " Use it as minconfig?") { + if (!defined($use_output_minconfig)) { + if (read_yn " Use it as minconfig?") { + $start_minconfig = $output_minconfig; + } + } elsif ($use_output_minconfig > 0) { + doprint "Using $output_minconfig as MIN_CONFIG\n"; $start_minconfig = $output_minconfig; + } else { + doprint "Set to still use MIN_CONFIG as starting point\n"; } } @@ -3247,6 +3761,11 @@ sub make_min_config { build "oldconfig" or $failed = 1; if (!$failed) { start_monitor_and_boot or $failed = 1; + + if ($type eq "test" && !$failed) { + do_run_test or $failed = 1; + } + end_monitor; } @@ -3311,6 +3830,39 @@ sub make_min_config { return 1; } +sub make_warnings_file { + my ($i) = @_; + + if (!defined($warnings_file)) { + dodie "Must define WARNINGS_FILE for make_warnings_file test"; + } + + if ($build_type eq "nobuild") { + dodie "BUILD_TYPE can not be 'nobuild' for make_warnings_file test"; + } + + build $build_type or dodie "Failed to build"; + + open(OUT, ">$warnings_file") or dodie "Can't create $warnings_file"; + + open(IN, $buildlog) or dodie "Can't open $buildlog"; + while (<IN>) { + + # Some compilers use UTF-8 extended for quotes + # for distcc heterogeneous systems, this causes issues + s/$utf8_quote/'/g; + + if (/$check_build_re/) { + print OUT; + } + } + close(IN); + + close(OUT); + + success $i; +} + $#ARGV < 1 or die "ktest.pl version: $VERSION\n usage: ktest.pl config-file\n"; if ($#ARGV == 0) { @@ -3356,7 +3908,7 @@ EOF read_config $ktest_config; if (defined($opt{"LOG_FILE"})) { - $opt{"LOG_FILE"} = eval_option($opt{"LOG_FILE"}, -1); + $opt{"LOG_FILE"} = eval_option("LOG_FILE", $opt{"LOG_FILE"}, -1); } # Append any configs entered in manually to the config file. @@ -3433,7 +3985,7 @@ sub set_test_option { my $option = __set_test_option($name, $i); return $option if (!defined($option)); - return eval_option($option, $i); + return eval_option($name, $option, $i); } # First we need to do is the builds @@ -3441,11 +3993,28 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) { # Do not reboot on failing test options $no_reboot = 1; + $reboot_success = 0; + + $have_version = 0; $iteration = $i; + undef %force_config; + my $makecmd = set_test_option("MAKE_CMD", $i); + $outputdir = set_test_option("OUTPUT_DIR", $i); + $builddir = set_test_option("BUILD_DIR", $i); + + chdir $builddir || die "can't change directory to $builddir"; + + if (!-d $outputdir) { + mkpath($outputdir) or + die "can't create $outputdir"; + } + + $make = "$makecmd O=$outputdir"; + # Load all the options into their mapped variable names foreach my $opt (keys %option_map) { ${$option_map{$opt}} = set_test_option($opt, $i); @@ -3453,18 +4022,26 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) { $start_minconfig_defined = 1; + # The first test may override the PRE_KTEST option + if (defined($pre_ktest) && $i == 1) { + doprint "\n"; + run_command $pre_ktest; + } + + # Any test can override the POST_KTEST option + # The last test takes precedence. + if (defined($post_ktest)) { + $final_post_ktest = $post_ktest; + } + if (!defined($start_minconfig)) { $start_minconfig_defined = 0; $start_minconfig = $minconfig; } - chdir $builddir || die "can't change directory to $builddir"; - - foreach my $dir ($tmpdir, $outputdir) { - if (!-d $dir) { - mkpath($dir) or - die "can't create $dir"; - } + if (!-d $tmpdir) { + mkpath($tmpdir) or + die "can't create $tmpdir"; } $ENV{"SSH_USER"} = $ssh_user; @@ -3473,13 +4050,17 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) { $buildlog = "$tmpdir/buildlog-$machine"; $testlog = "$tmpdir/testlog-$machine"; $dmesg = "$tmpdir/dmesg-$machine"; - $make = "$makecmd O=$outputdir"; $output_config = "$outputdir/.config"; if (!$buildonly) { $target = "$ssh_user\@$machine"; if ($reboot_type eq "grub") { dodie "GRUB_MENU not defined" if (!defined($grub_menu)); + } elsif ($reboot_type eq "grub2") { + dodie "GRUB_MENU not defined" if (!defined($grub_menu)); + dodie "GRUB_FILE not defined" if (!defined($grub_file)); + } elsif ($reboot_type eq "syslinux") { + dodie "SYSLINUX_LABEL not defined" if (!defined($syslinux_label)); } } @@ -3490,9 +4071,9 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) { $run_type = $bisect_type; } elsif ($test_type eq "config_bisect") { $run_type = $config_bisect_type; - } - - if ($test_type eq "make_min_config") { + } elsif ($test_type eq "make_min_config") { + $run_type = ""; + } elsif ($test_type eq "make_warnings_file") { $run_type = ""; } @@ -3507,6 +4088,10 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) { doprint "\n\n"; doprint "RUNNING TEST $i of $opt{NUM_TESTS} with option $test_type $run_type$installme\n\n"; + if (defined($pre_test)) { + run_command $pre_test; + } + unlink $dmesg; unlink $buildlog; unlink $testlog; @@ -3528,6 +4113,10 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) { $no_reboot = 0; + # A test may opt to not reboot the box + if ($reboot_on_success) { + $reboot_success = 1; + } if ($test_type eq "bisect") { bisect $i; @@ -3541,10 +4130,15 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) { } elsif ($test_type eq "make_min_config") { make_min_config $i; next; + } elsif ($test_type eq "make_warnings_file") { + $no_reboot = 1; + make_warnings_file $i; + next; } if ($build_type ne "nobuild") { build $build_type or next; + check_buildlog or next; } if ($test_type eq "install") { @@ -3568,12 +4162,20 @@ for (my $i = 1; $i <= $opt{"NUM_TESTS"}; $i++) { success $i; } +if (defined($final_post_ktest)) { + run_command $final_post_ktest; +} + if ($opt{"POWEROFF_ON_SUCCESS"}) { halt; -} elsif ($opt{"REBOOT_ON_SUCCESS"} && !do_not_reboot) { +} elsif ($opt{"REBOOT_ON_SUCCESS"} && !do_not_reboot && $reboot_success) { reboot_to_good; +} elsif (defined($switch_to_good)) { + # still need to get to the good kernel + run_command $switch_to_good; } + doprint "\n $successes of $opt{NUM_TESTS} tests were successful\n\n"; exit 0; diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf index 5ea04c6a71b..172eec4517f 100644 --- a/tools/testing/ktest/sample.conf +++ b/tools/testing/ktest/sample.conf @@ -158,7 +158,7 @@ # # TEST_START IF (DEFINED ALL_TESTS || ${MYTEST} == boottest) && ${MACHINE} == gandalf # -# Notice the use of paranthesis. Without any paranthesis the above would be +# Notice the use of parentheses. Without any parentheses the above would be # processed the same as: # # TEST_START IF DEFINED ALL_TESTS || (${MYTEST} == boottest && ${MACHINE} == gandalf) @@ -328,12 +328,29 @@ # For a virtual machine with guest name "Guest". #CONSOLE = virsh console Guest +# Signal to send to kill console. +# ktest.pl will create a child process to monitor the console. +# When the console is finished, ktest will kill the child process +# with this signal. +# (default INT) +#CLOSE_CONSOLE_SIGNAL = HUP + # Required version ending to differentiate the test # from other linux builds on the system. #LOCALVERSION = -test +# For REBOOT_TYPE = grub2, you must specify where the grub.cfg +# file is. This is the file that is searched to find the menu +# option to boot to with GRUB_REBOOT +#GRUB_FILE = /boot/grub2/grub.cfg + +# The tool for REBOOT_TYPE = grub2 to set the next reboot kernel +# to boot into (one shot mode). +# (default grub2_reboot) +#GRUB_REBOOT = grub2_reboot + # The grub title name for the test kernel to boot -# (Only mandatory if REBOOT_TYPE = grub) +# (Only mandatory if REBOOT_TYPE = grub or grub2) # # Note, ktest.pl will not update the grub menu.lst, you need to # manually add an option for the test. ktest.pl will search @@ -343,8 +360,33 @@ # For example, if in the /boot/grub/menu.lst the test kernel title has: # title Test Kernel # kernel vmlinuz-test +# +# For grub2, a search of top level "menuentry"s are done. No +# submenu is searched. The menu is found by searching for the +# contents of GRUB_MENU in the line that starts with "menuentry". +# You may want to include the quotes around the option. For example: +# for: menuentry 'Test Kernel' +# do a: GRUB_MENU = 'Test Kernel' +# For customizing, add your entry in /etc/grub.d/40_custom. +# #GRUB_MENU = Test Kernel +# For REBOOT_TYPE = syslinux, the name of the syslinux executable +# (on the target) to use to set up the next reboot to boot the +# test kernel. +# (default extlinux) +#SYSLINUX = syslinux + +# For REBOOT_TYPE = syslinux, the path that is passed to to the +# syslinux command where syslinux is installed. +# (default /boot/extlinux) +#SYSLINUX_PATH = /boot/syslinux + +# For REBOOT_TYPE = syslinux, the syslinux label that references the +# test kernel in the syslinux config file. +# (default undefined) +#SYSLINUX_LABEL = "test-kernel" + # A script to reboot the target into the test kernel # This and SWITCH_TO_TEST are about the same, except # SWITCH_TO_TEST is run even for REBOOT_TYPE = grub. @@ -376,6 +418,24 @@ # DEFAULTS # DEFAULTS SKIP +# If you want to execute some command before the first test runs +# you can set this option. Note, it can be set as a default option +# or an option in the first test case. All other test cases will +# ignore it. If both the default and first test have this option +# set, then the first test will take precedence. +# +# default (undefined) +#PRE_KTEST = ${SSH} ~/set_up_test + +# If you want to execute some command after all the tests have +# completed, you can set this option. Note, it can be set as a +# default or any test case can override it. If multiple test cases +# set this option, then the last test case that set it will take +# precedence +# +# default (undefined) +#POST_KTEST = ${SSH} ~/dismantle_test + # The default test type (default test) # The test types may be: # build - only build the kernel, do nothing else @@ -408,6 +468,14 @@ # (default "") #BUILD_OPTIONS = -j20 +# If you need to do some special handling before installing +# you can add a script with this option. +# The environment variable KERNEL_VERSION will be set to the +# kernel version that is used. +# +# default (undefined) +#PRE_INSTALL = ssh user@target rm -rf '/lib/modules/*-test*' + # If you need an initrd, you can add a script or code here to install # it. The environment variable KERNEL_VERSION will be set to the # kernel version that is used. Remember to add the initrd line @@ -426,6 +494,18 @@ # (default 0) #NO_INSTALL = 1 +# If there is a command that you want to run before the individual test +# case executes, then you can set this option +# +# default (undefined) +#PRE_TEST = ${SSH} reboot_to_special_kernel + +# If there is a command you want to run after the individual test case +# completes, then you can set this option. +# +# default (undefined) +#POST_TEST = cd ${BUILD_DIR}; git reset --hard + # If there is a script that you require to run before the build is done # you can specify it with PRE_BUILD. # @@ -459,7 +539,7 @@ #POST_BUILD_DIE = 1 # Way to reboot the box to the test kernel. -# Only valid options so far are "grub" and "script" +# Only valid options so far are "grub", "grub2", "syslinux" and "script" # (default grub) # If you specify grub, it will assume grub version 1 # and will search in /boot/grub/menu.lst for the title $GRUB_MENU @@ -467,6 +547,13 @@ # your setup, then specify "script" and have a command or script # specified in REBOOT_SCRIPT to boot to the target. # +# For REBOOT_TYPE = grub2, you must define both GRUB_MENU and +# GRUB_FILE. +# +# For REBOOT_TYPE = syslinux, you must define SYSLINUX_LABEL, and +# perhaps modify SYSLINUX (default extlinux) and SYSLINUX_PATH +# (default /boot/extlinux) +# # The entry in /boot/grub/menu.lst must be entered in manually. # The test will not modify that file. #REBOOT_TYPE = grub @@ -657,6 +744,14 @@ # (default 60) #BISECT_SLEEP_TIME = 60 +# The max wait time (in seconds) for waiting for the console to finish. +# If for some reason, the console is outputting content without +# ever finishing, this will cause ktest to get stuck. This +# option is the max time ktest will wait for the monitor (console) +# to settle down before continuing. +# (default 1800) +#MAX_MONITOR_WAIT + # The time in between patch checks to sleep (in seconds) # (default 60) #PATCHCHECK_SLEEP_TIME = 60 @@ -705,15 +800,37 @@ # Example for a virtual guest call "Guest". #POWER_OFF = virsh destroy Guest +# To have the build fail on "new" warnings, create a file that +# contains a list of all known warnings (they must match exactly +# to the line with 'warning:', 'error:' or 'Error:'. If the option +# WARNINGS_FILE is set, then that file will be read, and if the +# build detects a warning, it will examine this file and if the +# warning does not exist in it, it will fail the build. +# +# Note, if this option is defined to a file that does not exist +# then any warning will fail the build. +# (see make_warnings_file below) +# +# (optional, default undefined) +#WARNINGS_FILE = ${OUTPUT_DIR}/warnings_file + # The way to execute a command on the target # (default ssh $SSH_USER@$MACHINE $SSH_COMMAND";) # The variables SSH_USER, MACHINE and SSH_COMMAND are defined #SSH_EXEC = ssh $SSH_USER@$MACHINE $SSH_COMMAND"; -# The way to copy a file to the target +# The way to copy a file to the target (install and modules) # (default scp $SRC_FILE $SSH_USER@$MACHINE:$DST_FILE) -# The variables SSH_USER, MACHINE, SRC_FILE and DST_FILE are defined. -#SCP_TO_TARGET = scp $SRC_FILE $SSH_USER@$MACHINE:$DST_FILE +# The variables SSH_USER, MACHINE are defined by the config +# SRC_FILE and DST_FILE are ktest internal variables and +# should only have '$' and not the '${}' notation. +# (default scp $SRC_FILE ${SSH_USER}@${MACHINE}:$DST_FILE) +#SCP_TO_TARGET = echo skip scp for $SRC_FILE $DST_FILE + +# If install needs to be different than modules, then this +# option will override the SCP_TO_TARGET for installation. +# (default ${SCP_TO_TARGET} ) +#SCP_TO_TARGET_INSTALL = scp $SRC_FILE tftp@tftpserver:$DST_FILE # The nice way to reboot the target # (default ssh $SSH_USER@$MACHINE reboot) @@ -911,6 +1028,20 @@ # BISECT_BAD with BISECT_CHECK = good or # BISECT_CHECK = bad, respectively. # +# BISECT_TRIES = 5 (optional, default 1) +# +# For those cases that it takes several tries to hit a bug, +# the BISECT_TRIES is useful. It is the number of times the +# test is ran before it says the kernel is good. The first failure +# will stop trying and mark the current SHA1 as bad. +# +# Note, as with all race bugs, there's no guarantee that if +# it succeeds, it is really a good bisect. But it helps in case +# the bug is some what reliable. +# +# You can set BISECT_TRIES to zero, and all tests will be considered +# good, unless you also set BISECT_MANUAL. +# # BISECT_RET_GOOD = 0 (optional, default undefined) # # In case the specificed test returns something other than just @@ -1031,6 +1162,12 @@ # can specify it with CONFIG_BISECT_GOOD. Otherwise # the MIN_CONFIG is the base. # +# CONFIG_BISECT_CHECK (optional) +# Set this to 1 if you want to confirm that the config ktest +# generates (the bad config with the min config) is still bad. +# It may be that the min config fixes what broke the bad config +# and the test will not return a result. +# # Example: # TEST_START # TEST_TYPE = config_bisect @@ -1097,10 +1234,56 @@ # and will not be tested again in later runs. # (optional) # +# MIN_CONFIG_TYPE can be either 'boot' or 'test'. With 'boot' it will +# test if the created config can just boot the machine. If this is +# set to 'test', then the TEST option must be defined and the created +# config will not only boot the target, but also make sure that the +# config lets the test succeed. This is useful to make sure the final +# config that is generated allows network activity (ssh). +# (optional) +# +# USE_OUTPUT_MIN_CONFIG set this to 1 if you do not want to be prompted +# about using the OUTPUT_MIN_CONFIG as the MIN_CONFIG as the starting +# point. Set it to 0 if you want to always just use the given MIN_CONFIG. +# If it is not defined, it will prompt you to pick which config +# to start with (MIN_CONFIG or OUTPUT_MIN_CONFIG). +# # Example: # # TEST_TYPE = make_min_config # OUTPUT_MIN_CONFIG = /path/to/config-new-min # START_MIN_CONFIG = /path/to/config-min # IGNORE_CONFIG = /path/to/config-tested +# MIN_CONFIG_TYPE = test +# TEST = ssh ${USER}@${MACHINE} echo hi +# +# +# +# +# For TEST_TYPE = make_warnings_file +# +# If you want the build to fail when a new warning is discovered +# you set the WARNINGS_FILE to point to a file of known warnings. +# +# The test "make_warnings_file" will let you create a new warnings +# file before you run other tests, like patchcheck. +# +# What this test does is to run just a build, you still need to +# specify BUILD_TYPE to tell the test what type of config to use. +# A BUILD_TYPE of nobuild will fail this test. +# +# The test will do the build and scan for all warnings. Any warning +# it discovers will be saved in the WARNINGS_FILE (required) option. +# +# It is recommended (but not necessary) to make sure BUILD_NOCLEAN is +# off, so that a full build is done (make mrproper is performed). +# That way, all warnings will be captured. +# +# Example: +# +# TEST_TYPE = make_warnings_file +# WARNINGS_FILE = ${OUTPUT_DIR} +# BUILD_TYPE = useconfig:oldconfig +# CHECKOUT = v3.8 +# BUILD_NOCLEAN = 0 # |