Firefox and Thunderbird testing

Bash

bash 3.x is the shell of choice in the framework since it provides great flexibility and is available across all platforms. Mac OS X makes life slightly more difficult since current releases have default installations of bash 2.x. In order to not interfere with the existing operating system on Mac OS X, bash is to be built and installed into /usr/local/bin. On the other platforms where bash 3.x is already the default, a link from /bin/bash to /usr/local/bin/bash is to be created.

Unless there is a reason to do otherwise, all scripts will use

#!/usr/local/bin/bash -e

to invoke the shell.

Script argument conventions

The framework uses a set of conventions to hopefully make life simpler both today and tomorrow.

Command line arguments

Scripts which require arguments will use bash's options processing as in the following example:

#
# options processing
#
# options which require a value are listed in $options with a trailing :
# options which do not take a value are listed at the end of $options
#
options="a:b:cd"
function usage()
{
    cat<<EOF
usage: 
$SCRIPT -a avalue [-b bvalue] [-c]

variable            description
===============     ===========================================================
-a avalue           required. sets the variable avalue to the specified value
-b bvalue           optional. sets the variable bvalue to the specified value
-c                  optional. sets the variable cvalue to ...
EOF
    exit 2
}

unset avalue bvalue cvalue

while getopts $options optname ; 
  do 
  case $optname in
      a) avalue="$OPTARG";;
      b) bvalue="$OPTARG";;
      c) cvalue="cvalue";;
  esac
done

if [[ -z "$avalue" ]]; then
    usage
fi

If a script deals with the installation or invocation of Firefox or Thunderbird it has the following arguments:

-p product

In order to insulate scripts from differences in command line syntax or behavior between various products and to more easily support additional products in the future all scripts that deal with Firefox or Thunderbird have a required argument -p product where product is currently firefox or thunderbird.

-b branch

In order to insulate scripts from differences in command line syntax or behavior between various branches and to more easily support new branches in the future all scripts that deal with Firefox or Thunderbird have a required argument -b branch where branch is currently 1.8.0, 1.8.1 or 1.9.0.

-x executablepath

Since the actual path to the executable file in an installation of Firefox or Thunderbird depends on the type of installation as well as the platform, paths to executables are specified as the path to a directory that contains somewhere underneath its hierarchy an executable file with a name determinable from the product. The bash function get_executable() is to be used to locate the executable file from the executable path.

-d datafiles

Many command line arguments associated tests and builds have associated with them a standard bash variable name. Combinations of these arguments and their values can be saved in special data files as name=value pairs and passed to scripts via the -d datafiles argument.

Thus scripts can be called either by specifying all of the required arguments on the command line or by specifying them in a file of variable definitions. These datafiles allow the saving of combinations of command-line arguments for various scripts in a central place which hopefully helps improve reuse.

create-directory.sh

create-directory.sh is simple script which is meant to make the creation and deletion of directories safer than simple calls to rm -fR dir.

create-directory.sh -d directory [-n]

-d directory

directory specifies the directory to be created.

create-directory.sh will actually try to create the directoy using mkdir -p $directory and then change the current working directory to that directory. This allows the script to handle absolute and relative paths in a unified way. The -p argument will create any parent directories if they are needed.

create-directory.sh will fail with an error if directory is the root directory /, a subdirectory of root, or a subdirectory of a subdirectory of root with the exception of /tmp. This means that /work/test/ is prohibited, but /tmp/test/ is not.

This should protect important directories such as /, /usr/, /usr/bin/, /home/user/ even on operating systems that do not protect such directories from deletion. You should never run this script as root!

If all such checks pass, then create-directory.sh will delete the directory and recreate it, thus ensuring it is initially empty and owned by the user.

-n

If the optional argument -n is not specifed, then create-directory.sh will cause rm to prompt the user before deleting anything. This is primarily used during testing to make sure the script does not doing anything unexpected. Once you are sure that nothing unexpected will be deleted, you can specify the -n argument to stop the prompt.

download.sh

download.sh is a simple wrapper script which uses curl to download a file.

download.sh -u url -f filepath [-t timeout] [-c credentials] [-d datafiles]

-u url

url specifies the location of the file to download.

-f filepath

filepath specifies the location where to save the file on the local filesystem.

-t timeout

timeout specifies how long to wait in seconds before timing out the download.

-c credentials

credentials are the optional user:password to be used when authenticating.

-d datafiles

datafiles is a list of paths to files which contain lists of variable definitions of the form:

url=urlvalue
filepath=filepathvalue
timeout=timeoutvalue

library.sh

library.sh is a script which is intended to be sourceed into other scripts. It provides common functions and environment variables for other scripts in the framework.

functions

error()

error() is used by scripts to perform error handling.

error "message"

simply displays

error in script $SCRIPT: "message"

then exits with exit code 2, if the script is not an instance of bash itself. SCRIPT is an environment variable containing the name of the running script and is set by library.sh

get_executable()

get_executable() is used by scripts to locate the executable binary for the specified product. It is intended to hide the differences between products and operating systems. get_executable() will write to standard out the path to the executable and is usually used with backtics. The returned path to the executable must be checked to see if it is non-empty and that it points to a unique executable file.

executable=`get_executable $product $branch $executablepath`

if [[ -z "$executable" ]]; then
    error "get_executable $product $branch $executablepath returned empty path"
fi

if [[ ! -x "$executable" ]]; then 
    error "executable \"$executable\" is not executable"
fi

On Windows and Linux, get_executable looks for the executable file with same name as the product.

On Mac OS X for product firefox, get_executable will look for the unique executable in a subpath Contents/MacOS/firefox. On Mac OS X for product thunderbird, it will look for the unique executable in a subpath of Contents/MacOS/thunderbird-bin since there is no script thunderbird corresponding to the script firefox on Mac.

environment variables

TEST_DIR: TEST_DIR contains the path to the test framework. The framework currently expects to live in the directory /work/mozilla/mozilla.com/test.mozilla.com/www however you should be able to place it anywhere and set TEST_DIR. If TEST_DIR is not set, the framework sets it to /work/mozilla/mozilla.com/test.mozilla.com/www.
TEST_BIN: TEST_BIN contains the path to the test framework's bin directory. By default it is located at $TEST_DIR/bin however you should be able locate the scripts elsewhere.
TEST_HTTP: TEST_HTTP contains the name of the web server that contains the web-based tests in the framework. By default, the name is test.mozilla.com. Typically, this web server is setup as a virtual web server on the same machine as the hosted framework with the document directory the same as TEST_DIR.
TEST_STARTUP_TIMEOUT: TEST_STARTUP_TIMEOUT contains the expected maximum time in seconds for basic operations such as profile creation, time to start and load a page. The default is 30.
PATH: library.sh appends $TEST_BIN to the PATH variable to simplify the invocation of the other scripts in the $TEST_BIN directory.
STARTDIR: library.sh captures the current working directory in STARTDIR when the script, which sourced library.sh, was initially executed.
SCRIPTDIR: library.sh captures the directory where the script , which sourced library.sh, is located.
SCRIPT: library.sh captures the name of the script in SCRIPT, which sourced library.sh, without the leading directory path.
OSID: library.sh sets OSID to win32 for Windows, linux for Linux and mac for Mac OS X.
EXE_EXT: library.sh sets EXE_EXT to .exe on Windows and empty for Linux and Mac OS X.
TIMEFORMAT: TIMEFORMAT is a bash variable that controls the format of output of the time keyword applied to pipelines.
MOZ_NO_REMOTE: MOZ_NO_REMOTE is set to 1 to allow Gecko products to invoke multiple instances.
NO_EM_RESTART: NO_EM_RESTART is set to 1 to prevent the extension manager in Gecko 1.8.0 and later from restarting the process.
MOZ_BYPASS_PROFILE_AT_STARTUP: MOZ_BYPASS_PROFILE_AT_STARTUP ....
MOZ_GDB_SLEEP: By default on Linux and Mac OS X, Firefox and Thunderbird will sleep for 5 minutes after a crash to allow a developer time to debug the crash. MOZ_GDB_SLEEP is set to 10 to prevent crashes from delaying a set of automatic tests.
XPCOM_DEBUG_BREAK: XPCOM_DEBUG_BREAK is set to warn to prevent any dialog from being presented when an assertion is hit in debug builds.
MALLOC_CHECK: MALLOC_CHECK_ is set to 2 to tell glibc to abort the program immediately in case of a heap error. This is only effective on Linux and Mac OS X where gcc is used.
MACHINE: MACHINE is set to the output of uname -n to identify the machine where the test is being executed.

uninstall-build.sh

uninstall-build.sh is a bash script that encapsulates the steps required to uninstall Firefox and Thunderbird on the various branches 1.8.0, 1.8.1 and 1.9.0.

uninstall-build.sh -p product -b branch -x executablepath [-d datafiles]

-p product: product is either firefox for Firefox or thunderbird for Thunderbird.
-b branch: branch is either 1.8.0 for Firefox 1.5.0.x or Thunderbird 1.5.0.x, 1.8.1 for Firefox 2.0.0.x or Thunderbird 2.0.0.x, or 1.9.0 for Firefox 3 and Thunderbird 3.
-x executablepath: executablepath is the directory underneath where to find the executable for the build. Note that this directory will be deleted and then recreated using create-directory.sh so it should not contain anything you wish to keep.
-d datafiles: datafiles is a list of paths to files which contain lists of variable definitions.

install-build.sh

Once you have a build, you must install it. install-build.sh is a bash script that encapsulates the steps required to install Firefox and Thunderbird on the various branches 1.8.0, 1.8.1 and 1.9.0.

install-build.sh -p product -b branch -x executablepath -f filename [-d datafiles]

-p product

product is either firefox for Firefox or thunderbird for Thunderbird.

-b branch

branch is either 1.8.0 for Firefox 1.5.0.x or Thunderbird 1.5.0.x, 1.8.1 for Firefox 2.0.0.x or Thunderbird 2.0.0.x, or 1.9.0 for Firefox 3 and Thunderbird 3.

-x executablepath

executablepath is the directory underneath which to install the build. Note that this directory will be deleted and then recreated using create-directory.sh so it should not contain anything you wish to keep.

Note that once the installation has been completed, the actual path to the Firefox or Thunderbird executable file will depend on the platform and type of installation file that was used (zip or installer for Windows; tar.gz, or tar.bz2 for Linux, or dmg for Mac OS X). executablepath introduces another convention used in this framework of specifying the path to an executable not by using a path to the exact executable file, but specifying a path to a directory that contains as a descendant the executable file. This fuzziness in specifying the path to the executable simplifies the issue of specifying the location of the executable.

-f filename

filename is the path to the downloaded build. The script will use the command line utility file to determine the type of the file along with the platform to determine how to install the build.

-d datafiles

datafiles is a list of paths to files which contain lists of variable definitions.

create-profile.sh

Once you have installed (or otherwise obtained) a build, you must create a profile for it. create-profile.sh is a bash script that encapsulates the steps required to create profiles for Firefox and Thunderbird on the various branches 1.8.0, 1.8.1 and 1.9.0.

create-profile.sh -p product -b branch -x executablepath -D directory -N profilename [-L profiletemplate] [-U user] [-d datafiles]

-p product: product is either firefox for Firefox or thunderbird for Thunderbird.
-b branch: branch is either 1.8.0 for Firefox 1.5.0.x or Thunderbird 1.5.0.x, 1.8.1 for Firefox 2.0.0.x or Thunderbird 2.0.0.x, or 1.9.0 for Firefox 3 and Thunderbird 3.
-x executablepath: executablepath is the directory containing the executable file for the build.
-D directory: directory is the location where to create the profile.
-N profilename: profilename is the name of the newly created profile.
-L profiletemplate: profiletemplate is the path to a template profile that will be copied over the newly created profile This is useful in circumstances where you wish to test a specific profile such as a Thunderbird profile with specific email accounts, etc.
-U user: user is the path to a user.js preferences file to be copied into the newly create profile. This is useful for setting preferences. Typically, one of the user preference files in the framework's preferences file is used since contain the necessary special permissions required to access XPCOM components.
-d datafiles: datafiles is a list of paths to files which contain lists of variable definitions.

edit-talkback.sh

Nightly and release builds of Firefox and Thunderbird have the capability of sending a talkback report to talkback.mozilla.org whenever a crash occurs. It is helpful to be able to automatically set some of the talkback parameters whenever a test is run in order that any talkback incidents for a particular crash can be located. edit-talkback.sh is a bash script that encapsulates the steps required to initialize the talkback ini files for Firefox and Thunderbird on the various branches 1.8.0, 1.8.1 and 1.9.0. It uses template talkback ini files from the talkback directory in the framework.

Note: at some point talkback will be replaced for the 1.9.0 branch.

edit-talkback.sh -p product -b branch -x executablepath -i talkbackid [-d datafiles]

-p product: product is either firefox for Firefox or thunderbird for Thunderbird.
-b branch: branch is either 1.8.0 for Firefox 1.5.0.x or Thunderbird 1.5.0.x, 1.8.1 for Firefox 2.0.0.x or Thunderbird 2.0.0.x, or 1.9.0 for Firefox 3 and Thunderbird 3.
-x executablepath: executablepath is the directory containing the executable file for the build.
-i talkbackid: edit-talkback.sh sets talkback's URLEdit field to mozqa:talkbackid so that any crash associated with the particular test can be located.
-d datafiles: datafiles is a list of paths to files which contain lists of variable definitions.

install-extensions.sh

The ability to install extensions is important not only for testing the extensions themselves, but the framework uses Spider as an essential testing tool.

install-extensions.sh is a bash script that encapsulates the steps required to install extensions for Firefox and Thunderbird on the various branches 1.8.0, 1.8.1 and 1.9.0. Extensions are installed globally (as part of the build) and not as part of the profile.

Note that since the extension manager is prevented from restarting by the environment variable NO_EM_RESTART=1, it is necessary to manually restart the build in order for the installed extensions to be properly initialized. This initialization uses additional html (install-extensions-1.html, and install-extensions-2.html) files located in the TEST_BIN directory in the framework during the extension initialization. Note that these html files use the quit.js script to automatically close the browser. This require special permissions which are normally granted by copying one of the default set of user preferences from the framework's prefs directory.

install-extensions.sh -p product -b branch -x executablepath -N profilename -E extensions [-d datafiles]

-p product

product is either firefox for Firefox or thunderbird for Thunderbird.

-b branch

branch is either 1.8.0 for Firefox 1.5.0.x or Thunderbird 1.5.0.x, 1.8.1 for Firefox 2.0.0.x or Thunderbird 2.0.0.x, or 1.9.0 for Firefox 3 and Thunderbird 3.

-x executablepath

executablepath is the directory containing the executable file for the build.

-N profilename

profilename is the name of the profile to be used when installing the extensions.

-E extensions

extensions is the path to a directory containing extensions to be installed. The directory has the following subdirectories:

all/
linux/
mac/
win32/

Extensions in the all/ directory will always be installed while the extensions in the other directories will only be installed on their respective platforms.

Spider's spider.xpi is typically located in all/.

-d datafiles

datafiles is a list of paths to files which contain lists of variable definitions.

install-plugins.sh

The ability to install plugins is also important for testing.

install-plugins.sh is a bash script that encapsulates the steps required to install plugins for Firefox and Thunderbird on the various branches 1.8.0, 1.8.1 and 1.9.0.

install-plugins.sh -p product -b branch -x executablepath -N profilename -E extensions [-d datafiles]

-p product

product is either firefox for Firefox or thunderbird for Thunderbird.

-b branch

branch is either 1.8.0 for Firefox 1.5.0.x or Thunderbird 1.5.0.x, 1.8.1 for Firefox 2.0.0.x or Thunderbird 2.0.0.x, or 1.9.0 for Firefox 3 and Thunderbird 3.

-x executablepath

executablepath is the directory containing the executable file for the build.

-N profilename

profilename is the name of the profile to be used when installing the extensions.

-D directory

directory is the path to a directory containing the plugins to be installed.

It is unfortunate that not all plugins have a cross-platform xpi-based installer. Therefore the approach taken in the framework is to perform an install on each platform and to extract the files from the build's plugins/ and components/ directories and to place them in a directory with the following layout:

linux/
    components/
    plugins/
mac/
    components/
    plugins/
win32/
    components/
    plugins/

The plugins will be installed on their respective platforms by copying the contents of the associated directories into the build's components/ and plugins/ directories.

-d datafiles

datafiles is a list of paths to files which contain lists of variable definitions.

check-spider.sh

check-spider.sh is a bash script that checks that the Spider extension is able to be started by attempting to load the html page start-spider.html using the Spider userhook script userhook-checkspider.js in the framework's TEST_BIN directory.

check-spider.sh -p product -b branch -x executablepath -N profilename [-d datafiles]

-p product: product is either firefox for Firefox or thunderbird for Thunderbird.
-b branch: branch is either 1.8.0 for Firefox 1.5.0.x or Thunderbird 1.5.0.x, 1.8.1 for Firefox 2.0.0.x or Thunderbird 2.0.0.x, or 1.9.0 for Firefox 3 and Thunderbird 3.
-x executablepath: executablepath is the directory containing the executable file for the build.
-N profilename: profilename is the name of the profile to be used when installing the extensions.
-d datafiles: datafiles is a list of paths to files which contain lists of variable definitions.

test-setup.sh

test-setup.sh is a bash script that exercises most of the framework by preparing a build for testing.

test-setup.sh -p product -b branch
       [-u url [-f filepath] [-c credentials]] 
       [-B buildcommands -T buildtype] 
       [-x executablepath]
       [-N profilename [-D profiledirectory [-L profiletemplate 
         [-U userpreferences]]]]
       [-E extensiondir]
       [-d datafiles]

-p product: required. one of firefox thunderbird
-b branch: required. one of 1.8.0 1.8.1 1.9.0
-u url: optional. url where to download build
-f filepath: optional. location to save downloaded build or to find previously downloaded build. If not specified, the default will be the basename of the url saved to the /tmp directory. If there is no basename, then the filepath will be /tmp/\$product-\$branch-file.
-B buildcommands: optional. one or more of clean checkout build
-T buildtype: optional. one of opt debug
-x executablepath: optional. directory tree containing executable with same name as product. If the build is downloaded and executable path is not specified, it will be defaulted to /tmp/\$product-\$branch. For cvs builds it will be defaulted to the appropriate directory in /work/mozilla/builds/\$branch/mozilla/\$product-\$buildtype/
-N profilename: optional. profilename. profilename is required if profiledirectory or extensiondir are specified.
-D profiledirectory: optional. If profiledirectory is specified, a new profile will be created in the directory.
-L profiletemplate: optional. If a new profile is created, profiletemplate is the path to an existing profile which will be copied over the new profile.
-U userpreferences: optional. If a new profile is created, userpreferences is the path to a user.js file to be copied into the new profile. If userpreferences is not specified when a new profile is created, it is defaulted to /work/mozilla/mozilla.com/test.mozilla.com/www/prefs/test-user.js
-E extensiondir: optional. path to directory tree containing extensions to be installed.
-d datafiles: optional. one or more filenames of files containing environment variable definitions to be included.

Example

test-setup.sh can be used to demonstrate an example where a Firefox nightly trunk build is downloaded, installed, profile created, extensions installed and the Firefox extension Spider is run.

$ ./bin/test-setup.sh -p firefox -b 1.9.0 \
   -u http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-trunk/firefox-3.0a3pre.en-US.linux-i686.tar.bz2 \
   -f /tmp/firefox-3.bz2 \
   -x /tmp/firefox-1.9.0 \
   -N firefox-1.9.0-profile \ 
   -D /tmp/firefox-1.9.0-profile \
   -E /work/mozilla/mozilla.com/test.mozilla.com/www/xpi \
    > test-setup.log 2>&1

The log file test-setup.log shows the result.