Month: September 2009

Browser JavaScript Reftests

The Bad Old Days

The JavaScript Tests have been the red headed stepchild of mozilla testing for some time. At least some of the reasons are:

  • The tests date to pre-mozilla days and do not follow the style of any of the other test suites.

  • The tests were originally designed to be run only in the JavaScript shell and had the ability to be run in the browser tacked on to them like a bad toupe

  • Not all of the tests pass and keeping track of the known failures has been a time consuming and error prone task.

  • Not having a good method of tracking failures, excluding long running tests or excluding tests which crashed the browser resulted in unacceptable run times for developers thus leaving a gap in the testing developers could perform prior to checking in changes to the JavaScript engine.

  • The automation used to run the tests in the shell and browser was not appropriate for running on the tinderboxes. As a result, the JavaScript engine did not benefit from immediate, cross-platform testing for each checkin but instead had to wait on me to detect regressions and file bugs.

  • The tests were not specialized for each branch but were instead maintained identically across branches. As the JavaScript engine has evolved, the differences between passing behavior on the branches has been problematic.

  • A single person was responsible for adding new tests causing the addition of new tests to increasingly lag behind new patches.

All of that has come to and end, because…

The Good Times have arrived!

(at least for the browser on tracemonkey)

David Baron’s reftest framework provides the solution to many of the problems of the Bad Old Days. It provides:

  • Familiar test invocation.

  • Relatively quick run times compared to the old Sisyphus-based browser JavaScript tests.

  • The reftest style manifests provide an excellent means of tracking obsolete tests, test failures, tests which take too long to run or tests which crash

  • Easy (hopefully) integration into the automated tests running on Tinderbox by leveraging the same approaches used to run the layout reftests

Currently, only the browser jsreftests are implemented. The same approach is planned to be implemented for the shell tests.

Running Browser JavaScript reftests

Running the jsreftests in the browser is familiar to anyone who has run the layout reftests or crash tests. Simply invoke make in the object directory with the target jstestbrowser.


$ make -C $(OBJDIR) jstestbrowser

This executes the following command. 


$ python $(OBJDIR)/_tests/reftest/runreftest.py \
       --extra-profile-file=js/src/tests/user.js js/src/tests/jstests.list

The user.js file is added to the run time profile to enable UniveralXPConnect and other privileges required for running the JavaScript tests. You can customize user.js to set other preferences you wish.

You can see example logs here:

No longer will you have to perform base-line runs to determine the known failures prior to testing your patches. If the jsreftests run to completion without crashing and without generating any UNEXPECTED results, your patch is good to go.

jsreftest manifests

jstests.listis the reftest manifest file used to tell the test runner which tests to execute. jstests.list is to be used for both the JavaScript shell tests and the JavaScript browser tests. url-prefix, a new feature for reftest manifests, is used to provide a single manifest file for both the shell and browser tests.

The manifest files are organized into a 3-level hierarchy.

The top level manifest files live in js/src/tests/. They incorporate the individual suites through the include manifest keyword. Whenever a new suite is added, the top level manifest files must be updated to include it. New suites must also be listed in js/src/tests/Makefile.in in order to be packaged with the other tests.

The second level manifest files are contained in the suite directories. They also include the sub-suites through the use of the include manifest keyword. Whenever a new sub-suite is added to a suite, the suite’s manifest files must be updated to include it.

The third and final level manifest files are contained in the sub-suite directories. These manifest files list each test using the script manifest keyword. This keyword tells the test runner to obtain the actual test results from the array returned by the getTestCases() function in the testcase. In order for the test results to be properly recorded in the test case results array, you must use one of the standard reporting functions, such as reportCompare(), reportMatch(), etc., to record the result of the test. If you do not use reportCompare() or one of its cousins, your test will fail with the message that No test results reported.

The first line of the manifest file contains a “url-prefix” which tells the reftest extension to prefix each test file with the url-prefix value before attempting to load it. The shell test driver will ignore the url-prefix, thus allowing a single manifest file to be used by both the shell and browser. The remainder of the manifest lists each test file preceded by the script keyword.

For example, js/src/tests/ecma_5/Date/jstests.list: 

url-prefix ../../jsreftest.html?test=ecma_5/Date/
script 15.9.4.2.js

The truly great thing about the reftest manifest files are their ability to give tests a failure-type. Each test can be marked as either:

fails

Tests marked with fails will be executed and will be marked as TEST-KNOWN-FAIL if they fail. However if they pass, they will be marked as TEST-UNEXPECTED-PASS.

random

Tests marked with random will be executed and will be marked as TEST-KNOWN-FAIL(EXPECTED RANDOM) if they fail or TEST-PASS(EXPECTED RANDOM) if they pass. This is useful for tests which may not reliably report a test failure but which you wish to continue to execute in the event they may find a regression through a crash or assertion.

random results are also used in the event that a test that returns multiple test results is marked with the failure-type fails. In that case, failures are marked with TEST-KNOWN-FAIL while successes are marked with TEST-PASS(EXPECTED RANDOM). This is necessary since it is not possible to mark the individual test case results in a multi-result test as either passing or failing.

skip

Tests marked with skip will not be executed at all. skip can be used for obsolete tests, for tests which are known to crash the browser, tests which do not terminate or tests which take too much time.

The even greater thing about the reftest manifest files are their ability to make the failure-types conditional. These means that you can conditionally mark tests as failing or to be skipped depending on the build type (whether you are testing a debug build), the operating system, or even the cpu type. The variables which can be used in the failure-type conditional are described in the reftest sandbox implementation. You can also find examples in the tree.

You can read more about the reftest manifests in the reftest README.txt

Checklist when adding New Tests

  1. New suites should be added to the top level jstests.list, and Makefile.in files in js/src/tests.

  2. New sub-suites should be added to the suite level jstests.list file.

  3. New tests should be listed in the appropriate sub-suite’s jstests.list file. Each sub-suite’s jstests.list manifest must begin with a url-prefix.

    url-prefix ../../jsreftest.html?test=suitedir/subsuitedir/

    Tests must call the reportCompare() function (or its cousins) to record test results.

Updated September 26, 2009

The original patch had to be backed out due to failures in the SpiderMonkey shell. As a result, the js/tests were moved to js/src/tests to facilitate their integration into the build system. There was some pushback also about the two different manifest files required by the original design. With a modification to reftest.js to allow the use of a url-prefix, the browser specific manifest was dropped.

See bug 469718 for more background.