test262.ecmascript.org shootout

Dave Fugate of Microsoft announced an update to test262.ecmascript.org, the test suite for ECMAScript 5. I thought I would check it out. Internet Explorer 9 was tested on 32bit Windows Vista, while the others were all tested on Mac OS X 10.5.

Browser Tests To Run Total Tests Ran Pass Fail Failed To Load
Internet Explorer 9 10456 10456 10439 17 0
Firefox 4.0 10456 10456 10155 301 0
Chrome 10 10456 10456 9959 497 0
Safari 5.0.4 10456 10456 9156 1300 72
Opera 11 10456 10456 6905 3551 66

The test suite looks very cool. Kudos to everyone involved in creating test262!

Browser JavaScript Reftests

The Bad Old Days

The JavaScript Tests have been the red headed stepchild of mozilla testing for some time. At least some of the reasons are:

  • The tests date to pre-mozilla days and do not follow the style of any of the other test suites.

  • The tests were originally designed to be run only in the JavaScript shell and had the ability to be run in the browser tacked on to them like a bad toupe

  • Not all of the tests pass and keeping track of the known failures has been a time consuming and error prone task.

  • Not having a good method of tracking failures, excluding long running tests or excluding tests which crashed the browser resulted in unacceptable run times for developers thus leaving a gap in the testing developers could perform prior to checking in changes to the JavaScript engine.

  • The automation used to run the tests in the shell and browser was not appropriate for running on the tinderboxes. As a result, the JavaScript engine did not benefit from immediate, cross-platform testing for each checkin but instead had to wait on me to detect regressions and file bugs.

  • The tests were not specialized for each branch but were instead maintained identically across branches. As the JavaScript engine has evolved, the differences between passing behavior on the branches has been problematic.

  • A single person was responsible for adding new tests causing the addition of new tests to increasingly lag behind new patches.

All of that has come to and end, because…

The Good Times have arrived!

(at least for the browser on tracemonkey)

David Baron’s reftest framework provides the solution to many of the problems of the Bad Old Days. It provides:

  • Familiar test invocation.

  • Relatively quick run times compared to the old Sisyphus-based browser JavaScript tests.

  • The reftest style manifests provide an excellent means of tracking obsolete tests, test failures, tests which take too long to run or tests which crash

  • Easy (hopefully) integration into the automated tests running on Tinderbox by leveraging the same approaches used to run the layout reftests

Currently, only the browser jsreftests are implemented. The same approach is planned to be implemented for the shell tests.

Running Browser JavaScript reftests

Running the jsreftests in the browser is familiar to anyone who has run the layout reftests or crash tests. Simply invoke make in the object directory with the target jstestbrowser.

$ make -C $(OBJDIR) jstestbrowser

This executes the following command.

$ python $(OBJDIR)/_tests/reftest/runreftest.py \
       --extra-profile-file=js/src/tests/user.js js/src/tests/jstests.list

The user.js file is added to the run time profile to enable UniveralXPConnect and other privileges required for running the JavaScript tests. You can customize user.js to set other preferences you wish.

You can see example logs here:

No longer will you have to perform base-line runs to determine the known failures prior to testing your patches. If the jsreftests run to completion without crashing and without generating any UNEXPECTED results, your patch is good to go.

jsreftest manifests

jstests.listis the reftest manifest file used to tell the test runner which tests to execute. jstests.list is to be used for both the JavaScript shell tests and the JavaScript browser tests. url-prefix, a new feature for reftest manifests, is used to provide a single manifest file for both the shell and browser tests.

The manifest files are organized into a 3-level hierarchy.

The top level manifest files live in js/src/tests/. They incorporate the individual suites through the include manifest keyword. Whenever a new suite is added, the top level manifest files must be updated to include it. New suites must also be listed in js/src/tests/Makefile.in in order to be packaged with the other tests.

The second level manifest files are contained in the suite directories. They also include the sub-suites through the use of the include manifest keyword. Whenever a new sub-suite is added to a suite, the suite’s manifest files must be updated to include it.

The third and final level manifest files are contained in the sub-suite directories. These manifest files list each test using the script manifest keyword. This keyword tells the test runner to obtain the actual test results from the array returned by the getTestCases() function in the testcase. In order for the test results to be properly recorded in the test case results array, you must use one of the standard reporting functions, such as reportCompare(), reportMatch(), etc., to record the result of the test. If you do not use reportCompare() or one of its cousins, your test will fail with the message that No test results reported.

The first line of the manifest file contains a “url-prefix” which tells the reftest extension to prefix each test file with the url-prefix value before attempting to load it. The shell test driver will ignore the url-prefix, thus allowing a single manifest file to be used by both the shell and browser. The remainder of the manifest lists each test file preceded by the script keyword.

For example, js/src/tests/ecma_5/Date/jstests.list:

url-prefix ../../jsreftest.html?test=ecma_5/Date/

The truly great thing about the reftest manifest files are their ability to give tests a failure-type. Each test can be marked as either:


Tests marked with fails will be executed and will be marked as TEST-KNOWN-FAIL if they fail. However if they pass, they will be marked as TEST-UNEXPECTED-PASS.


Tests marked with random will be executed and will be marked as TEST-KNOWN-FAIL(EXPECTED RANDOM) if they fail or TEST-PASS(EXPECTED RANDOM) if they pass. This is useful for tests which may not reliably report a test failure but which you wish to continue to execute in the event they may find a regression through a crash or assertion.

random results are also used in the event that a test that returns multiple test results is marked with the failure-type fails. In that case, failures are marked with TEST-KNOWN-FAIL while successes are marked with TEST-PASS(EXPECTED RANDOM). This is necessary since it is not possible to mark the individual test case results in a multi-result test as either passing or failing.


Tests marked with skip will not be executed at all. skip can be used for obsolete tests, for tests which are known to crash the browser, tests which do not terminate or tests which take too much time.

The even greater thing about the reftest manifest files are their ability to make the failure-types conditional. These means that you can conditionally mark tests as failing or to be skipped depending on the build type (whether you are testing a debug build), the operating system, or even the cpu type. The variables which can be used in the failure-type conditional are described in the reftest sandbox implementation. You can also find examples in the tree.

You can read more about the reftest manifests in the reftest README.txt

Checklist when adding New Tests

  1. New suites should be added to the top level jstests.list, and Makefile.in files in js/src/tests.

  2. New sub-suites should be added to the suite level jstests.list file.

  3. New tests should be listed in the appropriate sub-suite’s jstests.list file. Each sub-suite’s jstests.list manifest must begin with a url-prefix.

    url-prefix ../../jsreftest.html?test=suitedir/subsuitedir/

    Tests must call the reportCompare() function (or its cousins) to record test results.

Updated September 26, 2009

The original patch had to be backed out due to failures in the SpiderMonkey shell. As a result, the js/tests were moved to js/src/tests to facilitate their integration into the build system. There was some pushback also about the two different manifest files required by the original design. With a modification to reftest.js to allow the use of a url-prefix, the browser specific manifest was dropped.

See bug 469718 for more background.

Assert This! – Top Site Assertions, October 27, 2007

Assert This! – Top Site Assertions, October 27, 2007

No matter where you go, Jesse’s been there already.

During some top site testing of about 7,500 pages with a debug Minefield build on Linux over the weekend, I ran into the following assertions ordered by decreasing count. This should give some idea of the frequency of each in the wild.

2939 ASSERTION: Computed overflow area must contain frame bounds: ‘aNewSize.width == 0 || aNewSize.height == 0 || aOverflowArea->Contains(nsRect(nsPoint(0, 0), aNewSize))’, file 1.9.0/mozilla/layout/generic/nsFrame.cpp, line 5222
1756 ASSERTION: bad argument: ‘(aRight > mLeft) && (aRight < mRight)’, file 1.9.0/mozilla/layout/generic/nsSpaceManager.cpp, line 1403
1333 ASSERTION: no user stylesheets in styleset, but we have one!: ‘numBefore > 0’, file 1.9.0/mozilla/layout/base/nsPresShell.cpp, line 1826
751 ASSERTION: invalid divisor: ‘Error’, file 1.9.0/mozilla/layout/tables/nsTableFrame.cpp, line 3287
345 ASSERTION: didn’t subtract all that we added: ‘totalSPref == 0 && totalSMin == 0 && totalSNonPctPref == 0 && nonPctCount == 0 && minOutsidePref == 0 && minWithinPref == 0 && (info.prefCoord == 0 || info.prefCoord == nscoord_MAX) && (info.prefPercent == 0.0f || !spanHasNonPct)’, file 1.9.0/mozilla/layout/tables/BasicTableLayoutStrategy.cpp, line 531
271 ASSERTION: nscoord addition will reach or pass nscoord_MAX: ‘(PRInt64)a + (PRInt64)b < (PRInt64)nscoord_MAX’, file ../../dist/include/gfx/nsCoord.h, line 153
258 ASSERTION: unconstrained widths no longer supported: ‘aContainingBlockWidth != NS_UNCONSTRAINEDSIZE’, file 1.9.0/mozilla/layout/base/nsLayoutUtils.cpp, line 1641
257 ASSERTION: Doing nscoord addition with values > nscoord_MAX: ‘a < nscoord_MAX && b < nscoord_MAX’, file ../../dist/include/gfx/nsCoord.h, line 151
211 ASSERTION: bad width: ‘Not Reached’, file 1.9.0/mozilla/layout/generic/nsLineLayout.cpp, line 177
204 ASSERTION: cannot call on a dirty frame not currently being reflowed: ‘!NS_SUBTREE_DIRTY(this) || (GetStateBits() & NS_FRAME_IN_REFLOW)’, file 1.9.0/mozilla/layout/generic/nsFrame.cpp, line 556
132 ASSERTION: shouldn’t use unconstrained widths anymore: ‘availableWidth != NS_UNCONSTRAINEDSIZE’, file 1.9.0/mozilla/layout/generic/nsHTMLReflowState.cpp, line 302
132 ASSERTION: nsStreamLoader not thread-safe: ‘_mOwningThread.GetThread() == PR_GetCurrentThread()’, file 1.9.0/mozilla/netwerk/base/src/nsStreamLoader.cpp, line 65
111 ASSERTION: RefreshURIList timer callbacks should only be RefreshTimer objects: ‘rt’, file 1.9.0/mozilla/docshell/base/nsDocShell.cpp, line 4635
93 ASSERTION: update-band called late: ‘psd->mX == psd->mLeftEdge’, file 1.9.0/mozilla/layout/generic/nsLineLayout.cpp, line 307
90 ASSERTION: shouldn’t have unconstrained widths anymore: ‘psd->mRightEdge != NS_UNCONSTRAINEDSIZE’, file 1.9.0/mozilla/layout/generic/nsLineLayout.cpp, line 815
48 ASSERTION: shouldn’t have unconstrained widths anymore: ‘NS_UNCONSTRAINEDSIZE != aReflowState.availableWidth’, file 1.9.0/mozilla/layout/generic/nsLineLayout.cpp, line 1099
45 ASSERTION: XPConnect is being called on a scope without a ‘Components’ property!
36 ASSERTION: Should be in an update while destroying frames: ‘mUpdateCount != 0’, file 1.9.0/mozilla/layout/base/nsCSSFrameConstructor.cpp, line 9572
35 ASSERTION: Adding child where we already have a child? This will likely misbehave: ‘Error’, file 1.9.0/mozilla/docshell/shistory/src/nsSHEntry.cpp, line 594
30 ASSERTION: deleting dirty buffer: ‘mBufDirty == PR_FALSE’, file 1.9.0/mozilla/netwerk/cache/src/nsDiskCacheStreams.cpp, line 750
30 ASSERTION: Unhandled border style!!: ‘Not Reached’, file 1.9.0/mozilla/layout/base/nsCSSRendering.cpp, line 1928
30 ASSERTION: Non-border-colors case with borderColorStyleCount < 1 or > 3; what happened?: ‘Error’, file 1.9.0/mozilla/layout/base/nsCSSRendering.cpp, line 2112
30 ASSERTION: Flush() failed: ‘NS_SUCCEEDED(rv)’, file 1.9.0/mozilla/netwerk/cache/src/nsDiskCacheStreams.cpp, line 461
30 ASSERTION: Failed to position iterator!: ‘NS_SUCCEEDED(rv)’, file 1.9.0/mozilla/content/base/src/nsContentIterator.cpp, line 997
27 ASSERTION: RemovedAsPrimaryFrame called after PreDestroy: ‘PR_FALSE’, file 1.9.0/mozilla/layout/forms/nsTextControlFrame.cpp, line 1202
22 ASSERTION: iconv failed: ‘Error’, file 1.9.0/mozilla/xpcom/io/nsNativeCharsetUtils.cpp, line 572
14 ASSERTION: intrinsic widths out of order: ‘aMinCoord <= aPrefCoord’, file 1.9.0/mozilla/layout/tables/nsTableColFrame.h, line 190
14 ASSERTION: PushBack: ‘Not Reached’, file 1.9.0/mozilla/netwerk/protocol/http/src/nsHttpConnection.h, line 117
13 ASSERTION: bad width: ‘metrics.width>=0’, file 1.9.0/mozilla/layout/generic/nsLineLayout.cpp, line 956
12 ASSERTION: this shouldn’t happen anymore: ‘NS_UNCONSTRAINEDSIZE != aComputedWidth && NS_UNCONSTRAINEDSIZE != aAvailWidth’, file 1.9.0/mozilla/layout/generic/nsHTMLReflowState.cpp, line 1924
12 ASSERTION: shouldn’t use unconstrained widths anymore: ‘(mFrameType == NS_CSS_FRAME_TYPE_INLINE && !frame->IsFrameOfType(nsIFrame::eReplaced)) || frame->GetType() == nsGkAtoms::textFrame || mComputedWidth != NS_UNCONSTRAINEDSIZE’, file 1.9.0/mozilla/layout/generic/nsHTMLReflowState.cpp, line 323
11 ASSERTION: Shouldn’t have pending bindings!: ‘mAttachedStack.Length() == 0’, file 1.9.0/mozilla/content/xbl/src/nsBindingManager.cpp, line 913
10 ASSERTION: invalid BC damage area: ‘PR_FALSE’, file 1.9.0/mozilla/layout/tables/nsTableFrame.cpp, line 3861
8 ASSERTION: evictionRank out of sync: ‘evictionRank == mapRecord->EvictionRank()’, file 1.9.0/mozilla/netwerk/cache/src/nsDiskCacheMap.cpp, line 484
7 ASSERTION: no frame, see bug #188946: ‘frame’, file 1.9.0/mozilla/editor/libeditor/base/nsEditor.cpp, line 4159
7 ASSERTION: didn’t subtract all that we added: ‘space == 0 && ((l2t == FLEX_PCT_LARGE) ? (-0.001f < basis.f && basis.f < 0.001f) : (basis.c == 0 || basis.c == nscoord_MAX))’, file 1.9.0/mozilla/layout/tables/BasicTableLayoutStrategy.cpp, line 1013
7 ASSERTION: can’t mark frame dirty during reflow: ‘!mIsReflowing’, file 1.9.0/mozilla/layout/base/nsPresShell.cpp, line 3132
6 ASSERTION: no unconstrained widths should be present anymore: ‘NS_UNCONSTRAINEDSIZE != aReflowState.ComputedWidth()’, file 1.9.0/mozilla/layout/generic/nsBlockReflowState.cpp, line 113
6 ASSERTION: This shouldn’t be called anymore: ‘Not Reached’, file 1.9.0/mozilla/layout/generic/nsLineLayout.cpp, line 2437
5 ASSERTION: subshell not in the map: ‘shellContent’, file 1.9.0/mozilla/docshell/base/nsDocShell.cpp, line 3782
4 ASSERTION: storage allocated for zero data size: ‘!mBinding->mRecord.DataLocationInitialized()’, file 1.9.0/mozilla/netwerk/cache/src/nsDiskCacheStreams.cpp, line 372
4 ASSERTION: cannot call on a dirty frame not currently being reflowed: ‘!NS_SUBTREE_DIRTY(this) || (GetStateBits() & NS_FRAME_IN_REFLOW)’, file 1.9.0/mozilla/layout/generic/nsFrame.cpp, line 583
4 ASSERTION: Uh, LeaveModalState() called w/o a reachable top window?: ‘Error’, file 1.9.0/mozilla/dom/src/base/nsGlobalWindow.cpp, line 5219
4 ASSERTION: Failed to create scaled font: ‘cairo_scaled_font_status(scaledFont) == CAIRO_STATUS_SUCCESS’, file 1.9.0/mozilla/gfx/thebes/src/gfxPangoFonts.cpp, line 672
3 ASSERTION: nsNSSComponent relies on profile manager to wait for synchronous shutdown of all network activity: ‘mIsNetworkDown’, file 1.9.0/mozilla/security/manager/ssl/src/nsNSSComponent.cpp, line 2311
2 ASSERTION: unexpected progress values: ‘progress <= progressMax’, file 1.9.0/mozilla/netwerk/protocol/http/src/nsHttpChannel.cpp, line 4483
1 ASSERTION: More UnblockOnload() calls than BlockOnload() calls; dropping call: ‘Not Reached’, file 1.9.0/mozilla/content/base/src/nsDocument.cpp, line 5625
1 ASSERTION: CachedChromeStream doesn’t receive data: ‘Not Reached’, file 1.9.0/mozilla/content/xul/document/src/nsXULDocument.cpp, line 4339