While I was traveling last week to the Automation and Tools work week, droid-pro-1, nexus-s-2 and samsung-gs3-1 had issues requiring reboots and/or battery resets. The phones are back up and attempting to work through their backlogs.
Bug 870876 – Create a remote mozrunner class for b2g caused a run-time error with the older version of mozbase the autophone server was using which caused the Mochitest-Canvas tests to fail with ImportError: cannot import name DEFAULT_PORTS. Updating mozbase on the server fixed the issue on Wednesday 2013-08-07.
The last failure for this device was on 2013-07-20
Update 09:21 AM PT – The device failed to run tests due its failure to locate the test root on the new sdcard which resulted in failures to install new fennec builds. After a factory reset and the re-installation of today’s SUTAgent and Watcher, the device appears to correctly place test root on the sdcard and is now running tests.
My T1 line went down during a thunderstorm on Sunday afternoon and has only just now been repaired. During this time Autophone was not able to test builds. I have restarted Autophone and initiated retesting of all tinderbox box builds from 2013-07-28 12:00:00 PM PT.
The Kingston 4G Class 4 cards continue to fail. I have a number of them remaining, but will place an order for microsdcards equivalent to those used in releng tegras and pandas.
nexus-s-1 Found wifi disabled; reenabled 11:40AM PT.
The wifi router relocation didn’t solve all of the wifi disconnection/disabling issues unfortunately.
nexus-s-2 Found Firefox hung, SUTAgent dead; restarted SUTAgent.
nexus-s-2 continually suffers low memory conditions running the Mochitest-Canvas tests which results in Android killing off many processes which leaves the phone requiring a restart of the SUTAgent. I don’t know why nexus-s-2 can’t recover while nexus-s-1 can.
16:30 PM PT – Power outage resulted in server restart.
On 2013-07-17, several changes were made to Autophone Haxxor Virginia:
The Wireless router used to communicate with the phones was repositioned to be next to the phones. This was done in an attempt to eliminate cases where the phones would lose communication with the router and disable the Wifi connection. So far, this appears to have helped.
The Autophone Mochitest-Canvas tests were enabled. These tests run only the content/canvas Mochitests with the preferences gfx.canvas.azure.backends=skia gfx.canvas.azure.accelerated=true enabled.
To determine if the relocation of the router and the addition of the Mochitest-Canvas tests changed the overall, I retested the tinderbox nightly builds from 2013-07-15. This resulted in the testing of additional tinderbox builds which were missed in the original runs as well as expanded the number of Mochitest-Canvas results for the period.
The “After” results are a bit smoother than the “Before” particularly on July 16. The “regression” in the throbber start time on July 16 is consistent in both sets of results and is a bit clearer in the “After” results. It appears the relocation of the router may have been helpful and that the addition of the Mochitest-Canvas tests did not have a detrimental effect.
The droid-pro-1(8132,0,230), nexus-one-1 (8068,0,232), appears orange on the autolog display due to a failure in pulling files from the phone, but inspecting the log shows it actually passed the Mochitest-Canvas tests.
The samsung-gs2-1 (8132,0,230)and samsung-gs2-2 (8132,0,230) both appear orange on the autolog display due to a failure in pulling files from the phones, but inspecting the logs show they actually pass the Mochitest-Canvas tests.
The nexus-one-2 (8068,1,231), again fails to pull files. When it does have data on the test results it fails 1 test but due to the incomplete logs it is not possible to determine which test.
nexus-s-1, nexus-s-2 fail to complete the tests due to Android and SUTAgent crashes due to low memory.
lg-revolution-1 fails to complete due to SIGSEGVs at Crash address: 0xdeadbaad with stack libc.so@0x15cb4,libEGL.so@0x8d0e.
Failures due to the inability to pull files from devices is a known issue that has been seen elsewhere. In Autophone’s case, it may be partly due to the network shaping using ipfw which may cause issues retrieving the files. If a solution to the failure to pull files could be found, then all of the galaxy s2 phones and at least one of the nexus one phones would be green.
It would be possible to skip the Mochitest-Canvas tests on the phones which are known to fail for whatever reason.