My T1 line went down during a thunderstorm on Sunday afternoon and has only just now been repaired. During this time Autophone was not able to test builds. I have restarted Autophone and initiated retesting of all tinderbox box builds from 2013-07-28 12:00:00 PM PT.
nexus-one-2 Found dead sdcard; replaced 11:50AM PT
The Kingston 4G Class 4 cards continue to fail. I have a number of them remaining, but will place an order for microsdcards equivalent to those used in releng tegras and pandas.
nexus-s-1 Found wifi disabled; reenabled 11:40AM PT.
The wifi router relocation didn’t solve all of the wifi disconnection/disabling issues unfortunately.
nexus-s-2 Found Firefox hung, SUTAgent dead; restarted SUTAgent.
nexus-s-2 continually suffers low memory conditions running the Mochitest-Canvas tests which results in Android killing off many processes which leaves the phone requiring a restart of the SUTAgent. I don’t know why nexus-s-2 can’t recover while nexus-s-1 can.
Updates
16:30 PM PT – Power outage resulted in server restart.
On 2013-07-17, several changes were made to Autophone Haxxor Virginia:
The Wireless router used to communicate with the phones was repositioned to be next to the phones. This was done in an attempt to eliminate cases where the phones would lose communication with the router and disable the Wifi connection. So far, this appears to have helped.
The Autophone Mochitest-Canvas tests were enabled. These tests run only the content/canvas Mochitests with the preferences gfx.canvas.azure.backends=skia gfx.canvas.azure.accelerated=true enabled.
To determine if the relocation of the router and the addition of the Mochitest-Canvas tests changed the overall, I retested the tinderbox nightly builds from 2013-07-15. This resulted in the testing of additional tinderbox builds which were missed in the original runs as well as expanded the number of Mochitest-Canvas results for the period.
Before
After
The “After” results are a bit smoother than the “Before” particularly on July 16. The “regression” in the throbber start time on July 16 is consistent in both sets of results and is a bit clearer in the “After” results. It appears the relocation of the router may have been helpful and that the addition of the Mochitest-Canvas tests did not have a detrimental effect.
The droid-pro-1(8132,0,230), nexus-one-1 (8068,0,232), appears orange on the autolog display due to a failure in pulling files from the phone, but inspecting the log shows it actually passed the Mochitest-Canvas tests.
The samsung-gs2-1 (8132,0,230)and samsung-gs2-2 (8132,0,230) both appear orange on the autolog display due to a failure in pulling files from the phones, but inspecting the logs show they actually pass the Mochitest-Canvas tests.
The nexus-one-2 (8068,1,231), again fails to pull files. When it does have data on the test results it fails 1 test but due to the incomplete logs it is not possible to determine which test.
nexus-s-1, nexus-s-2 fail to complete the tests due to Android and SUTAgent crashes due to low memory.
lg-revolution-1 fails to complete due to SIGSEGVs at Crash address: 0xdeadbaad with stack libc.so@0x15cb4,libEGL.so@0x8d0e.
Next Steps
Failures due to the inability to pull files from devices is a known issue that has been seen elsewhere. In Autophone’s case, it may be partly due to the network shaping using ipfw which may cause issues retrieving the files. If a solution to the failure to pull files could be found, then all of the galaxy s2 phones and at least one of the nexus one phones would be green.
It would be possible to skip the Mochitest-Canvas tests on the phones which are known to fail for whatever reason.
Again had to restart the OSX 10.8 mac mini hosting the Autophone server due to it becoming unresponsive via vnc. I have switched back to running the phonedash and autophone servers via ssh and screen; however my previous experience with running Autophone using ssh and screen leads me to believe this may be a reoccurring issue. This behavior is new since the switch from a Macbook running OSX 10.7 to a Mac Mini running OSX 10.8.
Phones
c8_aa_21_ac_0c_b5_droid-pro
The Kingston Class 4 4G sdcard failed and was replaced at 2013-07-09 16:30 PM PT
00_23_76_96_cc_6f_nexus-one
The wireless connection became disabled and was restarted at approximately 2013-07-09 16:15 PM PT