Autophone Status Update 2015-02-23

Autophone is one of the few test frameworks for measuring page load performance and testing video playback on real Android devices. Although Autophone failures will not cause checkins to the tree to be backed out, developers need to be aware of Autophone and how to respond when their patches have been identified as regressing Autophone.

Mozilla developers can find more information about Autophone, what the tests actually measure and how to run try builds against Autophone at Autophone for developers on wiki.mozilla.org.

Much has changed in Autophone since my last update:

  • Moved to the Mountain View QA lab.
  • Tests Nexus S, Nexus 4, Nexus 5 and Nexus 7 (2013) devices.
  • Uses adb to issue commands to the devices rather SUTAgent.
  • Runs video related Mochitests
  • Submits test results to Treeherder and test logs to S3.
  • Supports testing try builds.
  • Tests mozilla-central, mozilla-inbound, fx-team, b2g-inbound, mozilla-aurora, mozilla-beta and mozilla-release respositories.

Future plans for Autophone include:

  • Support for job retriggers and cancels via the Treeherder UI.
  • Addition of Nexus 6 and Nexus 9 devices.
  • Replacement of the mac mini host with rack mounted linux servers.
  • Improved graphing solution to better support the increased number of repositories and devices.

Autophone bugs are filed under Testing:Autophone in bugzilla. If you have problems using Autophone or would like to make suggestions, please file an Autophone bug. If you would like to reach out directly, I am usually available on irc.mozilla.org as bc in #ateam or #mobile.

Splitting and Packing Android Boot images

After wandering twisty little passages, all alike concerning rooting Android devices I decided the best way forward for devices with unlocked bootloaders was to root via modifying the default.prop values in the boot image. There are a number of different scripts available on various sites which purport to unpack and pack Android boot images, but it seemed that the best approach was to go to the source and see how Android dealt with boot images.

I created spbootimg and pkbootimg from the official mkbootimg in order that it would properly handle official android boot images at least.

spbootimg

usage: spbootimg
       -i|--input 

spbootimg splits an Android boot image file into separate files:

* <bootimg-filename>-kernel – kernel
* <bootimg-filename>-first-ramdisk – ramdisk
* <bootimg-filename>-second-ramdisk – only created if it existed in the input boot image file.
* <bootimg-filename-header> – text file containing constants discovered in the boot image

You can download the source for spbootimg and pkbootimg and build it yourself. I don’t provide binaries because I do not wish people who do not understand the consequences of their actions to brick their devices.

pkbootimg

usage: pkbootimg
       --kernel 
       --kernel-addr 
--ramdisk --ramdisk-addr
[ --second <2ndbootloader-filename>] [ --cmdline ] [ --board ] [ --pagesize ] --second-addr
--tags-addr
-o|--output

pkbootimg takes the output of spbootimg: a kernel file, a ramdisk file, an optional ramdisk file; and using the command line, board, page size and address information discovered in the original boot image, packs them into a new boot image which can be flashed onto a device using fastboot.

default.prop

Looking at the source for adb.c, we see that at a minimum we need a build of adb where ALLOW_ADBD_ROOT was set and either of the following was set in the ramdisk’s default.prop:

ro.secure=0

or

ro.debuggable=1
service.adb.root=1

ro.secure=0 will result in adbd running as root by default. ro.debuggable=1 and service.adb.root=1 will allow you to run adbd as root via the adb root command.

If your device’s version of adb was not built with ALLOW_ADBD_ROOT set, you will need to build your own version and place it in the ramdisk at sbin/adbd.

Once you are able to run adb as root via adb root, you will be able to remount the /system/ directory as writable and can install anything you wish.

su

Some of the automation used in mozilla requires the use of a version of the su command which has the same command line syntax as sh. In particular, we require that su support calling commands via su -c "command args...". You can possibly build your own version of su which will run commands as root without access control or you can use one of the available “Superuser” apps which manage access to su and provide some degree of security. I’ve found Koushik Dutta’s Superuser to work well with Android up to 4.4.

If you use Koushik’s Superuser, you will need at least version 1.0.3.0 and will need to make sure that the install-recovery.sh script is properly installed so that Koushik’s su runs as a daemon. This is automatically handled if you install Superuser.apk via a recovery image, but if you install Superuser manually, you will need to make sure to unpack Superuser.apk and manually install:

  • su to /system/xbin/su
  • install-recovery.sh to /system/etc/install-recovery.sh
  • Superuser.apk to /system/app/Superuser.apk

Autophone Status Update 2013-11-02

  • 7:15 AM PT – Swapped Kingston Class 4 4G sdcards for 00_23_76_96_cc_6f_nexus-one and c8_aa_21_ac_0c_b5_droid-pro after repeated failures checking for the device root.

  • 19:00 PM PT – Both 00_23_76_96_cc_6f_nexus-one and c8_aa_21_ac_0c_b5_droid-pro showed an inability to properly detect and use the external sdcard. I cycled through most of the previously used sdcards attempting to find ones that worked. After checking almost all of the remaining older cards, I found one that worked in the droid-pro but could not find a card that would work with the nexus-one. In my attempt to get a working patch for bug 933842 – Add ability to specify test root in SUTAgent.ini, I realized that mozdevice’s devicemanager.getDeviceRoot() caches the reported value of the test root. Since the autophone workers survive the rebooting of a device, the cached test root value is reused until the device’s worker is restarted. This means that once an incorrect test root is detected, it will not be corrected by rebooting the device. I am testing an preliminary version of the patch for bug 933842 on the nexus one which forces the test root to /mnt/sdcard. I have resubmitted all builds from 2013-10-31 to now for both nexus ones and the droid pro. Hopefully the devices will be more reliable. It may also be the case that the older sdcards are not as unreliable as initially thought.

    I also noticed that Autophone’s workers use devicemanager’s getDeviceRoot() to determine if a device is still responsive. Since the value is cached, this test is ineffective in determining if a device is still responding.

    Before
    phonedash results prior to re-testing nexus one and droid pro
    phonedash results prior to re-testing nexus one and droid pro

Autophone Status Update 2013-11-01

droid-pro-1 and nexus-one-2 sdcard change

nexus-one-2 (00_23_76_96_cc_6f_nexus-one) and droid-pro-1 (c8_aa_21_ac_0c_b5_droid-pro) appear to have difficulties with the new SanDisk Extreme Pro UHS-I. I have reverted back to the old Kingston class 4 4G cards on these devices. I have also discarded the older jobs which were pending so as to not confuse the graphs with a mixture of sdcards.

Any apparent regression from 2013-10-31 to 2013-11-01 for these two phones should be disregarded.

Kingston class 4 4G cards

I went through the older Kingston cards and performed a disk verification and repair using my Mac Book Pro’s disk utility. Those cards which could not be repaired have been removed from the active inventory. I will attempt to use the remaining cards in the devices which can not reliably run using the newer SanDisk cards. If a device appears to have issues with accessing the sdcard, I will attempt to swap in another and will move the offending card into an ‘unreliable’ category.

New Phones

I received a number of phones from Haxxor Canada and am in the process of bringing them online to help partition devices between the s1s2, canvas SkiaGL and regression testing.

New Phones
Phone Type DNS Name Device Name Notes
atrix-4g mb860 atrix-4g-1 40_fc_89_4c_95_3f_atrix-4g cyanogen mod7 – Android 2.3.7, new SanDisk sdcard
htc-sensation-4g htc-sensation-4g-1 18_87_96_a7_89_70_htc-sensation-4g Android 2.3.4, new SanDisk sdcard
htc-sensation-4g htc-sensation-4g-2 c0_3f_0e_ba_5f_34_htc-sensation-4g Android 2.3.4, new SanDisk sdcard, Rooted but unable to reboot via SutAgent.
sony-experia sony-experia-1 5c_b5_24_b6_7d_1f_sony-experia Android 2.3.4, new SanDisk sdcard
droid droid-1 a4_ed_4e_59_6c_b2_droid Android 2.1, new SanDisk sdcard
droid-pro droid-pro-4 98_4b_4a_13_01_84_droid-pro Android 2.3.3, new SanDisk sdcard
nexus-one nexus-one-3 90_21_55_09_87_95_nexus-one cyanogen mod7 – Android 2.3.7, new SanDisk sdcard
samsung-gs2 samsung-gs2-4 14_7d_c5_af_c8_e3_samsung-gs2 Android 2.3.5
samsung-gs2 samsung-gs2-5 04_46_65_fd_2f_e1_samsung-gs2 Android 2.3.4
samsung-gs3 samsung-gs3-3 38_aa_3c_1c_f6_94_samsung-gs3 Android 4.0.4

Autophone Status Update 2013-10-22

2013-10-22 08:45 PT

Obtained new Micro SDHC cards (SanDisk Extreme Pro UHS-I cards) which match those in use for Panda testing and installed in the following phones which use external cards:

  • droid-pro-1
  • nexus-one-1
  • nexus-one-2

Any performance or reliability improvement in these phones in the next round of tests should be attributable to the new sdcards.

The shipment of phones from Haxxor Canada to Haxxor Virginia is being held up in due to an import clearance hold since Friday October 18. I’m not sure what is the cause for the delay but it may be due to a backlog caused by the U.S. Government “shutdown” this month. Hopefully the phones will be released soon.