1086672 - Running android_emulator_unittest.py fails locally due to missing files

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Chris Manchester (limited bugmail, email directly)]

Running the command:

scripts/android_emulator_unittest.py --cfg android/androidarm.py --test-suite mochitest-1 --blob-upload-branch try --download-symbols ondemand --cfg developer_config.py --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-7efbbe6ffad6/try-android/fennec-36.0a1.en-US.android-arm.apk --test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-7efbbe6ffad6/try-android/fennec-36.0a1.en-US.android-arm.tests.zip

To attempt to debug a weird failure from this try run: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=7efbbe6ffad6

Fails with this stack trace: http://pastebin.mozilla.org/6841147

My apologies in advance if there's something obvious about running mozharness locally that I have overlooked.

Comment 1

[Armen [:armenzg]]

Adding simone as well since he has touched this part of the code recently.

Comment 2

[Armen [:armenzg]]

Attachment: wip - android.diff

I started messing with this.

chmanchester, do you want to give it a try?

Do you have an up-to-date mozharness repo? simone landed a change recently to retrieve the manifest from in-tree.
It might even be inside the test.zip (that is why I moved download and extract higher).

Comment 3

[Chris Manchester (limited bugmail, email directly)]

I think you identified the issue in the filed dependent, but I get this far now (updated to default and with your patch applied): 

14:50:14     INFO - retry: Calling <bound method Proxxy._download_file of <mozharness.mozilla.proxxy.Proxxy object at 0x7fedf9d5d950>> with args: ('http://talos-remote.pvt.build.mozilla.org/tegra/tegra-host-utils.Linux.742597.zip', '/home/chris/mozharness/build/tegra-host-utils.Linux.742597.zip'), kwargs: {}, attempt #1
14:50:14     INFO - Downloaded 1116 bytes.
14:50:14     INFO - Running command: ['unzip', '-q', '-o', '/home/chris/mozharness/build/tegra-host-utils.Linux.742597.zip'] in /home/chris/mozharness/build/hostutils
14:50:14     INFO - Copy/paste: unzip -q -o /home/chris/mozharness/build/tegra-host-utils.Linux.742597.zip
14:50:14     INFO - Calling ['unzip', '-q', '-o', '/home/chris/mozharness/build/tegra-host-utils.Linux.742597.zip'] with output_timeout 1760
14:50:14     INFO -  [/home/chris/mozharness/build/tegra-host-utils.Linux.742597.zip]
14:50:14     INFO -    End-of-central-directory signature not found.  Either this file is not
14:50:14     INFO -    a zipfile, or it constitutes one disk of a multi-part archive.  In the
14:50:14     INFO -    latter case the central directory and zipfile comment will be found on
14:50:14     INFO -    the last disk(s) of this archive.
14:50:14     INFO -  unzip:  cannot find zipfile directory in one of /home/chris/mozharness/build/tegra-host-utils.Linux.742597.zip or
14:50:14     INFO -          /home/chris/mozharness/build/tegra-host-utils.Linux.742597.zip.zip, and cannot find /home/chris/mozharness/build/tegra-host-utils.Linux.742597.zip.ZIP, period.
14:50:14    ERROR - Return code: 9

Comment 4

[Armen [:armenzg]]

I have to spend a bit more time on this to get it right.
It now tries to download the file correctly (after adding host_utils_url to developer_config.py), however, I have to make it an authenticated download.

Comment 5

[Armen [:armenzg]]

Attachment: wip - android.diff

The patch is *almost* there.
We just need to read the manifest to download the avd.

This patch goes on top of the mass refactoring in bug 1087664.

Comment 6

[Armen [:armenzg]]

Attachment: wip - android.diff

I think all there is left is to stop compiz from being killed, verify is that symlink is needed and verify that emulator is in the PATH.
I think the Android SDK is what is needed to have emulator.

I will post dependent patches on dep bugs.

I run this locally [1] and I got almost there:
14:43:58     INFO - #####
14:43:58     INFO - ##### Running start-emulators step.
14:43:58     INFO - #####
14:43:58     INFO - Running pre-action listener: _resource_record_pre_action
14:43:58     INFO - Running main action method: start_emulators
14:43:58     INFO - Let's kill every process called compiz
14:43:58     INFO - Killing pid 2980.
14:43:58     INFO - Let's kill every process called xpcshell
14:43:58     INFO - Attempting to establish symlink for /home/armenzg/repos/mozharness/build/libGL.so
14:43:58     INFO - Symlinking /home/armenzg/repos/mozharness/build/libGL.so -> /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1
14:43:58     INFO - Attempt #1 to launch emulators...
14:43:58     INFO - Created temp file /tmp/tmppNMkBx.
14:43:58     INFO - Trying to start the emulator with this command: emulator -avd test-1 -debug init,console,gles,memcheck,adbserver,adbclient,adb,avd_config,socket -port 5554 -qemu -m 1024 -cpu cortex-a9
14:43:58     INFO - Running post-action listener: _resource_record_post_action
14:43:58    FATAL - Uncaught exception: Traceback (most recent call last):
14:43:58    FATAL -   File "/home/armenzg/repos/mozharness/mozharness/base/script.py", line 1145, in run
14:43:58    FATAL -     self.run_action(action)
14:43:58    FATAL -   File "/home/armenzg/repos/mozharness/mozharness/base/script.py", line 1087, in run_action
14:43:58    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
14:43:58    FATAL -   File "/home/armenzg/repos/mozharness/mozharness/base/script.py", line 1028, in _possibly_run_method
14:43:58    FATAL -     return getattr(self, method_name)()
14:43:58    FATAL -   File "scripts/android_emulator_unittest.py", line 543, in start_emulators
14:43:58    FATAL -     emulator_proc = self._launch_emulator(emulator_index)
14:43:58    FATAL -   File "scripts/android_emulator_unittest.py", line 224, in _launch_emulator
14:43:58    FATAL -     proc = subprocess.Popen(command, stdout=tmp_stdout, stderr=tmp_stdout, env=env)
14:43:58    FATAL -   File "/usr/lib/python2.7/subprocess.py", line 709, in __init__
14:43:58    FATAL -     errread, errwrite)
14:43:58    FATAL -   File "/usr/lib/python2.7/subprocess.py", line 1326, in _execute_child
14:43:58    FATAL -     raise child_exception
14:43:58    FATAL - OSError: [Errno 2] No such file or directory
14:43:58    FATAL - Running post_fatal callback...
14:43:58     INFO - Let's kill every process called emulator64-arm
14:43:58    FATAL - Exiting -1
14:43:58     INFO - Running post-run listener: _resource_record_post_run
14:43:58     INFO - Running post-run listener: _upload_blobber_files
14:43:58     INFO - Blob upload gear active.
14:43:58  WARNING - Blob upload directory does not exist!

[1]
python scripts/android_emulator_unittest.py --cfg android/androidarm.py --test-suite mochitest-1 --blob-upload-branch try --download-symbols ondemand --cfg developer_config.py --installer-url ~/moz/builds/fennec/fennec-36.0a1.en-US.android-arm.apk --test-url ~/moz/builds/fennec/fennec-36.0a1.en-US.android-arm.tests.zip

Comment 7

[Armen [:armenzg]]

I cannot any progress on this until the dep bugs are fixed.
Putting this on the side for now.

Comment 8

[Armen [:armenzg]]

Attachment: [mh] allow running Android test jobs locally

It's a big patch. I have no rush to get review on it.

Comment 9

[Armen [:armenzg]]

This can be run locally like this:
python scripts/android_emulator_unittest.py --cfg android/androidarm.py --test-suite mochitest-gl-1 --blob-upload-branch try --download-symbols ondemand --cfg developer_config.py --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-d933eaa88e60/try-android-api-9/fennec-37.0a1.en-US.android-arm.apk --test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-d933eaa88e60/try-android-api-9/fennec-37.0a1.en-US.android-arm.tests.zip

Comment 10

[Geoff Brown [:gbrown]]

I tested this against my own binaries:

python scripts/android_emulator_unittest.py --cfg android/androidarm.py --test-suite mochitest-gl-1 --blob-upload-branch try --download-symbols ondemand --cfg deeveloper_config.py --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-fc12d403667f/try-android-api-9/fennec-37.0a1.en-US.android-arm.apk --test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-fc12d403667f/try-android-api-9/fennec-37.0a1.en-US.android-arm.tests.zip

It worked perfectly -- nice!

Comment 11

[Geoff Brown [:gbrown]]

Comment on attachment 8540394 [details] [diff] [review]
[mh] allow running Android test jobs locally

Review of attachment 8540394 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/developer_config.py
@@ +35,5 @@
> +    "tooltool_servers": [
> +        "https://secure.pub.build.mozilla.org/tooltool/pvt/build"
> +    ],
> +    "tooltool_cache": os.path.join(LOCAL_WORKDIR, "builds/tooltool_cache"),
> +    "tooltool_cache_path": os.path.join(LOCAL_WORKDIR, "builds/tooltool_cache"),

tooltool_cache_path is the same as tooltool_cache?

I do not see where tooltool_cache_path is used.

@@ +40,5 @@
> +    "tooltool_py_url": "https://raw.githubusercontent.com/mozilla/" + \
> +        "build-tooltool/master/tooltool.py",
> +
> +    # Android related
> +    "host_utils_url": "https://secure.pub.build.mozilla.org/tooltool" + \

Consider making this relative to tooltool_servers....it might not be worth the effort.

::: scripts/android_emulator_unittest.py
@@ +210,5 @@
>          env['LD_LIBRARY_PATH'] = self.abs_dirs['abs_work_dir']
>  
> +        if self.config.get("developer_mode"):
> +            avds_dir = os.path.join(self.abs_dirs['abs_avds_dir'], 'avd')
> +            env['ANDROID_AVD_HOME'] = avds_dir 

Is ANDROID_AVD_HOME used by the emulator? A comment here might be nice.

Trailing space here.

Comment 12

[Armen [:armenzg]]

Comment on attachment 8540394 [details] [diff] [review]
[mh] allow running Android test jobs locally

Review of attachment 8540394 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/developer_config.py
@@ +35,5 @@
> +    "tooltool_servers": [
> +        "https://secure.pub.build.mozilla.org/tooltool/pvt/build"
> +    ],
> +    "tooltool_cache": os.path.join(LOCAL_WORKDIR, "builds/tooltool_cache"),
> +    "tooltool_cache_path": os.path.join(LOCAL_WORKDIR, "builds/tooltool_cache"),

I don't know either. The only place that I see it is inside of the _dev.py which I'm going to delete since it is not needed anymore.

@@ +40,5 @@
> +    "tooltool_py_url": "https://raw.githubusercontent.com/mozilla/" + \
> +        "build-tooltool/master/tooltool.py",
> +
> +    # Android related
> +    "host_utils_url": "https://secure.pub.build.mozilla.org/tooltool" + \

Fixed.

::: scripts/android_emulator_unittest.py
@@ +210,5 @@
>          env['LD_LIBRARY_PATH'] = self.abs_dirs['abs_work_dir']
>  
> +        if self.config.get("developer_mode"):
> +            avds_dir = os.path.join(self.abs_dirs['abs_avds_dir'], 'avd')
> +            env['ANDROID_AVD_HOME'] = avds_dir 

IIRC when using a directory different than $HOME/.android/avd we need to set this value. This is important since we're running in a developer's machine

Comment 13

[Armen [:armenzg]]

https://hg.mozilla.org/build/mozharness/rev/27e55b4b5c9a

Comment 14

[Armen [:armenzg]]

Landed as: http://hg.mozilla.org/build/mozharness/rev/27e55b4b5c9a

If anyone wants to try it in the future you can do so by running it as:
python scripts/android_emulator_unittest.py --cfg android/androidarm.py --test-suite mochitest-gl-1 --blob-upload-branch try --download-symbols ondemand --cfg developer_config.py --installer-url ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/fennec-37.0a1.en-US.android-arm.apk --test-url ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/fennec-37.0a1.en-US.android-arm.tests.zip

Filed bug 1117954 for Android 10/11 jobs.

http://armenzg.blogspot.ca/2015/01/run-android-test-jobs-locally.html

Comment 15

[Armen [:armenzg]]

Comment on attachment 8540394 [details] [diff] [review]
[mh] allow running Android test jobs locally

Jordan, don't look at this before you come back out of vacation.

This code has already landed.
This is to determine if there is any follow up clean up or changes you would like to see. Maybe I did something you don't believe is the right way.

Thanks!

Comment 16

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/27e55b4b5c9a

Comment 17

[Armen [:armenzg]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=011055026725

Comment 18

[Armen [:armenzg]]

This patch got backed out.
I'm retesting by moving back to subprocess.

Comment 19

[Armen [:armenzg]]

Attachment: [mh] go back to subprocess

Comment 20

[Armen [:armenzg]]

Comment on attachment 8546223 [details] [diff] [review]
[mh] go back to subprocess

This patch adds the missing __init__.py files and goes back to subprocess.

Unfortunately, the interdiff tool is not helping.

The job has passed in here:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=011055026725

I will do a full run as well.

Comment 21

[Armen [:armenzg]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=83b30051e449

Comment 22

[Geoff Brown [:gbrown]]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #21)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=83b30051e449

This has lots of "We have not been able to establish a telnet connection with the emulator" failures.

Comment 23

[Armen [:armenzg]]

So sad :/

Re-triggering. I will try again.

Comment 24

[Geoff Brown [:gbrown]]

It looks like the emulator is not finding the avd. I think your patch sets abs_avd_dir, for the non-developer case, to <work-dir>/avds_dir but does not set ANDROID_AVD_HOME.

Comment 25

[Armen [:armenzg]]

I have landed this:
http://hg.mozilla.org/users/armenzg_mozilla.com/mozharness/rev/51f276da7476

What I don't understand is why any of the jobs would have succeeded.
Maybe lingering emulators?
Maybe lingering ~/.android with avds in there?
I hope not!

Comment 26

[Armen [:armenzg]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e447d97da745

Comment 27

[Armen [:armenzg]]

It seems that in my user repo I have both a branch called "android_locally" and a bookmark (it cannot be seen on hg-web).

I've pushed a new bookmark to make sure my latest changes are being tested.

It seems that the bookmark points to 39b64a4e0457 while the branch points to 51f276da7476.

Comment 28

[Armen [:armenzg]]

Attachment: [mh] allow running Android test jobs locally

Never mix bookmarks and branches on your user repo. Never...

I think I will have in an hour the results to say that this is working.

Comment 29

[Armen [:armenzg]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d6587c5a057

Comment 30

[Armen [:armenzg]]

Note to self: only use branches and never bookmarks.

Comment 31

[Armen [:armenzg]]

https://hg.mozilla.org/build/ash-mozharness/rev/80891f815b2d

Comment 32

[Armen [:armenzg]]

Attachment: [mh] allow running Android test jobs locally

The latest code.
I have seen a green job.
I'm pushing a new try push to confirm.

Comment 33

[Armen [:armenzg]]

Comment on attachment 8555991 [details] [diff] [review]
[mh] allow running Android test jobs locally

The interdiff should be minor.

The jobs are starting to come out green.

Comment 34

[Geoff Brown [:gbrown]]

Comment on attachment 8555991 [details] [diff] [review]
[mh] allow running Android test jobs locally

Review of attachment 8555991 [details] [diff] [review]:
-----------------------------------------------------------------

I'd really like to see the credentials comment addressed; otherwise, just some suggestions.

::: configs/developer_config.py
@@ +39,5 @@
> +    "tooltool_cache": os.path.join(LOCAL_WORKDIR, "builds/tooltool_cache"),
> +    "tooltool_cache_path": os.path.join(LOCAL_WORKDIR,
> +                                        "builds/tooltool_cache"),
> +    # Android related
> +    "host_utils_url":

Does this have the exact same content as tegra-host-utils.Linux.1109310.2.zip?

::: mozharness/lib/python/authentication.py
@@ +30,5 @@
> +        https_password = getpass.getpass()
> +
> +        with open(CREDENTIALS_PATH, "w+") as file_handler:
> +            file_handler.write("%s\n" % https_username)
> +            file_handler.write("%s\n" % https_password)

I really don't like this. Is it just for convenience? I'd much prefer to type my password every time, rather than worry about my LDAP password being available in cleartext in a file.

::: mozharness/mozilla/tooltool.py
@@ +13,5 @@
>  
>  
> +TOOLTOOL_PY_URL = \
> +    "https://raw.githubusercontent.com/mozilla/" + \
> +    "build-tooltool/master/tooltool.py"

I appreciate that you are trying to keep the line length reasonable, but in this case I think this would be more readable if the line was not split.

@@ +75,5 @@
>  
> +    def _fetch_tooltool_py(self):
> +        """ Retrieve tooltool.py
> +        """
> +        file_path = os.path.join(os.getcwd(), "tooltool.py")

Consider using something like abs_work_dir, rather than os.getcwd(). That way you know at a glance what location we are talking about.

Comment 35

[Armen [:armenzg]]

Comment on attachment 8555991 [details] [diff] [review]
[mh] allow running Android test jobs locally

Review of attachment 8555991 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozharness/lib/python/authentication.py
@@ +30,5 @@
> +        https_password = getpass.getpass()
> +
> +        with open(CREDENTIALS_PATH, "w+") as file_handler:
> +            file_handler.write("%s\n" % https_username)
> +            file_handler.write("%s\n" % https_password)

Mainly convenience. Specially when trying to run it multiple times.

Mind if we open a separate bug to try to figure out what would work best?

Comment 36

[Armen [:armenzg]]

Attachment: [mh] addressing comments

Addressing issues bar authentication prompting concern.

Comment 37

[Geoff Brown [:gbrown]]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #35)
> Mind if we open a separate bug to try to figure out what would work best?

Sounds good to me.

Comment 38

[Geoff Brown [:gbrown]]

Comment on attachment 8556524 [details] [diff] [review]
[mh] addressing comments

Review of attachment 8556524 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

::: configs/developer_config.py
@@ +41,5 @@
>                                          "builds/tooltool_cache"),
>      # Android related
>      "host_utils_url":
> +        "http://talos-remote.pvt.build.mozilla.org/tegra/"
> +        "tegra-host-utils.Linux.1109310.2.zip"

I want to make sure the right/same-as-production files are downloaded, but I also want to ensure developer availability. I can never seem to access http://talos-remote.pvt.build.mozilla.org, so I'd actually prefer https://secure.pub.build.mozilla.org/tooltool, if the same file is available there.

Comment 39

[Armen [:armenzg]]

Comment on attachment 8556524 [details] [diff] [review]
[mh] addressing comments

Review of attachment 8556524 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/developer_config.py
@@ +41,5 @@
>                                          "builds/tooltool_cache"),
>      # Android related
>      "host_utils_url":
> +        "http://talos-remote.pvt.build.mozilla.org/tegra/"
> +        "tegra-host-utils.Linux.1109310.2.zip"

Silly me. I forgot why I put that there.
I will see who can get me a copy of it.

The tooltool file I uploaded is from the 18th and might be the old one.
I will update the new one and use that instead.
I assume I can land this for now while it works on the CI.

::: mozharness/mozilla/tooltool.py
@@ +12,5 @@
>  }]
>  
>  
> +TOOLTOOL_PY_URL =
> +"https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py"

I don't know how pep8 did not complain about this being a syntax error.
I've fixed it.

Comment 40

[Armen [:armenzg]]

Attachment: [mh] allow running Android test jobs locally

As landed:
https://hg.mozilla.org/build/mozharness/rev/89039ca68621

Carried over r=gbrown.

Comment 41

[Armen [:armenzg]]

(In reply to Geoff Brown [:gbrown] from comment #37)
> (In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from
> comment #35)
> > Mind if we open a separate bug to try to figure out what would work best?
> 
> Sounds good to me.

Filed as bug 1127449.

(In reply to Geoff Brown [:gbrown] from comment #38)
> Comment on attachment 8556524 [details] [diff] [review]
> ::: configs/developer_config.py
> @@ +41,5 @@
> >                                          "builds/tooltool_cache"),
> >      # Android related
> >      "host_utils_url":
> > +        "http://talos-remote.pvt.build.mozilla.org/tegra/"
> > +        "tegra-host-utils.Linux.1109310.2.zip"
> 
> I want to make sure the right/same-as-production files are downloaded, but I
> also want to ensure developer availability. I can never seem to access
> http://talos-remote.pvt.build.mozilla.org, so I'd actually prefer
> https://secure.pub.build.mozilla.org/tooltool, if the same file is available
> there.

Filed as bug 1127442.

Comment 42

[Armen [:armenzg]]

I always do the wrong thing when there are new files:
https://hg.mozilla.org/build/mozharness/rev/04ff04530e9a

At least it was caught before production.

Comment 43

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/mozharness/rev/89039ca68621

Comment 44

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/mozharness/rev/04ff04530e9a