Facebook has developed a testing library called "screenshot-tests-for-android" [1] (yeah, I know :P) that has similar functionality, but has been built for reliability on Android. Notably:
- It runs in a JUnit environment where you directly open inflate a View to be tested – i.e. no Activity lifecycles, or click event handling to struggle with (unclear how views backed by Adapters are rendered) - `gradle recordMode screenshotTests` will run all of the tests - `gradle verifyMode screenshotTests` will run the tests and compare the results to ^, alerting you of any failures. This is the function intended to be run in a Continuous Integration environment. - Gradle integration I filed [2] but haven't had a chance to dig in further yet. As for running this on different devices, it should be pretty cheap to run this on a different variety of emulators (as compared to anything dealing with Gecko profiles) or, since it works on JUnit & gradle, we can probably get it running on something like Google Cloud Test Lab. - Mike (:mcomella) [1]: https://facebook.github.io/screenshot-tests-for-android [2]: https://bugzilla.mozilla.org/show_bug.cgi?id=1232863 On Fri, Feb 12, 2016 at 5:29 AM, Margaret Leibovic <mleibo...@mozilla.com> wrote: > This pretty cool, and it sounds similar to the work Emily has been doing > to generate iOS screenshots. I wonder if it would make sense to investigate > doing something similar for Android as well. > > For Android, it would be really cool to get screenshots on different > Android device configurations. I wonder if we could do that with different > emulator versions. > > Margaret > > > ---------- Forwarded message ---------- > From: Matthew N. <mattn+firefox-...@mozilla.com> > Date: Fri, Feb 12, 2016 at 12:10 AM > Subject: Firefox screenshots can now be easily captured and compared in > automation > To: firefox-dev <firefox-...@mozilla.org>, dev-developer-tools < > dev-developer-to...@lists.mozilla.org> > > > Back in 2013, during Australis[1] development, I created a tool called > mozscreenshots. The purpose of this tool was to help detect UI > regressions and make it easier to review how the UI looks in various > configurations on each of our supported platforms. The main hindrance > to its regular usage was that it required developers and designers to > install it on each machine, tying it up while the images were > captured. > > The great news is that mozscreenshots is now running in automation on > Nightlies and on-demand for try pushes, making it much easier to > capture images. Comparing the captured images to a reference/base > version can now be done via a web interface[2] which means the images > don’t need to be downloaded for review. > > mozscreenshots has already detected two issues[3][4] during its first > week in automation, and I hope that this tool will improve the quality > of desktop Firefox by surfacing UI issues sooner. You can read about > it at > https://developer.mozilla.org/en-US/docs/Mozilla/QA/Browser_screenshots, > but here’s a quick start guide: > > Change detection via Nightly captures > In order to detect UI regressions, screenshots of some sets run on > every Nightly. These are compared to the previous day’s Nightly, using > the compare_screenshots tool[5] (which uses ImageMagick’s `compare`). > For now I’m manually kicking this off each day but soon I hope to > automate this and have it send an email to interested parties for > investigation. > > Currently, only one run of > “TabsInTitlebar,Tabs,WindowSize,Toolbars,LightweightThemes” occurs on > Nightlies. I plan to add runs with the DevEdition theme, DevTools, and > Preferences shortly, as the code for those are already written. > > Capturing on Try > You can request screenshots be captured on a Try push for UI review or > comparison to a know-good base by requesting the > “mochitest-browser-screenshots” test job. You can specify what you > would like captured by setting the MOZSCREENSHOTS_SETS environment > variable with a comma-separated list of configurations like so: > > try: -b o -p linux,linux64,macosx64,win32,win64 -t none > -u mochitest-browser-screenshots[Ubuntu,10.6,10.10,Windows XP,Windows > 7,Windows 8] > --setenv MOZSCREENSHOTS_SETS=TabsInTitlebar,WindowSize,LightweightThemes > > Note that the job is currently Tier 3 and “excluded” on TreeHerder so > you will need to toggle both of those filters to see the jobs there > with the symbol: “M[Tier-3] (ss)”[6]. Unlike Nightlies, Try pushes > won’t capture any images by default if MOZSCREENSHOTS_SETS isn’t > specified. This avoids capturing images when developers request all > mochitest runs, but don’t really want the screenshots. The capturing > is implemented as a mochitest-bc test[7] in the “screenshots” > subsuite, meaning configurations can use things like BrowserTestUtils > and such. See fetch_screenshots[8] if you would like to download the > captured images. > > Comparing screenshots > The simplest way to compare images is via the web UI at > http://screenshots.mattn.ca/compare/ (Example: http://bit.ly/1Qv4uWD > ). Simply provide the project and revision of a base push with images, > like the Nightly rev. from about:buildconfig, and your new revision, > like a try push with some patches to review. > > In the background, the images are fetched from automation via > fetch_screenshots[8], and then compared using compare_screenshots[5] > with the output displayed on the page. The first comparison for a pair > of revisions can take several minutes, as around one thousand (5 > platforms x 2 revisions x 100 screenshots) images need to be > downloaded and compared for the default set of screenshots. The > results are cached, so subsequent comparisons for the same revision > are much faster. > > There are a lot of opportunities for this tool (e.g. pulse > integration, notifications, simplified bug filing based on > differences, etc.), and I hope to continue to improve the workflow. > Please file bugs on the capturing infrastructure blocking the > “mozscreenshots” meta bug[9] (1169179) and bugs related to the web UI, > comparison and fetching at > https://github.com/mnoorenberghe/mozscreenshots/issues. See also the > list of issues and ideas at > https://public.etherpad-mozilla.org/p/mozscreenshots which haven’t > been filed yet. > > Thank you to everyone who has helped with getting mozscreenshots to > this point, specifically Felipe Gomes, Armen Zambrano Gasparnian, Joel > Maher, Brian Grinstead, Kit Cambridge, and Justin Dolske. > > Cheers, > Matthew Noorenberghe (:MattN) > > [1] Australis was the code name of the Firefox UI redesign that > launched April 29, 2014. > [2] http://screenshots.mattn.ca/compare/ > [3] https://bugzilla.mozilla.org/show_bug.cgi?id=1246316 > [4] > http://screenshots.mattn.ca/comparisons/try/dc5a8fa40cf5/try/288bdb0dd9b4/windows8-64/6_jsdebugger.png > [5] > https://developer.mozilla.org/en-US/docs/Mozilla/QA/Browser_screenshots#Command_line > [6] > https://treeherder.mozilla.org/#/jobs?repo=try&revision=bcd8a2bed2d4&filter-tier=1&filter-tier=2&filter-tier=3&exclusion_profile=false > [7] > https://dxr.mozilla.org/mozilla-central/source/browser/tools/mozscreenshots/browser_screenshots.js > [8] > https://developer.mozilla.org/en-US/docs/Mozilla/QA/Browser_screenshots#Fetching_from_automation > [9] https://bugzilla.mozilla.org/show_bug.cgi?id=mozscreenshots > > > _______________________________________________ > mobile-firefox-dev mailing list > mobile-firefox-dev@mozilla.org > https://mail.mozilla.org/listinfo/mobile-firefox-dev > >
_______________________________________________ mobile-firefox-dev mailing list mobile-firefox-dev@mozilla.org https://mail.mozilla.org/listinfo/mobile-firefox-dev
