目录
- APhotoManageradd new scripts3年前
- ActivityDiaryadd bug reporduction scripts for ActivityDiary, AFM, AnkiDroid, and3年前
- AmazeFileManageradd bug reporduction scripts for ActivityDiary, AFM, AnkiDroid, and3年前
- AnkiDroidadd all reproduction scripts3年前
- FirefoxLiteadd all reproduction scripts3年前
- Frostadd new scripts3年前
- MaterialFBookadd new scripts3年前
- Omni-Notesadd new scripts3年前
- Phonographadd all reproduction scripts3年前
- Scarlet-Notesadd all reproduction scripts3年前
- WordPressadd all reproduction scripts3年前
- and-bibleadd bug reporduction scripts for ActivityDiary, AFM, AnkiDroid, and3年前
- collectadd new scripts3年前
- commonsadd new scripts3年前
- documentsadd presentation video3年前
- geohashdroidadd new scripts3年前
- nextcloudMerge branch 'script-jue'3年前
- open-event-attendee-androidadd all reproduction scripts3年前
- openlauncheradd all reproduction scripts3年前
- osmeditor4androidadd all reproduction scripts3年前
- scriptsAdd files via upload3年前
- sunfloweradd all reproduction scripts3年前
- toolsadd more scripts3年前
- INSTALL.txtrevise3年前
- LICENSE.txtrevise3年前
- README.mdadd documents3年前
- REQUIREMENTS.txtrevise3年前
- STATUS.txtUpdate STATUS.txt3年前
- Themis_evaluation_data.xlsxAdd files via upload3年前
邀请码
The Themis Benchmark
Themis is a collection of real-world, reproducible crash bugs (collected from open-source Android apps) and a unified, extensible infrastructure for benchmarking automated GUI testing for Android and beyond.
Publication (Presentation Video)
[1] “Benchmarking Automated GUI Testing for Android against Real-World Bugs“. Ting Su, Jue Wang, Zhendong Su. 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2021)
News
- We are now actively developping an automated analysis tool to understand the effectiveness of GUI testing tools. We will release the tool soon!
- Themis is using by ByteDance’s FastBot to evaluate and improve its bug finding abilities! (Nov. 2021）
- Themis’s paper was accepted to ESEC/FSE’21! (Aug. 2021）
- We released Themis’s dataset and infrastructure (Mar. 2021）
1. Contents of Themis
Themis’s bug dataset
Themis now contains 52 reproducible crash bugs. All these bugs are labeled by the app developers as “critical bugs” (i.e., important bugs), which affected the major app functionalities and the larger percentage of app users.
For each bug, we provide:
The original bug report of the bug
An executable APK (Jacoco-instrumented for coverage collection),
A bug-reproducing script and a video
The stack trace of the bug
Metadata for supporting evaluation (e.g., app login scripts and configuration files used by Themis for code coverage computation)
The app source code w.r.t each bug
List of crash bugs
Themis’s Infrastructure
Themis contains a unified, extensible infrastructure for benchmarking automated GUI testing for Android. Any testing tool can be easily integrated into this infrastructure and deployed on a given machine with one line of command.
List of Supported Tools
The command line for deployment:
Implementation details
The directory structure of Themis is as follows:
2. Instructions for Using Themis
The instructions in this section was used for artifact evaluation. In the artifact evaluation, we run Themis in Virtual Machine with all the required stuffs already installed and prepared. You can follow the instructions in this section to get familar with Themis. You can download the VM image
Themis.ova(15GB) from this link on Google Drive.For using Themis for your own research, we recommend you to build and run Themis on a native machine (see 3. Instructions for Reusing Themis).
Prerequisite
Setup Virtual Machine
Themis.ova(this step may take about five to ten minutes to complete).themisand the password isthemis-benchmark.Getting Started
Take the quick test to get familar with Themis and validate whether it is ready.
Step 1. open a terminal and switch to Themis’s scripts directory
Step 2. run Monkey on one target bug for 10 minutes
Here,
--no-headlessshows the emulator GUI.--avd Android7.1specifies the name of the emulator (which has already been created in the VM).--apk ../ActivityDiary/ActivityDiary-1.1.8-debug-#118.apkspecifies the target bug which isActivityDiary‘s bug#118inv1.1.8.--time 10mallocates 10 minutes for the testing tool to find the bug-o ../monkey-results/specifies the output directory of testing results--monkeyspecifies the testing toolExpected results: you should see (1) an Android emulator is started, (2) the app
ActivityDiaryis installed and started, (3) Monkey is started to test the app, (4) the following sample texts are outputted on the terminal during testing, and (5) the emulator is automatically closed at the end.**click to see the sample output on the terminal of a successful run.**
Step 3. inspect the output files
If Step 2 succeeds, you can see the outputs under
../monkey-results/(i.e.,/home/themis/the-themis-benchmark/monkey-results).How to validate: If you can see all these files and these files are non-empty (use
ls -lto check), the quick test succeeds. Note that the number of coverage data files (e.g.,coverage_1.ec) varies according to the testing time. In practice, Themis notifies an app to dump coverage data every five minutes. Please note that the outuput files of different testing tools may vary (but all the other tools have these similar types of output files likeMonkey).Detailed Instructions
I. The supported tools
Themis now supports and maintains 6 state-of-the-art fully-automated testing tools for Android (see below). These tools can be cloned from Themis’s repositories and are put under
the-themis-benchmark/tools.Monkey: distributed with Android SDKsApe: https://github.com/the-themis-benchmarks/ape-bincombodroid: https://github.com/the-themis-benchmarks/combodroidHumanoid: https://github.com/the-themis-benchmarks/Humanoid, which depends ondroidbot(https://github.com/the-themis-benchmarks/droidbot/tree/themis-branch)Q-testing: https://github.com/the-themis-benchmarks/Q-testingTimeMachine: https://github.com/the-themis-benchmarks/TimeMachineNote that these tools are the modified/enhanced versions of their originals because we coordinated with the authors of these tools to assure correct and rigorous setup (e.g., report the encountered tool bugs to the authors for fixing). We tried our best efforts to minimize the bias and ensure that each tool is at “its best state” in bug finding (see Section 3.2 in the Themis’s paper).
Specifically, we track the tool modifications to facilitate review and validation. We spent slight efforts to integrate
Monkey,Ape,HumanoidandQ-testinginto Themis.Combodroidwas modifled by its author and intergrated into Themis, whileTimeMachinewas modified by us and later verified by its authors (view this commit to check all the modifications/enhancements made inTimeMachine).II. Running the supported tools
** In the following, we take
Monkeyas a tool andActivityDiary-1.1.8-debug-#118.apkas a target bug to illustrate how to replicate the whole evaluation, and how to validate the artifact if you do not have enough resources/time**[Replicate the whole evaluation]:
Step 1. run
MonkeyonActivityDiary-1.1.8-debug-#118.apkfor 6 hours and repeat this process for 5 runs. This step will take 30 hours to finish because of 5 runs of testing on one emulator. We do not recommend to run more than one emulators in the VM because of the limited memory and performance. If you do not have enough resource/time, see the instruction below.Here,
--no-headlessshows the emulator GUI--avd Android7.1specifies the name of the emulator (which has already been created in the VM).--apk ../ActivityDiary/ActivityDiary-1.1.8-debug-#118.apkspecifies the target bug which isActivityDiary‘s bug#118inv1.1.8.-n 1denotes one emulator instance will be created (in practice at most 16 emulators are allowed to run in parallel on one native machine)--repeat 5denotes the testing process will be repeated for 5 runs (these 5 runs will be distributed to the available emulator instances)--time 6hallocates 6 hours for the testing tool to find the bug-o ../monkey-results/specifies the output directory of testing results--monkeyspecifies the testing tool[Validate the artifact if you do not have enough resources/time: run 1-2 tools on 1-2 bugs at your will with limited testing time]:
For example, in Step 1, we recommend you to shorten the testing time (e.g., use
--time 1hfor 1 hour or--time 30mfor 30 minutes). Thus, you can use the following command (this step will take 2 hours to finish because of 2 runs of testing on one emulator).Step 2. When the testing terminates, you can inspect whether the target bug was found or not in each run, how long does it take to find the bug, and how many times the bug was found by using the command below.
Here,
--app ActivityDiaryspecifies the target app (You can omit this option to check all the tested apps and their bugs).--id \#118specifies the target bug id (You can omit this option to check all the target bugs of appActivityDiary).--simpleoutputs the checking result to the terminal for quick check (You can substitute--simplewith--csv FILE_PATHto output the checking results into a CSV file).-hto see the detailed list of command options.An example output could be (In this case, the target bug,
ActivityDiary‘s#118, was not found by Moneky in all the five runs):**click to see the sample output.**
Another example output could be (In this case, the target bug,
AnkiDroid‘s#4451, was found by 1 time after running Monkey for 55 minutes in one run):**click to see the sample output.**
Notes
(1) You can substitute
--monkeywith--apeor--comboin the command line in Step 1 to directly run the corresponding tool. You may need to change the output directory-o ../monkey-results/to a distinct directory, e.g.,-o ../ape-results. You can follow the similar steps described in Step 2 to inspect whether the target bug was found or not and the related info.(2) Specifically, for
humanoid, before running, you need to setup the specific virtual Python environment of Humanoid. Open a terminal, and run:Open a new terminal, run
Humanoid(which internally runsdroidbot) on the emulatorAndroid7.1_Humanoid(with specific screen size):Remember to execute
deactivatewhen you finish the running to exit from the specific virtual Python environment.(3) For
Q-testing, before running, you need to setup the specific virtual Python environment ofQ-testing. Open a terminal, and run:And then, run the command line:
Remember to execute
deactivatewhen you finish the running to exit from the specific virtual Python environment.(4) For
TimeMachine, we cannot buildTimeMachinewithin this VM becauseTimeMachinetests apps by using Docker on which it runs another layer of VirtualBox. Thus, we strongly recommend to buildTimeMachineon the native machines if you want to evaluate it (see the instructions in its repo: https://github.com/the-themis-benchmarks/TimeMachine).(5) If the app under test requires user login (see the Table of bug dataset), you should specify the login script. Themis will call the login script before testing. For example, if we run
Monkeyon../nextcloud/nextcloud-#5173.apkwhich requires user login, the command line should be:Here,
--login ../nextcloud/login-#5173.pyspecifies the login script (which will be executed before GUI testing)3. Instructions for Reusing Themis
Build and Use Themis from Scratch
In practice, we strongly recommend the users to setup our artifact on local native machines or remote servers rather than virtual machines to ensure (1) the optimal testing performance and (2) evaluation efficiency. Thus, we provide the instructions to setup Themis from scratch.
Prerequisite
TimeMachine)Steps
Android7.1with SDK version 7.1 (API level 25), X86 ABI image and Google APIs:In our evaluation, we set an emulator with 2GB RAM, 1GB SdCard, 1GB internal storage and 256MB heap size (the file for modification usually is:
~/.android/avd/Android7.1.avd/config.ini)--no-headlesswhich turns off the emulator GUI.Welcome your contribution!
Extend Themis for our research community
1. Add new crash bugs into Themis
Take
ActivityDiary-1.1.8-debug-#118.apkas an example, the basic steps to add such a bug into Themis’s dataset include:ActivityDiary/ActivityDiary-1.1.8-debug-#118.apk, whereActivityDiaryis the app name,1.1.8is the code version,#118is the original issue id.)ActivityDiary/crash_stack_#118.txt)uiautomator2and record a bug-triggering video (i.e.,ActivityDiary/script-#118.pyandActivityDiary/video-#118.mp4)TimeMachine) which describes its class files and source files (i.e.,ActivityDiary/class_files.json)The basic steps to add such a bug into Themis’s infrastructure include:
scripts/check_crash.py, one should add the app name into listALL_APPSand the crash signature (i.e., the crash type info and the partial crash trace related to the app itself) into dictapp_crash_data.In the future, we plan to (1) add the crash bugs from other existing benchmarks, (2) add crash bugs with different levels of severity rather than only critical ones.
2. Add new testing tools into Themis
Take
Monkeyas an example, the basic steps to add a new tool into Themis’s infrastructure include:scripts/run_monkey.sh) which defines (1) the concrete command line of invoking the tool, (2) the outputs of the tool, and (3) the tool-specific configurations before runningscripts/themis.py(see functionrun_monkey)scripts/check_crash.py.In fact, we already have successfully integrated FastBot, an industrial testing tool developed by ByteDance, into Themis. See the script for supporting FastBot.
3. Optimize and enhance existing supported tools
In Themis’s paper, Section 4.2 and 4.3 point out many optimization opportunities and future research for improving existing testing tools. By using Themis,
4. Coverage profiling and analysis
Themis now supports coverage profiling by running
The main usage is:
By leveraging the results, one can inspect the detailed coverage report generated by Jacoco.
5. Oher research purposes
Themis can also benefit other research (e.g., fault localization, program repair, etc.)
Main maintainers/contributors
Ting Su, East China Normal University, China
Jue Wang, Nanjing University, China
Zhendong Su, ETH Zurich, Switzerland
We appreciate the contributions from:
Enze Ma, Beijing Forestry University, China
Weigang He, East China Normal University, China
Shan Huang, East China Normal University, China
We welcome any feedback or questions on Themis. Free feel to share your ideas, open issues or pull requests. We are actively maintaining Themis to benefit our community.