Running Calabash Tests on Local Devices
- PDF for offline use
- Related Articles:
Let us know how you feel about this
last updated: 2016-10
This guide will discuss how to run Calabash tests on local devices attached to your computer.
This guide will cover how to run automated tests created with Calabash to physical devices. The process to do so is different between Android and iOS so each platform will be discussed in their own sections below. In general, the process consists of the floowing steps
Ensure the device is enabled for developement – By default, mobile devices are not enabled to support app development. It is necessary to configure or enable developer mode on the device. This step is typically performed only once per device.
Package the Mobile App – Calabash requires that an APK or IPA must be built (and possibly pre-installed on the device).
Set Environment Variables – It may be necessary to provide some information for a test run via environment variables.
Calabash Android and/or Calabash iOS must be installed on the computer running the tests. The device must be connected to the computer running the tests with a USB cable.
Each section may list additional requirements for testing on that platform.
Testing on Android Devices
Before tests can be run on an Android device, the device must be configured for testing.The guides Set Up Device for Development and Debug on Device describe the steps necessary to configure an Android device for local development.
It is necessary to build the APK and install it on the device. Xamarin Test Cloud Agent cannot interact with APK’s that are built using the Shared mono runtime. The Shared mono runtime is enabled by default in DEBUG builds and it is disabled by default in RELEASE builds. If you wish to use a DEBUG build of your APK, ensure that you turn off the Shared mono runtime:
Once the device is configured and the APK is packaged, it is time to run the tests on the device using
$ calabash-android run <APK>
If there is more one device connected to the computer, you can specify which device to run the tests on with the following command line:
$ ADB_DEVICE_ARG=<DEVICE ID> calabash-android run <APK>
You can obtain a list of the devices and emulators using the Android Debug Bridge, as shown in the following snippet:
$ adb devices List of devices attached 0756edf000620ace device
Testing on iOS Devices
It is not possible to run tests on iOS devices using Windows.
Testing on iOS devices is a much more involved process than testing on Android device. Let's first discuss the prerequistes that must be met before testing, and then the actual steps involved with testing.
Requirements for Testing on iOS Devices
This guide is for testing iOS apps using Xcode 8 and calabash-cucumber 0.20.0 or higher. For testing with Xcode 7 and older versions of the calabash-cucumber gem, please consult the deprecated guides for Xcode 7.
It is necessary to connect the device to the Mac using a USB cable, and both the computer and the device must be on the same WiFi network. You can improve your network stability by creating a Wifi network on your Mac.
The most recent released version of Xcode must be installed on the Mac. It is necessary to reboot the Mac if you have recently performed any of these activities:
- installed or updated Xcode
- changed the location of Xcode.app or renamed Xcode.app
- installed a iOS Simulator
The device must be enabled for development. The guide Device Provisioning describes the first steps necessary to set up an iOS device for local development. A fast way to check if the device is enabled for development is to inspect the output of
xcrun instruments -s devices at the command line. If the name of the device appears in the output, then it has been enabled for development.
The device must enable UIAutomation, described in detail below.
Finally the IPA must be installed on the device. This is also described in more detail below.
Installing the IPA
Unlike Calabash Android, Calabash iOS cannot install apps on a physical device. The easiest way to install the app on the device is to use the IDE (Xcode or Xamarin Studio) and have it package and install the IPA for you. If this is not possible, but you have the IPA, then there are two other options:
- Drag and drop the IPA onto the device using the Xcode Device Manager window (Command + Shift + 2 in Xcode).
- Use a third-party command line tool such as ideviceinstaller. ideviceinstaller is available through Homebrew by running
brew install ideviceinstallerat the command line.
Running the Tests
The Calabash tests are run using the
cucumber command from the directory that contains your features folder. You will need to set the
DEVICE_TARGET environment variables:
$ export BUNDLE_ID=com.xamarin.calabash.example.creditcardvalidation $ export DEVICE_ENDPOINT=http://<IP OF DEVICE >:37265 $ export DEVICE_TARGET=<UDID OF DEVICE> $ cucumber
It is also possible to set the environment variables at once, when running cucumber:
$ BUNDLE_ID=com.xamarin.calabash.example.creditcardvalidation \ DEVICE_ENDPOINT=http://< ip >:37265 \ DEVICE_TARGET=< UDID > \ cucumber
Details on how to obtain these environment variables are provided below.
Starting in iOS 8, it is necessary to enable UIAutomation on the device. Go to the devices Settings, and select Developer:
Once in the Developer settings, turn Enable UI Automation:
Collecting and Setting Environment Variables
Calabash requires four pieces of information to launch and interact with an app that is installed on a physical device. This information is typically provided via environment variables that are set before a test is run:
BUNDLE_ID– The bundle identifier of the app to test. It is also known as the CFBundleIdentifier. You can find this in Project Options > Build > iOS Application as the Bundle Identifier:
DEVICE_ENDPOINT– This IP Address off the device that will run the test. You can find the IP address of the device under Settings > WiFi > Network and clicking on the blue I that is next to the network name:
DEVICE_TARGET– This is the UDID of the device that is connected to the computer. The section Identifying Available Simulators describes how to retrieve the device ID for iOS devices.
This guide discussed how to run automated test on physical devices connected to your local workstation.