Xamarin Test Cloud is now part of Visual Studio App Center! Get Started.

Submitting Calabash Tests to Xamarin Test Cloud

PDF for offline use
Related Articles:

Let us know how you feel about this

Translation Quality


last updated: 2015-11

Testing on physical devices is an important part of creating a mobile application. Calabash allows developers to create automated tests for mobile applications and Xamarin Test Cloud provides a service that will run these tests on hundreds of physical devices. This guide will discuss how to submit tests to Xamarin Test Cloud.


This guide discusses how to submit your Calabash tests to Xamarin Test Cloud.


It is assumed that Calabash tests have been written and are running in your local environment, and that you have the xamarin-test-cloud gem installed.

Internet access to the Test Cloud web servers is required. Firewalls should be configured to allow traffic to the Test Cloud servers at the following IP addresses:


The Test Cloud servers require port 80 and 443.

It is assumed that a Test Cloud project has been created, your account has been added to that team, and you have the Team API Key. Please see the Organizations & Teams document for more details.

Android applications must request the internet permission in AndroidManifest.XML.

Xamarin Test Cloud uses Ruby 2.3.1p112. For maximum compatibility with tests run locally and tests run in Test Cloud it is recommended that you use the same version while developing tests.

Submitting Calabash Tests

Tests are submitted using the command line tool test-cloud, which is distributed via the xamarin-test-cloud gem. test-cloud is a multi-purpose utility that will upload iOS and Android tests. It works on OS X and Windows.

The following table describes some of the more important command line parameters for test-cloud. It is not a complete list of all parameters.

Parameter Description
--devices=<DEVICES> The device ID that was provided in the Test Cloud Upload dialog.
--user=<EMAIL> The e-mail address of the team member submitting the tests.
--config=<CONFIG> The path to the Cucumber configuration file. i.e. **./config/cucumber.yml**
--profile=<PROFILE> This is the Cucumber profile to use when running the tests. Optional, but recommended.
--dsym=<DSYM-FILE> iOS only. Will upload the dSYM files along with the application and tests. This will allow for more detail in the log files. Optional

It is important that all commands be run from the project directory that holds the config and features directory. As an example, consider the following screenshot of the TaskyPro-Calabash sample project:

Calabash scripts may only reference local gems that are installed on your machine. It is not possible to use Gems that are referenced with :github=> or :path=>. At the time of submission, test-cloud will package up those gems and submit them along with the test scripts to Xamarin Test Cloud.

As a part of the submission process, test-cloud will run some quick validation checks on the project before submitting the test to Xamarin Test Cloud. It is also possible to use Cucumber's dry run feature to validate that the tests are correct using either of the two following command line snippets in your project directory:

cucumber --dry-run


cucumber --profile <your XTC profile> --config <path to cucumber.yml> --dry-run

Cucumber will not load the features/support/env.rb class when the --dry-run parameter is specified. This may cause problems for more complex test projects that rely on this feature.

Note: Xamarin Test Cloud will terminate test runs that take longer than 180 minutes.

Creating A Gemfile

If you are using Bundler to maintain the gems in use by your scripts, then you may ignore this section as you already have a Gemfile. If you are not using Bundler, then you should create a Gemfile along with your tests. A Gemfile is a text file that tells Test Cloud what version of the gems your project uses.

Create a text file in the directory that holds your Calabash features folder, and name it Gemfile. A sample Gemfile is shown below:

source 'https://rubygems.org'
gem 'calabash-cucumber', '0.16.3'
gem 'calabash-android', '0.5.14'
gem 'xamarin-test-cloud', '1.1.2'

The gems and their version number listed should match the version number of the gems you have installed.

Submitting iOS Applications to Test Cloud

iOS tests and applications are submitted at the command line using the test-cloud application:

test-cloud submit <PATH-TO-IPA> <TEST-CLOUD-API-KEY> --user=<EMAIL> --devices <DEVICE-ID> --profile=<PROFILE> --dsym=<PATH-TO-DSYM>

Submitting Android Applications to Test Cloud

Submitting Android applications to Test Cloud requires the following two steps:

  1. Build the Android Test Server for the application. If you do not build the test server for the application, test-cloud will not be able to submit the tests to Test Cloud. You can build the Android Test Server using the following command:

    $ calabash-android build <PATH TO YOUR APK>

    If this step is omitted, test-cloud will halt the upload and display an error message.

  2. Run test-cloud to upload the application and tests. The following snippet is an example of a command line that should be used to submit tests to Test Cloud:

    $ test-cloud submit <PATH-TO-APK <TEST-CLOUD-API-KEY> --user=<EMAIL> --devices <DEVICE-ID> --profile=<PROFILE>

Signing Android APK's

An Android APK must be signed before test-cloud will upload the tests to Xamarin Test Cloud. test-cloud will use it’s own keystore for resigning the APK and the test server APK. It is possible to change the default keystore used for signing these two APK’s by running calabash-android setup as shown below:

$ calabash-android setup
Please enter keystore information to use a custom keystore instead of the default
Please enter keystore location
Please enter the password for the keystore
Please enter the alias
Saved your settings to calabash_settings. You can edit the settings manually or run this setup script again
Note: The calabash-settings file contains confidential information, such as the keystore password, and for this reason should not be included in source code control.


This guide discussed how to use test-cloud to upload Calabash tests and applications to Xamarin Test Cloud for testing.

Xamarin Workbook

If it's not already installed, install the Xamarin Workbooks app first. The workbook file should download automatically, but if it doesn't, just click to start the workbook download manually.