Automated UI testing for iOS apps: UIAutomation + jasmine + jenkins

A couple of months ago I was doing some iOS work at Sensis with regards to the Yellow Pages iPad app + SAPI integration. Rochana Attale and I needed to come up with a way to make it easy to hook up UI Automation acceptance tests written in Jasmine to our Jenkins CI environment.  Please keep in mind at the time the zucchini testing framework wasn’t released which is why we put together our own solution.

And to be fair what we put together has worked out quite well.  It is a bit rough around the edges but at the same time extremely quick to get acceptance tests up and running.

Acceptance tests are critical for any application which is intended to be around for a while, and they are even more important if your intending to make some major changes (such as swapping out a backend API … which is what we did on Yellow iPad). So before we started making any massive changes to our app we wanted to find a way to automate the UI testing of our iOS application and ideally hook it up to jenkins.

Hopefully you find this post useful, as we were surprised as to how quick writing the tests were once you had your environment up and running. At the end of the day its only UI Automation with some BDD love via Jasmine.

Download the example projects

I have created 2 projects, one for writing / running our acceptances tests and the other is the app which our tests will run against.

So go ahead and clone the following two projects from github into the same parent directory:

jasmine-ios-acceptance-tests
Make sure you remember to run git submodule update –init to grab the jasmine-reporters dependency.

jasmine-ios-acceptance-tests-basic-app
Build and run the app to make sure it is running in the simulator correctly.

Running the tests in Instruments

Firstly we will run our acceptance tests in Instruments so you can see how easy it is.

  1. Make sure you have Xcode 4.2 installed as a minimum.
  2. Start Instruments
  3. Select the Automation template and click Choose
  4. Select the Choose Target drop down -> Choose Target -> Choose Target…
  5. A prompt will appear. Browse to the location of the BasicApp.app file and click Choose. For example mine is located at: /Users/shaune/dev/blog/jasmine-ios-acceptance-tests-basic-app/build/Debug-iphonesimulator/BasicApp.app 
  6. Now back to the Instruments window on your left hand panel you should have a section called Scripts under it is a drop down button called Add. Click Add -> Import…
  7. Browse to the location of our ci.js file and click Open. For example mine is located at: /Users/shaune/dev/blog/jasmine-ios-acceptance-tests/ci.js
  8. Now that we are all setup you should be able to click the little red button to launch the simulator and see our tests running.

Running the tests from the command line

Also checked in is a basic shell script that you can use to kick off the acceptance tests from the command line OR from a Jenkins job.

$ ./runTests.sh ci.js “/Users/shaune/dev/blog/jasmine-ios-acceptance-tests-basic-app/build/Debug-iphonesimulator/BasicApp.app”

Hopefully the output you see is: Build Passed

Viewing test results in Jenkins

To add this to Jenkins simply setup a new unix shell job and execute the runtTests shell script with your own test script + app.

If you now look in your jasmine-ios-acceptance-tests directory a new directory called test-reports has hopefully been created with a test-results.xml file.

All you need to do is now add this into Jenkins for your acceptance test job to look for JUnit reports and you will then be able to see your Jasmine + UI Automation acceptance tests reports in Jenkins!

Super simple… but gets the job done

Keep in mind that our solution is quite simple, but it was super quick to integrate with Jenkins and it works. Also I found that writing the Jasmine specs were really quick and using the script recorder in Instruments does half the job for you.

What would I like to change?

I would like to update the jasmine.uiautomation_junit_reporter.js so that it actually writes the test-reports/test-results.xml file instead of simply logging it and having the runTests.shfile grep for the output. This could be done via the UIAHost.performTaskWithPathArgumentsTimeout call that can execute system commands.

Still hungry for other test frameworks?

If you are serious about setting up acceptance tests for your iOS application you should really consider checking out zucchini as well. It came out after we had already setup our acceptance test environment and is quite polished. It gives you the ability to write Gherkin’ish feature files and image comparison for verifying successful scenarios. If you have a UX team on board this will be ideal because you can test down to the pixel!

2 Responses to Automated UI testing for iOS apps: UIAutomation + jasmine + jenkins
  1. Stewart Gleadow Reply

    Thanks Shaun. I just worked through this post and it worked well. I made a small modification to the script to run on device and run with Xcode 4.3, see pull request https://github.com/shaune/jasmine-ios-acceptance-tests/pull/1

    I didn’t know about the JUnit reporter for Jasmine, that’s really handy. Assuming that makes it easy to integrate with other tools, like Jenkins plugins for visualising test output as if it were JUnit?

    I also could only get the simulator mode running if I provided the .app path as an absolute path within derived data. Relative path didn’t work. Copying the .app to a known location also didn’t work, can’t work out why… oh well, I guess I’ll just run from derived data for now.

    • Shaun Reply

      Hey Stewart, cheers for the mod to allow the tests to be executed on device I have merged the changes.

      The jasmine reporters project is pretty cool. For us it was perfect because we were already using ocunit2junit to convert out unit tests to JUnit output, so made sense to do the same with our acceptance tests so we could visualize our test output in Jenkins like your mentioned.

      From what I have found with running instruments from CLI (against the simulator anyways) you must always specify the absolute location of your .app your trying to execute your tests against.

      Thanks for trying out the post :)

Leave a Reply

Your email address will not be published. Please enter your name, email and a comment.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>