A couple of months ago I was doing some iOS work at Sensis with regards to the Yellow Pages iPad app + SAPI integration. Rochana Attale and I needed to come up with a way to make it easy to hook up UI Automation acceptance tests written in Jasmine to our Jenkins CI environment. Please keep in mind at the time the zucchini testing framework wasn’t released which is why we put together our own solution.
And to be fair what we put together has worked out quite well. It is a bit rough around the edges but at the same time extremely quick to get acceptance tests up and running.
Acceptance tests are critical for any application which is intended to be around for a while, and they are even more important if your intending to make some major changes (such as swapping out a backend API … which is what we did on Yellow iPad). So before we started making any massive changes to our app we wanted to find a way to automate the UI testing of our iOS application and ideally hook it up to jenkins.
Hopefully you find this post useful, as we were surprised as to how quick writing the tests were once you had your environment up and running. At the end of the day its only UI Automation with some BDD love via Jasmine.
Download the example projects
I have created 2 projects, one for writing / running our acceptances tests and the other is the app which our tests will run against.
So go ahead and clone the following two projects from github into the same parent directory:
Make sure you remember to run git submodule update –init to grab the jasmine-reporters dependency.
Build and run the app to make sure it is running in the simulator correctly.
Running the tests in Instruments
Firstly we will run our acceptance tests in Instruments so you can see how easy it is.
- Make sure you have Xcode 4.2 installed as a minimum.
- Start Instruments
- Select the Automation template and click Choose
- Select the Choose Target drop down -> Choose Target -> Choose Target…
- A prompt will appear. Browse to the location of the BasicApp.app file and click Choose. For example mine is located at: /Users/shaune/dev/blog/jasmine-ios-acceptance-tests-basic-app/build/Debug-iphonesimulator/BasicApp.app
- Now back to the Instruments window on your left hand panel you should have a section called Scripts under it is a drop down button called Add. Click Add -> Import…
- Browse to the location of our ci.js file and click Open. For example mine is located at: /Users/shaune/dev/blog/jasmine-ios-acceptance-tests/ci.js
- Now that we are all setup you should be able to click the little red button to launch the simulator and see our tests running.
Running the tests from the command line
Also checked in is a basic shell script that you can use to kick off the acceptance tests from the command line OR from a Jenkins job.
$ ./runTests.sh ci.js “/Users/shaune/dev/blog/jasmine-ios-acceptance-tests-basic-app/build/Debug-iphonesimulator/BasicApp.app”
Hopefully the output you see is: Build Passed
Viewing test results in Jenkins
To add this to Jenkins simply setup a new unix shell job and execute the runtTests shell script with your own test script + app.
If you now look in your jasmine-ios-acceptance-tests directory a new directory called test-reports has hopefully been created with a test-results.xml file.
All you need to do is now add this into Jenkins for your acceptance test job to look for JUnit reports and you will then be able to see your Jasmine + UI Automation acceptance tests reports in Jenkins!
Super simple… but gets the job done
Keep in mind that our solution is quite simple, but it was super quick to integrate with Jenkins and it works. Also I found that writing the Jasmine specs were really quick and using the script recorder in Instruments does half the job for you.
What would I like to change?
I would like to update the jasmine.uiautomation_junit_reporter.js so that it actually writes the test-reports/test-results.xml file instead of simply logging it and having the runTests.shfile grep for the output. This could be done via the UIAHost.performTaskWithPathArgumentsTimeout call that can execute system commands.
Still hungry for other test frameworks?
If you are serious about setting up acceptance tests for your iOS application you should really consider checking out zucchini as well. It came out after we had already setup our acceptance test environment and is quite polished. It gives you the ability to write Gherkin’ish feature files and image comparison for verifying successful scenarios. If you have a UX team on board this will be ideal because you can test down to the pixel!