Bug Blog

Simulate testing with emulators

iOS Emulators provide a simulated test environment without the overhead of purchasing and managing multiple devices and you’ve likely been using them for several years, but are they effective enough to be useful? Do they find defects that affect the end users experience?

Emulators are useful for testing application behaviour that is not device or carrier specific early in the development cycle. However, their effectiveness for testing beyond the basics is suspect. Relying too much on emulators for testing is risky. It’s difficult to execute all the testing types needed for a new iOS release. Consider doing a risk analysis on your testing types - basic emulation (smoke testing), performance, functional, load and then decide on the method used for each.  

Automating functional testing

The advantage of automated testing tools is speed and regression testing repeatability.

A few disadvantages are their dependency on the platform so duplicate tests may be needed for each device or platform type. Additionally, automated tests need tuning to get timings correct. Most importantly, automated test maintenance is significant when application code changes frequently. The maintenance effort involved is the leading killer of automated testing projects. 

Another consideration is ensuring automated tests have valid verification points included.  Otherwise, you may be running the exact same test multiple times which wastes valuable time and fails to discover defects.

Automated testing is stronger and less frail when built at the unit testing level rather than the GUI level. GUI level automated tests frequently produce false failures when a software application changes frequently as they require constant maintenance.  If your mobile apps change frequently, plan on how to handle test maintenance. Both emulators and automated tests are useful as complements to manual and device testing, but not as a total replacements even when testing time is limited.