Mobile Testing: Out of the Darkness

“Hang on a second… how do we test mobile?”

Blank. Other testers stop their tasks and raise their heads up, staring at each other and frowning, as if they were trying to find the solution to a 3,000-year old history problem, like trying to discover the origins of the Pyramids. The Great Mobile Lord has cast his confusion.

This confusion is a major risk to software quality, and this is why it is time to cast some light over these mysteries and show you how easy and accessible mobile testing is. So please join me in diving into the darkness of mobile testing.

The Twin Brothers

Mobile solutions are composed of 2 parts, which will have to be considered for your investigation: mobile websites and native mobile applications. This is where the first ambiguity usually occurs: what are these different terminologies?

Mobile Website:

Think of a mobile website as a website that you access through a browser on your mobile device, as you would access a regular website through a browser on your desktop computer or your laptop.

”Mmm, OK, but then do I have to enter a different URL to display the mobile website? Because if I go to on my mobile, I’ll probably see the regular website on my tiny screen”.

Not at all! And this is where the darkness fades out: the mobile website and desktop website are the same web application, and the latter will automatically display one or the other depending on your device screen size. More specifically, your browser, acting on your behalf as your User Agent, will send the device type to the server in the header of the request, which should deduce which presentation layout to display.

Native Mobile Applications:

People who are native of a country speak a language specific to that country. In the same way, think of a native mobile application as an application designed and developed to speak/interact with a specific mobile Operating System platform. They are coded with a language specific to that system, and therefore follow particular rules, including for example memory management, menu interaction or device permissions for feature execution.

These are the apps that you can download for example via the Apple Store and the Google Play Store.

On the contrary, mobile websites use the web technology languages which are understood by various platforms.

Now that we have identified these two mobile shadows, let’s find out how to tackle the testing.

Will my story affect mobile?

This is the number one question at the box-office, as the answer will determine your test efforts and test strategy. As we have just seen above, the investigation will be comprised of two parts: assess the impact on the mobile website and the impact on the native apps.

Mobile website:

Any code change on the web app means a high chances of making the Mobile Lord unhappy: the questions that a tester can ask to make a risk analysis are:

1. Does the affected page have an equivalent mobile version?

Not all pages of a website both have desktop and mobile version: If you request a page on a mobile device which only has a desktop version, the server will serve you that unique page. If you can answer yes to that question, then it is worth considering question 2. If not, you can jump to question 3.

2. Does the affected feature also exist on Mobile?

Having both versions of the same page on desktop & mobile does not imply that features are identical; this being mostly due to the constraint of a small screen device which makes the feature selection for mobile crucial. If the feature exists, its functional & technical behaviour might also differ from desktop; You will have to study how it behaves to determine how it is affecting your testing.

3. Are the resources affected by the code change also used on the mobile website?

The following question can be considered as a mandatory step, no matter what is the outcome of the two previous ones. Whether you have a mobile version of the page or not, you will need to discover if the resources that are modified/removed/added by the developers for the need of the story are used by the mobile website: this can include text, css layout properties, scripts, etc. As an example, a script can be reused by several different pages, triggering a risk of regression on other mobile pages that no one may have thought about.

4. Are there other deadly traps I should be aware of?

Of course, these previous three questions only cover the most common examples, based on experience. Your investigation and your questioning, combined with your growing experience, will lead you to uncover other deadly traps, such as web services and content fields being reused by both desktop and mobile, and any new imagery requiring its own specific version for iOS device with Retina displays (Retina display started from the iPhone4 and iPad3 models, not including iPad mini).

Native mobile applications:

Due to their programming language specifics, most parts of our native apps code won’t be affected by the changes to the web app, which is a good news as the amount of potential collateral damage is strongly reduced.

However, like a certain movie demonstrated, a dormant “He-who-must-not-be-named” should still raise suspicion. Let’s have a look at the questions that you will need to consider:

1. Do the changes affect an API?

APIs are a key asset in the mobile world: they serve enough data to have critical features accessible to the user, and at the same time limit the quantity of data available: the latter restrains from the temptation of overloading an app and making it unusable with too many features, the ease-of-access and smaller screen size being two major constraints.

Several clients can use the same API, which is where the deadly trap lies : modifying a web service means that all the clients using the latter need to be tested – that includes all the mobile clients on different mobile OS platforms such as Android or iOS. The changes could include addition/update/removal of parameters, modification of a URL, or could affect the way the data is rendered (for example, displaying a decimal number instead of an integer, or using a JSON format instead of a XML). However, that risk of regression should be greatly reduced as the APIs are developed following backwards compatibility guidelines, which means that a newer version should equally work with an older version of the same client.

2. Do the changes affect the part of the web app used by native apps?

Depending on the technical infrastructure available in a company, the development of some features might be limited. In order to offset that drawback, the native apps can reuse some of the existing solutions from the web application, which means that a change on the latter can have a side-effect within the native app: an additional test analysis needs to be carried out to know the amount of regression tests needed.

What do I need in order to do mobile testing in my team?

Now that we have investigated if our stories require testing on mobile, and that the darkness has become lighter, let’s use our magic tools to make it fade away altogether.

Mobile website:

As mentioned previously, the mobile website is simply a different layout and presentation of the web app depending on the user’s device. Therefore, mobile website testing is accessible to anyone in any team. It is strongly recommended to test on a real mobile device: indeed, the behaviour of mobile User Agent on a desktop browser is very far from being representative; Using a mobile simulator/emulator might be effective but only in a limited set of test cases which testers learn with the experience, and depending on the available framework for each mobile OS.

All you will need to do is allow budget to order mobile test devices, which should be the most representative from the customers. You also need to order enough test devices to not obstruct the progression of the testing, and to not represent a risk to the project – if there are two testers in the team testing different features on Mobile, make sure that you have at least two devices.

Mobile native apps

The mobile website is not difficult to set up on a test environment, and that is a very good news! But what about his evil twin, the native apps, which you will also have to provide testing for? Well, let’s say the Mobile Lord has definitely not made things easy for us! In order to get the native app on a test device pointing to the correct test environment, you will need to consider two different steps.

1. Build the app onto a test device

Building the software onto real device is the most time-consuming; A native application requires a specific framework depending on the mobile Operating System.

On Android, it can be straightforward: ask the developers to send you the build as an .apk file, transfer the file on the Android device and simply open it to install the build. This however does not allow you to control your builds, and you might want to consider setting up the Android SDK for that purpose, without any cost.

On the contrary, iOS has basic free tools, including the Xcode IDE, but the free options will limit you to build and run the software on a simulator. You will need to enroll in the Apple developer program by paying a membership, creating a developer account and requiring a developer certificate for each team member in order to do more. If you enroll in a standard program for example, the yearly membership will allow the team to publish the native apps on the Apple Store, and will allow the use of up to 100 iOS test devices per year. Once a team member has a valid certificate approved, he will also need to be added to an ad-hoc or development provisioning profile, whose purpose is to tie the devices to the team. Then you just need to load the source code of the app into the Xcode IDE and connect your test device to install the build.

(Please note that these OS mentioned are not exhaustive and if you need to test on a different mobile OS, you will need the relevant framework).

2. Configure the app to use the test environment.

This part should be quick, if you know how to configure the environment used by the app. You will probably have to ask your developers at first to find out which files to modify, and to improve the testability and avoid multiple changes in different files, it is strongly recommended to ask them to put the configuration in only one file – ideally within 1 line of code. To facilitate your task, you can use an Integrated Development Environment, such as Xcode for iOS, which gives you a nice and usable interface to do these changes.

What if I find a bug in the mobile clients, what is the process for releasing bug fixes?

Since mobile and desktop websites are 2 presentations of the same web application, bug fixes follow the same process and will be released live on the normal release frequency.

Releasing a new version of a native app, however, is completely different. The updates are uploaded to each Store by the account administrators, and the delay between the upload and the approval depends on each platform: uploading a new version of the Android app to the Google Play Store is very quick and requires no validation from the Store, whereas publishing a new iOS app version can take up to 10 days to pass the approval from Apple, providing that you have followed all their guidelines. This approval time should never be neglected, and should be accordingly reflected in the project planning.


Mobile testing is very often misunderstood and seems out of reach of any team who are not familiar with it, which dramatically increases the risk on the software quality. The mobile clients are in fact always embedded within the global software solution and have an impact on the architecture/infrastructure strategy and for this reason, the changes coming from or affecting them can not be ignored and must be evaluated.

This article, I hope, provided you with enough useful information to understand the mechanics of mobile website and mobile native applications, in order to increase the testing coverage and quality of a release. Mobile testing is not a privilege limited to a specific team, but more a badge that every tester should carry and add to their testing skills. It expands a tester’s vision beyond web applications, getting them more familiar with the concepts of data and user journey, two critical features in that world.

Mobile testing of native apps also has its own challenges in terms of frameworks and release processes, which should be highlighted among the project risks, and should be considered in parallel with other projects. Your vision of mobile tester will contribute to assess these risks, and also inform in time, and with accuracy, the other teams affected by these changes.

Print Friendly

Leave a Reply