As opposed to desktop computers, there is a specific framework that applies to mobile devices such as cells or smartphones: There is not only a smaller screen size; mobile services and applications (apps) are also frequently used in a completely different context. While office users have high band-width internet access and a perfectly balanced workstation environment, users of mobile applications have to cope with plenty of disadvantageous factors. That is why there are extended requirements for mobile applications, such as:
- Smartphones are exposed to many environmental influences: temperature, humidity and wetness, dust and filth or mechanical demands. Still it is supposed to work impeccably.
- Users do not want to deal with complex features: A down-to-earth handling of both the smartphone and the apps is a basic requirement ensuring user acceptance and evading distractions from the daily routine and the environment.
- Selecting the features is expected to be straightforward: For this, it takes keys large enough to avoid operating errors.
- Functionality should perform in a stable and accurate manner, and be likewise smart: Instead of inquiring many parameters with the user, the mobile application should obtain these details by retrieving them from GPS coordinates, environment parameters collected by sensor technology and by managing user profiles upon feature call. Thus, a complex user input can be avoided.
- A good readability of information should be ensured even with difficult lighting conditions, and it should not wear the user out. This important feature is realized for example by high-contrast displays and automatic contrast adjustments when the lighting conditions change. In line with the application design, both the colors and the object arrangement of data and info forms are to be chosen in such a way that the information can be well-captured by the user even under difficult lighting conditions.
- Market requirements are on the rise. While essential applications were able to trigger enthusiasm in the beginning of the smartphone era, these signs have also considerably changed in this regard by now. Today, it is understood that the range of features of mobile applications are available in a flawless and stable manner. This means that software producers are to develop their smartphone applications for the three major platforms – iOS, Android and Phone
- If they intend to dominate this appealing and seminal market segment, the producers are supposed to provide these platforms with the same range and in the very same excellent quality. User acceptance analysis of the apps within the respective stores of the aforementioned technology platforms visualize the trend that the customer focus lays on the first two apps within a user segment. Software companies planning to establish and develop their market position in this segment are advised to realize effective quality management in the long run, aimed at meeting user expectations (see also the box Quality Features) as well as possible. Albeit position 4 or 5 represents a good standing in terms of the app popularity ranking of one segment, for software companies this might be already unviable from the economical point of view and challenging the software’s advancement.
Within the mobile environment, apps are subject to different statutes than for instance software applications destined for office use. Please find below the 10 most important quality aspects representing the users’ decision whether an app is chosen, installed and used or not. If they intend to follow these quality features in particular, mobile applications should meet the respective market requirements in order to fully realize the following aims:
- Accuracy: Delivering correct or stipulated results or impressions, for example the required accuracy of calculated data.
- Stability: Low probability that unexpected impressions occur when altering the environment, software or user interaction.
- Operability: Low effort for the user to operate the application.
- Security: Ability to impede unauthorized access to programs and memory both accidentally and intentionally.
- Maturity: Low failure frequency caused by error statuses.
- Fault-Tolerance: Ability to maintain a specified level of performance when bugs occur or your specified interface has not been complied with.
- Understandability: Effort for the user to understand both the concept and the application.
- Resource Behavior: Resource-saving use of necessary tools (like CPU time and RAM) when carrying out features.
- Time Behavior: Brief response and process times and excellent performance during the execution of features.
- Installability: Easy to install considering the defined environment. This also applies to migrations.
Testing Apps is Different – And Yet the Same!
As the specifications for mobile applications differ significantly from office applications, this finding is reflected within the composition of the test. The test concept should therefore deal with the following topics:
-
Usability tests i. a. comprising the topics:
a. Handling
b. Data collection, appraisal and presentation
c. Accessibility - Mobility tests i.a. comprising the topics:
a. Connectivity
b. Environmental influences
c. Sensor technology - Functional tests i. a. comprising the topics:
a. Accuracy
b. Stability
c. Robustness against operating errors - Security tests i. a. comprising thetopics:
a. Permissions
b. Encoding
c. Data protection
Accordingly, several test scenarios unfold which are expected to be professionally carried out with appropriate tool support – both effectively and
efficiently.
Manual Tests Need to Be Conducted
Due to the wide range of topics and the diverse task structures, both automatic tool-based tests and manual tests shouldbe provided: While automatic, tool-based tests for instance are ideal for test automation, the topic Usability demands a skilled test engineer who evaluates the quality of user guidance and information presentation in light of manual operating simulations. Special tools, like Eye-Tracking, will surely be considered in the test setup and provide useful information. However, there are no tools that could be used for an expedient, automated execution of the test in the usability environment.
The Test Automation Is the Key Though!
I was just pointing out that test
automation cannot deal with all topics when it comes to “Mobile Testing”. Still I believe that introducing automated test processes properly within the development process supports the success in the app market segment or even make this success possible in the first place. On this a couple of thoughts:
- Test automation creates supplemental, available capacities on the part of the test team during the crucial test execution phase. This allows test engineers to focus on the topics of usability or field tests for instance. Note: Needless to say that the test automation’s efforts shift from the execution to the specification and preparation phase lying temporarily before the test execution though, and usually do not involve shortages or critical paths.
- Present-day test automation frameworks allow creating and maintaining test processes in a test tool or rather test-driver neutral language also supporting several technology platforms of the test items. As to the app development, this means that the test processes concerned with testing features need to be both created and maintained only once, even if the application is realized on Android, iOS and Mobile 7. The specification and preparation of efforts is therefore downsized to one third of the effort conventionally required for quality assurance of all three application versions for the Android, iOS and Mobile 7 platforms.
- The test automation is not only able to spot weaknesses of the test item on its own, but also to identify weaknesses in the specification and development concepts – if test results are evaluated properly. These findings are important to render possible optimization of the development processes cutting down the development expenses of the subsequent software versions.
Key Requirements for Test Tools Dealing with “Mobile Testing“
To enable successful test automation, it is advised to use test drivers specifically developed for this purpose that are able to work on the following technology platforms:
- HTML5
- Android (Google)
- iOS (Apple)
- Phone7 (Microsoft)
- RIM (Blackberry)
- Symbian (Nokia)
The Technical Standards are in Transition
Currently, the technical requirements of the test automation for test drivers and tools are changing: While Java ME and Symbian were important requirements a couple of years ago, already today this is not the case. Looking to the future, the three platforms Android, Phone 7 and iOS will represent the most relevant technical platforms on cells, smartphones and tablets. Additionally, the test systems are developed to deal with HTML5 as well.
The test automation itself has also been revised for years: These days, model and interaction driven test approaches are preferred over test technologies based on keywords or “capture/replay”. These new technological test approaches
- allow a closer interaction with development processes,
- renderpossible realizing check and test routines parallel to the product development,
- support more advantageous time-to-market conditions and
- allow the creation and maintenance of test cases with less effort than it would be possible with conventional technological test automation methods.
Presently, the test automation tools of the renowned major tool producers are only of limited assistance in the test environment as far as the requirements for “Mobile Testing” are concerned. Fortunately, there are special test frameworks and test drivers specifically created to suit “Mobile Testing”. Hence, systems by Communology and i3 Consult are available on the market.
The i3 Consult solution provides for automation tests to operate on both a smartphone and a virtual unit the smartphone simulates (see graphic). The advantage is that all the tests do not depend on the respective smartphone. Tests can also often be carried out faster in virtual environments than on the smartphone itself.
In order that different test items can be accessed through a test framework, using a – preferably abstract – test language, the particular test process definitions need to be transformed into proprietary test scripts via a language transformation before they are passed on to the test drivers or test tools to be carried out.
Ill. 4: The T3 principle – Accessing several test drivers using a test model language
After the test management application has activated the test suite, the T3 approach transforms for example the test procedures, having been defined in the abstract T3 TML test model language, into the intended test driver’s respective language by means of a script engine. This test driver eventually completes the test steps and transmits the test results to the test management system. Many test drivers are able to interpret the aforementioned proprietary test scripts. There are exceptions, however, compiling the code in a way that the hereby generated binary code itself is executable and carrying out the testing.
How do test drivers work in the context of app test automation? The following overview of this principle illustrates the most important functions.
Test drivers of the new generation have an adapter whose set-up options allow a successful integration in test tool platforms and test frameworks. The tools can thus not only be used as a stand-alone, they can act as a part of larger test factories using external controls to operate test management systems. The test drivers are the ones that carry out all preparation, execution and conclusion tasks. In the beginning, the test objects are built using the respective code from the version management tool. This application is translated into the respective test object and the corresponding driver version. The test execution follows the encoded test steps and checks the expected results against the actual values. All relevant information concerning the test execution is put on record in a testlog and either filed as test results or transferred to the test management system.
The Cell and Smartphone Market has Changed
Since 2009, the cell and smartphone market has changed considerably: Symbian suffered the loss of its dominant market position which was taken over by Android. Today’s users look for easy-to-use and adaptable operating systems for cells, smartphones and tablets that can be extended by supplemental modules and software. Herein lies the reason why the former sovereigns were replaced by today’s top dogs Android and iOS.
Another change has already been prefigured on the market though. These days, users prefer open systems over closed ones. Specialists therefore expect that Android will strengthen their market shares and that Microsoft 7 will move up to the group of pursuers. Market mavens furthermore assume that RIM and iOS will become less important.
Test Management Systems
| Product | Producer | Brief Description | Link |
| HP QualiyCenter (Automation tools, e.g. Astra Quicktest Professional, Loadrunner) | Hewlett- Packard | The top dog: For years, the Number 1 on the market in terms of test management applications. For a while, competitors oriented themselves towards the QualityCenter range of features and its predecessor TestDirector. That way, the phase model, which is based on the latter, has virtually become standard. And that is not everything: Test data management, conditional test step directions, Business Process Testing and so on – On the whole, topics that have always inspired competitors to enhance their own solutions. | http://www8.hp.com/de/de/software/software-product.html?compURI=tcm:144-937045 |
| SilkCentral TestManager (Automation tools, e.g. SilkTest, SilkPerformer) | Microfocus | The pursuer: A couple of years ago, Microfocus took over Borland enterprise and thus other software products related to software quality management of the Compuware corporation. For Microfocus, the most important step at first was the successful harmonization of all products towards an efficient portfolio. This challenge appears to be completed. The result of this, for example, is the SilkCentral TestManager that on the one hand created new features and on the other hand also adopted solution elements from both Borland’s former Silk product line and Compuware’s QA director. | http://www.microfocus.com/products/silk/silkcentral_test_ manager/index.aspx |
| Rational Quality Manager (Automation tools, e.g. Rational Functional Tester, Rational Performance Tester) | IBM | The one moving up: Rational Quality Manager is based on the JAZZ platform and has therefore good options to be integrated into IBM’s iCASE developer platform. So if development is generally taking place on the JAZZ tool platform, the Rational products are an interesting option because in them the link between development and test teams has been well-solved: The usual, commonly identifiable loss of information has been reduced considerably. That way, the developer is able to take a look at current test results from the system test in the corresponding code range while he is developing. In recent years, their Rational products have enabled IBM to gradually improve their market position in terms of software quality and is why they again rank among the cream of the crop. | http://www-01.ibm.com/software/rational/products/rqm/ |
| Visual Studio Test Professional 2010 | Microsoft | The Newcomer: With Visual Studio Test Professional 2010, Microsoft entered the test management market. Like IBM had done with Jazz, Mircrosoft integrated the system into the Team Foundation Server. This may be of interest for the internet community. Although it lacks some typical test management features, the solution offers some real highlights, such as the Intellitrace for instance. This feature breaks up the usual type of black-box test during the system test and eventually allows the realization of the white-box test: A true quantum leap! For example, it renders it possible to determine either the corresponding line of code related to the identified malfunction or the test coverage of the system test having been carried out – both online and in retrospect. In the context of this comparison, only IBM is able to keep up with solutions like PurifyPlus.The MS Office interface is another exemplary highlight, for this interface operates in both directions. The applications can thus be used not only for reporting but also for fast data collection and management. The necessary alignment of thetest management tool is settled by the Team Foundation Server in the background. | http://www.microsoft.com/visualstudio/en-us/products/2010-editions/testprofessional/overview |
This article was featured in the December 2012 Issue of Software Test & QA Magazine.
About the Author
Torsten Zimmermann Since 1985 Torsten Zimmermann developed software applications for the business and administration. After completing his degree as “Diplom Wirtschaftsinformatiker” (1993), he became familiar with the quality management within the software lifecycle. As of 1995 he has been contributing to international projects in the area of software quality and quality / test management at various corporations amongst them BMW, DaimlerChrysler, Hewlett-Packard, Hoffmann-La Roche and Logica. Over the years, he has become an expert at european level.
He has developped the risk-based testing approach which has been amongst others presented in the professional publication “QZ” and today is established as state-of-the-art knowledge in the area of software quality assurance. His accumulated experiences have led to the creation of the T1 TFT (Test Framework Technologies, 2001) for the advent of a new area of testing systems.
Today Torsten Zimmermann is developing new approaches to enhanced test concepts and test frameworks as the T2 TFT (2004) and T3 TFT (2006). In cooperation with a network of universities this creates new solutions for rules and model based testing systems. Hereby he also considers near-shore / off-shore concepts in terms of software development. In the role of speaker at conventions and specialist author in well-known magazines he periodically presents his conclusions, results, and concepts at national and internalional level.