With the vast majority of mobile apps being developed by startups, it is critical for these early-stage companies to be prepared to address the challenges that apps bring to the development and quality processes. The mobile web and native apps present distinct challenges and require new skill sets for testing.
If app developers do not focus on the functionality, usability, reliability, and security of their mobile and tablet apps, they may find themselves in the awkward position of getting their application rejected by the apps store, or finding disgruntled users tweeting their dissatisfaction very publicly.
Sure, your staff is probably underpaid, overworked and over-caffeinated. Yes, you have an incomplete product built for customers who don’t yet know you exist. And yes, your software has more holes in it than your business plan. But guess who doesn’t care about any of that? Your customers, prospects and investors.
With so much to do – and so many fast-approaching deadlines – it’s easy for entrepreneurs to dismiss tasks that would otherwise be considered mission-critical. Specifically, tasks like software testing.
The purpose of this article is to provide some essential tips and examples of how the world’s most promising companies test their mobile apps. Let’s get started…
Mobile Testing Checklist
A common mistake for startups is focusing too much on how they plan to test, as opposed to what they want to test. So, before getting into the various testing methodologies for mobile apps and mobile websites, let’s take a quick look at which features and functionalities you should consider prioritizing:
- Install and Uninstall: This may seem like a no-brainer, but if users cannot easily access your application, your efforts will have been wasted. Pay close attention to the fields.
- Menu Options: Oftentimes, menu options can be difficult to access and decipher. Make sure that menu items like Help, About, etc. are easy to find and navigate.
- Keys: Any problems related to scrolling, text selection, back button, etc. will lead to trouble. Make sure key functionality is clear and consistent; and that the app will run with a keyboard and touch-screen.
- Data Handling: Important for all mobile apps (games and social in particular) is data handling. Does the app store save information properly? What about data deletion and unintended data deletion?
- Connection Speed/Carrier: With the iPhone now running on Verizon, testing on one carrier is no longer an option. Make sure the most basic app tests include the most popular (and likely) carriers.
- Operating System: Like carriers, the mobile OS running your app will have a huge impact on its relative performance. Here, it’s best to identify 3-5 of the most popular versions and test accordingly.
- Screen Size: While this mainly applies to tablets, screen size discrepancies should be a top priority when testing a mobile app. How much should users have to scroll from left-to-right/top-to-bottom?
- Interrupts: How does your mobile app behave when the device battery is at full strength, medium strength and low strength? How about incoming calls, SMS, MMS and video calls?
- Geo-Location: If the app incorporates check-ins, maps, or GPS, make sure you test the functionality “in the wild.” Labs offer a sterile environment. Testing needs to be done by real people with real handsets.
- Error Messages: Your error messages should be clear, concise and actionable. Do this, and you’re a step ahead of virtually all mobile applications on the market today.
Mobile Functional Testing: Manual or Automated
Before you start crafting test cases or user journeys, you must answer another important question: should I use manual testing or automation?
For established companies, the answer to that question would be a resounding “both.” But for startups with limited testing budgets and rapidly-evolving applications, manual testing is the preferred option. Although there are several open-source automated solutions, many of them are exclusively made for one operating system (iOS). Testing author and expert James Bach advises:
“I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.”
Other advantages of manual testing include:
- Finding real bugs: Automation suites will highlight some errors, but most bugs within mobile apps – especially usability and layout issues – are only discovered under true real-world scenarios. Testing application design and function is best with handson device interaction.
- Adaptability: Manual testing can be altered much more quickly and effectively than an elaborate automated test. Chances are, if you’re working within a startup environment, your testing requirements are likely to change as new features are added.
- Real feedback: Unfortunately, automated tests can’t give you an honest (human) opinion about your app’s performance, usability and functionality. We’ll let you know when this changes. In the meantime, you need to see results from real users with real devices.
Mobile testing for startups is all about discovering new areas of concern. To rehash an old quote from mobile testing expert Karen N. Johnson: “If you don’t know what you need to test, then how are you supposed to automate it?”
Of course, it’s somewhat unreasonable to manually test every functional scenario of your application, which makes this process of open-ended discovery all the more important. User Reviews Usability testing is one – if not the —most vital task for a startup planning to launch a mobile application. Before you expose a new application (or version) to the scrutiny of real users, it’s important to conduct a trial beforehand. While enterprise companies spend hundreds of thousands on lab tests, focus groups and beta communities, startups must make do with less.
In a recent interview with usability expert Jakob Nielsen, he explained:
If you don’t have customers yet, locating software savvy professionals for a brief session is your next best option. Once you’ve selected your focus group, the next step is to determine what you want them to focus on. Here are a few ideas to get you started:
- Page Layout: Page Layout: Ask your users to evaluate the overall effectiveness of the page layout. Are they instinctively drawn to your application’s main features, or are they directed elsewhere? Also, are they required to scan/scroll for any important features? In short, the layout might seem intuitive to you, but that’s probably because you designed it! Therefore, a fresh set of eyes is a must.
- Color Schemes: This can include an evaluation of the use of colors in the background, text, links, icons, buttons and other aspects of your mobile app.
- Findability: When conducting your user tests, give them a set of 4-5 items to find. This can include Help, About, Instructions, Search or others. This will help you determine your application’s level of accessibility. Remember, mobile users have far less patience than web users, so these items need to be placed in the best spot possible. Find out where that is.
- Localization: If your application is going to be translated into other languages (or made available in non-English speaking markets), you’ll want to make sure that your app is consistent in terms of text, messages and symbols (e.g. dollar signs).
So, when should user testing be conducted? Ideally, usability testing is done throughout the development cycle. But for startups, it’s recommended to be done once the application is available, but before it’s being downloaded by end users (or paying customers).
Mobile App Strategy
Your mobile testing objectives will depend heavily on the overall strategy you choose to employ. In this instance, the main choice is between developing a native app or a mobile website. Are you looking for one common platform with a consistent look and feel, or are you looking for lights out usability at the expense of alienating device families? Let’s take a quick look at the benefits and drawbacks of both approaches:
Mobile Web Arguments For: n Single platform (the web)
- One app to build, launch and maintain
- All that’s needed for some companies Arguments Against:
- Less controlled user experience
- Slower user experience
- No app store distribution
- Lack of standards across mobile browsers
Native Apps Arguments For:
- Rich media functionality
- Controlled user experience
- Faster experience for users
- App store distribution Arguments Against:
- Multi-platform
- Must build, test and maintain multiple apps
- Not necessary for some companies
The lesson here: Choose your mobile path wisely, as your testing strategy will depend entirely on this choice.
For a mobile app to work as intended, consider the multi-dimensional testing matrix. There are multiple operating systems, different mobile browsers, a myriad of smartphone device makers and models, a host of carriers, and worldwide locations. With all these criteria, new apps for the Android, iPhone, iPad, BlackBerry and Windows phones are driving the need for fast, effective testing that mimic your user base.
As the mobile market doubles and triples in size over the next decade, it’s the startups (and enterprises alike) that have made testing coverage a priority that will enjoy their ROI in terms of increased market share, profitability and above all, user loyalty.
About the Author
Doron Reuveni Doron Reuveni co-founded uTest in February, 2007. Since its inception, uTest has provided functional, usability and load testing services through a community that includes more than 30,000 professional testers from 165 countries around the globe.