Skip to main content

Mobile Testing: Frameworks, Device Inventory and Virtual Testing

By July 22, 2014Article

A mobile app architect who consults to the wireless carriers recently lamented the complete absence of a contract-for-hire mobile testing center in Silicon Valley. We talked about the pros and cons of the community-sponsored Open Device Lab model for mobile testing and, feature limited or not, the absence of one in the Valley. That got me thinking about the why not? That led further to questions about cost, proximity and timely access to the ever-expanding, ever-changing mobile device inventory needed to test efficiently. What is the present state of virtual testing for mobile? 

I quizzed mobile testing management at my company and other industry insiders, and I discovered some surprising answers.  

Let’s look at the current mobile app and website testing challenges. 

Mobile device access 

It’s a foregone conclusion that in today’s testing landscape most mobile app developers will not have direct access to all the devices they want to test on. There are simply too many devices available in the market. Starting with Acer on the alphabetized OEM list and ending with ZTE, there are dozens of brands with hundreds of smartphones and tablet versions in between. The total number of devices potentially needed for comprehensive app testing is mind-boggling. Rapid device obsolescence only adds to that complication. 

Mobile testing tools 

TestFlight, Keynote, Gigwalk, dotMobi … these are just four of the many mobile testing tools available. Some cloud based, some not. This link will take you to a list of 25. Not enough to choose from? There are more available on the market. 

What criteria should be employed to select the right tool? Some to consider are open source versus commercial, automation versus manual, cross platform or not, scalability, testing speed and last, but most of the time never least, cost. 

What about mobile testing frameworks? Where do they fit? Robotium, selendroid and AvanTest Accelerator frameworks are options among many. These also should be considered. 

Virtual testing solutions (emulators and simulators) 

The complexity of replicating various mobile test-to environments is already established — it’s expensive and time-consuming. One path to take is the use of emulators and simulators. Emulators alone are only an approximation of real devices, at best. However, they can be effective particularly when used during the early stages of mobile testing but less so for final testing right before the app or website moves to production. 

One model employed by some is emulator use early on in the testing cycle, switching to real devices for smoke, performance and regression testing, and interoperability later. Free mobile device testing emulators available include iPhone Tester and Mobile Phone Emulator.  

Solutions like DeviceAnywhere and Perfecto Mobile provide cloud-based platforms for testing mobile apps and websites, essentially hardwiring devices into a centralized data center creating a virtual view to the end user wirelessly through an app.  

This approach mostly eliminates the limited approximations of devices found with an emulator solution as well as the very large step of creating and maintaining an expensive physical testing lab. 

Time to test versus time to market 

The numbers don’t lie. A recent mobile survey shows that 58 percent of respondents said they do most of their mobile app testing on real devices, while only 14 percent have automated more than 50 percent of their mobile testing. 

Translation: time to test is winning over time to market. And companies are spending a lot of money on purchasing mobile devices needed for testing — or worse, acquiring only a few devices due to budget limits, then releasing the narrowly tested app to predictable, multi-device access problems. 

If not approached correctly, poorly executed mobile testing will lead to quality problems resulting in uninstalled apps and abandoned websites. Mobile users are easily dissatisfied and know how to quickly uninstall their apps. Loyalty to the app or website can be quickly lost through poor performance.  

One way to close this inefficiency gap lies in forming a partnership with the right mobile testing partner. If you’ve standardized your testing methodology on Agile, an effective partnership category to consider is the nearshore mobile testing services specialists located in Latin America.  Countries noted for vendors in this area include Costa Rica, Peru and Argentina. 

The benefits include access to:

  • Broad mobile device inventory owned and maintained by the vendor
  • Mobile QA engineers that can easily be scaled up or down on demand
  • Comprehensive skills access that even reaches into expertise with functional requirements wireframes, UX and graphical artifacts creation, and app store submission
  • Same U.S. time zone; for example, Costa Rica is one hour ahead of San Francisco, Peru two hours ahead. Contrast that to 12.5 hours for Bangalore and 10 or more hours for Eastern Europe
  • Experience with mobile-centric platforms such as iOS, Android, Windows Mobile, PhoneGap and HTML5
  • Partner-created testing frameworks and third-party vendor testing tools your partner has experience in
  • Deep Agile knowledge of tools (Jira, Rally) and process (Scrum, Extreme) 

Surveys repeatedly show that the availability of mobile testing tools, enough time to test and mobile device access are, in order, the most serious challenges faced. Gaining access to a nearshore mobile testing partner that can check off these “knowledge boxes,” provide customer references with production apps, strong engineer retention metrics and attractive rates, can make all the difference. 

The road to effective mobile testing solutions has multiple forks. Cloud-based testing solution vendors provide SaaS platforms enabling testers to access real smartphones and tablets as well as emulators integrated with proven testing tools. Nearshore mobile testing companies in Costa Rica and other countries offer human capital expertise that is both on demand and cost-effective. There are other forks in that road worth exploring as well. 

It is important to understand the reality of poor performance in the marketplace: mobile users are easily dissatisfied and know how to quickly uninstall their apps or abandon a website. Choose wisely.    

John Hitchcock has over 20 years of experience in the software industry. Currently a director at Avantica Technologies, one of Latin America’s largest nearshore software engineering services companies, for the past decade he managed teams in India, China,  Europe and the United States. He held leadership positions at growth startups funded by Sequoia Capital, Norwest Venture Partners, Austin Ventures and the technology group at GE.

 

 

 

 

 

Copy link
Powered by Social Snap