End-to-end testing from sensors to cloud with 100% test coverage is an ideal test scenario in an IoT ecosystem. Test automation becomes the obvious choice to achieve the desired metrics at low cost. However, with a diverse set of products entering the IoT market, QAs face many challenges to effectively meet the quality standards of these IoT applications and embedded systems.
Common Challenges in End-to-End Testing of IoT Systems
Problems in scaling of the number of connected sensors/devices/networks to be tested
Difficulty in testing multiple configurations
High maintenance of fragmented test environments with multiple versions, devices, and communication protocols
Difficulty in checking unit level code for the security of IoT devices
Problems in testing distributed Microservices-based architecture from the SOA architecture for IoT
Troubles in delivering the right performance in a live environment
These are some of the hard pressing challenges for IoT test automation. Apart from these, the requirements to test end-to-end use-cases to deliver great user experience in a live environment of IoT ecosystem shoots up the test infrastructure costs in the form of test labs.
Virtualization in such a scenario becomes helpful to optimize the test environment usage and lower setup costs. Virtualization lets you simulate software components and hardware set-up for testing applications (which are dependent on these components) when these components are not accessible for testing. Interestingly, it emulates the dependencies to help imitate the behavior and data interaction to derive the actual testing output. It is also one of the best ways to speed up testing and time-to-market.
When it comes to implementing virtualization, there are various components that play different roles. Below are the components of setting up a virtualized test environment for IoT test automation. Along with this, the entire use-case of ‘How sending messages from the device under test to server and validating the response along with business rules can be automated through virtualization’ has been also discussed below:
1. Sensor Virtualization
Sensors can be virtualized by deploying firmware on low cost USB sticks and using Raspberry Pi for computational needs. These virtualized sensors can then pass signals to the server through a gateway for end-to-end testing. Applications can be tested at length with all possible versions of virtualized hardware.
2. Service Virtualization
Virtualizing web services, backend, messaging protocols, IoT endpoints in the cloud, cloud services, network services remove functional testing and integration testing service component dependencies and ensures total system testing coverage early in the development cycle rather than waiting for the end product. It also keeps the continuous integration/continuous delivery pipeline intact, virtualizing the constrained components.
3. API Virtualization
Virtual APIs can be developed to mirror production environment and test much earlier in the development cycle. These are also needed to enable scaling and partner with cloud devices. API virtualization comes in handy to test negative test scenarios on performance, downtime etc. Also in today’s on-demand API economy, where a developer has less control over how the users behave with the end product, API virtualization helps reduce the random test scenarios.
Case in Point
For testing of a home automation end-to-end use-case, virtualizing the water, smoke, humidity, temperature etc., with sensors connected to the primary home security and consumer premise equipment save large hardware and interoperability testing environment costs. Also, virtualizing web services and REST APIs to collaborate with partner cloud devices helps to scale rapidly and reduce time-to-market. Total coverage for functional testing can be ensured right from the sensor to the end user mobile app.
Another use case where virtualization can aid is of smart city solution requiring hundreds of field video nodes to supply the right video input to the platform. Video node virtualization using Raspberry Pi and API virtualization to capture the video input ensures continuous testing and monitoring, improving the uptime.
Virtualization can also lend a path to analytics, where the volume of data collected increases significantly due to a number of iterations and scaling it enables. This data can then be analyzed and fed back to the various test scenarios.
Virtualization forms a major part of automating the regression test cases of an IoT system that takes as much as 30% of overall testing efforts, typically during any particular release. It helps in reducing risk, time, and cost and achieving scale. In addition, virtualization enables continuous testing of applications.
Give us your take on virtualization for IoT test automation in the comments section below.
Urvashi Babaria is a Product Manager at eInfochips working in new age areas like IoT, DevOps and CloudOps. She is a techno commercial marketer with nine years of experience in product and project management. Start a conversation with her at firstname.lastname@example.org