IoT device testing in real world: best practices of the 2Smart’s QA team

Table of contents:

Testing the Internet of Things devices has its own specificity, setting it apart from similar processes in other industries. A crucial part of the business logic for any IoT device is its interaction with other devices and management and monitoring systems. All these processes must function flawlessly, ensuring the necessary privacy and security. To craft this article, we asked our QA team to provide more details about how IoT device testing is conducted at 2Smart, which devices they have worked on, and any amusing stories that may have occurred. Let’s delve deeper into IoT device testing in both laboratory and field conditions.

Features of IoT device testing

The specifics of the IoT influence the approaches used by the testing team when working with devices. Here are five key features of IoT testing.

1. Understanding the firmware-level operation

Testing IoT devices requires a deep understanding of how the device operates at the firmware level. This is crucial for proper testing and checking potential scenarios, adding an extra layer of complexity and interactivity to this type of testing.

2. Real-world testing conditions

2Smart testers emphasize the challenges of working in real-world conditions, where devices may undergo prolonged testing or be subjected to atypical scenarios. This requires special attention to the long-term stability and functionality of the devices.

3. Interaction with other devices and scenarios

One of the key features of IoT is the interaction between devices. Testing involves checking not only individual devices but also how they interact with each other through scripts or scenarios.

4. Device reactions and interaction with the platform

IoT testing includes not only checking web interfaces or mobile applications but also evaluating how devices react to changes happening on the server side and vice versa. This involves interacting with real devices and verifying their correctness in responding to changes from the user or the platform.

5. Creative approach to testing

Testing IoT devices requires a creative approach as testers are constantly looking for new methods and ways to verify functionality. This is particularly important given the diversity of devices and their functionalities.

IoT testing at 2Smart: process overview and key considerations

The process of testing Internet of Things devices typically varies depending on the project. The purpose of the device, its architecture, firmware business logic – each of these aspects can have significant features that influence the work of the QA team. At the same time, there is a common testing flow applicable to all cases, and we will outline it below.

1. Device introduction

Before a tester begins testing a device, they need to create a work plan. To do this, it is necessary to understand the tasks the device will perform, the functions embedded in its firmware, and the hardware and third-party software it will interact with.

A new device can be relatively simple with limited functionality, such as the 2Smart smart lamp. Alternatively, it can be a complex device capable of moving in space and executing various scenarios depending on the set mode, similar to a security robot.

The tester must break down the functionality of any device into specific functions and create a plan to check the operability of each one individually as well as in combination.

2. Initial device setup

Since we are talking about IoT devices, it is impossible to discuss their comprehensive testing without connecting to the network. Therefore, upon receiving a prototype or a finished device, the tester must connect it to the IoT platform (in the overwhelming majority of cases – to its MQTT broker) and ensure that it publishes messages indicating its connection, that the platform recognizes it, and that communication is established without interruptions.

It is logical that the QA team at 2Smart mainly tests devices that connect to the 2Smart Cloud IoT platform (less frequently those interacting with the open-source automation platform 2Smart Standalone). Therefore, the process of connecting them to the platform corresponds to what all 2Smart Cloud vendors and business platform users use. In general, it looks like this:

1. Creating a new product in the vendor panel.
2. Installing firmware on the device. If the developer has access to the repository, they can build the firmware themselves. More often, the development team provides the ready firmware. If the device will work with the 2Smart business platform, the vendor can upload the binary files for the product there, and firmware installation is done directly from the admin panel.
3. After the firmware is installed on the device, it launches a Wi-Fi access point, and the tester needs to initiate pairing, provide the device with the Wi-Fi network credentials (if this connection method is used), and ensure that the platform detects the device, accepts sensor readings, and is ready to send control commands.

3. Device testing

Speaking about the general case of IoT device testing, we can list universal aspects that require attention regardless of the specific device or its functionality:

Connectivity

This term describes the interaction of the device with external objects and services. This interaction includes data exchange over wired or wireless means at any distance, in any environment, or under any configuration. For example, it can involve interactions via Wi-Fi, Bluetooth, Modbus, and other protocols. This encompasses interaction with the MQTT broker of the IoT platform, communication with other third-party services, as well as interactions with other devices.

Developers may embed certain business logic into the firmware for cases when the device goes offline. In such instances, testers verify whether pre-set functions are executed correctly in the absence of a network. If the device accumulates any data during this offline period, which it should transmit to the cloud upon re-establishing a connection, testers need to check various scenarios and ensure that this data is transmitted correctly, regardless of how long the connection was lost.

Security

One of the principles adhered to by firmware developers for any IoT device is ensuring the security of the device and its communication. This involves using data encryption, authentication and authorization, as well as protective measures against attacks, such as buffer overflow.

Testers must ensure that the necessary security measures are incorporated into the firmware architecture, preventing unauthorized access to the device.

Updates delivery

The primary method of delivering firmware updates for IoT is over-the-air updates. During the device testing phase, it’s crucial to ensure that the delivery is robust, even if the Wi-Fi signal strength at the device’s installation location is weak. A common scenario is when the update fails or proceeds extremely slowly with high energy costs if the device operates far from the Wi-Fi router.

There are cases where the device is connected to the network via a cable, and in such instances, updates are delivered through this method. However, regardless of the method used, testers always check all possible scenarios to ensure that the device, operating in the real world, can always receive updates.

Speaking of updates, it’s also important to mention backward compatibility. The new firmware version should be compatible with the old backend version and vice versa. While this nuance often arises during product development, it is extremely important for the QA team.

Stress testing

Testing a single instance of a device is important, but it’s crucial not to forget that IoT devices work in conjunction with each other, often mediated by an IoT platform. The number of devices can be quite large, and the entire system should maintain stability under this load. Potential weak points might be the backend and the database, both of which should remain stable under any level of load. Stress testing is essential to ensure that the system can handle the expected volume of devices without compromising performance or stability.

Testing in real conditions

After addressing bugs identified during lab testing, it’s always important to verify the device’s performance in real-world conditions. For example, when testing Propuskator access controllers, it was crucial to ensure that these devices continued to operate reliably with weak Wi-Fi signals or during temporary network outages. Testing under real conditions helps validate the device’s robustness and its ability to handle various environmental challenges that might not be fully replicated in a controlled lab setting.

Battery-powered devices

While the aspects mentioned earlier apply to any IoT device, discussing battery-powered devices delves into specific cases. Nevertheless, there are quite a few such devices in practice, making this topic worthy of mention in this article.

Powering devices from batteries significantly influences their operational logic. Developers aim to minimize energy consumption, resulting in some specific nuances in how these devices operate.

Such devices often do not connect to the MQTT broker but communicate with the IoT platform via HTTP, independently initiating requests. They typically remain in sleep mode, waking up on a certain trigger, performing some action, connecting to the IoT platform to transmit and receive information, and then going back to sleep.

An essential task for the QA team is to assess how the slightest device action affects battery life. For example, in the case of an automatic watering system, it’s crucial to understand the battery consumption for each activation. Delivering updates can potentially be stressful for battery power and needs thorough testing.

Interesting stories from the 2Smart QA team’s practice

Testing the Internet of Things devices is a job that requires great concentration and the constant execution of similar tasks. Unusual situations arise when the actual performance of the device doesn’t quite align with the expectations set during its ideation. Here are two examples from 2Smart’s practical experience.

Testing smart lamps: seeing the colors through rose-colored glasses

The 2Smart QA team participated in the testing of several smart lamp projects. These devices varied in complexity, but each one had the same pitfall – the color reproduction of the LEDs used was not always ideal, which was crucial for the project.

In one instance, a lamp emitted a pink light instead of red. Dealing with this issue proved challenging, but it was clear that this aspect could be significant for many users. In another case, testers relied on the colors produced by the RGB strip for a long time, but after placing the lamp in its casing, it turned out that the colors and shades were perceived differently. Additional testing had to be conducted.

One lamp’s stress test involved illumination with a bright white light. Many power supplies couldn’t handle this, and the testers’ additional task became selecting parameters for power supplies and specific models that matched the requirements.

Testing the waters… without knowing its characteristics

Right now, the 2Smart QA team is working on a device with the task of dispensing a specific amount of liquid triggered by a certain event. Since the client is still finalizing the composition of this liquid, we don’t have the exact parameters, which might lead to surprises in the future.

At the moment, testers are using regular water to assist developers in calibrating the device. The device should dispense a small volume of liquid, as configured in the settings. The QA team collects this liquid and measures the dispensed volume to understand how long the device dispenser should activate. If the working liquid turns out to be more viscous, this process will need to be redone.

Wrapping up

The world of IoT device testing is a complex yet fascinating journey, as evidenced by our exploration of testing practices at 2Smart. We’ve delved into the critical nuances of firmware-level operations, the challenges posed by real-world testing environments, and the importance of seamless device interactions with platforms and other devices.

Our experiences have highlighted the necessity for meticulous attention to detail throughout the testing process. From resolving unexpected color discrepancies in smart lamps to adapting to fluid compositions in liquid dispensing devices, our QA team’s dedication ensures that every aspect of device functionality is thoroughly examined and refined.

As we look ahead, our commitment to innovation and creative testing methodologies remains steadfast. By maintaining a balance between technical precision and forward-thinking strategies, we empower our customers to navigate the complexities of the IoT landscape confidently. Our goal is clear: to deliver IoT solutions that not only meet but exceed expectations, providing unparalleled performance, security, and reliability for our clients.

FAQ

What is IoT device testing, and why is it important?

IoT device testing involves evaluating the functionality, security, and reliability of Internet of Things devices. It is crucial to ensure that these devices operate flawlessly, interact seamlessly with other devices and platforms, and maintain privacy and security standards.

What are some key considerations in IoT device testing?

Key considerations include understanding firmware-level operations, testing in real-world conditions, ensuring seamless interactions between devices and platforms, implementing security measures, and conducting stress testing to ensure system stability under load.

How does 2Smart approach IoT device testing?

At 2Smart, we employ a comprehensive testing process that involves understanding device functionality, setting up devices in real-world scenarios, testing connectivity and security measures, and conducting stress testing to validate system stability.

How does 2Smart ensure the quality of its IoT solutions?

2Smart ensures quality through rigorous testing methodologies, a dedicated QA team focused on identifying and addressing issues, and a commitment to continuous improvement and innovation in testing practices.

What are the benefits of thorough IoT device testing?

Thorough testing ensures that IoT devices meet performance, security, and reliability standards, leading to enhanced user experiences, reduced risks of malfunctions or security breaches, and increased customer satisfaction.

Can 2Smart assist with IoT device testing for custom projects?

Yes, 2Smart offers comprehensive testing services for custom IoT projects, leveraging our expertise in firmware operations, real-world testing, security measures, and stress testing to ensure the quality and reliability of your IoT solutions.