Home security robot: commercial R&D at its finest

Table of contents:

Introduction

Why read this White Paper?

This document will be helpful if you are looking for a partner with expertise in building IoT solutions.

The document describes the experience of the 2Smart Cloud team in solving problems of designing and creating a workable model of a complex IoT device — a mobile robot performing security tasks. The platform team shares how the tasks set to develop a minimum viable product (MVP) device were solved according to the requirements set by the customer. You will learn how several Proof of concept (POC) prototypes were created based on the research and development stage, and then a minimum viable product was assembled based on the best solutions. As a result, the customer received an MVP prototype that solves the tasks agreed upon in the final list of requirements for the robot. The 2Smart Cloud team also provided the customer with a schematic diagram of the device and software or firmware for the microcomputer.

The solution’s components included the following:

• conducting research and developing the customer’s ideas;
• creating several Proof of concept prototypes to test various solutions to the tasks set;
• offering optimal solutions for the implementation of the tasks assigned;
• developing firmware for the Raspberry Pi microcomputer controlling the robot;
• creating an MVP;
• integrating the product into the 2Smart Cloud platform;
• handing over documentation and software to the customer to establish serial production of devices.

Task formulated by the customer

The idea of the security robot concept, voiced by the customer

The customer contacted 2Smart Cloud with the goal of developing a mobile robot capable of performing security functions in an apartment or a private residence. The customer was dissatisfied with the market’s offerings for home security video surveillance using stationary IP cameras. A robot with a video surveillance function was supposed to become an alternative to such systems.

The following are some of the drawbacks of stationary video surveillance:

• to cover all zones in an apartment or house with video surveillance, a large number of cameras is required — this means significant equipment costs and also an increased number of potential system failure points;
• security problem — there is a theoretical possibility that an attacker will gain access to the broadcast from stationary cameras (controlling the status and location of a single mobile device is significantly easier).

The main scenario of the security robot proposed by the customer was the robot to be located at the charging station by default and control one selected zone. If necessary, it can take off from the charging station and start patrolling the available space. At the same time, it is crucial that the video surveillance camera on the robot body can be rotated in multiple directions.

The customer also suggested installing the maximum number of sensors on the device, the readings of which are helpful for monitoring the safety of housing (including fire safety control). Based on the sensor readings, in the future, it will be possible to configure a dashboard with the current state of home security in a mobile application and configure the automatic sending of urgent notifications to the owner’s smartphone when specified events occur, etc.

Additional wishes of the customer to the functionality of the security robot:

• patrolling a given territory offline using the capabilities of artificial intelligence;
• manual remote control of the robot’s movement;
• automatic sending of the robot to the charging station when the battery charge is reduced to a given level;
• video surveillance during patrolling;
• recognition of people on video and the selection of faces;
• the possibility of group interaction of several robots.

Solving problems using 2Smart Cloud

The customer-specified task received a positive response from the team. Already during the first brainstorming session, it became obvious that implementing the idea would require the employment of a wide range of technologies. Because a complete list of such technologies may not be readily apparent, the initial focus should be on research and development. Thus, it was decided to propose a three-stage approach to completing the task:

1. Proof of concept: assembling several variations of the robot based on existing elements; conducting R&D and determining the list of technologies necessary for the implementation of the tasks; getting a clear understanding of what the finished product should consist of and how it will work.
2. MVP (minimum viable product): assembling a prototype that is close to the real product based on Proof of concept results.
3. Established production: creating a board design for the element base defined at the Proof of concept stage; organizing the manufacture of a robot body and quality control of finished products.

The implementation of the third stage depended on the customer’s decision and had to be mainly carried out by his forces. It was assumed that the 2Smart Cloud team would provide support only if necessary improvements to the design and software of the robot are needed. Overall, the main task was to implement the first two stages of device development.

Proof of concept prototype

The first task to be solved was to design a base for a security robot. That is, to create a robot capable of moving in space, with the possibility of remote control. Based on the capabilities of the 2Smart Cloud platform, it was decided to implement remote control of the robot’s functions, receiving information from its sensors, and viewing video from a surveillance camera.

Using the 2Smart Cloud functionality makes it possible to design a security robot as a full-fledged IoT device that can be connected from anywhere where there is Internet access.

Given that real-time video monitoring was one of the major duties of the security robot, it was necessary to achieve the shortest possible latency in video transmission. To successfully handle the robot’s movement, the operator must receive a visual image from the camera almost instantly.

Thus, as part of the creation of the Proof of concept prototype, we had to verify the feasibility of two hypotheses:

1. There is a technical solution that will allow the creation of a platform for controlling a robot within the framework of the basic functionality of 2Smart Cloud.
2. There is a technical solution that allows ensuring a minimum delay when transmitting a video signal.

Creating a robot platform

A simple way to confirm the first hypothesis (the possibility of creating a software platform with integration into 2Smart Cloud) was using children’s constructors for robotics. Two models of constructors were used in work, each of which has its specific advantages:

• Lego Mindstorm is a constructor with excellent mechanics: servos, several types of sensors, wheels, and body parts.
• GoPiGo is a construction set with a Raspberry Pi–based microcomputer with proprietary software.

After initially turning to Lego Mindstorms, the team’s developers realized that the set’s microcomputer is unsuitable for building a 2Smart Cloud compatible programming platform. It was necessary to create a new or find a ready-made solution based on a microcontroller compatible with our IoT platform.

At this stage, the preferred option was to find a ready-made solution, which would save time and finances. It has been hypothesized that there are robotics kits based on Raspberry Pi, a popular microcomputer often used in robotics schools and fully compatible with 2Smart Cloud. This way, the Gigo constructor was found, which was precisely a solution we needed.

Another advantage of GoPiGo is the proprietary software, which code describes the logic of controlling standard robot parts. First of all, the developers were interested in a library describing the control of servos, that is, the movement of the device in space and the movement of its parts. Having a ready-made library, there was no need to spend resources on its creation.

The team’s programmers created firmware for Raspberry Pi using this library. The firmware was developed according to the principles of compatibility with 2Smart Cloud, described in the documentation for the platform. As a result, the microcomputer is connected to the platform, and a full-fledged Internet of Things device can be created on its basis.

A microcomputer with this firmware was installed on the Lego Mindstorm all-terrain vehicle. As a result of the search for the optimal design, the device’s circuitry was finalized – additional control boards built by us were installed, as well as a voltage divider and a power battery.

Immediately after connecting the assembled robot to the platform, it was possible to confirm the hypothesis about the possibility of creating a platform for controlling such a device compatible with 2Smart Cloud. The mobile application interface for robot controls was configured in the developer’s dashboard, and the device responded correctly to commands at first power-up. Initially, management was carried out via an application emulator via the 2Smart Cloud browser interface, which is available to all developers.

Broadcast from a surveillance camera with minimum latency

One of the previously implemented projects of our team is a cloud-based access control system (detailed white paper). The video surveillance functionality in that ACS was implemented based on the popular RTSP multimedia data transfer protocol. Therefore, we turned to the protocol we were familiar with, starting work on the video broadcasting functionality for the security robot.

During the first tests, our initial hypothesis that the latency in transmitting the video signal could be critically large was confirmed. The video broadcast on the mobile app screen lagged for 1 to 5 seconds. This was unacceptable since remote control of the robot became almost impossible.

When exploring alternatives to RTSP, we also applied protocols such as RTMP, HLS, and WebSocket video streaming. However, even in these cases, the best outcome was a delay of one second, which is quite a lot.

As a result, the team chose the WebRTC protocol used by the most popular video conferencing services. The main difference between WebRTC is using a peer-to-peer connection when no server is used for data exchange. Additionally, WebRTC makes it possible to adapt the multimedia signal depending on the Internet connection quality due to flexible data compression.

Multimedia data is transmitted directly from one device to another. Together with flexible stream compression, this reduces the broadcast delay time to only 100-300 ms. This is an excellent result, which makes it possible to control the robot remotely confidently.

An additional advantage of WebRTC protocol is the low demands on device resources. The load on the Raspberry Pi when using the video broadcast mode remains moderate, but only if we are talking about connecting one or two clients.

A serious issue that the 2Smart Cloud team encountered when testing the WebRTC protocol was a periodic signal freeze when watching a broadcast from more than two devices at the same time. The fact is that initially the team’s programmers used a MESH network to organize data transfer. With this option of using the WebRTC protocol, the load on connected devices increases in proportion to the increase in the number of devices.

Consequently, it was decided to use an alternative option — WebRTC SFU. With this method of organizing data transfer using WebRTC, it is necessary to use an additional server: we used the cloud resources of the 2Smart Cloud platform. This decision did not affect the data transfer latency time, although it increased the load on the platform servers.

Thus, WebRTC SFU has become the optimal solution for broadcasting video and audio signals from the surveillance camera of a security robot. The potential of such a solution is sufficient for any user scenarios within the task of creating a robot for home use. Therefore, it was decided to use WebRTC SFU when creating a minimum viable product.

Building an MVP

Before moving to this stage, the team additionally agreed with the customer on the list of requirements for the robot to have. It was decided to postpone the implementation of AI-based automatic patrolling, the function of automatically returning the robot to the charging station, as well as the possibility of group interaction of several robots. All these tasks are solved by updating the software and do not require changes to the design of the device. If necessary, they could be solved later if the customer decides to launch mass production of security robots.

To start creating a minimum viable product, it was necessary to choose a body for the robot. As a result, a commercially available crawler chassis for robots was found, the characteristics of which met the requirements of 2Smart Cloud for the body for the MVP prototype.

The choice in favor of a caterpillar engine was made consciously – even at the POC stage, it turned out that this option is preferable to a wheel drive. Models with tracks move better on carpets and overcome thresholds, which is important when used in residential areas.

To simplify the serial production of robots, the following solution for installing firmware on the device was proposed:

• the customer or his contractors assemble the device on Raspberry Pi, connecting the necessary sensors according to the device’s schematic diagram;
• a memory card with an image for Raspberry Pi is installed on the microcomputer and after deploying the image, the device is ready to work.
• a memory card with an image for Raspberry Pi is installed on the microcomputer and after deploying the image, the device is ready to work.

The firmware code has been improved — in the final version, the library for the children’s constructor GoPiGo is not used since other libraries have been optimized and improved.

To control the robot and receive data from its sensors, it was proposed to use the 2Smart Cloud mobile application. After purchasing the robot, the end user needs to install a mobile application and start the pairing standard for all IoT devices. After completing this procedure, the robot connects to the home Wi-Fi network and begins to perform its functions.

In addition to direct control of the device from its interface, the mobile application offers users additional functionality: control using voice commands, a phone call, a Telegram bot, the ability to share access to robot control with family members.

If you need a branded mobile application, it is possible to create a white label application based on 2Smart Cloud.

The main robot control features available in the mobile app are:

• controlling the robot’s movements using a visual joystick;
• watching video with sound from the robot’s surveillance camera in real-time;
• the ability to turn on the siren, light source, RGB illumination of the robot;
• the ability to launch one of the preset modes of operation of the robot.

In the first version of the firmware of the robot, three modes of its operation were provided:

1. Silent mode: the robot is in a stationary state and does not respond to any events. At the same time, the user can connect to his surveillance camera at any time and start manual control of the device.
2. Monitoring mode: provides the ability to configure a set of events to which the robot will respond by sending notifications to the owner. Such events can be a motion sensor triggering, motion detection on a surveillance camera, a shock sensor triggering, an obstacle sensor triggering, and the room air temperature exceeding the specified limits.
3. Security mode: the robot reacts to the same events as in surveillance mode, but in addition to sending notifications to the user, it actively reacts to what is happening in the house. The user can choose one of the preset modes of the robot’s response to events: only sending notifications (analogous to the monitoring mode), launching a siren, simulating the presence of the owners.

The second stage concluded with the delivery to the customer of the MVP prototype, schematic diagrams for the security robot, and the Raspberry Pi firmware image. Currently, the customer is using the prototype for personal purposes and assessing the economic viability of launching the full product on the market.

You can also use the cloud software for turning your ideas for free if you need to connect a prototype device similar to the one described above.