How we implemented the video surveillance functionality for the security robot

Table of contents:

So let’s continue. For those who have not read the previous posts on how we created the security robot, we advise you to start with them. We have already talked about the idea of the device and the development of a Proof of concept prototype based on children’s construction kits. We settled on the fact that we turned a child’s toy into a full-fledged Internet of Things device that can be remotely controlled from anywhere in the world. Let’s move on to another interesting point: how we looked for the best solution for implementing video surveillance functionality.

Watching video from a robot’s camera is not an easy task

One of the main tasks that the robot must solve is video surveillance in the territory it protects. The idea of ​​the device was initially that the security robot should replace the stationary cameras installed in the house. First, several cameras to monitor every corner of the house are expensive. Secondly, the more devices there are, the more potential points for system failure. Finally, stationary cameras are unsafe; theoretically, intruders can hack them. You can always put it in a closet when it is a robot. Let hackers admire your fishing gear, not your personal life.

In fact, there are no serious difficulties with video surveillance as such – you can use any modern multimedia data transfer protocol. Choose your taste.

We started with a protocol familiar to our team, namely RTSP. A couple of years ago, we developed a custom access control system with video surveillance. It was logical to use popular RTSP cameras on the market, which we did.

However, what is suitable for stationary cameras is not suitable for a mobile robot.

The main problem is that we need more than just video surveillance. The robot must be controlled from another point of the planet, and at the same time, we wanna see what is happening through its eyes. But this is not easy since the signal delay becomes a critical technical parameter.

When using RTSP, the video delay ranged from 1 to 5 seconds. It would seem quite tolerable. But now, imagine how to remotely control a mobile device if you see on the smartphone screen not what is happening right now but what happened five seconds ago. The robot has already hit the wall, and you don’t know about it yet. Or you send him to the closet, but he goes to the nursery. It’s no good.

Experiments with RTMP, HLS, and WebSocket video streaming protocols were similar. The signal delay was consistently at least a second, which is a lot for our project.

Robot Zoomer

Discussing these problems with the whole team in Zoom, we thought – stop, and what kind of multimedia data transfer protocol this Zoom use. In this and similar services, the problem of multimedia delay has been quite successfully solved.

Bingo! This is where the solution had to be found. Zoom and other video-telephony services use the WebRTC protocol or its modifications. This protocol can flexibly adapt to the connection quality, and if it is bad, the video quality decreases, but the connection does not disappear. WebRTC is not demanding device resources and works great on the Raspberry Pi. But its main advantage is that the video delay is minimal here, in total from 100 to 300 ms. That is the almost instantaneous transmission of video and sound even to the other end of the planet. Just perfect!

So our robot suddenly became not just a robot but a robot zoomer. A little moody, though.

A new problem has arisen during testing. If you start broadcasting from the robot’s camera on three or four smartphones, the picture begins to freeze. These are the features of peer-to-peer connections without using a server when devices communicate directly.

This problem was solved very simply. At first, we used WebRTC inside a MESH network, which does not require a server. But the disadvantage of this solution is the increase in load with an increase in the number of connected devices.

There is also a second option – WebRTC SFU. In this case, a server appears on the network that takes on such a load. We just used the server capacity of the 2Smart Cloud platform. This does not affect the delay in any way, and it remains minimal.

From that moment on, the robot’s video surveillance functionality became a reference. Several users can connect to the robot’s camera from their smartphones at once. For example, members of a large family may want to control what happens at home while no one is inside. At the same time, the robot can be confidently controlled – the video and sound are transmitted with minimal delay, which does not interfere with the control of the device. This is exactly what we need.

The first stage of work is completed!

With this, we have completed the first stage of work on the security robot. We assembled a Proof of concept prototype using Lego Mindstorm and GoPiGo children’s constructors. We turned a child’s wheeled robot into an IoT device by writing 2Smart Cloud compatible firmware for the Raspberry Pi microcontroller. Finally, we have implemented a video surveillance solution that allows users to view the image and control the robot remotely comfortably.

The next challenge is to turn the Proof of concept prototype into a minimum viable product. Children’s constructors are good. An attached mobile phone to use its camera as a surveillance camera is even better. But it is impossible to sell such a device on the market because buyers will laugh.

We will tell you what solutions were used to build the MVP. And how we reassembled the robot as a result. Now it looks much more like a serious device that will provide security at home.

Also, we remind you about our Telegram chat if you are interested in technical details! Join us, and ask your questions. We will try to answer everything!