Integrating CCTV into the 2Smart Standalone Automation Platform

Table of contents:

The functionality of video surveillance is traditionally in demand among users of building monitoring and automation systems. In addition to remotely monitoring sensor readings describing the environmental conditions or equipment statuses, users want the ability to visually monitor what is happening, especially when sensors and connected devices cannot fully describe the situation in or around the building. Today’s manufacturers of surveillance equipment offer proprietary software and a set of hardware for powerful and reliable video surveillance. However, when talking about a unified ecosystem that includes both remote video surveillance and remote management of IoT devices, monitoring their statuses, and configuring automation, there are not many such comprehensive solutions. The 2Smart team conducted R&D on the possibility of integrating video surveillance systems into the 2Smart Standalone automation and monitoring platform to create an MVP of BMS, and we will discuss the results in this article.

2Smart’s Experience in Integrating Video Surveillance into IoT Solutions

One of 2Smart’s products is the Access Control System called “Propuskator,” where our developers have already worked on integrating video surveillance into the admin panel for access control system administrators and the mobile application for end users. After studying existing protocols, the 2Smart team chose RTSP as the optimal one, which is a prevalent standard in the market and supported by a large number of cameras.

In the access control system’s admin panel, there is a dedicated section for “Cameras,” where administrators can add links to RTSP streams from cameras and associate them with specific access points. For example, a camera installed at an office entrance can be linked to the access point “office”.

The administrator can view the camera stream in the same admin panel section, but the primary purpose of this functionality is to broadcast the video in the mobile application for end users. Security and access control system users with access to control the door’s electric lock in our example can see the camera stream and remotely open the door, for instance, when a trusted individual arrives.

Here are a few basic points we considered when implementing video surveillance into the access control system based on RTSP:

• Our goal is not to work directly with the camera video stream; we receive the RTSP stream from the network video recorder (NVR).
• The NVR broadcasts a public or local RTSP stream that we can use at the application level. If the stream is available locally, the application must be on the local network and have access to it. If it’s a public stream, the video can be viewed from anywhere with internet access.
• We have no task to record video, save it as an archive, or use it beyond direct streaming in the applications.
• We found a wide range of desktop software capable of streaming RTSP out of the box, such as the popular VLC player.
  
• We also discovered that there are no difficulties with streaming the RTSP stream from the NVR to mobile applications. Special publicly available tools exist for this purpose.
• At the same time, we found that there are no straightforward ways to display the raw stream in browsers, which became our main challenge.
• In addition to camera demonstrations, we also had experience in R&D with streaming, specifically implementing the recording of short video clips at the moment a door or another access point is opened. This experience gave us an understanding that working with streams requires a very powerful infrastructure. Therefore, firstly, it should only be used when absolutely necessary, and secondly, it is preferable to perform this task on separate resources to avoid affecting the operation of other services.

During several demonstrations showcasing the capabilities of the automation and monitoring platform 2Smart Standalone for potential clients, we received feedback requesting video surveillance functionality. Recognizing the demand for an integrated solution that combines device monitoring tools, remote management, automation, and real-time video surveillance, we observed an opportunity to build a comprehensive BMS. In response to this request, we initiated R&D efforts to create an MVP for such a solution, leveraging the existing architecture of the 2Smart Standalone platform.

Research and Development of Multimedia Data Transmission Protocols

There are three options for viewing IP camera streams:

• HTTP Live Streaming (HLS).
• RTSP stream via WebRTC.
• RTSP stream in MPEG1 Video.

HTTP Live Streaming

We dismissed this option immediately upon discovering that not all IP cameras support HLS. We aimed for a more universal solution.

RTSP stream via WebRTC

Our team is familiar with WebRTC, as we used it in creating a prototype of a security robot. At that time, we chose this technology for streaming video from the robot’s camera to the user’s mobile app because WebRTC ensures minimal signal delay. Since we were working on the functionality of remotely controlling the robot from the mobile app, it was crucial for the user, even if located on the other side of the planet, to be able to promptly manage the robot. To achieve this, they needed to see the actions with millisecond-level latency.

However, for the current R&D task, this advantage of WebRTC is not decisive. At the same time, there are several drawbacks when using this technology on desktop devices.

WebRTC works only with H.264 or VP8 codecs. Typically, cameras are set up for streaming using H.264, but there can be exceptions. Therefore, we cannot claim that this tool is universal. Additionally, not all browsers have native support for these codecs, so some users may not have access to video surveillance functionality when choosing this technology.

Nevertheless, we consider streaming via WebRTC a promising direction, partly because this technology imposes a minimal load on the server. Therefore, we continue our R&D, and we will present the results regarding WebRTC in a separate article a bit later.

RTSP stream in MPEG1 Video

Opting for this option involves on-the-fly conversion of video to MPEG1 Video from any codec (H.264, H.265, VP8, AV1, etc). This offers the following advantages:

• The camera’s codec for streaming becomes irrelevant.
• The user’s browser choice doesn’t matter. We use WebGL & Canvas2D renders for video rendering in the browser. Thus, we are not tied to whether the browser supports the MPEG-1 codec, as we decode and render the data ourselves.
• No complexities in implementing the server-side UI part.

While this option proved most suitable for our current R&D, it has one significant drawback – it heavily consumes server resources. Video and audio conversion from one format to another is a resource-intensive process.

To implement this option within the 2Smart Standalone platform, we created a new “RTSP to MPEG1VIDEO” bridge, as well as a dashboard widget called “RTSP-MPEG.”

Bridge for Integrating RTSP Video Surveillance into the Platform

The microservices architecture of the automation and monitoring platform 2Smart Standalone implies that users connect any necessary functionality by installing the corresponding bridge from the built-in market. To integrate video surveillance into the platform, we developed a new bridge with the task of converting the RTSP stream into MPEG-1. After conversion, the video should be rendered in a widget on the platform’s dashboard.

To add an RTSP stream, the user only needs to specify its URL in the settings. The bridge utilizes the ffmpeg solution for stream processing and a WebSocket server for data transmission.

Here is the ffmpeg command used by the bridge:

ffmpeg -rtsp_transport tcp -i rtsp:// -f mpegts -codec:v mpeg1video -r 30

More about the FFmpeg command options based on the solution’s documentation:

1. -rtsp_transport tcp: This option specifies the transport protocol to be used for RTSP (Real Time Streaming Protocol). In this case, it is set to TCP. RTSP can use different transport protocols such as UDP or TCP, and this option explicitly chooses TCP.
2. -i rtsp://<url>: This specifies the input file or stream. In this case, it’s an RTSP stream located at the specified URL (rtsp://<url>). You need to replace <url> with the actual URL of your RTSP stream.
3. -f mpegts: This option sets the output format to MPEG-TS (Transport Stream). MPEG-TS is a standard format for the transmission and storage of audio, video, and data.
4. -codec:v mpeg1video: This sets the video codec to MPEG-1 video. MPEG-1 is a widely used video compression standard. You can replace mpeg1video with other video codecs if needed.
5. -r 30: This sets the frame rate of the output video to 30 frames per second (fps). You can adjust this value based on your requirements.

In summary, this FFmpeg command is capturing an RTSP stream over TCP, converting it to MPEG-TS format with MPEG-1 video compression at a frame rate of 30 fps. Make sure to replace <url> with the actual URL of your RTSP stream.

By executing this command, we can “broadcast raw data” to the UI. To achieve this, it is necessary to launch a WebSocket server to which the UI will connect, receiving data for video rendering.

Here’s what the pseudocode implementation of the bridge looks like:

// Start WS server on 6060 port
const wsServer = new ws.Server({
    port: 6060
});

// Add callback function to handle new WS connections
wsServer.on("connection", (socket) => {
    return onConnect(socket);
});

// Start ffmpeg command, that will be listening RTSP stream and convert
// incomming data to MPEG1 format
const stream = child_process.spawn("ffmpeg", [
	'-rtsp_transport',
	'tcp',
	'-i',
	'rtsp://',
	'-f',
	'mpegts',
	'-codec:v',
	'mpeg1video',
	'-r',
	'30'
], {
	detached: false
});

// Handle incomming converted data and send to WS clients
stream.stdout.on('data', (data) => {
    return wsBroadcast(data);
});


// Callback function for new connected clients
// Should send the “header” packet with specified video resolution
function onConnect(socket) {
	const width = 320; //could be adjusted or parsed from the RTSP data stream
	const height = 320; //could be adjusted or parsed from the RTSP data stream


	let streamHeader;
	// Send magic bytes and video size to the newly connected socket
	// struct { char magic[4]; unsigned short width, height;}
	streamHeader = new Buffer(8)
    	    .write('jsmp') // STREAM_MAGIC_BYTES, must be 4 bytes
    	    .writeUInt16BE(width, 4)
    	    .writeUInt16BE(height, 6);

	socket.send(streamHeader, { binary: true });
	console.log(`New WebSocket Connection (${wsServer.clients.size} total)`);
}

// send MPEG1 data to all WS clients
function wsBroadcast(data) {
    wsServer.clients.forEach(client => {
        client.send(data);
    });
}

The Widget for Displaying the Video Stream on the Platform Dashboard

To ensure video decoding and rendering in the dashboard widget, we opted for the JSMpeg solution. It provides a refreshing alternative to the complexities of WebRTC, offering a JavaScript-based Video Player with significant advantages. In contrast to the intricate landscape of WebRTC, JSMpeg simplifies the process of setting up video and audio formats that seamlessly function across all modern browsers. It comes with the advantage of no licensing fees, leveraging the expiration of all MPEG1 and MP2 patents. It also excels in delivering very low-latency real-time streaming with high frame rates and acceptable quality and bitrates.

The only challenge in creating the widget was inputting the stream URL because widgets in 2Smart Standalone are solely created through MQTT topic binding. Therefore, within the bridge, we established a sensor that conveys the path to obtain the stream.

Users need to input the path (“/service/xxxxxxxxx”) rather than the full URL. Why did we opt for this solution? In the bridge’s config (when adding it as an additional service in 2Smart Standalone), we can specify that it opens supplementary functionality through a public endpoint. For instance, the Grafana bridge opens a web interface, and the RTSP bridge opens a WS endpoint, enabling the widget to receive the video stream.

Hence, the widget can construct the full URL on its own:

`<protocol>://<hostname><sensor-value>`

• “<protocol>” – ws or wss depending on the browser’s protocol (http or https).
• “<hostname>” – IP address or domain of the platform.
• “<sensor-value>” – value from the sensor, obtained from the bridge.

As for the video delay when using our selected solution, it can be assessed in the video below. On the right side of the monitor, there is a website open with the current time at the moment the video was captured, and on the left side, there is a video surveillance widget in the 2Smart Standalone platform. As you can see, while it is noticeable, it is less than one second. This is not critical for the purposes of video surveillance.

Results

It’s important to understand that we utilized the existing architecture of 2Smart’s automation platform, specifically leveraging our bridges. We added a new bridge to the market responsible for stream conversion and developed a new widget to showcase the already converted stream.

At this point, we have achieved an MVP for BMS with monitoring and video surveillance functionality, which can be further developed and optimized to meet clients’ needs.

We conducted testing on both home and professional cameras. Throughout the tests, we verified streams from domestic and industrial cameras, confirming that the majority of modern cameras with RTSP stream support can be easily integrated into the 2Smart Standalone monitoring system.

FAQ

What is the primary goal of integrating video surveillance into the 2Smart Standalone platform?

Our aim is to create a comprehensive Building Management System (BMS) that seamlessly combines device monitoring, remote management, automation, and real-time video surveillance functionalities.

What protocols are utilized for integrating video surveillance into the platform?

We primarily utilize the Real Time Streaming Protocol (RTSP) for its widespread support among cameras and network video recorders (NVRs). This protocol allows us to receive video streams efficiently and reliably.

What are the key considerations when implementing video surveillance into the platform?

Our focus is on real-time streaming rather than recording or archiving video. We leverage existing RTSP streams from NVRs and employ protocols like MPEG1 Video for efficient browser rendering. Additionally, we ensure compatibility with various camera codecs and browser types.

How is the video stream rendered within the platform’s dashboard?

To display video streams on the dashboard, we utilize the JSMpeg solution, which offers low-latency streaming with high frame rates and broad browser compatibility. Users input the stream path through MQTT topic binding, enabling seamless integration into the platform’s widget ecosystem.

What challenges were encountered during the integration process?

One notable challenge was enabling browser-based rendering of the RTSP stream, which required on-the-fly conversion to MPEG1 Video format. This approach, while effective, can be resource-intensive and necessitates careful server management.

What are the current capabilities of the integrated video surveillance solution?

At present, we have achieved an MVP for BMS with monitoring and video surveillance functionality. Our solution has been tested with a variety of cameras, demonstrating compatibility with both domestic and industrial setups. We continue to refine and optimize the system to meet the evolving needs of our clients.