The functionality of video surveillance is traditionally in demand among users of building monitoring and automation systems. In addition to remotely monitoring sensor readings describing the environmental conditions or equipment statuses, users want the ability to visually monitor what is happening, especially when sensors and connected devices cannot fully describe the situation in or around the building. Today’s manufacturers of surveillance equipment offer proprietary software and a set of hardware for powerful and reliable video surveillance. However, when talking about a unified ecosystem that includes both remote video surveillance and remote management of IoT devices, monitoring their statuses, and configuring automation, there are not many such comprehensive solutions. The 2Smart team conducted R&D on the possibility of integrating video surveillance systems into the 2Smart Standalone automation and monitoring platform to create an MVP of BMS, and we will discuss the results in this article.
One of 2Smart’s products is the Access Control System called “Propuskator,” where our developers have already worked on integrating video surveillance into the admin panel for access control system administrators and the mobile application for end users. After studying existing protocols, the 2Smart team chose RTSP as the optimal one, which is a prevalent standard in the market and supported by a large number of cameras.
In the access control system’s admin panel, there is a dedicated section for “Cameras,” where administrators can add links to RTSP streams from cameras and associate them with specific access points. For example, a camera installed at an office entrance can be linked to the access point “office”.
We’re empower your business with our technology expertise
The administrator can view the camera stream in the same admin panel section, but the primary purpose of this functionality is to broadcast the video in the mobile application for end users. Security and access control system users with access to control the door’s electric lock in our example can see the camera stream and remotely open the door, for instance, when a trusted individual arrives.
Here are a few basic points we considered when implementing video surveillance into the access control system based on RTSP:
During several demonstrations showcasing the capabilities of the automation and monitoring platform 2Smart Standalone for potential clients, we received feedback requesting video surveillance functionality. Recognizing the demand for an integrated solution that combines device monitoring tools, remote management, automation, and real-time video surveillance, we observed an opportunity to build a comprehensive BMS. In response to this request, we initiated R&D efforts to create an MVP for such a solution, leveraging the existing architecture of the 2Smart Standalone platform.
There are three options for viewing IP camera streams:
We dismissed this option immediately upon discovering that not all IP cameras support HLS. We aimed for a more universal solution.
Our team is familiar with WebRTC, as we used it in creating a prototype of a security robot. At that time, we chose this technology for streaming video from the robot’s camera to the user’s mobile app because WebRTC ensures minimal signal delay. Since we were working on the functionality of remotely controlling the robot from the mobile app, it was crucial for the user, even if located on the other side of the planet, to be able to promptly manage the robot. To achieve this, they needed to see the actions with millisecond-level latency.
However, for the current R&D task, this advantage of WebRTC is not decisive. At the same time, there are several drawbacks when using this technology on desktop devices.
WebRTC works only with H.264 or VP8 codecs. Typically, cameras are set up for streaming using H.264, but there can be exceptions. Therefore, we cannot claim that this tool is universal. Additionally, not all browsers have native support for these codecs, so some users may not have access to video surveillance functionality when choosing this technology.
Nevertheless, we consider streaming via WebRTC a promising direction, partly because this technology imposes a minimal load on the server. Therefore, we continue our R&D, and we will present the results regarding WebRTC in a separate article a bit later.
We’re empower your business with our technology expertise
Opting for this option involves on-the-fly conversion of video to MPEG1 Video from any codec (H.264, H.265, VP8, AV1, etc). This offers the following advantages:
While this option proved most suitable for our current R&D, it has one significant drawback – it heavily consumes server resources. Video and audio conversion from one format to another is a resource-intensive process.
To implement this option within the 2Smart Standalone platform, we created a new “RTSP to MPEG1VIDEO” bridge, as well as a dashboard widget called “RTSP-MPEG.”
The microservices architecture of the automation and monitoring platform 2Smart Standalone implies that users connect any necessary functionality by installing the corresponding bridge from the built-in market. To integrate video surveillance into the platform, we developed a new bridge with the task of converting the RTSP stream into MPEG-1. After conversion, the video should be rendered in a widget on the platform’s dashboard.
To add an RTSP stream, the user only needs to specify its URL in the settings. The bridge utilizes the ffmpeg solution for stream processing and a WebSocket server for data transmission.
Here is the ffmpeg command used by the bridge:
ffmpeg -rtsp_transport tcp -i rtsp:// -f mpegts -codec:v mpeg1video -r 30
More about the FFmpeg command options based on the solution’s documentation:
In summary, this FFmpeg command is capturing an RTSP stream over TCP, converting it to MPEG-TS format with MPEG-1 video compression at a frame rate of 30 fps. Make sure to replace <url> with the actual URL of your RTSP stream.
By executing this command, we can “broadcast raw data” to the UI. To achieve this, it is necessary to launch a WebSocket server to which the UI will connect, receiving data for video rendering.
Here’s what the pseudocode implementation of the bridge looks like:
// Start WS server on 6060 port
const wsServer = new ws.Server({
port: 6060
});
// Add callback function to handle new WS connections
wsServer.on("connection", (socket) => {
return onConnect(socket);
});
// Start ffmpeg command, that will be listening RTSP stream and convert
// incomming data to MPEG1 format
const stream = child_process.spawn("ffmpeg", [
'-rtsp_transport',
'tcp',
'-i',
'rtsp://',
'-f',
'mpegts',
'-codec:v',
'mpeg1video',
'-r',
'30'
], {
detached: false
});
// Handle incomming converted data and send to WS clients
stream.stdout.on('data', (data) => {
return wsBroadcast(data);
});
// Callback function for new connected clients
// Should send the “header” packet with specified video resolution
function onConnect(socket) {
const width = 320; //could be adjusted or parsed from the RTSP data stream
const height = 320; //could be adjusted or parsed from the RTSP data stream
let streamHeader;
// Send magic bytes and video size to the newly connected socket
// struct { char magic[4]; unsigned short width, height;}
streamHeader = new Buffer(8)
.write('jsmp') // STREAM_MAGIC_BYTES, must be 4 bytes
.writeUInt16BE(width, 4)
.writeUInt16BE(height, 6);
socket.send(streamHeader, { binary: true });
console.log(`New WebSocket Connection (${wsServer.clients.size} total)`);
}
// send MPEG1 data to all WS clients
function wsBroadcast(data) {
wsServer.clients.forEach(client => {
client.send(data);
});
}
To ensure video decoding and rendering in the dashboard widget, we opted for the JSMpeg solution. It provides a refreshing alternative to the complexities of WebRTC, offering a JavaScript-based Video Player with significant advantages. In contrast to the intricate landscape of WebRTC, JSMpeg simplifies the process of setting up video and audio formats that seamlessly function across all modern browsers. It comes with the advantage of no licensing fees, leveraging the expiration of all MPEG1 and MP2 patents. It also excels in delivering very low-latency real-time streaming with high frame rates and acceptable quality and bitrates.
The only challenge in creating the widget was inputting the stream URL because widgets in 2Smart Standalone are solely created through MQTT topic binding. Therefore, within the bridge, we established a sensor that conveys the path to obtain the stream.
Users need to input the path (“/service/xxxxxxxxx”) rather than the full URL. Why did we opt for this solution? In the bridge’s config (when adding it as an additional service in 2Smart Standalone), we can specify that it opens supplementary functionality through a public endpoint. For instance, the Grafana bridge opens a web interface, and the RTSP bridge opens a WS endpoint, enabling the widget to receive the video stream.
Hence, the widget can construct the full URL on its own:
`<protocol>://<hostname><sensor-value>`
<protocol>
” – ws or wss depending on the browser’s protocol (http or https).<hostname>
” – IP address or domain of the platform.<sensor-value>
” – value from the sensor, obtained from the bridge.As for the video delay when using our selected solution, it can be assessed in the video below. On the right side of the monitor, there is a website open with the current time at the moment the video was captured, and on the left side, there is a video surveillance widget in the 2Smart Standalone platform. As you can see, while it is noticeable, it is less than one second. This is not critical for the purposes of video surveillance.
It’s important to understand that we utilized the existing architecture of 2Smart’s automation platform, specifically leveraging our bridges. We added a new bridge to the market responsible for stream conversion and developed a new widget to showcase the already converted stream.
At this point, we have achieved an MVP for BMS with monitoring and video surveillance functionality, which can be further developed and optimized to meet clients’ needs.
We conducted testing on both home and professional cameras. Throughout the tests, we verified streams from domestic and industrial cameras, confirming that the majority of modern cameras with RTSP stream support can be easily integrated into the 2Smart Standalone monitoring system.
Our aim is to create a comprehensive Building Management System (BMS) that seamlessly combines device monitoring, remote management, automation, and real-time video surveillance functionalities.
We primarily utilize the Real Time Streaming Protocol (RTSP) for its widespread support among cameras and network video recorders (NVRs). This protocol allows us to receive video streams efficiently and reliably.
Our focus is on real-time streaming rather than recording or archiving video. We leverage existing RTSP streams from NVRs and employ protocols like MPEG1 Video for efficient browser rendering. Additionally, we ensure compatibility with various camera codecs and browser types.
To display video streams on the dashboard, we utilize the JSMpeg solution, which offers low-latency streaming with high frame rates and broad browser compatibility. Users input the stream path through MQTT topic binding, enabling seamless integration into the platform’s widget ecosystem.
One notable challenge was enabling browser-based rendering of the RTSP stream, which required on-the-fly conversion to MPEG1 Video format. This approach, while effective, can be resource-intensive and necessitates careful server management.
At present, we have achieved an MVP for BMS with monitoring and video surveillance functionality. Our solution has been tested with a variety of cameras, demonstrating compatibility with both domestic and industrial setups. We continue to refine and optimize the system to meet the evolving needs of our clients.
Share with us your business idea and expectations about the software or additional services.