Homie-based MQTT Convention 2Smart: Benefits and Practical Applications

Table of contents:

MQTT (Message Queuing Telemetry Transport) is a lightweight messaging protocol designed for efficient data exchange between devices. Alongside HTTP, MQTT stands as one of the primary protocols for message exchange at the application level in the Internet of Things (IoT) technology. Among its key advantages are properties such as low network overhead, implementation simplicity, capability for asynchronous communication, and scalability. Additionally, MQTT possesses two more characteristics that influenced the 2Smart team’s decision to adopt this protocol as the primary choice for their IoT projects:

1. The capability to interact directly with devices and deliver real-time messages. Deploying an MQTT broker in the cloud, on a local server, or elsewhere allows IoT devices to connect to it via internet connections, typically over Wi-Fi. Mobile applications, backend services, and interfaces also connect to this broker as clients, subscribing to topics of any connected devices. This enables two-way communication, where connected devices transmit data to the broker, and mobile applications, etc., send control commands to the devices. The broker serves as the intermediary in this communication, ensuring quick response to received commands since the device is always connected to the broker. This is not achievable with HTTP, as, in that case, a command will reach the device only when it connects to the server and sends a request.
2. The ability to connect any number of control interfaces to an IoT device, such as mobile and web applications. All interfaces will display the real-time status of each sensor and respond instantly to status updates. When a command is sent from one interface instance, we observe the corresponding sensor’s status change on all other instances with virtually no delay. This ensures synchronized and immediate updates across multiple platforms, enhancing the overall control and monitoring experience for users.

MQTT Conventions and Their Purpose

Within the realm of utilizing MQTT in specific projects or systems, certain agreements or conventions may exist, defining the rules for interaction and protocol usage.

One of the key components of MQTT conventions is topics. A topic is a string that serves as an address or name to which devices can subscribe for receiving or sending data. Topics categorize data into logical groups, providing convenient management and structuring of messages within the system. These conventions play a crucial role in facilitating organized communication and data flow, ensuring an efficient and streamlined operation of the MQTT protocol in diverse applications.

Topics form a hierarchical structure, separated by a forward slash (“/”). For instance, “home/living_room/temperature” represents the topic for the temperature in the living room of a house. A temperature sensor installed in the room publishes values to this topic. A mobile application or another user interface subscribes to this topic and retrieves the sensor-published data. This enables the user to monitor the real-time air temperature in the living room. Moreover, if the MQTT broker is in the cloud, monitoring is possible from any location worldwide with internet access.

In addition to the topic structure, MQTT conventions define message formats and other aspects to ensure consistency and compatibility across devices.

Today, there is an extensive array of MQTT conventions, broadly categorized into two groups: proprietary and common. Common conventions are universal standards utilized across a wide spectrum, adhered to by various applications and products as originally created by their authors. They are not tied to any specific project. On the other hand, proprietary conventions are specific to particular implementations or applications, employed exclusively within the scope of a defined project.

Developing a device according to a convention requires developers to delve into documentation and implement firmware, user interfaces, and backend services in accordance with the rules outlined therein.

For popular common conventions or products utilizing proprietary conventions, developers benefit from tools that simplify the aforementioned tasks. This is primarily where Software Development Kits (SDKs) come into play, offering an essential set of functions that can be leveraged to expedite and simplify the definition of the necessary firmware or mobile application business logic in alignment with the specific requirements of an MQTT convention.

MQTT Convention 2Smart: Homie with Extensions

2Smart is a developer team that coalesced around the idea of building a universal IoT platform in 2018, aiming to bring together a multitude of devices into a cohesive ecosystem. Our primary objectives were universality and the swift and straightforward expansion of the platform. Therefore, a crucial requirement was the selection of a set of rules for connecting devices and interfaces. We opted against creating our own convention, foreseeing that developers joining us in the future might already be engaged in projects based on various conventions. Consequently, we chose the Homie MQTT convention, which was already quite popular among IoT developers and continued to actively evolve.

The foundation of the 2Smart convention is the Homie version 3.0.1, which was current in 2018. We have also developed, and continue to develop, extensions to this convention to address specific tasks that cannot be resolved within the framework of the Homie base version.

More about Homie

Homie Convention is a set of abstract rules that describe procedures for automatic discovery, configuration, and use of devices and services. This convention is strictly typed, supporting a defined set of data types (float, string, boolean, enum, etc.) and provides clarity on how to interact with them.

This convention delineates the attributes of a device, its lifecycle with all possible status states. The hierarchical structure of topics in Homie is implemented through nodes, each having its attributes (name, type, properties) and properties. For instance, the topic hierarchy for a device like a coffee machine might look like the following, where the device comprises nodes, each incorporating specific sensors and telemetry:

• The “milk_tank” node has sensors for “volume,” “temperature,” an option to set the milk storage temperature, and a choice of milk type (regular, soy, peanut, etc.).
• The “coffee_tank” node includes sensors for “mass,” “temperature,” and an option to adjust the grind level.
• The “water_tank” node incorporates sensors for “volume,” “temperature,” and an option to set the target water storage temperature.
• The coffee machine itself, as a device, includes telemetry sensors like “wifi_level,” “battery_level,” “cpu_temperature,” as well as operational mode options.

An essential mechanism within the Homie Convention is the property command topic, utilized for controlling device properties using commands via MQTT. This mechanism allows for altering the state or executing specific actions on a device by utilizing external commands transmitted through the MQTT broker. For example, if a device has a property named “brightness” with an associated command topic “brightness/set,” another device or system can send a message to the “brightness/set” topic with a new value to adjust the brightness of the device. When an external device or system publishes a message to the “brightness/set” topic, a device implementing the Homie Convention should interpret this message and perform the corresponding action, such as adjusting the brightness.

Once the device modifies the brightness value, the updated value will be published to the “brightness” (without “set”) topic and all clients subscribed to the topic will instantly be informed of the change. This bidirectional communication mechanism allows for efficient and dynamic control of device properties within the MQTT ecosystem.

Extensions for Homie by 2Smart

As extensions to the Homie Convention, we have implemented the following mechanisms that were necessary for the realization of our projects:

1. Options for Device: A group of attributes, named “options,” has been added for a device. This group is used to store device settings, both editable and non-editable.
2. Telemetry for Device: Another group of attributes, named “telemetry,” has been added for a device. This group is used to store various telemetry and statistics related to the device.
3. State Attribute for Node: Each node has a “state” attribute, similar to a device, providing information about the current state of the node.
4. Options and Telemetry for Node: A node also has groups of attributes, “options” and “telemetry,” similar to those of a device, allowing for node-specific settings and telemetry data.
5. Validation Mechanism: A mechanism for validating the set values has been incorporated into the application. In case of receiving incorrect messages, the system sends a validation error notification to a specially designated topic. Clients subscribed to this topic will receive the notification, gaining insight into the reason for the command failure.
6. Fixed Telemetry: Certain telemetry attributes such as “signal” and “battery” have been strictly defined and fixed. These are relevant to the majority of devices encountered in our projects.

In addition to these modifications, we expanded Homie’s native list of mandatory states required for a device to be considered valid in our system. This list, beyond those included in Homie, encompasses additional states for devices, nodes, and properties.

Furthermore, we devised the mechanism of the device heartbeat, utilized at 2Smart for monitoring the connection of devices to the network. Diverging from the MQTT-native “last will” mechanism, our heartbeat relies not on the state of the TCP socket with the device but on messages sent periodically by the device to signal its online status. This approach allows the system to promptly detect when a device disconnects from the broker. Additionally, we integrated additional functionality into the heartbeat for handling events related to device disconnection from the system.

Lastly, we created our Notifications API, enabling devices to send real-time notifications. Additionally, we designated a specialized topic for device versioning to facilitate over-the-air firmware updates.

Practical Application of the 2Smart Convention

Homie-based 2Smart convention described above is a standard across all products developed by the 2Smart team. This convention is employed in the 2Smart Standalone automation platform, the “Propuskator” (Access Control System), and the 2Smart Cloud IoT platform. Notably, 2Smart Cloud serves as a scalable turnkey solution delivered to clients through a white-label model. This means that devices from various manufacturers, often with firmware developed independently, can connect to the platform. However, adherence to the Homie convention with 2Smart extensions is essential for seamless integration with the platform.

All devices interacting with our platforms inherently benefit from the additional functionalities we have integrated into our solutions. This includes features such as storing historical sensor values, notification mechanisms, remote control capabilities, and more.

SDK

To simplify the lives of developers and expedite the development of services and interfaces in accordance with the 2Smart Convention, we have created an SDK based on Node.js. This SDK not only facilitates the creation of firmware logic for devices but also enables the publication of messages to relevant topics following our convention. Additionally, we’ve implemented boilerplates in C and C++ for various chips, further streamlining the development process and allowing developers to swiftly implement devices compatible with the 2Smart Convention.

The Node.js-based SDK serves as a comprehensive toolkit for developers, offering tools and resources to seamlessly integrate their devices with the 2Smart ecosystem. With these SDKs and boilerplates, developers can focus on implementing the unique functionalities of their devices while ensuring compatibility with the standardized communication protocols established by the 2Smart Convention. This approach accelerates the development lifecycle, promotes consistency, and empowers developers to efficiently bring their IoT solutions to market.

Device Self-Declaration

Thanks to standardization within a unified convention, every device provides information about itself, its properties, and its interfaces. This enables other devices or systems to automatically identify and interact with it without prior configuration. Any developer working with such a device understands how to control it, the logic for reading sensor values, sending commands, receiving notifications or validation errors, and determining the device’s current status (lost, ready, disconnected, etc.).

The strict data typing and topic structures also eliminate the need to develop a control interface for each new device, as a generic interface can be created once and reused. For example, if a device adheres to the Homie convention, we understand that a boolean sensor can only have true or false states, and it requires a switch for control. The same logic applies to sensors with any other data type.

Thus, device self-declaration facilitates the automatic detection and integration of new devices into the system. Importantly, self-declaration allows the creation of universal products, rendering universal interfaces, and working consistently with all devices, regardless of the complexity of the tasks they solve. The practical implementation of this concept will be demonstrated below.

Public Implementations of Interfaces

Homie offers specific standards and conventions for describing devices and their interfaces. When referring to “public implementations of interfaces,” it means that device interfaces are described in accordance with Homie standards and can be easily understood and used by other devices that support Homie.

For example, Homie provides structured ways to define device nodes and properties. Public implementations of interfaces allow devices to automatically provide information about their structure and capabilities, facilitating interaction with other devices or systems on the network.

Examples of Homie interfaces that can be publicly implemented:

• Sensors: Temperature, humidity, light level, etc.
• Control: Switches, relays, brightness adjustment, etc.
• Status: Information about the current state of the device.

When a device provides public implementations of interfaces, it becomes more convenient to integrate it into smart home systems, the Internet of Things, and other applications that support Homie.

Examples of public implementations of interfaces include projects like FlutterHomie, whose mobile application allows the discovery and management of Homie devices. Additionally, home automation software such as HomeAssistant, openHAB, or Node-RED can be mentioned as platforms that support public implementations of Homie interfaces.

2Smart MQTT Convention: Practical Usage

Developers and QA engineers at 2Smart actively utilize the MQTT Explorer application, which stands out as one of the best tools for monitoring and interacting with MQTT brokers. MQTT Explorer provides a convenient way to track MQTT messages, subscribe to topics, send messages, and much more. Let’s make use of it and see how real devices, connected to the 2Smart broker according to our convention, declare themselves.

There are two devices linked to the account: a lamp with multiple effects and an access controller named “Propuskator.” In MQTT Explorer, we can observe their identifiers:

Opening the details of one of the devices, we see its name – 2Smart Lamp, along with the current state, microcontroller model, and other information.

Here is the telemetry from the device, which is strictly typed (each parameter includes data type, name, retained, and settable parameters):

We also see nodes with names firmware, lenta, notifications, and rstbutton. The lenta node represents the sensors of the lamp (brightness, color, mode, state, text). It also includes mandatory attributes like name and state.

Opening the details of the state sensor, we can see that it is a boolean sensor with the current value of true, and it is editable. This information is sufficient to understand that we are dealing with a switch, and its state can be changed, with the firmware reacting accordingly.

Similarly, we can see that brightness is an integer sensor with values ranging from 0 to 100, and it is both retained and settable. The color sensor operates with the color data type, taking values in RGB format, and is also retained and settable.

Similarly, we can explore the nodes and sensors of the “Propuskator” access controller and see detailed information about them, all presented within the same MQTT convention. It’s worth noting that this is a different device addressing entirely different functionalities.

Thanks to the strict structure and typing within the MQTT convention, it becomes possible to automatically create interfaces for interacting with devices – to retrieve telemetry from them and send control commands.

Utilizing the Convention in 2Smart Standalone

Our initial product in the lineup was the 2Smart Standalone automation platform. Let’s connect the same two devices we discussed earlier – the lamp and the access controller. Right after adding these devices to the dashboard, cards appear featuring controls for interacting with the sensors.

Thanks to the clear structuring, we promptly observe the device’s name, its current status, and the Wi-Fi signal level at its installation location.

We can open a modal window displaying all available telemetry from the device.

The device card is logically divided into separate sections based on nodes, each including controls for all sensors within that node.

Controls are automatically assigned based on the information sensors publish about themselves through the self-declaration mechanism. For instance, if it’s a boolean sensor with a retained parameter, a toggle switch control is assigned to it.

If it’s a non-retained boolean sensor, a Push Button is assigned to it.

If it’s a color sensor that is retained, the automation platform assigns a color picker control to it. If it’s a sensor with enum data type, it gets a control in the form of a drop-down list. Readings from sensors with non-editable string or integer types are displayed as numbers or text. In other words, for each possible scenario, we have provided standard controls that are displayed on the automation platform’s dashboard interface.

The next step in enhancing the user interface in 2Smart Standalone was the introduction of widgets. Standard controls for sensors of different types are simple, clear, and functional, but they may be perceived as not the most modern and not meeting users’ design preferences for control elements. Additionally, in various scenarios, sensors with the same set of parameters may have different control elements, which would appear more logical.

Widgets are UI components designed to interact with sensors that have specific sets of properties. When configuring a widget, users can see the full range of compatible sensors and assign it to any of them.

We’ve developed a library of widgets that allows users of 2Smart Standalone to customize dashboards for device control, monitoring, and home automation settings.

Utilizing the Convention in ACS Propuskator

The next major project our team worked on was the cloud-based access control system called “Propuskator”. The access controllers we developed also communicate with the cloud using the MQTT protocol following the 2Smart convention based on Homie.

Here’s what the automatically generated settings and monitoring interface for the Propuskator access controller looks like in the ACS software. Each sensor is assigned a specific control depending on its type.

In fact, for the Propuskator project, we reused the developments created for the 2Smart Standalone automation platform, turning them into a UI kit that can be applied in any other IoT projects operating based on the 2Smart MQTT convention.

For comparison, here’s how the control interface for the same Propuskator controller looks when connected to both the automation platform and the cloud-based ACS. The only difference is how we’ve grouped the controls in the ACS, distributing them across tabs and arranging them more compactly. But the controls themselves remain the same.

If we connect to the MQTT broker of the Propuskator access control system using MQTT Explorer, we will also see that the device structure is exactly the same as in the examples above with the automation platform.

Utilizing the Convention in 2Smart Cloud

The 2Smart Cloud platform also utilizes the MQTT convention described above. However, during its development, we elevated the interaction with device sensors to another level in the interface.

The only element we reused is the modal window displaying the list of device sensors. It remains the same as in the 2Smart Standalone home automation software. The folders in the modal window represent nodes, and their contents include the sensors belonging to those nodes. Standard telemetry metrics such as IP and Wi-Fi signal level are separately displayed.

However, unlike 2Smart Standalone and the Access Control System “Propuskator,” 2Smart Cloud does not use controls. Instead, compatible widgets are immediately assigned to sensors, allowing end-users to manage devices from the mobile or web application. Upon adding a device, a widget is assigned to each sensor, but developers have the flexibility to reassign them, choosing more suitable widgets from the library.

The 2Smart Cloud widget library is extensive and continually updated. To facilitate developers in finding appropriate widgets, we leveraged the benefits of the self-declarative device mechanism and added filters to the library.

Developers can sort widgets based on data type:

They can use sorting based on editability:

Finally, the developer can choose the specific sensor of the device for which they need to configure a widget, and the library will display only the compatible widgets:

Cross-Platform Compatibility

Since the same MQTT convention is employed across all 2Smart projects, the devices essentially become cross-platform. They are not only easy to integrate into each platform, but also, any changes in the state of a particular sensor are instantly reflected in different interfaces.

Additional Functionality Based on the MQTT Convention

The extensions to the Homie MQTT convention that we have developed enable the use of the services described below, enhancing the functionality of all software platforms created by 2Smart.

Device Status Tracking

As mentioned earlier, we introduced the heartbeat mechanism to monitor the device’s connection to the network and detect any sudden disconnection. Since a device that suddenly goes offline cannot report this itself, we utilize the heartbeat mechanism to notify users. After the communication with a device is interrupted, a message is sent to the mobile application or another interface to inform the user. Similarly, the user receives a notification when the device comes back online.

Historical Data Collection

To allow users to monitor the historical data of sensor values, we introduced a service that subscribes to sensors via MQTT and records all their values in InfluxDB, associating each value with the timestamp when it arrived at the broker. Subsequently, users can view and analyze this history in a convenient format using the open-source analytics and interactive visualization web application Grafana.

Firmware Over-the-Air Updates

As an extension of the Homie convention by 2Smart, we have implemented a specialized topic where the device’s firmware version is specified. This feature facilitates the delivery of firmware updates over the air.

Conclusion

The Internet of Things (IoT) is a technology that continues to transform our lives. Unfortunately, this transformation is not happening as quickly as desired, and one of the reasons is the lack of standardization. Different development teams create amazing things and unique products, but it’s often very challenging or impossible for users to unite them into a single ecosystem. At 2Smart, we advocate for IoT standardization that would enable the seamless collaboration of diverse devices and services, regardless of manufacturers and brands.

Standardizing IoT could simplify the adoption and compatibility of various devices, creating a cohesive and easily manageable Internet of Things ecosystem. Unified standards would facilitate rapid deployment and scaling of systems since devices and applications adhering to standards could easily interact with each other and integrate into a shared infrastructure. Standards would establish a common set of rules and norms, simplifying the development of new devices and applications and increasing the confidence of consumers and enterprises in using IoT technology. Ultimately, standardization could reduce the resources spent on IoT technology development and implementation.

A unified MQTT convention could serve as the foundation for such standardization, ultimately leading to the best possible user experience. The Internet of Things could genuinely enhance people’s lives rather than merely existing as isolated manifestations often unrelated to each other. Implementing unified standards in IoT could result in broader acceptance of this technology, lower development and support costs, increased security, and smoother integration into people’s everyday lives.