About MQTT
MQTT is a Client-Server publish/subscribe messaging transport protocol. A central service called an MQTT Broker allows clients to connect and either publish or subscribe to events. This architecture decouples the producers of data from the consumers of data - one or more consumers have a subscription to a message topic and receive those messages, but they do not need to know the source of the messages.
For ease of organization, MQTT events can be sub-divided into "topics” that clients can subscribe to Clients thus have the flexibility to subscribe to events from particular types of sensors, as well as different categories of information reported by those sensors. In the Itron environment, our hosted MQTT Broker captures data reported by all sensors in the field, and publishes that data to client subscribers implemented by our partners. This page describes the process for connecting client subscribers to our MQTT Broker.
Itron strongly encourages integrators and developers to consume alerts/alarms and periodic measurements through MQTT instead of CoAP OBSERVE. Major advantages of adopting MQTT include the following:
- Asynchronous, event-driven interaction model
- Eliminate the difficulty of maintaining a client/server CoAP session across a self healing RF mesh due to changing IPv6 addresses when this occurs
- Familiar security and firewall demands
- MQTT uses TCP/SSL similar to HTTPS, which is easier for Operations to administer than CoAP UDP
Note that the use of MQTT for reporting periodic measurements from sensors does not preclude the use of CoAP for other, more interactive operations. Back office applications should still implement CoAP for command and control of the sensor as well as on-demand requests to the sensor.
The Itron back office includes an MQTT broker that allows applications to publish messages on topics and other applications to subscribe to messages on particular topics. Itron uses a Java based enterprise software called HiveMQ. The following image describes how HiveMQ communicates with Gateway and the MQTT client.
ADDITIONAL RESOURCES
Connecting a Client: Connection Security
Itron requires all clients to use TLS to connect to the HiveMQ MQTT broker. The client connection URL must be be in the following format:
ssl://{MQTT_BROKER_HOST}:{MQTT_BROKER_SSL_PORT}
The TLS port is typically 8883, although it is configurable through deployment properties.
CONNECT Message
MQTT clients initiate a connection to a broker by sending a CONNECT message with various fields. HiveMQ supports all of these options as defined by the following MQTT protocol:
Choosing a ClientID
The client identifier (short ClientId) is an identifier of each MQTT client connecting to an MQTT broker. It must be unique per broker. The broker uses it for identifying the client and the current state of the client.
If an application can have multiple instances connecting to the MQTT broker, each instance must use a unique clientID. If the application cannot, the connections will steal the clientID from each other and repeatedly break each other’s connection. Consider avoiding this by using a clientID prefix (to make the clientID human readable) but padded with a randomized string. According to the MQTT RFC section 3.1.3.1, the clientID must be between 1 and 23 characters long.
You can also namespace clientIDs to an application to avoid collisions with other application MQTT clients. Consider choosing a prefix that starts all clients from a particular application.
Example
The Gateway application has two MQTT client connections that publish messages: a CoAP-to-MQTT bridge and a presence reporter. To help make these unique, each clientID begins with the string "gw" to identify that it comes from Gateway. The two clientID prefixes are:
- gw-coap-mqtt-bridge (19 characters)
- gw-presence (11 characters)
At configuration time, these two prefixes are padded to reach 23 characters to help avoid colliding with the connections from other Gateway instances. The actual clientIDs used to connect to the broker look like:
- gw-coap-mqtt-bridge-112
- gw-presence-112b25b-5f
Note that Gateway generates the randomized clientID at configuration time; not at start up time. This allows a Gateway instance to have a stable clientID across restarts, which is important for persistent session connections and some client side message buffering strategies.
Clean Session
The clean session flag tells the broker whether the client want to establish a persistent session. In a persistent session (CleanSession = false), the broker stores all subscriptions for the client and all missed messages for the client that subscribed with a Quality of Service (QoS) level 1 or 2. If the session is not persistent (CleanSession = true), the broker does not store anything for the client and purges all information from any previous persistent session.
Persistent Sessions and QoS 0
Connecting to the MQTT broker with a persistent session causes the broker to queue messages while the client is offline, but only for QoS 1 or QoS 2. Messages from subscriptions established with QoS 0 are not queued while the client is disconnected.
Clients are expected to use the cleanSession value set to true unless otherwise negotiated.
CONNECT Authentication
As of version 1.1.0 of the HiveMQ-Plugins package (SSC 2.6+), clients authenticate their CONNECT request using a valid JWT token. The JWT string must be provided in the CONNECT message password field (the username field is optional in this mode). The broker validates the signature of the JWT token against a configured certificate or shared secret, and validates that the token is not expired.
JWT Token Authentication for the Itron Developer Program
The Itron Developer Program uses the Starfish Stage network and JWT token authentication for MQTT connections to the broker on this network. There is a token API which is called using your API clientID/secret key. This returns a token with all the necessary information to connect to the MQTT broker. The token JWT string is then provided in the CONNECT message password field (the username field is not needed).
Further Developer Program API authentication details including the token API can be found on the API Overview page.
CONNECT Authorization
As of version 1.1.0 of the HiveMQ-Plugins package (SSC 2.6+), the MQTT broker limits the actions that clients are allowed to perform based on the credentials they provide in the CONNECT message. The defined privileges per client are:
- Activity - whether the client is allowed to publish, subscribe, or both.
- QoS - The MQTT QoS level the client is allowed to publish/subscribe at.
- Message Retention - Whether the client is allowed to publish retained messages (applies only to clients permitted to publish).
- Topic - The topic prefix the client is allowed to publish/subscribe to.
Predefined combinations of these privileges are built into CAAS roles that apply to different types of clients:
|
Role Name
|
Activity
|
QoS
|
Retained Allowed
|
Topic
|
|---|---|---|---|---|
|
BUBBLE_UP_ALARM_CONSUMER
|
Subscribe
|
0 or 1
|
N/A
|
{tenantId}/alert/#
|
|
BUBBLE_UP_READ_CONSUMER
|
Subscribe
|
0
|
N/A
|
{tenantId}/read/#
|
|
DEVICE_PRESENCE_CONSUMER
|
Subscribe
|
0
|
N/A
|
{tenantId}/last_heard/#
|
For Itron's use of MQTT, external clients are not authorized to publish events to the MQTT broker. Clients can connect using roles appropriate to their function, such as BUBBLE_UP_ALARM_CONSUMER for clients that need to process critical alarms from sensors, and BUBBLE_UP_READ_CONSUMER for clients that need to consume bubble-up sensor data.
Client role assignment is handled differently depending on the authentication mechanism used:
- CAAS - the CAAS roles for the provided CAAS credentials are loaded by the HiveMQ plugin when the CONNECT message is received and are used for privilege enforcement for the duration of the client's session. Each MQTT client can have one or more roles assigned to it and the appropriate privileges will be granted. Applications can also establish more than one MQTT client connection (as long as they use unique clientIDs), each with the appropriate roles for that client.
- JWT - the CAAS roles listed by name in the JWT "mqttScope" claim will be loaded and granted to the client.
Permissions Violations
Clients that attempt to perform an action that they are not authorized for will be rejected by the broker. The rejection behavior depends on the activity:
- Unauthorized subscription - The broker will return a SUBACK response with a "subscribe failed" error code of 0x80 (128).
- Unauthorized publication - As described in section 3.3.5 of the MQTT spec, there is no message mechanism in the MQTT protocol to inform the client that a publication is rejected. The broker must either make a positive acknowledgement of the publication according to the normal QoS rules or close the Network Connection if the publication is rejected.
Last Will
For the purposes of Itron's use of MQTT brokers, clients are not expected to provide any "lastWill" information.
SUBSCRIBE Message
Subscribing to topics
Clients are expected to subscribe to as many topics (in wildcard form) as required and authorized. If a client subscribes to a topic they are not authorized to receive, they might receive a failure code for that topic.
A client will often provide a wildcard form topic subscription. For example, device data might be published with a topic format of ssni/alert/<mac address>/<device type>. A single level wildcard subscription like "ssni/alert/+/gas" would subscribe to gas device alerts across all possible MAC IDs. Alternatively, a multi-level wildcard subscription like "ssni/alert/#" would subscribe to alerts across all MAC IDs for all device types.
Subscription QoS
Clients can select the following quality of service (QoS) options when publishing or subscribing to topics:
- QoS 0 is often called “fire and forget” and provides the same guarantee as the underlying TCP protocol.
- QoS 1 guarantees that a message is delivered at least one time to the receiver using an ACK mechanism in the MQTT protocol.
- QoS 2 is the highest level of service in MQTT and guarantees that each message is received only once by the intended recipients.
QoS 1 and QoS 2 require additional round trips between the broker and the client and therefore introduce additional latency to the messages. They also require additional state tracking on the broker side, causing CPU and heap usage to increase.
Itron recommends QoS 0 for most use cases. QoS 1 can be used for more critical traffic but should only be used on lower volume topics.
PUBLISH Message
For Itron's use of MQTT, external clients are not authorized to publish at all.
MQTT Client: Mqtt-Spy
Mqtt-Spy is an open source utility for connecting to an MQTT broker and publishing or subscribing to topics. The tool provides an easily navigable user interface for managing connections and subscriptions, and can be scripted to publish messages to the broker for testing. Itron recommends this tool for for testing and development.
Connection Settings
This section provides an example of setting up a connection to the Itron HiveMQ broker configuration using MQTT-Spy.
- Server URI - the host name of the MQTT broker, should match SHARED_MQTT_BROKER_URI.
- Client ID - a unique client ID that will not collide with other clients.
- Clean Session - check the box to establish a clean session (no subscription persistence on the broker).
- Reconnect on failure (optional) - if this is checked then MQTT-Spy will automatically attempt to reconnect to the broker when the connection is lost.
The Itron HiveMQ configuration requires valid CAAS credentials or JWT token string to be provided when connecting to the broker. These can be entered under the "Security" tab in the "User auth" section.
- User name - the CAAS user to connect as when using CAAS authentication; leave blank with JWT token authentication
- Password - the password for that CAAS user when using CAAS authentication; JWT token string when using JWT token authentication
The Itron HiveMQ Configuration requires that connections are established using TLS security. This can be configured under the "Security" tab in the "TLS" section.
- Protocol - select TLSv1.2
Paho Library
Eclipse Paho provides a Java MQTT client library that can be integrated into applications that need to connect to the broker. It's a nice API and is easy to work with. For example, establishing a connection to the broker can be as simple as the following:
|
public MqttAsyncClient createAndConnectClient() { try { MqttDefaultFilePersistence persistence = new MqttDefaultFilePersistence(getPersistenceDirectory()); MqttAsyncClient mqttClient = new MqttAsyncClient(getBrokerUri(), getClientId(), persistence);
DisconnectedBufferOptions bufferOptions = new DisconnectedBufferOptions(); bufferOptions.setBufferEnabled(true); bufferOptions.setPersistBuffer(true); mqttClient.setBufferOpts(bufferOptions);
MqttConnectOptions connectOptions = new MqttConnectOptions(); connectOptions.setCleanSession(isUseCleanSessions());
// We're relying on the library to reconnect to the broker if we disconnect. The documentation says this will // initially wait 1 second before it attempts to reconnect, for every failed reconnect attempt, the delay will // double until it is at 2 minutes. connectOptions.setAutomaticReconnect(true); connectOptions.setKeepAliveInterval(2); connectOptions.setUserName(getCaasUser()); connectOptions.setPassword(getDecryptedCaasPassword().toCharArray());
// Default connection timeout is 30 seconds. mqttClient.connect(connectOptions).waitForCompletion();
logger.info("MQTT client connected to broker %s as client '%s'", getBrokerUri(), getClientId());
return mqttClient; } catch (MqttException e) { logger.error("Failed to connect to MQTT broker: %s", e, e.toString()); throw Throwables.propagate(e); } } |
For more information about Gateway's MqttClientConfiguration class, see GitLab.
From there, you can use Paho's publish() API to submit messages to the broker. To receive messages, the application should call the subscribe() API and configure the client with an MqttCallback instance where the messageArrived() callback will be invoked.
MQTT.fx Client
MQTT.fx is a free Windows-based MQTT client that can receive MQTT messages published by Milli-enabled sensors through the Starfish platform. MQTT.fx provides an intuitive user interface that allows you to subscribe to MQTT messages published by your Milli devices, as well as publish messages to other MQTT subscribers. MQTT messages received from Starfish are displayed in a window, as well as sent to your PC as notifications on the desktop. The list of MQTT messages received can also be exported to a folder. MQTT.fx does not run as a Windows service, so the MQTT broker connection is only maintained as long as the application is open. In addition, unlike Itron's Java MQTT client reference application, messages received by MQTT.fx are not uploaded to the Starfish Data Platform.
Install and Configure MQTT.fx
Download the MQTT.fx Windows client application here. Once the client application has been installed on your PC, open the application, and set up a connection to the Starfish MQTT broker using the instructions here.