Me: This module gives you the flexibility to have multiple sensors sending from one module. This can be configured by the InstanceCount. The existing solution allows you to set the send interval in seconds only. With this solution, you can specify this in milliseconds.
You: Now I understand. Thank you
Me: Sure. Feel free to use the code and adjust it to your needs.
The solution contains a tweaked Simulated Temperature Module for Azure IoT Edge. It continuously creates simulated temperature data and sends that to an Azure IoT Hub.
This post shows a way to find out how many IoT (Edge) devices have been provisioned by a specific enrolment group within the last x minutes.
The solution could be much simpler if I just wanted to know how many devices are registering themselves. In this case the build in metrics are enough to get that information.
IoT Hub Metrics
The use case required a more sophisticated solution that is able to reflect the tenants, identified by tags.
Solution Architecture
Device Provisioning Service
Different Enrolment Groups separate devices in this scenario into Tenants. To be able to identify the customers, an initial tag CustomerId is added to the enrolment group. It is then applied to the devices that are create by DPS in the IoT Hub.
The metrics from DPS did not allow me to distinguish the tags/customers. But IoT Hub will make them available and offers events for newly created devices.
IoT Hub
Within IoT Hub I created an event subscription, that passed on all necessary events to an EventHub.
Event Subscription in IoT Hub
The event will include the device twin, which has been prepopulated with the tags specified in the enrolment group.
Device Twin in IoT Hub
As seen in the architecture diagram, Event Grid has been connected to an Event Hub. #plugandplay 😉 It will fire an event with the documented schema: Azure IoT Hub and Event Grid | Microsoft Docs
Event Hub
Why the additional Event Hub? Event Grid cannot be used as input for an Azure Stream Analytics Job and Event Hub is the universal connector in this case.
You can use the smallest tier (which is Basic) as there is not a lot of events flowing through it. The default 2 partitions is also fine.
As you can see, the query is pretty simple and can be adjusted easily.
Azure Stream Analytics Job Query
The example uses a blob storage as output, but you can choose to write to an Azure Function or whatever you want to do with the know how that one customer has onboarded lots of devices in a short period of time.
In this post I would like to show some tweaks you can (and might need to) apply to influence the behavior of your IoT Edge device, when it comes to message retention on devices that are limited in resources.
The setup of this scenario is not uncommon, as it uses a module to retrieve telemetry from machines, parses them in another module and sends the messages to an IoT Hub.
The problem
After a while the device is not sending data anymore and is not accessible via SSH. The logs reveal lots of message still in the queue.
Lots of messages in the queue in edgeHub logs
But why? And how can I find out what causes the problem? Spoiler: Disk full 🙁
Troubleshoot
Looking at logfiles helps a lot – if you have access to the logfiles. Fortunately IoT Edge can expose data in the Prometheus exposition format for the edgeHub and edgeAgent. These endpoints are enabled by default for IoT Edge 1.0.10 (upgrade to this version if you haven’t) and can be enabled for 1.0.9.
The data can then be uploaded to Log Analytics for further analysis and to create alerts with a sample metrics-collector module.
For analyzation and to display the metrics, you can use a Workbook in Azure Monitor.
Azure Monitor Workbook with edgeHub log extract
In this particular case I could see that the available disk space was going down, down, down until the whole device did not respond anymore (no SSH access possible, no data sent to Azure).
What to change?
Adding more space to the disk was not an option. Other solutions needed to solve the issue. There are 2 options I looked at and adjusted to be a better fit for the usage scenario and resource limitation.
After tweaking the settings, the following graph shows that now the device cleans up data before the disk runs full.
I can not give you values for you particular setup. You’ll need to figure them out for your setup depending on the amount of messages going though the Edge device and hardware sizing. Here are some pointers to settings which you might want to investigate, if you hit a similar problem on your devices:
The above image shows setting for RocksDB (orange: 512MB, blue, 128MB, green 256MB). With the default setting the device is running out of disk space.
What can I do to prevent the device crashing?
Well, it depends 🙂 You can find a setting from the above that will prevent a full disk for a known scenario. But if you don’t know which modules with which setting is deployed?
In this case an alarm for low disk space is an option. It then needs to trigger a function that calls a method on the device to restart the edgeHub. This will clear the cache.
Finally I got it up and running by creating a docker group, adding iotedge to it and changed the group ownership of the /var/run/docker.sock file sudo chown root:docker /var/run/docker.sock
This post is meant to be found via search engines if you (or me again) has the same startup problems.
My context: Ubuntu 20.04 with snap installed docker
In this post I want to show how to use properties that are added to messages that IoT devices are sending to Azure IoT Hub in Stream Analytics. And while talking about properties, let’s even use message enrichment 🙂
Stream Analytics Architecture
Sample Message
The green properties will be added by the Message enrichment feature of IoT Hub, as the data is not most likely not known on the IoT device or does not need to be transferred with each message.
The code that sends the message with the alert property has been adjusted to this:
string dataBuffer = $"{{\"messageId\":{count},\"temperature\":{_temperature},\"humidity\":{_humidity}}}";
using (var eventMessage = new Message(Encoding.UTF8.GetBytes(dataBuffer)))
{
eventMessage.Properties.Add("temperatureAlert", (_temperature > TemperatureThreshold) ? "true" : "false");
Configure IoT Hub
Device Twin
In most cases the IoT (Edge) device does not know which customer it is associated, as it does not need to know. For further processing of the data – or for device management – this information is relevant. Therefore we add this information to the device twin in Azure IoT Hub.
The property names do not need to match the desired properties that will be added via message enrichment. You can choose a structure that fits best.
Message Enrichment
We want to add the customer number and id from the device twins to the message before it is being passed along to an endpoint.
Message Enrichment settings in IoT Hub
As you can see the name of the property that is added does not need to match the name of the twin properties. Make sure you add the message enrichment to the right endpoint(s). You can decide to add different properties to messages that are routed to different endpoints.
Azure Stream Analytics
In the Stream Analytics job we use a SQL like query to filter the incoming message stream and route the messages to endpoints. The query will work fine as long as you use only the columns that are in the body of the messages (like “temperature” or “humidity” in this examle).
To be able to use the values in the properties, we need to use the GetMetadataPropertyValue function. Please take not of the sentence on the docs page: This function cannot be tested on the Azure portal using sample data
Query
SELECT
GetMetadataPropertyValue([IoTHub-Messaging], '[User].[temperatureAlert]') AS temperaturealert,
GetMetadataPropertyValue([IoTHub-Messaging], '[User].[CustomerName]') AS customername,
GetMetadataPropertyValue([IoTHub-Messaging], '[User].[CustomerId]') AS customerid,
*
The first three columns are our property and message enrichment columns while the other columns are all added as well.
Output
Let’s assume we want to add all message to a storage account where the customer id is part of the path.
Stream Analytics Blob storage output
This will work, as we added the customerid column in the query and it can be used for the path. Remember this is a demo and we only use the customerid as part of the path.
In the architecture diagram at the beginning of the post an Alert route is drawn. You can achieve this by adding a second query to the job which routes certain messages to that output.
With Azure IoT Edge you can deploy modules (also known as Docker Containers) to a server. I’ve created a sample solution on GitHub that deploys a module which monitors the temperature of the harddisk that the server is running on.
Today my VisionAI DevKit was not deploying a module. In the logs (sudo journalctl -u iotedge -f) I could see the deployment was received:
Successfully pulled image machinelearndfd8df7d.azurecr.io/mobilenetimagenet:3 Creating module VisionSampleImagenet… Could not create module VisionSampleImagenet caused by: No such image: machinelearndfd8df7d.azurecr.io/mobilenetimagenet:3
Strange. During troubleshooting I started docker images and saw a lot of older images and versions. After deleting a log of them with docker image rm xyz the deployment succeeded and the module started. 🙂