Category: Uncategorised

Count provisioned devices by DPS

This post shows a way to find out how many IoT (Edge) devices have been provisioned by a specific enrolment group within the last x minutes.

The solution could be much simpler if I just wanted to know how many devices are registering themselves. In this case the build in metrics are enough to get that information.

IoT Hub Metrics

The use case required a more sophisticated solution that is able to reflect the tenants, identified by tags.

Solution Architecture

Device Provisioning Service

Different Enrolment Groups separate devices in this scenario into Tenants. To be able to identify the customers, an initial tag CustomerId is added to the enrolment group. It is then applied to the devices that are create by DPS in the IoT Hub.

{
  "tags": {
    "CustomerId": "AnotherCustomer"
  },
  "properties": {
    "desired": {}
  }
}

This tag can then be used for e.g. message enrichment. I’ve written previously about using it: https://www.hezser.de/blog/2020/05/13/properties-for-iot-messages-in-azure-stream-analytics/ (opens in a new tab)

The metrics from DPS did not allow me to distinguish the tags/customers. But IoT Hub will make them available and offers events for newly created devices.

IoT Hub

Within IoT Hub I created an event subscription, that passed on all necessary events to an EventHub.

Event Subscription in IoT Hub

The event will include the device twin, which has been prepopulated with the tags specified in the enrolment group.

Device Twin in IoT Hub

As seen in the architecture diagram, Event Grid has been connected to an Event Hub. #plugandplay 😉 It will fire an event with the documented schema: Azure IoT Hub and Event Grid | Microsoft Docs

Event Hub

Why the additional Event Hub? Event Grid cannot be used as input for an Azure Stream Analytics Job and Event Hub is the universal connector in this case.

You can use the smallest tier (which is Basic) as there is not a lot of events flowing through it. The default 2 partitions is also fine.

Stream Analytics Job

I chose Stream Analytics for the further analysis of the events, because it offers an out-of-the box functionality for queries on time windows: Introduction to Azure Stream Analytics windowing functions | Microsoft Docs

As you can see, the query is pretty simple and can be adjusted easily.

Azure Stream Analytics Job Query

The example uses a blob storage as output, but you can choose to write to an Azure Function or whatever you want to do with the know how that one customer has onboarded lots of devices in a short period of time.

Azure IoT Edge on constraint devices

Introduction

In this post I would like to show some tweaks you can (and might need to) apply to influence the behavior of your IoT Edge device, when it comes to message retention on devices that are limited in resources.

The setup of this scenario is not uncommon, as it uses a module to retrieve telemetry from machines, parses them in another module and sends the messages to an IoT Hub.

The problem

After a while the device is not sending data anymore and is not accessible via SSH. The logs reveal lots of message still in the queue.

picture with logfile lines like Cleaned up messages from queue for endpoint iothub and messages from message store
Lots of messages in the queue in edgeHub logs

But why? And how can I find out what causes the problem?
Spoiler: Disk full 🙁

Troubleshoot

Looking at logfiles helps a lot – if you have access to the logfiles. Fortunately IoT Edge can expose data in the Prometheus exposition format for the edgeHub and edgeAgent. These endpoints are enabled by default for IoT Edge 1.0.10 (upgrade to this version if you haven’t) and can be enabled for 1.0.9.

The data can then be uploaded to Log Analytics for further analysis and to create alerts with a sample metrics-collector module.

For analyzation and to display the metrics, you can use a Workbook in Azure Monitor.

Azure Monitor Workbook with edgeHub log extract

In this particular case I could see that the available disk space was going down, down, down until the whole device did not respond anymore (no SSH access possible, no data sent to Azure).

What to change?

Adding more space to the disk was not an option. Other solutions needed to solve the issue. There are 2 options I looked at and adjusted to be a better fit for the usage scenario and resource limitation.

  1. The Time to live setting defines how long messages will be kept on the device: Operate devices offline – Azure IoT Edge | Microsoft Docs (which is set to 2h per default).
  2. The not so obvious Rocks DB size configures the size of the logfiles: https://github.com/Azure/iotedge/issues/2431#issuecomment-582089419

After tweaking the settings, the following graph shows that now the device cleans up data before the disk runs full.

I can not give you values for you particular setup. You’ll need to figure them out for your setup depending on the amount of messages going though the Edge device and hardware sizing. Here are some pointers to settings which you might want to investigate, if you hit a similar problem on your devices:

RocksDB sizes

The above image shows setting for RocksDB (orange: 512MB, blue, 128MB, green 256MB). With the default setting the device is running out of disk space.

What can I do to prevent the device crashing?

Well, it depends 🙂 You can find a setting from the above that will prevent a full disk for a known scenario. But if you don’t know which modules with which setting is deployed?

In this case an alarm for low disk space is an option. It then needs to trigger a function that calls a method on the device to restart the edgeHub. This will clear the cache.

Configure Azure IoT Edge for downstream devices

A lot of documentation and posts are available to setup an Azure IoT Edge to act as an IoT Hub for downstream devices. In order to get it up and running in a dev environment, I had to do some more research.

My setup is a RaspberryPi 3 with Raspbian stretch and an Azure IoT DevKit which looks like this. And please remember the setup I used is for development only. I’ve used symmetric key authentication for the IoT Device. In a production scenario you would probably use certificate based authentication and no self signed certificates for the TLS encryption.

Transparent Gateway
Source: https://docs.microsoft.com/en-us/azure/iot-edge/iot-edge-as-gateway

Some starting points for reading are:

And here are my findings with the solutions that worked for my setup

  1. The downstream IoT devices should be able to connect to port 443 on the Edge module. But that port was not open/listening.
  2. How to verify the gateway certificate after the connection has been established?

The connectionstring for connecting to the gateway instead of an IoT Hub you can add ;GatewayHostName=hostname and the device should then go to the gateway. Take a note of the hostname and make sure it matches the name you specified when you were creating the certificates.

Looking at the serial output of the DevKit, I noticed it could not connect to the gateway. A quick analysis revealed that it does not accept connections on port 443. Hmm. Maybe a firewall on the Pi? As it turned out you have to tell the edge container to listen to 443 if you want to use it as a gateway.

Port bindings for the Edge module
{
    "HostConfig": {
        "PortBindings": {
            "8883/tcp": [{ "HostPort": "8883" }],
            "443/tcp": [{ "HostPort": "443" }],
            "5671/tcp": [{ "HostPort": "5671" }]
        }
    }
}

This will allow incoming connections not only for HTTPS. After the change was pushed to the Edge device, I could connect to it on port 443. Hurray.

The next challenge was to get the downstream device accept the certificate, that the gateway offered. In order to be able to verify the certificate, it has to trust the root certificate. This was, in my case, the file azure-iot-test-only.root.ca.cert.pem from the ~/certividates/certs directory. Open it with an editor, paste the content into the ino file and use the certificate.

// declare a constant with the content of 
// azure-iot-test-only.root.ca.cert.pem from ~/certificates/certs
static const char edgeCert [] =
"-----BEGIN CERTIFICATE-----\r\n"
...
"-----END CERTIFICATE-----";

// set trusted certs for the client
DevKitMQTTClient_SetOption(OPTION_MINI_SOLUTION_NAME, "something");
DevKitMQTTClient_SetOption("TrustedCerts", edgeCert);

Now the IoT device should be able to connect to the gateway. Have fun with IoT 😉

Azure Meetup OWL

Am 13.7. wird das erste Treffen des Azure Meetups OWL stattfinden. Wir sind derzeit noch auf der Suche nach einem Ort und den Themen 🙂
Vermutlich wird es um Chatbots und Machine Learning gehen. Wir werden aber auch auf euer Feedback eingehen um Themen für die nächsten Treffen zu finden.

Wenn ihr dabei sein möchtet, meldet euch bitte über die Meetup Seite an.

Update:

Ort: Arvato Systems, An der Autobahn 100, Gütersloh, Tower I

Zeit: 19:00 – 21:00 Uhr

Es wird um eine namentliche Anmeldung gebeten, da die Besucher beim Pförtner angemeldet werden müssen. Bitte entweder über Meetup, oder sendet mir eine E-Mail.

Multi Level Navigation

The out-of-the-box WSS navigation offers one level of navigation. The navigation elements can be configured in the site settings.

There is a way to make it have multi level navigation, as MOSS does, by changing the DataSource. But this “hack” is not flexible, as you can not reorder the item within the navigation menu.

While I was reading the post How to create your own Cascading Navigation using the ASPMenu control by Yvonne Harryman, the decision was made to build a custom navigation which supports multiple navigation levels.

default WSS toplevel navigation 

configure top link bar 

 

multi level navigation I wanted a more flexible way of realizing a navigation, which allows you to have more than one level with a plain Windows SharePoint Services site. So I build a navigation control, which uses the entries of a SharePoint list as source for the navigation elements.

Of coarse multiple navigation levels are supported. You can configure a parent element for each navigation element.To customize the order of appearance, you can set the ‘Link Order’.

The feature list:

  • Custom navigation control
  • Webpart shows the navigation
  • SharePoint list as DataSource
  • Multiple navigation levels
    • Assign a parent navigation element
    • Expand navigation nodes with child elements
  • Reorder navigation elements
  • Hide navigation elements
  • Access check on relative links for web, list and items
  • Cache
    • Items from the navigation list
    • Invalidate cache if there is a change within the listitems
    • Access to navigation Urls is stored in the session
  • Import SharePoint TopLevel navigation entries to the list
  • Get navigation list items with elevated privileges, so the list can be hidden from users
  • Grouped view “By Parent” for the navigation list
  • Url Parameter
    • Refresh=true
      will force a reload of the navigation list items
    • Debug=true
      will show how long (in ms) the navigation control needs to render

Each navigation element is stored in a SharePoint list. This list is created when the feature is activated on the SiteCollection.

SharePoint list

The list can be filled with the current SharePoint navigation by clicking on the link “Import Navigation to Multi Level Navigation List” in the site settings. It brings up another page, where you can import the navigation and check for updates for the solution.

site collection administration

Installation:

  1. Add the solution with “stsadm –o addsolution –filename RH.Navigation.wsp”
  2. Deploy the solution in the Central Administration
  3. Active the feature “RH.Navigation” for a site collection
    sitecollection feature
  4. Add the Webpart to a page, or alter the masterpage to use the custom navigation instead of the build-in SharePoint navigation

Customize the masterpage

To use the Multi Level Navigation instead of the out-of-the box SharePoint navigation, register a tagprefix, add the control and remove the SharePoint Navigation.

  • Add an entry to the top of your masterpage
    <%@ Register TagPrefix="Navigation" Namespace="RH.Navigation.Control" Assembly="RH.Navigation, Version=1.0.0.0, Culture=neutral, PublicKeyToken=2d07bbf19c37a6cf" %>
  • Alter the SharePoint navigation
    <SharePoint:AspMenuNavigation:MultiLevelNavigation
          ID="TopNavigationMenu"
          Runat="server"
          DataSourceID="topSiteMap"
          EnableViewState="false"
          AccessKey="<%$Resources:wss,navigation_accesskey%>"
          Orientation="Horizontal"
          StaticDisplayLevels="21"
          MaximumDynamicDisplayLevels="1"
          DynamicHorizontalOffset="0"
          StaticPopoutImageUrl="/_layouts/images/menudark.gif"
          StaticPopoutImageTextFormatString=""
          DynamicHoverStyle-BackColor="#CBE3F0"
          SkipLinkText=""
          StaticSubMenuIndent="0"
          CssClass="ms-topNavContainer">
            <StaticMenuStyle/>
            <StaticMenuItemStyle CssClass="ms-topnav" ItemSpacing="0px"/>
            <StaticSelectedStyle CssClass="ms-topnavselected" />
            <StaticHoverStyle CssClass="ms-topNavHover" />
            <DynamicMenuStyle  BackColor="#F2F3F4" BorderColor="#A7B4CE" BorderWidth="1px"/>
            <DynamicMenuItemStyle CssClass="ms-topNavFlyOuts"/>
            <DynamicHoverStyle CssClass="ms-topNavFlyOutsHover"/>
            <DynamicSelectedStyle CssClass="ms-topNavFlyOutsSelected"/>
        </SharePoint:AspMenuNavigation:MultiLevelNavigation>
  • OR remove the SharePoint navigation (<SharePoint:AspMenu…) and insert the MultiLevelNavigation
    <Navigation:MultiLevelNavigation runat="server"></Navigation:MultiLevelNavigation>

Note:

If your AspMenu contains the StaticDisplayLevels property, remove it or set it to 1! (Quote from MSDN: The StaticDisplayLevels property indicates how many levels to display statically from the root of the menu. For example, if StaticDisplayLevels is set to 3, your menu will be expanded to statically display its first three levels. The minimum static display level is 1, and the control will throw an exception if the value is set to 0 or a negative number.)

Download RH.Navigation.wsp

Download Sourcecode