Critical Equipment Monitoring

Project Information

Authors

  • Rio Akbar
  • Calvin Nguyen
  • Erik Stefan
  • James Tarrant
  • Dylan Turner

1. Introduction

1.1 Purpose

The purpose of this document is to describe the end-to-end design of the radio repeater remote monitoring system. This is intended to provide information for troubleshooting the existing system, or to build on the design to add additional functionality.

1.2 Background

The WA State Emergency Service (SES) provides a wide range of volunteer services during natural disaster or emergency situations, to help the community cope and recover from the impacts. The Communications Support Unit (CSU) is a function of the SES that provides vital radio and data links to restore and maintain safety critical communications. The CSU uses radio repeaters in remote or hilly terrain to extend two-way radio communications between Users that are otherwise out of line-of-site propagation range, as shown in Figure 1.

Figure 1: Repeater Functionality
Figure 1: Repeater Functionality

The repeater set-up is mobile, temporary, and typically independently powered. Once the Field Operative has installed the equipment, it will be left online and unmanned. Therefore, the Operators are only aware of an issue with the repeater (such as loss of power) once the quality of communication degrades. It may be several hours before the Field Operative can reach the repeater to restore functionality.

1.3 Initiative

It is an initiative of the CSU to improve availability of deployed repeaters by obtaining advanced warning of potential issues. The status of key components such as normal and backup power supplies should be monitored. The Operator should be aware of changes in state, with sufficient time to take remedial action.

The functionality in this Design and Testing Document, in particular the monitoring of the normal and back-up power supplies, are to support this initiative.

1.4 Previous System

The CSU operates a range of radio repeaters of varying make and model. They are typically powered via 240VAC from a generator, or a grid connected outlet. This is the “Normal Supply” shown in Figure 2.

Figure 2: Repeater Cable Block Diagram (Existing)
Figure 2: Repeater Cable Block Diagram (Existing)

If the normal supply fails, such as failure of the generator, the repeater will automatically changeover to a 12VDC supply. This is the “Back-up Supply” shown in Figure 2, and is usually provided by a 12V battery and solar panel.

Figure 3: Approximate Battery Voltage Profile
Figure 3: Approximate Battery Voltage Profile

When the normal supply is unavailable, the back-up supply may only provide power for several hours. The actual duration relies on the sunlight received by the solar panel and the level of usage of the repeater. Figure 3 illustrates the effect on battery voltage when normal supply is lost, and the threshold at which the repeater functionality deteriorates. The existing system does not incorporate remote monitoring functionality. Therefore, there is no advanced warning of repeater performance degradation. The following sections describe the system designed and implemented to provide the remote monitoring improvement.

2. System Architecture

Figure 4: System Architecture
Figure 4: System Architecture

Figure 4 shows the new architecture implemented by the project team. The field device samples the battery voltage and mains status. This telemetry is sent to the IoT Central service. IoT Central provides the API used by the web application (hosted in Azure App Services) to read and display the telemetry. IoT Central will also initiate the SMS notifications, via Azure Logic Apps and SMS Broadcast, to notify the Client of issues with the repeater power supplies.

3. Hardware

The hardware consists of an Arduino MKR WIFI 1010 module, a carrier board into which the Arduino is inserted, and a sensor which is placed near the mains or generator power cord going into the repeater to detect 50Hz 240V AC. The Arduino module is responsible for the analysis and reporting of data to the IoT Central Application. It collects data from the 12V battery line and the mains/generator line and, using its onboard Wi-Fi module, reports this to the website as shown in Section 2.

The carrier board supplies power to the system from the 12V battery, stepping it down to 5V through a switch-mode power supply. It also provides a means of connecting the mains sensor output to the Arduino through screw terminals, protection circuitry to prevent damage to the device from incorrect connections, and a USB Type A charging port which can supply power to a mobile hotspot or mobile phone.

The mains sensor amplifies and low-pass-filters any signal picked up by the sensor wire. This amplified signal is then sent to the Arduino for analysis. The sensor is designed to be placed as close to the cable-to-be-monitored as possible, as this limits the amount of unwanted noise which may be picked up by a longer sensor wire. It takes power directly from the 12V battery line (bypassing the carrier board’s power supply).

The final device is shown in Figure 5 and Figure 6. Refer to Appendix A for a hardware schematic, and Appendix B for the corresponding Bill of Materials.

Device Internal
(a) Device Internal
Mains Sensor Internal
(b) Main Sensor Internal
Figure 5: Hardware Device Photos
Figure 6: Hardware External, and Mains/Battery Interfaces
Figure 6: Hardware External, and Mains/Battery Interfaces

The latest firmware for the device can be found at the following git repository: https://github.com/erik6K/remote-monitor/tree/master/firmware

The functionality of the firmware is too extensive to describe in detail in this design document. Therefore, the logic flow diagram shown in Appendix C has been produced to graphically describe the functionality at a high-level.

During the “Analyse” state the device takes a burst of 1024 samples of the signal picked up by the mains sensor (AC amplifier) and performs a Fast Fourier Transform (FFT) on the data in order to find the relative magnitudes of different frequencies present. It then checks the magnitude of the 50Hz frequency against the noise floor to determine whether mains voltage is present.

4. IoT Central Application

The IoT Central Application is used as the gateway to bring the device telemetry into the Azure Cloud. The application performs the key functions described in the following sections.

4.1 MQTT Client

MQTT is the Messaging Querying Telemetry Transport protocol. This is a lightweight publish and subscribe protocol, preferred in internet of things applications such as this project. The field device is programmed to periodically publish its telemetry over MQTT to the IoT Central Application. IoT Central is subscribed to these topics and will store the data to be displayed on the dashboard or retrieved via the REST API.

4.2 REST API

The IoT Central Application provides a REST API that allows other applications to perform operations that are usually performed from within the IoT Central GUI. This project uses the following APIs within the web application:

  1. Devices - List: Retrieve the list of devices currently deployed in IoT Central
  2. Devices - Get Component Telemetry Value: Retrieves the most recent mains status and battery voltage, to be displayed on the web application
  3. Devices - Get Cloud Properties: Retrieves whether notifications are currently enabled or disabled for each device and telemetry type.
  4. Devices - Update Cloud Properties: Updates whether notifications should be enabled/disabled for a particular device telemetry type.

For a full list of REST APIs refer to: https://docs.microsoft.com/en-us/rest/api/iotcentral/

4.3 Dashboards

The dashboards provide a secondary way to view device telemetry, in addition to the web application (Refer section 5). An example of this dashboard is shown in Figure 7.

Figure 7: Azure IoT Central Device Dashboard
Figure 7: Azure IoT Central Device Dashboard

4.4 Device Templates

A template of the remote monitoring device has been pre-configured in the cloud. New devices can be provisioned using this template, with minimal configuration required.

4.5 Rules

Rules are used within IoT Central to trigger events when certain conditions are met. These rules have been used to implement the SMS notification functionality described in Section 6.

5. Web Application

A web application has been developed to provide the functionality described in Figure 8. Its primary functions are to view the devices, most recent telemetry, and manage SMS notifications.

Figure 8: Web Application Purpose and Layout
Figure 8: Web Application Purpose and Layout

The application is written in Python, using Flask as the web application framework. To avoid the need to update the website when deploying new devices, templating is used to dynamically create the HTML by pulling the data straight from IoT Central. Jinja2 is used as the templating language. The high-level design of the web application has been described in the logic flow diagram in Figure 9.

Figure 9: Web Application Logic Flow Diagram
Figure 9: Web Application Logic Flow Diagram

The full code is contained in the following git repository. Please contact a member of the team for access, if required: https://github.com/tarrantja/remote-monitor-web

The application is hosted using Azure App Services, using Gunicorn as the internal Web Server Gateway Interface (WSGI) HTTP server: https://repeater-monitor.azurewebsites.net/

6. Notifications

Since the team was on track to complete the core functionality described in the Scope of Work, attention began to turn to possible stretch goals and additional features. One of these was a method for notifying an operator (via SMS or Email) of an issue, rather than needing to periodically check the web application.

6.1 IoT Central Rules

The IoT Central application allows setting of ‘rules’ to trigger a subsequent action when certain conditions are met. The two rules shown in Table 1 were added to implement the notification functionality.

Table 1: IoT Central Rule Triggers
TriggerFilterAction
Mains Status OFFIf ‘Cloud Mains Status Notification’ property is set to ‘OFF’, do not trigger.Trigger Logic App ‘remote-monitor-notifications’. Refer Section 6.2.
Battery Voltage < 11.9VIf ‘Cloud Battery Voltage Notification’ property is set to ‘OFF’, do not trigger.Trigger Logic App ‘remote-monitor-notifications-battery’. Refer Section 6.2.

6.2 Logic Apps and SMS Broadcast

The project team then created two simple Logic Apps within Azure, one to trigger SMS notifications on mains failure, and the other to trigger SMS notifications on low battery voltage. Each logic app has a “trigger” which comes from the IoT Central Application when the rules in Table 1 are satisfied. The subsequent “action” is to initiate a HTTPS POST request to the SMS Broadcast API. SMS Broadcast has been selected at the request of the Client, who have an existing account with credits to cover the cost of the SMS notifications.

6.3 Notification Silencing

The rules will be triggered whenever telemetry is received that matches the relevant criteria. To prevent flooding of SMS alerts, each telemetry type (battery and mains) has an associated ‘Cloud Telemetry Status Notification’ property assigned. When this property is set to ‘OFF’, the notifications would be filtered out, as stated in Table 1. This filter can be toggled on and off within the web application, as shown previously in Figure 8.

7. Security

To manage security, a large cloud service provider, like Azure, was preferred because of their built-in cyber security controls. Table 2 shows the possible system security threats, and controls to manage.

Table 2: Security Threats and Associated Controls
ThreatConsequenceControl
Access to the device via the internetDamage to device or connected equipmentDevice is publishing-only and does not receive commands/data.
Access to IoT Central via the internetAbility to add/change/delete all devices and associated data.Devices connect securely to IoT Central using a Secure Access Token. Applications connect securely using a unique access key.
Unauthorized access to the IoT Central PortalAs aboveAuthentication and access authorization to IoT Central is fully managed by Microsoft. Permissions can be set within the application to control access if required.
Loss or theft of the deviceThe secret credentials within the firmware can also be stolen, and the firmware can be changed. If lost or stolen, recommend:
  1. Blocking and deleting the device within IoT Central (refer User Manual)
  2. Changing hotspot WiFi password
Malicious access to web application source codeChange, delete or deface the web applicationChanges pushed to the master branch of the web application git repository will be automatically published. Therefore, access to the git repository should only be provided to trusted parties. App Services can be used to de-link the git repository if required. Refer to User Manual - disaster recovery for web application.
Unauthorised access to web application.Disable or enable SMS notifications.Access to the web application shall be restricted to users within the Client’s applicable Azure Active Directory.

8. IP Exploitation Model

8.1 Distribution Model

As agreed in the Scope of Work, the project has been modelled via a creative commons distributing rights model. The Creative Commons (CC) license enables free distribution of an otherwise copyrighted "work". The Team has licensed the project deliverables as Creative Commons 3.0 Non-Commercial Australian License. In essence, the Team has given the Client the right to share, use, and build upon what the Team have created, in line with the conditions below.

8.2 Client Rights

9. Cost

9.1 Ongoing Monthly Costs

Each cloud service represents an ongoing monthly cost. The monthly cost, at point of handover, is shown in Table 3. This will change as more devices are deployed. Before adding more devices is it recommended to refer to the Azure pricing overview information for each service.

Table 3: Ongoing Costs
ServiceTier$/month
Azure App Service PlanFree$0/month
Azure Logic AppsUsage based. $0.000218 per SMS~$0/month
Azure IoT Central ApplicationStandard Tier 1 - First 2 devices free~$0/month
SMS Broadcast3.7c per SMS$3.70/month for 100 SMS
TOTAL<$4/month

9.2 Device Cost

Each device represents a once-off cost. Refer to the bill of materials in Appendix B for a list of parts. The approximate cost for all equipment for a new device is $90-$110, depending on suppliers.

10. User Acceptance Tests

This section documents the user acceptance tests executed to prove the project design (described in this document) has met all requirements laid out at the beginning of the project. A summary of all tests is provided in Table 4.

Table 4: User Acceptance Test Summary
StoryCore/AdditionalTest Pass/Fail
Story 1.1 Powering DeviceCorePASS
Story 1.2 Powering Mobile HotspotAdditionalPASS
Story 2.1 Detects Main VoltageCorePASS
Story 2.2 Detects Battery VoltageCorePASS
Story 3.1 View Web PageCorePASS
Story 3.2 View Live DataCorePASS
Story 4.1 Reporting of Data Via RadioAdditionalNOT TESTED
Story 5.1 Receiving NotificationsAdditionalPASS
Story 5.2 Silencing NotificationsAdditionalPASS
Figure 10: Final Demonstration Set-up
Figure 10: Final Demonstration Set-up
Figure 11: Repeater Monitor Device Set-up Cable Block Diagram
Figure 11: Repeater Monitor Device Set-up Cable Block Diagram

Figure 10 and Figure 11 show the equipment used during the tests. Not all equipment was required for each test.

10.1 STORY 1.1 - POWERING DEVICE

10.1.1 Story

As a Field Operative, I want to power the device using the existing battery connector, so that it is standardised across batteries.

10.1.2 Test and Results

Table 5: Device Power and Battery Voltage Test Procedure
StepActionResult
1.Connect T-Connector to battery connectorOK
2.Connect device to T-connectorOK
3.Confirmed device power indicator LED illuminatesCONFIRMED

10.2 [STRETCH] STORY 1.2 - POWER MOBILE HOTSPOT

10.2.1 Story

As an Operator, I want a way of powering a mobile hotspot or mobile phone from the device, so I do not need to rely on the hotspot/mobile internal battery.

10.2.2 Test and Results

Table 6: Power Mobile Hotspot Test Procedure
StepActionResult
1.Connect the device to the 12V DC supplyOK
2.Connect the mobile phone to the USB A charging port provided on the deviceOK
3.Confirm the mobile charges.CONFIRMED

10.3 STORY 2.1 - DETECTS MAINS VOLTAGE

10.3.1 Story

As a Field Operative, I want a device which I can interface with my repeater’s main power source in an isolated fashion, so that it can detect the presence of power.

10.3.2 Test and Results

The first test was completed with the repeater powered from the mains generator. The results are shown in Table 7.

Table 7: Generator 240VAC Voltage Detection Test Procedure
StepActionResult
1.Wrap sensor onto 240VAC cableOK
2.Ensure generator is running, cable plugged in, and connected to repeaterOK
3.Power on deviceOK
4.Confirm indication LED lights up to show presence of 50Hz mainsCONFIRMED
5.Unplug generator cable and confirm LED turns offCONFIRMED

The test was repeated using a normal grid-connected 240V mains outlet. The results are shown in Table 8.

Table 8: Grid 240VAC Voltage Detection Test Procedure
StepActionResult
1.Connect the repeater to a 240VAC general purpose outletOK
2.Wrap the antenna around the 240VAC cableOK
3.Power on deviceOK
4.Turn outlet on and confirm LED indicator shows presence of voltageCONFIRMED
5.Turn outlet off and confirm LED indicator turns offCONFIRMED

10.4 STORY 2.2 - DETECT BATTERY VOLTAGE

10.4.1 Story

As a Field Technician, I want a device which can interface with the backup 12V battery using the existing connector, so that it can measure its voltage.

10.4.2 Test and Results

Table 9: Battery Voltage Detection Test Procedure
StepActionResult
1.Connect the device to the battery using the T-connectorOK
2.Connect the device to the serial monitor to monitor outputOK
3.Compare the voltage level on the serial monitor to the voltage read from a reliable multimeter, and confirm they are within +/- 0.1V.CONFIRMED
Figure 12: Multimeter Reading from Battery
Figure 12: Multimeter Reading from Battery
Figure 12: Reading from Device
Figure 12: Reading from Device

Figure 12: (a) Multimeter Reading from Battery (b) Reading from Device

10.5 STORY 3.1 - VIEW WEB PAGE

10.5.1 Story

As an Operator, I want to go to a web page and see an on/off status for the main repeater 240VAC supply and a battery voltage, so that I have a convenient user interface for tracking repeaters.

10.5.2 Test and Results

Table 10: Web Application Design Test Procedure
StepActionResult
1.Open the Web ApplicationOK
2.Confirm the list of deployed devices can be viewed.CONFIRMED
3.Confirm there are values displayed for the device telemetry.CONFIRMED
4.[Stretch] Confirm the duration since data was last report is displayed.CONFIRMED
5.[Stretch] Open the application on a mobile device and ensure layout is optimised for mobile.CONFIRMED

10.6 STORY 3.2 - VIEW LIVE DATA

10.6.1 Story

As an Operator, I want the web page to be updated with field data in real time, so I can have the most up to date information for decision making.

10.6.2 Test and Results

Table 11: Live Data Reporting Test Procedure
StepActionResult
1.Open the Web ApplicationOK
2.Power the battery via the deviceOK
3.Confirm, via the Serial monitor, that the device has connected to the WiFi hotspotCONFIRMED
4.Confirm, via the Serial monitor, that the device has connected to the MQTT ClientCONFIRMED
5.Change the battery voltage and confirm the value updates live within the Web ApplicationCONFIRMED

10.7 [STRETCH] STORY 4.1 - REPORTING OF DATA VIA RADIO

10.7.1 Story

As an Operator, I want to be informed of when the repeater’s main and back-up power source fails by radio, so that I can be aware of impending repeater failure.

10.7.2 Test and Results

To ensure core functionality could be implemented to a high level of quality, and to focus on higher priority stretch targets (refer stories 2.1, 5.1 and 5.2) this stretch target was left partially completed. Therefore, this story has not been tested. Logic has been built into the firmware to allow this functionality to be added in a future project. Refer to Appendix C for more details.

10.8 [STRETCH] STORY 5.1 - RECEIVING NOTIFICATIONS

10.8.1 Story

As an operator I want to be immediately notified of a mains voltage failure or low battery level, so I can act.

10.8.2 Test and Results

Table 12: Receiving Notifications Test Procedure
StepActionResult
1.Connect the device to the battery and mains supplyOK
2.Turn the mains supply offOK
3.Confirm an SMS alert is received describing the eventCONFIRMED
4.Reduce the battery voltage below 11.9VOK
5.Confirm an SMS alert is received describing the eventCONFIRMED
Figure 13: Mains Failure SMS
Figure 13: Mains Failure SMS
Figure 13: Battery Failure SMS
Figure 13: Battery Failure SMS

Figure 13: (a) Mains Failure SMS (b) Battery Failure SMS

10.9 [STRETCH] STORY 5.2 - SILENCING NOTIFICATIONS

10.9.1 Story

As an operator I want the ability to disable notifications of the main voltage status, so I can deploy a repeater without a generator and not become flooded with irrelevant notifications, or prevent alarms once I am already aware of an issue.

10.9.2 Test and Results

Table 13: Silencing Notifications Test Procedure
StepActionResult
1.Repeat steps 1-5 from Table 12.OK
2.On the web application, toggle the notifications for the Mains Status to OFF.OK
3.Confirm the notifications related to the mains status are no longer received.CONFIRMED
4.On the web application, toggle the notifications for the Battery Voltage to OFF.OK
5.Confirm the notifications related to the battery voltage are no longer received.CONFIRMED

11 HANDOVER

A handover is a process, not a date such as a unit deadline. The purpose of this section is to describe how the project will be handed over to the Client, and what support or interactions may be expected between the Client and Team following the completion of the project.

11.1 DOCUMENTATION

The knowledge needed to sustain the system is contained within the Installation and Operation Manual. This document captures all activities for the intended use of the device. Additionally, it contains instructions for changing key system parameters. The knowledge to understand and further develop the system is contained within this Design and Testing Document, and the linked code repositories.

11.2 TRAINING

The Team has provided a final face-to-face demonstration of the device to the Client. The system itself is designed to be intuitive to use, with the operating manual created to provide back-up. The team will remain available to answer queries or further demonstrate how to use the system.

11.3 FEEDBACK

The Team is eager to receive feedback on how the device performs when deployed in a real-world scenario, and what material benefit was seen by the Client. If the Client is happy with the continued performance, the team has offered to build and deploy additional devices.

12 FUTURE STORIES

This section describes the “next steps” or future stories that would support the Client’s initiative to remotely monitor their communication equipment. It is intended to set the next person or project team up for success.

12.1 RADIO REPORTING

As an Operator, I want to be informed of when the repeater’s main and back-up power source fails by radio in case the device does not have access to 4G for web reporting.

12.2 RADIO INTERROGATION

As an Operator, I want to be able to interrogate the health of the normal and back-up supply over the radio in case I miss an automated status update.

12.3 LIVE VIDEO STREAM

As an Operator, I want to be able to view a live video stream from the repeater set-up, so I can confirm whether the equipment has been misused, or a hazard is approaching.

APPENDIX A: DEVICE SCHEMATICS

Appendix A: Device Schematics
Appendix A: Device Schematics
Appendix A: Device Schematics
Appendix A: Device Schematics

APPENDIX B: BILL OF MATERIALS

Appendix B: Bill of Materials
Appendix B: Bill of Materials

APPENDIX C: FIRMWARE FLOW CHART

Appendix C: Firmware Flow Chart
Appendix C: Firmware Flow Chart