Log Parser Connector for easy Integration of your Log Based Sensors/Metrics - TeamViewer Support
<main> <article class="userContent"> <p><em>This article applies to TeamViewer IoT users who monitor their IoT networks.</em></p><p><em>The solution presented in this article requires TeamViewer IoT Agent v 1.1.102 (or higher). To identify the version of your agent, run: </em><strong><em>sudo teamviewer-iot-agent info </em></strong><em>(an error, or no result, indicates an older version of the agent is installed).</em></p><h4 data-id="description"><strong>Description</strong></h4><p>As described in the <a href="https://community.teamviewer.com/t5/TeamViewer-IoT-Labs/Custom-Plugin-Connector-How-to-easily-integrate-your-IoT-Sensors/td-p/36439" rel="nofollow noreferrer ugc">Custom Plugin Connector</a> article, it's quite easy to monitor your sensor network via TeamViewer IoT. This article describes how to use the TeamViewer IoT Log Parser Connector to easily extract & integrate sensor data from your log files. The Log Parser is generic, in that, it can be used to parse any log. For the context of this article, it is assumed your IoT sensor values are written to a log file which you would like to connect to the TeamViewer IoT Agent.</p><h4 data-id="step-1-install-dependencies"><strong>Step 1 - Install dependencies</strong></h4><p>The connector requires the following to be installed on the device:</p><ul><li>python 3</li><li>For details, reference: <a href="https://www.python.org/downloads/" rel="nofollow noreferrer ugc">https://www.python.org/downloads/</a></li><li>Note: python v2 is not supported </li></ul><h4 data-id="step-2-download-and-install-the-log-parser-connector"><strong>Step 2 - Download and install the Log Parser Connector</strong></h4><ol><li>From your device running the TeamViewer IoT Agent, download the connector: </li></ol><pre class="code codeBlock" spellcheck="false" tabindex="0">wget https://download.teamviewer-iot.com/connectors/teamviewer-iot-log-parser/1.0.0/teamviewer_iot_log_parser-1.0.0.tar.gz </pre><ol><li>Upon download completion, extract the contents of the connector:</li></ol><pre class="code codeBlock" spellcheck="false" tabindex="0">sudo mkdir -p /usr/local/teamviewer-iot-agent/monitoring/ tar -xzvf teamviewer_iot_log_parser-1.0.0.tar.gz -C /usr/local/teamviewer-iot-agent/monitoring/ </pre><ol><li>Install the Connector</li></ol><pre class="code codeBlock" spellcheck="false" tabindex="0"> teamviewer_iot_log_parser-1.0.0 python3 setup.py installcd </pre><ol><li>Verify the installation was successful by identifying a log file to parse and using the following command to parse a string. Replace <em>logFilePath & justTextorRegex </em>with the path of your log file and a string to search for, respectively.</li></ol><pre class="code codeBlock" spellcheck="false" tabindex="0">teamviewer_iot_log_parser -f logFilePath --regex justTextOrRegex </pre><p><strong> </strong></p><h4 data-id="step-3-get-to-know-the-teamviewer-log-parser-connector"><strong>Step 3 – Get to know the TeamViewer Log Parser Connector</strong></h4><p>The connector will integrate with the TeamViewer IoT Agent as will be discussed in the next section. Prior to integrating the connector, the connector configuration and syntax should be reviewed.</p><p>The connector can be configured given the following configuration arguments:</p><p><strong>TeamViewer IoT Log Parser Connector</strong></p><p> <strong>Argument / flag</strong></p><p> <strong>Required?</strong></p><p> <strong>Description</strong></p><p> -p, --parser</p><p> True</p><p> The log parser can operate in 2 modes:</p><p><em>matches – (Default) </em>returns a count of strings matching a specified regex since the last run</p><p><em>metrics – </em>returns an aggregated value for each specified metric since the last run</p><p> -t, --timeout</p><p> True</p><p> Duration to wait for parsing to complete before terminating and returning an error.</p><p>Values are in seconds; Default, if not specified, is 5 seconds.</p><p> </p><p> </p><p>The two modes of operation, (<em>Matches & Metrics), each require additional parameters to be specified:</em></p><p><strong>Matches</strong></p><p> <strong>Argument / flag</strong></p><p> <strong>Required?</strong></p><p> <strong>Description</strong></p><p> -f, --file </p><p> True</p><p> File path of the log to parse</p><p> -R, --regex</p><p> True</p><p> String or regex expression to match</p><p><strong><em>For regex details, reference</em></strong>: <a href="https://docs.python.org/3/library/re.html" rel="nofollow noreferrer ugc">https://docs.python.org/3/library/re.html</a></p><p> </p><p> </p><p><strong>Metrics</strong></p><p> <strong>Argument / flag</strong></p><p> <strong>Required?</strong></p><p> <strong>Description</strong></p><p> -f, --file</p><p> True</p><p> File path of the log to parse</p><p> -M, --metrics</p><p> True</p><p> Space delimited list of metrics to extract</p><p> -A, --aggfunctions</p><p> True</p><p> Space delimited list of aggregation functions for each respective metric</p><p>Possible values: avg, sum, min, max, median</p><p> -T, --template</p><p> False</p><p> Template to parse metrics</p><p>Default template is :metric=.(.*?).[\s], which will match log entries of “metricName"="value"</p><p> </p><p> </p><h4 data-id="step-4-integrate-the-log-parser-connector-with-the-teamviewer-iot-agent"><strong>Step 4 – Integrate the Log Parser Connector with the TeamViewer IoT Agent</strong> </h4><p>The TeamViewer IoT Agent includes its own system monitoring configuration file. This configuration file will be extended to include the Log Parser configuration. As you are going to maually edit a system configuration file, ensure to <strong><em>create a backup of the file in the case the file gets corrupted by user error.</em></strong></p><ol><li>Open for editing the TeamViewer IoT Agent monitoring configuration file located at:</li></ol><ul><li>/var/lib/teamviewer-iot-agent/system_monitors.conf</li><li>sensors are hosted under the "sensors" node of the configuration file</li><li>For each sensor you want to integrate, append its registration information to the configuration file. Reference the template below and replace the placeholders with your sensor details.</li><li>Save the configuration file.</li><li>Restart the agent to reload the configuration file.</li><li>teamviewer-iot-agent restart</li><li> Log into your <a href="https://community.teamviewer.com/English/home/leaving?target=https%3A%2F%2Fteamviewer-iot.com%2F" rel="nofollow noreferrer ugc">TeamViewer IoT dashboard</a> and verify that the Sensors/Metrics added to the configuration file now appear on your <strong><em>Metrics </em></strong>tab.</li><li>Pin the metrics to your dashboard and ensure the data is being updated per the frequency specified in the config file</li></ul><h4 data-id="teamviewer-iot-log-parser-connector-template">TeamViewer IoT Log Parser Connector Template</h4><p>Use the following template to add your connector configuration for each sensor to the monitoring configuration file. The template is an example of adding a single sensor with two (2) metrics (Temperature & Humidity).</p><p>Ensure to replace the following placeholders with the specifics of your sensor. Note: the frequency in the template is set to <em>10,</em> which will trigger the Log Parser Connector to run every 10 seconds, aggregating the values from each interval.</p><p>More details about the JSON schema can be found in the <a href="https://community.teamviewer.com/t5/TeamViewer-IoT-Labs/Custom-Plugin-Connector-How-to-easily-integrate-your-IoT-Sensors/td-p/36439" rel="nofollow noreferrer ugc">Custom Plugin Connector</a> article.</p><pre class="code codeBlock" spellcheck="false" tabindex="0">{ "sensors": [ { "name": "temperature_humidity_parser", "monitoringService": "python3 /usr/local/teamviewer-iot-agent/monitoring/log_parser/teamviewer_iot_log_parser.py", "monitoringParams": "--file /tmp/file.log --parser metrics -–metrics temperature humidity --aggfunctions max avg –-template ':metric=.(.*?).[\\s]'", "frequency": 10, "metrics": [{ "name": "Temperature", "key": "temperature", "valueType": "double", "valueAnnotation": "C" }, { "name": "Humidity", "key": "humidity", "valueType": "double", "valueAnnotation": "" } ] }, ... ] } </pre><p><em>Note:</em> the <em>metrics.key</em> in the config file should be same as in the <em>monitoringParams</em> parameter. The <em>metrics.name</em> can be any string as it will be the display name of the metric in the TeamViewer IoT Cloud dashhboard.</p><h4 data-id="monitoringparams-examples"><strong><em>monitoringParams</em> Examples</strong></h4><p>The following examples describe use cases and their respective configuration of the connector as specified by the <strong><em>monitoringParams </em></strong>parameter:</p><ol><li>Parse a log file to retrieve the <em>maximum</em> value of a metric called “<em>temperature</em>” and the a<em>verage</em> value of a metric called “humidity”.</li></ol><pre class="code codeBlock" spellcheck="false" tabindex="0">>> Log File (log_file.log) Contents: some text temperature=18 some text humidity=80 some other text temperature=19 some text humidity=82 text >> monitoringParams: --file log_file.log --parser metrics --metrics temperature humidity --aggfunctions max avg --template=":metric=(.*?)[\\s]" >> Result: {"temperature": 19, "humidity":81} </pre><p> </p><ol><li>Parse a log file to retrieve the number of occurrences of the string “findMe".</li></ol><pre class="code codeBlock" spellcheck="false" tabindex="0">>> Log File (log_file.log) Contents: some text findMe some text findMe some other text temperature=19 some findMe text humidity=82 text >> monitoringParams: --file log_file.log --parser matches --regex someRegex >> Result: {"count": 3} </pre><p> </p><p> </p> </article> </main>
