# obinsect - a highly configurable energy meter M-BUS to MQTT proxy ## command line options ``` $ obinsectd obinsectd version 0.01, using libmosquitto 1.5.7 and libjson-c 0.12.1 Usage: ./obinsectd [-d] [-c configfile] -s device [-b hostname] [-p port] [-u username [-P password]] [-i id] [-k keepalive] [--insecure] [{--cafile file | --capath dir} [--cert file] [--key file] [--unscaled | --units ] -c : Configuration file. Default: /etc/obinsect.conf -d : Enable debugging -s : Serial device connected to M-Bus. E.g /dev/ttyUSB0. Use '-' to read from stdin MQTT client options: -b : Broker hostname or IP address. Default: localhost -i : Client Id. A random id is generated by default -k : Keepalive in seconds. Default: 60 -p : Broker TCP port. Default: 1883 (8883 when using MQTTS) -P : Password. Default is no authentication -u : Username MQTT over TLS (MQTTS): --insecure : Do not validate brokername against certificate --cafile : Trusted CA certificates PEM file --capath : Trusted CA certificates directory --cert : Client certificate --key : Client private key Published value format: --unscaled : Do not scale numbers. Default: false (scaling enabled) --units : Include units. Implies value scaling. Default: false Example: ./obinsectd -s /dev/ttyUSB0 -b broker.example.com ``` The serial device is the only option required. This should normally be an USB to M-BUS bridge connected to a supported meter. Using **-** as serial device makes obinsectd to read packes from **stdin** instead. This can be used to feed it pre-recorded samples. **obinsectd** will attempt an anonymous MQTT connection to a broker running on the same host by default. It will continue without MQTT if this fails for any reason. It will not subscribe to any MQTT topics, and it will only publish if configured to do so using the configuration file. ## JSON config file **obinsect** can be used for debugging the HAN port without any configuration. Simply run it with the -d (--debug) option. The serial port option is mandatory: ``` obinsectd -d -s /dev/ttyUSB0 ``` In debug mode, obinsectd prints a hexdump of all received packets and the native structure of all sucessfully parsed packets as JSON. But its real advantage lies in the configurable mapping of data to MQTT topics, as well as fully configurable OBIS code lists. The configuration file has two sections: 1. "topicmap" - mapping MQTT topics to values 2. "obisdefs" - list of JSON files with OBIS code defintions This is an example: ```javascript { "topicmap" : { "/obinsect/json/alias" : "alias", "/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ], "/obinsect/websocket/power" : "Power", "/obinsect/counters" : [ "timestamp", "MeterTime", "CumulativeEnergy", "CumulativeEnergyExport", "CumulativeReactiveEnergy", "CumulativeReactiveEnergyExport" ] }, "obisdefs" : [ "/etc/obinsect/aidon_v0001.json", "/etc/obinsect/kfm_001.json", "/etc/obinsect/kamstrup_v0001.json" ] } ``` ### topicmap Any number of MQTT topics can be given, including none. No specific topic structure is assumed. The example above is simply an example. You do not have to use a common prefix, or organize topics in a tree at all. After parsing a packet, data will be published to all topics having one or more values received from the meter. I.e. **metadata** is not considered when deciding whether to publish to a topic or not. The **value** specification can be 1. an OBIS code in text representation, e.g **"1-1:0.2.129.255"** 2. a code alias defined in the currently selected code list, e.g. **"Power"** 3. the **"date-time"** field of the DLMS/COSEM packet 4. a metadata variable (see below for spec), e.g. **"timestamp"** 5. one of two predefined objects: **"normal"** or **"alias"** 6. a list combining one or more of the above 7. special debug values: **"rawpacket"**, **"parserdata"** or one of the loglevels **"debug"**, **"info"** or **"error"** #### OBIS code The value is converted to its matching JSON representation, considering scaling and unit conversion as described below. By default, numeric values are returned as numbers scaled according to the OBIS list definition. Note that Kaifa meters transmits arrays of values only. These are mapped to codes by **obinsectd** using the **"alias"** section of the matching OBIS list (**"KFM_001"**). This list must therefore be defined in the correct order. **"date-time"** values are converted to UNIX epoch time. Some meters a binary object type for **"MeterTime"**. This is corrected to **"date-time"** by obinsectd. #### aliases The OBIS code lists include an **"alias"** section. These names can be used instead of the actual code. The value transformation rules are as described above. These aliases are intended to be consistent across code lists, but this is not verified. They can be changed as you like by editing the list definition, provided a few rules are followed: 1. one alias for each code must be defined, and in the exact order they sent used by the meter. See the list specs for this, or edit the tables of the included lists without changing the order 2. an alias must be an atom. Space and other special characters are not allowed 3. aliases must not collide with any other value code word. An easy way to ensure this is by capitalizing the first letter. #### date-time DLMS/COSEM packets include an optional **"date-time"** field in their header. This is available as a value, **if** sent by the meter. Note that many meters skip this. Dump the **"parserdata"** if you are unsure, and look at the keys **"data-notification"** field. If there is no **"date-time"**, then **"notification-body"** follows immediately after **"long-invoke-id-and-priority"**: ``` "data-notification": { "long-invoke-id-and-priority": 1073741824, "notification-body": [ ... list contents ... ``` A packet with **"date-time"** will look similar to this: ``` "data-notification": { "long-invoke-id-and-priority": 0, "date-time-bug": true, "date-time": 1508474270, "notification-body": { ... list contents ... ``` The **"date-time-bug"** field is informative only and can be ignored. Some meter firmwares prefix the **"date-time"** with an object type code. This is wrong, and is simply ignored by **obinsectd**. **"date-time-bug"** existing and being true indicates that this happened. This prepares for a possible future feature where the exact binary packet received from the meter can be recreated from the parsed JSON object, including this bug. Possibly in the other end of an MQTT publish-subscribe chain... #### metadata obinsectd generates a number of internal variables every time a packet is parsed: 1. **"timestamp"** - local packet arrival time as UNIX epoch 2. **"framlength"** - length of packet, excluding frame markers (0x7e) 3. **"framenumber"** - frame count since **obinsectd** was started 4. **"srcprog"** - application name (**"obinsectd"**) 5. **"srchost"** - host name where **obinsectd** is running 6. **"version"** - **obinsectd** version 7. **"serialport"** - configured serial port, or **"stdin"** 8. **"parsetime"** - total processing time for this packet, in microseconds All of these can be included at once as a separate JSON object by specifying the value **"metadata"**. #### predefined objects **obinsectd** can publish all values received from the meter as a single JSON object, as a normalized and flattened list of **key => value** pairs. Each value is formatted as specified in the **OBIS code** section. There are two such lists available: 1. **"normal"** - using the original OBIS code as key 2. **"alias"** - using the OBIS code alias as key Both objects include the internal **"timestamp"** metadata field, but no other internal fields. The number of keys will obviously depend on the received packets, and should match the OBIS list documentation. An example **"normal"** value for a 10 second list: ```javascript { "1-1:0.2.129.255": "Kamstrup_V0001", "1-1:0.0.5.255": "5706567274389702", "1-1:96.1.1.255": "6841121BN243101040", "1-1:1.7.0.255": 1.828, "1-1:2.7.0.255": 0, "1-1:3.7.0.255": 0, "1-1:4.7.0.255": 0.444, "1-1:31.7.0.255": 6.67, "1-1:51.7.0.255": 2.22, "1-1:71.7.0.255": 6.21, "1-1:32.7.0.255": 233, "1-1:52.7.0.255": 228, "1-1:72.7.0.255": 234, "timestamp": 1559049474 } ``` The exact same packet published using the **"alias"** value: ```javascript { "ListId": "Kamstrup_V0001", "SerialNumber": "5706567274389702", "Model": "6841121BN243101040", "Power": 1.828, "PowerExport": 0, "ReactivePower": 0, "ReactivePowerExport": 0.444, "CurrentL1": 6.67, "CurrentL2": 2.22, "CurrentL3": 6.21, "VoltageL1": 233, "VoltageL2": 228, "VoltageL3": 234, "timestamp": 1559049474 } ``` #### complex objects Specifying a list of values from the sets defined above makes obinsect publish all the values as a single JSON object. For example, this value definition: ``` "/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ], ``` results in an object like this ```javascript { "1-0:1.7.0.255": 4.821, "timestamp": 1559045686 } ``` published to **"/obinsect/json/powerandtime"** every time a **"1-0:1.7.0.255"** value is received. Nesting of complex objects is not allowed, but the pre-defined objects **"normal"** and **"alias"** can be included in a complex object. It is also possible to use the debug objects **"rawpacket"** or **"parserdata"** in a complex object, but the usefulness of this is questionable... #### publishing debug info to MQTT **obinsectd** can also use MQTT for log and debug messages. This includes ##### rawpacket The received packet as a hexbyte string. This includes any unparseable packets. Packets are strings of bytes starting and ending with a frame marker, **0x7e**. Valid packets will have two bytes defining HDLC frame type 3 and the frame length following the frame start marker. "HDLC type 3" is 1010b, or 0xa, so the first hex digit following the frame start marker should always be **a** An example of a published **"rawpacket"** value, broken into lines for readability: ``` 7e a0 e2 2b 21 13 23 9a e6 e7 00 0f 00 00 00 00 0c 07 d0 01 01 06 16 21 00 ff 80 00 01 02 19 0a 0e 4b 61 6d 73 74 72 75 70 5f 56 30 30 30 31 09 06 01 01 00 00 05 ff 0a 10 35 37 30 36 35 36 37 30 30 30 30 30 30 30 30 30 09 06 01 01 60 01 01 ff 0a 12 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 09 06 01 01 01 07 00 ff 06 00 00 00 00 09 06 01 01 02 07 00 ff 06 00 00 00 00 09 06 01 01 03 07 00 ff 06 00 00 00 00 09 06 01 01 04 07 00 ff 06 00 00 00 00 09 06 01 01 1f 07 00 ff 06 00 00 00 00 09 06 01 01 33 07 00 ff 06 00 00 00 00 09 06 01 01 47 07 00 ff 06 00 00 00 00 09 06 01 01 20 07 00 ff 12 00 00 09 06 01 01 34 07 00 ff 12 00 00 09 06 01 01 48 07 00 ff 12 00 00 5b e5 7e ``` ##### parserdata This value specification makes **obinsectd** publish the parsed JSON object, with all the structure and most of the types and values as received from the meter. Scaling and units are not used, and fields have the type specified by the meter. The only exception to the no-touch rule is that OBIS codes are converted from their 6 byte binary original to the more readable OBIS string format. Unique field names for the DLMS/COSEM **"notification-body"** are generated by using the field type together with an index into the object. The packet structure from some meters can be rather complex, using for example nested objects inside arrays. The **"parserdata"** objects are therefore not suitable for direct use, but can be very useful for debugging both meter and parser. The internally generated **"metadata"** object is also included in the **"parserdata"** object. An example: ```javascript { "metadata": { "timestamp": 1559050860, "framelength": 226, "framenumber": 1, "srcprog": "./obinsectd", "srchost": "miraculix", "version": "0.01", "serialport": "stdin", "parsetime": 1029 }, "hdlc": { "format": 10, "segmentation": false, "length": 226, "src": 43, "dst": 33, "control": 19, "hcs": 39459, "fcs": 58715 }, "llc": { "lsap": 230, "dsap": 231, "quality": 0 }, "data-notification": { "long-invoke-id-and-priority": 0, "date-time": 946762380, "notification-body": { "visible-string-0": "Kamstrup_V0001", "obis-1": "1-1:0.0.5.255", "visible-string-2": "5706567000000000", "obis-3": "1-1:96.1.1.255", "visible-string-4": "000000000000000000", "obis-5": "1-1:1.7.0.255", "double-long-unsigned-6": 0, "obis-7": "1-1:2.7.0.255", "double-long-unsigned-8": 0, "obis-9": "1-1:3.7.0.255", "double-long-unsigned-10": 0, "obis-11": "1-1:4.7.0.255", "double-long-unsigned-12": 0, "obis-13": "1-1:31.7.0.255", "double-long-unsigned-14": 0, "obis-15": "1-1:51.7.0.255", "double-long-unsigned-16": 0, "obis-17": "1-1:71.7.0.255", "double-long-unsigned-18": 0, "obis-19": "1-1:32.7.0.255", "long-unsigned-20": 0, "obis-21": "1-1:52.7.0.255", "long-unsigned-22": 0, "obis-23": "1-1:72.7.0.255", "long-unsigned-24": 0 } } } ``` ##### log output **obinsectd** will log some debug info to **stderr** by default, but can also publish the log messages to MQTT topics, using one of the loglevels **"debug"**, **"info"** or **"error"** as value for the topic. Log messages related to MQTT are never published, for obvious reasons... ## reading multiple meters Run one **obinsectd** process per meter if you want to monitor more than one meter. The configuration files may be shared between the processes, as long as the same publishing rules and OBIS lists are wanted.