path: root/README.md

# obinsect - a highly configurable energy meter M-BUS to MQTT proxy


## command line options

```
$ obinsectd 
obinsectd version 0.01, using libmosquitto 1.5.7 and libjson-c 0.12.1

Usage: ./obinsectd [-d] [-c configfile] -s device
                     [-b hostname] [-p port] [-u username [-P password]]
                     [-i id] [-k keepalive] [--insecure]
                     [{--cafile file | --capath dir} [--cert file] [--key file]
                     [--unscaled | --units ]

 -c : Configuration file.  Default: /etc/obinsect.conf
 -d : Enable debugging
 -s : Serial device connected to M-Bus. E.g /dev/ttyUSB0. Use '-' to read from stdin

MQTT client options:
 -b : Broker hostname or IP address. Default: localhost
 -i : Client Id. A random id is generated by default
 -k : Keepalive in seconds. Default: 60
 -p : Broker TCP port. Default: 1883 (8883 when using MQTTS)
 -P : Password.  Default is no authentication
 -u : Username

MQTT over TLS (MQTTS):
 --insecure : Do not validate brokername against certificate
 --cafile   : Trusted CA certificates PEM file
 --capath   : Trusted CA certificates directory
 --cert     : Client certificate
 --key      : Client private key

Published value format:
 --unscaled : Do not scale numbers. Default: false (scaling enabled)
 --units    : Include units. Implies value scaling. Default: false

Example: ./obinsectd -s /dev/ttyUSB0 -b broker.example.com
```

The serial device is the only option required.  This should normally
be an USB to M-BUS bridge connected to a supported meter. Using **-**
as serial device makes obinsectd to read packes from **stdin**
instead.  This can be used to feed it pre-recorded samples.

**obinsectd** will attempt an anonymous MQTT connection to a broker
running on the same host by default. It will continue without MQTT if
this fails for any reason. It will not subscribe to any MQTT topics,
and it will only publish if configured to do so using the
configuration file.


## JSON config file

**obinsect** can be used for debugging the HAN port without any
configuration.  Simply run it with the -d (--debug) option.  The
serial port option is mandatory:

```
obinsectd -d -s /dev/ttyUSB0
```
In debug mode, obinsectd prints a hexdump of all received packets and
the native structure of all sucessfully parsed packets as JSON.

But its real advantage lies in the configurable mapping of data to
MQTT topics, as well as fully configurable OBIS code lists.  The
configuration file has two sections:

 1. "topicmap" - mapping MQTT topics to values
 2. "obisdefs" - list of JSON files with OBIS code defintions

This is an example:

```javascript
{
	"topicmap" :
		{
			"/obinsect/json/alias"       : "alias",
			"/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ],
			"/obinsect/websocket/power"  : "Power",
			"/obinsect/counters"   :
				[
					"timestamp",
					"MeterTime",
					"CumulativeEnergy",
					"CumulativeEnergyExport",
					"CumulativeReactiveEnergy",
					"CumulativeReactiveEnergyExport"
				]
},
	"obisdefs" :
		[
			"/etc/obinsect/aidon_v0001.json",
			"/etc/obinsect/kfm_001.json",
			"/etc/obinsect/kamstrup_v0001.json"
		]
}
```


### topicmap

Any number of MQTT topics can be given, including none.  No specific
topic structure is assumed.  The example above is simply an example.
You do not have to use a common prefix, or organize topics in a tree
at all.

After parsing a packet, data will be published to all topics having
one or more values received from the meter. I.e. **metadata** is not
considered when deciding whether to publish to a topic or not.

The **value** specification can be
 1. an OBIS code in text representation, e.g **"1-1:0.2.129.255"**
 2. a code alias defined in the currently selected code list, e.g. **"Power"**
 3. the **"date-time"** field of the DLMS/COSEM packet
 4. a metadata variable (see below for spec), e.g. **"timestamp"**
 5. one of two predefined objects: **"normal"** or **"alias"**
 6. a list combining one or more of the above
 7. special debug values: **"rawpacket"**, **"parserdata"** or one of the
    loglevels **"debug"**, **"info"** or **"error"**


#### OBIS code

The value is converted to its matching JSON representation,
considering scaling and unit conversion as described below.  By
default, numeric values are returned as numbers scaled according to
the OBIS list definition.

Note that Kaifa meters transmits arrays of values only.  These are
mapped to codes by **obinsectd** using the **"alias"** section of the
matching OBIS list (**"KFM_001"**). This list must therefore be
defined in the correct order.

**"date-time"** values are converted to UNIX epoch time. Some meters a
binary object type for **"MeterTime"**.  This is corrected to
**"date-time"** by obinsectd.


#### aliases

The OBIS code lists include an **"alias"** section. These names can be
used instead of the actual code.  The value transformation rules are
as described above.

These aliases are intended to be consistent across code lists, but
this is not verified. They can be changed as you like by editing the
list definition, provided a few rules are followed:

1. one alias for each code must be defined, and in the exact order
   they sent used by the meter.  See the list specs for this, or edit
   the tables of the included lists without changing the order
2. an alias must be an atom. Space and other special characters are
   not allowed
3. aliases must not collide with any other value code word. An easy
   way to ensure this is by capitalizing the first letter.

#### date-time

DLMS/COSEM packets include an optional **"date-time"** field in their
header. This is available as a value, **if** sent by the meter.  Note
that many meters skip this.  Dump the **"parserdata"** if you are
unsure, and look at the keys **"data-notification"** field.

If there is no **"date-time"**, then **"notification-body"** follows
immediately after **"long-invoke-id-and-priority"**:

```
  "data-notification": {
    "long-invoke-id-and-priority": 1073741824,
    "notification-body": [
	... list contents ...
```

A packet with **"date-time"** will look similar to this:

```
  "data-notification": {
    "long-invoke-id-and-priority": 0,
    "date-time-bug": true,
    "date-time": 1508474270,
    "notification-body": {
	... list contents ...
```

The **"date-time-bug"** field is informative only and can be
ignored. Some meter firmwares prefix the **"date-time"** with an
object type code.  This is wrong, and is simply ignored by
**obinsectd**.  **"date-time-bug"** existing and being true indicates
that this happened.  This prepares for a possible future feature where
the exact binary packet received from the meter can be recreated from
the parsed JSON object, including this bug. Possibly in the other end
of an MQTT publish-subscribe chain...


#### metadata

obinsectd generates a number of internal variables every time a packet
is parsed:

 1. **"timestamp"** - local packet arrival time as UNIX epoch
 2. **"framlength"** - length of packet, excluding frame markers (0x7e)
 3. **"framenumber"** - frame count since **obinsectd** was started
 4. **"srcprog"** - application name (**"obinsectd"**)
 5. **"srchost"** - host name where **obinsectd** is running
 6. **"version"** - **obinsectd** version
 7. **"serialport"** - configured serial port, or **"stdin"**
 8. **"parsetime"** - total processing time for this packet, in microseconds
 
All of these can be included at once as a separate JSON object by
specifying the value **"metadata"**.


#### predefined objects

**obinsectd** can publish all values received from the meter as a
single JSON object, as a normalized and flattened list of **key => value**
pairs.  Each value is formatted as specified in the **OBIS code** section.

There are two such lists available:
 1. **"normal"**  - using the original OBIS code as key
 2. **"alias"** - using the OBIS code alias as key
 
Both objects include the internal **"timestamp"** metadata field, but
no other internal fields.

The number of keys will obviously depend on the received packets, and
should match the OBIS list documentation. An example **"normal"**
value for a 10 second list:

```javascript
{
  "1-1:0.2.129.255": "Kamstrup_V0001",
  "1-1:0.0.5.255": "5706567274389702",
  "1-1:96.1.1.255": "6841121BN243101040",
  "1-1:1.7.0.255": 1.828,
  "1-1:2.7.0.255": 0,
  "1-1:3.7.0.255": 0,
  "1-1:4.7.0.255": 0.444,
  "1-1:31.7.0.255": 6.67,
  "1-1:51.7.0.255": 2.22,
  "1-1:71.7.0.255": 6.21,
  "1-1:32.7.0.255": 233,
  "1-1:52.7.0.255": 228,
  "1-1:72.7.0.255": 234,
  "timestamp": 1559049474
}
```


The exact same packet published using the **"alias"** value:

```javascript
{
  "ListId": "Kamstrup_V0001",
  "SerialNumber": "5706567274389702",
  "Model": "6841121BN243101040",
  "Power": 1.828,
  "PowerExport": 0,
  "ReactivePower": 0,
  "ReactivePowerExport": 0.444,
  "CurrentL1": 6.67,
  "CurrentL2": 2.22,
  "CurrentL3": 6.21,
  "VoltageL1": 233,
  "VoltageL2": 228,
  "VoltageL3": 234,
  "timestamp": 1559049474
}
```



#### complex objects

Specifying a list of values from the sets defined above makes obinsect
publish all the values as a single JSON object.  For example, this
value definition:

```
	"/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ],
```

results in an object like this

```javascript
{
  "1-0:1.7.0.255": 4.821,
  "timestamp": 1559045686
}
```

published to **"/obinsect/json/powerandtime"** every time a
**"1-0:1.7.0.255"** value is received.

Nesting of complex objects is not allowed, but the pre-defined objects
**"normal"** and **"alias"** can be included in a complex object.

It is also possible to use the debug objects **"rawpacket"** or
**"parserdata"** in a complex object, but the usefulness of this is
questionable...



#### publishing debug info to MQTT


**obinsectd** can also use MQTT for log and debug messages. This includes

##### rawpacket

The received packet as a hexbyte string.  This includes any
unparseable packets.  Packets are strings of bytes starting and ending
with a frame marker, **0x7e**. Valid packets will have two bytes
defining HDLC frame type 3 and the frame length following the frame
start marker. "HDLC type 3" is 1010b, or 0xa, so the first hex digit
following the frame start marker should always be **a**

An example of a published **"rawpacket"** value, broken into lines for
readability:

```
7e a0 e2 2b 21 13 23 9a e6 e7 00 0f 00 00 00 00
0c 07 d0 01 01 06 16 21 00 ff 80 00 01 02 19 0a
0e 4b 61 6d 73 74 72 75 70 5f 56 30 30 30 31 09
06 01 01 00 00 05 ff 0a 10 35 37 30 36 35 36 37
30 30 30 30 30 30 30 30 30 09 06 01 01 60 01 01
ff 0a 12 30 30 30 30 30 30 30 30 30 30 30 30 30
30 30 30 30 30 09 06 01 01 01 07 00 ff 06 00 00
00 00 09 06 01 01 02 07 00 ff 06 00 00 00 00 09
06 01 01 03 07 00 ff 06 00 00 00 00 09 06 01 01
04 07 00 ff 06 00 00 00 00 09 06 01 01 1f 07 00
ff 06 00 00 00 00 09 06 01 01 33 07 00 ff 06 00
00 00 00 09 06 01 01 47 07 00 ff 06 00 00 00 00
09 06 01 01 20 07 00 ff 12 00 00 09 06 01 01 34
07 00 ff 12 00 00 09 06 01 01 48 07 00 ff 12 00
00 5b e5 7e
```

##### parserdata

This value specification makes **obinsectd** publish the parsed JSON
object, with all the structure and most of the types and values as
received from the meter. Scaling and units are not used, and fields
have the type specified by the meter.

The only exception to the no-touch rule is that OBIS codes are
converted from their 6 byte binary original to the more readable OBIS
string format.

Unique field names for the DLMS/COSEM **"notification-body"** are
generated by using the field type together with an index into the
object.

The packet structure from some meters can be rather complex, using for
example nested objects inside arrays.  The **"parserdata"** objects
are therefore not suitable for direct use, but can be very useful for
debugging both meter and parser.

The internally generated **"metadata"** object is also included in the
**"parserdata"** object.

An example:

```javascript
{
  "metadata": {
    "timestamp": 1559050860,
    "framelength": 226,
    "framenumber": 1,
    "srcprog": "./obinsectd",
    "srchost": "miraculix",
    "version": "0.01",
    "serialport": "stdin",
    "parsetime": 1029
  },
  "hdlc": {
    "format": 10,
    "segmentation": false,
    "length": 226,
    "src": 43,
    "dst": 33,
    "control": 19,
    "hcs": 39459,
    "fcs": 58715
  },
  "llc": {
    "lsap": 230,
    "dsap": 231,
    "quality": 0
  },
  "data-notification": {
    "long-invoke-id-and-priority": 0,
    "date-time": 946762380,
    "notification-body": {
      "visible-string-0": "Kamstrup_V0001",
      "obis-1": "1-1:0.0.5.255",
      "visible-string-2": "5706567000000000",
      "obis-3": "1-1:96.1.1.255",
      "visible-string-4": "000000000000000000",
      "obis-5": "1-1:1.7.0.255",
      "double-long-unsigned-6": 0,
      "obis-7": "1-1:2.7.0.255",
      "double-long-unsigned-8": 0,
      "obis-9": "1-1:3.7.0.255",
      "double-long-unsigned-10": 0,
      "obis-11": "1-1:4.7.0.255",
      "double-long-unsigned-12": 0,
      "obis-13": "1-1:31.7.0.255",
      "double-long-unsigned-14": 0,
      "obis-15": "1-1:51.7.0.255",
      "double-long-unsigned-16": 0,
      "obis-17": "1-1:71.7.0.255",
      "double-long-unsigned-18": 0,
      "obis-19": "1-1:32.7.0.255",
      "long-unsigned-20": 0,
      "obis-21": "1-1:52.7.0.255",
      "long-unsigned-22": 0,
      "obis-23": "1-1:72.7.0.255",
      "long-unsigned-24": 0
    }
  }
}
```


##### log output

**obinsectd** will log some debug info to **stderr** by default, but
can also publish the log messages to MQTT topics, using one of the
loglevels **"debug"**, **"info"** or **"error"** as value for the
topic.

Log messages related to MQTT are never published, for obvious
reasons...



## reading multiple meters

Run one **obinsectd** process per meter if you want to monitor more
than one meter. The configuration files may be shared between the
processes, as long as the same publishing rules and OBIS lists are
wanted.