path: root/README.md

# obinsect - a highly configurable energy meter M-BUS to MQTT proxy


## command line options

```
$ obinsectd 
obinsectd version 0.01, using libmosquitto 1.5.7 and libjson-c 0.12.1

Usage: ./obinsectd [-d] [-c configfile] -s device
                     [-b hostname] [-p port] [-u username [-P password]]
                     [-i id] [-k keepalive] [--insecure]
                     [{--cafile file | --capath dir} [--cert file] [--key file]
                     [--unscaled | --units ]

 -c : Configuration file.  Default: /etc/obinsect.conf
 -d : Enable debugging
 -s : Serial device connected to M-Bus. E.g /dev/ttyUSB0. Use '-' to read from stdin

MQTT client options:
 -b : Broker hostname or IP address. Default: localhost
 -i : Client Id. A random id is generated by default
 -k : Keepalive in seconds. Default: 60
 -p : Broker TCP port. Default: 1883 (8883 when using MQTTS)
 -P : Password.  Default is no authentication
 -u : Username

MQTT over TLS (MQTTS):
 --insecure : Do not validate brokername against certificate
 --cafile   : Trusted CA certificates PEM file
 --capath   : Trusted CA certificates directory
 --cert     : Client certificate
 --key      : Client private key

Published value format:
 --unscaled : Do not scale numbers. Default: false (scaling enabled)
 --units    : Include units. Implies value scaling. Default: false

Example: ./obinsectd -s /dev/ttyUSB0 -b broker.example.com
```

### -d or --debug

Turns on very verbose debug output to stderr. Note that this is
independent of the MQTT debug channel described [below](#Logging).


### -c or --config

Change location of the JSON formatted configuration file.  See
[below](#Config) for details

### -s or --serial

The serial device is the only mandatory option.  This should normally
be an USB to M-BUS bridge connected to a supported meter

Using **"-"** as serial device to read packes from **stdin** instead.
This is useful for feeding it replayed recorded samples.

### MQTT client related options

**obinsectd** will attempt an anonymous MQTT connection to a broker
running on the same host by default. It will continue without MQTT if
this fails for any reason. It will not subscribe to any MQTT topics,
and it will only publish if configured to do so using the
[configuration file](#Config).


### <a name="Unscaled"></a>--unscaled

**obinsectd** will by default scale all numeric values according to
the OBIS list definition. This means that many values end up as floats
instead of integers, which might be undesirable.  The scaling will
also suffer from the usual floating point division issues, where the
number is off by a small fraction way outside the precision.


Typical example of a published **"alias"** object (see [below](#Predefined)):
```javascript
{
  "ListId": "KFM_001",
  "SerialNumber": "MA304H3E",
  "Model": 1176,
  "Power": 0,
  "PowerExport": 131,
  "ReactivePower": 0,
  "ReactivePowerExport": 2208,
  "CurrentL1": 3.841,
  "CurrentL2": 3.794,
  "CurrentL3": 2.3890000000000002,
  "VoltageL1": 0,
  "VoltageL2": 239.70000000000002,
  "timestamp": 1559117502
}
```

Setting this option will disable the scaling, making **obinsectd**
publish the raw integer values as received.

Same example as above, with **--unscaled** set:

```javascript
{
  "ListId": "KFM_001",
  "SerialNumber": "MA304H3E",
  "Model": 1176,
  "Power": 0,
  "PowerExport": 131,
  "ReactivePower": 0,
  "ReactivePowerExport": 2208,
  "CurrentL1": 3841,
  "CurrentL2": 3794,
  "CurrentL3": 2389,
  "VoltageL1": 0,
  "VoltageL2": 2397,
  "timestamp": 1559117502
}
```

### --units

By default all values are publish as JSON numbers, either integer or
double. This makes the values usable on the other end without further
parsing.  This suits most usecases like updating gauges, databases etc
best.  The unit is not published at all in this case.


Setting **"--units"** will change all values with a unit defined in
the OBIS list definiton, to strings.  The strings are formatted with
the number of decimals specified in the list definition, followed by a
space and the unit.

This can be useful for usecase where published values are used as
strings anyway, for example in a simple text log or in a text element
on a web page.

Note that this setting is global.  It is currently not possible to
publish some topics or values with units and some without.

**--units** implies scaling, and will silently re-enable scaling even
if [**--unscaled**](#Unscaled) is set.


Reusing the [**"alias"**](#Predefined) object example, but this time with units
enabled:

```javascript
{
  "ListId": "KFM_001",
  "SerialNumber": "MA304H3E",
  "Model": 1176,
  "Power": "0 W",
  "PowerExport": "131 W",
  "ReactivePower": "0 Var",
  "ReactivePowerExport": "2208 Var",
  "CurrentL1": "3.841 A",
  "CurrentL2": "3.794 A",
  "CurrentL3": "2.389 A",
  "VoltageL1": "0.0 V",
  "VoltageL2": "239.7 V",
  "timestamp": 1559117502
}

```


## <a name"Config"></a>JSON config file

**obinsect** can be used for debugging the HAN port without any
configuration.  Simply run it with the -d (--debug) option.  The
serial port option is mandatory:

```
obinsectd -d -s /dev/ttyUSB0
```
In debug mode, obinsectd prints a hexdump of all received packets and
the native structure of all sucessfully parsed packets as JSON.

But its real advantage lies in the configurable mapping of data to
MQTT topics, as well as fully configurable OBIS code lists.  The
configuration file has two sections:

 1. ["topicmap"](#Topicmap) - mapping MQTT topics to values
 2. ["obisdefs"](#Obisdefs) - list of JSON files with OBIS code defintions

This is a complete configuration example:

```javascript
{
    "topicmap" : {
		"/obinsect/log/debug"        : "debug",
		"/obinsect/log/info"         : "info",
		"/obinsect/log/error"        : "error",
		"/obinsect/websocket/power"  : "Power",
		"/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ],
		"/obinsect/json/realtime"    : "alias",
		"/obinsect/json/counters"    : [
			"timestamp",
			"MeterTime",
			"CumulativeEnergy",
			"CumulativeEnergyExport",
			"CumulativeReactiveEnergy",
			"CumulativeReactiveEnergyExport"
		]
	},
	"obisdefs" : [
		"/etc/obinsect/aidon_v0001.json",
		"/etc/obinsect/kfm_001.json",
		"/etc/obinsect/kamstrup_v0001.json"
    ]
}
```


### <a name="Topicmap"></a>topicmap

Any number of MQTT topics can be given, including none.  No specific
topic structure is assumed.  The example above is simply an example.
You do not have to use a common prefix, or organize topics in a tree
at all.

After parsing a packet, data will be published to all topics having
one or more values received from the meter. I.e. **metadata** is not
considered when deciding whether to publish to a topic or not.

The **value** specification can be
 1. an [OBIS code](#Obiscode) in text representation, e.g **"1-1:0.2.129.255"**
 2. a [code alias](#Codealias) defined in the currently selected code list, e.g. **"Power"**
 3. the [**"date-time"**](#Datetime) field of the DLMS/COSEM packet
 4. a [metadata](#Metadata) variable (see below for spec), e.g. **"timestamp"**
 5. one of two [predefined objects](#Predefined): **"normal"** or **"alias"**
 6. a [list combining](#Complex) one or more of the above
 7. special [debug](#Debug) values: **"rawpacket"**, **"parserdata"** or one of the
    [loglevels](#Logging) **"debug"**, **"info"** or **"error"**


#### <a name="Obiscode"></a>OBIS code

The value is converted to its matching JSON representation,
considering scaling and unit conversion as described above.  By
default, numeric values are returned as numbers scaled according to
the OBIS list definition.

Note that Kaifa meters transmits arrays of values only.  These are
mapped to codes by **obinsectd** using the **"alias"** section of the
matching OBIS list (**"KFM_001"**). This list must therefore be
defined in the correct order.

**"date-time"** values are converted to UNIX epoch time. Some meters a
binary object type for **"MeterTime"**.  This is corrected to
**"date-time"** by obinsectd.


#### <a name="Codealias"></a>aliases

The OBIS code lists include an **"alias"** section. These names can be
used instead of the actual code.  The value transformation rules are
as described above.

These aliases are intended to be consistent across code lists, but
this is not verified. They can be changed as you like by editing the
list definition, provided a few rules are followed:

1. one alias for each code must be defined, and in the exact order
   they sent used by the meter.  See the list specs for this, or edit
   the tables of the included lists without changing the order
2. an alias must be an atom. Space and other special characters are
   not allowed
3. aliases must not collide with any other value code word. An easy
   way to ensure this is by capitalizing the first letter.

#### <a name="Datetime"></a>date-time

DLMS/COSEM packets include an optional **"date-time"** field in their
header. This is available as a value, **if** sent by the meter.  Note
that many meters skip this.  Dump the **"parserdata"** if you are
unsure, and look at the keys **"data-notification"** field.

If there is no **"date-time"**, then **"notification-body"** follows
immediately after **"long-invoke-id-and-priority"**:

```
  "data-notification": {
    "long-invoke-id-and-priority": 1073741824,
    "notification-body": [
	... list contents ...
```

A packet with **"date-time"** will look similar to this:

```
  "data-notification": {
    "long-invoke-id-and-priority": 0,
    "date-time-bug": true,
    "date-time": 1508474270,
    "notification-body": {
	... list contents ...
```

The **"date-time-bug"** field is informative only and can be
ignored. Some meter firmwares prefix the **"date-time"** with an
object type code.  This is wrong, and is simply ignored by
**obinsectd**.  **"date-time-bug"** existing and being true indicates
that this happened.  This prepares for a possible future feature where
the exact binary packet received from the meter can be recreated from
the parsed JSON object, including this bug. Possibly in the other end
of an MQTT publish-subscribe chain...


#### <a name="Metadata"></a>metadata

obinsectd generates a number of internal variables every time a packet
is parsed:

 1. **"timestamp"** - local packet arrival time as UNIX epoch
 2. **"framlength"** - length of packet, excluding frame markers (0x7e)
 3. **"framenumber"** - frame count since **obinsectd** was started
 4. **"srcprog"** - application name (**"obinsectd"**)
 5. **"srchost"** - host name where **obinsectd** is running
 6. **"version"** - **obinsectd** version
 7. **"serialport"** - configured serial port, or **"stdin"**
 8. **"parsetime"** - total processing time for this packet, in microseconds
 
All of these can be included at once as a separate JSON object by
specifying the value **"metadata"**.


#### <a name="Predefined"></a>predefined objects

**obinsectd** can publish all values received from the meter as a
single JSON object, as a normalized and flattened list of **key => value**
pairs.  Each value is formatted as specified in the **OBIS code** section.

There are two such lists available:
 1. **"normal"**  - using the original OBIS code as key
 2. **"alias"** - using the OBIS code alias as key
 
Both objects include the internal **"timestamp"** metadata field, but
no other internal fields.

The number of keys will obviously depend on the received packets, and
should match the OBIS list documentation. An example **"normal"**
value for a 10 second list:

```javascript
{
  "1-1:0.2.129.255": "Kamstrup_V0001",
  "1-1:0.0.5.255": "5706567274389702",
  "1-1:96.1.1.255": "6841121BN243101040",
  "1-1:1.7.0.255": 1.828,
  "1-1:2.7.0.255": 0,
  "1-1:3.7.0.255": 0,
  "1-1:4.7.0.255": 0.444,
  "1-1:31.7.0.255": 6.67,
  "1-1:51.7.0.255": 2.22,
  "1-1:71.7.0.255": 6.21,
  "1-1:32.7.0.255": 233,
  "1-1:52.7.0.255": 228,
  "1-1:72.7.0.255": 234,
  "timestamp": 1559049474
}
```


The exact same packet published using the **"alias"** value:

```javascript
{
  "ListId": "Kamstrup_V0001",
  "SerialNumber": "5706567274389702",
  "Model": "6841121BN243101040",
  "Power": 1.828,
  "PowerExport": 0,
  "ReactivePower": 0,
  "ReactivePowerExport": 0.444,
  "CurrentL1": 6.67,
  "CurrentL2": 2.22,
  "CurrentL3": 6.21,
  "VoltageL1": 233,
  "VoltageL2": 228,
  "VoltageL3": 234,
  "timestamp": 1559049474
}
```


#### <a name="Complex"></a>complex objects

Specifying a list of values from the sets defined above makes obinsect
publish all the values as a single JSON object.  For example, this
value definition:

```
	"/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ],
```

results in an object like this

```javascript
{
  "1-0:1.7.0.255": 4.821,
  "timestamp": 1559045686
}
```

published to **"/obinsect/json/powerandtime"** every time a
**"1-0:1.7.0.255"** value is received.

Nesting of complex objects is not allowed, but the pre-defined objects
**"normal"** and **"alias"** can be included in a complex object.

It is also possible to use the debug objects **"rawpacket"** or
**"parserdata"** in a complex object, but the usefulness of this is
questionable...



#### <a name="Debug"></a>publishing debug info to MQTT


**obinsectd** can also use MQTT for log and debug messages. This includes

##### rawpacket

The received packet as a hexbyte string.  This includes any
unparseable packets.  Packets are strings of bytes starting and ending
with a frame marker, **0x7e**. Valid packets will have two bytes
defining HDLC frame type 3 and the frame length following the frame
start marker. "HDLC type 3" is 1010b, or 0xa, so the first hex digit
following the frame start marker should always be **a**

An example of a published **"rawpacket"** value, broken into lines for
readability:

```
7e a0 e2 2b 21 13 23 9a e6 e7 00 0f 00 00 00 00
0c 07 d0 01 01 06 16 21 00 ff 80 00 01 02 19 0a
0e 4b 61 6d 73 74 72 75 70 5f 56 30 30 30 31 09
06 01 01 00 00 05 ff 0a 10 35 37 30 36 35 36 37
30 30 30 30 30 30 30 30 30 09 06 01 01 60 01 01
ff 0a 12 30 30 30 30 30 30 30 30 30 30 30 30 30
30 30 30 30 30 09 06 01 01 01 07 00 ff 06 00 00
00 00 09 06 01 01 02 07 00 ff 06 00 00 00 00 09
06 01 01 03 07 00 ff 06 00 00 00 00 09 06 01 01
04 07 00 ff 06 00 00 00 00 09 06 01 01 1f 07 00
ff 06 00 00 00 00 09 06 01 01 33 07 00 ff 06 00
00 00 00 09 06 01 01 47 07 00 ff 06 00 00 00 00
09 06 01 01 20 07 00 ff 12 00 00 09 06 01 01 34
07 00 ff 12 00 00 09 06 01 01 48 07 00 ff 12 00
00 5b e5 7e
```

##### parserdata

This value specification makes **obinsectd** publish the parsed JSON
object, with all the structure and most of the types and values as
received from the meter. Scaling and units are not used, and fields
have the type specified by the meter.

The only exception to the no-touch rule is that OBIS codes are
converted from their 6 byte binary original to the more readable OBIS
string format.

Unique field names for the DLMS/COSEM **"notification-body"** are
generated by using the field type together with an index into the
object.

The packet structure from some meters can be rather complex, using for
example nested objects inside arrays.  The **"parserdata"** objects
are therefore not suitable for direct use, but can be very useful for
debugging both meter and parser.

The internally generated **"metadata"** object is also included in the
**"parserdata"** object.

An example:

```javascript
{
  "metadata": {
    "timestamp": 1559050860,
    "framelength": 226,
    "framenumber": 1,
    "srcprog": "./obinsectd",
    "srchost": "miraculix",
    "version": "0.01",
    "serialport": "stdin",
    "parsetime": 1029
  },
  "hdlc": {
    "format": 10,
    "segmentation": false,
    "length": 226,
    "src": 43,
    "dst": 33,
    "control": 19,
    "hcs": 39459,
    "fcs": 58715
  },
  "llc": {
    "lsap": 230,
    "dsap": 231,
    "quality": 0
  },
  "data-notification": {
    "long-invoke-id-and-priority": 0,
    "date-time": 946762380,
    "notification-body": {
      "visible-string-0": "Kamstrup_V0001",
      "obis-1": "1-1:0.0.5.255",
      "visible-string-2": "5706567000000000",
      "obis-3": "1-1:96.1.1.255",
      "visible-string-4": "000000000000000000",
      "obis-5": "1-1:1.7.0.255",
      "double-long-unsigned-6": 0,
      "obis-7": "1-1:2.7.0.255",
      "double-long-unsigned-8": 0,
      "obis-9": "1-1:3.7.0.255",
      "double-long-unsigned-10": 0,
      "obis-11": "1-1:4.7.0.255",
      "double-long-unsigned-12": 0,
      "obis-13": "1-1:31.7.0.255",
      "double-long-unsigned-14": 0,
      "obis-15": "1-1:51.7.0.255",
      "double-long-unsigned-16": 0,
      "obis-17": "1-1:71.7.0.255",
      "double-long-unsigned-18": 0,
      "obis-19": "1-1:32.7.0.255",
      "long-unsigned-20": 0,
      "obis-21": "1-1:52.7.0.255",
      "long-unsigned-22": 0,
      "obis-23": "1-1:72.7.0.255",
      "long-unsigned-24": 0
    }
  }
}
```


##### <a name="Logging"></a>log output

**obinsectd** will log some debug info to **stderr** by default, but
can also publish the log messages to MQTT topics, using one of the
loglevels **"debug"**, **"info"** or **"error"** as value for the
topic.

Log messages related to MQTT are never published, for obvious
reasons...



### <a name="Obisdefs"></a>OBIS code lists

Each meter vendor has defined specifics sets of OBIS codes and the
intervals these are published.  These lists define the relative order
of the values, resolution and scaling, and units.

**obinsect** extends the list definitions with a code alias for each
OBIS code. These aliases are examples only, and can be changed as
required.

The order of the OBIS code to alias map is significant, however, and
must not be changed. Values are sometimes sent by the meter without
any key or code label. The relative index into the **alias** map is
used to map such values to their respective OBIS codes.


**obinsect** comes with three OBIS code lists which can be used as
examples when adding support for other meters:

	1. aidon_v0001.json
	2. kfm_001.json
	3. kamstrup_v0001.json



## reading multiple meters

Run one **obinsectd** process per meter if you want to monitor more
than one meter. The configuration files may be shared between the
processes, as long as the same publishing rules and OBIS lists are
wanted.