docs: update

Signed-off-by: Bjørn Mork <bjorn@mork.no>

1 files changed, 392 insertions, 77 deletions

diff --git a/README.md b/README.md
index 3a9b1ce..d24f297 100644
--- a/README.md
+++ b/README.md
@@ -1,131 +1,446 @@
-# obinsect - configurable energy meter HAN M-BUS to MQTT proxy
+# obinsect - a highly configurable energy meter M-BUS to MQTT proxy
 
-## configuration
 
-The HAN reader can be used for debugging without any
-configuration. Publishing data to MQTT requires a configuration file
-in JSON format. This maps the meter values to MQTT topic(s).
+## command line options
+
+```
+$ obinsectd 
+obinsectd version 0.01, using libmosquitto 1.5.7 and libjson-c 0.12.1
+
+Usage: ./obinsectd [-d] [-c configfile] -s device
+                     [-b hostname] [-p port] [-u username [-P password]]
+                     [-i id] [-k keepalive] [--insecure]
+                     [{--cafile file | --capath dir} [--cert file] [--key file]
+                     [--unscaled | --units ]
+
+ -c : Configuration file.  Default: /etc/obinsect.conf
+ -d : Enable debugging
+ -s : Serial device connected to M-Bus. E.g /dev/ttyUSB0. Use '-' to read from stdin
+
+MQTT client options:
+ -b : Broker hostname or IP address. Default: localhost
+ -i : Client Id. A random id is generated by default
+ -k : Keepalive in seconds. Default: 60
+ -p : Broker TCP port. Default: 1883 (8883 when using MQTTS)
+ -P : Password.  Default is no authentication
+ -u : Username
+
+MQTT over TLS (MQTTS):
+ --insecure : Do not validate brokername against certificate
+ --cafile   : Trusted CA certificates PEM file
+ --capath   : Trusted CA certificates directory
+ --cert     : Client certificate
+ --key      : Client private key
+
+Published value format:
+ --unscaled : Do not scale numbers. Default: false (scaling enabled)
+ --units    : Include units. Implies value scaling. Default: false
+
+Example: ./obinsectd -s /dev/ttyUSB0 -b broker.example.com
+```
+
+The serial device is the only option required.  This should normally
+be an USB to M-BUS bridge connected to a supported meter. Using **-**
+as serial device makes obinsectd to read packes from **stdin**
+instead.  This can be used to feed it pre-recorded samples.
+
+**obinsectd** will attempt an anonymous MQTT connection to a broker
+running on the same host by default. It will continue without MQTT if
+this fails for any reason. It will not subscribe to any MQTT topics,
+and it will only publish if configured to do so using the
+configuration file.
+
+
+## JSON config file
+
+**obinsect** can be used for debugging the HAN port without any
+configuration.  Simply run it with the -d (--debug) option.  The
+serial port option is mandatory:
+
+```
+obinsectd -d -s /dev/ttyUSB0
+```
+In debug mode, obinsectd prints a hexdump of all received packets and
+the native structure of all sucessfully parsed packets as JSON.
+
+But its real advantage lies in the configurable mapping of data to
+MQTT topics, as well as fully configurable OBIS code lists.  The
+configuration file has two sections:
+
+ 1. "topicmap" - mapping MQTT topics to values
+ 2. "obisdefs" - list of JSON files with OBIS code defintions
+
+This is an example:
 
-The configuration file must specificy two fields:
- 1. "topicmap" - a map of MQTT "topic" to "value"
- 2. "obisdefs" - a list of files defining the supported OBIS Lists.
-                 one file for each known list name
- 
-This is a sample configuration file:
 ```javascript
 {
 	"topicmap" :
 		{
-			"/obinsect/debug/channel"   : "debug",
-			"/obinsect/debug/raw"       : "rawhexdump",
-			"/obinsect/json/normalized" : "normal",
-			"/obinsect/json/full"       : "full",
-			"/obinsect/value/power"     : "1-1:0.2.129.255",
-			"/obinsect/value/export"    : "PowerExport",
-			"/obinsect/value/custom"    : [ "timestamp", "1-1:0.2.129.255" ]
-		},
+			"/obinsect/json/alias"       : "alias",
+			"/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ],
+			"/obinsect/websocket/power"  : "Power",
+			"/obinsect/counters"   :
+				[
+					"timestamp",
+					"MeterTime",
+					"CumulativeEnergy",
+					"CumulativeEnergyExport",
+					"CumulativeReactiveEnergy",
+					"CumulativeReactiveEnergyExport"
+				]
+},
 	"obisdefs" :
 		[
 			"/etc/obinsect/aidon_v0001.json",
-			"/etc/obinsect/kaifa_v0001.json",
+			"/etc/obinsect/kfm_001.json",
 			"/etc/obinsect/kamstrup_v0001.json"
 		]
 }
 ```
 
-Any number of MQTT topics can be given, including zero. Data will be
-published to the topics with an updated value every time a DLMS/COSEM
-packet is received from the energy meter.
 
-Note that Kaifa meters transmits arrays of values only.  The OBIS list
-definition is used to map these to the correct OBIS code keys, with
-any associated scale and units. This implies that list of codes for
-these meters must be complete and in the exxact order received from
-the meter.
+### topicmap
+
+Any number of MQTT topics can be given, including none.  No specific
+topic structure is assumed.  The example above is simply an example.
+You do not have to use a common prefix, or organize topics in a tree
+at all.
 
+After parsing a packet, data will be published to all topics having
+one or more values received from the meter. I.e. **metadata** is not
+considered when deciding whether to publish to a topic or not.
 
-### value specification
+The **value** specification can be
+ 1. an OBIS code in text representation, e.g **"1-1:0.2.129.255"**
+ 2. a code alias defined in the currently selected code list, e.g. **"Power"**
+ 3. the **"date-time"** field of the DLMS/COSEM packet
+ 4. a metadata variable (see below for spec), e.g. **"timestamp"**
+ 5. one of two predefined objects: **"normal"** or **"alias"**
+ 6. a list combining one or more of the above
+ 7. special debug values: **"rawpacket"**, **"parserdata"** or one of the
+    loglevels **"debug"**, **"info"** or **"error"**
 
-Each topic points to the key used to look up values for that topic.
 
-These keys can:
+#### OBIS code
 
+The value is converted to its matching JSON representation,
+considering scaling and unit conversion as described below.  By
+default, numeric values are returned as numbers scaled according to
+the OBIS list definition.
 
-#### an obis code
+Note that Kaifa meters transmits arrays of values only.  These are
+mapped to codes by **obinsectd** using the **"alias"** section of the
+matching OBIS list (**"KFM_001"**). This list must therefore be
+defined in the correct order.
 
-Any obis code listed in the "obisdef" file.  The value will divided by
-the "scale" field, if present, and used directly without any JSON
-wrapping.
+**"date-time"** values are converted to UNIX epoch time. Some meters a
+binary object type for **"MeterTime"**.  This is corrected to
+**"date-time"** by obinsectd.
 
-Integer or float values are converted to their text representation.
 
-Date and time values are converted the text representation of an UNIX
-epoch time.
+#### aliases
 
-Units are not published with the value.
+The OBIS code lists include an **"alias"** section. These names can be
+used instead of the actual code.  The value transformation rules are
+as described above.
 
-#### a named obis code
+These aliases are intended to be consistent across code lists, but
+this is not verified. They can be changed as you like by editing the
+list definition, provided a few rules are followed:
 
-The name of an obis code, as it is defined in the **"obisdef"** file,
-can be used as an alias for the code. See above for value specification.
+1. one alias for each code must be defined, and in the exact order
+   they sent used by the meter.  See the list specs for this, or edit
+   the tables of the included lists without changing the order
+2. an alias must be an atom. Space and other special characters are
+   not allowed
+3. aliases must not collide with any other value code word. An easy
+   way to ensure this is by capitalizing the first letter.
 
+#### date-time
 
-#### "timestamp"
+DLMS/COSEM packets include an optional **"date-time"** field in their
+header. This is available as a value, **if** sent by the meter.  Note
+that many meters skip this.  Dump the **"parserdata"** if you are
+unsure, and look at the keys **"data-notification"** field.
 
-Text representation of the UNIX epoch time we received the packet
-causing this publish event
+If there is no **"date-time"**, then **"notification-body"** follows
+immediately after **"long-invoke-id-and-priority"**:
 
-#### "debug"
+```
+  "data-notification": {
+    "long-invoke-id-and-priority": 1073741824,
+    "notification-body": [
+	... list contents ...
+```
 
-Debugging messages from obinsectd.  Any format.  ASCII text can be
-assumed.
+A packet with **"date-time"** will look similar to this:
 
+```
+  "data-notification": {
+    "long-invoke-id-and-priority": 0,
+    "date-time-bug": true,
+    "date-time": 1508474270,
+    "notification-body": {
+	... list contents ...
+```
 
-#### "rawhexdump"
+The **"date-time-bug"** field is informative only and can be
+ignored. Some meter firmwares prefix the **"date-time"** with an
+object type code.  This is wrong, and is simply ignored by
+**obinsectd**.  **"date-time-bug"** existing and being true indicates
+that this happened.  This prepares for a possible future feature where
+the exact binary packet received from the meter can be recreated from
+the parsed JSON object, including this bug. Possibly in the other end
+of an MQTT publish-subscribe chain...
+
+
+#### metadata
+
+obinsectd generates a number of internal variables every time a packet
+is parsed:
+
+ 1. **"timestamp"** - local packet arrival time as UNIX epoch
+ 2. **"framlength"** - length of packet, excluding frame markers (0x7e)
+ 3. **"framenumber"** - frame count since **obinsectd** was started
+ 4. **"srcprog"** - application name (**"obinsectd"**)
+ 5. **"srchost"** - host name where **obinsectd** is running
+ 6. **"version"** - **obinsectd** version
+ 7. **"serialport"** - configured serial port, or **"stdin"**
+ 8. **"parsetime"** - total processing time for this packet, in microseconds
+ 
+All of these can be included at once as a separate JSON object by
+specifying the value **"metadata"**.
 
-The packet as a string of hex codes.
 
+#### predefined objects
 
-#### "full"
+**obinsectd** can publish all values received from the meter as a
+single JSON object, as a normalized and flattened list of **key => value**
+pairs.  Each value is formatted as specified in the **OBIS code** section.
 
-A JSON struct with all parts of the received packet, including headers
-and checksum. The received DLMS/COSEM structure is preserved in JSON
-as received, so the exact format is meter dependent.
+There are two such lists available:
+ 1. **"normal"**  - using the original OBIS code as key
+ 2. **"alias"** - using the OBIS code alias as key
+ 
+Both objects include the internal **"timestamp"** metadata field, but
+no other internal fields.
 
-This is mostly useful for debugging. The original packet can be
-reconstructed from this JSON struct.
+The number of keys will obviously depend on the received packets, and
+should match the OBIS list documentation. An example **"normal"**
+value for a 10 second list:
 
+```javascript
+{
+  "1-1:0.2.129.255": "Kamstrup_V0001",
+  "1-1:0.0.5.255": "5706567274389702",
+  "1-1:96.1.1.255": "6841121BN243101040",
+  "1-1:1.7.0.255": 1.828,
+  "1-1:2.7.0.255": 0,
+  "1-1:3.7.0.255": 0,
+  "1-1:4.7.0.255": 0.444,
+  "1-1:31.7.0.255": 6.67,
+  "1-1:51.7.0.255": 2.22,
+  "1-1:71.7.0.255": 6.21,
+  "1-1:32.7.0.255": 233,
+  "1-1:52.7.0.255": 228,
+  "1-1:72.7.0.255": 234,
+  "timestamp": 1559049474
+}
+```
 
-#### "normal"
 
-A JSON struct where the received data has been flattened to an a set
-of **"obiscode" : "value"**.  A timestamp field is also included
- 
-This format is similar for all meter types, but the values are not
-sscaled so they cannot be directly compared.
+The exact same packet published using the **"alias"** value:
 
+```javascript
+{
+  "ListId": "Kamstrup_V0001",
+  "SerialNumber": "5706567274389702",
+  "Model": "6841121BN243101040",
+  "Power": 1.828,
+  "PowerExport": 0,
+  "ReactivePower": 0,
+  "ReactivePowerExport": 0.444,
+  "CurrentL1": 6.67,
+  "CurrentL2": 2.22,
+  "CurrentL3": 6.21,
+  "VoltageL1": 233,
+  "VoltageL2": 228,
+  "VoltageL3": 234,
+  "timestamp": 1559049474
+}
+```
 
-#### array of keys
 
-Will publishing a JSON struct with the given set of values, regardless
-of what the meter sends in each packet. Each element of the array
-becomes a key in the JSON, having values as defined above.
 
-Only fields having an updated value will be included, so the actual
-number of fields can vary from packet to packet.
+#### complex objects
 
-A JSON struct will be published when any of the included fields are
-updated, except for **"timestamp"**.
+Specifying a list of values from the sets defined above makes obinsect
+publish all the values as a single JSON object.  For example, this
+value definition:
+
+```
+	"/obinsect/json/powerandtime": [ "timestamp", "1-0:1.7.0.255" ],
+```
+
+results in an object like this
+
+```javascript
+{
+  "1-0:1.7.0.255": 4.821,
+  "timestamp": 1559045686
+}
+```
+
+published to **"/obinsect/json/powerandtime"** every time a
+**"1-0:1.7.0.255"** value is received.
+
+Nesting of complex objects is not allowed, but the pre-defined objects
+**"normal"** and **"alias"** can be included in a complex object.
+
+It is also possible to use the debug objects **"rawpacket"** or
+**"parserdata"** in a complex object, but the usefulness of this is
+questionable...
+
+
+
+#### publishing debug info to MQTT
+
+
+**obinsectd** can also use MQTT for log and debug messages. This includes
+
+##### rawpacket
+
+The received packet as a hexbyte string.  This includes any
+unparseable packets.  Packets are strings of bytes starting and ending
+with a frame marker, **0x7e**. Valid packets will have two bytes
+defining HDLC frame type 3 and the frame length following the frame
+start marker. "HDLC type 3" is 1010b, or 0xa, so the first hex digit
+following the frame start marker should always be **a**
+
+An example of a published **"rawpacket"** value, broken into lines for
+readability:
+
+```
+7e a0 e2 2b 21 13 23 9a e6 e7 00 0f 00 00 00 00
+0c 07 d0 01 01 06 16 21 00 ff 80 00 01 02 19 0a
+0e 4b 61 6d 73 74 72 75 70 5f 56 30 30 30 31 09
+06 01 01 00 00 05 ff 0a 10 35 37 30 36 35 36 37
+30 30 30 30 30 30 30 30 30 09 06 01 01 60 01 01
+ff 0a 12 30 30 30 30 30 30 30 30 30 30 30 30 30
+30 30 30 30 30 09 06 01 01 01 07 00 ff 06 00 00
+00 00 09 06 01 01 02 07 00 ff 06 00 00 00 00 09
+06 01 01 03 07 00 ff 06 00 00 00 00 09 06 01 01
+04 07 00 ff 06 00 00 00 00 09 06 01 01 1f 07 00
+ff 06 00 00 00 00 09 06 01 01 33 07 00 ff 06 00
+00 00 00 09 06 01 01 47 07 00 ff 06 00 00 00 00
+09 06 01 01 20 07 00 ff 12 00 00 09 06 01 01 34
+07 00 ff 12 00 00 09 06 01 01 48 07 00 ff 12 00
+00 5b e5 7e
+```
+
+##### parserdata
+
+This value specification makes **obinsectd** publish the parsed JSON
+object, with all the structure and most of the types and values as
+received from the meter. Scaling and units are not used, and fields
+have the type specified by the meter.
+
+The only exception to the no-touch rule is that OBIS codes are
+converted from their 6 byte binary original to the more readable OBIS
+string format.
+
+Unique field names for the DLMS/COSEM **"notification-body"** are
+generated by using the field type together with an index into the
+object.
+
+The packet structure from some meters can be rather complex, using for
+example nested objects inside arrays.  The **"parserdata"** objects
+are therefore not suitable for direct use, but can be very useful for
+debugging both meter and parser.
+
+The internally generated **"metadata"** object is also included in the
+**"parserdata"** object.
+
+An example:
+
+```javascript
+{
+  "metadata": {
+    "timestamp": 1559050860,
+    "framelength": 226,
+    "framenumber": 1,
+    "srcprog": "./obinsectd",
+    "srchost": "miraculix",
+    "version": "0.01",
+    "serialport": "stdin",
+    "parsetime": 1029
+  },
+  "hdlc": {
+    "format": 10,
+    "segmentation": false,
+    "length": 226,
+    "src": 43,
+    "dst": 33,
+    "control": 19,
+    "hcs": 39459,
+    "fcs": 58715
+  },
+  "llc": {
+    "lsap": 230,
+    "dsap": 231,
+    "quality": 0
+  },
+  "data-notification": {
+    "long-invoke-id-and-priority": 0,
+    "date-time": 946762380,
+    "notification-body": {
+      "visible-string-0": "Kamstrup_V0001",
+      "obis-1": "1-1:0.0.5.255",
+      "visible-string-2": "5706567000000000",
+      "obis-3": "1-1:96.1.1.255",
+      "visible-string-4": "000000000000000000",
+      "obis-5": "1-1:1.7.0.255",
+      "double-long-unsigned-6": 0,
+      "obis-7": "1-1:2.7.0.255",
+      "double-long-unsigned-8": 0,
+      "obis-9": "1-1:3.7.0.255",
+      "double-long-unsigned-10": 0,
+      "obis-11": "1-1:4.7.0.255",
+      "double-long-unsigned-12": 0,
+      "obis-13": "1-1:31.7.0.255",
+      "double-long-unsigned-14": 0,
+      "obis-15": "1-1:51.7.0.255",
+      "double-long-unsigned-16": 0,
+      "obis-17": "1-1:71.7.0.255",
+      "double-long-unsigned-18": 0,
+      "obis-19": "1-1:32.7.0.255",
+      "long-unsigned-20": 0,
+      "obis-21": "1-1:52.7.0.255",
+      "long-unsigned-22": 0,
+      "obis-23": "1-1:72.7.0.255",
+      "long-unsigned-24": 0
+    }
+  }
+}
+```
+
+
+##### log output
+
+**obinsectd** will log some debug info to **stderr** by default, but
+can also publish the log messages to MQTT topics, using one of the
+loglevels **"debug"**, **"info"** or **"error"** as value for the
+topic.
+
+Log messages related to MQTT are never published, for obvious
+reasons...
 
-Note that including any of **"normal"**, **"full"** or
-**"rawhexdump"** will result in a publish to the topic for every
-received packed, which might not be what you wanted.
 
-Array mesting is not allowed.
 
- 
 ## reading multiple meters
 
-Run one process per meter. The configuration files may be shared if
-applicable.
+Run one **obinsectd** process per meter if you want to monitor more
+than one meter. The configuration files may be shared between the
+processes, as long as the same publishing rules and OBIS lists are
+wanted.