Categories
Batteries

Ingesting Mate3 JSON into node-red

Get data from your Mate3 data into your InfluxDB database.

This guide assumes that you have already set up a RPi4 with InfluxDB + “batrium” user, node-red, grafana, etc. and know how to use the Pi’s CLI.

Head into your Pi4 CLI, we need to create another Influx database for the Mate3 data we’re about to grab.

influx 

#type 'influx' you will notice a different CLI prompt. This is normal, you are now in the influx CLI.

create database mate

grant all on mate to batrium

show databases

show users

exit

influx -username 'batrium' -password '123456' -database mate

exit

Congratulations, you now have an empty Mate3 database for all of your tasty data!

Importing the Mate3 flow into node-red

Open Node-Red in your PC web browser.
IP address of your Pi4 into your URL bar, suffixed with :1880- ie, 10.7.0.47:1880
In the top-right menu icon, click on Manage palette.

  • Click HERE to download the Mate3 flow JSON

Import the Mate3 flow json by clicking the top right menu icon, then click on import.

Past in the Mate3 JSON file, import to “new flow”, and click “import”

Your page should look something like this.

How this flow works, and what you need to change.

From left to right;

  1. Blue inject node “5 sec” tells the flow to run every five seconds.
  2. Yellow http request node “Mate3” “GET”s data from your Mate3 JSON, and returns a parsed JSON object
  3. Orange function nodes grab individual lines of data from the Mate3 JSON
  4. Grey influxdb out nodes send those individual lines of data from node-red into your “mate” Influxdb database.

So the first step we need to fix is to tell the http request node where your JSON file with all of the Mate3 data is – ie, what the IP address of your Mate3 address is.

For what it’s worth, here is Outback Solar’s documentation for programming the Mate3.

For most people, there is no easy way to find this address. I personally use a Unifi network, so I can easily check the IP address of the Mate3 device (called AlphaTec) in my Unifi app.

You could perhaps log into your router and check a list of connected devices, find “AlphaTec“, and copy the IP address down.

Alternatively, you could use a piece of free-trial software called “Solarwinds IP Address Manager”.

However you do it, you NEED to get this IP address – in my case, it’s 10.0.0.66

Our Mate3 IP address is 10.0.0.66

Using Chrome, install JSON Viewer, or any other extension that auto-formats JSON files.

Take this IP address, append /Dev_status.cgi?Port=0 to the end of it, and paste it (http://<mateIPaddress>/Dev_status.cgi?Port=0%20A) into your web browser to confirm that it’s the correct address.

If all goes to plan, you’ll be greeted by something like this:

Therefore, our Mate3 JSON file is located at
http://<mateIPaddress>/Dev_status.cgi?Port=0%20A

If it didn’t work, then you may want to double check to ensure you got the correct IP address. If you’re SURE that it’s the correct IP address, then perhaps you have a firewall blocking traffic across your network and I can’t help you – may as well give up now.

Assuming it DID work, it’s VERY important that you remove the “&” in front of “Port=0”, if it’s present when using the address URL from here on in.

For whatever reason, it’s an error introduced to the URL that Chrome can remove automatically, but it’ll stop the URL from working once inside node-red.


Take the address of your Mate3’s JSON (in our case, it’s http://10.1.1.66/Dev_status.cgi?Port=0%20A) and enter it in the Mate3 node settings page, as per below.
Ensure that;

  1. Method = GET
  2. Payload = Ignore
  3. Return = a parsed JSON object
  4. Name = Mate3

Congrats! If everything went to plan, then your node-red flow is ingesting data from your Mate3!

And now is the hard part…


Telling node-red to grab the right JSON data.

Everyone’s setup is different, so here is where you need to get “coding”.
Below is the JSON data from my Mate3.

{
  "devstatus": {
    "Gateway_Type": "Mate3s",
    "Sys_Time": 1610996763,
    "Sys_Batt_V": 25.8,
    "ports": [
      {
        "Port": 1,
        "Dev": "FX",
        "Type": "230V",
        "Inv_I": 4,
        "Chg_I": 0,
        "Buy_I": 0,
        "Sell_I": 0,
        "VAC_in": 0,
        "VAC_out": 232,
        "Batt_V": 25.8,
        "AC_mode": "NO AC",
        "INV_mode": "Inverting",
        "Warn": [
          "Temp Sensor Fail"
        ],
        "Error": [
          "none"
        ],
        "AUX": "disabled"
      },
      {
        "Port": 2,
        "Dev": "FX",
        "Type": "230V",
        "Inv_I": 0,
        "Chg_I": 0,
        "Buy_I": 0,
        "Sell_I": 0,
        "VAC_in": 0,
        "VAC_out": 224,
        "Batt_V": 26.0,
        "AC_mode": "NO AC",
        "INV_mode": "Off",
        "Warn": [
          "Temp Sensor Fail"
        ],
        "Error": [
          "none"
        ],
        "AUX": "disabled"
      },
      {
        "Port": 3,
        "Dev": "FX",
        "Type": "230V",
        "Inv_I": 0,
        "Chg_I": 0,
        "Buy_I": 0,
        "Sell_I": 0,
        "VAC_in": 0,
        "VAC_out": 226,
        "Batt_V": 26.0,
        "AC_mode": "NO AC",
        "INV_mode": "Off",
        "Warn": [
          "Temp Sensor Fail",
          "Fan Failure"
        ],
        "Error": [
          "none"
        ],
        "AUX": "disabled"
      },
      {
        "Port": 9,
        "Dev": "CC",
        "Type": "FM60/80",
        "Out_I": 4.2,
        "In_I": 1,
        "Batt_V": 25.9,
        "In_V": 73.0,
        "Out_kWh": 8.7,
        "Out_AH": 330,
        "CC_mode": "Bulk  ",
        "Error": [
          "none"
        ],
        "Aux_mode": "Manual",
        "AUX": "disabled"
      },
      {
        "Port": 10,
        "Dev": "CC",
        "Type": "MX60",
        "Out_I": 2.0,
        "In_I": 0,
        "Batt_V": 25.9,
        "In_V": 68.8,
        "Out_kWh": 6.0,
        "CC_mode": "Bulk  ",
        "Error": [
          "none"
        ],
        "Aux_mode": "Disabled",
        "AUX": "disabled"
      }
    ]
  }
}

You can see that ports 1,2,3 are VFX3524 inverters, while my two charge controllers are on ports 9 and 10.

If we look at the “Function” tab of the orange Inv1 kW function node – which tells me how many kW my inverter is currently using – in node-red (pictured below), you’ll see some… code. Frankly, I don’t know what kind of code it is… heck, I can’t even remember how I wrote it – all I know is that it works. (but I’ll try to explain, anyway).

var inv1kw  = 0;

inv1kw = Number(msg.payload.devstatus.ports[0].VAC_out * msg.payload.devstatus.ports[0].Inv_I);
msg.payload =
{
    inv1kw: inv1kw,
}
return msg;

First we have to create/specify a variable which we want node-red to send to our influxDB “mate” database, let’s call it “inv1kw“, and assign it a starting value of 0.

Now that node-red understands that inv1kw is a variable that exists, we want it to calculate a value for it.

We tell it to assign inv1kw the value resulting from multiplying the VAC_out (volts) by the Inv_I (amps), which results in the wattage out.

Then there is some other code, I don’t really know what it does but it works, so #yolo. If you’re copying this code to use on other variables you want to report on, just make sure that you change the “inv1kw” name to something else relevant – ie, if you wanted to know what amps are being output by your MX60 solar charge controller, you might call that variable “mxamps“.


Understanding how to point to data in your JSON.

Below is a stripped-down example of the JSON from my Mate3 inverter – I’ve removed all valued except from “Port” and “Batt_V”, and will illustrate how to refer to each of these Batt_V values in your node-red flow.

If you can’t figure out how to write code to refer to other values after this explanation, then I recommend you go find some resources on the internet regarding “JSON dot notation”.

{
  "devstatus": {
    "Gateway_Type": "Mate3s",
    "Sys_Time": 1610996763,
    "Sys_Batt_V": 25.8,
    "ports": [
      {
        "Port": 1,
        "Batt_V": 25.8,
      },
      {
        "Port": 2,
        "Batt_V": 26.0,
      },
      {
        "Port": 3,
        "Batt_V": 26.0,
      },
      {
        "Port": 9,
        "Batt_V": 25.9,
      },
      {
        "Port": 10,
        "Batt_V": 25.9,
      }
    ]
  }
}

VERY basically, JSON works upwards from 0.

On my Mate3 I am using ports 1, 2, 3, 9, 10. But any JSON query you do will “see” them in order 0, 1, 2, 3, 4

For example,
msg.payload.devstatus.ports[0].Batt_V
will see the value “25.8“, as it’s looking at the “Batt_V” of the port 0nth (aka 1st) on the list.

msg.payload.devstatus.ports[1].Port
will return “2”

msg.payload.devstatus.ports[2].Batt_V
will return “26.0”

msg.payload.devstatus.ports[3].Port
will return “2”

msg.payload.devstatus.ports[4].Batt_V
will return “25.9”

msg.payload.devstatus.ports[3].Port
will return “10”

If you can’t intuitively work out how this code works from the node-red flow, my examples above, and playing around for an hour… then I give up.

Ok, fine, last example.

To create a new output value to be displayed in grafana;

1. Look at the Mate3’s JSON to find the value you want to display – for example, we’ll use Port 1, “VAC_out”: 234
2. Create a function node to grab that value from the JSON, call it “inv1VAC”
3. Edit the function to something like this:

var inv1VAC= 0;

inv1VAC= Number(msg.payload.devstatus.ports[0].VAC_out);
msg.payload =
{
    inv1VAC: inv1VAC,
}
return msg;

4. Edit the output influxdb out node to be called “inv1VAC”, make the measurement “inv1VAC” and hit “done”. The output node will grab the “inv1VAC” and send it to your database, easy peasy.


Great, now we can move onto setting up Grafana!

Categories
Batteries

Batrium + Mate3 + Grafana : Part 4

Big thanks to;

Daniel Römer
for pioneering this method of getting Batrium data to Grafana with the watchmonUDPlistener.

Bloats
for the original Batrium node-red flow.

Wolfgang Attwenger / Dirty Optics
for building 90% of this workflow, albeit with a few speeling mistakes and for a completely different platform.

sdmtr
for helping me hack everything together.

Second Life Storage Forum

Binary-Parser info

Final thoughts.

Jesus fuck.

Table of contents.
Part 1 : Hardware configuration
Part 2 : Installing software
Part 3 : Configuring Grafana
Part 4 : Credits and final thoughts
Categories
Batteries

Batrium + Mate3 + Grafana : Part 3

Nearly there, let’s set up Grafana!

Go to Grafana in your PC browser, using the Pi4’s IP address suffixed with “:3000”, for example:

http://<Pi4ipaddress>:3000

We now need to set up Grafana. In a nutshell, we need to do two things. First, add the influxdb data source. Secondly, import the Dashboard.

Skipped too far ahead?
Part 1 : Hardware configuration
Part 2 : Installing software
Part 3 : Configuring Grafana
Part 4 : Credits and final thoughts
Click on the cog, and then “Data Sources”
Add new data source (ignore the existing batrium datasource in this screenshot)
Search for “InfluxDB” in the search bar, and click on “select”
URL: http://127.0.0.1:8086
Database: batrium
User: batrium
Pass: 123456
Doing the same for Mate3? Simply create another database connection and put “mate” in the database field.

Now to install the dashboard

Click on Dashboards -> Manage
Click on the blue “Import” button.
Paste in the dashboard JSON data linked here, and hit the blue “Load” button.

And tharrr she blows!

Click here to go to Part 4: Credits and tidbits

Categories
Batteries

Batrium + Mate3 + Grafana : Part 2

Finally, the fun part.

In this guide, I will be using code blocks to show you what to enter into the command line interface (CLI), illustrated below.

sudo example-code-block -butts

# stuff prefixed with a hash is my comments

When a code block like this appears, it means that you have to enter each line individually into your CLI and hit enter.

I might add notes, just for you, which are prefaced with the hash symbol (#) – don’t put these into the CLI if you can help it.

Wanna skip ahead?
Part 1 : Hardware configuration
Part 2 : Installing software
Part 3 : Configuring Grafana
Part 4 : Credits and final thoughts

Changing your password, and enabling SSH

First we need to connect your Pi4 to your Wifi network, using the little Wifi icon on the taskbar. Select your network SSID, and enter your password.

Next we need to change your Pi4’s user password from the default, and turn on SSH so we can do the rest of the hard work from the comfort of your personal computer.

Once we get through these early stages of setting up the Pi4, you’ll be using PuTTY to access your Pi4 remotely using SSH, so you should probably install in on your PC.

Now, open the Pi4’s CLI (called terminal)

Remember, each line has to be entered separately and ENTER hit before going to the next line!

Change your password:

sudo raspi-config
#hit enter

passwd

#you will be prompted to enter your password. 
#For the purposes of this tutorial I'll use 123456 - but please, for the love of god, use something a little harder to crack.
# Do it, and hit enter. 
# You need to do this so that you're not hacked immediately once we enable SSH. 

Pi4 username and password is now Pi/1234.

Turn on SSH:

sudo systemctl enable ssh
sudo systemctl start ssh

Find out your Pi’s IP address

ifconfig
Loading Debian (Ubilinux) on the Edison - learn.sparkfun.com
There should be a block of text with “wlan0” or something similar to the left of it. You need the addr: number, in this illustration it’s 10.7.0.47.

If you’re using a wired connection, you want the “eth0” IP address. “eth” stands for ethernet. You should probably know that.

The reason it says eth0 is because in the computer language world, stuff counts from 0.

So if you have three bananas, you would count them Banana 0, Banana 1, Banana 2.

I promise this will become important later, when we’re dealing with JSON files.

It’s a very good idea to write this IP address down somewhere, as we’ll be referring to it often.

Head back to your computer, and install PuTTY.

Open PuTTY, and create a new connection by;

  1. Entering your Pi4’s IP address in the Host Name box (our example uses 10.7.0.47)
  2. Typing Pi (or whatever you want) in the Saved Sessions box.
  3. Hit “Save”, so you don’t have to type the IP address again in the future.
  4. Either double click the now saved Pi session, or make sure it’s selected and click on “Open”

to paste code into PuTTY’s CLI, copy the code from this page and right-click inside the CLI – the code should paste auto-magically!

You’ll now be presented with a CLI window, asking for your username.

Type in;

pi

The CLI will now ask you for your password, this is the password we set earlier, as an example we used 1234.

1234

then hit enter again (the password text won’t be displayed).

if everything does well, you should see something like this.

Updating everything on your before we go on.

On the now logged-in and authenticated PuTTY terminal, enter the following:

sudo apt-get update
sudo apt-get upgrade -y

If at any stage through this tutorial the CLI asks for confirmation to install/update, type Y and hit enter.

Installing Grafana

sudo apt-get install -y apt-transport-https

sudo apt-get install -y software-properties-common wget

wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

# that paragraph above is all a single line

sudo systemctl enable --now grafana-server

sudo /bin/systemctl daemon-reload 

sudo /bin/systemctl enable grafana-server

sudo /bin/systemctl start grafana-server 

systemctl status grafana-server.service 

sudo apt -y install ufw

sudo ufw enable

sudo ufw allow ssh

sudo ufw allow 3000/tcp

#enter admin and admin for default username, (Default will need to be reset after first login.)





Check to make sure Grafana is working

Open your PC’s web browser, and enter the IP address of your Pi4 into your URL bar, suffixed with :3000 – ie, 10.7.0.47:3000
You’ll be asked to choose a new password, for this example we’ll choose 123456.

Grafana username and password is now admin/123456.

Installing InfluxDB

get -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/os-release

echo "deb https://repos.influxdata.com/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

sudo apt-get update && sudo apt-get install influxdb

sudo service influxdb start

sudo systemctl enable --now influxdb 
#enable and run on system boot


sudo systemctl status influxdb 
#check status

#hit ctrl-c to close


sudo ufw allow 8086/tcp 
#This command will open the default port for Influxdb
Create InfluxDB Database for your Batrium data.
influx 

#type 'influx' you will notice a different CLI prompt. This is normal, you are now in the influx CLI.


create database batrium

create user batrium with password '123456'

grant all on batrium to batrium

show databases

show users

exit

influx -username 'batrium' -password '123456' -database batrium

exit

Congratulations, you now have an empty Batrium database!

InfluxDB username and password is now batrium/123456.

Installing node-red

get -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -source /etc/os-release

echo "deb https://repos.influxdata.com/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

sudo apt-get update && sudo apt-get install influxdb

sudo apt install nodejs

nodejs -v

sudo apt install npm

sudo npm install -g --unsafe-perm node-red node-red-admin

sudo ufw allow 1880

sudo ufw allow 18542/udp 
#Opening this port allows Node-Red to "listen" for the data from the Batrium

node-red 
#This starts node-red.


Check to make sure node-red is working

Open your PC’s web browser, and enter the IP address of your Pi4 into your URL bar, suffixed with :1880– ie, 10.7.0.47:1880

Tell node-red to boot on startup.
sudo nano /etc/systemd/system/node-red.service 
# this should open up a different graphical interface in your CLI, paste in the following
.

[Unit]
Description=Node-RED
After=syslog.target network.target

[Service]
ExecStart=/pi/local/bin/node-red
Restart=on-failure
KillSignal=SIGINT

# log output to syslog as 'node-red'
SyslogIdentifier=node-red
StandardOutput=syslog

# non-root user to run as
WorkingDirectory=/home/pi/
User=pi
Group=pi

[Install]
WantedBy=multi-user.target

#once you have pasted in the text above, hit:

ctrl-o
#this saves the file

ctrl-x
#this exits the file

sudo systemctl enable node-red 

sudo systemctl start node-red 

sudo systemctl stop node-red

#Reboot and test to ensure node-red starts up.
Installing binary-parser, required by node-red.
#Navigate to your Node-Red install folder. 

sudo -i

cd /home/pi/.node-red 
#substitute 'pi' for you current Linux user.

npm install binary-parser

#We now need to edit our Node-Red settings.js file to enable the binary-parser

sudo nano /home/pi/.node-re/settings.js 
#again substitute 'pi' for your current linux user.



#this will open up a blue text editor screen with a lot of information. You need to add some text after the following line: "functionGlobalContext:"

#This line can be found if you hit "page-down" on your keyboard around six or seven times. It's annoying.

binary_parser:require('binary-parser').Parser,

ctrl-o
ctrl-X

Y
enter

Importing the Batrium flow into node-red

Open Node-Red in your PC web browser.
IP address of your Pi4 into your URL bar, suffixed with :1880- ie, 10.7.0.47:1880
In the top-right menu icon, click on Manage palette.

You need to install two new palettes (plug-ings, basically)

  • node-red-contrib-influxdb
  • node-red-dashboard

Put these two palette names into the install tab search box, and click on “install“. Accept any prompts that come up.

  • Click HERE to copy the Batrium flow JSON

Import Batrium flow json by clicking the top right menu icon, then click on import.

Past the Batrium JSON file, import to “new flow”, and click “import”

Your node-red flow should look something like this.

Of note here is that this flow is set up to listen for the Batrium UDP data on port 18542. We opened up this UDP port during the Node-Red install. This flow is also set up for a string of 16 longmons. If you have more, you will need to edit some of the nodes to reflect this.

Feel free to hit the big red deploy button!

Now we need to Deploy the Node-Red flow and check the debug tab for any errors. If you see some “TypeErrors” then it is usually because the Binary-Parser was not installed correctly. Also, you can toggle the green msg.payload debug filters on and off to see the data being received in the debug window.

If all you’re interested in is getting data from a Batrium, click here to go to Part 3: Setting up Grafana


…or click this link to read how to set up the Mate3 JSON ingest.