Monday, November 22, 2021

Make an ESP8266 WiFi Temperature Sensor & Python Flask Backend - Part 4

Last time, we combined our example sketches into a functioning temperature sensor and refactored the backend to receive temperature data. This time, let's get basic plotting working. To do that, we need to:

  • Save temperature data on the backend
  • Implement an API for getting that data
  • Write a client-side JavaScript app to plot the data with Chart.js

Here's our familiar diagram; we'll focus on the items marked with an orange star.

Let's start with saving temperature data... 

Save Temperature Data

Eventually you'd want to save temperature data to a database. To keep things simple, I'm simply going to log the data to a file in CSV (comma-separated) format. Each entry will include temperature (C), a timestamp, the units of measurement ("C"), and the sender's IP address.

Logging the data is simply a matter of appending the data to a local file each time it's posted to the API. To get the timestamp, we need to import datetime.

from datetime import datetime

We also need a format string for the timestamp. 

date_fmt = "%y-%m-%d %H:%M:%S" 

The timestamp will be in the form: YYYY-MM-DD hh:mm:ss

The IP address is available from Flask's request object via request.remote_addr. The code looks like this:

now = datetime.strftime(datetime.utcnow(), date_fmt)
with open("temp.log", "a") as f:
f.write("{ts},{ip},{tc:4.2f},C\n".format(
ts=now, ip=request.remote_addr, tc=tc))
f.close()

Once everything is running, our temp.log begins filling up with entries:

21-11-14 17:55:47,192.168.1.82,20.75,C
21-11-14 17:55:58,192.168.1.82,20.75,C
21-11-14 17:56:08,192.168.1.82,20.75,C

The next step is to get the backend API working to retrieve this data.

Backend GET API

We already have a route defined in our Flask app to POST data. We need another route to GET data. I've stubbed it out:

@app.route("/temp", methods=["GET"])
def temp_get():
return ""

Reading The File

First, we open the file. One of the principles of software engineering is Don't Repeat Yourself (DRY). We could add to our new route the following open statement.

with open("temp.log", "r") as f:

The problem is if we decide to change the log file, we have to do it in two places: here and in our temp_post() route. Instead, following DRY, make a single variable for the logfile name

log_file="temp.log"

and use that in both routes like this:

with open(log_file, "r") as f:


Now we read the lines out of the file into an array using readlines().

with open(log_file, "r") as f:
data = f.readlines()

Converting to JSON

Our API is supposed to return JSON data. We'll return an array of JSON objects that look like this:

{
"tempC": 20.75,
"unit": "C",
"ip": "192.168.1.82",
"timestamp": "21-11-14 17:55:58"
}

We can either construct the JSON by creating an array of formatted strings, or we can create a data structure and use jsonify(). The former is preferable, because we're not reinventing the wheel with one more opportunity to goof up the format.

We'll loop through the lines of data, converting each to a dict and adding it to an list. Then we'll convert the list of dicts JSON and return the text.

We need to import jsonify.

from flask import Flask, request, jsonify

With that, I came up with the following route function:

@app.route("/temp", methods=["GET"])
def temp_get():
result = []
with open(log_file, "r") as f:
data = f.readlines()
for d in data:
ts, ip, tc, unit = d.rstrip().split(",")
entry = {}
entry["tempC"] = float(tc)
entry["unit"] = unit
entry["timestamp"] = ts
entry["ip"] = ip
result.append(entry)
return jsonify(result)

Note that python reads in the newline (\n) for each line so we use rstrip() to remove it. 

When I visit the endpoint with the browser, I see properly formatted JSON data in the form of an array of objects with the expected keys and values:

All that's left is to write our client-side plotting application.

Plotting

Our Flask app not only implements a backend API but also will implement a Single Page Application. That simply means it will serve a single webpage which loads client-side JavaScript to perform the plotting. So let's start with the webpage.

index.html

Flask allows you to serve static pages for a given route, or to render a page template, server-side. We'll use the approach.

First of all, let's create a subdirectory for our static content called static.

We'll create a boilerplate index.html with an H1 header.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" />
<meta http-equiv="Pragma" content="no-cache" />
<meta http-equiv="Expires" content="0" />
<title>Temperature Plot</title>
<link rel="shortcut icon" href="{{ url_for('static', filename='favicon.ico') }}">
</head>
<body>

<h1>Temperature Plot</h1>

</body>
</html>

To return this page, instead of "hello world", for the "/" route in our Flask app, have to configure the static_dir for app when our app is instantiated.

app = Flask(__name__, static_folder="./static")

In our "/" route, we change the return statement, to call the send_static_file() method on app.

@app.route("/")
def index():
return app.send_static_file('index.html')

Testing with your browser shows it works!

plot.js

The only thing left is to make use of chart.js and JavaScript to get the data and plot it.

I created plot.js in the static directory and populated it with the above chart object.

So let's obtain the data from our API that we made earlier. I'm going to use jQuery to call the API asynchronously. If you're not familiar with the concept, basically the query runs in a thread and when it's done, it executes the code you specify.

For now we'll just print the data to the console. But before we get into the JavaScript we have to make two additions to our index.html. One loads jQuery, the other loads the plot.js file. I added the following to the head section of index.html.

<script type="text/javascript"  
src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js">
</script>
<script type="text/javascript" src="static/plot.js"></script>

Next, in plot.js, we add the code to perform an asynchronous get request against our API. The anonymous function simply logs the response data to the console.

const url = "http://localhost:5000/temp";

$.getJSON(url).done((response) => {
console.log(response);
});

When you visit the index page, you can view console output by pressing F12 to open developer tools. You can expand the data to see that, indeed, we're retrieving the JSON data.

Now that we have that working, let's plot it in Chart.js.

It took me awhile to sort it out, but Chart.js is looking for a line chart object that looks like this:

{
type: "line",
data: {
labels: [],
datasets: [{}],
},
options: {
scales: {
y: {
beginAtZero: true,
},
},
},
});

The data consists of a label for each data point, an array of data sets. Each data set consists of data points. We'll only have one data set, temperature from our single device.

First, import the necessary JChart scripts in our index.html, like so:

<head>
...
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
<script src="https://d3js.org/d3-color.v1.min.js"></script>
<script src="https://d3js.org/d3-interpolate.v1.min.js"></script>
<script src="https://d3js.org/d3-scale-chromatic.v1.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/chart.js@3.6.0/dist/chart.min.js"></script>
<script type="text/javascript" src="static/plot.js"></script>
<title>Temperature Plot</title>
</head>

To begin working on the plotting, we need to add a canvas with id "myChart" to our html body below the H1 tags.

<canvas id="myChart"></canvas>

In jQuery we grab the 2d context for the canvas like this.

const ctx = $("#myChart").get(0).getContext("2d");

Except, if you just put this at the top of the script it won't work because it executes before the document is fully loaded. So we wrap our code in a $().ready() handler to ensure it runs after the document has been loaded.

$().ready(() => {
const url = "http://localhost:5000/temp";
const ctx = $("#myChart").get(0).getContext("2d");
const chart = new Chart(ctx, {
type: "line",
data: {
labels: [],
datasets: [{}],
},
options: {
scales: {
y: {
beginAtZero: true,
},
},
},
});

$.getJSON(url).done((response) => {
console.log(response);
});
});

With that working, adding chart.update() within the getJSON response handler makes the chart appear. Next, I populated the Chart object's labels. To keep it simple, I just numbered them 1..n. 

Finally, I added each temperature entry to the Chart dataset. All together, the code looks like this:

$.getJSON(url).done((data) => {
chart.data.labels = Array.from({ length: data.length }, (_, i) => i);

var ds = chart.data.datasets[0].data;
ds = [];
data.forEach((entry) => {
ds.push(entry.tempC);
});
chart.update();
});

Rather than plotting all the data, I decided to only plot the most recent 50 entries. I define a variable N, set it to 50, and then use slice(-N) on the array.

$.getJSON(url).done((r) => {
var data = r.slice(-N);
var dataset = [];

chart.data.labels = Array.from({ length: data.length }, (_, i) => i);

data.forEach((entry) => {
dataset.push(entry.tempC);
});
chart.data.datasets[0].data = dataset;
chart.update();
});

Et voila, it works!

Done For Now

Yay! We've got a sensor that sends data to a backend API, the data is saved, then retrieved and plotted by our front end web application. We've worked with ESP8266 using Arduino, Python with Flask, JSON data formats, jQuery, JavaScript and Chart.js.

Hopefully you've found this journey helpful if you're new to software engineering or some of these technologies. Or, at least, hopefully it was fun to follow along. Now you kind of get a general idea of how I approach these sorts of projects.

So what's next?

My plans to refine the project include accommodating multiple sensors, multiple types of measurements (temp, humidity, soil moisture, etc.). I'd like to use an off-the-shelf plotting app like ThingsBoard. The ESP8266 configuration needs some improvements as well.

Stay tuned for further updates.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.