ESP and MongoDB, sitting in a tree.
Part 3 - Into the realm
Part 1 of this series looked at the hardware life support for one of my sick house plants. In part 2, I talked in far more depth than was even remotely interesting about the code that runs on said device. In this instalment, I have hopefully learned my lesson and will give you a brief review of the code that runs on the backend to create the web service and receive the JSON packets.
For the sake of complete transparency, I work at MongoDB. I love the company, and I love the database, I have literally drunk ALL the cool-aid. I’m not interested in engaging in any debates on why MongoDB, because it’s a foregone conclusion that I’ll win, convincingly. That being said, everything I’m doing here is done in the free tier of MongoDB Atlas, so you can take this idea and run with it at zero cost to you. You can of course take these ideas and try to deploy them on other technologies, but frankly, you’d be mad.
MongoDB for those who haven't encountered the technology yet, is first and foremost a database, but unlike a lot of databases that you may be familiar with, it’s not rows and columns and tables and schemas and things that look like REALLY complicated spreadsheets all joined together, it’s JSON documents. The same JSON documents we create on the device in the previous post, are the ones that get stored in the MongoDB without any translation or messy, brittle ORM layers. It’s the way working with data was supposed to be.
You can download and run MongoDB on your home computer / server / nas / cloud instance / Raspberry Pi, or whatever. But like a normal, sane, I-don’t-want-to-mess-around-installing-a-billion-different-things kind of person, you probably just want to use MongoDB Atlas, which, as I mentioned above has a free tier we’re going to stay in.
Atlas also has the added benefit of giving us access to MongoDB Realm which is both the Mobile version of MongoDB and a Serverless platform for running your own code on. Realm allows us to build services, and in this case, I used it to build a Web Service that receives data from the ESP32 via a POST request, and then slaps a timestamp on it and pushes it into MongoDB.
Step 1: Register for a free account on MongoDB Atlas
Go to cloud.mongodb.com and register. It’s not difficult.
Step 2: Create a cluster in the free tier (official tutorial here)
There is a big green button that says “Create Cluster”, click that. Choose which cloud you want to deploy into (AWS, Azure or GCP), and choose a region. Make sure to choose a region that has the little blue tag under it which says free tier available.
Choose the M0 Sandbox in the cluster tier, give it a name and hit the create cluster button at the bottom. In about 3 minutes you’ll have a MongoDB cluster that you can use for free, forever (although it does only have 512MB of storage, so let's not get too carried away!)
You will also need whitelist the IP that you are connecting from and create a user. Both of which should take less then 60 seconds if you follow those links.
If you aren't the kind of RTFM person, check out my good chum Mike Lynn as he explains it all in televisual format:
Right, so we’ve got a free cluster in MongoDB Atlas, lets go build a Web Service using Realm. At the top of the Atlas page, you’ll see a tab for Realm, lets click that
Once you have got to the Realm tab, click the button to create a new Realm App
The Data Source in this context is the name you gave the free cluster in the step above. Give the app a name. If you want to change the region that your Realm app will be deployed into, it’s in the advanced configuration drop-down, but it’s not necessary for an app like this, as we’re not particularly latency-sensitive.
I’m not planning on giving you a tour of Realm, or indeed going into the complete detail on how to create a webservice, others have done it much better than me. (cue Mike again…)
The short version of it is, create a 3rd party service using the HTTP Service module. Then create an incoming webhook. The code I use is at commented out at the bottom of the Arduino code on github that we were looking at in part 2, but to save you going back, here it is:
In Realm, you’ll see it here:
As with all my attempts at programming, it is very simple and fairly laughable. It is invoked (serverless, don’t you know 😉) when a POST is made to the webhooks endpoint. The function takes the payload of the POST request, and looks at the body I take the opportunity to add a timestamp to the body body['ts'] = new Date(); and then unceremoniously dump the whole lot into my Atlas collection using coll.insertOne(body);
If you move to the Settings page of the webhook function, you can see the URL near the middle of the page (the bottom of this screenshot above) which is what you put in the secrets.h file we discussed in part 2. This is the endpoint that the ESP32 will connect to.
If you scroll further down this settings page, you’ll find a couple more settings of note:
I’m using a POST, but if you wanted to, you could build this as a GET request, but I don’t think that makes much sense. Realm has some really, really sophisticated security controls, that control access at many levels. I’ve chosen to largely ignore these, and instead protect my endpoint from all you crazy internet people posting random JSON up my endpoints, with a secret token that is needed as part of the POST request.
Thats it.
I can’t begin to tell you how amazingly simple it is to build really complex apps with Realm, but in the next part, we’ll discuss some data visualisation.
A couple of notes on this before I sign off:
- I think I’m going to need to write a Part 5 too. There is something really, really, really amazing here inside Realm that I’ve not mentioned yet. The Web Services and functions can work in the other direction too. We can set up a trigger in Realm to watch our moisture levels, and call a serverless function to send a Twilio push notification, or call another API when they fall below a threshold. By ‘setting up’ I mean point and click. It's amazing.
- Some people may wonder about MongoDB’s suitability for this kind of time-series work, and to be fair I’m not even beginning to scratch the surface of its capabilities. The system I’ve built here is no better than what you’d see in a relational database, which is to say a very nieve, one row per reading data model where all the metadata around the sensor is stored over and over again in a very wasteful manner. MongoDB really shines for time-series when you do it properly using some of the well-established patterns for data modelling that have come from working with customers in Finance, Automotive Engineering, Video Games and many other verticals. MongoDB has some great blog posts on how to do this sort of thing properly.
