Making an ESP8266 Web-Accessible

I’ve spent a few hours recently making an ESP8266 web-accessible. Here I’m documenting my progress for the benefit of others.


My project is to control some lamps around my house from my smartphone.  There are lots of ways of solving this, such as WiFi-enabled power sockets or having a slave with an email account to sit next to the switch.  My requirements boil down to these:

  • Reasonably cheap.  WiFi-enabled switches seem to start at around £25 a go.  £75 to control three lamps is too much.
  • Accessible from outside my home network.  It can’t depend on being on the same WiFi network.
  • Secure.  No-one else should be able to twiddle my lights.  This is not to be another IoT project where security is an afterthought (or non-thought).
  • It’s not a hard requirement, but I didn’t want to depend on a free messaging server.  Too many of them don’t worry too much about privacy or security.

The Setup

This diagram shows the overall architecture:


I’ve bought some radio-controlled power sockets off eBay.  They come with a remote control which uses a 433MHz radio to send a 24-bit control signal.  I’ve bought an ESP8266 and a 433MHz transmitter/receiver module pair.  The receiver is only useful for sniffing the codes sent by the remote.  When you first plug one of the sockets in, you have to hold down a button on the remote to ‘programme’ the switch to recognize that button on the remote.  It’d be possible to make the ESP8266 send a new code when you press a particular button on the website, to programme a socket, but I haven’t bothered so far.

I’m programming the ESP8266 using the Arduino ESP8266 platform available on GitHub here.

The ESP8266 connects to my home WiFi network and, through that, to my cloud virtual private server (VPS).  This is a virtual machine running on someone’s cloud (in my case WebSound).  It costs me £2.50 per month.  I actually already had this server available for a different project I maintain, so it hasn’t added anything to the cost.  I have a DNS registration through GoDaddy, so I just added a new host name to the DNS configuration so the same machine now has two separate names (ie. and

The communication from the ESP8266 to the VPS uses the MQTT protocol.  MQTT originally stood for MQ Telemetry Transport.  You can read about it online, but basically it’s a protocol for publish-subscribe communication.  Some clients subscribe to topics and others publish messages on those topics; the subscribers receive the messages published by the publishers.  In my case, I’ve opted for MQTT encrypted using TLS.  By requiring that clients connect using certificates that I’ve signed using my CA or master key, this solves both the authentication and encryption problems on the ESP8266 side.

Also running on the VPS is a web server.  The front end of this is an nginx instance, mainly because it does everything I need and I was already using it for the other project.  The webserver is running two “virtual servers” (it’s easy to get lost in the various types of “virtual” here).  This means that you get a different “virtual server” depending on which DNS name you use for the host – if someone looks up they get my web-enable ESP8266, if they look up they get my other project.  The nginx part handles this as well as encryption using TLS.

The back end of the webserver is using Flask, a Python micro-web-framework.  Flask uses an API called WSGI which is a generic interface to Python web services.  You can run a Flask application by just starting a Python interpreter and loading your web service module.  This is fine for debugging, but in production you want cool features like multi-threading and resilience to exceptions; you get these for free with a WSGI container, in my case gunicorn.  This loads your web service module and instantiates as many instances as it needs to handle the incoming load.

The web server uses the Paho library to connect to the MQTT server, again encrypted using TLS.

Web Server Setup

My VPS is running Ubuntu Server 14.04 (my VPS host spun up a new instance for me with this already installed).  This makes installing the various bits and pieces fairly straightforward:

$ sudo apt-get install nginx python3.4 python-virtualenv mosquitto git supervisor openssl
$ virtualenv -p /usr/bin/python3.4 ha
$ . ha/bin/activate
(ha) $ pip install Flask Flask-SQLAlchemy python-social-auth paho-mqtt gunicorn

Here’s a quick overview of what I’ve installed:

  • nginx – front end web server
  • python3.4, python-virtualenv – the Python interpreter and a tool called virtualenv for setting up isolated Python configurations.  This lets me have a different set of installed Python packages for each project.  In this case, I’ve created a new virtualenv called ‘ha’ and activated it.
  • mosquitto – MQTT server
  • git – source control system used for getting the Let’s Encrypt software
  • supervisor – a service monitoring framework
  • openssl – tools for creating certificates etc
  • Flask – back end WSGI framework
  • Flask-SQLAlchmey – a Python framework for accessing databases
  • python-social-auth – a Python framework for authenticating against social media services, such as Google.
  • paho-mqtt – a library for accessing MQTT servers from Python
  • gunicorn – a Python WSGI container


All the encryption involved uses the public key infrastructure (PKI).  Public-private key cryptography relies on a pair of keys.  Anything encrypted using one of the keys can only be decrypted by using the other key in the pair.  So you can publish your public key and keep your private key private.  Now you can encrypt someone with your private key and anyone with your public key can check that it was really you that encrypted it; this is a sort of digital signature.  And someone with your public key can use it to encrypt a message and send it to you; only you can read it.

This is extended to the concept of signed certificates.  A certificate is a public key that is signed with someone else’s private key.  That “someone else” is a certificate authority or CA.  The CA is saying that this public key really belongs to the person claiming to own it and, so long as no-one has managed to steal either of their private keys, it’s cryptographically verifiable.

Who you choose to be your CA depends on what you’re using the certificate for.  Web browsers have a list of “trusted” CAs and their public keys built in to them.  If you want to run an encrypted website, you really need to have a certificate that’s signed by one of these trusted CAs or your users will get horrible warnings when they try to access your site.  It used to be that getting a certificate signed by one of these trusted CAs was hideously expensive – tens of thousands of pounds.  These days, there is the Let’s Encrypt project.  They are a CA who give away certificates.  You can’t use them to prove that you are who you say you are, but you can use them to prove that you control the server the certificate is issued for.  This is enough to set up a trusted HTTPS web server.  The only people who are likely to have trouble with it are those running Windows XP and frankly they deserve everything they get.

On the MQTT side, I have created my own root certificate and use it to sign the certificates used by the ESP8266 and the web server.  These are never publicly visible and all I want to be certain of is that the certificates used to connect to mosquitto are ones that I have signed, not someone trying to connect without a certificate I issued.

Create your CA certificate/key pair like this:

$ openssl req -new -x509 -days 1095 -extensions v3_ca -keyout ca.key -out ca.crt

This creates a new key and certificate pair that we will use to sign other certificates.  A couple of things to note here:

  • Keep your key (ca.key) safe!  Anyone who gets hold of it can sign certificates and you won’t be able to tell you didn’t sign them.
  • Mark in your diary now a day just short of three years from now to renew your certificates.  That 1095 on the command line is three years and after that your CA certificate will expire.  It is not likely you will remember this until everything stops working – rather embarrassing.

Next, create a key and a certificate request for mosquitto:

$ openssl genrsa -out mosquitto.key 2048
$ openssl req -out mosquitto.csr -key mosquitto.key -new

And sign it with your CA key:

$ openssl x509 -req -in mosquitto.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out mosquitto.crt -days 1095

You now have a key/certificate pair, called mosquitto.key and mosquitto.crt.  Put them in /etc/mosquitto/certs.  Also, put ca.crt (NOT ca.key) in /etc/mosquitto/ca_certificates.

Repeat the process above to create a key pair for your ESP8266 (esp8266.key / esp8266.crt) and python framework (server.key / server.csr).

Nginx configuration

We start by getting a certificate from Let’s Encrypt.  Stop nginx:

$ sudo service stop nginx

Next, download the Let’s Encrypt software and request a certificate

$ git clone $ cd letsencrypt
$ sudo ./letsencrypt-auto certonly --standalone

I had to run this a few times before it worked, apparently just because the Let’s Encrypt servers are a bit overloaded. It will ask you some questions along the way, most importantly what the server you want the certificate for is called. You have to get this right, and it has to be the server where you are actually running these commands!

Once this completes successfully, you should have a directory called /etc/letsencrypt that contains the key and certificate that has been created.  In particular, you should have /etc/letsencrypt/live/ (substitute in the name of your site) which will contain:

  • cert.pem – your web server’s certificate
  • chain.pem – not sure what this is
  • fullchain.pem – combination of cert.pem and chain.pem
  • privkey.pem – your web server’s private key

Now create a file called /etc/nginx/sites-available/a.example.conf (the actual name of this file doesn’t matter, so long as it’s in the right directory) and put this in it:

# Declare a back-end server listening on port 8100 that we will
# hand requests on to
upstream ha {

# Declare our server
server {
    listen 443 ssl; # Listening on the default SSL port for SSL connections
    client_max_body_size 1000M; # You could reduce this...
    keepalive_timeout 15;

    # Configure it to use our shiny new certificate
    ssl_certificate /etc/letsencrypt/live/
    ssl_certificate_key /etc/letsencrypt/live/
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Only use newer protocols
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH'; # Only use reasonably secure ciphers
    ssl_prefer_server_ciphers on;

    location / {
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Protocol $scheme;
        proxy_pass http://ha; # This is the back end we declared above

    location /static/ {
        root /home/me/ha/static;
    location /robots.txt {
        root /home/me/ha/static;
    location /favicon.ico {
        root /home/me/ha/static;

This sets up the server so that it serves /robots.txt, /favicon.ico and anything under /static/ straight from the filesystem, and forwards everything else on to our Python framework which is (or will be) listening on port 8100.  We serve these files directly in nginx because it is a lot more efficient at doing so than Flask is.

Python configuration

Log in to the VPS and activate our Python virtualenv:

$ . ha/bin/activate
(ha) $

Now, create a basic web service in ~/ha/project/ha/ with this content:

from flask import Flask
application = Flask(__name__)
def index():
    return 'Hello, world!'

This just returns the string ‘Hello, world!’ whenever someone tries to access our server.  By the way, the ‘ha’ name that keeps coming up isn’t special, it’s just what I decided to call my project (originally it stood for ‘home automation’ though my ambitions have got less grand).

Next create ~/ha/project/ha/

command = '/home/tkcook/ha/bin/gunicorn'
pythonpath = '/home/tkcook/ha'
bind = ''
workers = 2
user = 'me'

This is just a Python file with variables to configure the WSGI container, gunicorn.  It will run two worker threads and listen on port 8100 (this has to match what you put in the nginx configuration).

Lastly, we’ll use a programme called ‘supervisor’ to run gunicorn as a service.  Create /etc/supervisor/conf.d/ha.conf:


command=/home/me/ha/bin/gunicorn -c -p ha.wsgi

Now you can start your Python framework service:

$ sudo supervisorctl start ha:*

Supervisor is actually quite sophisticated and we’re not really using much of its capabilities here.  It can set up groups of programmes that can be started and stopped together but we have just one group with one programme.

Finally, restart nginx:

$ sudo service nginx restart

and you should be able to open (or whatever your server is called) and it should respond with ‘Hello, world!’.  Phew!

Mosquitto configuration

Create /etc/mosquitto/conf.d/local.conf (actually the name doesn’t matter, so long as it’s in that directory and ends in ‘.conf’):

allow_anonymous false
password_file /etc/mosquitto/passwd

listener 8883
cafile /etc/mosquitto/ca_certificates/ca.crt
keyfile /etc/mosquitto/certs/mosquitto.key
certfile /etc/mosquitto/certs/mosquitto.crt
require_certificate true

Create the password database like this:

$ sudo mosquitto_passwd -c /etc/mosquitto/passwd esp8266_user
$ sudo mosquitto_passwd /etc/mosquitto/passwd python_user

Enter a suitable password when prompted.  Restart mosquitto:

$ sudo service mosquitto restart

You should now have an MQTT server listening on port 8883.  It will only accept clients that present a certificate signed by your CA.


I won’t present all my ESP8266 code here, just the bit to do with connecting to the MQTT server.

I’m using the Arduino platform support for ESP8266; if you’re using some other programming system for the ESP8266, you’ll have to figure out things for yourself.

Note that the ESP8266 is just powerful enough to do TLS encryption.  You’ll need to make sure the CPU is running at 160MHz, not 80MHz or else the hardware watchdog trips while connecting to the server and the thing reboots.  Even running at 160MHz I’ve had it reboot occasionally if the server takes too long to respond.

I’m using the pubsubclient Arduino library available here for the MQTT connection.  It has a port to ESP8266 and a reasonable ESP8266 example; the hard thing is that the example doesn’t cover using TLS.  Here’s a brief sketch of how it’s done:

#include <ESP8266WiFi.h>
#include <PubSubClient.h>
#include "certificates.h"

WiFiClientSecure espClient;
PubSubClient client(espClient);

void setup() {
    WiFi.begin("my_ssid", "my_password");
    while(WiFi.status() != WL_CONNECTED)
    espClient.setCertificate(certificates_esp8266_bin_crt, certificates_esp8266_bin_crt_len);
    espClient.setPrivateKey(certificates_esp8266_bin_key, certificates_esp8266_bin_key_len);
    client.setServer("", 8883);

void reconnect() {
    while(!client.connected()) {
        if(client.connect("ESP8266Client", "esp8266_user", "your_password_here")) {
            // Resubscribe to all your topics here so that they are
            // resubscribed each time you reconnect
        } else {

void loop() {
    if(!client.connected()) {
    // Your control logic here

Basically, to connect to MQTT using TLS, you just use WiFiClientSecure instead of WiFiClient and set the certificate and private keys before connecting.  The tricky bit really is generating the file certificates.h.  Openssl by default generates keys and certificates in a human-readable text format, but here we need them in binary.  Openssl can get us part of the way by converting a text file to a binary one; Ubuntu provides the xxd utility to get us the rest of the way by converting the binary file to a C array definition we can use directly in our code.  First convert the text files to binary ones, putting the result in a directory called certificates:

$ mkdir certificates
$ openssl x509 -in esp8266.crt -out certificates/esp8266.bin.crt -outform DER
$ openssl rsa -in esp8266.key -out certificates/esp8266.bin.key -outform DER
$ xxd -i certificates/esp8266.*
$ cat certificates/esp8266.* > certificates.h

Now if you look in certificates.h, you will find two C arrays called certificates_esp8266_bin_crt and certificates_esp8266_bin_key, and two variables with their lengths, which you can use directly when setting up the WiFiClientSecure instance.

There are quite a few things that can go wrong in this process and I can’t remember all of the dead ends I went down.  Here are a few:

  • All the certificates have to be signed by your CA!
  • Remember to set the ESP8266 to run at 160MHz.
  • You may also have to disable the software watchdog timout – use ESP.wdtDisable() to do this.
  • Error messages at both the mosquitto end and the ESP8266 end are pretty unhelpful.  Make sure all your certificates are set up right and that your username and password is correct; otherwise you just get generic TLS failure messages with no indication what is wrong.

Python MQTT Connection

Again I’m not going to paste all my code here, but here’s a sketch of how to connect to the MQTT server from Python:

import paho.mqtt.client as paho
client = paho.Client()
client.tls_set("ca.crt", certfile="server.crt", keyfile="server.key")
client.username_pw_set("python_user", password="your_password_here")
client.connect("", port=8883)

There are a couple of options for how to make the client loop; you can either set off a background thread as I have done above, or you can call .loop() regularly to process incoming messages.

Using MQTT

MQTT, as I said above, is a publish-subscribe messaging system.  So, in my example, I’ve assigned an ID to each ESP8266.  Each ESP8266 then subscribes to a topic whose name is this ID, in hexadecimal.  So if the device ID is 0xDEADBEEF1234, it subscribes to ‘/DEADBEEF1234’.  The webserver then publishes commands on this topic which the ESP8266 broadcasts over the 433MHz radio.

For more complex scenarios, more complex topics can be constructed.  For instance, I could have a topic called ‘/DEADBEEF1234/temperature’ where the ESP8266 could publish a measured temperature, ‘/DEADBEEF1234/command’ where the webserver could publish commands and so on.  You can subscribe to ‘/DEADBEEF/+’, which would match either ‘/DEADBEEF1234/temperature’ or ‘/DEADBEEF1234/command’, or you can subscribe to ‘/DEADBEEF/#’ which will match either of them as well as ‘/DEADBEEF1234/measurements/measurement1’ and so on.  That is, a ‘+’ matches any string but only one path level, while ‘#’ matches any string and any number of path levels.

At the moment, I’m using the value returned by ESP.getChipId() as the unique identifier.  This has a couple of issues:

  • It’s only 24 bits, so there are only ~16.9 million unique devices possible.  That’s not a small number, but I don’t think it’s impossible that more ESP8266 devices than that will be made.  Collisions would be bad – it would let two people turn each others lights on and off (for example).
  • It’s the bottom 24 bits of the MAC address.  This is a bit more serious.  Anyone who can connect to the WiFi network can see the devices MAC address, and once you know the MAC address you are about half way to being able to control it.  My database setup means that I record which users are associated with which devices and only those users can control those devices through the web interface.  But if someone got hold of a client key and certificate then they could use them to connect directly to the MQTT server and control any device for which they know the MAC address.  Or they could just flood the system with messages for random device IDs and see what havoc they could cause!  And where would they get the client key and certificate?  Well, they come burned into the flash of the ESP8266 device.  So once I start selling these, I’m also giving away the key to connect directly to the MQTT server, if someone is keen enough to download the flash from the ESP8266 and figure out which bits are the key, certificate, username, password and hostname for the MQTT server.  I haven’t figured out a good way around this yet!

Web Client Authentication

Authenticating web users is one of the hardest bits of this to get working, and one of the hardest bits to get right.  Once again I’ll sketch how I went about it here, but whether what I’ve done will be suitable for you will depend a lot on what you want to achieve.  In particular, I just wanted to require that a user have a Google account to for Google to authenticate them for me.  If you want to use other services, you’re on your own from here.

First you need to set up an application in your Google developer’s console.  Go to  Create a new project.  Open the API Manager screen and click on Credentials.  Create an OAuth Client ID.  You will have to fill in some details for the OAuth consent screen the user will be presented with when logging in.  Note the client ID and client secret.  The type should be set to ‘Web Application’.

You need an object type to store user details in.  I’m using a PostgreSQL database to store my data and SQLAlchemy to access it in Python.  That’s not very important, so long as you have some sort of User object.  Here’s the guts of mine:

from sqlalchemy import create_engine, Column, Integer, String, Boolean
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from ha.settings import DB_USER, DB_PASS
from flask.ext.login import UserMixin

# ha is the database instance name
engine = create_engine('postgresql+psycopg2://' + DB_USER + ':' + DB_PASS + '@localhost/ha')
db = declarative_base()
db_session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine))

class User(db, UserMixin):
    __tablename__ = 'users'
    id = Column(Integer, primary_key = True)
    email = Column(String)
    active = Column(Boolean, nullable = False, default = False)
    fname = Column(String)
    lname = Column(String)
    accesstoken = Column(String)
    username = Column(String(400))

    def is_authenticated(self):
        return True

    def is_active(self):

    def is_anonymous(self):
        return False

    def get_auth_token(self):
        return self.accesstoken

I’ve put this in, putting it in a module called ha.models.  Don’t forget to create the matching schema in your database.

Then in my

import ha.models, ha.settings

from flask import Flask, redirect, render_template, session, jsonify, g, url_for, abort
from flask.ext.login import LoginManager, current_user, login_user, logout_user, login_required

from social.apps.flask_app import routes
from social.apps.flask_app.routes import social_auth
from social.apps.flask_app.template_filters import backends
from social.apps.flask_app.default.models import init_social
from social.apps.flask_app.default import models

application = Flask(__name__)
init_social(application, ha.models.db_session)

login = LoginManager()
login.login_view = 'login'

def load_user(id):
    return ha.models.db_session.query(ha.models.User).get(int(id))

def load_user_from_token(token):
    return ha.models.db_session.query(User).filter(User.accesstoken == token).first()

def global_user():
    g.user = current_user

def commit_on_success(error=None):
    if error is None:


def inject_user():
        return {'user': g.user}
    except AttributeError:
        return {'user': None}


def login():
    return redirect(url_for("social.auth", backend="google-oauth2"))

def index():
    if g.user.is_anonymous:
        return redirect(url_for('login'))
    if not g.user.is_active:
        return redirect(url_for('nouser'))
    return render_template('index.html')

def logout():
        return redirect('/')

def nouser():
    return render_template('nouser.html')

There’s a lot to get your head around here.  Essentially, the python_social_auth package handles authentication for you.  Given the configuration at the beginning, you can then decorate any web endpoint with ‘@login_required’ and Flask will automatically redirect any non-logged-in user to the Google login service before letting them access that page.

When someone new accesses your service, they’ll get registered in your database.  You’ll then have to have some mechanism to check that they really are someone you want accessing your service and make them active.  At the moment I’m the only user of my service and I’ve manually updated my database to make my account active; if you’re aiming for the big time, you’ll need something better than this.

I think the only thing left that is very mysterious here is the ‘ha.settings’ module.  It is just a file called which contains:

SECRET_KEY='something nice and random here'


SOCIAL_AUTH_GOOGLE_OAUTH2_KEY='Your Google oauth2 key here'
SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET='Your Google oauth2 secret here'

DB_USER='Your database username here'
DB_PASS='Your database password here'


Randomness really does matter in your SECRET_KEY.  Try this to get some reasonable randomness:

$ head -1 /dev/urandom | base64

And, as a final trap for your players, don’t forget to exclude this file from source control before you push it all to github!


Putting all these together took me quite a few hours of puzzling through stuff.  I hope a description of a real-world setup, with encryption throughout, is helpful to people.  If there’s things that are not clear please comment below and I’ll try to clear it up.