The amount of resources and tutorials available for complex asynchronous programming in Rust and Tokio is abysmal. When I was developing this project, I was very frustrated with the lack of good examples and I promised myself that I would create my own tutorial. That’s what you are reading right now.

There’s going to be very little code in this first section; I like to understand what I’m dealing with and how I’m going to approach it before I write any code. If you skip the planning and just get straight to the blasting, you’re setting yourself up for a load of hurt, especially with Rust and Tokio.

The end of goal of this project is to have two things. First, we want a minimally spec compliant MQTT broker. Something that you can actually use in your day to day, but you should probably extend before doing so. Second, we want to have a better understanding of using Tokio and designing complex asynchronous systems. So even if you don’t care about MQTT at all, maybe you want to make an asynchronous HTTP server or an Discord bot, you can still apply the knowledge of Tokio that you gain from this tutorial.

Purpose

When I began this project, many months ago, there were no good resources on Tokio. I procrastinated on writing this blog series, but my intention was for it to be a guide on using Tokio and implementing a real project. A reference of sorts. When I began populating this blog in March, there still weren’t any resources. That is no longer the case. I recently discovered that Tokio added a much better tutorial in June.

This project/tutorial series was my way of learning Rust and Tokio, and I’ve gained a lot out of it. However, the mini-redis tutorial that I linked above seems to cover everything that I would. By the end of part 3, we’ll have a client-server architecture that is easily extensible to be a compliant MQTT broker, but I won’t continue the tutorial to create a full-blown MQTT server. If people are interested, I can continue the series, but I don’t see the need for it anymore.

The MQTT Protocol

MQTT (MQ Telemetry Transport) is surprisingly not too complex and very cool. We will be dealing with MQTT Version 3.1.1, solely because it makes our life much easier. I highly suggest you read through the documentation yourself, but I will be going over the basics of the protocol too.

The goal of this tutorial is to teach you about the basics of asynchronous programming with a real life example. So, we’re not going to spend very much (or any) time handling some of the things that are unrelated to this. Instead, we’re going to make use of any libraries that make our goal easier.

Extension

Whenever we cheat our way out of doing something, I’ll make a note of it in a box like this one, and if you want to turn this tutorial into something more fully fledged, you should try implementing that thing on your own. In fact, the first thing we’re going to cheat our way out of is the bit banging and lower level manipulation of the MQTT protocol. We’ll care about the flags and the fields transmitted in each packet, but it doesn’t matter to us how the packet goes from bytes to struct, we’ll use a handy library to do that for us.

So, what in the world is MQTT? It’s a very cool protocol used by things like IoT devices because of its low overhead. Tons of clients can connect to one centralized server and send messages back and forth between each other through a message queue. A client can subscribe to a topic and it will receive any messages that are published by other clients.

As an example, let’s say you want to create a smart lamp that will turn on and off whenever it is told to, maybe by an Arduino or by a motion sensor. The lamp might subscribe to the topic /home/lamp1/status and when the motion sensor desires, it can publish the message “OFF” to /home/lamp1/status. The lamp will recieve this message and turn itself off. These topics are represented as a hierarchy, sort of like a file’s location. This is because things are allowed to subscribe to wildcards: /home/lamp1/*. That wildcard will send everything under /home/lamp1 to whoever is subscribed to it.

You could probably imagine how you would implement something like this using HTTP, but there’s a ton of overhead in HTTP. I highly recommend you scroll through the MQTT documentation and get a feel for this yourself, but the average packet has ~3 bytes of bookkeeping. An HTTP header response is >200 bytes and usually tends to be reported in KB, not bytes.

The job of the broker, what we will be building, is to oversee this communication and connect clients together. There are 14 packet types that we will need to handle. Most of them are ACK responses, just acknowledging that we recieved what the client sent us. But the main packets we will need to respond to are the Connect, Subscribe, and Publish packets.

Reading the documentation

I urge you to read the documentation and understand these packets yourself, it’s a good skill, but let’s do one together. Let’s go over the Publish packet, starting in section 3.3.

So the packet has some flags that are shown in 3.3.1. We won’t worry about how these are read or represented, the library will do that for us, but we do need to know what they are. The next 3 sections detail what the flags all represent. The DUP flag seems pretty self explanatory, as does the RETAIN flag. These two signify whether a message is being sent again (due to not recieving an ACK) and whether the message should be sent to new subscribers, respectively. But what is this QoS? There’s a little table there but it’s not super helpful.

Indeed, the explanation for QoS is found later in the documentation: Section 4.3. The explanation in section 4.3 is much better than what we saw in the publish section, but the gist is that that the QoS determines how aggresively we confirm that the client has recieved our message. We’re not going to worry too much about this until later, but it is something we should keep in mind.

That was the fixed header. Every packet also has a variable header that contains some more information, and a payload. The variable header contains two values: the topic name and the packet identifier. The topic name is what we discussed earlier, something like /home/lamp1/status, and the packet identifier is just a unique number so that we can quickly compare packets and make sure we are handling the correct ones. The payload is just our message, in our previous example, something like “OFF”.

Afterwards the documentation describes how the client and server use the Publish packet. We, as the server, will recieve a publish message and send a similar publish message to all clients that are subscribed. I hope this little foray into a specific packet will help you read and understand the other packets. We will go over them lightly whenever we implement the handling of one.

Architecture

So now that we have some understanding of what we are doing, how will we do it? We’re going to be using Rust and Tokio, but what is our architecture going to look like?

For every single client, we will need some sort of communication channel, a way to communicate over the internet back to the client. It is tempting to create a central Broker struct and let it control every single one of these, but trust me, it gets messy. Instead the easiest and most intuitive way to handle something like this is to mimic the MQTT protocol in our code.

We’re going to create a Broker struct that will handle accepting clients and storing information about things like what they’re subscribed to. But rather than handle communication on the Broker, every time we get a new connection, we’re going to spawn a Client struct. The Broker will communicate to our internal representation of the Client by using a tokio::mpsc channel, sending it custom messages that it then converts to packets and sends to the client. The Client will talk to the centralized broker with another tokio::mpsc, sending it any packets that the internal client recieves.

We will have three layers to our architecture. The broker at the top will handle all the state, and it will be running as one single thread. Whenever we get a new connection, our broker will spawn another thread for that client. This client thread will act like a middleman. It’ll talk to the actual client and to the broker, without every actually touching the state. This architecture will be completely thread safe, because all the information that we want to be mutable is going to sit on one thread.

That’s all for now. In the next section, we’ll implement a basic event-loop that can connect to clients and reports all the messages that it recieves.