Denis Papathanasiou

I’ve been rewriting critical server components that were originally written in Python to use Go instead. Unlike Python, which is interpreted and uses a global lock because the interpreter is not thread-safe, Go has built-in support for concurrency, and is statically compiled.

One of the first things I tackled was implementing a state machine. The Python version was based on this article by David Mertz.

Mertz uses an object oriented approach, defining a class with both data and methods. His code, syntax aside, will be familiar to anyone who has worked with objects in C++, C#, and Java.

Go, however, does not provide a mechanism for coupling methods to specific data structures. Instead, Go allows you to associate methods with data structures, so that any method can be applied to any struct.

It’s a model which is closer to what Alan Kay meant when he defined the term object oriented in the first place.

With that in mind, here’s how I originally wrote the state machine class as a Go struct:

type Machine struct {
    Handlers   map[string]func(interface{}) (string, interface{})
    StartState string
    EndStates  map[string]bool
}

Just as in Mertz’s definition, Handlers is a map whose keys are name strings, and whose values are functions which accept a “cargo” value, and return a next state name string, along with the updated cargo value.

Go treats functions as first-class objects, so storing and passing them from state to state works exactly as it does in Python.

The only change I made was in the end state list: whereas Mertz used a list of strings, I used a map, since there’s no fast equivalent of detecting presence in a list (in Go, the only way to do it would be to iterate through the entire list of strings until or unless a match is found).

Since the handler function signature is a little unwieldy, I created a user-defined function type for it:

type Handler func(interface{}) (string, interface{})

type Machine struct {
    Handlers   map[string]Handler
    StartState string
    EndStates  map[string]bool
}

All that leaves is the definition of the methods associated with the Machine struct.

The first two provide a way to define which handler function is associated with which name string, and which name strings constitute end states:

func (machine *Machine) AddState(handlerName string, handlerFn Handler) {
    machine.Handlers[handlerName] = handlerFn
}

func (machine *Machine) AddEndState(endState string) {
    machine.EndStates[endState] = true
}

It’s worth noting that since EndStates is a map (or a list, in Mertz’s original), it’s possible to have more than one state terminate processing.

The final method is the one which executes the state machine, by applying the appropriate handler function to the cargo value, stopping when an end state has been reached.

Since the set of functions are stored as first-class objects in a map, looking them up based on their names and invoking them is trivial:

func (machine *Machine) Execute(cargo interface{}) {
    if handler, present := machine.Handlers[machine.StartState]; present {
        for {
            nextState, nextCargo := handler(cargo)
            _, finished := machine.EndStates[nextState]
            if finished {
                break
            } else {
                handler, present = machine.Handlers[nextState]
                cargo = nextCargo
            }
        }
    }
}

The only possible fly in the ointment is Go’s strong typing, since the type of the cargo value in the handler function signature needs to be specified.

By using the generic interface{} as the type, however, all the handler functions need to do is invoke a type assertion on the incoming cargo value, and they can handle any data (the test example uses a float for the cargo, but it can be any data type, even user-defined structs).

The complete state machine is available as a Go package.

Update: February 21, 2012

I’ve extended this concept into a framework for creating an intelligent email-based agent server, whereby email sent to designated inboxes get dynamic, or custom replies.

It’s the same logic used by the TeamWork.io web service and I’ve decided to open source it on github: https://github.com/dpapathanasiou/intelligent-smtp-responder

Paul Tyma, the creator of Mailinator, once wrote about its architecture. He said that after starting with sendmail, he found it necessary to write his own SMTP server from scratch.

While he never released the Java source code of his server, I wanted to see if I could re-create it using Python, since I also wanted to understand how state machines work in that language.

The Basic Server

To start, I needed some code that would listen on a specific port, and read and respond to clients.

Python’s SocketServer module makes this simple.

Here, in a few lines, is a multi-threaded TCP server that listens on port 8888 of the local machine and echoes back what a connected client sends to it:

#!/usr/bin/python

import SocketServer

cr_lf = "\r\n"

class SMTPRequestHandler (SocketServer.StreamRequestHandler):
    def handle (self):
        try:
            while 1:
                client_msg = self.rfile.readline()
                self.wfile.write(client_msg.rstrip()+cr_lf) # a simple echo
        except Exception, e:
            print e

# server hostname and port to listen on
server_config = ('localhost', 8888) 

if __name__ == '__main__':
    tcpserver = SocketServer.ThreadingTCPServer(server_config, SMTPRequestHandler) 
    tcpserver.serve_forever() 

Start it from a command line prompt (if the port number you choose is less than 1025, then you need to do this as root):

$ python server.py

And test it using telnet:

$ telnet localhost 8888
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
This is an echo
This is an echo
Ok, I get it
Ok, I get it
What next?
What next?

Handling SMTP

Now I needed to be able to understand and reply to SMTP requests. The protocol is fairly simple, with only a handful of commands.

Each command consists of four letters, which appear at the start of the stream sent by the client, and terminated with “\r\n”.

Tyma did not, however, implement the full list of SMTP commands, since RSET (Reset), VRFY (Verify), NOOP (No operation), and others are used by spammers to abuse or even take over a server, and are rarely required by legitimate email clients.

The server needs to be able to handle the basic interaction, so HELO (Hello) / EHLO (Extended Hello), MAIL (Mail from), RCPT TO (Recipient To), and DATA all need to be supported.

At first glance, it’s tempting to try to implement it like this:

class SMTPRequestHandler (SocketServer.StreamRequestHandler):
    def handle (self):
        try:
            data = {}
            while 1:
                client_msg = self.rfile.readline()
                if client_msg.startswith('MAIL FROM:'):
                    data['sender'] = get_email_address(client_msg)
                elif client_msg.startswith('RCPT TO:'):
                    data['recipient'] = = get_email_address(client_msg)

                ...

                elif client_msg.startswith('QUIT'):
                    break
        except Exception, e:
            print e

Where get_email_address() is defined as, for example, something like this:

def get_email_address (s):
    """Parse out the first email address found in the string and return it"""
    for token in s.split():
        if token.find('@') > -1:
            # token will be in the form:
            # 'FROM:' or 'TO:'
            # and with or without the <>
            for email_part in token.split(':'): 
                if email_part.find('@') > -1:
                    return email_part.strip('<>')

But this gets messy in a hurry. While some commands fit within the neat single-line /^CMND rest of data\r\n/ pattern, others do not.

RCPT, for example, can be repeated multiple times, and once DATA is seen, every subsequent line must be collected until the final /^\.$/ appears.

State Machines to the rescue

A state machine provides a much better way of handling SMTP requests. In his excellent article, David Mertz defines a state machine as:

a directed graph, consisting of a set of nodes and a corresponding set of transition functions. The machine “runs” by responding to a series of events. Each event is in the domain of the transition function belonging to the “current” node, where the function’s range is a subset of the nodes. The function returns the “next” (perhaps the same) node. At least one of these nodes must be an end-state. When an end-state is reached, the machine stops.

And that corresponds exactly to what happens when a client interacts with an SMTP server:

Brass Tacks

Creating a state machine in Python is simple, since Python allows you to pass functions as higher-order objects. The statemachine.py implementation in Mertz’s article was done in just a few lines of code.

To handle each SMTP node, I defined a series of functions, one for each server response or command.

Here are the function prototypes, where the cargo parameter is a tuple, containing both the stream from/to requests are read and responses written, and a dict of data collected from the request:

def greeting (cargo):
def helo (cargo):
def mail (cargo):
def rcpt (cargo):
def data (cargo):
def process (cargo):

The state machine is defined within the SMTPRequestHandler class like this:

class SMTPRequestHandler (SocketServer.StreamRequestHandler):
    def handle (self):
        try:
            m = StateMachine()
            m.add_state('greeting', greeting)
            m.add_state('helo', helo)
            m.add_state('mail', mail)
            m.add_state('rcpt', rcpt)
            m.add_state('data', data)
            m.add_state('process', process)
            m.add_state('done', None, end_state=1)
            m.set_start('greeting')

            m.run((self, {}))
        except Exception, e:
            print e

So that each function knows how to recognize its assigned command, I defined and compiled these regular expressions. These are created as globals, since it’s more efficient to initiate them once, and have each subsequent method call use the already-existing version.

import re
helo_pattern = re.compile('^HELO', re.IGNORECASE)
ehlo_pattern = re.compile('^EHLO', re.IGNORECASE)
mail_pattern = re.compile('^MAIL', re.IGNORECASE)
rcpt_pattern = re.compile('^RCPT', re.IGNORECASE)
data_pattern = re.compile('^DATA', re.IGNORECASE)
end_pattern = re.compile('^.$')

The greeting() function, which begins the interaction with the client, sends a simple message and passes control to the helo() function. It looks like this:

def greeting (cargo):
    stream = cargo[0]
    stream.wfile.write('220 localhost SMTP'+cr_lf)
    return ('helo', cargo)

Later in the sequence, the mail() function, which is the first node from which data is collected (in this case, the email address of the sender), is the first to save information in the cargo’s dict. It looks like this:

def mail (cargo):
    stream = cargo[0]
    client_msg = stream.rfile.readline()
    if mail_pattern.search(client_msg):
        sender = get_email_address(client_msg)
        if sender is None:
            stream.wfile.write(bad_request+cr_lf)
            return ('done', cargo)
        else:
            email_data = cargo[1]
            email_data['sender'] = sender
            return ('rcpt', (stream, email_data))
    else:
        stream.wfile.write(bad_request+cr_lf)
        return ('done', cargo)        

Here, if the request is not recognized or invalid, the client sees the bad_request message, and the connection is closed, since control passes to the done end-state.

I followed Tyma’s example and defined bad_request as “550 No such user” (which, as he notes, is ironic, since Mailinator accepts email sent to any user).

It also doesn’t conform to the protocol, since I’m supposed to give different error messages at different nodes, but since clients are always disconnected after any type of invalid request, it hardly matters what they see in that scenario.

If a client is well-behaved, the final method called is process() which decides what to do with the client’s email. The data dict will contain three parameters: ‘sender’ (the email address of the sender), ‘recipients’ (a list of email addresses), and ‘data’ (the contents which followed the DATA command ahead of the final ‘.’).

def process (cargo):
    email_data = cargo[1]
    # do something with the email_data dict here

    return ('done', cargo)

Basically, this is where the data can be saved to disk/db (so that it can be served by a web browser later, e.g.), MIME-parsed (to remove attachments, etc.), or just trashed (if you have reason to believe the sender is a spambot or zombie network, e.g.).

Tyma describes various measures for dealing with attacks from spambots and zombies which I haven’t implemented here, but would be relatively easy to add to both the data() and process() functions.

Obtaining the ip address of the client is done using the stream.client_address[0] attribute.