Mixpanel Engineering

In my previous post covering OpenVPN, I said that we needed to restrict access to most of our servers – they will only be accessible to each other, rather than open to the outside world.

How do we do this? iptables. You can add iptables rules that explicitly state the ip addresses that are allowed through the firewall, and then disallow everything else.

If our network was static – meaning we would never have to add more machines – then this would be really simple. All you’d need to do is update your iptables file once with the ip of every server you own, and you’re done. No worries.

In the real world, the network isn’t static. We’re adding new machines all the time, and if we don’t update iptables at the same time, the new machines won’t be able to communicate with the old ones. To solve this problem, I dynamically generate iptables files and deploy them with Fabric.

Note: all code mentioned in this post can be found on github here: http://github.com/ttrefren/firewall

Creating an iptables template

We use Django, so I can leverage the django templating engine. Here’s a sample iptables template:

*filter

#  Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT

#  Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allows all outbound traffic
#  You can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

# Only allow eth1 access from our internal network.
{% for server in servers %}
-A INPUT -i eth1 -s {{ server }} -j ACCEPT
{% endfor %}

# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

You can see that we’re passing a list of server ips to the template, and then creating one rule per server of the format

-A INPUT -i eth1 -s {{ server }} -j ACCEPT

This rule allows traffic from a specific ip address (contained in the {{ server }} template variable) on a specific interface (eth1, the Rackspace internal network). Any traffic that doesn’t match one of the server ip/interface combinations is then rejected. We don’t have to open specific ports for our different services – this allows connections on any port, as long as the ip is valid.

Note that this will be modified slightly for servers that need to be externally accessible, such as your web server. This is easy to do though, you just have to open one or two ports:

# Append this to the ip-level filtering
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

Generating the rules

Once you have your template set up, it’s really easy to generate the files you need – all it takes is a list of ips.

from django.template import Context, Template

def generate_iptables(filename, ips):
    """
        Generate iptables files from templates.
    """
    f = open('templates/%s' % filename)
    f = ''.join([line for line in f])
    t = Template(f)
    c = Context({ 'servers': ips })
    rendered = t.render(c)

    out = open('rendered/%s' % filename, 'w')
    out.write(rendered)

If you checked out the git repo, an extended version of this function is located in iptables.py. It accepts a text file with ip addresses, one per line.

To be honest, this is the easy part. Any competent programmer can figure it out. The more difficult part – maintaining the list of ip addresses – is project-dependent. The simplest thing to do is maintain a list of all the ip’s you own, adding and removing ips as needed, but that’s pretty fragile.

Personally, I wrote a parser for our settings.py file (where we define all our server groups) which makes it so you only have to add new ips in once place. You probably have a different setup. The important thing is to figure out a good way to manage this process, because it’s critical to this firewall scheme working out.

Updating the firewall

Once you have iptables files being generated based on a dynamic list of servers, you are ready to set up your deployment system. I use fabric, and it’s dead simple.

@roles('all')
def ship_iptables():
    put('iptables.up.rules', '~/')
    sudo('mv /etc/iptables.up.rules /etc/iptables.up.rules.bak.%s' % int(time.time()))
    sudo('mv ~/iptables.up.rules /etc/iptables.up.rules')
    sudo('iptables-restore < /etc/iptables.up.rules')

This fab script ships the iptables file, backs up the previous one, and updates the current ruleset on every server. Easy.