Mixpanel Engineering

Real-time scaling

Secure your codebase: OpenVPN in the (Rackspace) Cloud

with 11 comments

If you're interested in what we work on, please apply - we're hiring: http://mixpanel.com/jobs/

Imagine this scenario: your company is growing rapidly and you’re hiring tons of engineers.  The passwords to many of your servers are stored in plaintext in configuration files — everyone has access to them.  Your main database – the one with all the user data – is available to anyone who has that file.

You fire an engineer.  Now what?  He knows the passwords to everything.  Do you trust him? What if you’re firing him because he’s just a bad egg? What can you do?

Well, you could change the passwords on every single server… but that’s a huge pain in the ass for everyone involved.  Luckily, there’s a solution for this problem, one that comes with quite a few other benefits as well: you set up a VPN.

What is OpenVPN?

OpenVPN is open-source software that allows you to set up a Virtual Private Network.  Basically, it allows you to treat remote servers as if they were on your internal network. This gives you fine-grained control over everyone who has access to your systems.

Each computer that has access to the VPN is a “client”, and they each have their own certificate. If you want to remove someone’s access to your network, you just revoke their certificate.  It only takes a few seconds, and boom – problem solved.  The engineer you fired can’t access the MySQL server even if he knows the password because it’s only accessible via the VPN.

The Situation

At Mixpanel we use the Rackspace Cloud for hosting, so this guide (as you may have guessed from the title) will be written from that perspective.

Servers hosted at Rackspace come with two interfaces by default: the Public interface (eth0), which is accessible from the internet, and the Private interface (eth1, also known as Rackspace ServiceNet) that is used from intercommunication between your own servers.  Bandwidth on the private interface is unmetered (read: free) and actually uses a whole different ethernet port than the public interface.

The Plan

Our first goal in setting up the server is to secure the codebase.  To do this we need to do two things:

  1. Set up the OpenVPN server and generate clients
  2. Update the firewall on your other servers to block external connections and only allow connections via OpenVPN or the local network

After accomplishing (2) our infrastructure should resemble this diagram:

Basically, your whole setup will be walled off from the internet except for the servers that really need to be able to communicate with the outside world, namely the vpn server and your actual web servers.  All communication between servers should happen over the private network, since it’s both cheaper and more secure.

Setting up the server

Helpful links:

The OpenVPN HOWTO is quite comprehensive and useful. It should be the first thing you read:
http://openvpn.net/index.php/open-source/documentation/howto.html

If you get stuck, this wiki is helpful – particularly these pages:

Setup

1. Install OpenVPN:

sudo aptitude install openvpn

This will put a bunch of files in /usr/share/doc/openvpn/, but you should make a copy so that your modifications don’t get lost in a software update (see next step)

2. Copy the examples folder to /etc/openvpn/:

cd /usr/share/doc/openvpn/examples && sudo cp * /etc/openvpn/

3. Follow the HOWTO.  It will walk you through the rest of the steps, which I will go over briefly.

First we need to build the key signing infrastructure and the initial server keys. (Tip: do this as root, otherwise too many sudo)

cd /etc/openvpn/easy-rsa/2.0
vim vars    # edit vars at the bottom to say whatever
. ./vars    # Source those vars you just edited
./clean-all # Remove any existing keys (none if this is fresh, but do it anyway)
./build-ca    # Build the certificate authority key/cert
./build-key-server server    # Build server.key and server.crt
./build-dh                    # Still not sure what this is, Diffie Hellman parameters. Apparently required.

Then we can build client keys – basically one for each computer that will connect to the VPN is a client.  I plan on using employee emails as client names, but you can use any unique naming scheme.

Just run the following command once for each client, replacing clientname with the name of your choice :)

./build-key clientname     # This will save files in /etc/openvpn/easy-rsa/2.0/keys

Your server needs to retain clientname.crt, and the client needs both clientname.key and clientname.crt to be able to connect.

Before we go any further, let’s set up the server.conf – create a server.conf file at /etc/openvpn/server.conf and paste in the following code:

port 1194
proto udp

dev tun

# Keys we generated earlier
ca      /etc/openvpn/easy-rsa/2.0/keys/ca.crt
cert    /etc/openvpn/easy-rsa/2.0/keys/server.crt
key     /etc/openvpn/easy-rsa/2.0/keys/server.key  # This file should be kept secret
dh      /etc/openvpn/easy-rsa/2.0/keys/dh1024.pem

# This will be the internal tun0 connection IP - choose whatever you want
server 10.37.73.0 255.255.255.0
ifconfig-pool-persist ipp.txt

# This will send all of a client's 10.x.x.x traffic through the VPN
push "route 10.0.0.0 255.0.0.0"
keepalive 10 120

# Compression - MUST be turned on at both ends. Should be an option on client side as well
comp-lzo
persist-key
persist-tun

# Prevent revoked certificates from accessing vpn
crl-verify easy-rsa/2.0/keys/crl.pem
status log/openvpn-status.log

# Verbose, good for testing.  Switch to 3 in production.
verb 6

Next, try running the server – sudo openvpn server.conf.  It should run, and you should be able to connect from a client, but it won’t forward traffic across the network yet.

To do that, you need to set up IP forwarding – this is required to route traffic to other nodes:

# temporary solution
echo 1 > /proc/sys/net/ipv4/ip_forward

# permanent solution
vim /etc/sysctl.conf
# uncomment net.ipv4.ip_forward = 1

The final step is to set up your iptables rules for the vpn server.  The most critical rule (and a curiously underdocumented one) applies to the NAT table – it will rewrite the packets that come through the vpn with the vpn’s ip address before passing them along.  If you don’t do this, the origin address on the packets is 10.37.73.XX, which the other servers on your network don’t know how to handle.

The important one is:

# Masquerade local subnet - run this on your VPN server.
iptables -t nat -A POSTROUTING -s $PRIVATE -o eth1 -j MASQUERADE

Here’s a sample iptables ruleset for your VPN server:

* filter
# Set default policies
-P OUTPUT ACCEPT
-P INPUT DROP
-P FORWARD DROP

# Prevent external packets from using loopback addr
-A INPUT -i eth0 -s 127.0.0.1 -j DROP
-A FORWARD -i eth0 -s 127.0.0.1 -j DROP
-A INPUT -i eth0 -d 127.0.0.1 -j DROP
-A FORWARD -i eth0 -d 127.0.0.1 -j DROP

# Check source address validity on packets going out to internet
-A FORWARD -s ! 10.0.0.0/8 -i eth1 -j DROP

# Allow local loopback
-A INPUT -s 127.0.0.1 -j ACCEPT
-A INPUT -d 127.0.0.1 -j ACCEPT

# Allow incoming pings (can be disabled)
-A INPUT -p icmp --icmp-type echo-request -j ACCEPT

# Allow services such as www and ssh (can be disabled)
-A INPUT -p tcp --dport ssh -j ACCEPT

# Allow incoming OpenVPN packets
-A INPUT -p udp --dport 1194 -j ACCEPT
-A INPUT -i tun+ -j ACCEPT
-A FORWARD -i tun+ -j ACCEPT

# Allow packets from private subnets
-A INPUT -i eth1 -j ACCEPT
-A FORWARD -i eth1 -j ACCEPT

# Keep state of connections from local machine and private subnets
-A OUTPUT -m state --state NEW -o eth0 -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -m state --state NEW -o eth0 -j ACCEPT
-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

* nat

# Masquerade local subnet
-t nat -A POSTROUTING -s 10.0.0.0/8 -o eth1 -j MASQUERADE

COMMIT

If you save this file as /etc/iptables.up.rules on the vpn server, and call sudo iptables-restore < /etc/iptables.up.rules then you should have a fully functional vpn server.  If you connect to it, you should be able to ping your other servers by their internal ip addresses from your local computer.

At this point we don’t have the rest of the network blocked off, but that’s the next step. Broadly, you will want to update the iptables firewall on all of your servers to restrict to a set of IP addresses that you own.  The rules will look something like this:
# Only allow eth1 from these IPs.
-A INPUT -i eth1 -s 10.177.160.190 -j ACCEPT
-A INPUT -i eth1 -s 10.176.143.1 -j ACCEPT
# ... continue, 1 line per IP

# Reject all other ips
-A INPUT -i eth1 -j REJECT

I’ll go into more detail on this part of the process in my next post. For now, you should enjoy your fully functioning OpenVPN server!

If you're interested in what we work on, please apply - we're hiring: http://mixpanel.com/jobs/

Written by Tim Trefren

September 8th, 2010 at 8:12 am

Posted in Operations

11 Responses to 'Secure your codebase: OpenVPN in the (Rackspace) Cloud'

Subscribe to comments with RSS or TrackBack to 'Secure your codebase: OpenVPN in the (Rackspace) Cloud'.

  1. test

    Tim Trefren

    8 Sep 10 at 8:19 am

  2. [...] Secure your codebase: OpenVPN in the (Rackspace) Cloud Good to know. Will need to play around with OpenVPN one of these days. [...]

  3. [...] my previous post covering OpenVPN, I said that we needed to restrict access to most of our servers – they will only be [...]

  4. [...] Secure your codebase: OpenVPN in the (Rackspace) Cloud at Mixpanel Engineering – Share/Save var a2a_config = a2a_config || {}; a2a_config.linkname="Links for November 1st"; a2a_config.linkurl="http://crankycoder.com/2010/11/01/links-for-november-1st/"; Categories: Uncategorized Tags: api, data, links, openvpn, rackspace, semanticweb, sparql, vpn, wikipedia Comments (0) Trackbacks (0) Leave a comment Trackback [...]

    404 Not Found

    1 Nov 10 at 9:02 pm

  5. [...] slices privately via OpenVPN Diagram: OpenVPN to Rackspace Cloud Servers and Slicehost A recent blog post from Mixpanel inspired me to write a quick how-to for Fedora users on using OpenVPN to talk to instances [...]

  6. [...] recent blog post from Mixpanel inspired me to write a quick how-to for Fedora users on using OpenVPN to talk to instances [...]

  7. Easily the post is actually the best on this deserving topic. I fit in with your conclusions and will eagerly look forward to your upcoming updates. Just saying thanks will not just be sufficient for the extraordinary lucidity in your writing.

  8. Publikacja elektroniczna jest od czasu do czasu ujmowana szerzej, skoro obejmuje materialy elektroniczne niebedace ksiazkami, niczym chociazby systemy pomocy. z trudem zrealizowac precyzyjna klasyfikacje oraz w gruncie rzeczy wolno przyklaskiwac rozmaite zakresy definicji publikacji oraz ksiazek elektronicznych. jest dozwolone jakkolwiek przychylic sie, iz ta ostatnia jest przeniesieniem klasycznej ksiazki czyli czasopisma az do swiata urzadzen komputerowych, co wyraza sie chocby w nazwie.

    ksiazki

    18 Jan 11 at 4:36 pm

  9. Hi,

    My name is Ananjan Chaudhuri and I work for Packt Publishing, a U.K. based publishing firm specializing in focused IT books. You can read more about us here: http://www.packtpub.com/

    I came across your blog through google search and noticed good amount of information on OpenVPN.

    I would like to inform you that Packt has recently published a new book OpenVPN 2 Cookbook, a new book which covers everything a system administrator needs to manage and run an OpenVPN network, from point to point networks to troubleshooting. Written by Jan Just Keijser, this book offers all the information you need to successfully manage your network. You may read more about the book here:- https://www.packtpub.com/openvpn-2-cookbook/book

    Keeping in mind your knowledge in this subject and having looked at your contributions, I feel you’d make an excellent reviewer of this book. In case you’d like to write a review on your blog/website and/or Amazon, simply let me know your shipping details (Required for the registration of new reviewers) and I’ll have the eBook added to your account and would provide you with the download instructions in my next e-mail. You can then download this book instantly.

    If you have any queries, do let me know and I’d be happy to assist.

    I look forward to hearing your thoughts on this.

    Have a good day!

    Ananjan.

    Ananjan Chaudhuri

    16 Mar 11 at 6:01 am

  10. I’m getting the following error with CentOS 5.6 running OpenVPN 2.2.2:

    [root@server 2.0]# iptables-restore < /etc/iptables.up.rules
    iptables-restore v1.3.5: Line 53 seems to have a -t table option.

    Error occurred at line: 53

    Any help is appreciated.

    Charles

    14 May 12 at 8:59 am

  11. OpenVPN is very handy when you cannot use common VPN via PPTP (due to firewalls for example). Thanks for a detailed manual!

    Abe Sohm

    3 Jul 12 at 2:50 am

Leave a Reply

Safe place to purchase clomid and buy weight loss pills