Building a (not so simple) expression language part II: Scope

(This is part II of a two part series of posts, you can find part I here)

One of the most powerful parts of the Mixpanel query language is the any  operator, which allows you to select events or profiles based on the value of any element in a list. The any  operator is just a bit more magical1 than the other operators in our query language, both in its power and in its implementation.

We’ve already written about building the Mixpanel expression language – the language we built inside of the Mixpanel data store to allow you to query and select data for reports. The model we built in the last post can do a lot of work, but parsing and interpreting the any  query takes the language to another level, both metaphorically and syntactically.

Like the basic expression language post, we’ll be using Python and JSON to talk about procedures and data, but won’t assume you’re a serious Pythonista. It will also be worth taking another look at the simple expression language post, since this post elaborates that model.

Continue reading “Building a (not so simple) expression language part II: Scope”

Straightening our Backbone: A lesson in event-driven UI development

Mixpanel’s web UI is built out of small pieces. Our Unix-inspired development philosophy favors the integration of lightweight, independent apps and components instead of the monolithic mega-app approach still common in web development. Explicit rather than implicit, direct rather than abstract, simple rather than magical: with these in-house programming ideals, it’s little surprise that we continue to build Single-Page Applications (SPAs) with Backbone.js, the no-nonsense progenitor of many heavier, more opinionated frameworks of recent years.

On an architectural level, the choice to use Backbone encourages classic Model-View designs in which control flow and communication between UI components is channeled through events, without the more opaque declarative abstraction layers of frameworks such as Angular. Backbone’s greatest strengths, however – its simplicity and flexibility – are a double-edged sword: without dictating One True Way to architect an application, the library leaves developers to find their own path. Common patterns and best practices, such as wiring up Views to listen for change events on their Models and re-render themselves, remain closer to suggestions than standard practices, and Backbone apps can descend into anarchy when they grow in scope without careful design decisions.

Continue reading “Straightening our Backbone: A lesson in event-driven UI development”

Diagnosing networking issues in the Linux Kernel

A few weeks ago we started noticing a dramatic change in the pattern of network traffic hitting our tracking API servers in Washington DC. From a fairly stable daily pattern, we started seeing spikes of 300-400 Mbps, but our rate of legitimate traffic (events and people updates) was unchanged.

Suddenly our network traffic started spiking like crazy.

Pinning down the source of this spurious traffic was a top priority, as some of these spikes were triggering our upstream routers into a DDos mitigation mode, where traffic was being throttled.

Continue reading “Diagnosing networking issues in the Linux Kernel”

Building data products that people will actually use

This was originally posted on High Scalability.

Building data products is not easy.

Many people are uncomfortable with numbers, and even more don’t really understand statistics. It’s very, very easy to overwhelm people with numbers, charts, and tables – and yet numbers are more important than ever. The trend toward running companies in a data-driven way is only growing…which means more programmers will be spending time building data products. These might be internal reporting tools (like the dashboards that your CEO will use to run the company) or, like Mixpanel, you might be building external-facing data analysis products for your customers.

Either way, the question is: how do you build usable interfaces to data that still give deep insights?

We’ve spent the last 6 years at Mixpanel working on this problem. In that time, we’ve come up with a few simple rules that apply to almost everyone:

  1. Help your users understand and trust the data they are looking at
  2. Strike the right balance between ease and power
  3. Support rapid iteration & quick feedback loops

Continue reading “Building data products that people will actually use”

Feb 2015 Mixpanel C++ meetup: Fun with Lambdas (Effective Modern C++ chapter 6)

We’ve been hosting a series of monthly meetups on C++ programming topics. The theme of the series is a chapter-by-chapter reading of Scott Meyers’ new book, “Effective Modern C++”.

The meetings so far have been

  1. December: Arthur O’Dwyer on “C++11’s New Pointer Types” (EMC++ chapter 4)
  2. January: Jon Kalb on “Rvalue References, Move Semantics, and Perfect Forwarding” (EMC++ chapter 5)
  3. February: Sumant Tambe on “Fun with Lambdas” (EMC++ chapter 6)

Next up, we’ll be continuing chapter 6 with a presentation on “Generic Lambdas from Scratch“. Come by the office and check it out!

Building a simple expression language

(This is part one of a two part series, you can find part II here)

The Mixpanel reporting API is built around a custom expression language that customers (and our main reporting application) can use to slice and dice their data. The expression language is a simple tool that allows you to ask powerful and complex questions and quickly get the answers you need.

The actual Mixpanel expression engine is part of a complex, heavily optimized C program, but the core principles are simple. I’d like to build a model of how the expression engine works, in part to illustrate how simple those core principles are, and in part to use for exploring how some of the optimizations work.

This post will use a lot of Python to express common ideas about data and programs. Familiarity with Python should not be required to enjoy and learn from the text, but familiarity with a programming language that has string-keyed hash tables, maps, or dictionaries, or familiarity with the JSON data model will help a lot.
Continue reading “Building a simple expression language”

Queuing and Batching on the Client and the Server

We recommend setting up work queues and batching messages to our customers as an approach for scaling upward server-side Mixpanel implementations, but we use the same approach under the hood in our Android client library to scale downward to fit the constraints–battery power and CPU–of a mobile phone.

The basic technique, where work to be done is discovered in one part of your application and then stored to be executed in another, is a simple but broadly useful; both for scaling up in your big server farm and scaling down for your customer’s smartphones.

Continue reading “Queuing and Batching on the Client and the Server”

Debugging MySQL performance at scale

On Monday we shipped distinct_id aliasing, a service that makes it possible for our customers to link multiple unique identifiers to the same person. It’s running smoothly now, but we ran into some interesting performance problems during development. I’ve been fairly liberal with my keywords; hopefully this will show up in Google if you encounter the same problem.

The operation we’re doing is conceptually simple: for each event we receive, we make a single MySQL SELECT query to see if the distinct_id is an alias for another ID. If it is, we replace it. This means we get the benefits of multiple IDs without having to change our sharding scheme or moving data between machines.

A single SELECT would not normally be a big deal – but we’re doing a lot more of them than most people. Combined, our customers have many millions of end users, and they send Mixpanel events whenever those users do stuff. We did a little back-of-the-envelope math and determined that we would have to handle at least 50,000 queries per second right out of the gate.
Continue reading “Debugging MySQL performance at scale”

How we handle deploys and failover without disrupting user experience

At Mixpanel, we believe giving our customers a smooth, seamless experience when they are analyzing data is critically important. When something happens on the backend, we want the user experience to be disrupted as little as possible. We’ve gone to great lengths to learn new ways for maintaining this level of quality, and today I want to share some of the techniques were employing.

During deploys runs Django behind nginx using FastCGI. Some time ago, our deploys consisted of updating the code on our application servers, then simply restarting the Django process. This would result in a few of our rubber chicken error pages when nginx failed to connect to the upstream Django app servers during the restart. I did some Googling and was unable to find any content solving this problem conclusively for us, so here’s what we ended up doing.
Continue reading “How we handle deploys and failover without disrupting user experience”

We went down, so we wrote a better pure python memcache client

Memcache is great. Here at Mixpanel, we use it in a lot of places, mostly to cache MySQL queries but also for other data stores. We also use kestrel, a queue server that speaks the memcache protocol.

Because we use eventlet, we need a pure python memcache client so that eventlet can patch the socket operations to be non-blocking. The de-facto standard for this is python-memcached, which we used until recently.
Continue reading “We went down, so we wrote a better pure python memcache client”