Mixpanel Engineering

Real-time scaling

Why We Moved Off The Cloud

with 58 comments

If you're interested in what we work on, please apply - we're hiring: http://mixpanel.com/jobs/

This post is a follow up to We’re moving. Goodbye Rackspace.

Cloud computing is often positioned as a solution to scalability problems. In fact, it seems like almost every day I read a blog post about a company moving infrastructure to the cloud. At Mixpanel, we did the opposite. I’m writing this post to explain why and maybe even encourage some other startups to consider the alternative.

First though, I wanted to write a short bit about the advantages of cloud servers since they are ideal for some use cases.

  • Low initial costs. Specifically, you can get a cloud server for less than $20. Even the cheapest dedicated servers (and I wouldn’t recommend the cheapest) will cost more than $50. For new companies, this can make a difference.
  • Fast deployment times and hourly billing.If you have variable traffic and you’re not having problems scaling your data persistence layer, you can fairly easy spin up and spin down servers quickly in response to usage patterns. It’s worth pointing out that I specifically mean variable traffic rather than growing traffic. From purely an ease of deployment standpoint, handling even quickly growing traffic is fairly easy on both cloud and dedicated platforms.
  • Cheap CPU performance.If your application is purely CPU bound, then you can end up with great price/performance ratios. Most cloud servers allow a single small node on a physical server to use more than its fair share of CPU resources if they are otherwise underutilized — and they often are. One of the last bits of our infrastructure still on the cloud is CPU bound and even though we pay for very small Rackspace cloud servers, we get the performance of dedicated hardware.

The cloud’s intractable problem

… is variable — no, highly variable — performance. We’ve spent a lot of effort designing our infrastructure to scale horizontally so poor performance is not much of a problem, it just means buying more machines. However, highly variable performance is incredibly hard to code or design around (think a server that normally does 300 queries per second with low I/O wait suddenly dropping to 50 queries second at 100% disk utilization for literally hours). It’s solvable, certainly, but with lots of time and money and it’s hard to justify the cost when there’s a better alternative available.

The fundamental problem with cloud servers is that you’re at the mercy of your neighbors. If they decide to “dd if=/dev/zero of=/dev/sda”, there’s not a whole lot you can do about it other than migrating to a different physical server (and it’s really hard to decide whether to wait it out or migrate, especially because zero down time migrations are always little painful). Even worse, at Mixpanel’s level of disk usage that migration can easily take more than a day. In other words, you better hope that your neighbor never runs that command or anything that even looks like it from a disk utilization perspective. Side note: based on observations over a few months, I’m pretty sure that Rackspace actually does the equivalent of a full virtual disk wipe every time a customer deletes a cloud server. Better hope that none of your neighbors ever decides to cancel their server!

To be clear, variable performance forced us off the cloud, but I thought I would point out a couple other cloud disadvantages too:

  • One size fits all. The cloud, even AWS, offers very little customization compared to dedicated hardware. We recently added a new backup machine with a crappy CPU, little RAM, and 24 2TB drives in a hardware RAID 6 configuration. You can’t get that from a cloud provider and if you find something similar it’s going to cost an order of magnitude more than what we’re paying.
  • No access to bleeding edge hardware. At Mixpanel, some of our codebase is highly optimized low level C. We’ve profiled, tweaked, and made sure we’re not missing anything obvious. My point is the only way this code is going to run in less time is if we get faster hardware. Dedicated hosting providers usually stay on top of new hardware (specifically, the latest CPU’s and SSD’s). On the cloud, you’re usually stuck with whatever the provider got a volume discount on.

Going dedicated

After getting fed up with variable cloud performance, I decided to make the move to dedicated hardware. This isn’t a decision to take lightly. It literally took months to move the most important parts of our infrastructure. Starting migrations at 8 p.m. on a Friday and waking up early Saturday morning to finish them off isn’t so much fun either.

After deciding to go dedicated, the next step is choosing a provider. We got competing quotes from a number of companies. One thing that I was surprised by — and this really doesn’t seem to be the case with the cloud — is that pricing is highly variable and you have to be prepared to negotiate everything. The difference between ordering at face value and either getting a competing quote or simply negotiating down can be as much at 50-75% off. As an engineer, this type of sales process is tiring, but once you have a good feel for what you should be paying and what kind of discount you can reasonably get, the negotiations are pretty quick and painless.

We ultimately decided to go with Softlayer for a number of reasons:

  • No contracts. I don’t think I really need to explain the advantage. You would think that you could get better prices by signing 1 or 2 year contracts, but interestingly enough, out of the initial 5 providers we talked to the two that didn’t require contracts had the best prices.
  • Wide selection. Softlayer seems to keep machines around for a while and you can get very good deals on last year’s hardware. Most of the other providers we contacted would only provision brand new hardware and you pay a premium.
  • Fast deployment. Softlayer isn’t quite at the cloud level for deployment times, but we usually get machines within 2-8 hours or so. That’s good enough for our purposes. On the other hand, a lot other hosting companies have deployment times measured in days or worse.

One last thing about getting dedicated hardware. It’s cheaper… a lot cheaper. We have machines that give us 2-4x performance that cost less than half as much as their cloud equivalents and we’re not even co-locating (which has its own set of hassles).

Mixpanel today

We’ve moved 100% of our machines that rely upon performant disks to dedicated servers hosted at Softlayer. Roughly speaking, this corresponds to about 80% of our hosting costs. Eventually, we’ll move everything both for ease of management and bandwidth savings (a lot of our traffic could be internal to a datacenter if all of our machines we’re hosted in the same place).

Since I started this migration, our traffic has grown more than ten-fold. At the same time, our infrastructure has gotten significantly faster, more reliable, and interestingly enough cheaper (at the per machine level). Most importantly, the amount of time I’ve spent fixing server issues late at night or on weekend has decreased to almost nothing.

I hope this post has convinced at least one growing to startup to consider dedicated hardware a lot sooner than we did. Honestly, as soon as you first start to see issues with inconsistent or poor disk performance, you should probably move. It will save you a lot of late nights, development time, and grief.

If you're interested in what we work on, please apply - we're hiring: http://mixpanel.com/jobs/

Written by mixpanel

October 27th, 2011 at 12:34 pm

Posted in Uncategorized

58 Responses to 'Why We Moved Off The Cloud'

Subscribe to comments with RSS or TrackBack to 'Why We Moved Off The Cloud'.

  1. Good luck with SoftLayer. We run a site using dedicated hardware with them and I cannot wait until we move to somewhere else. We are completely fed-up with random hardware issues, SoftLayer goofing up their network topology, and us always having to prove the problem is SoftLayer’s fault before they take action. Almost anything in the “cloud” will be better than this.

    Cliff McCollum

    27 Oct 11 at 1:13 pm

  2. Hey,

    While I generally agree, I don’t think you can say clouds are bad because you’ve got people doing stupid stuff in a very poorly designed cloud architecture.

    The entire point of clouds is to NOT allow a single cloud to affect another one. And you are clearly not getting that.

    But clouds where never entended to handle amazon, a dedicated server farm with replication/load balancers/etc is what was intended to handle amazon. Clouds are just a good solution for a small set of websites which require 100% uptime, and the ability to scale, at least to the point where one would desire to be separate from the rest, or where ones cloud specs are higher then that of multiple dedicated servers.

    But then buying three dedicated servers in a single database is a single point, if that datacenter had power issues, or something (which is rare) you have guaranteed downtime with no ability to “quickly move to another cloud”.

    There are pluses and minuses for sure, but to suggest that dedicated servers give 2-4x the performance is a bit absurd and I would suggest yelling at your cloud provider for sucking.

    Steven Verbeek

    27 Oct 11 at 1:22 pm

  3. Softlayer is way too expensive for what they offer. Hetzner.de is way better on both hardware and price.

  4. Clearly you got it wrong this time. I think you’re missing the whole point of the cloud. Good luck with SoftLayer. I’m sure we are seeing another blog post after a few months about why you’re ditching SoftLayer and moving back again to the cloud.

    Jason Shen

    27 Oct 11 at 1:51 pm

  5. That’s odd to hear Cliff, I have been at SL for years, with multiple deployments, and they’ve never messed up my network configuration, in fact, they’ve been extremely responsive and flexible with issues that have come up (one of our sites triggered their automated DDoS, and they moved it to a non-DDoS-monitored port with zero downtime) within only 20 minutes or so from it being reported to me that the IP was inaccessable…

    As always, all companies will have people with issues. I have had issues at every other company I’ve been with, and so far my track record at SL has been flawless; except when I tried their BMI (bare metal instance) servers when they offered them. They seemed to provision any random equipment they had around for those, which were supposed to be like a cloud-like physical machine; mine did have random hardware issues there.

    However, all my dedicateds have worked flawlessly, for years now – in fact, most of my servers have an uptime now of actually two years on the dot – no power cycles, reboots, or anything; I did have one hard drive that got reported as failing through their system, which sent them a ticket, and they emailed me to ask when a good time would be to swap the disks out because it may require a short downtime/performance degredation.

    mike

    27 Oct 11 at 2:08 pm

  6. I’ve had a similar experience with VPS/Cloud Hosting. After experiencing a crash on my Cloud VPS, I’ve also decided to go to dedicated hosting. It feels safer, and is familiar ground (at work we have dedicated servers instead)

    VPS/Cloud’s can be great for a lot of things, however with the intention of having e-commerce stores hooked up, we need something known and reliable. So a dedicated server seems to be the right choice at the moment.

    Before seeing this article on Twitter I had just signed up to Tagadab, I highly recommend for people looking for UK based hosting. Service is top notch, their interfaces need work, however after that everything is great! :)

    Ash Smith

    27 Oct 11 at 2:15 pm

  7. Agree with Steven – hetzner.de has very good prices / support / … Happy customer since 3 years now.

    mark

    27 Oct 11 at 2:18 pm

  8. Spending hundreds of hours on workarounds for a virtual solution that is supposed to make your life easier is terrible, I did that several years ago and became fed up with unexpected out of the blue issues and now have two co-located dell servers. I pay $198 for 15mbps and the co-location space, now I no longer have to even think about my hosting… it just works.

    George Scott

    27 Oct 11 at 2:21 pm

  9. Steven, you don’t seem to have much of a clue. Of course you are going to get more performance out of dedicated hardware, for the amount of development and management going into cloud architectures, you don’t think they’re going to charge a premium over the hardware cost?

    Also, 100% uptime is an impossibility. Whether you go cloud or dedicated, you need to architect your own systems with redundancy and failover plans. If you are shooting for 100% uptime and are willing to pay for it then you will want hot servers in at least 2 clouds (ie. from different providers) / datacenters and quality DNS services in order to be able to fail over quickly.

    You seem to be referring to the fact that you can quickly provision new instances in case of hardware failure, and that is indeed a benefit, but if you have a large database you need hot failover instances or you will be down for hours / days even if you can commission a new instance in minutes.

    As for “clouds not being intended to handle Amazon,” that is just nonsense. Amazon were the major creative force behind modern web hosting cloud services. They generalized their infrastructure to be saleable and then dogfooded heavily. So actually, the prototype cloud service was exactly designed to handle Amazon.

    Gabe da Silveira

    27 Oct 11 at 2:40 pm

  10. I can understand having frustration with a specific cloud vendor. But, it seems a bit of a leap to cast judgment on cloud architectures in general.

    Believe me, I understand the drive for a customized hardware solution. But, coupling an architecture to specific hardware would give me pause. Shouldn’t the goal be to decouple our solutions?

    What happens when the CPU you coded against is no longer manufactured? Will you get trapped in an expensive latest-and-greatest chipset loop? It would seem more prudent to throw commodity hardware at your problem than expensive hardware.

    I’d be interested to know your thoughts on this.

    -Best

    TVD

    27 Oct 11 at 2:44 pm

  11. With my experience, private cloud and/or hybrid cloud solutions can be a happy medium. In the end I chose to go back to managed/dedicated for my cluster and chopped up one of the machines into VM’s for my purposes. I used Lighcrest out of Los Angeles. Ryan hooked up a good deal and the migration was relatively painless

    D.Masterson

    27 Oct 11 at 2:46 pm

  12. Thanks so much for the kind words – To Suhail and the entire Team at MixPanel –> We are very happy to have you guys on the SoftLayer Platform!

    And thank you for laying out so nicely – why the cloud is great, but not great for everything.
    __________________

    This is for Cliff –

    Cliff – Can we talk some time? I would like to better understand what issues you are having at SoftLayer. Perhaps I can help.

    If you would please ping me at pford at softlayer dot com, I would welcome the discussion.

    Paul Ford
    VP, Community Development
    SoftLayer

    Paul Ford

    27 Oct 11 at 3:02 pm

  13. In my humble opinion it’s either The Cloud or your own data center that’s the best options.

    Moving to a hosted solution usually bases itself on one of three things:

    - You don’t know how to, or do not want to run your own hardware
    - You want to shift responsibility for your service outside of the company
    - You believe setting up and running your own environment is to expensive

    In my opinion, the only reason that is truly valid and can constitute a good reason is the third. Though I believe that for almost any company the cost over time will become less then going with a hosted solution.

    The first two are valid reasons, but as a consumer/customer something I really do not like. If something breaks, I’d rather the company I am working with fess up and fix it then blame a third party, ALWAYS.

    As for the competency issue, I would be worried if the company that developed the service do not have the competency to run their own environment.

    Daniel Larsson

    27 Oct 11 at 3:06 pm

  14. It’s good to know that all the time I spent learning how to build out a dedicated hosting environment in the early 2000′s wasn’t a complete waste of time after all ;)

    Jon

    27 Oct 11 at 3:09 pm

  15. Great point. The cloud is not for everybody. No one product is suitable for all applications. Thanks for the alternative view.

    Bloice Davison

    27 Oct 11 at 3:18 pm

  16. Did you investigate going with a simple managed colo? We had good success with that at Hive7. You rent some on call guys when needed to do the grunt HW work. How many servers are we talking here? In my experience, dedicated hardware is the worst of both worlds if you have any interesting amount of HW (10′s of servers). You have a huge set of things out of your control, most of the headaches of owning hardware, and a high cost that is easily recouped within a few years of owning your own stuff.

    JD Conley

    27 Oct 11 at 3:32 pm

  17. +1 on colo. After you get over the fact that it’s the least sexy hosting possible, there is a lot to love, starting with the fact that every time I’ve run the numbers, it has ended up significantly cheaper than any other hosting option.

    I am too lazy atm to provide specific examples, but I run the numbers every six months or so, and every time, I find that the net costs for a year of colo (=buying a server then running it for a year) are extremely similar to the net costs for a year of cloud, given comparable hardware. The difference is that, after year one, you’re saving a ton of money given that an average colo server will run $100-$200 per month for unlimited performance, whereas that souped up cloud server will keep running you $800+ monthly until you die.

    I’m sure detractors would say that colo means having to deal with more hardware issues yourself, but in our experience, the redundancies built into hardware mean those problems happen way less than one would expect. We have grown to 10 colo servers over 3 years, and in that time, I believe there have been two total incidences where we had to buy a replacement part for one of our servers. Assuming you architect for failover (which you’d need to do in the cloud anyway), such hardware probs aren’t a big deal anyway.

    Have been meaning to write a blog about the virtues of colo for awhile, but I think it would resonate with a pretty narrow audience (i.e., don’t think it makes sense unless you’re doing at least 500k-1m monthly uniques)

    Bill Harding

    27 Oct 11 at 4:34 pm

  18. I had a client with similar problems using SoftLayer’s CloudLayer. Performance issues during peak times kept popping up and SL tech support had nothing useful to add until after a week of this someone said our disk I/O was too much for the cloud. While I don’t have many numbers to back me up, the disk I/O was not enough to be too much for the cloud. Seems to be SL was overselling the cloud hardware and not managing it well. We moved the box that was on CloudLayer to a dedicated server and things smoothed out. While I would have like to have left SL due to how frustrating their support had been, there was a database server on a dedicated server that the client could not afford the downtime to move.

    After that, I tend to avoid dealing with SL, unless I have no choice.

    Scott Blaydes

    27 Oct 11 at 4:50 pm

  19. I moved off Softlayer to AWS because I couldn’t count on Softlayer to do anything right.

    One time I had them add a new disk to my machine, and the tech formatted /dev/sdc, not /dev/sdc2 (no partition table.) When the machine rebooted, the superblock got overwritten and the system went down. It turned out when I put the ticket in to get that change made, they screwed up my record in the ticket system so I couldn’t put tickets in anymore.

    Making any kind of configuration change was like playing Russian Roulette. I’d get it all FUBARed and then I’d call up and talk to some guy who knew nothing and then they’d send me to another guy who knew nothing and eventually end up talking to some guy who was pretty good, but along the way I’d end up wasting a few hours of my time.

    At least AWS when it gets screwed up it’s because I screwed up, not because some newb sysadmin screwed it up.

    Paul A. Houle

    27 Oct 11 at 5:01 pm

  20. Sorry to hear about Rackspace, I have had good luck with them, but I guess it all depends on your application and architecture. Two thoughts:

    1) You can have the best of both worlds (depending on your app) by mixing cloud and dedicated. Providers like GoGrid allow you to deploy dedicated machines on demand for your data i/o, and use cloud servers for your web and app servers. It is all on the same VLAN for fast performance, but you don’t lose the ability to scale quickly if you need to. (Full disclosure: I was a founder of GoGrid but am no longer involved, and I worked at Rackspace briefly)

    2) Did you try getting larger size cloud servers? Obviously they are not cheap, but all of the cloud providers are now rolling out really large instances, where you would be sharing with a small number of other customers, making your i/o scenario less likely. I think some cloud providers may even have instances that are only 1 or 2 customer VMs per server.

    As for Softlayer, good choice if you are going dedicated.

    And colo? Ouch, feels like a bad “Datacenter Time Machine” movie taking me back to 1999. Server huggers. ;-)

    Good luck!

    Hectic

    27 Oct 11 at 5:07 pm

  21. I have been with softlayer for over 4 years, most of that time on a dedicated server and just recently migrated to their cloud environment, which, has gone good so far.

    My websites are “medium” sized, so, I don’t have to worry too much about scaling, but none the less, I have nothing to complain about.

    Best of luck. I look forward to a follow up post, no matter how the cookie crumbles. :-)

    _Zack

    Zack B

    27 Oct 11 at 5:29 pm

  22. Questions:

    - Did you look at any Private Cloud Solutions?
    - do you really require perfect score performance on a per node basis? (As engineers, its easy for us to get tunnel vision on server performance, ignoring the trade-offs of flexibility, reliability, and cost)
    - Is SoftLayer giving you a kickback for writing this article? (Mostly kidding ;) Mostly..)

    GJ

    27 Oct 11 at 5:32 pm

  23. I’m surprised that so many people here have had trouble with Softlayer. I’ve had perhaps one incident that I attribute to idiocy on the part of the techs, but it was with one of the old EV1/Rackshack servers, so I assumed they let the brain damaged people who used to work there keep those servers operational. Other than that I have very little negative to say about Softlayer.. and we spend thousands of dollars with them a month.

    thomas lackner

    27 Oct 11 at 6:47 pm

  24. Take a look at AWS. Their only server with variable performance is the micro. All the others have fixed CPU and I/O and network. There is some variability between instances because they run on different hardware. But you never get hosed by a neighbor.

    BraveNewCurrency

    27 Oct 11 at 6:48 pm

  25. [...] Mixpanel 從 AWS 搬到 Softlayer Posted on October 28, 2011 by Gea-Suan Lin Tweet 在 Mixpanel Engineering 上提到 Mixpanel 從 AWS 搬到 SoftLayer:「Why We Moved Off The Cloud」。 [...]

  26. I would like to know how do you handle following challenges.
    01.catastrophic failures like lightning/ earthquakes
    02.how to balance requests based on their geographically patterns

    sanj

    27 Oct 11 at 8:11 pm

  27. I have banged my head with dedicated, cloud and colo and I call colo the winner in our case.
    I put our stuff in a colo 2 years ago and never looked back. Pretty much all servers come with some kind of remote console interface IPMI, and that’s not terminal redirection, thats actually a totally self contained microprocessor and ether port that you can run on a separate subnet and control your server even if it’s off. I updated the bios, reinstalled OS’s, all via IPMI which is part of the motherboards. Add to that power strips that you can also control remotely and you’re all set. Our servers are in the Bay Area, I’m in Canada. I have NEVER had to drive/fly to fix anything. Never even had to use remote hands for anything. Sure some drives died, but standby drives are in place.
    The costs are dirt cheap these days. You can get a full rack, power and a gigabit feed for about $800 in many colos in texas. We opted for equinix in san jose, which is all fancy with work areas, meeting rooms, etc when you are there, but the funny part is, we’re never there!
    I do like the virtualization for some maintenance/flexibility so we have a few servers that are hosts and we run our own private cloud where we get to decide where/what runs. Database servers on bare metal with ssd drives in other cases. Best of both worlds.
    It’s so cheap you get a second colo in a different part of the country to house a second copy of your backups, and some redundant systems just in case something really bad happens.
    Oh yeah and don’t get me started on storage. We store about 100TB of data. How much is that on S3 per month? $12,000/month! A fancy enterprise storage system pays for itself every couple of month of s3 fees.

    Bukrat

    27 Oct 11 at 8:46 pm

  28. It’s usuallythe other way around but can understand how it works for you.

    I rent some dedicated servers with support but am using the cloud increasingly. I even have some very powerful Dell PowerEdge servers I want to sell but am unable to get rid of them at a good price! I use to have a colocation space but it was too stressful as I was also the tech support.

    Jerome Paradis

    27 Oct 11 at 9:05 pm

  29. BraveNewCurrency said:

    “Take a look at AWS. Their only server with variable performance is the micro. All the others have fixed CPU and I/O and network. There is some variability between instances because they run on different hardware. But you never get hosed by a neighbor.”

    I’m eager to hear why you think that is so, because it is very much not so.

    Tim McClarren

    27 Oct 11 at 9:26 pm

  30. Jason, the author talked about IO-bound servers and those are the ones they migrated off the cloud. It’s unlikely a cloud (ie, unspecified) machine will ever reach the performance of a dedicated machine with specialized hardware.

    Clayten H

    27 Oct 11 at 10:39 pm

  31. +1 for SoftLayer. We’ve been with them for many years. I can’t even remember being down once in all these years. I heard their cloud is not as good as their dedicated service, we have dedicated. You have to negotiate with them, and don’t go through a reseller.

    Peter

    27 Oct 11 at 10:55 pm

  32. We actually have about 50 machines at Softlayer.

    We started working on a new project and did an EC2+Rackspace comparison and Softlayer STILL came out on top :-)

    Kevin Burton

    28 Oct 11 at 1:04 am

  33. I heart Softlayer is great, my friend use them for her company.

  34. The most important thing is the right tool for the job and obviously you’re in the best position to know what that is for you.

    I have to say, though, that my experience Rackspace’s cloud has not been bad at all (I’ve never seen the variable performance that nipped you). You definitely get more performance from a dedicated server but not having to worry about discrete hardware failure and backups is a HUGE bonus to me, as is the availability of more RAM/processing resources on-demand. I have a mix of dedicated and cloud server though – each have their uses!

    ThePlanet errr SoftLayer has given me great service for years, so has Peer1, He.Net and Rackspace, and LeaseWeb if you’re looking for hosting in Europe.

    Mitchell Vincent

    28 Oct 11 at 7:45 am

  35. In time high performance clouds will rise to solve this problem. High IO requirements is a big enough portion of the market to dictate specialized cloud computing services based around them.

    Take a look at cloudharmony, you will see that there exists cloud providers with very good disk IO and even some that put fairness mechanisms in place to make this very reliable.

    In time it will become so much cheaper to use cloud resources due to massive multi tenancy benefits that dedicated servers won’t present attractive TCO.

    Cloud is not about scaling your application or any of that BS that people people. Cloud is about numbers, hard cash and ease of deployment, elasticity and transparency of costs. In the end the most important metric of all is the TCO of whatever solution you decide to employ.

    Joseph

    28 Oct 11 at 1:51 pm

  36. While I understand there is a point where moving off the cloud makes sense. Especially in the analytics and big data spaces. I would have chosen colo over a dedicated provider. Especially if you wanted to get into serious optimizations in disk IO and CPU / GPU computations.

  37. @Steven, Catalin, Mark: I disagree about Hetzner. Their prices are competitive but their network is very flaky and customer support is non-existent. In addition they recently left online customer data openly accessible, including bank data and passwords to servers (just google for “Hetzner leak”).

    sepp

    29 Oct 11 at 12:26 pm

  38. [...] via:Mixpanel Engineering [...]

  39. [...] via:Mixpanel Engineering [...]

  40. [...] via:Mixpanel Engineering http://www.leiphone.com/how-about-cloud.html 30 10月 2011 in 业界 [...]

  41. [...] Why We Moved Off The Cloud Cloud computing is often positioned as a solution to scalability problems. In fact, it seems like almost every day I read a blog post about a company moving infrastructure to the cloud. At Mixpanel, we did the opposite. I’m writing this post to explain why and maybe even encourage some other startups to consider the alternative. [...]

  42. [...] via:Mixpanel Engineering [...]

  43. [...] via:Mixpanel Engineering [...]

  44. [...]   via:Mixpanel Engineering [...]

  45. [...]   via:Mixpanel Engineering [...]

  46. This is a great case study for the next generation of cloud computing, IMHO. You mention “variable performance” being a showstopper and being a victim to multi-tenancy. I couldn’t agree more.

    So another way forward is to look at DATA across clouds and evaluate solutions that leverage the best provider, at the right place at the right time. Have a look at our Free Data portal where you can see how Rackspace performs the other cloud providers.

    https://portal.cedexis.com/dashboard/public/home.html?siteLanguage=en

    We have nearly 200 customers using both our data and our intelligent load-balancing solution to help them solve the problems that you described. Let me know if we can help.

    Ed Sarausad
    http://www.cedexis.com

    Ed

    31 Oct 11 at 10:09 am

  47. If you’re looking for reliable, cost competitive dedicated hosting for US/EU, including an extremely extensive network, I’d be more than happy to organise comparison quotes.

    p(dot)grimwood(at)leasweb(dot)com.
    Business Development Manager

    Paul Grimwood

    31 Oct 11 at 12:22 pm

  48. On my experience, you should try VMware enabled clouds. Clouds built on their vSphere + vCloud platform have built-in controls to truly avoid the “noisy neighbor” issue you faced. I don’t Rackspace builds on VMware.

    Andre

    31 Oct 11 at 1:38 pm

  49. I have banged my head with dedicated, cloud and colo and I call colo the winner in our case.
    I put our stuff in a colo 2 years ago and never looked back. Pretty much all servers come with some kind of remote console interface IPMI, and that’s not terminal redirection, thats actually a totally self contained microprocessor and ether port that you can run on a separate subnet and control your server even if it’s off. I updated the bios, reinstalled OS’s, all via IPMI which is part of the motherboards. Add to that power strips that you can also control remotely and you’re all set. Our servers are in the Bay Area, I’m in Canada. I have NEVER had to drive/fly to fix anything. Never even had to use remote hands for anything. Sure some drives died, but standby drives are in place.
    The costs are dirt cheap these days. You can get a full rack, power and a gigabit feed for about $800 in many colos in texas. We opted for equinix in san jose, which is all fancy with work areas, meeting rooms, etc when you are there, but the funny part is, we’re never there!
    I do like the virtualization for some maintenance/flexibility so we have a few servers that are hosts and we run our own private cloud where we get to decide where/what runs. Database servers on bare metal with ssd drives in other cases. Best of both worlds.
    It’s so cheap you get a second colo in a different part of the country to house a second copy of your backups, and some redundant systems just in case something really bad happens.
    Oh yeah and don’t get me started on storage. We store about 100TB of data. How much is that on S3 per month? $12,000/month! A fancy enterprise storage system pays for itself every couple of month of s3 fees.

    +1

    Catlin

    31 Oct 11 at 5:36 pm

  50. I’ve seen/helped a number of companies move both in and out of various clouds, with good and bad results in any case you want to mention. Among the problems that have prompted some to leave various clouds are the variability mentioned at the beginning of this post. For those who doubt, I have measured this extensively and it’s real. I’ve even been hired to perform the same metrics by clients who didn’t believe the results I got for other clients.

    One write-up of my findings is here: http://www.mysqlperformanceblog.com/2011/02/21/death-match-ebs-versus-ssd-price-performance-and-qos/

    Baron Schwartz

    4 Nov 11 at 1:09 pm

  51. [...] good to see every now and then that someone else has an experience that matches our own, such as Mixpanel’s decision to move off Rackspace’s cloud and onto dedicated servers. I’d love to know how to negotiate 50%-75% off a vendor’s [...]

  52. [...] off the cloud Mixipanel may have lost their angel status. Why would they do such a thing? Read Why We Moved Off The Cloud for the details. The reason for the fall:  highly variable performance. Highly variable [...]

  53. [...] the cloud isn’t the right choice. Mixpanel’s CTO explains why they decided to move their application out of the cloud and into their own dedicated [...]

  54. [...] off the cloud Mixipanel may have lost their angel status. Why would they do such a thing? Read Why We Moved Off The Cloud for the details. The reason for the fall:  highly variable performance. Highly variable [...]

  55. [...] Why We Moved Off The Cloud at Mixpanel Engineering After getting fed up with variable cloud performance, I decided to make the move to dedicated hardware. [...]

  56. [...] cost so we started looking at alternatives. After talking to our friends at Mixpanel (and reading their blog) and at one of our customers, Struq, we selected Softlayer. Specs of one of our dedicated MongoDB [...]

  57. [...] via:Mixpanel Engineering [...]

  58. [...] of cloud dropouts that will soon be blogging on why they moved back to physical hardware (e.g. Mixpanel, Zynga). If you are going to the cloud, do so with eyes wide open, fully embracing the design [...]

Leave a Reply

Safe place to purchase clomid and buy weight loss pills