Server Farm Setup for Knife Depot

OK… the title is a little misleading.  This post actually discusses the setup for my entire company, Internet Retail Connection.  We run a few other sites, like Picnic World and Backyard Chirper, but Knife Depot is our bread and butter.

A very abridged background story is that we run our own ecommerce websites on a custom platform developed over the last 10 years.  It started as a legacy platform that ran on PHP 4 (yikes) and is now almost completely run on the Symfony Framework.  We have unique business requirements that make a custom platform the best choice for us.

Current Infrastructure - New Page

old setup

Our Old Setup

For a long time, we ran our entire business on two machines:

  1. A single dedicated server at WestHost
  2. A VM at VPS.net

The VM at VPS.net was only used as a MySQL slave for backup purposes, so we pretty much ran everything on one machine.

The huge benefit of this server farm setup is it’s simplicity.  Everything is on one machine, so it’s really easy to monitor, maintain, deploy code, etc. The KISS principle really applies here.

The big downside is that it’s too simple and it’s a single point of failure.  If a hard drive died, the entire infrastructure goes down.  Need to take down a web server for maintenance? The entire infrastructure goes down.  Suddenly getting a lot of traffic? Impossible to scale up quickly.

Needless to say, it was time for a change, and I wanted to do it on my timetable and not during an emergency.  After a bunch of research, I decided to move over to the cloud servers on DigitalOcean, although AWS was a close second.

Moving To A Multi-Server Setup

I wanted to have multiple web/application servers, but our current application was designed to run on a single server.  These were the primary roadblocks to having a multi-tier web application.

Assets (product images, videos)

These need to be available to all web servers.  To solve this, we moved them to Amazon S3 and serve them via Amazon’s Cloudfront CDN.

Session Data

When someone browses our site, we keep track of certain information (which products they viewed, items in their cart, etc) via a session.  Traditionally, session data is stored as a file on the web server.  Since someone browsing our site might be directed to different web servers in the same session, this doesn’t work.

At first, I tried configuring Varnish (load balancer) to use sticky sessions and always send the same customer to the same application server.  This didn’t work very well because some ISPs dynamically change their customers’ IP addresses. If a customer would get directed to a different application server, they would lose their session.  This manifested itself by phone calls and emails from customers telling us that their cart contents were “lost”.  Not good.

Next, I tried storing the session data in MySQL using Symfony’s PDOSessionHandler, but it is broken for high traffic sites.

Finally, I ended up storing the session data in MongoDB using Symfony’s poorly-documented but well-written MongoDBSessionHandler.  It works perfectly for this type of storage and is very fast.

Our New Setup

Here is a diagram of our new server farm infrastructure. I’ll explain more below.  I’m not going to discuss the actual transition between our old and new setups, because that’s a whole different story. All in all, we only had 1 hour of downtime in the middle of the night. I’m either lucky or I planned well.

IRC DigitalOcean Infrastructure - New Page (2)

Terminating Server (nginx + Varnish)

The only server that is public facing is our terminating server and it runs on a DigitalOcean 8GB Droplets ($80/month).  It uses Varnish for caching and load balancing and nginx for SSL termination.

nginx

I wish I didn’t have to use nginx, but at the time of this writing, Varnish doesn’t “speak” SSL. So, nginx is solely used for SSL termination. If you are connecting to one of our sites with HTTPS, nginx will take that connection and convert it to non-encrypted internal traffic on port 80 which it passes to Varnish.

Varnish

Varnish has two purposes. First, it acts as a load balancer between our different web servers, spreading out the traffic evenly among the active servers. Second, it is an HTTP Cache which dramatically decreases page load time on cached pages.  It was VERY complicated to get this working with Symfony (I wanted long cache times and selective cache invalidation), but it works unbelievably well. I might write another post about how I got this working.

Application/Web Servers (apache)

These servers are the meat and potatoes of our company.  They run the actual code that powers our sites and backend systems.  As traffic increases or decreases, we can scale up or down by adding or removing these servers.  We use DigitalOcean 16GB Droplets ($160/month), and the number that we use varies with the time of year.

Session Database (MongoDB)

As mentioned above, this server stores shared session data for use by the application servers. We use a single 4GB DigitalOcean Droplet ($40/month).

Master Database (MySQL) + Message Queue (RabbitMQ)

Our main data storage server – We use a single DigitalOcean 16GB Droplet ($160/month).

MySQL

Our data is stored in MySQL and all reads and writes go through this server.  That is sufficient for our usage, but the data can be replicated to multiple slave servers if we need to increase capacity.

RabbitMQ

This is a recent addition to our infrastructure, and I love it.  We primarily use it to store cache invalidation requests from our application servers, which are handled by consumers on a different server.

Slave Database (MySQL)

This server is a replication slave from our master database.  We use it to take hourly database backups without disrupting the production database.  We also use it for miscellaneous background jobs, such as the cache invalidation jobs mentioned earlier. We use a single DigitalOcean 16GB Droplet ($160/month).

Monitoring (Scout)

Our new setup is a lot more complicated than our old setup, but it’s also a lot more robust and flexible.  Believe it or not, the cost is almost the same. In order to keep tabs on things, I have been using Scout to monitor things and send alerts when the inevitable problems occur.

9 Comments

  1. I had trouble scaling my website away so what I did was pretty similar with what you did in there.
    Just that when you are working with big data and mysql I started to do reads from mysql slaves and writes in the master.

    I did this to ensure that master on mysql is not under big cpu load and I can always add more slaves to it in case I need to do even more things.

    Another thing that I started doing is to separate an API which does most of the common jobs in my website. You’ve done something like this as well ?

    • steven

      I have used the mysql slave for reads in the past, but under our current load, it’s not needed. I have not broken out the functionality to an API yet, but it is on the roadmap.

  2. Mxx

    Why nginx for ssl termination/lb instead of ELB or CloudFlare or Haproxy?
    And if you already use nginx, why keep Apache?

    • steven

      We are not on AWS, so I can’t use ELB. Cloudflare is an external service I don’t want to rely on. Haproxy would do the same thing as nginx so I just made a decision to use nginx for SSL termination. If HAProxy did the advanced caching that Varnish does, I could just use HAProxy for everything which would be awesome.

    • steven

      I’ve used Apache for many years and I know it well. The plan is to replace it with nginx in the future if performance is a problem. I have some legacy code that relies on .htaccess files.

  3. Interesting, thanks for sharing. What about local cache such as twig, route and other cache? How do you clear all cache on all web servers at once when you deploy new code?

Leave a Reply

Your email address will not be published. Required fields are marked *