200 Terabytes Served in 81 Days = 2.47TB per day!

We interrupt our regular programming with a bit of geek braggery. What? We don’t have regular programming here? Sheesh, what kind of blog is this? It’s not a blog? It’s a “Web Log”? Whatever… back to the topic at hand…

A year and a half ago the company where I was contracting decided to build a video site for gamers. It was all the rage, and they didn’t want to fall behind. The programming team was tasked with building the application, and I was tasked with designing the download systems. The manager at that time was very interested in reducing costs, and was more than happy to listen to my suggestions for a cluster of inexpensive commodity boxes that would each have a copy of the video repository on a large RAID volume and be kept in sync with rsync. I am a fan of the Google approach to colo hardware, where you build it cheap and easy to replace.

While I was out of town dealing with my father’s funeral my manager’s manager trashed all my ideas, said we were a Dell shop and were going to buy Dell gear, and he ordered two Dell servers and a cheap EMC SAN that he dictated would be our video download system. Managers who don’t know their tech should stay out of the server room. He spent $25k for an entry level SAN that couldn’t handle the sustained activity of our video download servers a mere two months after we went live. I wish he hadn’t moved on before then, because I am the sort of person to remind a manager of my original proposal and ask them to explain again why it was rejected.

Anyway… With the ‘big man’ out of the picture, and the system he mandated collapsing miserably under the load, I was able to once again pitch my idea. We did use Dell machines for the commodity boxes, but that’s because the Dell PowerEdge 1800 is a very un-Dell-like machine. Well, it was, before Dell dropped it. For around $1400 you could get a PowerEdge 1800 with dual processors, dual power supplies, and a CERC SATA RAID controller, all with a three year on-site warranty. It was a sweet deal, and unlike most Dell boxes the PowerEdge 1800 didn’t use special drive rails; so we could pack it with six Seagate 500G SATA drives fairly cheaply. Configured as RAID-5, that gave us a little under 2.5TG per server for storing videos. A simple rsync script keeps the video archives in sync after user and editor uploads. Apache 2.2 configured with the worker MPM lets me do about 600 simultaneous connections per machine before I start running out of memory. Six of these boxes behind a HAProxy load balancing system can completely saturate a 1Gbps fiber Cogent network drop with plenty of cycles to spare, and we’ve held that level of bandwidth for amazingly sustained periods of time.

For the HAProxy box I used a Dell PowerEdge 2850 with PCIe slots. In order to make the most of our bandwidth I used an Intel PCIe 4x Gigabit Fiber card for the cogent drop and Intel PCIe 4x Gigabit Copper card for the server side. The load balancer and six servers were connected via a Cisco Gig-E switch.

In the past 81 days we have served 200 Terabytes from this little server cluster that I designed and built on a tight budget. I think that is cool. :)

-Chris Knight

9 thoughts on “200 Terabytes Served in 81 Days = 2.47TB per day!”

  1. Hi Chris,

    If you are serving static content (videos) it might be better to use Cherokee web server instead of Apache. I suspect its peak performance is much better. Of course you cant do many fancy things with it, but that might be done with some Apache’s running the PHP whatever side and then Cherokees for the videos.

    Just my 2 cents

    Regards

  2. Hey Chris,

    Actually, any single-threaded web server (thttpd, mathopd) would totally outperform Apache when it comes to purely static content (as I understand is your case) – and I’m talking more than 10 times faster under heavy load.

    A single thttpd can saturate a gigabit connection if files are big enough (such as yours).

    Of course, while keeping haproxy to load balance.

    Regards

  3. There are quite a few other load balancing software in the open source community. Do you guys tried LVS Layer 3/4 IP load balancing?

  4. Just to throw another hat in the ring, I would suggest you check out Lighttpd. A number of large sites (including wikipedia, youtube and meebo) use it for serving static content. We have had in production use since almost 1.5 years and we have never had a problem.

  5. Has there been an update to this article as mentioned in the final paragraph?

    Others have already given fine recommendations, but I’d also suggest looking at Nginx. Its at least as fast as the others when serving static content, can use sendfile to serve the files extremely quickly, and I like how clean and stable it is in very different configurations.

    LightTPD had alot of problems few years back when it and FastCGI were “the way” to deploy RoR apps, Nginx never suffered from the same stability problems, which is why most of the RoR community now uses Nginx in combination with Mongrel or Thin on the backend. Not that you’re doing anything dynamic, it’s just a testament to how rock solid Nginx is as a front-end server.

  6. Hello, I was looking around for a while searching for ip san and I happened upon this site and your post regarding 200 Terabytes Served in 81 Days = 2.47TB per day!, I will definitely this to my ip san bookmarks!

  7. Its good to see that you used HAProxy on a heavy enviroment as describe above. I liked reading your post.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>