open beta buildout

[mark]

For the curious, when Thursday rolls around, Open Beta will have...

1x Global Cluster (master/slave)
3x User Clusters (master/master)
20x Web Servers
4x Memcache Servers
6x MogileFS Stores
4x Job Servers
4x Perlbal Servers
+ a few miscellaneous machines

For a monthly cost of ~$8,000. This is most certainly overkill for Open Beta. Probably quite a bit overkill. But we decided we'd rather be safe than offline, and we will adjust the spend as appropriate during the rest of May, to bring things down to a sustainable level.

This brings the total number of servers (virtual servers, but still) that will be handling Dreamwidth.org up to nearly fifty. That's a lot of oomph!

Comments

open beta buildout

[piranha]

AWEfuckingSOME.

there is no such thing as too much POWER (*channels tim allen*).

Re: open beta buildout

[denise]

MORE POWER!

Re: open beta buildout

[chris]

*arf arf arf arf arf*

Re: open beta buildout

[ladyvox]

Wow!

Re: open beta buildout

[ladyvox]

Oops, that comment was meant for the OP, ohwell.. tis still WOW! "_

[catchmyfancy]

I'm a great believer in having more than you need then dialling it back - no interruptions to service and everybody is happy and 50 servers - golly!

[princessofgeeks]

*cheering*

[squeaky.insanejournal.com]

Woah,

I think you might be going a bit overboard with your costs. I wonder if you are not properly estimating your true costs. If you want to talk about what IJ does, please let me know. I really want to see you guys succeed, I think the online journaling community could use another quality site, I just wonder if you are over estimating your hardware needs. Your business plan will be much more succesful if you can pare those costs some.

I am available on IM or email, you can get my IM from my IJ profile page. And my email is the most obvious you can think of.

[mark]

Oh for sure, I know it's overboard. That was a strategic decision. I'd rather overspend a few thousand to make sure Open Beta is successful. I'd rather scale down than have to hurriedly scale up.

There are also other considerations. You are using dedicated hardware, are you not? You get guaranteed performance. Since we're on virtual slices, it's very likely that at least some percentage of our infrastructure is going to be hindered by other people we have no control over.

Also, our databases are physically on SATA drives. Slow I/O. We have to go overkill on the memcache to reduce I/O. My benchmarks (bonnie++) show a relative lot of variability to the read/write speed of the slices. I'm pretty paranoid about it, so I'm going overkill.

Another point against us is that Slicehost has limits on the throughput of your network interfaces. Each machine is limited to 50Mbps, and I'm kind of paranoid about having a few large slices and saturating the network. Spreading out gives us more flexibility to keep the overall system load very low and hopefully not run into that problem.

But yeah. For sure it's overkill, probably by a lot.