The "entities" of a Dreamwidth setup

[szabgab]

Hi,

Seeing how complex the deployment of applications developed in various corporations I am doing a little research to see how open source application manage the complexity. For that I was looking for complex (in terms of deployment and development environment) applications to see how they manage the task and how they test their system. Dreamwidth seems like a good case so I jumped on the IRC channel where pauamma gave me a quick list of the entities. Then I was volunteered to raise the issue here to get further clarifications.

So here is the list I got:

1. perlbal on the load balancer(s)


2. webservers running Apache+modperl+DW webapp code


3. database cluster for global data


4. several (currently 3) database clusters for per-user data. Each cluster holds the same data for different users


5. Gearman servers, to offload some web-synchronous compute-intensive tasks


6. TheSchwartz server(s) for asynchronous or non-web tasks.


7. MogileFS servers for blobby data not stored in a SQL database for various reasons.


8. Postfix

I found the Production Notes to be also a good source for information but I'd appreciate to get further details if the list is not complete. What would be very interesting to me is to understand what are the responsibilities of the various entities in the system. What kind of data is kept in the various databases. Which entities can be duplicated if load requires it and how the jobs can be divided?

Comments

[pauamma]

Ooooh, nice, Thanks.

A few t5hings that (IIRC) are no longer current for LJ from thisby diagram are dmtpd and qbufferd. (Replaced bu TheSchwartz and maybe Gearman.) Also missing for LJ are FotoBilder/ScrapBook and its insane interface to LiveJournal (oh, the horror, the horror!) and DJabber/LJTalk, but neither of those is used on Dreamwidth.

AFAIK, Asterisk and phonepostd are still current for LJ, but not used on Dreamwidth.

Also also, in the list I gave szabgab, I forgot memcache and pound.