![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png){kind=small}
Entry tags:
The "entities" of a Dreamwidth setup
Hi,
Seeing how complex the deployment of applications developed in various corporations I am doing a little research to see how open source application manage the complexity. For that I was looking for complex (in terms of deployment and development environment) applications to see how they manage the task and how they test their system. Dreamwidth seems like a good case so I jumped on the IRC channel where![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
pauamma gave me a quick list of the entities. Then I was volunteered to raise the issue here to get further clarifications.
So here is the list I got:
I found the Production Notes to be also a good source for information but I'd appreciate to get further details if the list is not complete. What would be very interesting to me is to understand what are the responsibilities of the various entities in the system. What kind of data is kept in the various databases. Which entities can be duplicated if load requires it and how the jobs can be divided?
Seeing how complex the deployment of applications developed in various corporations I am doing a little research to see how open source application manage the complexity. For that I was looking for complex (in terms of deployment and development environment) applications to see how they manage the task and how they test their system. Dreamwidth seems like a good case so I jumped on the IRC channel where
pauamma gave me a quick list of the entities. Then I was volunteered to raise the issue here to get further clarifications.So here is the list I got:
- perlbal on the load balancer(s)
- webservers running Apache+modperl+DW webapp code
- database cluster for global data
- several (currently 3) database clusters for per-user data. Each cluster holds the same data for different users
- Gearman servers, to offload some web-synchronous compute-intensive tasks
- TheSchwartz server(s) for asynchronous or non-web tasks.
- MogileFS servers for blobby data not stored in a SQL database for various reasons.
- Postfix
I found the Production Notes to be also a good source for information but I'd appreciate to get further details if the list is not complete. What would be very interesting to me is to understand what are the responsibilities of the various entities in the system. What kind of data is kept in the various databases. Which entities can be duplicated if load requires it and how the jobs can be divided?
no subject
http://pics.livejournal.com/xb95/pic/001hfx62
That is a diagram I made when I worked at LiveJournal showing the complexity of the infrastructure. I believe that covered everything, but that was circa 2005. Half a decade later, there have been a lot of changes to how things work.
Dreamwidth is not that complex. It could be, if we were to get as popular as LJ was at the time. We would certainly have to scale things up. Right now we operate a lot of services on one node and we double and triple pack things so that we need fewer servers.
no subject
A few t5hings that (IIRC) are no longer current for LJ from thisby diagram are dmtpd and qbufferd. (Replaced bu TheSchwartz and maybe Gearman.) Also missing for LJ are FotoBilder/ScrapBook and its insane interface to LiveJournal (oh, the horror, the horror!) and DJabber/LJTalk, but neither of those is used on Dreamwidth.
AFAIK, Asterisk and phonepostd are still current for LJ, but not used on Dreamwidth.
Also also, in the list I gave szabgab, I forgot memcache and pound.