alierak: (Default)
alierak ([personal profile] alierak) wrote in [site community profile] dw_dev 2013-04-06 02:57 am (UTC)

I think gearman and theschwartz workers might be categorically exempt, since they have well-defined ways to assign a specific task to a worker.

I would look at things like schedule-synsuck and subscribe-hubbub that grab a chunk of rows from some table, iterate over them updating something, and then sleep. If you have more than one running for some reason, they might be working on the same chunk of rows and duplicating tasks.

Some workers of this type (e.g., birthday-notify and expunge-users) use DDLockClient via LJ::locker->trylock() to avoid stepping on each other if running multiple instances. This requires running ddlockd and setting @LJ::LOCK_SERVERS if the instances are on different servers. It might be pointless anyway to run more than one because of the timing of their sleep/work cycles.

Post a comment in response:

If you don't have an account you can create one now.
HTML doesn't work in the subject.
More info about formatting

If you are unable to use this captcha for any reason, please contact us by email at support@dreamwidth.org