Worker configuration!

[fu]

etc/workers.conf now contains a list of all the workers, grouped by functionality, and some basic documentation for the most commonly used ones.

Should make it easier the next time anyone has to test crossposting/importing/notifications, etc.

Feel free to adjust further as required, or if you discover that doing something needs a worker that isn't listed!

Comments

[alierak]

Nice! I vaguely remember there being some workers that should only have one instance running, that would be a good place to document those if they still exist.

[fu]

Oooh yeah. I can't think of any like that, but here would be great to put that information.

[alierak]

I think gearman and theschwartz workers might be categorically exempt, since they have well-defined ways to assign a specific task to a worker.

I would look at things like schedule-synsuck and subscribe-hubbub that grab a chunk of rows from some table, iterate over them updating something, and then sleep. If you have more than one running for some reason, they might be working on the same chunk of rows and duplicating tasks.

Some workers of this type (e.g., birthday-notify and expunge-users) use DDLockClient via LJ::locker->trylock() to avoid stepping on each other if running multiple instances. This requires running ddlockd and setting @LJ::LOCK_SERVERS if the instances are on different servers. It might be pointless anyway to run more than one because of the timing of their sleep/work cycles.

[afuna]

Aha!


Hmm, I've only put the gearman/theschwartz workers in here since that's what workers.conf is for. I think that the other workers will have to be in a separate README or up on the wiki somewhere, in that case.

[alierak]

http://wiki.dwscoalition.org/wiki/index.php/Workers?

But the ones marked 'Manual' are neither gm nor schwartz and still run from worker-manager.

[afuna]

!!!

Hmm so I'm guessing all the manual workers always need just the one, yeah?

[alierak]

Some of them will work with more than one because of the locking mechanism. I could probably come up with a pathological case where you had so many users in so many clusters that needed birthday notifications every day that one worker on one server just wouldn't cut it, but that seems pretty unlikely.

[alierak]

That said, a worker that doesn't have a good way of handling parallelism could just be considered a bug.

[ninetydegrees]

Is there anything else one needs to do to be able to make the shop work? It won't let me buy anything (I'm talking about step 1: putting something in the cart).

[fu]

Hmm, make sure that the %SHOP variable is set up in your config. See doc/config-local.pl.txt for an example!

[ninetydegrees]

Done, thanks, but it still won't work. Is there anything else?

[fu]

Hmmm. That's the only one I can think of that fits the symptoms.

I'm assuming it's something like: you add to cart, then the time mysteriously disappears / isn't processed?

Can you check (by printing in your apache logs), that the shop variable is indeed set properly? As a just in case1

[ninetydegrees]

Oh I think I figured it out. Thanks a lot for reminding me about the logs!

[fu]

Woo glad that helped!

[ninetydegrees]

For the record, and in case anybody has the same issue, workers never worked for me because I didn't have my TheSchwartz database set up as my hack was too old and I didn't know I should do this. Sophie had me run this:

$LJHOME/bin/ljdb < /usr/share/doc/libtheschwartz-perl/schema.sql

then restart apache and workers. It wouldn't work otherwise.