Memory leak heads up

[alierak]

Anyone remember when we deployed some servers with Perl 5.10.0 and saw all sorts of problems with memory usage due to a memory leak in map? I ended up drastically reducing the number of requests each Apache process would handle before it exits, mark and I set up various cron jobs that would unceremoniously kill off workers every so often, etc. As we upgraded and phased out older servers, I think it was no longer a problem (e.g., web08 had Perl 5.10.1).

Well, it's looking like there's another Perl memory leak on the newest group of servers. They're running 5.14.2, and things like synsuck and resolve-extacct (really?) are using several gigs of RAM after a few days. I've set up an array of lovely new worker-restarting cron jobs, and I think this is the relevant Perl bug: perl: memory leak in Encode::decode. It is likely that we'll be able to patch this in production.

I guess if you have a choice, just skip to 5.14.4.

Comments

[alierak]

So, I think the quick fix for production is just this:

cd $LJHOME
cpanm -L extlib Encode

(then sync this to the other servers). Would anyone like to confirm whether Encode 2.51 is suitable for use with Dreamwidth?

[mark]

I don't know if 2.51 is acceptable, but I think that would probably fix it if this is the source of our problem. We could try rolling it out on admin01 and then run a local synsuck for a few days and see if a) it has problems and b) if it fixes the memory leak?

[alierak]

Okay, giving it a shot.

[mark]

FWIW, I did upgrade my hack last night to use 2.51 with the proposed cpanm flow, and it seems to work. Granted, I don't have syndicated feeds so it's not a great test... but the web site itself seems to work and not explode.

[alierak]

Ok, good. So far that synsuck is growing more slowly than the others, so that's a good sign. I'll give it a day or so (and disable the cron job on web13 for a control) to be sure.

[alierak]

All right. The admin box (patched) has VSZ 686840 and RSS 429424. The synsuck that started at the same time on web13 (unpatched) has VSZ 1077616 and RSS 827524. I have observed the process on the admin box to grow over time, but it is not growing as quickly. I guess that could mean either that there is another memory leak in addition to the one in Encode, or that it's showing a high water mark effect over time as the "largest feed it's ever downloaded" increases, or it keeps on loading more syn users or whatever.

[mark]

Okay, I'm rolling Encode 2.51 it out to the rest of the servers (sync-dw-home2) and restarting things. I'll keep an eye on the error logs and make sure things look happy.