Memory leak heads up

[alierak]

Anyone remember when we deployed some servers with Perl 5.10.0 and saw all sorts of problems with memory usage due to a memory leak in map? I ended up drastically reducing the number of requests each Apache process would handle before it exits, mark and I set up various cron jobs that would unceremoniously kill off workers every so often, etc. As we upgraded and phased out older servers, I think it was no longer a problem (e.g., web08 had Perl 5.10.1).

Well, it's looking like there's another Perl memory leak on the newest group of servers. They're running 5.14.2, and things like synsuck and resolve-extacct (really?) are using several gigs of RAM after a few days. I've set up an array of lovely new worker-restarting cron jobs, and I think this is the relevant Perl bug: perl: memory leak in Encode::decode. It is likely that we'll be able to patch this in production.

I guess if you have a choice, just skip to 5.14.4.

Comments

[alierak]

All right. The admin box (patched) has VSZ 686840 and RSS 429424. The synsuck that started at the same time on web13 (unpatched) has VSZ 1077616 and RSS 827524. I have observed the process on the admin box to grow over time, but it is not growing as quickly. I guess that could mean either that there is another memory leak in addition to the one in Encode, or that it's showing a high water mark effect over time as the "largest feed it's ever downloaded" increases, or it keeps on loading more syn users or whatever.

[mark]

Okay, I'm rolling Encode 2.51 it out to the rest of the servers (sync-dw-home2) and restarting things. I'll keep an eye on the error logs and make sure things look happy.