Still a memory leak

[alierak]

I previously noted an apparent memory leak in several workers under Perl 5.14.2. We upgraded the Encode module because it had a known memory leak bug and seemed relevant, and that did actually slow down the memory leak somewhat. However, I now need help brainstorming what could still be causing this. The affected workers seem to be synsuck, send-email, and resolve-extacct, so I've set cron jobs to restart those every so often.

Honestly, what can resolve-extacct be doing that causes it to grow to 9GB in size over the course of a week? We will probably have to resort to profiling memory usage to figure this one out.

Comments

[pauamma]

Honestly, what can resolve-extacct be doing that causes it to grow to 9GB in size over the course of a week?

Semirandom guess: DW::External::Userinfo uses a process cache somewhere that's never cleared.

[alierak]

I think I figured out resolve-extacct and synsuck, at least.

ETA: found same problem in send-email

[dreamatdrew]

OK, that is a wicked odd set of processes to be memoryhogging. And I think the only thing they have in common is that they touch the user db. Any objections to me trying to unspool this from the outside?

[alierak]

But then the question is, don't we have other long-running workers that touch the user db, like all the ESN stuff? No, of course, any help is appreciated.

[pauamma]

ESN uses TheSchwartz. resolve-extacct uses Gearman. What do the other memory hogs use?

[alierak]

synsuck and send-email use TheSchwartz.

[alierak]

I think I figured out synsuck and resolve-extacct at least. They're doing DNS lookups on arbitrary numbers of external domains, and we have a known leaky Net::DNS 0.72. I ran a modified resolve-extacct for a while last night, using Test::LeakTrace to identify modules that might be leaking, and Net::DNS dominated the output.

ETA: send-email is doing DNS lookups too; code is in external TheSchwartz::Worker::SendEmail module rather than LJ/Sendmail.pm as I had thought.

[pauamma]

Time to test the patch at https://rt.cpan.org/Ticket/Attachment/1237442/654131/net-dns-0.72-mem-leak.patch ? (Dunno what timezone rt.cpan uses, so it's possible the patch came after you posted your comment.)

[alierak]

Yeah, I applied that on one machine, ran my modified resolve-extacct again and saw nothing leaked by Net::DNS anymore. I've propagated it to the rest of them now.