Memory leaks can be a big issue with gevent workers because several tasks share the same python process.
Thankfully, MRQ provides tools to track down such issues. Memory usage of each worker is graphed in the dashboard and makes it easy to see if memory leaks are happening.
When a worker has a steadily growing memory usage, here are the steps to find the leak:
- Check which jobs are running on this worker and try to isolate which of them is leaking and on which queue
- Start a dedicated worker with
--trace_memory --gevent 1on the same queue : This will start a worker doing one job at a time with memory profiling enabled. After each job you should see a report of leaked object types. - Find the most unique type in the list (usually not 'list' or 'dict') and restart the worker with
--trace_memory --gevent 1 --trace_memory_type=XXX --trace_memory_output_dir=memdbg(after creating the directory memdbg). - There you will find a graph for each task generated by objgraph which is incredibly helpful to track down the leak.