--max_jobs off by one

[mark-99]

I'm using --max-jobs to control worker lifetime, it seems like there's an off-by-one bug in that the workers process 1 more job than the param to max_jobs.

This option is marked as a "temp workaround", but it's a useful feature so hopefully it won't disappear!

Even better would be an option to control worker lifetime by elapsed time, is that possible?

[sylvinus]

Thanks for reporting this! I added a test and fixed the counting issue.

The "temp" in the docs referred to the fact that we don't think it's a long-term solution to memory leaks, but I guess your usecase is more about burst mode, so I reworded the docs a bit.

What would be the usecase for limiting the lifetime of workers? Couldn't this be done easily with an external tool to MRQ?

[mark-99]

Thanks for the fix!

Our use case for max lifetime is we launch thousands of workers via Condor but we need to load balance between users, ie we want 1 user to be able to take the whole cluster if is idle, but not lock it out for hours with long-running generic worker processes.

As such I need the workers expire after a while and the control script can then respawn some more - this maintains a worker pool around the requested size if there's no contention, but allows other users a chance to grab some nodes if they need to.

BTW I figured out a get_num_workers() based on the dashboard source which would be a good candidate for a 1st class API also (can't post now am on my phone).

It might be possible via Condor, but the whole point of using MRQ is that Condor becomes unresponsive and generally doesn't work well when we run very large numbers of processes/jobs. MRQ on the other hand works great as mostly the control script is just talking to Regis and Mongo. This is why burst mode is great, and max jobs, but max lifetime would be ideal.

On 29 Oct 2015, at 21:00, Sylvain Zimmer notifications@github.com wrote:

Thanks for reporting this! I added a test and fixed the counting issue.

The "temp" in the docs referred to the fact that we don't think it's a long-term solution to memory leaks, but I guess your usecase is more about burst mode, so I reworded the docs a bit.

What would be the usecase for limiting the lifetime of workers? Couldn't this be done easily with an external tool to MRQ?

—
Reply to this email directly or view it on GitHub.

[sylvinus]

Ok. What about running jobs? Would you want to worker to wait for them to finish or be killed off and interrupt running jobs?

[mark-99]

Wait to finish. Just check runtime >= lifetime between jobs.

On 30 Oct 2015, at 20:46, Sylvain Zimmer notifications@github.com wrote:

Ok. What about running jobs? Would you want to worker to wait for them to finish or be killed off and interrupt running jobs?

—
Reply to this email directly or view it on GitHub.