Information about the MrQ status

[frankrousseau]

Hello PricingAssistant,

We are implementing a task-based processing for our infrastructure management API requests. We are looking for Python queue frameworks. At first, we were interested in using Celery+Redis for it. Then we remembered that RQ was a lighter alternative. After digging into the subject, we discovered your project.

From what says your README, it looks like the proper solution for us: simple as RQ, stores results in a MongoDB instance, comes with a beautiful dashboard.

So my question is can we try to use MrQ for our infrastructure API? Is it easy to start with?

[sylvinus]

Hey Frank :)

Yes, we are just getting ready for our first officially supported release (0.9.0)!

MRQ has been used in production at Pricing Assistant for 1 year, processed more than 1 billion jobs, so we're pretty confident by now! Several other 3rd party projects have begun using it, some in "serious" production too.

Let us know if you need help! The "get started" should make some sense:
https://github.com/pricingassistant/mrq/blob/master/docs/get-started.md

Cheers,

[frankrousseau]

Ok we'll play with it and will come back to you when we decide to use it or not.

Thank you for the information.

[frankrousseau]

Ok MrQ looks very good. But I have two questions:

• Can you give me a snippet + explanation for scheduled task (I didn't find one in the docs and tests are not self-explanatory)? My use case is simple: run a task in the next 24. From what I understand I must use a raw queue with the _timed suffix. But after that, I don't understand how you give the schedule params to the queue (the config file example is not clear).
• Is it ok to start the worker via Supervisord?

[frankrousseau]

One more question:

• Is it possible to sequence jobs? I mean run a job only when another job is finished (status = success). RQ calls it dependency and provides a depends_on parameter.

[sylvinus]

Sorry for the lag - Friday was dotSwift :/

• Scheduled tasks: Do you want a task to run each 24h but don't care when (in which case MRQ's scheduler is the solution) or do you want a task to run precisely 24h after some event (in which case a timed_set queue is the solution)? I'll add snippets for both cases to the doc anyway!
• You can start the worker via supervisord or any other way (by default it's just 1 regular unix python process) but we also have it integrated directly into MRQ if you launch mrq with mrq-worker --processes 2
• We don't have job dependencies because in this case we just schedule jobs at the end of the job if it has completed successfully. Would that work for you? If not I'd be happy to learn about your usecase.

Thanks!

[frankrousseau]

No problem for the delay and thank you for you answer. Finally, we developed our own solution. We don't have much jobs to handle currently and we were in a rush. So, it will be temporarly fine.

Our use case is simple : we spawn containers on which we apply a sery of actions. That actions should be runned sequentially and there is something like 16 actions. We want to track every result to be able to restart failed jobs. Some jobs must be runned 24h later the previous job.

At first, we tried your solution to run the job the previous one finished. But, because of the numerous jobs, it made things a little bit messy. And we prefer to describe the whole workflow in one place.

NB: To be honest, we do it sequentially to keep things simple. But, a workflow engine would probably be better

NB: Congrats for the DotSwift, I saw the pictures, it looked great! I want the Dot stickers too ;)

[sylvinus]

Thanks!

[frankrousseau]

And thank you for the upcoming snippets, it will be useful when we will try to integrate MrQ again.