|
| 1 | +This tutorial will guide you through the configuration of a MRQ queue for maximum performance. |
| 2 | + |
| 3 | +Code is available in the `examples/queue_performance` folder. To be able to run the commands below, you should enter the container first: |
| 4 | + |
| 5 | +``` |
| 6 | +make shell |
| 7 | +make stack |
| 8 | +cd examples/queue_performance |
| 9 | +``` |
| 10 | + |
| 11 | + |
| 12 | + |
| 13 | +## Regular queue |
| 14 | + |
| 15 | + |
| 16 | + |
| 17 | + |
| 18 | +### Default setup |
| 19 | + |
| 20 | +Let's start with a simple task that squares integers, from the `tasks.py` file: |
| 21 | + |
| 22 | +``` |
| 23 | +class Square(Task): |
| 24 | + def run(self, params): |
| 25 | + return int(params["n"]) ** 2 |
| 26 | +``` |
| 27 | + |
| 28 | +You can enqueue it 200 times on a regular, MongoDB-backed queue named `square` with this code: |
| 29 | + |
| 30 | +``` |
| 31 | +from mrq.job import queue_jobs |
| 32 | +queue_jobs("tasks.Square", [{"n": 42} for _ in range(200)], queue="square") |
| 33 | +``` |
| 34 | + |
| 35 | +For convenience, we will use the `enqueue.py` file to do this. Here is the command to enqueue the jobs and launch a worker to dequeue them: |
| 36 | + |
| 37 | +``` |
| 38 | +./enqueue.py square 200 && mrq-worker square |
| 39 | +``` |
| 40 | + |
| 41 | +You should see the output of the worker, with a line like this one at the end (performance numbers from a 2015 MacBook Pro): |
| 42 | + |
| 43 | +``` |
| 44 | +[INFO] Worker spent 2.398 seconds performing 200 jobs (83.403 jobs/second) |
| 45 | +``` |
| 46 | + |
| 47 | +As we have `DEQUEUE_STRATEGY = "burst"` in the `mrq-config.py` file, the worker exits as soon as there are no jobs left on the queue, which is more convenient for this tutorial. |
| 48 | + |
| 49 | +80 jobs per second is rather slow. The main bottleneck is that by default, `mrq-worker` uses a single process and a single greenlet. With this setup, jobs are executed sequentially and between each, the worker must fetch the next one from MongoDB. As a consequence, most of the time of the worker is spent on blocking I/O to MongoDB: not good! |
| 50 | + |
| 51 | + |
| 52 | + |
| 53 | + |
| 54 | +### Multi-greenlet worker |
| 55 | + |
| 56 | +Fortunately, MRQ uses [gevent](http://gevent.org) and allows us to start many greenlets at once in the same worker. Let's try with 5 greenlets: |
| 57 | + |
| 58 | +``` |
| 59 | +./enqueue.py square 200 && mrq-worker square --greenlets 5 |
| 60 | +... |
| 61 | +[INFO] Worker spent 0.652 seconds performing 200 jobs (306.554 jobs/second) |
| 62 | +``` |
| 63 | + |
| 64 | +We got an almost linear increase in performance! What if we tried 50 greenlets? |
| 65 | + |
| 66 | +``` |
| 67 | +./enqueue.py square 200 && mrq-worker square --greenlets 50 |
| 68 | +... |
| 69 | +[INFO] Worker spent 0.382 seconds performing 200 jobs (523.174 jobs/second) |
| 70 | +``` |
| 71 | + |
| 72 | +A nice increase again, but definitely not linear anymore. Depending on your workload, the performance gains will stop at some point either because you hit a CPU bottleneck on the worker, or the concurrency limit of your MongoDB server. |
| 73 | + |
| 74 | +If MongoDB is the limiting factor, you have 2 choices to go further: |
| 75 | + |
| 76 | + - Scale your MongoDB instance ([many options](https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/) are available, including [sharding](https://docs.mongodb.com/manual/sharding/)) |
| 77 | + - Switch to a Redis-backed queue (also called a *raw queue* in MRQ). |
| 78 | + |
| 79 | + |
| 80 | + |
| 81 | +## Raw queue |
| 82 | + |
| 83 | + |
| 84 | + |
| 85 | + |
| 86 | +### Default setup |
| 87 | + |
| 88 | +A raw queue must be configured in `mrq-config.py` with its job factory function, which will transform a "raw" parameter string into a complete job definition: |
| 89 | + |
| 90 | +``` |
| 91 | +RAW_QUEUES = { |
| 92 | + "square_raw": { |
| 93 | + "job_factory": lambda rawparam: { |
| 94 | + "path": "tasks.Square", |
| 95 | + "params": { |
| 96 | + "n": rawparam |
| 97 | + } |
| 98 | + } |
| 99 | + } |
| 100 | +} |
| 101 | +``` |
| 102 | + |
| 103 | +The only thing that will be queued in redis will be the raw parameter. This has the benefit of using much less storage than MongoDB-backed queues, but also of being faster to dequeue: |
| 104 | + |
| 105 | +``` |
| 106 | +./enqueue.py square_raw 2000 && mrq-worker square_raw --greenlets 30 |
| 107 | +... |
| 108 | +[INFO] Worker spent 1.804 seconds performing 2000 jobs (1108.419 jobs/second) |
| 109 | +``` |
| 110 | + |
| 111 | +Much better! If you use `top` while launching these commands (you can open a second shell in the same container with the `make reshell` command from the host), you will see that the python worker process is now maxing-out a CPU. |
| 112 | + |
| 113 | + |
| 114 | + |
| 115 | + |
| 116 | +### Multi-process worker |
| 117 | + |
| 118 | +As you know, a single Python process can only use a single CPU. Let's try to use all the cores you have at your disposal to get better performance! |
| 119 | + |
| 120 | +mrq-worker can start multiple processes with the ```--processes``` flag. In this case it will use `supervisord` to manage the processes. If you use this option you will have to manually terminate the worker with a `ctrl-C` keystroke once it is finished: |
| 121 | + |
| 122 | +``` |
| 123 | +./enqueue.py square_raw 20000 && mrq-worker square_raw --greenlets 30 --processes 5 |
| 124 | +... |
| 125 | +[INFO] Worker spent 6.697 seconds performing 4239 jobs (632.986 jobs/second) |
| 126 | +... |
| 127 | +[INFO] Worker spent 6.505 seconds performing 4307 jobs (662.075 jobs/second) |
| 128 | +... |
| 129 | +``` |
| 130 | + |
| 131 | +Each of the 5 worker processes handled its share of the jobs. The performance numbers aren't aggregated but you can see that the global throughput is now more than 3000 jobs per second. |
| 132 | + |
| 133 | +`top` reveals that the bottleneck is once again MongoDB. We are using a Redis-backed queue so jobs are not queued in MongoDB anymore but by default they are still inserted there once they are started. This is done to be able to see them in MRQ's dashboard as well as to store their results once they reach the `success` state. |
| 134 | + |
| 135 | + |
| 136 | + |
| 137 | + |
| 138 | +### Redis-only queue |
| 139 | + |
| 140 | +If you don't need visibility on started jobs or on their results, you can actually bypass MongoDB altogether with this configuration: |
| 141 | + |
| 142 | +``` |
| 143 | +RAW_QUEUES = { |
| 144 | + "square_nostorage_raw": { |
| 145 | + "statuses_no_storage": ("started", "success"), |
| 146 | + "job_factory": lambda rawparam: { |
| 147 | + "path": "tasks.Square", |
| 148 | + "params": { |
| 149 | + "n": rawparam |
| 150 | + } |
| 151 | + } |
| 152 | + } |
| 153 | +} |
| 154 | +``` |
| 155 | + |
| 156 | +Let's try that with a single-process worker: |
| 157 | + |
| 158 | +``` |
| 159 | +./enqueue.py square_nostorage_raw 20000 && mrq-worker square_nostorage_raw --greenlets 50 |
| 160 | +... |
| 161 | +[INFO] Worker spent 8.449 seconds performing 20000 jobs (2367.030 jobs/second) |
| 162 | +``` |
| 163 | + |
| 164 | +Redis should be at less than 1% CPU load, so we can definitely keep adding processes: |
| 165 | + |
| 166 | +``` |
| 167 | +./enqueue.py square_nostorage_raw 20000 && mrq-worker square_nostorage_raw --greenlets 50 --processes 5 |
| 168 | +... |
| 169 | +[INFO] Worker spent 11.884 seconds performing 20850 jobs (1754.444 jobs/second) |
| 170 | +... |
| 171 | +[INFO] Worker spent 11.851 seconds performing 20950 jobs (1767.750 jobs/second) |
| 172 | +... |
| 173 | +``` |
| 174 | + |
| 175 | +We are now close to 9000 jobs per second, maxing-out the local CPUs again! |
| 176 | + |
| 177 | +From there on, the sky is the limit! You should be able to run thousands of workers accross hundreds of machines before maxing-out a high-performance Redis instance. |
| 178 | + |
| 179 | +Beyond that, using using multiple queues on a [Redis Cluster](https://redis.io/topics/cluster-tutorial) will definitely allow you to run several million jobs per second. If you do, please drop us a line ;-) |
| 180 | + |
| 181 | + |
| 182 | +## Choosing the right kind of queue |
| 183 | + |
| 184 | +### Queue types |
| 185 | + |
| 186 | +With the different settings explored in this tutorial, MRQ allows you to choose how much data you want to store in MongoDB and Redis. |
| 187 | + |
| 188 | +By choosing the right kind of queue for your jobs, you will strike a balance between performance, visibility in the dashboard, and safety guarantees. |
| 189 | + |
| 190 | +Here is a table to sum up the choices: |
| 191 | + |
| 192 | +| **Queue type** | **Regular** | **Raw** | **Raw with no_storage config** | |
| 193 | +|----------------------------------------|-------------|-------------|--------------------------------| |
| 194 | +| **Storage for queued jobs** | MongoDB | Redis | Redis | |
| 195 | +| **Storage for started & success jobs** | MongoDB | MongoDB | None | |
| 196 | +| **Performance** | + | ++ | +++ | |
| 197 | +| **Visibility in the dashboard** | Full | After start | Job counts & failed jobs | |
| 198 | +| **Safety** | +++ | ++ | + | |
| 199 | + |
| 200 | + |
| 201 | + |
| 202 | + |
| 203 | +### Job safety |
| 204 | + |
| 205 | +A regular queue is guaranteed not to lose any jobs once they have been inserted in MongoDB. |
| 206 | + |
| 207 | +A raw queue can lose jobs if the worker abruptly exits in a short time window, between the dequeue from Redis and the insertion in MongoDB. |
| 208 | + |
| 209 | +A raw queue backed by Redis only won't be able to guarantee that a job is finished once it has been dequeued, if the worker abruptly exists. |
| 210 | + |
| 211 | +There are several ways to make raw queues safer. The easiest one is to use a `timed_set` raw queue backed by a Redis ZSET. We'll expand on this in an upcoming tutorial! |
0 commit comments