Skip to content

Commit ae7f938

Browse files
adipasqualeadipasquale
committed
improved README and docs, external links and script to sync both
1 parent 906607f commit ae7f938

6 files changed

Lines changed: 229 additions & 18 deletions

File tree

‎README.md‎

Lines changed: 93 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,125 @@
11
# MRQ
22

3-
Mongo Redis Queue - A distributed worker task queue in Python
3+
[MRQ](http://pricingassistant.github.io/mrq) is a distributed task queue for python built on top of mongo, redis and gevent.
44

5-
Full documentation is available on [readthedocs]()
5+
Full documentation is available on [readthedocs](http://mrq.readthedocs.org/en/latest/)
66

77
/!\ MRQ is not yet ready for public use. Soon!
88

99
# Why?
1010

11-
MRQ is an opinionated task queue. It aims to be simple and beautiful like http://python-rq.org while having performance close to http://celeryproject.org
11+
MRQ is an opinionated task queue. It aims to be simple and beautiful like [RQ](http://python-rq.org) while having performances close to [Celery](http://celeryproject.org)
1212

13-
MRQ was first developed at http://pricingassistant.com and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound & CPU-bound, lots of small tasks & a few large ones).
13+
MRQ was first developed at [Pricing Assistant](http://pricingassistant.com) and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound & CPU-bound, lots of small tasks & a few large ones).
1414

15-
The main features of MRQ are:
15+
# Main Features
1616

1717
* **Simple code:** We originally switched from Celery to RQ because Celery's code was incredibly complex and obscure ([Slides](http://www.slideshare.net/sylvinus/why-and-how-pricing-assistant-migrated-from-celery-to-rq-parispy-2)). MRQ should be as easy to understand as RQ and even easier to extend.
18-
* **Great dashboard:** Have visibility and control on everything: queued jobs, current jobs, worker status, ...
18+
* **Great [dashboard](dashboard.md):** Have visibility and control on everything: queued jobs, current jobs, worker status, ...
1919
* **Per-job logs:** Get the log output of each task separately in the dashboard
2020
* **Gevent worker:** IO-bound tasks can be done in parallel in the same UNIX process for maximum throughput
2121
* **Supervisord integration:** CPU-bound tasks can be split across several UNIX processes with a single command-line flag
2222
* **Job management:** You can retry, requeue, cancel jobs from the code or the dashboard.
2323
* **Performance:** Bulk job queueing, easy job profiling
24-
* **Easy configuration:** Every aspect of MRQ is configurable through command-line flags or a configuration file
24+
* **Easy [configuration](configuration.md):** Every aspect of MRQ is configurable through command-line flags or a configuration file
2525
* **Job routing:** Like Celery, jobs can have default queues, timeout and ttl values.
26-
* **Thorough testing:** Edge-cases like worker interrupts, Redis failures, ... are tested inside a Docker container.
26+
* **Thorough [testing](tests.md):** Edge-cases like worker interrupts, Redis failures, ... are tested inside a Docker container.
2727
* **Builtin scheduler:** Schedule tasks by interval or by time of the day
2828
* **Greenlet tracing:** See how much time was spent in each greenlet to debug CPU-intensive jobs.
2929
* **Integrated memory leak debugger:** Track down jobs leaking memory and find the leaks with objgraph.
3030

31-
32-
# Dashboard Screnshots
31+
# Dashboard Screenshots
3332

3433
![Job view](http://i.imgur.com/xaXmrvX.png)
3534

3635
![Worker view](http://i.imgur.com/yYUMCbm.png)
3736

3837
# Get Started
3938

40-
39+
- Make sure you have installed the [dependencies](https://mrq.readthedocs.org/en/latest/dependencies/) : Redis and Mongo
40+
- Install MRQ with `pip install mrq`
41+
- Start a mongo server with `mongod &`
42+
- Start a redis server with `redis-server &`
43+
44+
45+
Create a sample project and write a simple task :
46+
```
47+
$ mkdir test-mrq && cd test-mrq
48+
$ touch __init__.py
49+
$ vim tasks.py
50+
```
51+
52+
53+
from mrq.task import Task
54+
import urllib2
55+
56+
class Fetch(Task):
57+
def run(self, params):
58+
f = urllib2.urlopen(params.get("url"))
59+
t = f.read()
60+
f.close()
61+
return len(t)
62+
63+
You can now run it using `mrq-run` :
64+
```
65+
$ mrq-run tasks.Fetch url http://www.google.com
66+
67+
2014-12-18 15:44:37.869029 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
68+
2014-12-18 15:44:37.880115 [DEBUG] mongodb_jobs: ... connected.
69+
2014-12-18 15:44:37.880305 [DEBUG] Starting tasks.Fetch({'url': 'http://www.google.com'})
70+
2014-12-18 15:44:38.158572 [DEBUG] Job None success: 0.278229s total
71+
17655
72+
```
73+
74+
You can also enqueue a few tasks with
75+
```
76+
$ mrq-run --async --queue fetches tasks.Fetch url http://www.google.com &&
77+
mrq-run --async --queue fetches tasks.Fetch url http://www.yahoo.com &&
78+
mrq-run --async --queue fetches tasks.Fetch url http://www.wordpress.com
79+
80+
2014-12-18 15:49:05.688627 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
81+
2014-12-18 15:49:05.705400 [DEBUG] mongodb_jobs: ... connected.
82+
2014-12-18 15:49:05.729364 [INFO] redis: Connecting to Redis at 127.0.0.1...
83+
5492f771520d1887bfdf4b0f
84+
2014-12-18 15:49:05.957912 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
85+
2014-12-18 15:49:05.967419 [DEBUG] mongodb_jobs: ... connected.
86+
2014-12-18 15:49:05.983925 [INFO] redis: Connecting to Redis at 127.0.0.1...
87+
5492f771520d1887c2d7d2db
88+
2014-12-18 15:49:06.182351 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
89+
2014-12-18 15:49:06.193314 [DEBUG] mongodb_jobs: ... connected.
90+
2014-12-18 15:49:06.209336 [INFO] redis: Connecting to Redis at 127.0.0.1...
91+
5492f772520d1887c5b32881
92+
```
93+
94+
Now start the dasbhoard with `mrq-dashboard &` and go check your newly created queue and job on [localhost:5555](http://localhost:5555/#jobs)
95+
96+
Instanciate a worker with `mrq-worker` and you can follow it on the dashboard as it executes in parallel all the enqueued jobs
97+
98+
```
99+
$ mrq-worker --gevent 10 fetches
100+
101+
2014-12-18 15:52:57.362209 [INFO] Starting Gevent pool with 10 worker greenlets (+ report, logs, adminhttp)
102+
2014-12-18 15:52:57.388033 [INFO] redis: Connecting to Redis at 127.0.0.1...
103+
2014-12-18 15:52:57.389488 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
104+
2014-12-18 15:52:57.390996 [DEBUG] mongodb_jobs: ... connected.
105+
2014-12-18 15:52:57.391336 [DEBUG] mongodb_logs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
106+
2014-12-18 15:52:57.392430 [DEBUG] mongodb_logs: ... connected.
107+
2014-12-18 15:52:57.523329 [INFO] Fetching 10 jobs from ['fetches']
108+
2014-12-18 15:52:57.537570 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.google.com'})
109+
2014-12-18 15:52:57.567311 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.google.com'})
110+
2014-12-18 15:52:57.567747 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.yahoo.com'})
111+
2014-12-18 15:52:57.568080 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.wordpress.com'})
112+
2014-12-18 15:52:57.798167 [INFO] Fetching 6 jobs from ['fetches']
113+
2014-12-18 15:52:58.432574 [INFO] Fetching 6 jobs from ['fetches']
114+
2014-12-18 15:52:58.670492 [DEBUG] Job 5492f771520d1887bfdf4b0f success: 1.135268s total
115+
2014-12-18 15:52:59.344227 [DEBUG] Job 5492f74b520d1887a38dd7c8 success: 1.816439s total
116+
2014-12-18 15:52:59.912086 [INFO] Fetching 8 jobs from ['fetches']
117+
2014-12-18 15:53:00.685727 [DEBUG] Job 5492f772520d1887c5b32881 success: 3.149119s total
118+
2014-12-18 15:53:01.578981 [INFO] Fetching 9 jobs from ['fetches']
119+
2014-12-18 15:53:01.897873 [DEBUG] Job 5492f771520d1887c2d7d2db success: 4.361895s total
120+
2014-12-18 15:53:03.415555 [INFO] Fetching 10 jobs from ['fetches']
121+
```
41122

42123
# More
43124

125+
Full documentation is available on [readthedocs](http://mrq.readthedocs.org/en/latest/)

‎docs/configuration.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
# Configuration
22

3-
Check all the [available config options](mrq/config.py)
4-
53
For each of these values, configuration is loaded in this order by default:
4+
65
- Command-line arguments (`mrq-worker --redis=redis://127.0.0.1:6379`)
76
- Environment variables prefixed by MRQ_ (`MRQ_REDIS=redis://127.0.0.1:6379 mrq-worker`)
87
- Python variables in a config file, by default `mrq-config.py` (`REDIS="redis://127.0.0.1:6379"` in this file)

‎docs/dependencies.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,14 @@
22

33
## Python
44

5-
MRQ has only been tested with Python 2.7+. Required external services dependencies are MongoDB >= 2.4 and Redis >= 2.6 (we use LUA scripting to boost performance and provide extra safety).
5+
MRQ has only been tested with Python 2.7+.
6+
7+
Required external services dependencies are
8+
9+
- [MongoDB >= 2.4](http://docs.mongodb.org/manual/installation/)
10+
- [Redis >= 2.6](http://redis.io/topics/quickstart)
11+
12+
We use LUA scripting in Redis to boost performance and provide extra safety.
613

714
You will need [Docker](http://docker.io) to run our unit tests. Our [Dockerfile](https://github.com/pricingassistant/mrq/blob/master/Dockerfile) is actually a good way to see a complete list of dependencies, including dev tools like graphviz for memleak images.
815

‎docs/get-started.md‎

Lines changed: 83 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,85 @@
11
# Get started
22

3-
Test
3+
- Make sure you have installed the [dependencies](https://mrq.readthedocs.org/en/latest/dependencies/) : Redis and Mongo
4+
- Install MRQ with `pip install mrq`
5+
- Start a mongo server with `mongod &`
6+
- Start a redis server with `redis-server &`
7+
8+
9+
Create a sample project and write a simple task :
10+
```
11+
$ mkdir test-mrq && cd test-mrq
12+
$ touch __init__.py
13+
$ vim tasks.py
14+
```
15+
16+
17+
from mrq.task import Task
18+
import urllib2
19+
20+
class Fetch(Task):
21+
def run(self, params):
22+
f = urllib2.urlopen(params.get("url"))
23+
t = f.read()
24+
f.close()
25+
return len(t)
26+
27+
You can now run it using `mrq-run` :
28+
```
29+
$ mrq-run tasks.Fetch url http://www.google.com
30+
31+
2014-12-18 15:44:37.869029 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
32+
2014-12-18 15:44:37.880115 [DEBUG] mongodb_jobs: ... connected.
33+
2014-12-18 15:44:37.880305 [DEBUG] Starting tasks.Fetch({'url': 'http://www.google.com'})
34+
2014-12-18 15:44:38.158572 [DEBUG] Job None success: 0.278229s total
35+
17655
36+
```
37+
38+
You can also enqueue a few tasks with
39+
```
40+
$ mrq-run --async --queue fetches tasks.Fetch url http://www.google.com &&
41+
mrq-run --async --queue fetches tasks.Fetch url http://www.yahoo.com &&
42+
mrq-run --async --queue fetches tasks.Fetch url http://www.wordpress.com
43+
44+
2014-12-18 15:49:05.688627 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
45+
2014-12-18 15:49:05.705400 [DEBUG] mongodb_jobs: ... connected.
46+
2014-12-18 15:49:05.729364 [INFO] redis: Connecting to Redis at 127.0.0.1...
47+
5492f771520d1887bfdf4b0f
48+
2014-12-18 15:49:05.957912 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
49+
2014-12-18 15:49:05.967419 [DEBUG] mongodb_jobs: ... connected.
50+
2014-12-18 15:49:05.983925 [INFO] redis: Connecting to Redis at 127.0.0.1...
51+
5492f771520d1887c2d7d2db
52+
2014-12-18 15:49:06.182351 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
53+
2014-12-18 15:49:06.193314 [DEBUG] mongodb_jobs: ... connected.
54+
2014-12-18 15:49:06.209336 [INFO] redis: Connecting to Redis at 127.0.0.1...
55+
5492f772520d1887c5b32881
56+
```
57+
58+
Now start the dasbhoard with `mrq-dashboard &` and go check your newly created queue and job on [localhost:5555](http://localhost:5555/#jobs)
59+
60+
Instanciate a worker with `mrq-worker` and you can follow it on the dashboard as it executes in parallel all the enqueued jobs
61+
62+
```
63+
$ mrq-worker --gevent 10 fetches
64+
65+
2014-12-18 15:52:57.362209 [INFO] Starting Gevent pool with 10 worker greenlets (+ report, logs, adminhttp)
66+
2014-12-18 15:52:57.388033 [INFO] redis: Connecting to Redis at 127.0.0.1...
67+
2014-12-18 15:52:57.389488 [DEBUG] mongodb_jobs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
68+
2014-12-18 15:52:57.390996 [DEBUG] mongodb_jobs: ... connected.
69+
2014-12-18 15:52:57.391336 [DEBUG] mongodb_logs: Connecting to MongoDB at 127.0.0.1:27017/mrq...
70+
2014-12-18 15:52:57.392430 [DEBUG] mongodb_logs: ... connected.
71+
2014-12-18 15:52:57.523329 [INFO] Fetching 10 jobs from ['fetches']
72+
2014-12-18 15:52:57.537570 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.google.com'})
73+
2014-12-18 15:52:57.567311 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.google.com'})
74+
2014-12-18 15:52:57.567747 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.yahoo.com'})
75+
2014-12-18 15:52:57.568080 [DEBUG] Starting tasks.Fetch({u'url': u'http://www.wordpress.com'})
76+
2014-12-18 15:52:57.798167 [INFO] Fetching 6 jobs from ['fetches']
77+
2014-12-18 15:52:58.432574 [INFO] Fetching 6 jobs from ['fetches']
78+
2014-12-18 15:52:58.670492 [DEBUG] Job 5492f771520d1887bfdf4b0f success: 1.135268s total
79+
2014-12-18 15:52:59.344227 [DEBUG] Job 5492f74b520d1887a38dd7c8 success: 1.816439s total
80+
2014-12-18 15:52:59.912086 [INFO] Fetching 8 jobs from ['fetches']
81+
2014-12-18 15:53:00.685727 [DEBUG] Job 5492f772520d1887c5b32881 success: 3.149119s total
82+
2014-12-18 15:53:01.578981 [INFO] Fetching 9 jobs from ['fetches']
83+
2014-12-18 15:53:01.897873 [DEBUG] Job 5492f771520d1887c2d7d2db success: 4.361895s total
84+
2014-12-18 15:53:03.415555 [INFO] Fetching 10 jobs from ['fetches']
85+
```

‎docs/index.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,17 @@
22

33
[MRQ](http://pricingassistant.github.io/mrq) is a distributed task queue for python built on top of mongo, redis and gevent.
44

5+
Full documentation is available on [readthedocs](http://mrq.readthedocs.org/en/latest/)
6+
7+
/!\ MRQ is not yet ready for public use. Soon!
8+
59
# Why?
610

7-
MRQ is an opinionated task queue. It aims to be simple and beautiful like http://python-rq.org while having performance close to http://celeryproject.org
11+
MRQ is an opinionated task queue. It aims to be simple and beautiful like [RQ](http://python-rq.org) while having performances close to [Celery](http://celeryproject.org)
812

9-
MRQ was first developed at http://pricingassistant.com and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound & CPU-bound, lots of small tasks & a few large ones).
13+
MRQ was first developed at [Pricing Assistant](http://pricingassistant.com) and its initial feature set matches the needs of worker queues with heterogenous jobs (IO-bound & CPU-bound, lots of small tasks & a few large ones).
1014

11-
The main features of MRQ are:
15+
# Main Features
1216

1317
* **Simple code:** We originally switched from Celery to RQ because Celery's code was incredibly complex and obscure ([Slides](http://www.slideshare.net/sylvinus/why-and-how-pricing-assistant-migrated-from-celery-to-rq-parispy-2)). MRQ should be as easy to understand as RQ and even easier to extend.
1418
* **Great [dashboard](dashboard.md):** Have visibility and control on everything: queued jobs, current jobs, worker status, ...

‎scripts/update_readme.py‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
import os
2+
import re
3+
CURRENT_DIRECTORY = os.path.dirname(os.path.realpath(__file__))
4+
5+
6+
""" This script overrides the README with parts contained in the docs :
7+
index.md => Intro, Why, Main Features
8+
get-started => Get Started
9+
"""
10+
11+
12+
def get_path(filename):
13+
return os.path.join(CURRENT_DIRECTORY, "..", "docs", "%s.md" % filename)
14+
15+
readme_path = os.path.join(CURRENT_DIRECTORY, "..", "README.md")
16+
17+
with open(readme_path, 'r') as readme_file:
18+
readme = readme_file.read()
19+
20+
for filename, title_begin, title_end in (
21+
("index", "MRQ", "Dashboard Screenshots"),
22+
("get-started", "Get Started", "More"),
23+
):
24+
with open(get_path(filename), 'r') as f:
25+
raw = f.read()
26+
# remove first 2 lines
27+
cropped = "\n".join(raw.split("\n")[2:])
28+
29+
readme = re.sub(
30+
"(# %s\n\n)(.*)(\n\n^# %s)" % (title_begin, title_end),
31+
r"\1%s\3" % cropped,
32+
readme,
33+
flags=re.MULTILINE | re.DOTALL
34+
)
35+
36+
with open(readme_path, 'w') as readme_file:
37+
readme_file.write(readme)

0 commit comments

Comments
 (0)