splitted README into proper doc generated with mkdocs

Author: adipasquale

README.md

@@ -1,12 +1,12 @@
-MRQ
-===
+# MRQ
 
 Mongo Redis Queue - A distributed worker task queue in Python
 
+Full documentation is available on [readthedocs]()
+
 /!\ MRQ is not yet ready for public use. Soon!
 
-Why?
-====
+# Why?
 
 MRQ is an opinionated task queue. It aims to be simple and beautiful like http://python-rq.org while having performance close to http://celeryproject.org
 
@@ -29,299 +29,15 @@ The main features of MRQ are:
  * **Integrated memory leak debugger:** Track down jobs leaking memory and find the leaks with objgraph.
 
 
-
-Dashboard
-=========
-
-A strong focus was put on the tools and particularly the dashboard. After all it is what you will work with most of the time!
+# Dashboard Screnshots
 
 ![Job view](http://i.imgur.com/xaXmrvX.png)
 
 ![Worker view](http://i.imgur.com/yYUMCbm.png)
 
-There are too much features on the dashboard to list, but the goal is to have complete visibility and control over what your workers are doing!
-
-
-Design
-======
-
-A talk+slides about MRQ's design is upcoming.
-
-A couple things to know:
-- We use Redis as a main queue for task IDs
-- We store metadata on the tasks in MongoDB so they can be browsable and managed more easily.
-
-
-
-Performance
-===========
-
-On a MacbookPro, we see 1300 jobs/second in a single worker process with very simple jobs that store results, to measure the overhead of MRQ. However what we are really measuring there is MongoDB's write performance.
-
-
-Install & dependencies
-======================
-
-MRQ can be installed via PIP:
-
-```pip install mrq```
-
-MRQ has only been tested with Python 2.7+. External service dependencies are MongoDB >= 2.4 and Redis >= 2.6 (we use LUA scripting to boost performance and provide extra safety).
-
-You will need [Docker](http://docker.io) to run our unit tests. Our [Dockerfile](https://github.com/pricingassistant/mrq/blob/master/Dockerfile) is actually a good way to see a complete list of dependencies, including dev tools like graphviz for memleak images.
-
-You may want to convert your logs db to a capped collection : ie. run db.runCommand({"convertToCapped": "mrq_jobs", "size": 10737418240})
-
-
-Configuration
-=============
-
-Check all the [available config options](mrq/config.py)
-
-For each of these values, configuration is loaded in this order by default:
-- Command-line arguments (`mrq-worker --redis=redis://127.0.0.1:6379`)
-- Environment variables prefixed by MRQ_ (`MRQ_REDIS=redis://127.0.0.1:6379 mrq-worker`)
-- Python variables in a config file, by default `mrq-config.py` (`REDIS="redis://127.0.0.1:6379"` in this file)
-
-Most of the time, you want to set all your configuration in a `mrq-config.py` file in the directory where you will launch your workers, and override some of it from the command line.
-
-On Heroku, environment variables are very handy because they can be set like `heroku config:set MRQ_REDIS=redis://127.0.0.1:6379`
-
-
-Command-line use
-================
-
-All the command-line tools support a set of common configuration flags, defined in [config.py](https://github.com/pricingassistant/mrq/blob/master/mrq/config.py). Use --help with any of them to see the full list.
-
- - `mrq-worker` starts a worker
- - `mrq-dashboard` starts the web dashboard on the default port
- - `mrq-run` runs a task. If you add the `--async` option that will enqueue it to be later ran by a worker
-
-Typical usage is:
-```
-$ mrq-run tasks.mylib.myfile.MyTask '{"param1": 1, "param2": True}'
-```
-
-
-Job maintenance
-===============
-
-MRQ can provide strong guarantees that no job will be lost in the middle of a worker restart, database disconnect, etc...
-
-To do that, you should add these recurring scheduled jobs to your mrq-config.py:
-
-```
-SCHEDULER_TASKS = [
-
-  # This will requeue jobs marked as interrupted, for instance when a worker received SIGTERM
-  {
-    "path": "mrq.basetasks.cleaning.RequeueInterruptedJobs",
-    "params": {},
-    "interval": 5 * 60
-  },
-
-  # This will requeue jobs marked as started for a long time (more than their own timeout)
-  # They can exist if a worker was killed with SIGKILL and not given any time to mark
-  # its current jobs as interrupted.
-  {
-    "path": "mrq.basetasks.cleaning.RequeueStartedJobs",
-    "params": {},
-    "interval": 3600
-  },
-
-  # This will requeue jobs 'lost' between redis.blpop() and mongo.update(status=started).
-  # This can happen only when the worker is killed brutally in the middle of dequeue_jobs()
-  {
-    "path": "mrq.basetasks.cleaning.RequeueLostJobs",
-    "params": {},
-    "interval": 24 * 3600
-  }
-]
-```
-
-Obviously this implies that all your jobs should be *idempotent*, meaning that they could be done multiple times, maybe partially, without breaking your app. This is a very good design to enforce for your whole task queue, though you can still manage locks yourself in your code that make sure a block of code will only run once.
-
-
-Raw queues
-==========
-
-With regular queues, MRQ stores the task metadata in MongoDB and the task IDs in a Redis list. This design allows a good compromise between performance and visibility.
-
-Raw queues give you more performance and some new features in exchange for a bit less visibility. In their case, only the parameters of a task are stored in serialized form in Redis when queued, and they are inserted in MongoDB only after being dequeued by a worker.
-
-There are 4 types of raw queues. The type of a queue is determined by a suffix in its name:
-
- * ```_raw``` : The simpliest raw queue, stored in a Redis LIST.
- * ```_set``` : Stored in a Redis SET. Gives you the ability to have "unique" tasks: only one (task, parameters) couple can be queued at a time.
- * ```_sorted_set``` : The most powerful MRQ queue type, stored in a Redis ZSET. Allows you to order (and re-order) the tasks to be dequeued. Like ```_set```, task parameters will be unique.
- * ```_timed_set``` : A special case of ```_sorted_set```, where tasks are sorted with a UNIX timestamp. This means you can schedule tasks to be executed at a precise time in the future.
-
-Raw queues need a special entry in the configuration to unserialize their "raw" parameters in a regular dict of parameters. They also need to be linked to a regular queue ("default" if none) for providing visibility and retries, after they are dequeued from the raw queue.
-
-This is an example of raw queue configuration:
-
-```python
-RAW_QUEUES = {
-  "myqueue_raw": {
-      "job_factory": lambda rawparam: {
-          "path": "tests.tasks.general.Add",
-          "params": {
-              "a": int(rawparam.split(" ")[0]),
-              "b": int(rawparam.split(" ")[1])
-          }
-      },
-      "retry_queue": "high"
-  }
-}
-```
-
-This task adds two integers. To queue tasks, you can do from the code:
-
-```python
-from mrq.queue import send_raw_tasks
-
-send_raw_tasks("myqueue_raw", [
-  ["1 1"],
-  ["42 8"]
-])
-```
-
-To run them, start a worker listening to both queues:
-
-```
-$ mrq-worker high myqueue_raw
-```
-
-Queueing on timed sets is a bit different, you can pass unix timestamps directly:
-
-```python
-from mrq.queue import send_raw_tasks
-import time
-
-send_raw_tasks("myqueue_timed_set", {
-  "rawparam_xxx": time.time(),
-  "rawparam_yyy": time.time() + 3600  # Do this in an hour
-})
-```
-
-For more examples of raw queue configuration, check https://github.com/pricingassistant/mrq/blob/master/tests/fixtures/config-raw1.py
-
-
-Hunting memory leaks
-====================
-
-Memory leaks can be a big issue with gevent workers because several tasks share the same python process.
-
-Thankfully, MRQ provides tools to track down such issues. Memory usage of each worker is graphed in the dashboard and makes it easy to see if memory leaks are happening.
-
-When a worker has a steadily growing memory usage, here are the steps to find the leak:
-
- * Check which jobs are running on this worker and try to isolate which of them is leaking and on which queue
- * Start a dedicated worker with ```--trace_memory --gevent 1``` on the same queue : This will start a worker doing one job at a time with memory profiling enabled. After each job you should see a report of leaked object types.
- * Find the most unique type in the list (usually not 'list' or 'dict') and restart the worker with ```--trace_memory --gevent 1 --trace_memory_type=XXX --trace_memory_output_dir=memdbg``` (after creating the directory memdbg).
- * There you will find a graph for each task generated by [objgraph](https://mg.pov.lt/objgraph/) which is incredibly helpful to track down the leak.
-
-
-Simulating network latency
-==========================
-
-Sometimes it is helpful in local development to simulate an environment with higher network latency.
-
-To do this we added a ```--add_network_latency=0.1``` config option that will add (in this case) a random delay between 0 and 0.1 seconds to every network call.
-
-
-Worker concurrency
-==================
-
-The default is to run tasks one at a time. You should obviously change this behaviour to use Gevent's full capabilities with something like:
-
-`mrq-worker --processes 3 --gevent 10`
-
-This will start 30 greenlets over 3 UNIX processes. Each of them will run 10 jobs at the same time.
-
-As soon as you use the `--processes` option (even with `--processes=1`) then supervisord will be used to control the processes. It is quite useful to manage long-running instances.
-
-On Heroku's 512M dynos, we have found that for IO-bound jobs, `--processes 4 --gevent 30` may be a good setting.
-
-
-Metrics & Graphite
-==================
-
-MRQ doesn't support sending metrics to Graphite out of the box but makes it extremely easy to do so.
-
-All you have to do is add this hook in your mrq-config file:
-
-```
-
-# Install this via pip
-import graphiteudp
-
-# Initialize the Graphite UDP Client
-_graphite_client = graphiteudp.GraphiteUDPClient(host, port, prefix, debug=False)
-_graphite_client.init()
-
-def METRIC_HOOK(name, incr=1, **kwargs):
-
-  # You can use this to avoid sending too many different metrics
-  whitelisted_metrics = ["queues.all.", "queues.default.", "jobs."]
-
-  if any([name.startswith(m) for m in whitelisted_metrics]):
-    _graphite_client.send(name, incr)
-
-
-```
-
-If you have another monitoring system you can plug anything in this hook to connect to it!
-
-
-Tests
-=====
-
-Testing is done inside a Docker container for maximum repeatability.
-We don't use Travis-CI or friends because we need to be able to kill our process dependencies (MongoDB, Redis, ...) on demand.
-
-Therefore you need to ([install docker](https://www.docker.io/gettingstarted/#h_installation)) to run the tests.
-If you're not on an os that supports natively docker, don't forget to start up your VM and ssh into it.
-
-```
-$ make test
-```
-
-You can also open a shell inside the docker (just like you would enter in a virtualenv) with:
-
-```
-$ make docker (if it wasn't build before)
-$ make ssh
-```
-
-
-PyPy
-====
-
-Earlier in its development MRQ was tested successfully on PyPy but we are waiting for better PyPy+gevent support to continue working on it, as performance was worse than CPython.
-
-
-Useful third-party utils
-========================
-
-* http://superlance.readthedocs.org/en/latest/
-
+# Get Started
 
-Credits
-=======
 
-Inspirations:
- * RQ
- * Celery
 
-JS libraries used in the Dashboard:
- * http://backbonejs.org
- * http://underscorejs.org
- * http://requirejs.org
- * http://momentjs.com
- * http://jquery.com
- * http://datatables.net
- * https://github.com/Jowin/Datatables-Bootstrap3/
- * https://github.com/twbs/bootstrap
+# More
 
-... as well as all the Python modules in requirements.txt!

docs/cli/index.md

@@ -0,0 +1,13 @@
+
+# Commands
+
+All the command-line tools support a set of common configuration flags, defined in [config.py](https://github.com/pricingassistant/mrq/blob/master/mrq/config.py). Use --help with any of them to see the full list.
+
+ - `mrq-worker` starts a worker
+ - `mrq-dashboard` starts the web dashboard on the default port
+ - `mrq-run` runs a task. If you add the `--async` option that will enqueue it to be later ran by a worker
+
+Typical usage is:
+```
+$ mrq-run tasks.mylib.myfile.MyTask '{"param1": 1, "param2": True}'
+```

docs/cli/mrq-dashboard.md

@@ -0,0 +1 @@
+# mrq-dashboard

docs/cli/mrq-run.md

@@ -0,0 +1 @@
+# mrq-run

docs/cli/mrq-worker.md

@@ -0,0 +1,18 @@
+# mrq-worker
+
+## Concurrency
+
+The default is to run tasks one at a time. You should obviously change this behaviour to use Gevent's full capabilities with something like:
+
+`mrq-worker --processes 3 --gevent 10`
+
+This will start 30 greenlets over 3 UNIX processes. Each of them will run 10 jobs at the same time.
+
+As soon as you use the `--processes` option (even with `--processes=1`) then supervisord will be used to control the processes. It is quite useful to manage long-running instances.
+
+
+## Simulating network latency
+
+Sometimes it is helpful in local development to simulate an environment with higher network latency.
+
+To do this we added a ```--add_network_latency=0.1``` config option that will add (in this case) a random delay between 0 and 0.1 seconds to every network call.

docs/configuration.md

@@ -0,0 +1,12 @@
+# Configuration
+
+Check all the [available config options](mrq/config.py)
+
+For each of these values, configuration is loaded in this order by default:
+- Command-line arguments (`mrq-worker --redis=redis://127.0.0.1:6379`)
+- Environment variables prefixed by MRQ_ (`MRQ_REDIS=redis://127.0.0.1:6379 mrq-worker`)
+- Python variables in a config file, by default `mrq-config.py` (`REDIS="redis://127.0.0.1:6379"` in this file)
+
+Most of the time, you want to set all your configuration in a `mrq-config.py` file in the directory where you will launch your workers, and override some of it from the command line.
+
+On Heroku, environment variables are very handy because they can be set like `heroku config:set MRQ_REDIS=redis://127.0.0.1:6379`

docs/dashboard.md

@@ -0,0 +1,11 @@
+Dashboard
+=========
+
+A strong focus was put on the tools and particularly the dashboard. After all it is what you will work with most of the time!
+
+![Queues view](http://i.imgur.com/H2nsgq2.png)
+
+![Jobs view](http://i.imgur.com/SNsYhuQ.png)
+
+
+There are too many features on the dashboard to list, but the goal is to have complete visibility and control over what your workers are doing!