improved script to update readme and index

Author: adipasquale

README.md

@@ -15,15 +15,15 @@ MRQ was first developed at [Pricing Assistant](http://pricingassistant.com) and
 # Main Features
 
  * **Simple code:** We originally switched from Celery to RQ because Celery's code was incredibly complex and obscure ([Slides](http://www.slideshare.net/sylvinus/why-and-how-pricing-assistant-migrated-from-celery-to-rq-parispy-2)). MRQ should be as easy to understand as RQ and even easier to extend.
- * **Great [dashboard](dashboard.md):** Have visibility and control on everything: queued jobs, current jobs, worker status, ...
+ * **Great [dashboard](http://mrq.readthedocs.org/en/latest/dashboard/):** Have visibility and control on everything: queued jobs, current jobs, worker status, ...
  * **Per-job logs:** Get the log output of each task separately in the dashboard
  * **Gevent worker:** IO-bound tasks can be done in parallel in the same UNIX process for maximum throughput
  * **Supervisord integration:** CPU-bound tasks can be split across several UNIX processes with a single command-line flag
  * **Job management:** You can retry, requeue, cancel jobs from the code or the dashboard.
  * **Performance:** Bulk job queueing, easy job profiling
- * **Easy [configuration](configuration.md):** Every aspect of MRQ is configurable through command-line flags or a configuration file
+ * **Easy [configuration](http://mrq.readthedocs.org/en/latest/configuration):** Every aspect of MRQ is configurable through command-line flags or a configuration file
  * **Job routing:** Like Celery, jobs can have default queues, timeout and ttl values.
- * **Thorough [testing](tests.md):** Edge-cases like worker interrupts, Redis failures, ... are tested inside a Docker container.
+ * **Thorough [testing](http://mrq.readthedocs.org/en/latest/tests):** Edge-cases like worker interrupts, Redis failures, ... are tested inside a Docker container.
  * **Builtin scheduler:** Schedule tasks by interval or by time of the day
  * **Greenlet tracing:** See how much time was spent in each greenlet to debug CPU-intensive jobs.
  * **Integrated memory leak debugger:** Track down jobs leaking memory and find the leaks with objgraph.

docs/dashboard.md

@@ -1,5 +1,4 @@
-Dashboard
-=========
+# Dashboard
 
 A strong focus was put on the tools and particularly the dashboard. After all it is what you will work with most of the time!
 

docs/dependencies.md

@@ -13,7 +13,11 @@ We use LUA scripting in Redis to boost performance and provide extra safety.
 
 You will need [Docker](http://docker.io) to run our unit tests. Our [Dockerfile](https://github.com/pricingassistant/mrq/blob/master/Dockerfile) is actually a good way to see a complete list of dependencies, including dev tools like graphviz for memleak images.
 
-You may want to convert your logs db to a capped collection : ie. run db.runCommand({"convertToCapped": "mrq_jobs", "size": 10737418240})
+You may want to convert your logs db to a capped collection : ie. run db.
+
+```
+runCommand({"convertToCapped": "mrq_jobs", "size": 10737418240})
+```
 
 ## Javascript
 

docs/index.md

@@ -15,15 +15,15 @@ MRQ was first developed at [Pricing Assistant](http://pricingassistant.com) and
 # Main Features
 
  * **Simple code:** We originally switched from Celery to RQ because Celery's code was incredibly complex and obscure ([Slides](http://www.slideshare.net/sylvinus/why-and-how-pricing-assistant-migrated-from-celery-to-rq-parispy-2)). MRQ should be as easy to understand as RQ and even easier to extend.
- * **Great [dashboard](dashboard.md):** Have visibility and control on everything: queued jobs, current jobs, worker status, ...
+ * **Great [dashboard](http://mrq.readthedocs.org/en/latest/dashboard/):** Have visibility and control on everything: queued jobs, current jobs, worker status, ...
  * **Per-job logs:** Get the log output of each task separately in the dashboard
  * **Gevent worker:** IO-bound tasks can be done in parallel in the same UNIX process for maximum throughput
  * **Supervisord integration:** CPU-bound tasks can be split across several UNIX processes with a single command-line flag
  * **Job management:** You can retry, requeue, cancel jobs from the code or the dashboard.
  * **Performance:** Bulk job queueing, easy job profiling
- * **Easy [configuration](configuration.md):** Every aspect of MRQ is configurable through command-line flags or a configuration file
+ * **Easy [configuration](http://mrq.readthedocs.org/en/latest/configuration):** Every aspect of MRQ is configurable through command-line flags or a configuration file
  * **Job routing:** Like Celery, jobs can have default queues, timeout and ttl values.
- * **Thorough [testing](tests.md):** Edge-cases like worker interrupts, Redis failures, ... are tested inside a Docker container.
+ * **Thorough [testing](http://mrq.readthedocs.org/en/latest/tests):** Edge-cases like worker interrupts, Redis failures, ... are tested inside a Docker container.
  * **Builtin scheduler:** Schedule tasks by interval or by time of the day
  * **Greenlet tracing:** See how much time was spent in each greenlet to debug CPU-intensive jobs.
  * **Integrated memory leak debugger:** Track down jobs leaking memory and find the leaks with objgraph.

requirements-dev.txt

@@ -14,3 +14,4 @@ requests==2.4.3
 # pytest-httpbin==0.0.3
 
 mkdocs==0.11.1
+mistune==0.5

scripts/update_readme.py scripts/propagate_docs.pyscripts/update_readme.py renamed to scripts/propagate_docs.py

@@ -1,5 +1,9 @@
 import os
 import re
+import subprocess
+import sys
+import mistune
+
 CURRENT_DIRECTORY = os.path.dirname(os.path.realpath(__file__))
 
 
@@ -8,6 +12,11 @@
       get-started => Get Started
 """
 
+res = subprocess.check_output(["git", "branch"])
+if "* master" not in res:
+  print "you have to be on master to run this !"
+  sys.exit(1)
+
 
 def get_path(filename):
   return os.path.join(CURRENT_DIRECTORY, "..", "docs", "%s.md" % filename)
@@ -35,3 +44,31 @@ def get_path(filename):
 
 with open(readme_path, 'w') as readme_file:
   readme_file.write(readme)
+
+subprocess.call(["git", "add", "-p", "README.md"])
+subprocess.call(["git", "commit"])
+
+# II. index.html
+raw = ""
+for filename in ["index", "dashboard", "get-started"]:
+  with open(get_path(filename), 'r') as f:
+    raw += f.read()
+    raw += "\n\n"
+subprocess.call(["git", "checkout", "gh-pages"])
+html = mistune.markdown(raw)
+
+index_path = os.path.join("index.html")
+
+with open(index_path, 'r') as index_file:
+  index = index_file.read()
+index = re.sub(
+  r"(<div class=\"row marketing\">)(.*)(</div><!-- row marketing -->)",
+  r"\1%s\3" % html,
+  index,
+  flags=re.MULTILINE | re.DOTALL
+)
+with open(index_path, 'w') as index_file:
+  index_file.write(index)
+subprocess.call(["git", "add", "-p", "index.html"])
+subprocess.call(["git", "commit"])
+# subprocess.call(["git", "checkout", "master"])