Simple Web Crawler with MRQ
===========================

This is a simple demo app that crawls a website, to demo some MRQ features.


How to use
==========

First, get into a Python virtualenv (`make virtualenv`) or into the docker image at the root of this directory (`make ssh`)

Then install MRQ and the packages needed for this example:
```
$ cd examples/simple_crawler
$ sudo apt-get install libxml2-dev libxslt1-dev libz-dev     # Needed for lxml
$ pip install -r requirements.txt
```

Launch MongoDB & Redis if they are not already started:
```
$ mongod &
$ redis-server &
```

Queue the first task via the command line:

```
$ mrq-run --queue crawl crawler.Fetch '{"url": "http://docs.python-requests.org/"}'
```

Then start a worker with 3 (or more!) greenlets:

```
$ mrq-worker crawl --greenlets 3
```

You should also launch a dashboard to monitor the progress:
```
$ mrq-dashboard
```

We also included 2 utility tasks:
```
$ mrq-run crawler.Report
$ mrq-run crawler.Reset
```

This is obviously a very simple crawler, production systems will be much more complex but it gives you an overview of MRQ and a good starting point.


Expected result for crawler.Report
==================================

As of 2015-02-20:

```
Crawl stats
===========
URLs queued: 26
URLs successfully crawled: 23
URLs redirected: 1
Bytes fetched: 608131
```