Find freelancers. Lose those costly delays.

With 1.7 million freelancers, we'll match you with the perfect talent.

or, Register with Elance »

Clojure Crawler
Sign in to Add to Watch List

IT & Programming > Other IT & Programming

View Change History

The client has made the following changes to the job.


Job awarded.

Mar 15, 2013

Job Description

  |  Change History >>


  • Posted: Wed, Mar 13, 2013
  • Time Left: Closed
  • Location: Anywhere
  • Client prefers freelancers from:

    You are still able to submit a proposal for this job.

    The client prefers freelancers from
    a different location.

    You're still able to submit a proposal for this job, regardless of your location.
  • Start: Immediately
  • Hourly Rate: Not Sure
  • Hrs/wk: 20 | Duration: 1-2 weeks
  • Work View™ Payment Protection
  • U.S. freelancers must have W9
Sign in to view client's details

I'm looking for a software professional to build a Clojure application that crawls URLs and caches to Amazon S3.

This is a great opportunity for a talented developer to build a simple, powerful web crawler in a modern language (Clojure) using great decentralized storage tools (Amazon S3, Riak). I hope that I find a great contractor who wants to work on an ongoing basis.

## Application Requirements:

1. Use Amazon S3 an an external file store.

2. Use Riak as an external working database.

3. Be responsible; (a) respect `robots.txt`; (b) throttle access based on domain; (c) use a configurable user agent string.

4. There is no GUI, only a Clojure API.

5. The API to add a URL to crawl:

(add-crawl-url {:url "  [obscured]  -url.co/stuff.html" :priority 5})

This sets the one or more initial URLs. It can also add a new URL if the crawl is already underway. URLs with higher priority get processed first.

6. The API to run the crawl is:


This instructs ...

Sign in or Register to see more

Job ID: 38989146
Hourly Rate: Avg $ | High $ | Low $ — Show Pricing
  • Submit Date (Latest)

 Italy  |  
I am developer, I enjoy most clojure but I love to work with python too.
 0.0   |  $0 Earnings   |  0 Jobs
Bid ID: 39082061  |  Submitted: Mar 15, 2013 12:51 ET 
Proposal SEALED

TokenMill is a small linguistic engineering company focusing on providing text analytics and vertical search solutions.
 0.0   |  Private   |  0 Jobs
Bid ID: 39013432  |  Submitted: Mar 14, 2013 02:44 ET 
Proposal SEALED

 Netherlands  |  
I'm an experienced software developer from the Netherlands. My two main specialties are search (e.g. web crawling, Apache Lucene) and building web...
 0.0   |  $0 Earnings   |  0 Jobs
Bid ID: 39001414  |  Submitted: Mar 13, 2013 19:17 ET 
Elance is now an Upwork company.
Upwork is the choice of 4M+ clients. Get started working on Upwork today.
Are you ready to post a job like this one?
Post a Similar Job »