Find freelancers. Lose those costly delays.

With 1.7 million freelancers, we'll match you with the perfect talent.

or, Register with Elance »

PHP cURL Crawler
Sign in to Add to Watch List

IT & Programming > Web Programming

View Change History

The client has made the following changes to the job.


Job awarded.

Oct 8, 2009

Job Description

  |  Change History >>


  • Posted: Wed, Oct 07, 2009
  • Time Left: Closed
  • Location: Anywhere
  • Client prefers freelancers from:

    You are still able to submit a proposal for this job.

    The client prefers freelancers from
    a different location.

    You're still able to submit a proposal for this job, regardless of your location.
  • Start: Immediately
  • Budget: Less than $500
  • Fixed Price Job
  • Elance Escrow Protection
  • W9 Not Required
Sign in to view client's details

I need an experience cURL programmer to write a highly efficient site crawler with the following features:

Takes a domain as start input
Crawls the domain for every (or optionally limited) page on it, by means of:
-Following internal links
-Reading existing sitemap file if one exists
-Scraping Google for "site:domain.com" hits and check for missed pages from the above steps.

The crawler should store the following to a MySQL database:
Anything from the curl_getinfo function.
Page meta tags and values (ie keywords, description and a few other "standard" tags)
All header tags (H1-H5) contents.
All of the above within reasonable character count limitations.

The crawler must be able to read various character sets and convert to UTF8 for storage, with html entities converted to plain text.

The crawler shold be written in well stuctured and commented PHP code, utilizing the cURL library and make use of it's ability to make multiple simultaneous requests, ie curl_multi_*...

Sign in or Register to see more

Desired Skills
MySQL Administration, PHP, curl
Job ID: 18117339
Avg $ | High $ | Low $ — Show Pricing
  • Submit Date (Latest)
Sign in to Elance and start working on jobs today.
Sign in to view more of the job details and submit a proposal. Once registered, you'll have access to thousands of jobs online or through email.
Are you ready to post a job like this one?
Post a Similar Job »