We are currently looking for a web crawler that can fit into our site.
If you know of an open source crawler that will work well then implement it for us.
We currently run a Joomla site but we tore it apart so a lot of it is just custom PHP.
What we need is a crawler module that can turn the site into an aggregate site. The search res...