An advanced development tool, Teleport Exec allows you to create and run complex spidering projects directly from your own applications. The program also exposes the entire project database, enabling you to get detailed information about websites directly from the spider.
Some Features :
Reads HTML 4.0 and most of the current extension languages.
Uses true multithreading to retrieve many files simultaneously.
Monitors the responsiveness of each server it contacts, in order to maximize retrieval speed.
Can rewrite HTML links to produce localized versions of web pages.
Provides inline filter functionality.
Uses a tuned, high-performance, multithreaded database.
"Teleport Exec/VLX (Very Large eXploration) ($4995 US) has the same programmatic, scriptable API and project-based design as Teleport Exec but it also has the ability to scan up to 40 million addresses in a single project."