Hi Guys, We have a 19K pages site, which I disclose later. Although Google shows there are 19,000 pages, it shows ONLY the first 1000. not more. I need to crawl the addresses, i.e the URL of any of the internal pages, includes subdomains, directories .asp pages, .htm pages and .PDFs. Don't crawl the content, as we don't need it. URLs only. It is hosted on a broadband server. Please use off-shelf software. For example: BlackWidow is a good one. scripting especially for this project is time consuming and not bug free. Please apply/bid only if you have done this before. Showing each page's PR - advantage. Thank you