can any one tell me how can i crawl my website?

can any one tell me how can i crawl my website?

 

You want to make a crawler to crawl your own website, or have another search engine crawl you?

If you want to make your own crawler, it would take a bit of coding or there are lots of places you can get some code to start with.

If you want to be crawled, that's much easier. First thing I would probably do is setup a sitemap. You can find out more at www.sitemaps.org. Then go to the various 'add url' of search engines and submit your site.

 

thx to help me i have one more question
i have created a site in a blogger can u plz help me out how can i create a sitemap?

 

You could use a service like this one: http://www.xml-sitemaps.com/

There was a discussion on WMW the other day about whether site scrapers were taking advantage of XML site maps. The consensus was that they are useful for scrapers, but even without them, a determined thief will still get your data.

 

If you meant how Google could crawl your site, then the answer is create a sitemap. There are resources at Google webmasters to help you. You also need a robots.txt file. Lastly, look for a site which has already been indexed by Google and link from there.

 

If you want to crawl you site and check it, grab a list of urls etc... try Xenu link slooth- google xenu its free to download but...be careful with your server, its heavy on ressources

 

Best is XML sitemap and use google webmaster tools and submit it.