Site Search

Discussion in 'Site & Server Administration' started by samo1215, Jun 24, 2009.

  1. #1
    I am in the process of setting up my first website and wanted to start uploading my pages to a server soon. However, I wanted to test them on the server and make sure that everyone involved with the project was happy before any search engines crawl the page.

    My question is, if I don't submit the page to Google, Yahoo and other search engines do I need to worry about them crawling my page or do I need to setup an htcacess file to keep them at bay?
     
    samo1215, Jun 24, 2009 IP
  2. kailash

    kailash Well-Known Member

    Messages:
    1,248
    Likes Received:
    42
    Best Answers:
    0
    Trophy Points:
    190
    #2
    Even if you do not submit your site to search engines, they may start crawling your web site. Use robots.txt to disallow search engine spiders.
     
    kailash, Jun 24, 2009 IP