1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

what is the code to write robots.txt files ?

Discussion in 'robots.txt' started by sandegownc, Nov 21, 2012.

0
  1. #1
    As we all knew about robots.txt file , which is a text file we include to instruct the search engine to crawl our webpages.

    So, how to write , create and what code is required to generate this file.
    All suggestions would be very welcome !!
     
    sandegownc, Nov 21, 2012 IP
  2. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #2
    No, it's not required to tell it to crawl. It's required to tell it where to not crawl.

    You're unclear about what you want. You should read http://www.robotstxt.org/ and learn how to do this. It's not difficult, even someone stupid can do it.
     
    ryan_uk, Nov 23, 2012 IP
  3. kashishjain

    kashishjain Peon

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    creating and using robots.txt depends only on your requirements.
    It is created to restrict crawlers from accessing certain parts of your blog which you dont want in search engines.
     
    kashishjain, Nov 24, 2012 IP