1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Wat is robots.txt??

Discussion in 'Search Engine Optimization' started by meenka, Feb 22, 2010.

  1. shawn.itchair

    shawn.itchair Guest

    Messages:
    162
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #21
    You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).
     
    shawn.itchair, Feb 23, 2010 IP
  2. alfredwinston

    alfredwinston Peon

    Messages:
    220
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #22
    Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It is helpful to keep out unwanted search engine spiders like email retrievers, image strippers, etc.
     
    alfredwinston, Feb 23, 2010 IP
  3. John2010

    John2010 Guest

    Messages:
    218
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #23
    The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages.
    For those spiders that obey the file, it provides a map of what they can, and cannot index. The file must reside in the root directory of your web. The URL path (web address) of your robots.txt file should look like this...
    www. your site name .com/robots.txt
     
    John2010, Feb 26, 2010 IP
  4. makwarne

    makwarne Peon

    Messages:
    54
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #24
    thanks for sharing your views about robots.txt
     
    makwarne, Feb 26, 2010 IP
  5. rupesh123

    rupesh123 Guest

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #25
    Hello Meenka ,

    Thanks for sharing your knowledge about Robots. It's very useful for me & other users, but you wrote less about it. Can you please explain in details


    Thank you,
    Dewang Jani
     
    rupesh123, Feb 26, 2010 IP
  6. Azher Memon

    Azher Memon Peon

    Messages:
    566
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #26
    Its gateway for search engines where you are giving them permission to crawl website or not
     
    Azher Memon, Feb 26, 2010 IP
  7. kuldep

    kuldep Peon

    Messages:
    400
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #27
    Robot.txt files is a sort of tag used to place in particular site for not indexing it.In simple words its the substitue of "No follow"
     
    kuldep, Feb 26, 2010 IP
  8. Website_Development_NJ

    Website_Development_NJ Guest

    Messages:
    180
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #28
    Hi,

    Robot.txt is working as commander for Search engine spiders, we can allow to search engine spiders through Robot.txt which page we want to crawl or which page we don't want to crawl.

    hope you have gotten what you want.
     
    Website_Development_NJ, Feb 26, 2010 IP
  9. cellhub

    cellhub Guest

    Messages:
    50
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
  10. pratoocool

    pratoocool Active Member

    Messages:
    241
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #30
    My site is showing crawl errors and a20 urls are shown here. Restricted by robots.txt ‎(120)‎. Please guide me that how this can be fixed and if not fixed what is the harm...
     
    pratoocool, Mar 18, 2010 IP
  11. alexjoan

    alexjoan Peon

    Messages:
    114
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #31
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
     
    alexjoan, Mar 19, 2010 IP
  12. asadiyah

    asadiyah Guest

    Messages:
    237
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #32
    maybe this information will explained more detail about robots.txt

     
    asadiyah, Mar 19, 2010 IP
  13. sherrylinx

    sherrylinx Guest

    Messages:
    71
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #33
    robots.txt is a file, resident on web-sites. It contains a list of filenames that should not be searched by search engines such as Google. You should use robots.txt to indicate pages that are not worth indexing - such as pages with a lot of variable data.
     
    sherrylinx, Mar 25, 2010 IP