about robots.txt file

Discussion in 'Search Engine Optimization' started by christinabob, Jul 30, 2007.

Thread Status:
Not open for further replies.
0
  1. #1
    I found this text in robots.txt file
    what it mean

    User-agent: *
    Disallow: /search


    what it means disallow: /search

    please explain in details:)
     
    christinabob, Jul 30, 2007 IP
  2. hasbehas

    hasbehas Well-Known Member

    Messages:
    726
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    190
    #2
    it tells the to the search engine bot (in this case, its any bot, (*) means that)
    to not to crawl or index the directory names as search

    if you were to say ;
    User-agent: *
    Disallow: copyright.htm
    Disallow: terms.php

    same restriction for the files copyright.htm and terms.php

    Also single slash (/) means the root directory.
     
    hasbehas, Jul 30, 2007 IP
  3. christinabob

    christinabob Banned

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    this robots.txt file find at blogger.com

    User-agent: *
    Disallow: /search

    disallow: /search what it means at blogger.com
     
    christinabob, Jul 30, 2007 IP
  4. qazu

    qazu Well-Known Member

    Messages:
    1,834
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    165
    #4
    It means any files in the search folder should not be crawled or indexed
     
    qazu, Jul 30, 2007 IP
  5. Ravi yadav

    Ravi yadav Guest

    Messages:
    31
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    it means google will not crawl search folder
     
    Ravi yadav, Jul 30, 2007 IP
  6. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,032
    Likes Received:
    437
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Dan Schulz, Jul 30, 2007 IP
  7. christinabob

    christinabob Banned

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #7
    yes but its robots.txt file for blog on blogspot.com

    soo my question is which post or part of blog reside in search folder.
     
    christinabob, Jul 30, 2007 IP
  8. webdesignerindia

    webdesignerindia Peon

    Messages:
    53
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    that means in your blog there is a topic starts with search it will not be crawl or cache by search engines....

    usually when we want to block any folder or file of our website

    we use

    User-agent : *
    Disallow :

    User-agent : *

    * denotes for welcoming every search engines

    If we left the disallow: it means search engines should to crawl each folder n files on the site and if we put like this
    User-agent: *
    disallow: /

    it means we are blocking our website files completely

    and if we put like this

    User-agent: *
    Disallow: /search

    That means we are blocking those files n folder which starts from "search"

    And to block any particular folder

    we put like this

    User-agent: *
    Disallow: /search/

    And to block any particular file, we put like this

    User-agent: *
    Disallow: /filename.html or /filename.php or /filename.asp etc

    I hope this help you lot n lot....


    regards,
    Website Designer
     
    webdesignerindia, Jul 30, 2007 IP
  9. christinabob

    christinabob Banned

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #9
    thank you, abhar,sukariya webdesigner inda

    but i just want to know which part of blog on blogspot reside in search folder.

    as robots.txt disallow crawling of /search folder.
    I am asking about robots.txt file for blogspot point of view only.

    please answer
     
    christinabob, Jul 31, 2007 IP
Thread Status:
Not open for further replies.