How to write the robots file

Discussion in 'Search Engine Optimization' started by getbestproduct, Jan 8, 2011.

  1. #1
    I want to disabled search engine to crawl the page such as

    /a_xxxxxx.html
    /b_xxxxxx.html

    This is the content of my robots file .Does it right?
    User-agent: *
    Disallow: /a_*.html
    Disallow: /b_*.html
     
    getbestproduct, Jan 8, 2011 IP
  2. seo.vijay

    seo.vijay Member

    Messages:
    391
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    30
    #2
    This example disallows all Web Spiders for the entire site:
    # Make changes for all web spiders
    User-agent: *
    Disallow: /
     
    seo.vijay, Jan 8, 2011 IP
  3. getbestproduct

    getbestproduct Peon

    Messages:
    140
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I just disallow the page begining with 'a_' or 'b_' .
    Not all page.
     
    getbestproduct, Jan 8, 2011 IP
  4. getbestproduct

    getbestproduct Peon

    Messages:
    140
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Please help me/
     
    getbestproduct, Jan 8, 2011 IP
  5. Reliable-SeoServices

    Reliable-SeoServices Peon

    Messages:
    441
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Are you planning to block any directory like this example

    User-agent: *
    Disallow: /administrator/
    Disallow: /cache/
    Disallow: /images/
     
    Reliable-SeoServices, Jan 8, 2011 IP
  6. rajeevranjan

    rajeevranjan Peon

    Messages:
    136
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Hi.. Go through the website robotstxt.org/robotstxt.html that will help u out.
     
    rajeevranjan, Jan 9, 2011 IP
  7. internetproz

    internetproz Peon

    Messages:
    136
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #7
    here is the code you you need to create

    User-agent: *
    Disallow: /a_xxxxxx.html
    Disallow: /b_xxxxxx.html
     
    internetproz, Jan 9, 2011 IP