Need Help in creating robots.txt

Discussion in 'Search Engine Optimization' started by Anjana Ratadiya, Oct 28, 2015.

  1. #1
    Hi Everyone,

    I want some help in creating robots.txt for my website. I want to disallow some URL patterns but I don't know, how to write that syntax in robots.txt

    Example:
    URLS of my websites are like:
    http://www.example.com/portfolio/portfolio1/
    http://www.example.com/portfolio/portfolio2/

    Now, I want disallow all URLs after http://www.example.com/portfolio/. I mean I want to disallow portfolio1 , portfolio2....... pages...

    Is there any syntax for it?

    Thank you in advance [​IMG]
     
    Anjana Ratadiya, Oct 28, 2015 IP
  2. iwebsocial

    iwebsocial Well-Known Member

    Messages:
    1,718
    Likes Received:
    69
    Best Answers:
    4
    Trophy Points:
    170
    #2
    Your robots.txt will be like this:
    User-agent: *
    Disallow: /portfolio/portfolio1/
    Disallow: /portfolio/portfolio2/
     
    iwebsocial, Oct 28, 2015 IP
  3. rsrikanth05

    rsrikanth05 Well-Known Member

    Messages:
    1,362
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    190
    #3
    Hi, you can use my robots.txt for reference:
    bestpedia[dot]in/robots[dot]txt
    I've prevented multiple directories from being indexed.

    You can ignore the Attracta lines because they're no longer in use.
     
    rsrikanth05, Oct 29, 2015 IP
  4. Anjana Ratadiya

    Anjana Ratadiya Member

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #4
    Thank you for the answer. But I have multiple url like portfolio1, portfolio2.........portfolio500... so I am looking for some pattern to disallow all using one line.
     
    Anjana Ratadiya, Oct 29, 2015 IP
  5. Anjana Ratadiya

    Anjana Ratadiya Member

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #5
    Thanks for the reply. I have get an idea from your reply and created a syntax like:
    Disallow: /portfolio/portfolio
    And it works for me because I want to index portfolio page but don't want to index it's sub pages like portfolio1, portfolio2.....Thank you. :)
     
    Anjana Ratadiya, Oct 29, 2015 IP
  6. rsrikanth05

    rsrikanth05 Well-Known Member

    Messages:
    1,362
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    190
    #6
    ^^Google Webmaster lets you check the Robots.txt. Do check it once and make sure it is all ok.
     
    rsrikanth05, Oct 29, 2015 IP
  7. Anjana Ratadiya

    Anjana Ratadiya Member

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #7
    Yes rsrikanth. I have checked it. It's working correctly.
     
    Anjana Ratadiya, Oct 29, 2015 IP