1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Issue Regarding Robots.txt

Discussion in 'robots.txt' started by Kamaldeep Singh SEO, Oct 24, 2018.

  1. #1
    Hi Webmasters,
    I have a question regarding robots.txt. I have a website which is running on java and I also installed a blog on that website. both has different robots file. Now please tell me which will google consider?
    Example:
    abc.com/robots.txt
    abc.com/blog/robots.txt
    I know google will consider 1st robots file but I just want to know 2 robots file can effect ranking? if it ranking factor?
    SEMrush
     
    Solved! View solution.
    Kamaldeep Singh SEO, Oct 24, 2018 IP
    SEMrush
  2. #2
    First and prior rule about robot is .. it should be on root folder. So its not good idea.
    And if you can manage complete website from single robot.txt (on root) then why need to manage multiple.
    For example
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /tmp/
    Above for homesite
    You can use same here for blog also like

    Disallow: /blog/image/

    Not need to manage multiple .... according to me its not good idea and will not effect your ranking or other ranking factors.
     
    RSSeosolution, Oct 25, 2018 IP
    Kamaldeep Singh SEO likes this.
  3. peter patel

    peter patel Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #3
    I also have a question regarding the robots.txt, i see some people putting

    Allow: /en/gm/

    Is this a requirement or not?

    Also if you leave a line in between like below format, is this okay?

    User-agent: *

    Disallow: /cgi-bin/
     
    peter patel, Oct 25, 2018 IP
  4. RSSeosolution

    RSSeosolution Banned

    Messages:
    144
    Likes Received:
    6
    Best Answers:
    3
    Trophy Points:
    140
    #4
    robot.txt generally used to block any folder or files online which are private or You dont need to index in search engines. So

    Allow: tag is not important because if you leave robot.txt blank after user-agent:* then search engines will index all according to his rules.

    Use only disallow if you want to block any folder or files from index.

    If you leave a line between two line then also it will not effect more according to me ... While i did not test this type of format but i think it can happen by mistake and search engines or bot can skip one line easily.
     
    RSSeosolution, Oct 25, 2018 IP
  5. sigen

    sigen Greenhorn

    Messages:
    77
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    13
    #5
    It won't affect your ranking, but you just don't need 2 robots.txt files since the only one which is in the root directory will be considered.
     
    sigen, Oct 25, 2018 IP