1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robot.txt for Google (error)

Discussion in 'robots.txt' started by wartes, Apr 10, 2017.

  1. #1
    Hello!
    Can somebody help me ?
    in webmaster google i see errors like
    site.com/mobile/page23/mobile
    i want allow only site.com/mobile/page23/
    how to Disallow double /mobile ?
    (becose the page site.com/mobile/page23/mobile does not exist)
    SEMrush
     
    Solved! View solution.
    wartes, Apr 10, 2017 IP
    SEMrush
  2. Nick Chernets

    Nick Chernets Member

    Messages:
    55
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    33
    #2
    Just paste
    Disallow:
    and what you need to disallow,
     
    Nick Chernets, Apr 11, 2017 IP
  3. Patricia Ann Lee

    Patricia Ann Lee Active Member

    Messages:
    1,190
    Likes Received:
    73
    Best Answers:
    1
    Trophy Points:
    95
    #3
    Disallow: site.com/mobile/page23/mobile
     
    Patricia Ann Lee, Apr 11, 2017 IP
  4. wartes

    wartes Member

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    33
    #4
    wtf people ? i need to Disallow pages like this site.com/mobile/page23/mobile (over 70000 pages)
     
    wartes, Apr 12, 2017 IP
  5. #5
    A better solution will be to create server-rules to redirect non-existing urls to an existing one, or just a general 404-error page, or just send them to the index or a search page. Manually redirecting or disallowing that many links are counterproductive.
     
    PoPSiCLe, Apr 12, 2017 IP
  6. wartes

    wartes Member

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    33
    #6
    (I found) If someones will be have the problem
    just add this is robots.txt
    Disallow: */*/mobile
     
    wartes, Apr 12, 2017 IP
  7. Md.Saddam Hossain

    Md.Saddam Hossain Active Member

    Messages:
    77
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    63
    #7
    Hello Wartes,

    The best solution, you can redirect this page (site.com/mobile/page23/mobile) to site.com/mobile/page23/ and reupdate it from search console. If you want to disallow it by robots.txt, then follow the below:

    User-agent: *
    Disallow: /site.com/mobile/page23/mobile/

    The search engine won't index this page again. If your site built on WordPress, there are many easy ways fix it out. Let me know if you have any question. :)
     
    Md.Saddam Hossain, May 7, 2017 IP