1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

google cannot crawl my page

Discussion in 'Google Sitemaps' started by arijit_ad, Aug 6, 2011.

  1. #1
    3.jpg 4.jpg w.jpg 1.jpg
    problems.
    1) google shows that my robots.txt is unreachable. though it suceeds to find it. and it can be viewied www.ultimate4m.com/robots.txt
    2) shows a cross mark in my uploaded site map.
    3) does not crawl my page. it crawl my page in july but didnt do in august.
    SEMrush
     
    arijit_ad, Aug 6, 2011 IP
    SEMrush
  2. rboa

    rboa Greenhorn

    Messages:
    16
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #2
    www.ultimate4m.com/robots.txt/ is really unreachable. Don't forget to add "/" in the end of URL as it's done at your screenshot. The question is why the crawler attempts to reach the file in this URL format. May be some problem with .htaccess? BTW, can other search engines access this file or the problem is only with Google?
     
    rboa, Aug 8, 2011 IP
  3. luffynuoc

    luffynuoc Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    are you using godaddy hosting ?
    some user hosting at godaddy get the same problem
     
    luffynuoc, Aug 10, 2011 IP
  4. Jaish_D

    Jaish_D Peon

    Messages:
    53
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Jaish_D, Aug 10, 2011 IP
  5. cm2010

    cm2010 Active Member

    Messages:
    319
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    95
    #5
    1) please remove Sitemap: http://www.ultimate4m.com/sitemap.xml.gz from your robots.txt file.
    2) don't add "/" at the end of .txt because it is file not the folder so "/" doesn't come at the end.
    3) Complete survey popup when visiting your website, you need to remove that. (that could be causing the problem)
     
    cm2010, Aug 11, 2011 IP
  6. unknownpray

    unknownpray Active Member

    Messages:
    3,832
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    70
    #6
    When you place a URL in the robots.txt file you are actually telling the search engine crawler that you do not want this page to be indexed or crawled. Usually the pages that you would place here are the printer friendly pages of your websites if any .
     
    unknownpray, Aug 14, 2011 IP