Crawling Issue Help

Discussion in 'Search Engine Optimization' started by HarryJackson, Aug 13, 2010.

  1. #1
    Some thing not going good with my site. All search engines stop crawling my website.
    Need quick help from SEO gurus. My developer add this code to stop search engine crawl admin
    User-Agent: *
    Disallow: http://cashforgoldleprechaun.com/admin/ , is this have any effect or any other reason, some big links are given so no issue of back links, but search engine are not showing any link.
     
    HarryJackson, Aug 13, 2010 IP
  2. Lever

    Lever Deep Thought

    Messages:
    1,823
    Likes Received:
    94
    Best Answers:
    0
    Trophy Points:
    145
    #2
    Are you sure your site is not being crawled? Are you checking in, for instance, Google webmaster Tools?

    You don't seem to have a /sitemap.xml so unless your xml sitemap has another name I'd create one and submit it to GWTs.

    Are you updating your site regularly and pinging the SEs if/when pages are added and the xml sitemap is updated?

    Also your robots.txt file should actually have the folder w/o the URL so it should just read

    User-Agent: *
    Disallow: /admin/
    Code (markup):
     
    Lever, Aug 13, 2010 IP
  3. HarryJackson

    HarryJackson Peon

    Messages:
    181
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I have site map, Analytic shows change in visitors, but non of search engine crawling my site, problem is not specific to Google. I also place some good back links but non of them are shown on Yahoo or Google.
     
    HarryJackson, Aug 13, 2010 IP
  4. robertthomas

    robertthomas Peon

    Messages:
    156
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    check google webmaster tools it will show current status of your site
     
    robertthomas, Aug 13, 2010 IP