Robots.txt

Discussion in 'WordPress' started by jacoboram, Apr 21, 2013.

  1. #1
    "A description for this result is not available because of this site's robots.txt – learn more" i am new on wordpress cms, so when i search on goolge shwoing this text , why not showing meta description? how can i add a Robots.txt file?
     
    jacoboram, Apr 21, 2013 IP
  2. Techking

    Techking Well-Known Member

    Messages:
    611
    Likes Received:
    9
    Best Answers:
    1
    Trophy Points:
    130
    #2
    Better to install any seo plugin such as yoast and add meta description there for each page .

    Following link will help you for robot.txt config :

    http://www.seomoz.org/learn-seo/robotstxt

    Good Luck :)
     
    Techking, Apr 21, 2013 IP
  3. jacoboram

    jacoboram Active Member

    Messages:
    17
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    88
    #3
     
    jacoboram, Apr 21, 2013 IP
  4. Techking

    Techking Well-Known Member

    Messages:
    611
    Likes Received:
    9
    Best Answers:
    1
    Trophy Points:
    130
    #4
    yes ,you have to wait 24-144 hours for search engines .Plus, submit your sitemap manually to google along with signup in google webmaster tools.
     
    Techking, Apr 22, 2013 IP
  5. Devtard

    Devtard Notable Member

    Messages:
    850
    Likes Received:
    133
    Best Answers:
    4
    Trophy Points:
    220
    #5
    If the file robots.txt is involved then no plugin will help you. The page is probably being blocked by the website owner (unless Google made an error).

    If that is your website, make sure that WP doesn't prevent bots from indexing it (Settings → Reading → Search Engine Visibility). You can view the file by opening example.com/robots.txt in your browser.
     
    Devtard, Apr 22, 2013 IP
  6. jacoboram

    jacoboram Active Member

    Messages:
    17
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    88
    #6
     
    jacoboram, Apr 22, 2013 IP
  7. Devtard

    Devtard Notable Member

    Messages:
    850
    Likes Received:
    133
    Best Answers:
    4
    Trophy Points:
    220
    #7
    I think that it is fine. Although I am not sure why do you want to block these folders (Google is "smart" enough to not try index them, besides there are usually no links pointing to these locations).
     
    Devtard, Apr 22, 2013 IP
    ryan_uk likes this.
  8. Techking

    Techking Well-Known Member

    Messages:
    611
    Likes Received:
    9
    Best Answers:
    1
    Trophy Points:
    130
    #8
    because some crawlers still oversmart than google and crawl ,wp-content,wp-admin data .
     
    Techking, Apr 22, 2013 IP
  9. Devtard

    Devtard Notable Member

    Messages:
    850
    Likes Received:
    133
    Best Answers:
    4
    Trophy Points:
    220
    #9
    I wouldn't call that "smart" because in this case there is nothing to crawl.
     
    Devtard, Apr 22, 2013 IP