Review Your Robots.txt File - Assess Your Meta Robots Tagging

Discussion in 'Search Engine Optimization' started by zanikos, Dec 2, 2013.

  1. #1
    If you have a robots.txt file on your site, check by visiting /robots.txt. You may be surprised to find out you are withholding pages, folders, images, etc. from search engines that can drive traffic to your site.

    Additionally, run a site scan with a tool such as Screaming Frog to assess if there are any pages on your site you are excluding via a meta robots tag. Both of these are a very quick fix if you do find issues.

    Unknowingly tagged pages or robots.txt entries are usually the culprit of a developer who forgot to remove the designations when a new page rolled live or a previous site administrator who deemed the quality content unimportant for the masses.

    Does anyone used this ?
     
    zanikos, Dec 2, 2013 IP
    neteater likes this.
  2. John Dave

    John Dave Active Member

    Messages:
    873
    Likes Received:
    35
    Best Answers:
    10
    Trophy Points:
    80
    #2
    Well thanks i've found this post useful !! Good work.:)
     
    John Dave, Dec 3, 2013 IP