Twitter's robots.txt question:

Discussion in 'Search Engine Optimization' started by Hema1, Dec 9, 2010.

0
  1. #1
    Twitter's robots.txt, It shows everything is disallowed, but surprisingly search engines are crawling and indexing everybody's profiles pages, Why?
     
    Hema1, Dec 9, 2010 IP
  2. oilarts

    oilarts Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I have long assumed that the syntax of a robots.txt file was pretty straightforward, My question or problem is with the filespec part.
     
    oilarts, Dec 9, 2010 IP