Hello friends, i have confussion about robots.txt file. so someone tell me which format is default and right for php websites. User-agent: * allow: / is this right or not??
Here is the right format..... # Allows all robots User-agent: * And you can use this if you want to disallow spidering of certain folders.... # Allows all robots User-agent: * Disallow:/cgi-bin/ In this case cgi-bin is the disallowed folder.
The correct "default format" is not to have one unless you need it. User-agent: * allow: / Code (markup): This is not valid and robots.txt is typically used to disallow rather than allow. In fact, avoid using allow as it is not a standard. Don't try to get clever by putting invalid entries in it. This is where you can end up making a mistake and disallowing your whole site. So unless you need to disallow a page/folder, don't use one. Read this.