In my robots.txt I have two parts, one for googlebot and the other for general spiders, I want to ask that will the googlebot also look for the rules present in the robots.txt section for all bots or will it only follow the ones defined specifically for it ? basically I want to define some global rules and some rules only for google but I also want google to follow the global rules .. I am worried that once I define some rules specifically for googlebot than it will descard the rest of the file thanks
It does if you read it carefully. Do this: User-agent: * Disallow: {insert global rules here} User-agent: Googlebot Disallow: {insert global rules here} Disallow: {insert Googlebot specific rules here} Code (markup): Just copy the global rules into the Googlebot section and any extras you need to the Googlebot section.