Hello, for my site www.example.com i dont want to allow google crawl my /admin pannel.. so robots.txt user name: * Disallow: /admin or i have to write user name: * Disallow: /admin/ kindly give suggestion.
The two examples you listed do slightly different things... If you have URLs on your site like: www.example.com/admin/ www.example.com/admin/subfolder/ www.example.com/admin/pagename.php www.example.com/admin/subfolder/pagename.php www.example.com/adminblue/ www.example.com/admin.php Then Disallow: /admin will block ALL of the above pages since it basically says block EVERY URI whose path starts with "/admin" (in other words /admin*). The Disallow: /admin/ will block ONLY: www.example.com/admin/ www.example.com/admin/subfolder/ www.example.com/admin/pagename.php www.example.com/admin/subfolder/pagename.php It will not block: www.example.com/adminblue/ www.example.com/admin.php Because it is directing the search engines to NOT index any URL that starts w/ "/admin/" (in other words, /admin/*).
ok I will do like this user name:* Disallow: /admin becoz i dont want to allow google to index the pages after admin .. right?
yes you can use this:- User-Agent: * Disallow: /admin/ and if you want more information then look here:-www.robotstxt.org
hi, I have one more confusion i.e. what is the meaning of user agent: * Allow: / then google will indexing our site?
hey buddies, I too was searching for this info, me too do not want google to crawl my admin..... to get an idea I had just searched on this forum and got the tips here..... thxs for sharing .....
Hi, dear friends, can anyone able to answer me that I found in many sites, they are putting sitemap location in robots.txt file what is the benefit to do like this .. or sitemap and robots.txt from both which is best for site ? .. can we create both on our site?