What to Allow and Disallow on robots.txt

What to Allow and Disallow on robots.txt in PHP index site

 

Don't use allow because everything is default allow.

Disallow your moderator pages, cgi bin, tmp folder.

I would need to take a look at your sites directory to tell you more.

 

It all depends on you, if there is nothing in your website that you want to hide then you can use following code in your robots.txt

User-agent: *
Disallow:

OR

User-agent: *
Allow: /

and in case if you want to restrict some pages/folders, you can use following code:

User-agent: *
Disallow: /restricted-pages.php
Disallow: /restricted-folder/

 

Thanks .. i Generate a robots.txt this
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
Disallow: /css/
Disallow: /date/
Disallow: /email/
Disallow: /img/
Disallow: /omer/
Disallow: /picture_library/
Disallow: /public_html/

Sitemap: http://www.astrohope.com/public_html/sitemap.xml

and Status is 200 (Success)

What you think ?

 

This is cool

 

hi,
it is up to you that what you want to not index in google. simply disallow all those file or folder that you want to hide from Search Crawler.

 

Hello,

Index Page won't be crawled except the page is accessed with the full qualified path.

User-agent: *
Disallow: /
Allow: /index.php
Allow: /$