Testing Robots.txt

* disallow
allow=Googlebot

In Robots.txt i wrote above code then What is result

 

Well, this one wouldn't do squat, because it's wrong.

I guess you want to block every bot except googlebot, right?

Below code would do that

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /

Code (markup):

I highly suggest you visit

http://www.robotstxt.org/

Code (markup):

and read through the site.

 

This code may i punt in htaccess file right?

 

No, it goes into the robots.txt file

 

ok but 301 redirection code/tag need to know

 

Yeah, 301 redirection code goes into the .htaccess file

Here is an example that redirects the whole website

Redirect 301 / http://newsite.com/

Code (markup):

 

Hi read this tutorial for robots.txt http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress
You will know many information about robots.txt file.

 

Well, if the OP is using wordpress, yes, it would be useful, but ONLY for Wordpress.

Actually, no. You will not get "many information" from this site. You get the bare minimum.




The best source is still the official site

http://www.robotstxt.org/

Code (markup):

 

I would suggest you not to block the search engines. You can use this casual code, and in order to place it you have to create a txt file with the below contents and place it in the root folder.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

 

And what if OP doesn't use Wordpress?

 

I am sorry GMF, I described it with respect to wordpress. I am a begineer for other platforms, and I would be happy if any one could reply with an answer for your question.

 

This code is Robot.txt related so your page is online but search engine is not crawling your page.

 

You can

Disallow: /your-private-directory/

 

robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.

 

Nobody asked what robots.txt is.