Site with no robots.txt

Hi. I have this website and I just realized that it's got no robots.txt!

Even without it, all content is indexed.

As I understand, the robots.txt file helps the search engine to index your site so I thought that uploading the robots file would be better for indexing and that it could help with my rankings.

Maybe with a robots.txt file in place, the "quality" of the site as a whole would be better and could rank better than other sites.

Am I right?

 

You would need a robots.txt file if you want to block Search Engines from crawling / indexing certain sections of your site.

If you don't have irrelevant / noise content on your website, or you don't have any duplicate content/URL issues (for example, if you have a dynamic PHP/MySQL driven site); then I think theres no need for you to have a robots.txt file.

 

you do not need a robots.txt to index your pages in search engine

 

robots.txt file will help you which pages you don't want to index in particular crawler. by default it will index all thing which concern with your website.

 

may I ask here?
is robot.txt can block some ip?
and prevent from hackers...?

 

good question rise here
but
as per my knowledge google not block IP. robots.txt is help us only related with our website root folder.

but some experience person give us proper answer for your question.
awaiting for more reply

 

I thought that I have read the article about that..
I waiting for the expert also to confirm that..

 

Sometimes you don't want a particular spider to index your site for some reason or other. Perhaps the robot is ill-behaved and spiders your site at such a high speed that it takes down your entire server. Or perhaps you prefer that you don't want the images on your site indexed in an image search engine. With a robots.txt file.

 

You would block the IPs or IP ranges with .htaccess, robots.txt is purely to disallow spiders access to certain pages within your site so, as an example, you want your "thank you for your purchase" page to remain hidden then you would block robots from indexing it.

There are two major problems with robots.txt:

1) Anyone with a little knowledge can access the information you want hidden simply by looking at your robots file (which is always in the root directory) therefore in attempting to hide your pages you are in fact showing them. Case in point : http://www.whitehouse.gov/robots.txt

2) Malicious spiders will ignore the robots.txt file and index your "hidden" pages anyway.

A solution would be to add the following line to your html header

<META NAME="ROBOTS"  CONTENTS=" noindex ,nofollow,noarchive">

Code (markup):

 

so i dont upload robots.txt i will cause any problem in indexing?

 

As others have said the robots.txt file isn't used for simple indexing purposes, it's to define parts of your site which you don't want crawled.

 

Not only that WebCatch, but having one also helps keep a site's server logs free of unnecessary 404 errors (same as having a favicon.ico file).

 

robots.txt is primary for telling search engines to not index set paths of your webpages, not oposite

so page without robots.txt is ok

 

Google will index your web pages by default if you are not using robots.txt...robots.txt is used in opposite case, where you don't want to index some private pages over search engines..

 

having no robots.txt means google bot will crawl each and every page of your site...

 

<META NAME="ROBOTS"  CONTENTS=" noindex ,nofollow,noarchive">

Code (markup):

Thanks for that

 

robots.txt file do not help in better indexing of the website. it restricts the good robots from crawling the web pages you specify. moreover, when a robot comes to the website the first thing it looks for is robots.txt file. on not finding the file it returns the File Not Found error. It is also recommended specifying the xml sitemap address in the robots file so that robot can move on to the xml sitemap and crawl and index the urls. but submitting your sitemap to Google Webmaster tools is enough