How can you tell a site is blocking the bots?

I am pretty sure its .robot txt that stops a site from being indexed but wanted to make sure and if it is how can I look at a source code to see if they are blocking a bot from indexing or is that not possible?

 

Just look at their robots txt - www.whatevertheirdomainnameis.com/robots.txt

Alternatively, they might have put is as a meta-tag - look for "noindex" in the source

 

dont get it so it should be in the url? or should I be looking for something in the source code?

 

Lets say you want to see the robots.txt file of widget.com you would simply go to www.widget.com/robots.txt

If they are allowing everything, then look in the meta data (should be at the top) within the source code and see if they are blocking anything from there.

 

Check nofollow tags on links as well. It blocks robots from crawling the link that page will most likely not get indexed unless having inbound links from other websites or pages that is not using nofollow tags.

 

still a little confused can I pull up the source code and look at that and see a .robts txt in there or no?

 

You can also try typing /robots.txt directly on your browser.

Ex. http://www.example.com/robots.txt

 

and what do I need to look for once I type that in to the domain?

 

This can help you a lot, check this Google webmaster support information.

Just watch out for this type of robots.txt files as they are meant to block any crawlers from crawling your website.

User-agent: *
Disallow: /

Code (markup):

 

<meta name="robots" content="noindex, nofollow" />

 

This is perfect....

 

You are absolutely right, it is the only way to check robots.txt and content in it.

 

As robots.txt is placed in the root folder, you can easily check your robots.txt file at yourdomain.com/robots.txt