Robots.txt

Hi guys,

I have a wapsite, I put advertising on Admob, MobGold, etc on it. I heard a case that people got their account banned on the ad network and the ad network support answers them that they might getting fraud clicks from robot.

What I want to ask you all, are how do I keep robot off clicking ads on my wapsites ? On the other hand I want the site to be SEO-Friendly as well. It is possible to do this ?

 

Please visit this example.. also visit.. http://www.robotstxt.org/robotstxt.html.
You will get your results...
Hope you will find your answer..

 

I think you do not want to fetch your ads by Google bot. I think it is possible to allow the specif path in robots.txt, guess its a better idea to keep all the ads in a particular folder and disallow Google from fetching it. the format is somewhat like

Agent : Google bot
Disallow : /folder name

 

Can you tell me how robots.txt prove to be helpful in websites.

And how it can be checked that the clicks we are getting are made by robots not human clicks?

 

Hello

Generally when I work, I used to check robot.txt file by opening view source and find robot.txt file and check whether it is (index, follow) or (no index, follow) or (no index, no follow) or (index, no follow).

 

hello dear

anything which you want not to crawled by search engine just put that url into robots.txt file and that particular file or image not crawled by nay search engine..

 

Malicious crawlers will bypass any setting you put up . If you're not getting a lot of traffic from this countries then ban the following : China , India , Russia , Nigeria & Romania . That will be a far better method of website defense .

 

What do you mean by Ban specifically these countries? Do they have bad traffic?

 

To keep those spammy robots away from your website, you would first analyze the traffic of your website through server log files. If you find frequently visits from unknown robots/source that you think is not useful for your website, just ban them using htaccess. It would restrict them to visit your website again and would save a lot of bandwidth as well.

 

hello if i upload only robot meta tag then anything wrong. can anybody suggest me

 

Search engines will look for the robots.txt file AS WELL as the meta tag data so it is best to use both.

You won't be penalised for this in any way and it can only help!

 

actually i want all page indexed and follow. then can you suggest me what will be my robots.text file

 

User-agent: *
Allow: /

This will basically allow everything to be crawled by Google bots