20% PPC Cost is wasted due to Crawlers

[akash.kava]

Eversince I started google AdSense on my websites, I didnt have much time to investigate about how much actual clicks are generated by my websites. I have some doubt by running AWStats I came to know that crawlers (Google bot, MSN bot, Archive bot, PAD crawlers) etc make 20% of hits on our web servers.

AWStats is good tool to distinguish actual hits of crawlers and visitors. It does distinguish it by analyzing User Agent.

Does PPC ignore User Agents?
Yes, most of Ad Management softwares like phpAds etc does not distinguish user agent of HTTP. So all crawler's hits are counted as Visitor clicks. And we keep on paying for these hits.

Is Crawler equal to visitor?
Yes, Hits of crawler are benefits because they crawl the website and they somehow improve search engine listings. But cost of bringing crawler and cost of bringing visitors are same? But wait, these hits do not improve your search engine list at all because the links are travelled through many servers before they reach destination link. And ranking changes according to direct links. Google or PPC ad provider itself may get higher ranking due to crawlers but you dont get any benefit if your ads on PPC are crawled by crawlers. So this 20% PPC cost is complete waste due to crawlers. And no Ad Server uses USER Agent to identify visitor.

Does Crawlers crawl Google AdSense?
Dont know, may be google can answer, or crawlers can answer. We have our crawler which does execute flow javascripts because many times content is built with the help of java script. If we use Micrsoft's DHTML parser, then it does download google's adsense script and builds document. And we search for all "A" tags, and we mark them to crawl. This way "HREF" of all "A" tags of html are marked to crawl. Resulting crawling of PPC ads.

How to improve PPC Ads to avoid Crawler Clicks?
Crawler crawls HREFs of A tags, Src of Frame tags. But crawlers do not generate actual mouse clicks. Crawlers do not POST Forms. Easiest way is since crawler can not simulate mouse clicks, the best way to write html ad is, simpley output only IMG tag with STYLE tag to show Hand Cursor and write an event for onclick. Example below.

<IMG width="480" height="60"
src="http://imgsrv.ads.com/ad?id=2"
onclick="window.open('http://clicks.ads.com?ad?id=2','_blank')"
STYLE="CURSOR:Hand">

We have implemented this on our ad network, this is so perfect, there is no chance of fraudulent clicks.

[aenigma]

We have implemented this on our ad network, this is so perfect, there is no chance of fraudulent clicks.


No fraudolent clicks???
I think that EVERY pay per click sistem CAN be fucked up by a well done script, as they are on client side, and on my client I can do anything...

This method is Good against spider and this is right, but not to avoid fraudolent clicks.

[mxlabs]

We have implemented this on our ad network, this is so perfect, there is no chance of fraudulent clicks.

this statement is simply not true. just because you don't know of fraudster tools does not mean there is "no chance to do it". Javascript might have stopped fake clicks in 1998. Today, there are advanced tools on the "fraud market" supporting everything from javascript down to flash.
I'm sorry to disappoint you, but there is no way a simple javascript tag will stop all fraudulent traffic nowadays.

[digitalpoint]

Does Crawlers crawl Google AdSense?
Dont know, may be google can answer, or crawlers can answer. We have our crawler which does execute flow javascripts because many times content is built with the help of java script. If we use Micrsoft's DHTML parser, then it does download google's adsense script and builds document. And we search for all "A" tags, and we mark them to crawl. This way "HREF" of all "A" tags of html are marked to crawl. Resulting crawling of PPC ads.
Well then it doesn't look like you know how to make a very good spider then, eh? Considering the JavaScript file for AdSense is disallowed by Google's robots.txt document.

[akash.kava]

Well then it doesn't look like you know how to make a very good spider then, eh? Considering the JavaScript file for AdSense is disallowed by Google's robots.txt document.

Yes thats good point.. I know abt it , but AdSense scripts are inline or flow scripts, I guess all spider or any webclient has to execute flow scripts in order to complete page because Tree like structures are created on html wth help of flow scripts. Downloading javascript as inline part of html page and crawling page are two different things. Robots.txt is for crawling, however if makers of spiders do block inline or flow javascripts then I guess it would be better.

I will also try to search more on this.

[digitalpoint]

They aren't two different things... If someone has a framed page on another server that is disallowed for spiders, does that mean it's okay for a spider to crawl it because you "need" it to complete the page?

[Chris Choi]

Are you sure the crawlers are clicking on the Adsense boxes?

This never seemed to happen to me.

[seochamp]

No. crawlers do not get to see the adsense boxes in the very first place.
They do not load a HTML DOM. They just make a http request and just flushed plain HTML code which is as good as plain text to them.
Only when loaded thru HTML DOM is the javascript able to render the google adsense box.
So there is no way that a crawler can get to your adsense links unless the crawler perhaps identifies itself as mozilla compatible and tries to request faking itself as a browser.
This may be true because google and other engines might actually do the same thing themselves to figure out whether the pages of a site are shown the same way to browsers as they are shown to the crawler.

For now its a safe assumption that crawlers dont eat up adsense links.

[Josh]

No. crawlers do not get to see the adsense boxes in the very first place.
They do not load a HTML DOM. They just make a http request and just flushed plain HTML code which is as good as plain text to them.
Only when loaded thru HTML DOM is the javascript able to render the google adsense box.
So there is no way that a crawler can get to your adsense links unless the crawler perhaps identifies itself as mozilla compatible and tries to request faking itself as a browser.
This may be true because google and other engines might actually do the same thing themselves to figure out whether the pages of a site are shown the same way to browsers as they are shown to the crawler.

For now its a safe assumption that crawlers dont eat up adsense links.

Exactly :)

AdSense, and almost every other PPC and CPM ad programs out there all use javascript, and bots dont have the ability to crawl javascript. While in the near future, I have no doubt that there will be a "javascript emulator" so to speak, but until then, I dont think you will have to wory about bots clicking the ads ;) Well, clicking wouldn't be the term anyway, since bots cant click ;)

Josh

[eves]

but why would crawlers "click" on ppc engines results if they dont input anything to be searched, so nothing will be displayed??

[Will.Spencer]

I only wish crawlers clicked on my AdSense ads. :D

[szynka]

Seems like great, now its enought to make spider that will spider ads and
google will report another revenue record.

Shity.

[szynka]

BTW. Fraudlent click are not by robots or spiders, but humans.

PM me i you have any addres of free adsense tracking script, please.