How to stop Google Spidering my Javascript...

[misohoni]

I put in a robots tag and it still seems to spider it after 3 months of wait, any ideas? Put in a redirect page on the directory index also

[simplexity]

I thought google couldn't spider javascript. Well shows how much i know ;)
Does your robot.txt look like this


User-agent: *
Disallow: /


Because i this is what you need :)

[misohoni]

I thought it's robots.txt? Yes my setup looks like that

[DangerMouse]

You can do it using htaccess...


RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://([-a-z0-9]+\.)?yourdomain\.com [NC]
RewriteRule \.(js|vbs)$ - [F,NC,L]


This would stop any access where your domain wasn't in the referrer field. :cool:

(this is useful as not all robots obey the robots.txt guidelines - even googlebot!)

[minstrel]

User-agent: *
Disallow: /
this is what you need
No it isn't. That tells spiders to disallow (ignore) everything. That's generally the last thing you want or need.

I thought it's robots.txt? Yes my setup looks like that
It IS robots.txt and if yours actually looks like that it needs work fast.

Which site has the robots.txt file in question? Post a URL and the relevant lines from your .htaccess file.

[simplexity]

Oops sorry misohoni, Please forgive me. That teaches me for doing multiple things at once! :eek:

I thought google couldn't spider javascript. Well shows how much i know ;)
Does your robot.txt look like this



Because i this is what you need :)

DangerMouse thats quite interesting, thanks

[draculus]

You can place your javascript into an external file, place the file in a folder and then disallow access to that folder in your robots.txt.

This has two benefits:

1. It reduces the code to content overhead;
2. Any bots than can and do read javascript will never see it.

[scottj]

I prevent any robot from spidering my CSS and JS files by using this:

User-agent: *
Disallow: /css
Disallow: /js

I put all CSS and JS files in the /css and /js directories and voila, no more spiders on those files. :)

-Scott

[DangerMouse]

I prevent any robot from spidering my CSS and JS files by using this:

User-agent: *
Disallow: /css
Disallow: /js

I put all CSS and JS files in the /css and /js directories and voila, no more spiders on those files. :)

-Scott

This implies that spiders adhere to the robots.txt standard... I have found this not to be the case!