How to stop Google Spidering my Javascript...

I put in a robots tag and it still seems to spider it after 3 months of wait, any ideas? Put in a redirect page on the directory index also

 

I thought google couldn't spider javascript. Well shows how much i know
Does your robot.txt look like this

Because i this is what you need

 

I thought it's robots.txt? Yes my setup looks like that

 

You can do it using htaccess...

RewriteEngine    on
RewriteCond    %{HTTP_REFERER} !^$
RewriteCond    %{HTTP_REFERER} !^http://([-a-z0-9]+\.)?yourdomain\.com [NC]
RewriteRule    \.(js|vbs)$ - [F,NC,L]

Code (markup):

This would stop any access where your domain wasn't in the referrer field.

(this is useful as not all robots obey the robots.txt guidelines - even googlebot!)

 

No it isn't. That tells spiders to disallow (ignore) everything. That's generally the last thing you want or need.

It IS robots.txt and if yours actually looks like that it needs work fast.

Which site has the robots.txt file in question? Post a URL and the relevant lines from your .htaccess file.

 

Oops sorry misohoni, Please forgive me. That teaches me for doing multiple things at once!

DangerMouse thats quite interesting, thanks

 

You can place your javascript into an external file, place the file in a folder and then disallow access to that folder in your robots.txt.

This has two benefits:

1. It reduces the code to content overhead;
2. Any bots than can and do read javascript will never see it.

 

I prevent any robot from spidering my CSS and JS files by using this:

User-agent: *
Disallow: /css
Disallow: /js

I put all CSS and JS files in the /css and /js directories and voila, no more spiders on those files.

-Scott

 

This implies that spiders adhere to the robots.txt standard... I have found this not to be the case!