1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Googlebot/Test Spider Getting External JavaScript Files

Discussion in 'Google' started by digitalpoint, Mar 18, 2004.

0
  1. #1
    In the last few days, I've noticed a Googlebot/Test spider spidering nothing but external JavaScript files. There have been rumors of Google trying to better understand JavaScript, and it looks like they may be in the testing phase of it:

    I've had requests for .js files within multiple domains from two different IP addresses:

    64.68.89.156
    64.68.89.191

    According to ARIN, the 64.68.89.* block is not owned by Google, but considering Google owns the following class-Cs (which it uses for Googlebot):

    64.68.80.*
    64.68.81.*
    64.68.82.*
    64.68.83.*
    64.68.84.*
    64.68.85.*
    64.68.86.*
    64.68.87.*

    ...I think it's fairly safe to assume that it really *is* Google. Plus it's probably just a new IP block assignment that has not been updated in ARIN yet.

    - Shawn
     
    digitalpoint, Mar 18, 2004 IP
  2. Mr T

    Mr T Guest

    Messages:
    62
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Cool, thanks for that. Looks like I will finally have to implement that PHP redirect rather than JS links for affilates :(
     
    Mr T, Mar 18, 2004 IP
  3. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #3
    The spider is requesting a robots.txt file, so you could always exclude your external JavaScript files that way. {shrug}

    - Shawn
     
    digitalpoint, Mar 18, 2004 IP
  4. Will.Spencer

    Will.Spencer NetBuilder

    Messages:
    14,789
    Likes Received:
    1,040
    Best Answers:
    0
    Trophy Points:
    375
    #4
    ... unless you <include> your .js from your .shtml files.

    If you do that, the <include> code executes before the robot exclusion code is checked.
     
    Will.Spencer, Apr 18, 2004 IP
  5. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #5
    No, it would still be blocked... if you have an image directory you choose to block, Google will not spider it, even though the images are "included" within a HTML file that is spiderable.

    - Shawn
     
    digitalpoint, Apr 18, 2004 IP
  6. Will.Spencer

    Will.Spencer NetBuilder

    Messages:
    14,789
    Likes Received:
    1,040
    Best Answers:
    0
    Trophy Points:
    375
    #6
    Test it before you deploy it...

    <img src> isn't the same as <!--#include virtual-->.

    I tested non-JavaScript includes and found out that Google indeed did find them, because the include is done server-side.
     
    Will.Spencer, Apr 18, 2004 IP
  7. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #7
    Oh, I thought you were talking about a JS include like so:

    <script type="text/javascript"
    src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
    </script>

    - Shawn
     
    digitalpoint, Apr 18, 2004 IP
  8. Will.Spencer

    Will.Spencer NetBuilder

    Messages:
    14,789
    Likes Received:
    1,040
    Best Answers:
    0
    Trophy Points:
    375
    #8
    That might work like your IMG SRC example, or it might work like the <include> example... I am merely recommending testing. :)
     
    Will.Spencer, Apr 18, 2004 IP
  9. mobile phones uk

    mobile phones uk Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    So would I be able to put my affiliate links into an external js file and stop google from spidering them with the robots text file?

    Then call them up like in your quote?

    Thanks
     
    mobile phones uk, Apr 22, 2004 IP
  10. compar

    compar Peon

    Messages:
    2,705
    Likes Received:
    169
    Best Answers:
    0
    Trophy Points:
    0
    #10
    I'm missing something here. Why do you want to hide your affiliate links from Google's bots?
     
    compar, Apr 22, 2004 IP
  11. jarvi

    jarvi Well-Known Member

    Messages:
    127
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    103
    #11
    Not necessarily related to Googlebot, but a reason why you may wish to use php or js redirects is because some ad blocking software identifies affiliate type links and doesn't display them. Was just reading that Norton Internet Security appears to filter out links with redir or redirect in them and doesn't display anything. I haven't seen this, and am merely passing on some comments from another webmaster who purchased a new computer with the software preinstalled and was alarmed when the text links on his own site weren't appearing.

    Compar, as I mentioned in a thread a while back, I'd rather not pass PR to the merchants when they blatantly compete with me in PPC and SEO, so why give them more of a headstart.
     
    jarvi, Apr 23, 2004 IP
  12. Catfish

    Catfish Peon

    Messages:
    117
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #12
    why not just use the new rel command?
     
    Catfish, Jan 25, 2005 IP
  13. symetrix

    symetrix Peon

    Messages:
    44
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    If you ask rwhois.exodus.net, that class C is allocated to Google from the Savvis/Exodus/C&W US family.

    The IP blocks 64.68.{80-87}.* you mentioned are anycasted, which means your packets are routed to whatever datacenter is closest to you (network wise). However 64.68.88.0/21 is being routed only to their San Francisco office, which further supports your experimental theory.
     
    symetrix, Jan 26, 2005 IP
  14. tycoonjo

    tycoonjo Banned

    Messages:
    125
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    goolge don't love me
     
    tycoonjo, Jun 13, 2006 IP
  15. netprophet

    netprophet Banned

    Messages:
    288
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #15
    cool stuff ........:cool:

    thanx
     
    netprophet, Sep 19, 2006 IP
  16. baybossplaya

    baybossplaya Active Member

    Messages:
    597
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    58
    #16
    useful info
     
    baybossplaya, Dec 21, 2007 IP
  17. killer2021

    killer2021 Peon

    Messages:
    872
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #17
    Was thinking the same thing.
     
    killer2021, Nov 11, 2008 IP
  18. IEmailer.com

    IEmailer.com Well-Known Member

    Messages:
    1,864
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    110
    #18
    Not again..! It's your second post in two old threads..
    Seems like a spam to me?
     
    IEmailer.com, Nov 11, 2008 IP
  19. killer2021

    killer2021 Peon

    Messages:
    872
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #19
    I found this thread through google search! DP's advice is like gospel to me.
     
    killer2021, Nov 11, 2008 IP