1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Yahoo Spider Behavior

Discussion in 'Yahoo' started by BoneHead, Jul 12, 2006.

0
  1. #1
    Is it normal for the yahoo spider to visit a site 100 or more times a day?
     
    BoneHead, Jul 12, 2006 IP
    _JmZ_ likes this.
  2. ash1

    ash1 Active Member

    Messages:
    2,240
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    88
    #2
    how do you know the yahoo spider visits your site?

    I can only tell you once i know how :(
     
    ash1, Jul 12, 2006 IP
    _JmZ_ likes this.
  3. BoneHead

    BoneHead Peon

    Messages:
    19
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    The site is run by Invison Power Board forum software. Under Admin tools, you can click on "View Bot Log", which will list each spider that visited, keep a count for each and keep track of which page they requested.

    The tool is telling me it's yahoo.com.

    I see the ipaddresses coming in are from the following...

    72.30.xx.xx
    and
    68.142.xx.xx
     
    BoneHead, Jul 12, 2006 IP
  4. Xitanto

    Xitanto Peon

    Messages:
    434
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Also, if you have Cpanel installed on your server (domain.com/cpanel), you can use the statistics program to show you bot activity as well.
     
    Xitanto, Jul 13, 2006 IP
  5. BoneHead

    BoneHead Peon

    Messages:
    19
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I would be interested to see what ash1 has to say as to why yahoo spider visits so often.
     
    BoneHead, Jul 13, 2006 IP
  6. theguywhoknowz

    theguywhoknowz Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Yahoo Slurp Spider has been known for unnecessary visits, people even had to set up robots.txt to stop yahoo going in or control it. Our site, Maternity.com has average of 150 Yahoo bots a day. However; google is a different story.
     
    theguywhoknowz, Jul 13, 2006 IP
  7. BoneHead

    BoneHead Peon

    Messages:
    19
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #7
    And how many of your pages has yahoo indexed?
     
    BoneHead, Jul 13, 2006 IP
  8. dcristo

    dcristo Illustrious Member

    Messages:
    19,776
    Likes Received:
    1,199
    Best Answers:
    7
    Trophy Points:
    470
    Articles:
    7
    #8
    Nice domain you got there :)
     
    dcristo, Jul 13, 2006 IP
  9. theguywhoknowz

    theguywhoknowz Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    We have 357 listing on yahoo atm, I had just rebuild the site 2 months ago. Usually we have 4,5 new pages lising to yahoo SERP every week.
     
    theguywhoknowz, Jul 14, 2006 IP
  10. theguywhoknowz

    theguywhoknowz Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Thanks hehe :D
     
    theguywhoknowz, Jul 14, 2006 IP
  11. BoneHead

    BoneHead Peon

    Messages:
    19
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #11
    My problem is that slurp visits 100+ times each day but still only indexes main page. If slurps wants to visit fine but atleast index some of the 100 pages he is requesting each day.:(
     
    BoneHead, Jul 14, 2006 IP
  12. theguywhoknowz

    theguywhoknowz Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Yahoo did the same thing to me for 2,3 weeks, only put the forum index page on the SERPs, all other pages are old, even non existance. After that time, I got 1,2 pages in then slowly moving. I think it just matter of time.
     
    theguywhoknowz, Jul 15, 2006 IP
  13. redhits

    redhits Notable Member

    Messages:
    3,023
    Likes Received:
    277
    Best Answers:
    0
    Trophy Points:
    255
    #13
    Yes, why not?! especialy if it' coming from 301 redirects....
     
    redhits, Jul 15, 2006 IP
  14. Gniuz

    Gniuz Peon

    Messages:
    126
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #14
    Gniuz, Jul 17, 2006 IP
  15. Skyline_GTR

    Skyline_GTR Well-Known Member

    Messages:
    442
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    130
    #15
    yea it is normal... at once there were about 180 bots....
     
    Skyline_GTR, Jul 17, 2006 IP
  16. Web Gazelle

    Web Gazelle Well-Known Member

    Messages:
    3,590
    Likes Received:
    259
    Best Answers:
    0
    Trophy Points:
    155
    #16
    I have a site that is currently getting tens of thousands of Yahoo bot visits a day.
     
    Web Gazelle, Jul 18, 2006 IP
  17. liquidboy

    liquidboy Peon

    Messages:
    281
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #17
    same.. i get heaps of yahoo lovin
     
    liquidboy, Jul 18, 2006 IP
  18. BoneHead

    BoneHead Peon

    Messages:
    19
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #18
    I will start feeling the love once #indexed_pages > 1

    It seems odd yahoo has been visiting since mid-april and still only 1 page. Forget about 2 - 3 weeks, I am talking 12+ weeks.

    I wonder what yahoo spider does with the pages it indexes.
     
    BoneHead, Jul 19, 2006 IP