1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Googlebot and PHPbb forums?

Discussion in 'HTML & Website Design' started by Redleg, Aug 25, 2004.

  1. #1
    I just wondered if any of you who have spiderable PHPbb forums have noticed any change in spider activity the last 14 days??

    My forums and posts have been spidered and indexed in google for several months now without any problems, but for the last 14 days googlebot has only spidered and indexed the index.php page, none of the forums or new posts appears to have been visited at all. (last ones from 10th August)
    Googlebot used to re-index the forums (and most of the new posts) at least once every week.
    SEMrush
    I've done no changes to the forum, and it doesn't use sessions for guests/spiders.
    I've also tried several different spider simulators, and none of them have any problems, so I can't understand why googlebot doesn't like me anymore...

    Anyone else who has experienced this?

    You can find my forum here:
    http://www.military-quotes.com/forum/
     
    Redleg, Aug 25, 2004 IP
    SEMrush
  2. payoutwindow

    payoutwindow Peon

    Messages:
    788
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Well ... its gonna take a bit of work to get it to be search engine friendly.

    You can find a step by step guide at PhpBB Hacks
     
    payoutwindow, Aug 25, 2004 IP
  3. Redleg

    Redleg Raider

    Messages:
    360
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #3
    That was not the question, it is search engine friendly (or at least it was).
    I've installed those mods before (+ several I've made myself).
    20,000+ pages indexed in google, but it stopped indexing everything except the index.php page around the 10th August...

    I just wondered if anyone else had noticed this, or is googlebot just having a little break from my forum now?
     
    Redleg, Aug 25, 2004 IP
  4. GuyFromChicago

    GuyFromChicago Permanent Peon

    Messages:
    6,731
    Likes Received:
    528
    Best Answers:
    0
    Trophy Points:
    0
    #4
    How do you know it's not indexing your pages? Is it not showing a cache for new(er) posts & threads?
     
    GuyFromChicago, Aug 25, 2004 IP
  5. jontelofot

    jontelofot Peon

    Messages:
    53
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    A while ago (can't remember how long) googlebot changed it's User Agent from:
    Googlebot/2.1 (+http://www.googlebot.com/bot.html)
    to
    Googlebot/2.1 (+http://www.google.com/bot.html)

    Maybe your SID hack doesn't recognize the bot correctly anymore...
     
    jontelofot, Aug 25, 2004 IP
  6. Help Desk

    Help Desk Well-Known Member

    Messages:
    1,365
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    180
    #6
    I applied the hack to mine where guests don't need an SID. It was very easy and is documented in phpBB's forums.
     
    Help Desk, Aug 25, 2004 IP
  7. Redleg

    Redleg Raider

    Messages:
    360
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #7
    No, and the cache for each of the forums shows that the newest posts in there are from 10-12 August.
    Example of cahced forum:
    http://66.102.9.104/search?q=cache:...ary-quotes.com/forum/viewforum.php?f=9+&hl=en
    Real forum:
    http://www.military-quotes.com/forum/viewforum.php?f=9


    The SID hack I use removes SIDs from all guests, not just the bots, so that should not be the problem..
    I've also tried different spider simulators on the posts/forums, and they look ok.. (No "sid" at the end of the links)
    http://www.submitexpress.com/analyzer/

    Edit: tried to install another spider hack as well now (that looks very good by the way)
    http://www.phpbb.com/phpBB/viewtopic.php?t=200361
    We'll see if that one works.
     
    Redleg, Aug 25, 2004 IP
  8. GuyFromChicago

    GuyFromChicago Permanent Peon

    Messages:
    6,731
    Likes Received:
    528
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Damn Bots:)

    Have you noticed any decrease in activity from the other search engine bots?

    From time to time I do notice a "lag" in the updating of the cached pages in my forums. I've never really kept track of the length of the delay or tried to isolate the cause, as Google always seems to make up for lost time when I least expect it. Most of the time new posts/pages are indexed and the cache is visible with in a couple days…but like I said every so often Google seems to take a break. I wouldn't really be concerned unless a lot of time, like 4 - 6 weeks, passed without noticeable updates.

    One thing I've been doing that seems to help with the indexing of posts in my forums is mentioning, and linking to, specific threads within the forum from one of my blogs or from another site I own.
     
    GuyFromChicago, Aug 25, 2004 IP
  9. Redleg

    Redleg Raider

    Messages:
    360
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Googlebot does visit and index the index.php page every 2-3 days.

    I've made a small sitemap (in html) on my site some days ago, linking to each forum and some of the posts, maybe that helps..

    I've just "discovered" some posts from the 15th that's cached, maybe googlebot's just having a week or two off now. :)

    That bot tool I mentioned is actually pretty cool..
    MSN has visited my forum, and spidered 175 pages since my last post here.. :)

    Anyone here who knows all of the IP addresses googlebot uses by the way??
     
    Redleg, Aug 25, 2004 IP
  10. Redleg

    Redleg Raider

    Messages:
    360
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Looks like this thread may have brought back googlebot from it's little vacation... :D

    Since my last post googlebot have visited the forum 4 times and spidered over 400 pages.
    MSNbot has also been quite active, over 600 pages spidered.


    Edit: Make that 5276 pages... :eek:
    Googlebot is REALLY back from it's vacation now...

    Another Edit: 6500+ pages, and still going strong... :D
     
    Redleg, Aug 26, 2004 IP
  11. TLDTrader.com

    TLDTrader.com Peon

    Messages:
    187
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Redleg, does it come with a step by step installation guide? I haven't installed a mod before.

    Are there no major bugs?

    Thanks.
     
    TLDTrader.com, Aug 28, 2004 IP
  12. Redleg

    Redleg Raider

    Messages:
    360
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Sorry, didn't see your question until now...

    It's real easy to install, and everything should be explained in the .txt file that comes with the mod.

    Feel free to PM me if you have any questions about it.
     
    Redleg, Sep 1, 2004 IP
  13. TLDTrader.com

    TLDTrader.com Peon

    Messages:
    187
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #13
    Thanks a lot :)

    Will be sending PM.
     
    TLDTrader.com, Sep 1, 2004 IP