1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

WebmasterWorld Out Of Google

Discussion in 'General Chat' started by pachecus, Nov 23, 2005.

0
Page 7 of 8
  1. DomainMagnate

    DomainMagnate Illustrious Member

    Messages:
    10,932
    Likes Received:
    1,022
    Best Answers:
    0
    Trophy Points:
    455
    #121
    I think his site costs much more than that even now.
    Google still doesn't rule the world, not even the internet.
    These guys can afford being out of the index..
     
    DomainMagnate, Nov 28, 2005 IP
  2. Crazy_Rob

    Crazy_Rob I seen't it!

    Messages:
    13,157
    Likes Received:
    1,366
    Best Answers:
    0
    Trophy Points:
    360
    #122
    from the link DA posted:

    I think anyone that has been around for a while realizes that .txt is usually ignored. Especially by "rogue bots" and rippers. :rolleyes:
     
    Crazy_Rob, Nov 28, 2005 IP
  3. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #123
    There's still the strange choice of using a robots.txt file -- think about it: what he has done is ban all the good bots that pay attention to robots.txt and hand the forum over to the bad bots that don't.

    If he doesn't want ANY bots, he's done it the wrong way. If he doesn't want any BAD bots, he's done it the wrong way.

    If he just wants to ban Google, MSNSearch, Yahoo!, Ask Jeeves, and the others that obey robots.txt, then he's done it the right way.

    That interview doesn't change my mind about that - it doesn't address the question at all.

    Edit: Damn that fast-typing Crazy_Rob! :D
     
    minstrel, Nov 28, 2005 IP
  4. Design Agent

    Design Agent Peon

    Messages:
    3,061
    Likes Received:
    154
    Best Answers:
    0
    Trophy Points:
    0
    #124
    Im confused. From the article the problem is too many spiders - but he is trying to make the site more spiderable according to that article. :confused:
     
    Design Agent, Nov 28, 2005 IP
  5. Shoemoney

    Shoemoney $

    Messages:
    4,474
    Likes Received:
    588
    Best Answers:
    0
    Trophy Points:
    295
    #125
    Here is what I do. I have a hidden link to a php file on my page. This php file is forbidden of course in my robots.txt.That file records all the ips to a text file which is checked every 5 mins by a cron job that blocks those ips in iptables.

    Now it gets a little bit trickier when you have people using random proxies but its still done...

    I currently have over 1 million wav files and several hundred thousand gifs on a server so trust I have experience with this ;)
     
    Shoemoney, Nov 28, 2005 IP
  6. Edz

    Edz Peon

    Messages:
    1,690
    Likes Received:
    72
    Best Answers:
    0
    Trophy Points:
    0
    #126
    Am i asking the impossible if you could make a tutorial on this Shoemoney?

    Your method sure sounds good.

    If you don't want or have the time to do a tut on this do you know a link or something that can help us that want to block out these type of bots and crawlers?
     
    Edz, Nov 28, 2005 IP
  7. markhutch

    markhutch Peon

    Messages:
    357
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    0
    #127
    I just checked alexa and his new traffic is down tremendously, almost one million per day if I'm reading the thing right and the trend looks like the NASDAQ did during the 1999 - 2000 stock market crash. Sure any forum can survive without search engine traffic. They might just survive a lot smaller which appears to be the goal here.
     
    markhutch, Nov 28, 2005 IP
  8. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #128
    WebmasterWorld has started to allow bots to spider again. Although their robots.txt file is cloaked, so you will only see it if you spoof your user agent.

    Was a fun month traffic-wise for digitalpoint.com though. :)

    Just goes to show what the difference between one ranking in search results can do for ya (there are thousands of terms where digitalpoint.com is 2nd to webmasterworld.com). ;)

    [​IMG]
     
    digitalpoint, Dec 17, 2005 IP
    Blogmaster likes this.
  9. Shoemoney

    Shoemoney $

    Messages:
    4,474
    Likes Received:
    588
    Best Answers:
    0
    Trophy Points:
    295
    #129
    well dp is still owning =P
     
    Shoemoney, Dec 17, 2005 IP
  10. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #130
    Will be interesting to see when they get fully indexed though. :)
     
    digitalpoint, Dec 17, 2005 IP
  11. Nintendo

    Nintendo ♬ King of da Wackos ♬

    Messages:
    12,890
    Likes Received:
    1,064
    Best Answers:
    0
    Trophy Points:
    430
    #131
    And it looks like DP is officially the most populer webmaster site on the internet, since WMW will keep crashing...

    [​IMG]

    I posted a link over there to there Alexa stats page showing there numbers crash in half, and with in two minutes...the post was deleted!!!! :D:D:D
     
    Nintendo, Dec 17, 2005 IP
  12. Shoemoney

    Shoemoney $

    Messages:
    4,474
    Likes Received:
    588
    Best Answers:
    0
    Trophy Points:
    295
    #132
    yes lets all make sure we blog about it. Seriously ;)
     
    Shoemoney, Dec 17, 2005 IP
  13. markhutch

    markhutch Peon

    Messages:
    357
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    0
    #133
    I wonder if Google will put them in the "sandbox" for awhile! :) Sorry, I couldn't help myself.
     
    markhutch, Dec 17, 2005 IP
  14. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #134
    Hehe... nope. Their traffic will be back to normal within a week or two.

    Perfect example though...

    [search=google]adsense forum[/search]
     
    digitalpoint, Dec 17, 2005 IP
  15. markhutch

    markhutch Peon

    Messages:
    357
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    0
    #135
    I noticed they are already showing over 280 thousand pages indexed in Google. However, most are just line enteries without a description. I think they had over two million internal pages indexed in Google before they put up the blanket "robots.txt" ban page. You may be right about it only taking a couple of weeks to get reindexed, but Google, Yahoo and MSN are going to eat up a bunch of bandwidth the next couple of weeks just getting many of the old archived pages back into the index.
     
    markhutch, Dec 17, 2005 IP
  16. Blogmaster

    Blogmaster Blood Type Dating Affiliate Manager

    Messages:
    25,924
    Likes Received:
    1,354
    Best Answers:
    0
    Trophy Points:
    380
    #136
    Their IBLs are very old and have grown steadily over the years, it will take a little longer to pass them by.
     
    Blogmaster, Dec 17, 2005 IP
  17. Nintendo

    Nintendo ♬ King of da Wackos ♬

    Messages:
    12,890
    Likes Received:
    1,064
    Best Answers:
    0
    Trophy Points:
    430
    #137
    Nintendo, Dec 22, 2005 IP
  18. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #138
    Rank favoritism... :eek:
     
    minstrel, Dec 22, 2005 IP
    Will.Spencer likes this.
  19. Dekker

    Dekker Peon

    Messages:
    4,185
    Likes Received:
    287
    Best Answers:
    0
    Trophy Points:
    0
    #139
    i bet their admin has soft pillow lips.
     
    Dekker, Dec 22, 2005 IP
  20. fryman

    fryman Kiss my rep

    Messages:
    9,604
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    370
    #140
    I noticed that all my flash files in my swf folder were showing up doing a site: check, so I added a disallow parameter in my robots.txt

    That was over a week ago, and absolutely nothing has changed... so I agree with Minstrel, they seem to be getting a special treatment from Google
     
    fryman, Dec 22, 2005 IP
Page 7 of 8