Is Cuil Killing Websites?

Discussion in 'All Other Search Engines' started by desilator, Sep 1, 2008.

  1. #1
    Is Cuil Killing Websites?
    http://www.techcrunch.com/2008/09/01/is-cuil-killing-websites/
    by Don Reisinger on September 1, 2008



    An anonymous tipster wrote to us this morning to tell us that Cuil, the ill-fated “Google Killer,” has unleashed its Twiceler indexing bot on websites across the globe and in the process, has brought many sites down.

    “I don’t know what spawned it, but when Cuil attempts to index a site, it does so by completely hammering it with traffic,” the tipster wrote. “So much, that it completely brings the site down. We’re 24 hours into this “index” of the site, and I’ve had to restrict traffic to the site down to 2 packets per second, while discarding the rest, or otherwise it makes the site unusable.”

    The Admin Zone forums are abuzz over Cuil’s overzealous method for indexing. Countless posters on the site have said that their websites have been brought down because of the Twiceler robot and one user said it “leeched enormous amounts of bandwidth — nearly 2GB this month until it was blocked. It visited nearly 70,000 times!”

    Website owners are also saying that the way Cuil indexes sites isn’t scientific in any way and is actually quite “amateurish.” According to those who experienced the Twiceler onslaught, the bot seems to “randomly hit a site and continue to guess and generate pseudo-random URLs in an attempt to find pages that aren’t accessible by links. And by doing this, they completely bring a site down to where it’s not functional.”

    Upset site owners contacted Cuil to see why Twiceler was hitting sites so often. James Akers, Cuil’s Operational Engineer responded to the issue by saying that “Twiceler is an experimental crawler that we are developing for our new search engine. It is important to us that it obey robots.txt, and that it not crawl sites that do not wish to be crawled. If you wish I will glad to add your site to our list of sites to exclude, but I need you to tell the site name to block as email return addresses frequently from the domains that wish to be blocked.”

    Akers also claims that Cuil has seen a “number of crawlers” that pretend to be Twiceler, and site owners should consult the company’s IP addresses page to determine if it’s really Cuil causing all the trouble.

    Cuil has yet to respond to a request for comment, but it doesn’t look like the pelting of sites by the company’s Twiceler bot is an isolated incident. And if it’s true that Twiceler is trying to find pages on sites that don’t even exist to simply increase the index size, Cuil should work quickly to modify the bot before it receives even more negative publicity.
     
    desilator, Sep 1, 2008 IP
  2. GRFblog

    GRFblog Peon

    Messages:
    32
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Doubtful. Such a method would bring loads of lawsuits, so if this is true we'll be hearing more about it.
     
    GRFblog, Sep 2, 2008 IP
  3. Resident79

    Resident79 Peon

    Messages:
    169
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Maybe they are testing a lot. And this could lead to some extreme results.
     
    Resident79, Sep 3, 2008 IP
  4. Ben Lambert

    Ben Lambert Peon

    Messages:
    55
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Gee they've had some terrible PR. Its a shame, I actually like their search results layout. I say give them some time to get the search results right. Google can do with more competition.
     
    Ben Lambert, Sep 3, 2008 IP
  5. bauerdude

    bauerdude Peon

    Messages:
    83
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    What?! Bad news about Cuil?!? I'm devastated. No, shocked. No, both!
     
    bauerdude, Sep 3, 2008 IP
  6. alinalin0

    alinalin0 Peon

    Messages:
    48
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Too bad about Cuil, I really hoped for a new, powerfull SE, I'm sick and tired of only Google, Google, Google...
     
    alinalin0, Sep 3, 2008 IP
  7. JasonJason

    JasonJason Banned

    Messages:
    108
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    cueil or whatever its called is rubbish, the images search is rubbish, its just pointless
     
    JasonJason, Sep 4, 2008 IP
  8. desilator

    desilator Peon

    Messages:
    2,220
    Likes Received:
    49
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Yea.. I think it was released before it was actually ready.
     
    desilator, Sep 4, 2008 IP
  9. jamesplato

    jamesplato Active Member

    Messages:
    359
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    60
    #9
    we are recommending webmasters to add delay crawl for cuil.com in robots.txt as a protection
     
    jamesplato, Sep 4, 2008 IP
  10. LakeCountry

    LakeCountry Well-Known Member

    Messages:
    509
    Likes Received:
    56
    Best Answers:
    0
    Trophy Points:
    120
    #10
    Twiceler has been around and busy for years now and has crawled my sites frequently without any problems.
     
    LakeCountry, Sep 4, 2008 IP
  11. ubamba

    ubamba Member

    Messages:
    155
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    28
    #11
    It's kind of funny all the bad pr cuil has gotten. But with a name that stupid it's not surprising.
     
    ubamba, Sep 4, 2008 IP
  12. atulperx

    atulperx Banned

    Messages:
    3,949
    Likes Received:
    196
    Best Answers:
    0
    Trophy Points:
    0
    #12
    If this news is right than they have to get prepared by large number of lawsuits and i think most of them from competitors members .
     
    atulperx, Sep 4, 2008 IP
  13. desilator

    desilator Peon

    Messages:
    2,220
    Likes Received:
    49
    Best Answers:
    0
    Trophy Points:
    0
    #13
    Agreed.. Will be interesting to see how this pans out.
     
    desilator, Sep 4, 2008 IP