1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robot.txt file crawling pages very fast

Discussion in 'robots.txt' started by Sunil Keshari, Apr 26, 2012.

0
  1. #1
    I have a video portal when i allow robot then its index very fast so my server crash
    so my hosting says remove the site but also suggest me to disallow from search engine

    so i have put these code

    but by these way my site is not index in search engine
    i have checked in webmaster there its show more then 300000 links are block(Googlebot is blocked from http://www.bwithyou.co.in/)

    I have also set google webmaster crawling (custom) minimum
    suggest me some one that what robot i use that my site get index in search engine and not consume more cpu usage.
    waiting for some positive reply
    Thanks
     
    Sunil Keshari, Apr 26, 2012 IP
  2. sarahk

    sarahk iTamer Staff

    Messages:
    28,500
    Likes Received:
    4,460
    Best Answers:
    123
    Trophy Points:
    665
    #2
    You have to allow some in but it may just be that you have selected cheap hosting on an overloaded server or a poor script. Hostgator, for instance, get ratty if you run openx on their servers - good hosting but they don't like resource heavy scripts. You may have to consider changing hosting or changing scripts.

    Unless you are being targetted by a rogue spider for some reasons (your logs will tell you which one) then you shouldn't be getting any more spider visits than anyone else..
     
    sarahk, May 1, 2012 IP
  3. Sunil Keshari

    Sunil Keshari Well-Known Member

    Messages:
    247
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    170
    #3
    Thank you for the nice tip but regarding hosting
    i am already using Hostgator services.
    Website is hosted on Hostgator server.
     
    Sunil Keshari, May 1, 2012 IP
  4. sarahk

    sarahk iTamer Staff

    Messages:
    28,500
    Likes Received:
    4,460
    Best Answers:
    123
    Trophy Points:
    665
    #4
    So what is the script they object to?
     
    sarahk, May 1, 2012 IP
  5. Sunil Keshari

    Sunil Keshari Well-Known Member

    Messages:
    247
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    170
    #5
    the script is php ( script name: PHPAutovideo)
     
    Sunil Keshari, May 1, 2012 IP
  6. bradleymarketer

    bradleymarketer Peon

    Messages:
    67
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    You are using CDN network, why you use Disallow: while you say that you allow all bot,
     
    bradleymarketer, Oct 29, 2012 IP
  7. selectaupairs

    selectaupairs Greenhorn

    Messages:
    59
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    8
    #7
    You use that hosting service that not use more cpu memory and if you do the disallow index definetly any search engine do not crawl your site .

    So you don't find your site in Search engine if you want to crawl by serach engine you must be allowed that.
     
    selectaupairs, Oct 30, 2012 IP