Slow Down Google Bot to Reduce Server Strain

Discussion in 'Google' started by websiteideas, Aug 13, 2008.

0
  1. #1
    Is there anything that I can put in my robots.txt file so that our server is not stressed out when Google comes to spider several links at the same time?
     
    websiteideas, Aug 13, 2008 IP
  2. shabbirbhimani

    shabbirbhimani Well-Known Member

    Messages:
    1,063
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    140
    #2
    I would never do that but then you can use Meta for doing such things. Its always good to pay for more bandwidth for Google Bot
     
    shabbirbhimani, Aug 13, 2008 IP
  3. live-cms_com

    live-cms_com Notable Member

    Messages:
    3,128
    Likes Received:
    112
    Best Answers:
    0
    Trophy Points:
    205
    Digital Goods:
    1
    #3
    Google webmaster tools lets you do that.

    GoogleBot was visiting one of my sites 5000 times per day so DreamHost complained to me, so I had to slow them. :(
     
    live-cms_com, Aug 13, 2008 IP
  4. rjpzf

    rjpzf Well-Known Member

    Messages:
    123
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    113
    #4
    I think there's a feature available in Google Webmaster Tools that can allow you to do this to a limited extent?
     
    rjpzf, Aug 13, 2008 IP
  5. scriptglue

    scriptglue Banned

    Messages:
    170
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Yes.. Webmaster Tools WILL give you the option to control the bots activities.

    Alternatively, you can also do some work on your own server to make it handle your site better. Like enabling some cache features.

    Let us know how it goes.
     
    scriptglue, Aug 14, 2008 IP
  6. derral

    derral Banned

    Messages:
    52
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Hey rjpzf, You Need to send me the down load link please.

    I haven't received the down load link for:
    EZ CASH$-282 Emails/Day-Traffic/Cash EXPLOSION-Better than PPC,SEO,Articles,etc-VIRAL


    Would you please send it.


    Thank you,
    Derral


    =============================================================

    -----Original Message-----
    From: <service@paypal.com>
    To: derrall <groves71955@aol.com>
    Sent: Tue, 30 Sep 2008 3:27 pm
    Subject: Receipt for Your Payment to RP

    Dear derrall,

    This email confirms that you have paid RP$37.00 USD using PayPal.

    This credit card transaction will appear on your bill as "PAYPAL *RP".

    ----------------------------------------------------------------

    -----------------------------------
    Payment Details
    -----------------------------------

    Transaction ID: 6GG02464YM165712B
    Item Price: $37.00 USD
    Total: $37.00 USD
    Order Description: Viral Marketing Project
    Item/Product Number: 1
    Buyer: derrall

    ----------------------------------------------------------------


    It may take a few moments for this transaction to appear in the Recent Activity
    list on your Account Overview.
    -----------------------------------
    Business Information
    -----------------------------------

    Business: RP
    Contact E-Mail:

    If you have questions about the shipping and tracking of your purchased item or
    service, please contact RP at .


    ----------------------------------------------------------------

    Thank you for using PayPal!
    The PayPal Team


    PayPal Email ID PP120
     
    derral, Oct 5, 2008 IP