How crawler crawls a website

Discussion in 'Search Engine Optimization' started by choicefreedom2000, Oct 29, 2007.

Thread Status:
Not open for further replies.
0
  1. #1
    Hi friends
    Merely tell me that my bandwith for the month sep of my website is showing 700mb in logfile analyser i.e advanced statistics of a of xyz website so i want to know that is it possible that merely crawling my website by xyz website crawler or spider or robot results in 700mb bandwith or these pages are visited by the actual person who comes to buy our product from xyz website.

    Thanks in advance
     
    choicefreedom2000, Oct 29, 2007 IP
  2. frankcow

    frankcow Well-Known Member

    Messages:
    4,859
    Likes Received:
    265
    Best Answers:
    0
    Trophy Points:
    180
    #2
    AWstats should be able to give you that info. Is it installed on your host?
     
    frankcow, Oct 29, 2007 IP
  3. excaliburwebdesigners

    excaliburwebdesigners Peon

    Messages:
    501
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Like frankcow say check in cpanel for awstats.

    If it any idication my google bot stats for this month is

    Hits=15368+46 / Bandwidth =482.11 MB

    I get around 13k UV Per Month and the sites using around 150 meg or bandwidth per hour.
     
    excaliburwebdesigners, Oct 29, 2007 IP
  4. cosmocentral

    cosmocentral Banned

    Messages:
    349
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #4
    you can use robots.txt to specify what files/folders crawlers should not read. That way you can save lot of unnecessary bandwidth usage.
    You can look into this article how to setup robots.txt cosmocentral.com/page/robotstxt-Setup-for-Search-Engine-Spiders.aspx
     
    cosmocentral, Oct 29, 2007 IP
Thread Status:
Not open for further replies.