1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Please Guid me

Discussion in 'robots.txt' started by mobixone, Apr 16, 2009.

0
  1. #1
    I generate & upload a google robotex.txt file with following code:

    User-Agent: *
    Allow: /
    Sitemap: <http://mobixone.com/sitemap.xml>

    i saw google bot on my site but never saw yahoo bot on my site or live bot is it any problem with robotex.txt file?
    in my yahoo site explorer there r only 3 pages they crawl.
     
    mobixone, Apr 16, 2009 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    manish.chauhan, Apr 17, 2009 IP
  3. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Generally Yahoo and especially MSN take longer to crawl a site than Googlebot.
     
    Lpe04, Apr 28, 2009 IP
  4. linkdealer

    linkdealer Active Member

    Messages:
    138
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    90
    #4
    after analyzing your robots.txt file, I found no reason that yahoo and MSN crawlers will not be following your website
     
    linkdealer, Jun 9, 2009 IP
  5. S.K.L.Technologies

    S.K.L.Technologies Banned

    Messages:
    56
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5



    yes m agree with you
     
    S.K.L.Technologies, Jun 14, 2009 IP
    earlier likes this.
  6. earlier

    earlier Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    I'm confused then, is it robot.txt or robotex.txt
    also I thought there was only disallow? :confused:

    thanks for the sitemap info

    Cher
     
    earlier, Jun 16, 2009 IP
  7. earlier

    earlier Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I mean, shouldn't it be

    User-agent: *
    Disallow:
    Sitemap: <http://mobixone.com/sitemap.xml>

    instead of

    User-Agent: *
    Allow: /
    Sitemap: <http://mobixone.com/sitemap.xml>

    This is the first time that I've had to do my own site map

    Cher
     
    earlier, Jun 16, 2009 IP
  8. seojenifer

    seojenifer Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8

    yes, I am agree with you
     
    seojenifer, Jul 6, 2010 IP
  9. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #9
    Both are same. Both would allow crawlers to visit your websites.
     
    manish.chauhan, Jul 7, 2010 IP
  10. newlogo

    newlogo Peon

    Messages:
    3,932
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #10
    manish is right i suppose, * will allow all search engine
     
    newlogo, Jul 20, 2010 IP