1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is it good to block BaiduSpider from accessing/crawling the site?

Discussion in 'robots.txt' started by MayurKawale, Jan 7, 2021.

0
  1. #1
    Hi all

    I saw a lot of sites that won't allow the baiduspider to access/crawl the site. Can you describe what might be the reason for this. Even the Digital Point site (TLD) is not allowing the BaiduSpider to crawl the site.
    i.e
    User-agent: BaiduSpider
    Disallow: /
     
    MayurKawale, Jan 7, 2021 IP
  2. qwikad.com

    qwikad.com Illustrious Member Affiliate Manager

    Messages:
    7,151
    Likes Received:
    1,656
    Best Answers:
    29
    Trophy Points:
    475
    #2
    If I am not mistaken it's an aggressive spider. If you're on a shared hosting but have a rather large site, you may want to restrict it. Otherwise you don't have to.
     
    qwikad.com, Jan 10, 2021 IP
    MayurKawale likes this.