1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt Question

Discussion in 'robots.txt' started by PaRaDiGm13, Mar 16, 2016.

0
  1. #1
    Hello,

    What does it mean in robots.txt ?

    User-agent: *
    Disallow: /
    Allow: /$

    Does it only allow index page domain?

    Also if i add:

    Allow: /category/

    Will it allow http://www.domain.com/category/catname/ ?

    Thank you.
     
    PaRaDiGm13, Mar 16, 2016 IP
  2. Anusha55

    Anusha55 Active Member

    Messages:
    123
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    73
    #2
    I am a bit confused in first as you already disallow the all page and then adding the page with $ express i don't know whether it will allow only index page.

    second one is clear allow category but it doesn't means disallow other pages.
     
    Anusha55, Mar 23, 2016 IP
  3. MayurKawale

    MayurKawale Well-Known Member

    Messages:
    221
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    170
    #3
    User-agent: *
    Disallow: / - (Disallow to all search engine for crawling)
    Allow: /$ - (There is ambiguity, what you want to allow ... be specific)

    Allow: /category/ - (Allow to all search engine for crawling)

    Check all details here http://www.robotstxt.org/
     
    MayurKawale, Aug 4, 2016 IP
  4. Alex D

    Alex D Greenhorn

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    6
    #4
    Allow: /category/ is definitely will allow all search engine to crawl http://www.domain.com/category/catname/
     
    Alex D, Aug 10, 2016 IP
  5. Cado Magenge

    Cado Magenge Greenhorn

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #5
    a) User Agent: *
    means all robots or search engines crawlers.
    b) Disallow: /
    means not allow whole website to crawl by crawlers.
    c) Allow: /$
    $ is wild card character to specify matching the end of the URL. To block an URLs that end with .asp, you could use the following entry:
    User-agent: Googlebot
    Disallow: /*.asp$

    Hope it will help you and enhance your knowledge.
    Have a nice time.;):);)
     
    Cado Magenge, Feb 28, 2017 IP