1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Help with robots.txt file

Discussion in 'robots.txt' started by Divvy, Jun 14, 2017.

  1. #1
    Hello guys,

    I need some help here...
    I have a wordpress site with a robots.txt file that has the following content:
    SEMrush
    
    User-agent: *
    Sitemap: https://domain.com/sitemap.xml
    Disallow: /go/
    Disallow: /temp/
    Disallow: /reviews/*
    Disallow: /out/*
    
    Code (markup):
    And I have a blog with this URL:
    https://domain.com/blog
    Code (markup):
    And all articles of the blog uses this URL:
    https://domain.com/news/title-here
    Code (markup):
    What I need is to disallow only the url:
    https://domain.com/news/
    Code (markup):
    I have tried this:
    Disallow: /news/
    Code (markup):
    and this:
    Disallow: /news
    Code (markup):
    But disallows all my blog articles...
    Is there anything I can do to disallow only the /news/ and not the /news/* ?

    Unfortunately I think is not possible in wordpress /blog for root and /blog/* for articles :/

    Help... :)

    Thanks!!
     
    Divvy, Jun 14, 2017 IP
    SEMrush
  2. robert4u

    robert4u Greenhorn

    Messages:
    46
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    8
    #2
    This will disallow the link:
    Disallow: /news

    /news/ means the whole folder, while keep in mind you cannot check instantly, you will need to wait untill the google index again.
     
    robert4u, Jun 14, 2017 IP
  3. robert4u

    robert4u Greenhorn

    Messages:
    46
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    8
    #3
    One more thing, also remove the link from sitemap.
     
    robert4u, Jun 14, 2017 IP
  4. Divvy

    Divvy Well-Known Member

    Messages:
    719
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    128
    #4
    Hello,

    Thank you for helping me, but I already tried that and didn't worked.
    Please take a look to this screenshot:

    ss.png

    Any more ideas?
     
    Divvy, Jun 15, 2017 IP
  5. Arun Kumar Soni

    Arun Kumar Soni Active Member

    Messages:
    266
    Likes Received:
    15
    Best Answers:
    2
    Trophy Points:
    90
    #5
    Why you always going to Create /news/title1
    /news/title2
    /news/title3


    Go with
    /title1
    /title2
     
    Arun Kumar Soni, Sep 24, 2017 IP