1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Block a folder except some URL's

Discussion in 'robots.txt' started by soseooooo, Mar 9, 2016.

  1. #1
    Hi,

    I have a folder /FR/ with 2607 useless duplicated URL's i need to block with robot.txt and 130 i will redirect in 301.
    Should i enter 2607 URL's line by line or there is a way to write in robot.txt :
    - Block all the folder /FR/ except these 130 URL's
    SEMrush
    Thanks !
     
    soseooooo, Mar 9, 2016 IP
    SEMrush
  2. MayurKawale

    MayurKawale Well-Known Member

    Messages:
    151
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    108
    #2
    Yes you can Disallow the folder, so all pages from these folder will get Disallow. Here is the syntax.

    User-agent:*
    Disallow: /FR/

    You can crosscheck in Google WebmasterTool in Robots.txt section
     
    MayurKawale, Jun 29, 2016 IP