robots.txt directive

Discussion in 'Search Engine Optimization' started by jeanmichel, Feb 8, 2011.

  1. #1
    Hi all,

    quick question, i have a site which uses a script to serve a lot of pages including search pages, most of which i don't want indexed. This would be dealt with in the robots file as such.

    robots.txt
    Disallow: /pages.php
    Code (markup):
    .htaccess
    rewriterule ^useful-page.html$ ^/pages.php?id=1&type=content [L]
    Code (markup):
    If in my .htaccess file i'm rewriting some urls which use the pages.php script, the question is are they also blocked by the search engines?

    i.e. will /useful-page.html be blocked from search engine spiders?

    It's late at night and my mind has gone blank when it comes to working outside of wordpress!!!
     
    jeanmichel, Feb 8, 2011 IP