How to make robots.txt file for my site

Discussion in 'robots.txt' started by Armaan143, Oct 24, 2007.

  1. #1
    I have site sujangarh.net
    it use a cPanel ( site is not made yet)
    i have addon domian
    rundir.com .---this is working fine but
    but it also work from rundir.sujangarh.net and sujangarh.net/rundir.com

    so i dont want serach engine to understand them as as duplicate URL
    what should i do
     
    Armaan143, Oct 24, 2007 IP
  2. Colbyt

    Colbyt Notable Member

    Messages:
    3,224
    Likes Received:
    185
    Best Answers:
    0
    Trophy Points:
    210
    #2
    Never link from the main domain to to the sub or the directory.

    If you link, link to the addon domain.com only.

    It has never been a problem.

    If you still can, I would edit your initial post and replace the . with dot.
     
    Colbyt, Oct 24, 2007 IP
  3. wisdomtool

    wisdomtool Moderator Staff

    Messages:
    15,826
    Likes Received:
    1,367
    Best Answers:
    1
    Trophy Points:
    455
    #3
    Just use a text editor

    # robots.txt
    User-agent: *
    Disallow:
    Disallow: /cgi-bin/

    Something like above, save it as robots.txt and upload to your base directory
     
    wisdomtool, Oct 24, 2007 IP
  4. Kuldeep1952

    Kuldeep1952 Active Member

    Messages:
    290
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    60
    #4
    In the root of your domain sujangarh.net, add the following
    robots.txt file:

    User-agent: *

    Disallow: /rundir.com/
     
    Kuldeep1952, Oct 30, 2007 IP
  5. Armaan143

    Armaan143 Well-Known Member

    Messages:
    1,273
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    160
    #5
    thanks for this but it still can be accessed from rundir.sujangarh.net
    how to stop this
     
    Armaan143, Oct 31, 2007 IP
  6. Kuldeep1952

    Kuldeep1952 Active Member

    Messages:
    290
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    60
    #6
    If you want to stop all access using sujangarh.net, one of the ways
    is to put this php code at the start of your index.php

    <?php
    if (strpos($_SERVER['PHP_SELF'],'sujangarh.net' ))// Return 404 error if URL contains sujangarh.net
    {
    	header("HTTP/1.0 404 Not Found");
    	exit;
    }
    ?>
    PHP:
    This must be the start of the file before any output is sent.
     
    Kuldeep1952, Oct 31, 2007 IP
  7. Armaan143

    Armaan143 Well-Known Member

    Messages:
    1,273
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    160
    #7
    thanks for this i will try
     
    Armaan143, Oct 31, 2007 IP
  8. webgk.com

    webgk.com Peon

    Messages:
    96
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Why don't you redirect[301] your subdirectory[folder] to your subdomain. I think that would be the best solution. Then you will be free from duplication and using any URL to link. This can be done through mod_rewrite rule in .htaccess file.
     
    webgk.com, Nov 2, 2007 IP
  9. Armaan143

    Armaan143 Well-Known Member

    Messages:
    1,273
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    160
    #9
    I have tried this.
    I am getting problem in this
     
    Armaan143, Nov 6, 2007 IP