1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Where we use robot.txt in SEO?

Discussion in 'Search Engine Optimization' started by AndresNeil, Mar 21, 2015.

0
  1. #1
    Where we use robot.txt in SEO?
     
    AndresNeil, Mar 21, 2015 IP
  2. ICA James

    ICA James Member

    Messages:
    45
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #2
    we should use robot.txt if we want to restrict crawlers to crawl any private pages. You can do it using notepad file and upload it in your server. Thanks
     
    ICA James, Mar 21, 2015 IP
  3. braulio

    braulio Active Member

    Messages:
    70
    Likes Received:
    7
    Best Answers:
    1
    Trophy Points:
    95
    #3
    robots.txt tells google, yahoo, bing crawlers what not to crawl ( take an inventory of ) on your pages. Robots.txt works in conjunction with your sitemap.xml file. The sitemap.xml tells the crawlers what pages you have and where they are located. If you do not have a robots.txt file, the crawler crawls all your site. If your robots.txt file has page A as a restriction and you declare page A on your sitemap.xml, you get a crawl error on your Google Webmaster Tools panel.

    An example of how we use it is the following.

    We have a test directory on our server where we upload sites to be tested before we go public. Obviously, we do not want the crawlers to take note ( inventory ) of this directory since it is a directory for testing only. Below is the robots.txt file contents.

    # www.robotstxt.org/

    User-agent: *

    Disallow:/testing/​

    Good luck AndresNeil.

    Braulio
     
    braulio, Mar 21, 2015 IP
  4. Md Tanvir Ahmed

    Md Tanvir Ahmed Member

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #4
    robot.txt is one kind of things that will help you on crawling in search engine. To add a robot.txt for your web just do the following thing :
    Google Webmaster link
    log in and add your site .Then Click on submit then click on 3 number option and you are done .
    See the picture :
    [​IMG]
     
    Last edited: Mar 21, 2015
    Md Tanvir Ahmed, Mar 21, 2015 IP