1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

May I use this...?

Discussion in 'Google Sitemaps' started by M-BenzForum.com, May 24, 2006.

  1. #1
    Folks,

    I have been using mod_rewrite and Guest Sessions mod. (http://www.my.com/....html)
    SEMrush
    May I use Google Sitemaps? are there problem if I use it (because I have been using mod_rewrite and Guest Sessions mod)?

    Also, Do I need <meta name="Googlebot" Content="index,follow">

    Thanks for help.
     
    M-BenzForum.com, May 24, 2006 IP
    SEMrush
  2. Slapyo

    Slapyo Well-Known Member

    Messages:
    266
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    108
    #2
    By default the bot will index and follow, you don't need to put that meta tag in. As long as you have a robots.txt file for the bot to read so it knows which pages should not be crawled you are ok.
     
    Slapyo, May 24, 2006 IP
  3. M-BenzForum.com

    M-BenzForum.com Peon

    Messages:
    221
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    This is my robots.txt file :

    User-agent: *
    Disallow: /admin/
    Disallow: /db/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /language/
    Disallow: /templates/
    Disallow: /index.php
    Disallow: /common.php
    Disallow: /config.php
    Disallow: /faq.php
    Disallow: /groupcp.php
    Disallow: /login.php
    Disallow: /memberlist.php
    Disallow: /modcp.php
    Disallow: /posting.php
    Disallow: /privmsg.php
    Disallow: /profile.php
    Disallow: /search.php
    Disallow: /viewonline.php
    Disallow: /linktous.php
    Disallow: /contact.php
    Disallow: /partner.php

    is this right? I want to avoid a duplicate pages...

    thanks
     
    M-BenzForum.com, May 24, 2006 IP
  4. AmmoX

    AmmoX Guest

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Looks good.

    /directory/
    /page.html

    What you have will prevent indexing of everything listed. All content in the listed directories and the individual pages.

    You can add to it as needed

    Good job, post that sucker :D
     
    AmmoX, May 24, 2006 IP
  5. M-BenzForum.com

    M-BenzForum.com Peon

    Messages:
    221
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    M-BenzForum.com, May 24, 2006 IP
  6. DP Most

    DP Most Well-Known Member

    Messages:
    478
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    108
    #6
    Can any body tell me benifits of robots.txt files?
     
    DP Most, May 24, 2006 IP
  7. AmmoX

    AmmoX Guest

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Take a look at M-Benz's file below.

    The User Agent line sets the file up to be used by all bots (search engines).

    Then it lists directories that they should not go into to index and then specific pages to skip.

    Basically you list any directory or page that you don't want indexed. Include files and directories, CGI, admin, etc...

    In his case he listed specific pages such as login and others that he didn't want indexed as well.

    This prevents robots and bots from listing pages and content in the list so that they wont get indexed.

    Do a search on robots.txt there are plenty of tutorials out there.
     
    AmmoX, May 24, 2006 IP