1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Site with no robots.txt

Discussion in 'robots.txt' started by bruce-banner, Nov 14, 2008.

  1. #1
    Hi. I have this website and I just realized that it's got no robots.txt!

    Even without it, all content is indexed.

    As I understand, the robots.txt file helps the search engine to index your site so I thought that uploading the robots file would be better for indexing and that it could help with my rankings.

    Maybe with a robots.txt file in place, the "quality" of the site as a whole would be better and could rank better than other sites.

    Am I right?
     
    bruce-banner, Nov 14, 2008 IP
  2. Shadab

    Shadab Peon

    Messages:
    376
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #2
    You would need a robots.txt file if you want to block Search Engines from crawling / indexing certain sections of your site.

    If you don't have irrelevant / noise content on your website, or you don't have any duplicate content/URL issues (for example, if you have a dynamic PHP/MySQL driven site); then I think theres no need for you to have a robots.txt file.
     
    Shadab, Nov 16, 2008 IP
  3. savvisg

    savvisg Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    you do not need a robots.txt to index your pages in search engine
     
    savvisg, Nov 16, 2008 IP
  4. pareshprince

    pareshprince Active Member

    Messages:
    156
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    66
    #4
    robots.txt file will help you which pages you don't want to index in particular crawler. by default it will index all thing which concern with your website.
     
    pareshprince, Nov 18, 2008 IP
  5. a3d

    a3d Active Member

    Messages:
    409
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    60
    #5
    may I ask here?
    is robot.txt can block some ip?
    and prevent from hackers...?
     
    a3d, Nov 18, 2008 IP
  6. pareshprince

    pareshprince Active Member

    Messages:
    156
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    66
    #6
    good question rise here
    but
    as per my knowledge google not block IP. robots.txt is help us only related with our website root folder.

    but some experience person give us proper answer for your question.
    awaiting for more reply
     
    pareshprince, Nov 18, 2008 IP
  7. a3d

    a3d Active Member

    Messages:
    409
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    60
    #7
    I thought that I have read the article about that..
    I waiting for the expert also to confirm that.. :D
     
    a3d, Nov 18, 2008 IP
  8. share_boy

    share_boy Active Member

    Messages:
    71
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #8
    Sometimes you don't want a particular spider to index your site for some reason or other. Perhaps the robot is ill-behaved and spiders your site at such a high speed that it takes down your entire server. Or perhaps you prefer that you don't want the images on your site indexed in an image search engine. With a robots.txt file.
     
    share_boy, Nov 22, 2008 IP
  9. davejug1

    davejug1 Banned

    Messages:
    374
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    0
    #9
    You would block the IPs or IP ranges with .htaccess, robots.txt is purely to disallow spiders access to certain pages within your site so, as an example, you want your "thank you for your purchase" page to remain hidden then you would block robots from indexing it.

    There are two major problems with robots.txt:

    1) Anyone with a little knowledge can access the information you want hidden simply by looking at your robots file (which is always in the root directory) therefore in attempting to hide your pages you are in fact showing them. Case in point : http://www.whitehouse.gov/robots.txt

    2) Malicious spiders will ignore the robots.txt file and index your "hidden" pages anyway.

    A solution would be to add the following line to your html header

    <META NAME="ROBOTS"  CONTENTS=" noindex ,nofollow,noarchive">
    Code (markup):
     
    davejug1, Nov 22, 2008 IP
  10. yajur

    yajur Banned

    Messages:
    193
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    so i dont upload robots.txt i will cause any problem in indexing?
     
    yajur, Nov 26, 2008 IP
  11. WebCatch

    WebCatch Peon

    Messages:
    16
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    As others have said the robots.txt file isn't used for simple indexing purposes, it's to define parts of your site which you don't want crawled.
     
    WebCatch, Nov 26, 2008 IP
  12. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,034
    Likes Received:
    432
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Not only that WebCatch, but having one also helps keep a site's server logs free of unnecessary 404 errors (same as having a favicon.ico file).
     
    Dan Schulz, Nov 28, 2008 IP
  13. adamhk

    adamhk Member

    Messages:
    41
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    43
    #13
    robots.txt is primary for telling search engines to not index set paths of your webpages, not oposite :)

    so page without robots.txt is ok
     
    adamhk, Nov 29, 2008 IP
  14. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,684
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #14
    Google will index your web pages by default if you are not using robots.txt...robots.txt is used in opposite case, where you don't want to index some private pages over search engines..:)
     
    manish.chauhan, Jan 20, 2009 IP
  15. ggmittal

    ggmittal Guest

    Messages:
    27
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #15
    having no robots.txt means google bot will crawl each and every page of your site...
     
    ggmittal, Feb 17, 2009 IP
  16. bankesyjp

    bankesyjp Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #16
    <META NAME="ROBOTS"  CONTENTS=" noindex ,nofollow,noarchive">
    Code (markup):
    Thanks for that
     
    bankesyjp, Feb 18, 2009 IP
  17. shailendra

    shailendra Peon

    Messages:
    1,225
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #17
    robots.txt file do not help in better indexing of the website. it restricts the good robots from crawling the web pages you specify. moreover, when a robot comes to the website the first thing it looks for is robots.txt file. on not finding the file it returns the File Not Found error. It is also recommended specifying the xml sitemap address in the robots file so that robot can move on to the xml sitemap and crawl and index the urls. but submitting your sitemap to Google Webmaster tools is enough
     
    shailendra, Feb 19, 2009 IP