Hello i want to make robots.txt file for my new website

Friends i am launching my new website and i want to know from all of you that how i should make the robots.txt file or what i should disallow and what do i allow ? http://www.aaateleshoping.in










Regards,

 

okey nice i give you example for the robots.txt wait a while please

User-agent: *
Disallow:


this is working fine for my wwe blog and is going great if you want to disallow any robot like media robot of google or adsense then you can go to webmasters tool of google there you will find link to generate a robots file and i prefer just do not allow admin pages but let other go file allowed ..

 

and include the sitemap link at the bottom like this example

User-agent: *
Disallow:

Sitemap: http://www.yourdomain.com/sitemap.xml

 

That was the best you can do it.

* is used for all types useragents and robots.
Please make sure to Disallow all the un useful content to search engines. As Search engines first accesses this file to check which porition of site is not accesible.

 

even iam agreed your post

 

What are you trying to do with it?

 

if there any image or specific page that you do not want search engine to crawl then disallow it. User-agent: * Disallow: /example.html User-agent: * Disallow: /example.jpg

 

Thanks for sharing Information..

 

you can allow, the web pages that you want to crawl and disallow the pages that you don't want to crawl !

 

Titles/descriptions look good. Content is going to be an issue as it is a dynamic website from what I can tell. Dynamic sites generally present a giant SEO problems. There is usually little in terms of semantic coherence (I see here everything from massage chairs to jewelry) I would try to add content that deals with some general language for those pages, if you can find something outside of 'buy now' or 'add to cart'. Good luck.

 

Oops, you weren't asking for SEO tips. Should have read the post better. Disregard...

 

thanks for the improvements...

 

very nice information regarding robot.txt file.I am also seeking for this informayion.So i got answer here.

 

you can try this ezine article on robots.txt ........
may be this can help you under standing what is robots.txt

 

hello...

robots.txt
when robots.txt is downloaded from
http//sites.google.com/site/(sitename)/robots.txt
it comes with
Useragent: *
Allow:/

but when I download from my domain:
http//www.mydomain.com/robots.txt
then the file is suddenly different
Useragent: *
Disallow*:/