robots.txt pointing to index page

The main page was cached October 28, 2004 -- how long has this site been up?

I could find only 3 backlinks, 2 from forums.

If it is a new site, Google has found it and will eventually be back to try to spider the remaining pages. In the meantime, you need to submit to directories and see what backlinks you can arrange.

 

Does that code above say to every useragent (ie: every spider) please do not come to my site rather than spider everything

I think that's why you are not showing - the code is incorrect

<edit> My mistake - I have just seen this on the robots site:

To exclude all robots from the entire server

User-agent: *
Disallow: /

To allow all robots complete access

User-agent: *
Disallow:

Or create an empty "/robots.txt" file.
To exclude all robots from part of the server

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/

To exclude a single robot

User-agent: BadBot
Disallow: /

To allow a single robot

User-agent: WebCrawler
Disallow:

User-agent: *
Disallow: /

 

No, leeds1.... the line "Disallow: " with nothing after the ":" says "Disallow nothing", i.e., "spider everything".

 

yep - saw that hence the follow up edit thingy

 

Ah, sorry, leeds1 -- I must have been posting while you were editing... when I replied that wasn't there... spooky

 

I have recently completed a makeover to a site which had excellent Google rating. I changed some of the page names to optimize them for keywords and used the .htaccess file to force the redirection of old pages to the new counterpart.Will this method pf redirection have any effect on the ranking?

 

No. As long as it is clear that this is a permaent redirect (301), that is exactly what Google recommends:

Google Information for Webmasters

 

Thanks for your reply, but I am unclear on where the 301 code should go and the syntax required. Should it be in the header of the .htaccess file. The pages from the old site had different names so don't exist anylonger. I first made a 404 error page which allowd them access to the new navigation, but this would be bypassed now with the redirect from the .htaccess.

 

After the rewrite rule you put [R=301] or [R=301,L] most likely.

 

Place one of the following lines in your .htaccess file for each file that you've renamed:

Redirect 301 /oldpage.html [url]http://www.yoursite.com/newpage.html[/url]

Code (markup):

or

Redirect permanent /oldpage.html http://www.yoursite.com/newpage.html

Code (markup):

 

could that be the reason http://www.imgascot.com is indexed with the url as the title instead of the title tags?

 

Are you using a redirect on that site, sitetutor?

I see this in Google: http://www.google.com/search?source...,GGLD:2005-12,GGLD:en&q=site:www.imgascot.com

while the meta tags say this:

<title>IMG Ascot - Indianapolis Mortgage and Loan Specialists Serving Indiana</title>
<meta name="keywords" content="Indianapolis Mortgages, Indianapolis Indiana Mortgages, Indianapolis IN Mortgages, Indianapolis Mortgages and Loans, Indianapolis Indiana Mortgages and Loans, Indianapolis IN Mortgages and Loans, Indianapolis Home Loans, Indianapolis Indiana Home Loans, Indianapolis IN Home Loans, Indianapolis Loans, Indianapolis Indiana Loans, Indianapolis IN Loans">
<meta name="description" content="Indiana mortgage, Indianapolis Mortgage Specialists Serving the State on Indiana, ">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

Code (markup):

So the "title" Google uses, if it IS reflecting the fact that it's a redirect, is interesting -- I've never seen that before.

 

Minstrel, sorry:
http://search.msn.com/results.aspx?FORM=MSNH&srch_type=0&q=indianapolis+mortgages
on msn

 

Wait... the snippet is from an old version of the page... not the current version.

Did the page EVER have that as a title?

 

Oh. I'm not sure what that means fopr MSN search but Google does that for newly found but not yet fully spidered sites.

How new IS this site?

DOES it use a redirect?

I'm still interested in that Google result...

 

That looks like a webhost's temp page when you activate an account. Your site seems to have been indexed before it was setup with what Minstrel shows.

 

no, we took the site over as a client but it has been indexed for a few months ... it seems like there is a robot.txt file in the way

 

It is ... http://www.imgascot.com/robots.txt

# robots, nothing to see here. keep on move n...

User-agent: *

disallow /

and it's written wrong too.

 

Hey, good find there, noppid! That robots.txt file is telling Google (and all other spiders) not to index anything on the site... assuming the bots can figure it out since, as noppid noted, the syntax is incorrect.

Change this:

# robots, nothing to see here. keep on move n...

User-agent: *

disallow /

Code (markup):

to this

User-agent: *
Disallow:

Code (markup):

or just delete the file.

 

Thank you guys, deleted that garbage!