First of all I just found this site and spent about half an hour reading all the different posts and wanted to say I'm excited to find a place where there are people are going through a lot of the same things I'm going through, as well as all the great insight everyone in here have helped provide them. I just recently launched a new site related to professional sports photos and have been running into some frustration. Google has indexed the site, however so far it's been limited and it's now been a month and it hasn't even done a full crawl yet. After reading one of the other posts it appears I'm in the "sandbox" on one of the terms, and it hasn't even indexed the majority of category pages. I have a simple link structure that I use on another site that has worked just fine, and this one is even more simplistic. Very frustrating. I went back and created a "site map" page and submitted it, and about four days later it indexed the site map page but still hasn't indexed the category pages yet. Google also hasn't recognized the site that links to it yet. Meanwhile MSN has crawled nearly the whole site, but after noticing I needed to make some changes I've noticed that they don't seem to come back as frequently as Google does. I'm kind of curious of how I could increase that frequency as I tweak and make changes. I've also noticed Yahoo grabbed the two pages I linked to from another site but hasn't crawled any of the other pages at all. I've also submitted a Froogle feed and am getting frustrated with the fact that despite the fact I have the name of each team in the title of the feed as well as the description, they don't seem to come up when I do a product search with that search term. Some do, some don't. It's odd. Anyway, I know this was long and drawn out, but I'm looking forward to hearing your thoughts and suggestions. Thanks again, Ian EDIT: I also noticed that rookies are known as "peons" around here. Funny stuff
First of all, welcome to the forum. There are many, many issues at play here. Everything from the sandbox to back links to the actual content. Without being able to take a look at your site to see how you have it structured and what kind of content you have, it's next to impossible to speculate on what could causing your issues. Give us the url and we can offer our opinions. And I hope you like what you find here and stay around for a while.
It's http://www.prosportspictures.com Thanks for the kind words and offering to help. When I was putting the site together I thought I had a good grasp on what I was doing. Apparently I've still got a lot to learn It's nice to be associated with people who know how tough this process is and are willing to help. Ian
Welcome to DP. Nice looking site btw. As to why some bots seem to vary from month to month as to how often they visit a site - I have no idea. Over the last 2 months the bots seem to be taking it in turns to visit pages in my site. 2 months ago it was google (they pulled about 150MB), last month it was yahoo (just over 100MB) this month it is msn (currently about 90MB). The number of IBL's definitely makes a difference, I believe but haven't found a way of proving this yet, that G'bot can pull the headers from a page without actually registering as a visit. I'm not sure how to put these thoughts into words, but Gbot seems to be able to magically tell when I've changed a page. If say Gbot comes heavily around mysite, it seems to ignore pages that haven't changed for quite a while - fair enough, that's what it's supposed to do. I'm pretty sure they keep somekind of index on a url so they know roughly how often it is updated. The higher PR your site is the more acturate the index is. But, I've changed a few pages a very odd times ie it'll go 2 months without a change and then get change every couple of days for a fortnight. I've noticed that everytime I change certain pages, Gbot will hit them within 48 hours of the change. Now there is no way they can know that that page suddenly gone from a 2 month update to a 2 day -it's not as if it's following any cycle or such like. The only thing I can think of is that they can check whether a page has been updated without actually registering a visit in the logs. Thus that url gets added to the queue for proper search bot caching. I've also noticed that just 'cos a bot hits a page doesn't necessarily mean that that page gets cached. Which is another reason I think there are different bots that do page finds, indexing and caching. The only bit of advice I'll give you is: Don't try and work out in writing what the 'bots do, especially when you're half way through a second bottle of wine You'll only end up getting as confused as my ramblings above. PS: remember to introduced yourself in the introductions forum - it took me 2 months to get around to it and I felt a bit silly doing it then.
Thanks jlawrence. MSN is leading the way in terms of number of crawls, and google is there but again seems to be limiting the results. It also seems to be picking and choosing which pages to list, as I said the sitemap is up but the league and team pages don't show up in the listings. It's posting individual photo pages but that's about it. Yahoo hasn't done much of anything yet and as I said it's been about a month. I'm also having difficulty with inbound links since most fan sites, etc don't want to exchange links, they just want paid advertising. Not that I blame them I guess.... Speaking of which does anyone have some solid suggestions for getting in-bound links? That's an area I wanted to start working on but I've been a bit be-fuddled as to the best way to go about it. Ian Also: I'll make sure I stop by the introductions forum as well.
So I clicked on to Yahoo this afternoon and looked at one of my keyword phrases and noticed the page came up #6. I had linked to it off one of my other pages that was related in content. Needless to say it appears that by linking to the page from that site Yahoo crawled it and ranked it well, so that's good news. I've been reading older posts for much of the afternoon and I think my biggest issue now are back links. As I said in my previous post I'd love any and all ideas you may have to start building those. Ian
You will eventually appear in Google. I had gone looking for my site on Google for over a month but only the domain name came up. Then voila just a week ago, the whole site appeared without Google recently trawling.. How much is MSN and Yahoo used though?