Hey all... Currently I am working on a Web Directory On site SEO. The Web Directory has different categories. If someone wants to post in a category, he has to log in/register first... Here comes my query.... If a Search Engine Robot is crawling my directory then it will also encounter this log in/register page... Obviously it will face a problem now... Will the SE Robot crawl further pages after the log in page as the further pages are very important pages where i can give keywords in meta tags... If not, then please suggest what can be the way to overcome this problem so that it crawls my all the pages after log in page..... Your comments are most awaited........
I will crawl fine. Make sure you allow it in your robots.txt and that your site has a sitemap registered with Google. If you want to get really techie, you could create a specialized .htaccess file, but that usually isn't necessary. The spiders will crawl as long as you authorize it, password login pages or not. You might want to investigate methods to prevent bad bots from entering as well. If yours is a Wordpress site, then make sure that you have the public allowed radio box checked in the "privacy" section. Wordpress has some pretty snazzy SEO plugins like All In One SEO and an automatic sitemap generator and pinger. If you are not using Wordpress just Google up sitemaps and robot.txt and study up on it.
Great post..great question..great answer!!! I am new to SEO...try to collect such knowledge and tricks...thanks!!
How would the spider know what pages come after the login page if it doesn't know the credentials to log in with? Usually you would use a sitemap or similar, to give the spiders a route through to the pages that you want spidered. Having said that, it's unusual to want to serve content to spiders that you don't want to show to non-authenticated visitors. This could be seen as some kind of cloaking - which is normally a black-hat technique.