How should I invite robots to crawl the internal pages of my website?

Hi all,

I am in big trouble. I need your advice asap.

I have a website related to "Fireworks". The robot crawls some of its pages very often but usually leaves the internal pages (consist of content and keywords). Because of this my website does not rank on 1st page in SERPs. I have applied revisit tag in all the pages but still the robot is not crawling internal pages.

Please let me know how should I invite robots to crawl internal pages of my website asap?

Thanks & Regards
Brian Melony

 

Have you created a sitemap and submitted it to Google?

http://www.google.com/webmasters/sitemaps/

 

Point more links to those internal pages

 

maybe you find the answers here:

http://www.robotstxt.org/wc/faq.html

 

Hi Sixsigma,

I have created Sitemap for the Site and submitted that to Google also. Robot crawled that sitemap earlier but now days robot is not crawling to that sitemap even.

 

Moreover my site already ranks on 11 th position but I want it to rank on atleast 1st page in google SERPs.

 

Thanks Chris for the wonderful link. It will definitely help ...

 

I would change your internal linking structure. You should always be less than 2 clicks from the home page. Find where your pages aren't being indexed, and point some links to them. Write a quick article and point it to the pages that aren't being indexed. With a really good internal linking structure the highest priority page should be your homepage, and your second highest page should be your sitemap. Then you can roll the pages that aren't getting indexed to the top of the sitemap.

What that means is that you change the structure of your sitemap, and put the pages you're having issues with at the top of your sitemap.

Honestly though, most indexing issues are due to internal linking structure + not enough incoming links. Get some high value links, and point them to your sitemap instead of your home page.

HTH,

M

 

Get links pointing to your internal pages and they'll get crawled.

 

Are you kidding there is no way a robot can crawl a web page. The robot has to be too small so that it can walk on the computer screen.

 

Be active in some forums and have sign links to you internal pages. Also, Link internal pages directly from home page and frequently indexing pages.

Cross check your robots.txt file and meta tags for absence of spider stopping code.

 

Hello,

Please give a try to link below

http://blogbusinessworld.blogspot.com/2005/07/crawling-webs-with-search-engine.html

Hopefully it should help you to solve your issue.

 

Thanks to all of you for solving my problem...

 

Start submission for your internal pages and add sitemap in google.

 

add site map to Google, all links in site map will be crawled by Google.

 

try to add sitemap to google, and get backlinks to your website.

 

Very simple Make a sitemap.xml and submit in the Google.

Your all pages will be crawled quickly

And important thing Submit your sitemap.xml in the other rss feeds

Thanks

 

Brian the best way is to submit to a directory that google crawls fast.
and my site have the answer....if you don't get crawled in less than a week,
come to this forum and say it. I promise you, you will get crawled.
and by the way, is FREE

www.muozi.org

Thanks

 

I haven't read every response but an xml sitemap would definitely help.

Additionally I think they would get crawled pretty fast if you social bookmarked them, especially from Digg and StumbleUpon