Make sure you 1. Have a sitemap. If you have a wp blog you can install a plugin to do this for you. 2.Ping it also.... Again if you have a WP Blog you can install a plugin for this also. 3. Submit your RSS Feed 4. Social Book marking will also bring the spiders Hope this helps.
There is no dead sure way you can force crawlers to crawl your website. However, if your website has some great content and some potential backlinks, crawlers will automatically starting following your website.
The one and foremost way to have your site crawled is submit your sitemap to search engine and they will quickly crawl your site. Hope this helps...
Don't mind manipulating or controlling robots of different SE's. Just keep your content fresh and linking pattern consistent. That's your target anyway...
Site crawlability is vital. Sitemaps are actually very good for this. Try to use an XML sitemap that is updatable. For very large sites you may only realistically be able to update down to the sub-category level. With regards robots.txt files, ensure they have zero mistakes, may sound obvious but it is very easy to misplace a character or number and end up with swathes of your site not being seen by the search engines.