1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Will robots file help to cache the content in the web page

Discussion in 'robots.txt' started by Prasseo, Jan 12, 2015.

  1. #1
    Prasseo, Jan 12, 2015 IP
    SEMrush
  2. Sycrid

    Sycrid Active Member

    Messages:
    120
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    53
    #2
    Hi Prasseo,

    From what I'm aware of, you can not get your robots.txt to instruct search engines to cache your website pages. What it does do though, is tell search engines which pages it can and can not visit.

    http://en.wikipedia.org/wiki/Robots_exclusion_standard

    What I'd recommend is configuring your robots.txt to allow the website pages to be visited by search engines and then in time they will cache your pages. Depending on which search engine, depends on how long this takes. Make sure you submit your website URL to the search engines, and also if you set up webmaster tools on google and bing, then you can submit a sitemap which will help index your web pages faster.

    Hope this helps.
     
    Sycrid, Jan 18, 2015 IP