Online Site Crawler

Discussion in 'Site & Server Administration' started by anjanesh, Jul 13, 2006.

0
  1. #1
    Hi

    I updated my site to include mod_rewrite so that the .php links would be .htm links and Google crawled my site recently but the update still has the old .php links instead of the new .htm links.

    Is there some online link spider out there where I can see my links getting crawled very similar to way Google does ?

    Thanks
     
    anjanesh, Jul 13, 2006 IP
  2. MaxPowers

    MaxPowers Well-Known Member

    Messages:
    264
    Likes Received:
    5
    Best Answers:
    1
    Trophy Points:
    120
    #2
    It's likely because your old .php URLs that they have in their system still work. Try to setup robots.txt with a *.php (or similar way to block them /send a 404) and see if they go away
     
    MaxPowers, Jul 13, 2006 IP
  3. websiteideas

    websiteideas Well-Known Member

    Messages:
    1,406
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    130
    #3
    I don't think there is any crawler that will show you this. Just be patient and Google will drop those urls and pick up your new ones.
     
    websiteideas, Jul 14, 2006 IP
  4. anjanesh

    anjanesh Well-Known Member

    Messages:
    344
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    165
    #4
    As MaxPowers as said It's likely because your old .php URLs that they have in their system still work which is correct as I've just added the entries in the htaccess file to use htm for php but the same php links still exists. So Google may not drop them since the cached links still work.

    dead-links.com is a good one but it doesnt show from which page it got the link.
     
    anjanesh, Jul 14, 2006 IP