Robots.txt or What I've done is enough ?

Discussion in 'Search Engine Optimization' started by AsadMoeen, Nov 3, 2010.

  1. #1
    Hello. My website as of course many know : Mixupdate.com ( Signature URL )

    I changed Default permalinks to a custom structure a while back. 301 redirects are already applied to the old ones but even today the old links are being indexed by Google. Not sure but maybe to change permalinks, I have to make some other changes to Wordpress settings ? Just wanted to know if everything is fine at my end before I blame Google for this . So if it is really Google who should be blamed for this then,

    I thought of another way of speeding up the process for Google to remove links.

    I've basically tried up these things up til now :

    --> 301 Redirected the old links to new ones ( Yet Google considers new and old as duplicate ) .
    --> Manually added Google parameters of old links to get them ignored through Webmaster Tool's Parameters ( Yet links are still in Google index ) .

    Hence, even after trying this, Google keeps up increasing my Duplicate pages although they are 301 redirected whenever Google scans . I can manually remove those pages from Google index but the removal requirements must be :

    -> 404 Error ( Not possible due to 301 redirection ) .
    -> No index tag ( Not possible since the pages are redirected to new links ) .
    -> Robots.txt ( Probably the one thing possible ) .

    So what do you guys say about Robots.txt, Should I add links all to robots.txt so they can not be Scanned by Google. Also, I usually like to keep my Robots.txt clean. So once I have added links to robots.txt, got them removed through URL Removal in Google, Can I clean-up my Robots.txt since the content is already 301 redirected ?


    What do you guys say about my new idea ? And what command is there to disallow specific parameters from Google like Mixupdate.com?p=1234 through Robots.txt?


    Also, if there's another way, Please share!
     
    AsadMoeen, Nov 3, 2010 IP
  2. AsadMoeen

    AsadMoeen Active Member

    Messages:
    540
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #2
    So anyone who thinks this is a Good step to achieve what I want and also knows the commands to be put into the Robots.txt, please advice .
     
    AsadMoeen, Nov 3, 2010 IP
  3. fgogo

    fgogo Active Member

    Messages:
    108
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #3
    Use TAG:
    <meta name="robots" content="NOINDEX"> for pages which you dont want have indexed.
    For all rest use redirect 301.
    Where been 404 html code, will use http 410 code - Gone.
    You can also create sitemap as xml or gzip file.
    Grab some new backlins to new urls and wait.
    Thats all.
     
    fgogo, Nov 3, 2010 IP
  4. AsadMoeen

    AsadMoeen Active Member

    Messages:
    540
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #4
    So what should I be doing,

    I assume putting the old 301 redirected links to Robots.txt ( No index ) . To get them removed right ?
     
    AsadMoeen, Nov 3, 2010 IP
  5. fgogo

    fgogo Active Member

    Messages:
    108
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #5
    "NOINDEX" meta tag you need to use in html code, not in robots.txt
    301, 410 Codes need to generate by PHP or mod-rewrite.
    Site map can generate by many free online tools.
    All process of reindexing is a bit long and can keep 2 - 6 weeks, depending of website.
    Robots.txt left as is now. Dont do nothing with it.
     
    fgogo, Nov 3, 2010 IP
  6. AsadMoeen

    AsadMoeen Active Member

    Messages:
    540
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #6
    Ok then! I will try to wait. Are you sure there's no way of removing ?
     
    AsadMoeen, Nov 3, 2010 IP
  7. fgogo

    fgogo Active Member

    Messages:
    108
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #7
    If you have everything right, it mean all codes and redirects to new urls then don't worry matte, everything will be fine.
    Just remember that after reindexing you need left all redirects and codes for longer time, minimum 3-6 months.
    New backlinks to new urls will make this process quicker.
     
    fgogo, Nov 3, 2010 IP
  8. AsadMoeen

    AsadMoeen Active Member

    Messages:
    540
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #8
    Oh Great then.

    Thanks a lot.
     
    AsadMoeen, Nov 3, 2010 IP
  9. AsadMoeen

    AsadMoeen Active Member

    Messages:
    540
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #9
    3-6 months is a lot although . I would try to increase my backlinks to reduce this amount of time
     
    AsadMoeen, Nov 4, 2010 IP
  10. nikznikz

    nikznikz Peon

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search

    Sitemap: www dot stylechanel dot com/feeds/posts/default?orderby=updated
     
    nikznikz, Nov 12, 2010 IP
  11. nikznikz

    nikznikz Peon

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    please anyone help me.. i am new to blogger
     
    nikznikz, Nov 12, 2010 IP
  12. AsadMoeen

    AsadMoeen Active Member

    Messages:
    540
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #12

    What's that and How do I make use of it ?
     
    AsadMoeen, Nov 12, 2010 IP
  13. nikznikz

    nikznikz Peon

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
     
    nikznikz, Nov 12, 2010 IP