1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Need Suggestion for Robots.txt

Discussion in 'robots.txt' started by albertthomas, May 23, 2012.

  1. #1
    Hi,

    Here I am discussing about my website which has a big problem. qwertyc.com is my personal website.
    Actually I found many unusual or wrong urls in my site like this:

    [TABLE="width: 656"]
    [TR]
    [TD="class: xl64, width: 656"]http://www.qwertyc.com/index.php?option=com_content&view=category&id=8

    [TABLE="width: 656"]
    [TR]
    [TD="class: xl65, width: 656"]http://www.qwertyc.com/products-and-services/website-design/8-promote[/TD]
    [/TR]
    [/TABLE]
    [/TD]
    [/TR]
    [/TABLE]

    When I found many urls like this then I disallowed these all in robots.txt file as:

    User-agent: *
    Disallow: /?*
    Disallow: /9-prevent
    Disallow: /8-promote
    Disallow: /7-home/
    Disallow: /10-protect

    I am unable to remove all these thats why I had to take help of robots.txt. My main problem is that all these links are showing like before in webmaster tool. Is there any solution for removing these links from Google Webmaster Tool. And I also want that whenever any user open this link, it should never show on my site.

    Please reply me soon. I am in waiting for yours positive reply.
     
    albertthomas, May 23, 2012 IP
  2. natostanco

    natostanco Greenhorn

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    21
    #2
    once it gets crawled you have to manually remove it otherwise you have to wait longer for it to disappear, I just suggest you to 301 redirect all the dead links to the homepage
     
    natostanco, May 24, 2012 IP
  3. jessicamoises

    jessicamoises Peon

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    on the other side you can redirect them to the permanent links. so that it will not appear in the results.
     
    jessicamoises, Jun 9, 2012 IP
  4. MrOSX

    MrOSX Active Member

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    58
    #4
    I suggest redirecting them to permanent links.
     
    MrOSX, Jun 27, 2012 IP
  5. Custom IDX/MLS

    Custom IDX/MLS Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Play around with .htaccess, setting up two or three all-catching rule rewrites, redirecting all with a 301 somewhere SEO safe.
     
    Custom IDX/MLS, Jun 28, 2012 IP
  6. atxsurf

    atxsurf Peon

    Messages:
    2,394
    Likes Received:
    21
    Best Answers:
    1
    Trophy Points:
    0
    #6
    robots will help these url not to be crawled in the future, but it may take awhile for google to remove them

    also your "Disallow: /8-promote" does not disallow hxxp://www.qwertyc.com/products-and-services/website-design/8-promote
     
    atxsurf, Jul 3, 2012 IP
  7. WebNet Host

    WebNet Host Peon

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #7
    I suggest redirecting them to permanent links as well.
     
    WebNet Host, Feb 21, 2013 IP
  8. WebNet Host

    WebNet Host Peon

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #8
    We also did the same on our corporate site www.webnethosting.net, its a good way to redirect to permanent links.
     
    WebNet Host, Feb 21, 2013 IP