1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Webmaster Blocked URL's Understanding Issue

Discussion in 'robots.txt' started by Jaden Alvin, Jul 9, 2014.

  1. #1
    Hello Guys!

    I'm using Robots.txt file in my site but Google webmaster tool showing only 1 blocked URL and i have mentioned more then 60 Urls in file. What is the cause of that??? Please Help.

    Thanks
    SEMrush
     
    Jaden Alvin, Jul 9, 2014 IP
    SEMrush
  2. blogdum

    blogdum Active Member

    Messages:
    518
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    75
    #2
    Have you checked using site:yourwebsitename.com on google and did you find those urls indexed.
     
    blogdum, Jul 9, 2014 IP
  3. Jaden Alvin

    Jaden Alvin Active Member

    Messages:
    89
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    78
    #3
    Yes URLs are still getting indexed.
     
    Jaden Alvin, Jul 9, 2014 IP
  4. ceebee

    ceebee Active Member

    Messages:
    112
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    83
    #4
    could be lag. there are trillions of pages now.

    google would likely tell you if there was a fault in the formatting of robot.txt.

    if you use a sitemap.xml file which excludes those pages you should be fine.

    if you MUST get them off Google, you can manually remove them in Webmaster Tools.. Google Index :: Remove URLs
     
    ceebee, Jul 21, 2014 IP
  5. ceebee

    ceebee Active Member

    Messages:
    112
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    83
    #5
    i think that the way Google works now... (oversimplifying it here) it can add/update pages constantly, but removals and penalties, etc. happen during a big update. At least, that has been my experience.
     
    ceebee, Jul 22, 2014 IP
  6. Avish Gabba

    Avish Gabba Member

    Messages:
    32
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #6
    Go to google and type addmyurl. It helps you alot.
     
    Avish Gabba, Aug 30, 2014 IP
  7. danish001

    danish001 Active Member

    Messages:
    596
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #7
    Hi Everyone i have a query. I have created sitemap.xml and robots.txt where should i submit this both the things.
     
    danish001, Sep 17, 2014 IP
  8. cathandrew

    cathandrew Member

    Messages:
    62
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #8
    You can submit it in Google Webmaster tools. Create an account in Google webmaster tools and submit your website first. In your webmaster tools you can see the option "Crawl" below that you can see the sitemaps. Click it you can see the button "Submit a sitemap" You can also test the sitemap whether it's a valuable one or not. You can modify your robots.txt file under the same "Crawl" option by clicking robots.txt tester
     
    cathandrew, Sep 18, 2014 IP
  9. Jaden Alvin

    Jaden Alvin Active Member

    Messages:
    89
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    78
    #9
    Easy to do this, just simply login to Google webmaster account and their you have an option to submit sitemap. But take care about the sitemap format, it should be in xml format.
    If you have implemented robots.txt file in your website then it will be automatically get detected by Google webmaster tool; you can also test your URL which one is blocked and which one not.
     
    Jaden Alvin, Sep 22, 2014 IP
  10. danish001

    danish001 Active Member

    Messages:
    596
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #10
    I got it now and i will do the same.Thanks Catandrew for your reply.
     
    danish001, Oct 1, 2014 IP
  11. cathandrew

    cathandrew Member

    Messages:
    62
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #11
    you're welcome :)
     
    cathandrew, Oct 13, 2014 IP
  12. danish001

    danish001 Active Member

    Messages:
    596
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #12
    What should be length of meta tags.
    Title:-
    Desp:-
    Keywords:-
     
    danish001, Oct 13, 2014 IP
  13. cathandrew

    cathandrew Member

    Messages:
    62
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #13
    Your title should not be more than 75 characters and your description should not exceed 150 characters. And it is better if you use only 3 keywords by analyzing using Google keyword tool. I'm using that for my website http://worldleaks.com for the past one year. It really helps me a lot
     
    cathandrew, Oct 18, 2014 IP
  14. danish001

    danish001 Active Member

    Messages:
    596
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #14
    I am having one one problem my client is telling me that use same content in the website that is mention on http://www.godigitalonline.com to http://www.godigitalonline.in . SO tell me can i use same content in both the website.
    I think that i will be copy content issue. am right.
     
    danish001, Oct 22, 2014 IP
  15. Jaden Alvin

    Jaden Alvin Active Member

    Messages:
    89
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    78
    #15
    No. Don't use the same content in two different site, it will be counted as duplicate content. No matter how much similar both the sites are.

    But you can try to use URL canonical tags for this kind of process but sorry i never did that so cant help you one that.
     
    Jaden Alvin, Oct 31, 2014 IP
  16. cathandrew

    cathandrew Member

    Messages:
    62
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #16
    Yep it's really a big issue for your website's SEO. You can do some changes in the content and post it. there should be some other technical ways to byepass this problem.. will have to look and help you out
     
    cathandrew, Nov 13, 2014 IP
  17. danish001

    danish001 Active Member

    Messages:
    596
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #17
    Hi Cathndrew,
    My website pages are getting crawled but it does not show me Title when it is crawled.
    What i can do in this situation.
    Thanks & Regards
    Danish Shaikh
     
    danish001, Nov 13, 2014 IP
  18. cathandrew

    cathandrew Member

    Messages:
    62
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #18
    Does not show me the title?? What does it mean?
     
    cathandrew, Nov 13, 2014 IP
  19. Jaden Alvin

    Jaden Alvin Active Member

    Messages:
    89
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    78
    #19
    well if I'm not wrong then you want to say that the Title tag you are using in sites on page, are different from the SERP results Title. If that is the case then i will let you know that all this depends on the search engines and due to some Title length issue also.
     
    Jaden Alvin, Nov 17, 2014 IP
  20. Jaden Alvin

    Jaden Alvin Active Member

    Messages:
    89
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    78
    #20
    Well at the end it all depends on Google that how it want to represent your title in SERP's. Not to worry about anything if following guideline.
     
    Jaden Alvin, Nov 24, 2014 IP