Log in or Sign up
Digital Point
Home
Forums
>
Development
>
Site & Server Administration
>
robots.txt
Find help with setting up your robots.txt file on your web site.
Page 1 of 19
1
←
2
3
4
5
6
→
19
Next >
Sort by:
Title
Start Date
Replies
Views ↓
Last Message
How to check robots.txt on site?
seoworld1234
,
Apr 21, 2008
...
2
Replies:
33
Views:
85,437
imranbashir.seo
Mar 7, 2023
how to block all other BOT except Google, MSN, Yahoo
danieloffice
,
Jan 9, 2008
...
2
Replies:
23
Views:
70,046
moneyroad
Jul 10, 2015
List of Bad Bots to disallow?
hmansfield
,
May 20, 2007
Replies:
14
Views:
25,843
Sergio Minozzi
Feb 22, 2017
Help needed - robts.txt and sitemap
jack_sparrow
,
Aug 3, 2007
...
3
4
5
Replies:
89
Views:
22,888
kevoh99
May 25, 2010
Robots txt file permission
Ralarash
,
Apr 30, 2010
Replies:
9
Views:
22,303
manish.chauhan
Jun 11, 2010
How to hide robots.txt from visitors
latoya
,
Aug 23, 2008
Replies:
16
Views:
21,513
mrgroove
Apr 1, 2010
robots.txt to deny all spiders for proxy?
Mysmasken
,
Nov 29, 2007
Replies:
3
Views:
21,000
Mysmasken
Nov 30, 2007
Robots.txt
Angelabailey
,
Nov 15, 2013
Replies:
15
Views:
20,405
Panda123
Mar 27, 2016
Name of yahoo, google and msn bot
blabla11
,
Aug 8, 2007
Replies:
9
Views:
19,926
seoworld
Sep 11, 2007
Robots.txt for SMF Forum
pradip200
,
Mar 23, 2013
Replies:
4
Views:
19,705
Svoboda
Jun 30, 2017
Is my robots.txt working fine?
aleale
,
Sep 10, 2014
Replies:
2
Views:
19,479
maddenitrous
Oct 3, 2014
Very Strange Problem
Marker1982
,
Jul 21, 2015
Replies:
3
Views:
19,440
Jacob.k
Jul 25, 2015
Fix to ban google search from Cloaking and/or sneaky redirects
MarcL
,
Jul 30, 2015
Replies:
4
Views:
19,317
MarcL
Aug 3, 2015
robots.txt for phpBB forum
Googles76
,
Dec 3, 2005
Replies:
18
Views:
19,308
linux-lover
May 14, 2009
How to control when a bot(yahoo,google) can crawl a website
enikram
,
Mar 18, 2015
Replies:
4
Views:
18,756
Nigel Lew
Mar 18, 2015
Why cant Google access my robots.txt ?????
MoneyPlanters
,
Sep 27, 2015
Replies:
1
Views:
18,692
Mit Thakkar
Sep 27, 2015
Advantages of robots.txt
HomeRun
,
Feb 24, 2008
...
2
Replies:
26
Views:
18,131
agitetech
Dec 27, 2012
Blocked Url's by robots.txt in google webmaster tool how to fix it?
Minesh Parikh
,
Apr 6, 2015
Replies:
0
Views:
17,931
Minesh Parikh
Apr 6, 2015
Remove this stupid robots.txt
netboy
,
Feb 22, 2008
Replies:
17
Views:
17,782
Rituja
Feb 15, 2012
Do you block Majestic bots?
Jeffr2014
,
Apr 9, 2015
Replies:
4
Views:
17,271
neteater
Feb 20, 2016
Is 32,000 Alexa Rank Good?
Ratman2050
,
Oct 8, 2006
...
2
Replies:
35
Views:
17,148
icyfiren
Aug 26, 2011
Robots.txt Help to Block a Particular File Name
Kushal Biswas
,
Feb 3, 2016
Replies:
1
Views:
17,120
Anubhav-soin
Feb 25, 2016
Is Dreamhost support stupid or am i missing something?
johan-cr
,
Dec 27, 2006
...
2
Replies:
33
Views:
16,951
danieloffice
Jan 9, 2008
Webmaster Blocked URL's Understanding Issue
Jaden Alvin
,
Jul 9, 2014
Replies:
19
Views:
16,909
Jaden Alvin
Nov 24, 2014
Twiceler wont stop even when banned from robots.txt
deluxdon
,
Jun 24, 2007
...
2
Replies:
24
Views:
15,581
elladrone
Dec 28, 2009
spiders.text & robot.text
sck
,
Aug 11, 2006
Replies:
3
Views:
15,536
explorer
Sep 15, 2006
Robots.txt is blocking my labels
Mrinal_Roy
,
Feb 5, 2015
Replies:
5
Views:
15,417
Gideon Duah
Jul 21, 2021
i can't stop Baidu, pls help!!!
web_master
,
Dec 27, 2011
Replies:
7
Views:
14,938
RoundShots
Sep 11, 2015
How to block archive.org bot with both robots.txt AND .htaccess
EducationLinks
,
Aug 9, 2007
Replies:
6
Views:
14,841
hans
Oct 21, 2007
Locked
robot.txt
Abdullah adil
,
Feb 11, 2017
Replies:
6
Views:
14,726
sarahk
Feb 16, 2017
Robot.txt file not found on my blog
myfastrupee
,
Jun 21, 2013
Replies:
4
Views:
14,628
meethi jain
Jun 3, 2015
Robot.txt for Google (error)
wartes
,
Apr 10, 2017
Replies:
6
Views:
14,243
Md.Saddam Hossain
May 7, 2017
How do I write my robots.txt to exclude SSL pages only?
amanamission
,
Sep 18, 2007
Replies:
5
Views:
14,168
Webnauts
Dec 5, 2008
Disallow subdirectory in robots.txt
consultsoft
,
Dec 24, 2008
Replies:
4
Views:
13,846
manish.chauhan
Jan 20, 2009
What bots ignore robots.txt?
Jon12345
,
Aug 16, 2005
Replies:
3
Views:
13,346
minstrel
Oct 15, 2005
Robot.txt file
iluvm
,
Nov 30, 2005
...
2
Replies:
20
Views:
12,555
Emily Stephens
Jan 3, 2012
robots.txt please check
gree124
,
Nov 27, 2014
Replies:
7
Views:
12,165
rajendra1682
Oct 6, 2015
How to create a robots file for PHP webpage to crawl
Prasseo
,
Jan 12, 2015
Replies:
3
Views:
12,092
abinjoy
Jul 14, 2015
Dangerous Crawler and User Agent! Must Read!
angellica2017
,
Sep 19, 2006
...
2
Replies:
22
Views:
11,689
manish.chauhan
Apr 6, 2008
How to Remove 404 Error?
prabhjot.singh
,
Aug 26, 2012
Replies:
18
Views:
11,665
Rakeshraj
Jul 15, 2013
How do I unblock my robots.txt
scott2342
,
Nov 24, 2014
Replies:
3
Views:
11,526
meethi jain
Jun 3, 2015
unknown bots.
aj845
,
Jul 6, 2007
Replies:
12
Views:
11,363
cquinndesign
Jul 3, 2011
Best Robots.txt For Wordpress Installed Blogs
eniva
,
Sep 23, 2008
Replies:
12
Views:
10,985
seo.lover
Aug 29, 2010
Would a robot.txt fix this SEO issue?
rob587
,
May 10, 2013
Replies:
19
Views:
10,980
msjaved
Nov 8, 2013
How to Download google robot.txt file?
offsitenoc
,
Jul 26, 2010
Replies:
3
Views:
10,940
-[z]-
Nov 7, 2012
Best Robots.txt For Wordpress Blog ....
style fashion
,
Oct 1, 2009
Replies:
13
Views:
10,698
mohisha
May 10, 2011
Need advice on how to block certain page using robots.txt
mariush
,
Aug 12, 2006
Replies:
7
Views:
10,493
learn2success
Jun 6, 2012
Blogger Robots.txt Problem- Google is not crawling
nikmahajan
,
Aug 5, 2008
Replies:
8
Views:
10,417
g4ngst3r
Sep 25, 2011
Robot text and partial fetch issues - need suggestions
full_time_writer
,
Jul 28, 2015
Replies:
0
Views:
10,058
full_time_writer
Jul 28, 2015
Error - URLs restricted by robots.txt errors How Fix IT
linkladdy
,
Jul 2, 2014
Replies:
1
Views:
9,983
evildon
Jul 2, 2014
Showing threads 1 to 50 of 911
Thread Display Options
Sort threads by:
Last message time
Thread creation time
Title (alphabetical)
Number of replies
Number of views
First message likes
Order threads in:
Descending order
Ascending order
Loading...
(You must log in or sign up to post here.)
Show Ignored Content
Page 1 of 19
1
←
2
3
4
5
6
→
19
Next >
Log in with Facebook
Your name or email address:
Do you already have an account?
No, create an account now.
Yes, my password is:
Forgot your password?
Stay logged in
Digital Point
Home
Forums
>
Development
>
Site & Server Administration
>
Log in to view your Analytics, AdSense & PayPal stats
Home
Forums
Search Forums
Recent Posts
Spy
Search Engines
Google
AdSense
Guidelines / Compliance
Placement / Reviews / Examples
Reporting & Stats
Payments
AdWords
Analytics
Google+
Sitemaps
Google API
Shopping
Yahoo
Publisher Network
Search Marketing
Yahoo API
Bing
Bing Ads
All Other Search Engines
Directories
Solicitations & Announcements
ODP / DMOZ
Business & Marketing
General Business
General Marketing
Search Engine Optimization
Keywords
Microdata
Payment Processing
PayPal
Social Networks
Facebook
Facebook API
Twitter
Google+
Link Development
Legal Issues
Domain Names
Appraisals
Copywriting
Article Marketplace
eCommerce
Pay Per Click Advertising
Digital Point Ads
Google AdWords
Yahoo Search Marketing
Bing Ads
Affiliate Programs
Affiliate Program Management
Commission Junction
Google
eBay
Amazon
ClickBank
Chitika
Design
HTML & Website Design
CSS
Graphics & Multimedia
Photoshop
Reviews
Content Management
Blogging
WordPress
Drupal
Joomla
XML & RSS
Scripts
Development
Programming
PHP
JavaScript
jQuery
C#
Ruby
Site & Server Administration
Apache
Nginx
Security
Traffic Analysis
Google Analytics
robots.txt
Forum Management
XenForo
vBulletin
phpBB
Cluster Computing
Google Sitemaps
Web Hosting
Outages
Advertising
Databases
MySQL
Buy, Sell or Trade
Sites
Filter Sites For Sale
Domains
Filter Domains For Sale
Design Contests
Advertising
Secondhand Licenses
Services
Content Creation
Design
Programming
Traffic
Web Hosting
Plug-ins/Add-ons
Templates/Styles
Scripts
Designs
eBooks
Articles
The Digital Point
Introductions
General Chat
Movies, Music & TV
Games
Sports
Hardware
Politics & Religion
Products & Tools
Keyword Tracker
Cookie Search
Better Analytics
Legacy
Co-op Advertising Network
Optigold ISP
Reviews
Websites
Design
SEO
Support & Feedback
Advertising
Keyword Tracker
Cookie Search
The Underground
Area 51
Sphinx Search Support
XenForo Search Support
Tools
Most used tools...
Rank Checker
Cookie Search
Site Search Engine
Marketplace
Sites
Domains
Plugins/Add-ons
Scripts
Templates/Styles
Graphic Designs
eBooks
Articles
Advertising
Buy Ad Space
Sell Ad Space On Your Sites
Advertise on digitalpoint.com
Members
Notable Members
Current Visitors
Recent Activity
User Map
Directory
New Profile Posts
Menu
Search
Search titles only
Posted by Member:
Separate names with a comma.
Newer Than:
Search this forum only
Display results as threads
Useful Searches
Recent Posts
More...