Hi, Im using PHPNuke on one of my sites and G is indexing pretty much everything but the forums. I have the many google mods out there which disable sessIDs and it has all the hacks needed to be seo friendly. If anyone could take a look Here I would be very appreciative. Here is a link of Gs indexed pages http://www.google.com.au/search?hl=en&lr=&q=site:www.metalthrone.net
How long have you had the site up for? Do you have much link popularity? The urls look clean and easy to follow so there should be no problems there. If it is a new site its going to take a while without some quality inbound links. Brad
The site used to be www.metal-forums.net which had about 12k indexed. But i had the same problem with the forums being the last indexed, and indexed very slowly. And the site was never fully indexed. Now that I have redirected, I dont want to wait a long time again. I was maybe thinking there is something wrong but i do not see it. Page size, urls and content all seem to be very friendly IMO but i may have been missing something.
If you recently changed the domain name, that could be one reason. Do you have a redirect in place? Your page titles are reversed from optimum: <title>Metal Throne >> Forums >> General Metal Chat >> What's up with Cradle Of Filth?</title> Code (markup): Better would be: <title>What's up with Cradle Of Filth? >> General Metal Chat >> Forums >> Metal Throne</title> Code (markup): Then get rid of all of these: <META HTTP-EQUIV="EXPIRES" CONTENT="0"> <META NAME="RESOURCE-TYPE" CONTENT="DOCUMENT"> <META NAME="DISTRIBUTION" CONTENT="GLOBAL"> <META NAME="AUTHOR" CONTENT="Metal Throne"> <META NAME="COPYRIGHT" CONTENT="Copyright (c) by Metal Throne"> <META NAME="ROBOTS" CONTENT="ALL"> Code (markup): But double check your .htaccess file...
Well here is the htaccess file. # $Author: zx $ # $Date: 2003/08/17 14:03:21 $ # deny most common except .php <FilesMatch "\.(inc|tpl|h|ihtml|sql|ini|conf|class|bin|spd|theme|module)$"> deny from all </FilesMatch> <Limit GET PUT POST> Order Allow,Deny Allow from all </Limit> RewriteEngine on #The next lines check for Email Spammers Robots and redirect them to a fake page RewriteCond %{HTTP_USER_AGENT} ^Alexibot [OR] RewriteCond %{HTTP_USER_AGENT} ^asterias [OR] RewriteCond %{HTTP_USER_AGENT} ^BackDoorBot [OR] RewriteCond %{HTTP_USER_AGENT} ^Black.Hole [OR] RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR] RewriteCond %{HTTP_USER_AGENT} ^BlowFish [OR] RewriteCond %{HTTP_USER_AGENT} ^BotALot [OR] RewriteCond %{HTTP_USER_AGENT} ^BuiltBotTough [OR] RewriteCond %{HTTP_USER_AGENT} ^Bullseye [OR] RewriteCond %{HTTP_USER_AGENT} ^BunnySlippers [OR] RewriteCond %{HTTP_USER_AGENT} ^Cegbfeieh [OR] RewriteCond %{HTTP_USER_AGENT} ^CheeseBot [OR] RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR] RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR] RewriteCond %{HTTP_USER_AGENT} ^CopyRightCheck [OR] RewriteCond %{HTTP_USER_AGENT} ^cosmos [OR] RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR] RewriteCond %{HTTP_USER_AGENT} ^Custo [OR] RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR] RewriteCond %{HTTP_USER_AGENT} ^DittoSpyder [OR] RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR] RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR] RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR] RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [OR] RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR] RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR] RewriteCond %{HTTP_USER_AGENT} ^EroCrawler [OR] RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR] RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR] RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR] RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR] RewriteCond %{HTTP_USER_AGENT} ^Foobot [OR] RewriteCond %{HTTP_USER_AGENT} ^FrontPage [NC,OR] RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR] RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR] RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR] RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Image [OR] RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR] RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR] RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR] RewriteCond %{HTTP_USER_AGENT} ^Harvest [OR] RewriteCond %{HTTP_USER_AGENT} ^hloader [OR] RewriteCond %{HTTP_USER_AGENT} ^HMView [OR] RewriteCond %{HTTP_USER_AGENT} ^httplib [OR] RewriteCond %{HTTP_USER_AGENT} ^HTTrack [NC,OR] RewriteCond %{HTTP_USER_AGENT} ^humanlinks [OR] RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR] RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR] RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR] RewriteCond %{HTTP_USER_AGENT} ^Indy\ Library [NC,OR] RewriteCond %{HTTP_USER_AGENT} ^InfoNaviRobot [OR] RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR] RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR] RewriteCond %{HTTP_USER_AGENT} ^JennyBot [OR] RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR] RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR] RewriteCond %{HTTP_USER_AGENT} ^Kenjin.Spider [OR] RewriteCond %{HTTP_USER_AGENT} ^Keyword.Density [OR] RewriteCond %{HTTP_USER_AGENT} ^larbin [OR] RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR] RewriteCond %{HTTP_USER_AGENT} ^LexiBot [OR] RewriteCond %{HTTP_USER_AGENT} ^libWeb/clsHTTP [OR] RewriteCond %{HTTP_USER_AGENT} ^LinkextractorPro [OR] RewriteCond %{HTTP_USER_AGENT} ^LinkScan/8.1a.Unix [OR] RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR] RewriteCond %{HTTP_USER_AGENT} ^lwp-trivial [OR] RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR] RewriteCond %{HTTP_USER_AGENT} ^Mata.Hari [OR] RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR] RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR] RewriteCond %{HTTP_USER_AGENT} ^MIIxpc [OR] RewriteCond %{HTTP_USER_AGENT} ^Mister.PiX [OR] RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR] RewriteCond %{HTTP_USER_AGENT} ^moget [OR] RewriteCond %{HTTP_USER_AGENT} ^Mozilla/2 [OR] RewriteCond %{HTTP_USER_AGENT} ^Mozilla/3.Mozilla/2.01 [OR] RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR] RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR] RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR] RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR] RewriteCond %{HTTP_USER_AGENT} ^NetMechanic [OR] RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR] RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR] RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR] RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR] RewriteCond %{HTTP_USER_AGENT} ^NPBot [OR] RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR] RewriteCond %{HTTP_USER_AGENT} ^Offline.Explorer [OR] RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR] RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR] RewriteCond %{HTTP_USER_AGENT} ^Openfind [OR] RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR] RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR] RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR] RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR] RewriteCond %{HTTP_USER_AGENT} ^ProPowerBot/2.14 [OR] RewriteCond %{HTTP_USER_AGENT} ^ProWebWalker [OR] RewriteCond %{HTTP_USER_AGENT} ^ProWebWalker [OR] RewriteCond %{HTTP_USER_AGENT} ^QueryN.Metasearch [OR] RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR] RewriteCond %{HTTP_USER_AGENT} ^RepoMonkey [OR] RewriteCond %{HTTP_USER_AGENT} ^RMA [OR] RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR] RewriteCond %{HTTP_USER_AGENT} ^SlySearch [OR] RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR] RewriteCond %{HTTP_USER_AGENT} ^SpankBot [OR] RewriteCond %{HTTP_USER_AGENT} ^spanner [OR] RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR] RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR] RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR] RewriteCond %{HTTP_USER_AGENT} ^suzuran [OR] RewriteCond %{HTTP_USER_AGENT} ^Szukacz/1.4 [OR] RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR] RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR] RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR] RewriteCond %{HTTP_USER_AGENT} ^Telesoft [OR] RewriteCond %{HTTP_USER_AGENT} ^The.Intraformant [OR] RewriteCond %{HTTP_USER_AGENT} ^TheNomad [OR] RewriteCond %{HTTP_USER_AGENT} ^TightTwatBot [OR] RewriteCond %{HTTP_USER_AGENT} ^Titan [OR] RewriteCond %{HTTP_USER_AGENT} ^toCrawl/UrlDispatcher [OR] RewriteCond %{HTTP_USER_AGENT} ^toCrawl/UrlDispatcher [OR] RewriteCond %{HTTP_USER_AGENT} ^True_Robot [OR] RewriteCond %{HTTP_USER_AGENT} ^turingos [OR] RewriteCond %{HTTP_USER_AGENT} ^TurnitinBot/1.5 [OR] RewriteCond %{HTTP_USER_AGENT} ^URLy.Warning [OR] RewriteCond %{HTTP_USER_AGENT} ^VCI [OR] RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR] RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR] RewriteCond %{HTTP_USER_AGENT} ^WebBandit [OR] RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR] RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR] RewriteCond %{HTTP_USER_AGENT} ^WebEnhancer [OR] RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR] RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR] RewriteCond %{HTTP_USER_AGENT} ^Web.Image.Collector [OR] RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR] RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR] RewriteCond %{HTTP_USER_AGENT} ^WebmasterWorldForumBot [OR] RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR] RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR] RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR] RewriteCond %{HTTP_USER_AGENT} ^Website.Quester [OR] RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR] RewriteCond %{HTTP_USER_AGENT} ^Webster.Pro [OR] RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR] RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR] RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR] RewriteCond %{HTTP_USER_AGENT} ^WebZip [OR] RewriteCond %{HTTP_USER_AGENT} ^Wget [OR] RewriteCond %{HTTP_USER_AGENT} ^Widow [OR] RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR] RewriteCond %{HTTP_USER_AGENT} ^WWW-Collector-E [OR] RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR] RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR] RewriteCond %{HTTP_USER_AGENT} ^Xenu's [OR] RewriteCond %{HTTP_USER_AGENT} ^Zeus RewriteRule ^.*$ emailsforyou.php [L] #RewriteCond %{HTTP_USER_AGENT} ^-?$ #RewriteRule ^.*$ /www/nukecops/noID.php [L] # Code (markup): Then there is a few more lines associated with each module. But yeh, anything wrong here?? This is a part of the googletap mod for phpnuke.
The biggest problem you have is your title tag, which minstrel has mentioned. Actually, the clue is in the google link you provided
I have shortened the thread titles and removed the unneccessary meta tags. Anything else you guys see?? Thanks guys. Ok thanks for the suggestion. I know it may be seen as duplicate content, but in theory every thread is different. Is there anything else i can do about this? SORRY TO MODS... I didnt know an indexing question would not be relevant to SEO..
No they don't. There is little in your entire htaccess file that is helping you and there are more than a few things that may well be hurting you. If it were me, I would dump that entire file. At the very least, remove these lines: $Author: zx $ # $Date: 2003/08/17 14:03:21 $ # deny most common except .php <FilesMatch "\.(inc|tpl|h|ihtml|sql|ini|conf|class|bin|spd|theme|module)$"> deny from all </FilesMatch> <Limit GET PUT POST> Order Allow,Deny Allow from all </Limit> Code (markup): Then delete these: RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Image [OR] RewriteCond %{HTTP_USER_AGENT} ^larbin [OR] RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR] RewriteCond %{HTTP_USER_AGENT} ^Mozilla/2 [OR] RewriteCond %{HTTP_USER_AGENT} ^Mozilla/3.Mozilla/2.01 [OR] RewriteCond %{HTTP_USER_AGENT} ^Xenu's [OR] Code (markup): Also, there is no redirect for your previous domain name there... that's probably the only thing that SHOULD be in .htaccess.
The htaccess was part of the googletap mod from nukemods... Here is the rest of the file. I thought that the redirect went on the first domain name(ie, redirect from metal-forums to metalthrone, put it in metal-forums htaccess???) Anyhow, here is the rest of the htaccess file that it wouldnt let me paste before. This stuff seems relevant but you may think differently. #RewriteCond %{HTTP_USER_AGENT} ^-?$ #RewriteRule ^.*$ /www/nukecops/noID.php [L] RewriteRule ^index.html index.php #Articles RewriteRule ^article-([0-9]*)-([a-z]*)-([0-9]*)-([0-9]*).html modules.php?name=News&file=article&sid=$1&mode=$2&order=$3&thold=$4 RewriteRule ^article-([0-9-]*)-([a-z]*)-([0-9]*)-([0-9]*).html([0-9#]*) modules.php?name=News&file=article&thold=$1&mode=$2&order=$3&sid=$4$5 RewriteRule ^article-topic([0-9]*)-page([0-9]*).html modules.php?name=News&new_topic=$1&pagenum=$2 RewriteRule ^article-category([0-9]*)-page([0-9]*).html modules.php?name=News&file=categories&op=newindex&catid=$1&pagenum=$2 RewriteRule ^article-category-([0-9]*).html modules.php?name=News&file=categories&op=newindex&catid=$1 RewriteRule ^article-print-([0-9]*).html modules.php?name=News&file=print&sid=$1 RewriteRule ^article-friend-([0-9]*).html modules.php?name=News&file=friend&op=FriendSend&sid=$1 RewriteRule ^article-page-([0-9]*).html modules.php?name=News&pagenum=$1 RewriteRule ^article([1-9][0-9]*).* modules.php?name=News&file=article&sid=$1 RewriteRule ^article-topic-([0-9]*).html modules.php?name=News&new_topic=$1 RewriteRule ^allnews.html modules.php?name=News&file=allindex RewriteRule ^news.html modules.php?name=News #AvantGo RewriteRule ^avantgo-print-([0-9]*).html modules.php?name=AvantGo&file=print&sid=$1 RewriteRule ^avantgo.html modules.php?name=AvantGo #Content RewriteRule ^content-([0-9]*)-page([0-9]*).html modules.php?name=Content&pa=showpage&pid=$1&page=$2 RewriteRule ^content-cat-([0-9]*).html modules.php?name=Content&pa=list_pages_categories&cid=$1 RewriteRule ^content-([0-9]*).html modules.php?name=Content&pa=showpage&pid=$1 RewriteRule ^content.html modules.php?name=Content #Downloads RewriteRule ^downloadview-(comments|details|editorial)-([0-9]*)-([[:punct:]/:\-\'{}()._&a-zA-Z0-9+=]*).html modules.php?name=Downloads&d_op=viewdownload$1&lid=$2&ttitle=$3 RewriteRule ^download-mod-([0-9]*).html modules.php?name=Downloads&d_op=modifydownloadrequest&lid=$1 RewriteRule ^download-broken-([0-9]*).html modules.php?name=Downloads&d_op=brokendownload&lid=$1 RewriteRule ^download-shownew-([0-9]*).html modules.php?name=Downloads&d_op=NewDownloads&newdownloadshowdays=$1 RewriteRule ^download-seldate-([a-zA-Z0-9+]*).html modules.php?name=Downloads&d_op=NewDownloadsDate&selectdate=$1 RewriteRule ^download-file-([0-9]*).html modules.php?name=Downloads&d_op=getit&lid=$1 RewriteRule ^ratedownload-([0-9]*)-([[:punct:]/:\-\'{}()._&a-zA-Z0-9+=]*).html modules.php?name=Downloads&d_op=ratedownload&lid=$1&ttitle=$2 RewriteRule ^ratedownload-([0-9]*).html modules.php?name=Downloads&d_op=ratedownload&lid=$1 RewriteRule ^download-(TopRated|MostPopular)-([0-9]*)-(num|percent).html modules.php?name=Downloads&d_op=$1&ratenum=$2&ratetype=$3 RewriteRule ^downloads-(AddDownload|NewDownloads|MostPopular|TopRated).html modules.php?name=Downloads&d_op=$1 RewriteRule ^download([0-9]*)-orderby([a-zA-Z0-9+]*).html modules.php?name=Downloads&d_op=viewdownload&cid=$1&orderby=$2 RewriteRule ^download([0-9]*)-([0-9]*)-([a-zA-Z0-9+]*)-([0-9]*).html modules.php?name=Downloads&d_op=viewdownload&cid=$1&min=$2&orderby=$3&show=$4 RewriteRule ^download-search-([[:punct:]/:\-\'{}()._&a-zA-Z0-9+=]*)-([0-9]*)-([a-zA-Z0-9+]*)-([0-9]*).html modules.php?name=Downloads&d_op=search&query=$1&min=$2&orderby=$3&show=$4 RewriteRule ^download-search-([[:punct:]/:\-\'{}()._&a-zA-Z0-9+=]*)-([a-zA-Z0-9+]*).html modules.php?name=Downloads&d_op=search&query=$1&orderby=$2 RewriteRule ^download-search-([[:punct:]/:\-\'{}()._&a-zA-Z0-9+=]*).html modules.php?name=Downloads&d_op=search&query=$1 RewriteRule ^download-outside-([0-9]*).html modules.php?name=Downloads&d_op=outsidedownloadsetup&lid=$1 RewriteRule ^downloads-cat([0-9]*).html modules.php?name=Downloads&d_op=viewdownload&cid=$1 RewriteRule ^downloads.html modules.php?name=Downloads #Encyclopedia RewriteRule ^encyclopedia-([0-9]*)-page([0-9]*)-([a-zA-Z0-9_-]*).html modules.php?name=Encyclopedia&op=content&tid=$1&page=$2&query=$3 RewriteRule ^encyclopedia-([0-9]*)-page([0-9]*).html modules.php?name=Encyclopedia&op=content&tid=$1&page=$2 RewriteRule ^encyclopedia-([0-9]*)-([a-zA-Z0-9_-]*).html modules.php?name=Encyclopedia&op=content&tid=$1&query=$2 RewriteRule ^encyclopedia-([0-9]*).html modules.php?name=Encyclopedia&op=content&tid=$1 RewriteRule ^encyclopedia-list-([0-9]*).html modules.php?name=Encyclopedia&op=list_content&eid=$1 RewriteRule ^encyclopedia-terms([0-9]*)-([A-Z]*).html modules.php?name=Encyclopedia&op=terms&eid=$1<r=$2 RewriteRule ^encyclopedia-search-([a-zA-Z0-9_-]*).html modules.php?name=Encyclopedia&file=search&query=$1 RewriteRule ^encyclopedia-search.html modules.php?name=Encyclopedia&file=search RewriteRule ^encyclopedia.html modules.php?name=Encyclopedia #Feedback RewriteRule ^feedback.html modules.php?name=Feedback #FAQ RewriteRule ^faq.html modules.php?name=FAQ RewriteRule ^faq-([0-9]*)-([a-zA-Z0-9\+\&\-\/%[:punct:]\.\;]*).html modules.php?name=FAQ&myfaq=yes&id_cat=$1&categories=$2 #Forums RewriteRule ^ftopic-([0-9]*)-([0-9]*)-days([0-9]*)-order(desc|asc)-([A-Za-z0-9_-]*).html modules.php?name=Forums&file=viewtopic&t=$1&start=$2&postdays=$3&postorder=$4&highlight=$5 RewriteRule ^ftopic-([0-9]*)-days([0-9]*)-order(desc|asc)-([0-9]*).html modules.php?name=Forums&file=viewtopic&t=$1&postdays=$2&postorder=$3&start=$4 RewriteRule ^ftopic-([0-9]*)-([0-9]*).html modules.php?name=Forums&file=viewtopic&t=$1&start=$2 RewriteRule ^ftopic-([0-9]*)-(previous|next).html modules.php?name=Forums&file=viewtopic&t=$1&view=$2 RewriteRule ^ftopic(t|p)-([0-9]*)-([A-Za-z0-9_-]*).html modules.php?name=Forums&file=viewtopic&$1=$2&highlight=$3 RewriteRule ^ftopic-([0-9]*)-(watch|unwatch)-([0-9]*).html modules.php?name=Forums&file=viewtopic&t=$1&$2=topic&start=$3 RewriteRule ^ftopic(t|p)-([0-9]*).html modules.php?name=Forums&file=viewtopic&$1=$2 RewriteRule ^ftopic(t|p)-([0-9]*).html#([0-9]*) modules.php?name=Forums&file=viewtopic&$1=$2#$3 RewriteRule ^forum-editprofile.html modules.php?name=Forums&file=profile&mode=editprofile RewriteRule ^forum-userprofile-([0-9]*).html modules.php?name=Forums&file=profile&mode=viewprofile&u=$1 RewriteRule ^forum-register-new.html modules.php?name=Forums&file=profile&mode=register&agreed=true RewriteRule ^forum-register-coppa.html modules.php?name=Forums&file=profile&mode=register&agreed=true&coppa=true RewriteRule ^forum-register.html modules.php?name=Forums&file=profile&mode=register RewriteRule ^forums-posting.html modules.php?name=Forums&file=modules&name=Forums&file=posting RewriteRule ^forum-faq-bbcode.html modules.php?name=Forums&file=faq&mode=bbcode RewriteRule ^ftopic-post-([0-9]*).html modules.php?name=Forums&file=posting&t=$1 RewriteRule ^ftopic-vote-([0-9]*).html modules.php?name=Forums&file=posting&mode=vote&t=$1 RewriteRule ^ftopic-new-([0-9]*).html modules.php?name=Forums&file=posting&mode=newtopic&f=$1 RewriteRule ^ftopic-reply-([0-9]*).html modules.php?name=Forums&file=posting&mode=reply&t=$1 RewriteRule ^ftopic-edit-([0-9]*).html modules.php?name=Forums&file=posting&mode=editpost&p=$1 RewriteRule ^ftopic-smilies.html modules.php?name=Forums&file=posting&mode=smilies&popup=1 RewriteRule ^ftopic-quote-([0-9]*).html modules.php?name=Forums&file=posting&mode=quote&p=$1 RewriteRule ^forum-posting.html modules.php?name=Forums&file=posting RewriteRule ^forums-group([0-9]*).html modules.php?name=Forums&file=groupcp&g=$1 RewriteRule ^fsearch-author-([a-zA-Z0-9_-]*).html modules.php?name=Forums&file=search&search_author=$1 RewriteRule ^fsearch-([0-9]*)-([0-9]*).html modules.php?name=Forums&file=search&search_id=$1&start=$2 RewriteRule ^fsearch-(unanswered|egosearch|newposts).html modules.php?name=Forums&file=search&search_id=$1 RewriteRule ^fsearch-results.html modules.php?name=Forums&file=search&mode=results RewriteRule ^forums-([^index][a-zA-Z0-9_-]*).html modules.php?name=Forums&file=$1 RewriteRule ^forum-c([0-9]*).html modules.php?name=Forums&file=index&c=$1 RewriteRule ^forum-mark.html modules.php?name=Forums&file=index&mark=forums RewriteRule ^forums.html modules.php?name=Forums&file=index RewriteRule ^forum-viewonline.html modules.php?name=Forums&file=viewonline RewriteRule ^forum-([0-9]*)-days([0-9]*)-([0-9]*).html modules.php?name=Forums&file=viewforum&f=$1&topicdays=$2&start=$3 RewriteRule ^forum-([0-9]*)-([0-9]*).html modules.php?name=Forums&file=viewforum&f=$1&start=$2 RewriteRule ^forum-([0-9]*)-mark.html modules.php?name=Forums&file=viewforum&f=$1&mark=topics RewriteRule ^forum-([0-9]*).html modules.php?name=Forums&file=viewforum&f=$1 RewriteRule ^forums.html modules.php?name=Forums #Journal RewriteRule ^journal-search-([a-zA-Z0-9]*).html modules.php?name=Journal&file=search&bywhat=aid&exact=1&forwhat=$1 RewriteRule ^journal(search|delete|commentkill)-(bywhat|jid|onwhat)([a-zA-Z0-9]*)-(forwhat|ref)([a-zA-Z0-9]*).html modules.php?name=Journal&file=$1&$2=$3&$4=$5 RewriteRule ^journal-search-([0-9]*)-([a-zA-Z0-9_-]*).html modules.php?name=Journal&file=search&bywhat=$1&forwhat=$2 RewriteRule ^journal-([a-zA-Z0-9+]*)-(jid|onwhat|disp|op|disp)-([a-zA-Z0-9+]*).html modules.php?name=Journal&file=$1&$2=$3 RewriteRule ^journal-edit.html modules.php?name=Journal&file=edit RewriteRule ^journal(file|op)-([a-zA-Z0-9]*).html modules.php?name=Journal&$1=$2 RewriteRule ^journal.html modules.php?name=Journal #Members List RewriteRule ^members.html modules.php?name=Members_List #NSN Group Downloads RewriteRule ^downloads2-cat([0-9]*)-([0-9]*)-order([a-zA-Z]*).html modules.php?name=Downloads2&cid=$1&min=$2&orderby=$3 RewriteRule ^downloads2-cat([0-9]*)-order([a-zA-Z]*).html modules.php?name=Downloads2&cid=$1&orderby=$2 RewriteRule ^downloads2-cat([0-9]*).html modules.php?name=Downloads2&cid=$1 RewriteRule ^downloads2-rate([0-9]*)-(percent|num).html modules.php?name=Downloads2&op=MostPopular&ratenum=$1&ratetype=$2 RewriteRule ^downloads2-mostpop.html modules.php?name=Downloads2&op=MostPopular RewriteRule ^downloads2-shownew([0-9]*).html modules.php?name=Downloads2&op=NewDownloads&newdownloadshowdays=$1 RewriteRule ^downloads2-new-([0-9]*).html modules.php?name=Downloads2&op=NewDownloadsDate&selectdate=$1 RewriteRule ^downloads2-new.html modules.php?name=Downloads2&op=NewDownloads RewriteRule ^downloads2-broken([0-9]*).html modules.php?name=Downloads2&op=brokendownload&lid=$1 RewriteRule ^downloads2-([0-9]*).html modules.php?name=Downloads2&op=getit&lid=$1 RewriteRule ^downloads2-mod([0-9]*).html modules.php?name=Downloads2&op=modifydownloadrequest&lid=$1 RewriteRule ^downloads2-search-([a-zA-Z0-9_-]*)-([0-9]*)-order([a-zA-Z]*).html modules.php?name=Downloads2&op=search&query=$1&min=$2&orderby=$3 RewriteRule ^downloads2-search-([a-zA-Z0-9_-]*)-order([a-zA-Z]*).html modules.php?name=Downloads2&op=search&query=$1&orderby=$2 RewriteRule ^downloads2.html modules.php?name=Downloads2 RewriteRule ^submit_downloads.html modules.php?name=Submit_Downloads #Private Messages RewriteRule ^messages-post-([0-9]*).html modules.php?name=Private_Messages&file=index&mode=post&u=$1 RewriteRule ^messages-post-([0-9]*).html modules.php?name=Private_Messages&mode=post&u=$1 RewriteRule ^messages-(sentbox|inbox|savebox|outbox).html modules.php?name=Private_Messages&file=index&folder=$1 RewriteRule ^messages.html modules.php?name=Private_Messages #Recommend Us RewriteRule ^recommend.html modules.php?name=Recommend_Us #Reviews RewriteRule ^reviews-new.html modules.php?name=Reviews&rop=write_review RewriteRule ^reviews-preview.html modules.php?name=Reviews&rop=preview_review RewriteRule ^reviews-([0-9]*)-page([0-9]*).html modules.php?name=Reviews&rop=showcontent&id=$1&page=$2 RewriteRule ^reviews-([0-9]*).html modules.php?name=Reviews&rop=showcontent&id=$1 RewriteRule ^reviews-comment-([0-9]*)-([[:punct:]?/:\-\'{}()._&a-zA-Z0-9+=\%]*).* modules.php?name=Reviews&rop=postcomment&id=$1&title=$2 RewriteRule ^reviews-([0-9]*)-delete.html modules.php?name=Reviews&rop=del_review&id_del=$1 RewriteRule ^reviews-([0-9]*)-edit.html modules.php?name=Reviews&rop=mod_review&id=$1 RewriteRule ^reviews-([0-9]*)-delcomment-([0-9]*).html modules.php?name=Reviews&rop=del_comment&cid=$1&id=$2 RewriteRule ^reviews-([a-zA-Z0-9]*)-orderby-([a-z]*)-([a-zA-Z]*).html modules.php?name=Reviews&rop=$1&field=$2&order=$3 RewriteRule ^reviews-sortby-([a-zA-Z0-9]*).html modules.php?name=Reviews&rop=$1 RewriteRule ^reviews.html modules.php?name=Reviews #Search RewriteRule ^search-([a-zA-Z0-9]*)-([0-9]*)-([0-9]*)-([a-zA-Z0-9]*)-([a-zA-Z]*)-([0-9]*).html modules.php?name=Search&author=$1&topic=$2&min=$3&query=$4&type=$5&category=$6 RewriteRule ^search-([a-zA-Z0-9]*)-([0-9]*)-([0-9]*)-([a-zA-Z0-9]*)-([a-zA-Z]*).html modules.php?name=Search&author=$1&topic=$2&min=$3&query=$4&type=$5 RewriteRule ^search-author-([a-zA-Z0-9]*).html modules.php?name=Search&author=$1 RewriteRule ^search-([a-zA-Z0-9_-]*)-([a-zA-Z0-9_-]*).html modules.php?name=Search&query=$1&author=$2 RewriteRule ^search-([0-9]*).html modules.php?name=Search&query=&topic=$1 RewriteRule ^search-users.html modules.php?name=Search&type=users RewriteRule ^search.html* modules.php?name=Search #Sections RewriteRule ^sections-viewarticle([0-9]*)-page([0-9]*).html modules.php?name=Sections&op=viewarticle&artid=$1&page=$2 RewriteRule ^sections-(printpage|viewarticle)-([0-9]*).html modules.php?name=Sections&op=$1&artid=$2 RewriteRule ^sections-listarticles-([0-9]*).html modules.php?name=Sections&op=listarticles&secid=$1 RewriteRule ^sections.html modules.php?name=Sections #Statistics RewriteRule ^stats-([0-9]*)-([0-9]*)-([0-9]*).html modules.php?name=Statistics&op=DailyStats&year=$1&month=$2&date=$3 RewriteRule ^stats-([0-9]*)-([0-9]*).html modules.php?name=Statistics&op=MonthlyStats&year=$1&month=$2 RewriteRule ^stats-([0-9]*).html modules.php?name=Statistics&op=YearlyStats&year=$1 RewriteRule ^advstats.html modules.php?name=Statistics&op=Stats RewriteRule ^stats.html modules.php?name=Statistics #Stories Archive RewriteRule ^archive.html modules.php?name=Stories_Archive RewriteRule ^archive-showall.html modules.php?name=Stories_Archive&sa=show_all RewriteRule ^archive-([0-9]*)-([0-9]*)-([a-zA-Z]*).* modules.php?name=Stories_Archive&sa=show_month&year=$1&month=$2&month_l=$3 #Surveys RewriteRule ^survey-results-([0-9]*)-([a-z]*)-([0-9]*)-([0-9\-]*).html modules.php?name=Surveys&op=results&pollID=$1&mode=$2&order=$3&thold=$4 RewriteRule ^survey-results-([0-9]*).html modules.php?name=Surveys&op=results&pollID=$1 RewriteRule ^survey-([0-9]*).html modules.php?name=Surveys&pollID=$1 RewriteRule ^survey-comment-([0-9]*)-(tid|pid)-([0-9]*)-([a-z]*)-([0-9]*)-([0-9\-]*).html modules.php?name=Surveys&file=comments&pollID=$1&$2=$3&mode=$4&order=$5&thold=$6 RewriteRule ^survey-commreply-([0-9]*)-([0-9]*)-([a-z]*)-([0-9]*)-([0-9\-]*).html modules.php?name=Surveys&file=comments&op=Reply&pid=$1&pollID=$2&mode=$3&order=$4&thold=$5 RewriteRule ^survey-showreply-([0-9]*)-([0-9]*)-([0-9]*)-([a-z]*)-([0-9]*)-([0-9\-]*).html modules.php?name=Surveys&file=comments&op=showreply&tid=$1&pollID=$2&pid=$3&mode=$4&order=$5&thold=$6 RewriteRule ^survey-showreply-([0-9]*)-([a-z]*)-([0-9]*)-([0-9\-]*).html modules.php?name=Surveys&file=comments&op=showreply&tid=$1&mode=$2&order=$3&thold=$4 RewriteRule ^survey-comments.html modules.php?name=Surveys&file=comments RewriteRule ^surveys.html modules.php?name=Surveys #Submit_News RewriteRule ^submit.html modules.php?name=Submit_News #Topics RewriteRule ^topics.html modules.php?name=Topics #Top10 RewriteRule ^top-([[:print:]]*).html modules.php?name=Top&zx=$1 RewriteRule ^top.html modules.php?name=Top #WebLinks RewriteRule ^viewlinkcomments-([0-9]+)-([[:punct:]/:\-\'(){}.&=_a-zA-Z0-9\ ]*).html modules.php?name=Web_Links&l_op=viewlinkcomments&lid=$1&ttitle=$2 RewriteRule ^viewlinkdetails-([0-9]+)-([[:punct:]/:\-\'(){}.&=_a-zA-Z0-9\ ]*).html modules.php?name=Web_Links&l_op=viewlinkdetails&lid=$1&ttitle=$2 RewriteRule ^vieweditorial-([0-9]+)-([[:punct:]/:\-\'(){}.&=_a-zA-Z0-9\ ]*).html modules.php?name=Web_Links&l_op=viewlinkeditorial&lid=$1&ttitle=$2 RewriteRule ^modifylink-([0-9]+).html modules.php?name=Web_Links&l_op=modifylinkrequest&lid=$1 RewriteRule ^brokenlink-([0-9]+).html modules.php?name=Web_Links&l_op=brokenlink&lid=$1 RewriteRule ^outsidelink-([0-9]+).html modules.php?name=Web_Links&l_op=outsidelinksetup&lid=$1 RewriteRule ^linkop-(MostPopular|TopRated)-([0-9]+)-(num|percent).html modules.php?name=Web_Links&l_op=$1&ratenum=$2&ratetype=$3 RewriteRule ^newlinks-([0-9]+).html modules.php\?name=Web_Links&l_op=NewLinks&newlinkshowdays=$1 RewriteRule ^linksnew-([0-9]+).html modules.php?op=modload&name=Web_Links&file=index&l_op=NewLinksDate&selectdate=$1 RewriteRule ^linkop-(AddLink|MostPopular|NewLinks|RandomLink|TopRated).html modules.php?op=modload&name=Web_Links&file=index&l_op=$1 RewriteRule ^ratelink-([0-9]*)-([[:punct:]/:\-\'(){}.&=_a-zA-Z0-9\ ]*).html modules.php?op=modload&name=Web_Links&file=index&l_op=ratelink&lid=$1&ttitle=$2 RewriteRule ^viewlink-([0-9]*).html modules.php?name=Web_Links&l_op=visit&lid=$1 RewriteRule ^links-([0-9]*)-([a-zA-Z0-9]*).html modules.php?name=Web_Links&file=index&l_op=viewlink&cid=$1&orderby=$2 RewriteRule ^links-([0-9]*)-([0-9]*)-([a-zA-Z0-9]*)-([0-9]*).html modules.php?name=Web_Links&file=index&l_op=viewlink&cid=$1&min=$2&orderby=$3&show=$4 RewriteRule ^link-([0-9]*).html modules.php?name=Web_Links&l_op=viewlink&cid=$1 RewriteRule ^links-search-([a-zA-Z0-9_-]*).html modules.php?name=Web_Links&l_op=search&query=$1 RewriteRule ^links-search-([a-zA-Z0-9_-]*)-orderby-([a-zA-Z]*).html modules.php?name=Web_Links&l_op=search&query=$1&orderby=$2 RewriteRule ^links-search-([a-zA-Z0-9]*)-([0-9]*)-orderby-([a-zA-Z]*)-([0-9]*) modules.php?name=Web_Links&l_op=search&query=$1&min=$2&orderby=$3&show=$4 RewriteRule ^links.html modules.php?name=Web_Links #Webmail RewriteRule ^webmail.html modules.php?name=WebMail #Your Account RewriteRule ^userinfo-([a-zA-Z0-9_-]*).html modules.php?name=Your_Account&op=userinfo&username=$1 RewriteRule ^account-([a-z_]*).html modules.php?name=Your_Account&op=$1 RewriteRule ^account.html modules.php?name=Your_Account # $Log: htaccess.GT,v $ # Revision 1.6 2003/11/16 bm # GT-NExtGEn Beta 0.1 # # Revision 1.5 2003/10/22 bm # Added Complete Reviews and Content # # Revision 1.4 2003/10/19 bm # Added Complete Sections and updated GT Distro with file changes # # Revision 1.3 2003/10/18 bm # Added Complete Downloads, Journals, Stats, Forums and Web Links Module Rewrites # Added Corrections/Fixes for FAQ, Reviews and Sections Modules (bm) # # Revision 1.2 2003/08/17 14:03:21 zx # Added for forums poll and category # # Revision 1.1 2003/07/30 01:12:53 zx # Master GoogleTap File -- Use if you desire a GoogleTap Universe # # Code (markup): Thanks for the help. I really appreciate it. This was the only mod i could find for phpnuke which would get url sizes down and get rid of the sess IDS and all the long urls which were unfriendly. Would you suggest a better mod or way of doing this?? Without using mod rewrite?? I know VB would be better as a forum, but honestly cant afford that yet.
Oops. Sorry. You are correct of course - still waking bup with coffee and trying to multitask (poorly as usual)... But honestly, that is a huge .htaccess file, the vast majority of which is completely unnecessary, IMO, and wading through all that rewrite junk is going to put additional loads on your server. Matt Cutts talked recently about ping times and spiders like Googlebot timing out - I wonder if that isn't part of the reason your indexing is slower than you think it should be. Also, you are blocking some things you don't want to block. Be particularly careful about blocking user-agent strings containing "mozilla" or "gecko" because legitimate spiders including Googlebot use that. I note that you're also blocking Xenu which is a link verifier used by many webmasters, including me, to weed out dead links. If I had a link to your site and ran Xenu Link Checker, I'd probably end up deleting your link as a dead link.
Please note, this was all part of the mod downloaded and installed which was set out to help nuke and phpbb forums to get indexed quicker. The contents of the htaccess is nothing to do with me really. Can i get rid of that whole list of blocking agents??? I have already deleted the ones you said? I will go through the rewrite codes and delete the ones for the modules i do not use. The will see if i have any more spider activity. Thanks heaps and really appreciate all the suggestions. Feel free to keep them coming.
I personally don't believe any of that mod_rewrite stuff is necessary or even helpful, and it may in some cases do more harm than good. Modern spiders like Googlebot have no trouble at all indexing dynamic pages. If it were my site, I delete the file, and the sooner the better.
Ok thanks. Just thought id add that Gbot has visited 3060+35 this month.. Is that good or bad in my situ?
It's not bad... the question is what pages is Googlebot accessing? A couple of years back, I installed a MOD on a phpBB forum that through an error was blocking Googlebot. My logs showed Googlebot continuing to show up regularly but over time it started to decline. Eventually, through Google alerts, I discovered that the MOD was serving up an error page to Googlebot. I removed the MOD and things went back to normal.