My plan is to add over 100,000 recipes to http://www.free-recipes.co.uk/ and currently at 50,000 and the site is struggling, running slow. I know the size of HTML/CSS files is on the high side, but I've got similar sites on the same server, but with far fewer posts that's running much faster. Ran a comparative test of these two sites that both run the same WordPress theme, similar plugins- www.free-recipes.co.uk www.google-adsense-templates.co.uk Result was recipe site took 0.18 seconds per KB to load vs 0.03 seconds per KB for the template site, so 6 times slower! My son owns www.free-recipe.co.uk (copying my idea with a different source of recipes, kids hey ) and it's got either 5,000 or 10,000 posts (think 5K) and it loads at 0.02 seconds per KB, so is much faster. Looking for ways to speed this site up and suspect the size of the database is a major issue. I don't think WordPress is designed to run with sites of this size, (I asked on the WordPress support forum a while back, but got no help) anyone familiar with WordPress databases of this size that's sped the site up through some form of database optimization? Note: I've never optimized a database before. Any ideas? David Law
WP is powered on My SQL and should be able to take heavy punishment, though you may need to get a better server.
Don't count me as WordPress expert, but look up WP Super Cache plug in. It stores html versions of pages in ache for fast loading, by passing dynamic scripting. Give it a try and see if it does you any good. It won't solve your problem, but it may make the site look like it loads fast.
Do try one of the caching plugins. It should help significantly. There is WP Super Cache as mentioned above and another which I find a bit easier to use called WP Cache. Links go to the download pages for each.
Your site does not run that slow to me. You must be doing a lot of things right since it is obviously heavily linked to (pr5)...I agree about using WP Super Cache plug in, because by caching html pages you will slowly build extra traffic through archiving
If you're using the latest WP, the database is as optimized as you will need it to be - some other things that may affect speed include extra plugins (the more plugins you have, the more extra code is loaded/run), your hosting load (shared hosting means someone else can be hogging the mySQL), theme complexity (a simple theme loads/runs faster), and so on. One big speed issue I notice is using a WordPress plugin to manage stats - turn it off, as it needs to update the db every page load, and updating mySQL is much slower than reading. Use your website logs instead. I've done a site with 11,000 pages, and WP can handle it - so it quite likely is something else.
I can second what David says about extra plugins... there are some especially that I noticed really slowed things down on my sites. Especially where the plugin has to talk back to it's servers. I agree as well about the theme. Some of not optimized for speed. Especially check the size of header graphics, and if the theme is graphically intense with background images as well you might want to find a simpler one. It's so easy to change the theme, you might want to get a very, very simple one - like veryplaintxt or similar and benchmark your site speed with that. Then you can compare to your theme of choice and if there is quite a difference look at optimizing your theme for speed. 11,000 pages! That's very useful to know. I'm building bigger sites now - stores actually and many will be up around the 10,000 page mark. So it's great to know that it's not going to choke WP. David Law: Your site's loading fairly quickly for me too. I'm using Flock (similar to firefox) and there are problems with the post headings though. The text is aligning wrong, it's colliding with other text. You might want to check it with this little tool (free) which lets you see your site like different browsers do: BrowserShots
Thanks for the responses. It's on a dedicated server which I recently (few months back) moved from another dedicated server that was playing up BADLY. With the old server I could barely add new posts to that site with the PHP script I had created, now it updates smoothly at 500 recipes added at a time. Got another 100K recipes I could add, if I'm happy with the speed. Dedicated Server stats Intel DUAL CORE E2140 2GB Ram 500 GB SATA HD Having no problems with the server, only had to reboot it once since getting it in March. Other times I've had server problems it's been about the memory (had a server with 500MB, that was my first server) and the problems been running the memory at around 99% causing Apache to crash. Current servers 2GB memory is handling things fine, not come close to any serious memory issues. I was running the site with WP-cache at 35,000 recipes for months and it helped a bit (still on the slow side which is why I stopped adding recipes at 35K). Was trying to run an ecommerce plugin, but couldn't get it working without disabling the cache, (cart didn't work correctly) so that's why it's not running currently. Can't use the cache if I use the ecommerce plugin. Recently added 15,000 more recipes and have had to disable the ecommerce plugin as well now to speed things up! The ecommerce plugin was heavy on the code usage on every single page of the site!! Currently running these plugins- Akismet Won't cause problems. Fold Category List http://www.webspaceworks.com/resources/cat/wp-plugins/31/ Used to only shows one level of category links otherwise would show 250+ category links on the menu)! I once tried to create a site using WP that was a book format, (using public domain books) so each post would have 30+ pages using the more tag. Figured out how to get the result I wanted, but the way WP works meant every page that was created accessed the entire book (some books are several MB in size) rather than the 50KB for a particular page (so was processing 1MB plus data to pull 50Kb from it for every page!). So had to scrap that idea. Concerned WP might be doing something similar with category links and 'adding' 250 links is quite a overhead. Even with this plugin the menus are on the large aide, but with 50,000 posts and 100,000+ planned I have to get the site navigated some way. Popularity Contest for WordPress 2.3 (with some updates I added to fix problems in WP 2.51) http://alexking.org/projects/wordpress Ranks posts according to how many comments, number of visitors etc... used on this site to create the "50 Most Viewed Recipes" menu links. I doubt this is an issue, though 50 links is on the high side, so might reduce it down a bit. aLinks http://www.headzoo.com/alinksg Used to interlink keywords to relevant posts. Have used this to link a LOT keywords (pretty much every link on a post you see is due to this plugin) This is one plugin that might cause problems due to the number of links it's creating, but I don't understand it enough to know either way to be honest. Post Teaser http://alexking.org/projects/wordpress Makes a fancy excerpt on archive pages. Much better than the default excerpt coding as allows archive pages to be used with the aLinks plugin (otherwise aLinks only works with post pages). Doubt it's an issue. Related Posts http://www.w-a-s-a-b-i.com/archives/2006/02/02/wordpress-related-entries-20/ Used to create a list of related posts, I use it at the bottom of posts to list multiple related recipes. Guess it could cause a problem with such a big site. WP-PageNavi http://lesterchan.net/portfolio/programming.php Archive page pagination in a format I prefer. Unlikely to cause a problem. As you can see there's a few plugins that have the potential to slow a site down, will have to disable plugins one by one and see if any are an issue. Currently it's running a lot faster than a few days back when I added another 5K recipes to break the 50K mark, but is still on the slow side if you ask me. Right after I got to 50K the site was really hard to access and I got worried Difficult to know if users are quiting the site early due to loading times, I've recently (before adding the latest 15K recipes) increased AdSense CTR to around 5 1/2% to 6% (up from under 2.5%, which is quite high). I've not seen a drop in CTR last few days while adding new recipes**, I suppose if pages were loading slower than a week ago I might see CTR drop as AdSense registers an impression for partly loaded pages (if an ad loads). This would suggest there's not as big a problem as I see. **while uploading recipes 500 at a time it gets really hard to connect to the site, (practically takes the site out of action for 5-10 mins for every 500 recipes added) I've lost traffic those days when adding recipes, but not CTR %. Just added some ads to try out, so that will add a little to the loading times, but that will be temporary. When I determine which convert best will reduce it down to a couple of ads. I read that WP 2.51 was better optimised database wise, so might be one of those plugins. I'd hate to turn aLinks off as adds thousands of internal links so hoping it's not that one. David Law
I've written about it in the past: http://activeblogging.com/info/how-to-speed-up-wordpress-blog/ In your case, the easiest solution is to time and turn off: -Set up timing - many themes display the time in the html; if not, add the code (available from header.php and footer.php in the /default/ theme) and start timing pages. -Turn plugins off one by one, and time page loads. load the pages more than once to get a good average. -Figure out how to handle the plugins that are causing problems. With a large site like yours, my guess is that the cache plugin is doing little, and might be hurting you - it's great for a site with one or two pages read frequently (like after being dugg), but for a large number you're actually having a performance penalty added. I'm almost certain the alinks plugin is killing you - the code involved in doing a lot of keywords requires a lot of data reading, and a lot of processing per page. And as you've mentioned, wp loads the whole article in, even to show just part. That's a db issue rather than wp (the whole record read in), but it adds up. Another hit - if you have javascript, loading it can take time.
are you using the default mysql settings or have you already changed those to higher values? my database although its not for wordpress has 8 million rows , 3 gbs of data and still loads fast even with 250k google bot hits per day
Thanks for the timing tip, makes checking speed so much faster than using external tools. Found with everything currently running it was taking around 6-8 seconds to load with over 600 queries! Disabling all plugins brought it down to around 4 seconds and still 500+ queries. Another site, same theme, not all the same plugins was running much faster. So something weird is going on here, not sure why I'm seeing so many queries. I've left the time code up for now, so you can go to http://www.free-recipes.co.uk/ and look at the bottom left corner of the page to see for yourselves. Last time I loaded it got 694 queries. 5.387 seconds. Which is one of the faster loads with all the original plugins still running. Refreshed the home page just now 4 times- 694 queries. 5.387 seconds. 694 queries. 6.146 seconds. 694 queries. 6.312 seconds. 694 queries. 6.977 seconds. 694 queries. 6.330 seconds. That is not good. Why would I have so many queries when this page http://www.google-adsense-templates.co.uk/tag/wordpress-themes which has the same theme and mostly the same plugins gives 22 queries 1.514 seconds (home page 27 queries 1.008 seconds)? This site has less than 20 posts and runs the same plugins minus Fold Category List (not installed) and aLinks is turned off. Checked a site with 2,500 posts same sort of setup 66 queries 1.321 seconds. That uses a social book mark plugin that might account for the extra queries. I can't think of anything that would cause this site to have 600+ queries. David Law
I'm using the default settings. Basically whatever Centos 5 default settings are that's what I'm using. I know nothing about optimising a database, what higher values are you referring to? David Law
Not sure why I didn't try this before posting my last 2 replies, but. Disabled the category sidebar item via widgets, this meant no category links shown (normally 27 of them on the home page) and got the following speed- 110 queries. 3.392 seconds. Although normally it shows 27 category links on the home page there are 296 categories in total and I've used a plugin to only show the top level. So it looks like even though over 250 of the categories aren't shown they are still accessed by WordPress resulting in hundreds of extra queries. Would that cause the slow speed, 400+ additional queries? If that makes sense at least I know what is causing the problem, now to find a fix! David Law
Plugins includes widgets - to really get a feel, I'd recommend switching in the default theme, turning off all widgets, and checking the speed again. And I can already tell you - I'm sure the 'top 50' plugin is consuming a LOT of database time. From a programming perspective, finding 'the most' of anything takes time, and 50 of the topmost is very time consuming. Even if the plugin optimizes its part of the database (and I doubt it), the data also needs to put INTO the database, and that takes time. For a plugin like this, 1,000 posts is probably the maximum it seems on a typical blog. in the 100,000 arena, you need heavily optimized plugins - and I'm guessing this isn't.
i changed my mysql settings to... key_buffer = 1500M table_cache = 4000 sort_buffer_size = 5M max_connections = 400 query_cache_type = 1 query_cache_limit = 1M query_cache_size = 1500M innodb_flush_log_at_trx_commit = 0 innodb_buffer_pool_size = 700M innodb_additional_mem_pool_size=50M innodb_log_file_size = 175 innodb_log_buffer_size=8M i have 4 gb memory though but with 8 million pages, 1 page was taking about 15 seconds to load before changes, after changes about 0.61 sec 0.02 per kb