Hi, I need to write some php to search a web folder for all files whose name contains a search string. The folder is HUGE with hundreds of thousands of files in it. The script needs to run every time a page is loaded, so needs to be fast. Is there a nice php function that can do this? file_exists isn't really appropriate. Is it even possible, or should I write a cron job that puts all the filenames into a text file/database (weekly) and then search through that every time instead? Any help appreciated.
How to deal with this depends on how the files are getting there and what kind of matching you're doing. Are most of the searches arbitrary substring, or do they tend to be from the beginning of the file or have some other characteristic that can be used to rule some of them out? Once you have understood the patterns of your data and usage it's easier to figure out an effective search strategy. One thing that's for sure, using filesystem functions to search through the list will be slower than searching a separate index containing just the names. On most filesystems, by the way, having hundreds of thousands of files in one directory will slow down all operations on those files, perhaps dramatically. You could at least hash out the first few characters (assuming those have reasonable variation) by putting them in subdirectories like a/a/aardvark, a/b/abacus, a/c/across, etc.