1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

best way to store data

Discussion in 'PHP' started by mkeen, May 19, 2005.

  1. #1
    Hi

    Can someone tell me what the best way to store data locally with php is. I want to use a flat file rather than a normal database.

    The data I will be storing is basicly

    1000's of keyword and urls and also some other data that determines when the data was last updated. I would imagine storing this data in a normal txt file could be very slow, I need it to be as fast as possible really.
    SEMrush
    Any help with be appreciated as I have no idea where to start with this.


    Thanks
    Matthew Keen
     
    mkeen, May 19, 2005 IP
    SEMrush
  2. noppid

    noppid gunnin' for the quota

    Messages:
    4,246
    Likes Received:
    232
    Best Answers:
    0
    Trophy Points:
    135
    #2
    MySQL and PHP are the ticket. Blazingly fast stuff.
     
    noppid, May 19, 2005 IP
  3. Stin

    Stin Guest

    Messages:
    264
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #3
    yep you should definatly put this stuff in a database of some kind. MySQL is the most widely used so you should probably stick with it. VERY easy to integrate mysql and php.
     
    Stin, May 19, 2005 IP
  4. mkeen

    mkeen Peon

    Messages:
    186
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Thanks for the reply guys but please please please

    Read what I said before answering.


    Thanks

    Matthew
     
    mkeen, May 19, 2005 IP
  5. noppid

    noppid gunnin' for the quota

    Messages:
    4,246
    Likes Received:
    232
    Best Answers:
    0
    Trophy Points:
    135
    #5
    Well then, let's see.

    You need PHP and MYSQL. You should know HTML, Preferably XHTML.

    You need to define the data you will collect and make tables to store it with a place for each field of data in records.

    You can define your DB in phpmyadmin, however, you will probably be writing php and mysql and html code by hand to manage and use the data.
     
    noppid, May 19, 2005 IP
  6. Stin

    Stin Guest

    Messages:
    264
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #6
    ok, then why do you want to use a flat file instead of a db?
     
    Stin, May 19, 2005 IP
  7. frankm

    frankm Active Member

    Messages:
    915
    Likes Received:
    63
    Best Answers:
    0
    Trophy Points:
    83
    #7
    @mkeen:
    you probably do not have mysql on your server (get it, or change hosting co), but for the time being:

    pick a (any) character that's not in either one of your fields (say TAB), do something like:


    $data = array();
    
    $fp = fopen("data", "r");
    if ($fp)
    {
      while(!feof($fp))
      {
        $buf = fgets($fp);
        if (strlen($buf) > 0)
        {
          $data[] = explode("\t", $buf);
        }
      }
      fclose($fp);
    }
    
    print_r($data);
    PHP:
    create a file called "data", fields should be TAB seperated
    (didn't test this, so please fix syntax errors yourself)
     
    frankm, May 19, 2005 IP
  8. sarahk

    sarahk iTamer Staff

    Messages:
    26,223
    Likes Received:
    3,856
    Best Answers:
    108
    Trophy Points:
    665
    #8
    Take a look at webanalyse

    They use flat files, it's fast but it's also a pain the moment you want to do any custom queries on your data. But they serialise the data and it works really well so trawl through the code and use it as a case study.

    For a while there I was saving data in individual text files - 1 per relevant hit. See a wee code snippet I had to write as a result :D

    Text files are text files at the end of the day.
    Sarah
     
    sarahk, May 20, 2005 IP
  9. mkeen

    mkeen Peon

    Messages:
    186
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Thanks for the replies everyone, I did clearly say "I want to use a flat file rather than a normal database."

    I have now used something similar to what sarahk said, Im using a directory called urldata/ and storing each seperate record as a new file $url.txt in that directory, its working out very qell and is blazing fast because it only has to load one small file at a time rather than 1 large file storing 1000's of records.


    Thanks
    Matt
     
    mkeen, May 20, 2005 IP
  10. palespyder

    palespyder Psycho Ninja

    Messages:
    1,254
    Likes Received:
    98
    Best Answers:
    0
    Trophy Points:
    168
    #10
    Just out of sick curiousity, why flat files as opposed to a dbms?
     
    palespyder, May 20, 2005 IP
  11. jbw

    jbw Peon

    Messages:
    343
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #11
    I always find the use of the term flat file as oppsed to a relational db odd, since most relational dbs people then talk about still are using the file systems (there are some that dont but usually the same people that talk in terms of 'flat files' don't have a clue of the impliementaion details).

    Anyway, show us the output of your phpinfo() and we may be able to make better suggestions, but off hand if it is available, take a look at http://us2.php.net/manual/en/ref.dba.php some of these hash and isam libraries can blow the doors of mysql for the right problem sets.

    Another option, though it is a db, it just uses a single file in your area to store the data, is sqlite.
     
    jbw, May 20, 2005 IP
  12. sadcox66

    sadcox66 Spirit Walker

    Messages:
    496
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    0
    #12
    PHP4 will allow .xml if the columns and structure are going to change in future

    PHP 5 bundles SQLite which allows you to perform SQL statement on Flat-Files !
     
    sadcox66, May 20, 2005 IP
  13. jbw

    jbw Peon

    Messages:
    343
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #13
    well not exactly, there are some perl modules that do that. sqlite just stores the db in a locally opened single file. On the other hand mysql uses client server, but in the back end, it is storing the data in files on the filesystems.

    The term 'flat-files' has no real tech. meaning.
     
    jbw, May 20, 2005 IP
  14. mkeen

    mkeen Peon

    Messages:
    186
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #14
    Reason im going with flat files is that the code I am writing is for a small advertising network, the user installs the files on his server and each url is assigned a random url, the url&anchor is requested from my server and then stored locally, I dont want a database because not everyone will have access to one.
     
    mkeen, May 21, 2005 IP
  15. jbw

    jbw Peon

    Messages:
    343
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #15
    another option for you then, since you are just doing random data access is fixed length records, and doing an fseek to the len*recno
     
    jbw, May 21, 2005 IP
  16. microtony

    microtony Peon

    Messages:
    281
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #16
    MySQL or Plain-text Database are the choices.
     
    microtony, May 23, 2005 IP
  17. noppid

    noppid gunnin' for the quota

    Messages:
    4,246
    Likes Received:
    232
    Best Answers:
    0
    Trophy Points:
    135
    #17

    How many records will be in this file?

    Looping through and entire file with regex's for search is really poor code. You won't be doing anyone any favors with a large file. I see no reason why a site worth advertising can't invest in the extra dollar for a database. :/

    Anywho, this could be ok for a small file. I lost track, where are you stuck?
     
    noppid, May 23, 2005 IP
  18. mkeen

    mkeen Peon

    Messages:
    186
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #18
    The reason you lost track is because you dont seem to be reading all of the post, are you just scanning over it? What I was asking was what the best way to store data *>>without<<* a database (like mysql etc) was. Also as I have allready said, Im not looping through an entire file with regex, As I have allready stated each record is now stored in its own file.

    In the following structure.

    linksdata/
    linksdata/url.com.txt
    linksdata/url.com-sub.txt
    linksdata/url.com-sub2.txt
    linksdata/url.com-page1.txt
    linksdata/url.com-page2.txt

    Etc etc etc, only one file is accessed and the file only holds one small record.

    Anyway as Ive allready said I have sorted this problem now.
     
    mkeen, May 23, 2005 IP
  19. yabsoft

    yabsoft Active Member

    Messages:
    118
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #19
    A good way is to serialize $var into string,then store in flat file.
    step1:
    $s=serialize($you_wanted_store_var);
    step2:
    $fp=fopen($CFG[scriptDir]."/cache/$name.txt",'w');
    fputs($fp,$s);
    fclose($fp);

    Restore from flat files:
    $fp=@fopen($CFG[scriptDir]."/cache/$name.txt",'r');
    $content=@fread($fp,@filesize($CFG[scriptDir]."/cache/$name.txt"));
    $you_wanted_store_var=@unserialize( $content );
     
    yabsoft, Jun 6, 2005 IP