SitePoint Sponsor

User Tag List

Results 1 to 8 of 8
  1. #1
    SitePoint Evangelist Tapan's Avatar
    Join Date
    May 2005
    Location
    India
    Posts
    555
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    How to process text files ?

    Hi,

    I have a text file in which i have about 152000 records. 1 line = 1 record. I am trying to add those into the mysql db but the script is unable to process all.

    So i was thinking if i can somehow just add 100 records at a time. For eg. the script opens the files picks 100 records and adds them to mysql db and makes a note that x records has been processed.

    Then it again opens the file and process the other 100 and marks them as done.

    For eg. if i do like: process.php?file=abc.txt&rec=100 so it should open the file abc.txt and get records from 101 to 200 an adds them to the db and then automatically redirects itself to process.php?file=abc.txt&rec=200 then 300 and 400 etc. until the file is completed.

    Can you help me ?

    Thanks.

  2. #2
    SitePoint Zealot
    Join Date
    May 2006
    Posts
    118
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Could you give us a example of what is in the text file?

  3. #3
    SitePoint Evangelist Tapan's Avatar
    Join Date
    May 2005
    Location
    India
    Posts
    555
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Hi,

    Its simple 9 digit numeric values like:

    123456798
    123456798
    123456798
    123456798
    123456798

    Thats it.

    Thanks.

  4. #4
    rajug.replace('Raju Gautam'); bronze trophy Raju Gautam's Avatar
    Join Date
    Oct 2006
    Location
    Kathmandu, Nepal
    Posts
    4,013
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I can show this much code:
    PHP Code:
        $content file("./file.txt");
        foreach(
    $contents as $value){
            
    $sql "insert into tablename set fieldname='$value'";
            
    mysql_query($sql) or die(mysql_error());
        } 
    Which just reads the lines from the file. Then you can manipulate the lines accordingly. I am not sure about creating processes to do only 100 and next 100 like wise except using looping and break statement to break the loop.
    Mistakes are proof that you are trying.....
    ------------------------------------------------------------------------
    PSD to HTML - SlicingArt.com | Personal Blog | ZCE - PHP 5

  5. #5
    SitePoint Evangelist Tapan's Avatar
    Join Date
    May 2005
    Location
    India
    Posts
    555
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Hi rajug,

    I already have done that. This is where the problem is occuring that it fails to do the taks due to so much data. That is why i need the 100 by 100 thing.

    Thanks anyways.

  6. #6
    Working on it... Contrid's Avatar
    Join Date
    Apr 2006
    Location
    Online
    Posts
    955
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Do you get a timeout error?

  7. #7
    SitePoint Evangelist Tapan's Avatar
    Join Date
    May 2005
    Location
    India
    Posts
    555
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    HI,

    Yes...but i noticed one thing that in backgound on server the mysql kept on working and it kept on adding records. Last time i checked it was on 50k records and now it was on 95k..this is very weird. Means that even after timing out the mysql kept on working ?!

    Thanks.

  8. #8
    play of mind Ernie1's Avatar
    Join Date
    Sep 2005
    Posts
    1,252
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    try with pagination
    PHP Code:
    <?php
    include("db.php");
    $self=$_SERVER['PHP_SELF'];
    $pageNum = isset($_GET['p']) ? (int) $_GET['p'] : 1;
    $q = isset($_GET['q']) ? (int) $_GET['q'] : 100;
    $offset = ($pageNum 1) * $q
    $fp file('file.txt'); 
    $num_lines count($fp);
    $maxPage ceil($num_lines/$q); 
    for(
    $i=0;($i<$q)&&(isset($fp[$i]));$i++){
        if(
    $pageNum>$maxPage) break;
        echo 
    "$fp[$i]<br />\n";
        
    mysql_query("INSERT INTO tabel SET field=$fp[$i]") or die(mysql_error());
    }
    if (
    $pageNum >= && ($pageNum $maxPage))
    {
        
    $p $pageNum 1;

    ?>
        <a title="next" href="<?php echo $self ?>?p=<?php echo $p ?>&amp;q=<?php echo $q ?>">next</a>  
    <?php 
    }
    echo 
    $pageNum." of ".$maxPage;
    ?>
    my mobile portal
    ghiris.ro


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •