RE: [PHP] Parsing large file

[Adrian Teasdale]

Hi everyone.

Thanks for advice so far.  Going to try that split command (looks
useful).  IN the meantime, here is the code:

<?php

include "config.php";

// open database
if (!mysql_connect($config['db']['host'], $config['db']['user'],
$config['db']['password'])) {
    die ('Can\' connect to DB!');
}

if (!mysql_select_db($config['db']['dbname'])) {
    die ('Can\' work with DB! Try do FLUSH PRIVILEGES.');
}

// clear table
mysql_query('TRUNCATE TABLE '.$config['table_name']);


$pattern = '/^([a-z0-9]+[a-z0-9-]*[a-z0-9]) NS [a-z0-9.-]+$/i';

$handle = fopen($config['parsefile'], 'r');

$count = 0;
while (!feof($handle)) {
  
        $sql = '';
    $buffer = fgets($handle);

        $count++;
    if ($count>$config['max_count']) break;

     // parse string
     if (preg_match($pattern, $buffer, $match)) {
        $sql .= '("'.$match[1].'")';
     }

     if ($sql) { 
        $sql = 'INSERT IGNORE '.$config['table_name'].' (domain) VALUES
'.$sql;
        echo $sql."<br>";
        mysql_query($sql);
        }

}
fclose($handle);

?>


Any ideas?  Works 100% ok on a smaller file

Thanks

Ade


> -----Original Message-----
> From: Jay Blanchard [mailto:[EMAIL PROTECTED] 
> Sent: 31 August 2004 20:29
> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> Subject: RE: [PHP] Parsing large file
> 
> 
> [snip]
> We have a text file that is 2.2gb in size that we are trying 
> to parse using PHP to put the content into a mysql database. 
> This file contains 40 million individual lines of data.  
> Basically PHP isn't parsing it. Any suggestions of how we 
> could do this? [/snip]
> 
> Post some code and we'll see if we can help.
> 
> -- 
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
> 

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php