[PHP] Need help adding dBase support to PHP
Hi all: I'm not versed at all with Linux, and I need some help in configuring PHP to allow me to use the dBase-related functions. From what I read in the help files, it seems that I need to recompile PHP with the "--enable-dbase" command, but I don't know how to do this. Can someone out there help me out? All help will be greatly appreciated! Thanks, Jon
Re: [PHP] Need help adding dBase support to PHP
Hi Nathan: I have no idea. Where would I look to find this out? Will phpinfo() give me this? And, if so, where would I find it? Jon - Original Message - From: Nathan Nobbe To: Jon Westcot Cc: PHP General Sent: Thursday, October 11, 2007 5:00 PM Subject: Re: [PHP] Need help adding dBase support to PHP On 10/11/07, Jon Westcot <[EMAIL PROTECTED]> wrote: Hi all: I'm not versed at all with Linux, and I need some help in configuring PHP to allow me to use the dBase-related functions. From what I read in the help files, it seems that I need to recompile PHP with the "--enable-dbase" command, but I don't know how to do this. Can someone out there help me out? All help will be greatly appreciated! what linux distribution are you using? -nathan
Re: [PHP] Need help adding dBase support to PHP
Hi again, Nathan: I'm not certain how to get to a command line. The server is a shared server provided by GoDaddy, if that's any help. They told me that I needed to change something in the .htaccess file, but that didn't sound right at all. Thanks again for your help. Jon - Original Message - From: Nathan Nobbe To: Jon Westcot Cc: PHP General Sent: Thursday, October 11, 2007 5:08 PM Subject: Re: [PHP] Need help adding dBase support to PHP On 10/11/07, Jon Westcot <[EMAIL PROTECTED]> wrote: Hi Nathan: I have no idea. Where would I look to find this out? Will phpinfo() give me this? And, if so, where would I find it? you might try uname -a from the command line; but theres no guarantee it will show the distribution. it will show the kernel, and sometime kernels are named after the distro since they are often modified by the distribution. -nathan
Re: [PHP] Need help adding dBase support to PHP
Hi again: Thanks for the info. From what I can see, GoDaddy does NOT provide access to ssh. Any other thoughts? Or is there some way to tell me, in general terms, how to configure PHP to allow the dbase functions to be used? Jon - Original Message - From: Nathan Nobbe To: Jon Westcot Cc: PHP General Sent: Thursday, October 11, 2007 5:34 PM Subject: Re: [PHP] Need help adding dBase support to PHP On 10/11/07, Jon Westcot <[EMAIL PROTECTED]> wrote: Hi again, Nathan: I'm not certain how to get to a command line. The server is a shared server provided by GoDaddy, if that's any help. They told me that I needed to change something in the .htaccess file, but that didn't sound right at all. they may be right. did they give you some sort of documentation you could point us to? also, to determine if you have command line access, look around and see if they provide ssh access to the machine. if so i can tell you how to ssh in, even if you have windows or mac. that will give you a command line prompt on the remote machine. -nathan
Re: [PHP] Need help adding dBase support to PHP
Hi Nathan: The page you referenced in the PHP documentation is exactly what I saw that indicated that the dBase functions needed to be compiled in with the --enable-dbase directive. What I need to know is HOW to do this, even in general terms. GoDaddy is being very little help (no big surprise there). Their last message indicated updating values in the php.ini file would let this work, but, as you pointed out, this isn't an option with the php.ini. I looked in the phpinfo data and couldn't find anything that matched "dbase" anywhere; certainly, no section in the output indicated any dBase settings whatsoever. I'm fearing that I'm going to have to find some other way to handle the extraction of data from DBF files if I can't get this to work. Thanks again for your help. Jon - Original Message - From: Nathan Nobbe To: Jon Westcot Cc: PHP General Sent: Thursday, October 11, 2007 6:06 PM Subject: Re: [PHP] Need help adding dBase support to PHP On 10/11/07, Jon Westcot <[EMAIL PROTECTED]> wrote: Hi again: Thanks for the info. From what I can see, GoDaddy does NOT provide access to ssh. Any other thoughts? Or is there some way to tell me, in general terms, how to configure PHP to allow the dbase functions to be used? if godaddy is recommending that you place values in a .htaccess file they probly mean you should upload a .htaccess file via ftp. im assuming thats how the give you access to the system (since there is no ssh access). im a bit unsure of the .htaccess setting, though i recently posted a small how-to on using .htaccess files to override settings in php.ini; this is common in shared hosting environments. http://gentoo-wiki.com/HOWTO_php.ini_overrides_w/_.htaccess basically, to have dbase support, php must be compiled w/ --enable-dbase, per the documentation. http://www.php.net/manual/en/rlef.dbase.php perhaps, it is compiled in and the php.ini settings are preventing you from using the functions (though that sounds hard to believe). you might try tossing a phpinfo script on the box and looking for dbase in the output; particularly look for --enable-dbase, or --disable-dbase. you should see a dbase section also, if its compiled in. if it is there we can get a better idea of what sort of values could be placed in the .htaccess file that would influence the behavior of the dbase component on that system. -nathan
Re: [PHP] Need help adding dBase support to PHP
Hi Nathan: Thanks for the info. Unfortunately, it's not my call which host to use. I'm just being asked to help write some code, and I know that there are DBFs in the process that need to be examined and brought into MySQL on a daily basis, if not more often. From what I've been told, the GoDaddy server is a shared server, which I guess means that we, the underlings using it, can't really get in and make whatever changes we'd like to make. I did use the browser search to look for any occurence of "dbase" in the phpinfo-returned data, but nothing showed up. I appreciate the help you've given. I'll just have to wait and see if the person who owns the site is able to get anywhere with the GoDaddy tech support. Thanks again, Jon - Original Message - From: "Nathan Nobbe" <[EMAIL PROTECTED]> To: "Jon Westcot" <[EMAIL PROTECTED]> Cc: "PHP General" Sent: Friday, October 12, 2007 12:27 AM Subject: Re: [PHP] Need help adding dBase support to PHP > On 10/12/07, Jon Westcot <[EMAIL PROTECTED]> wrote: > > > > Hi Nathan: > > > > The page you referenced in the PHP documentation is exactly what I saw > > that indicated that the dBase functions needed to be compiled in with the > > --enable-dbase directive. What I need to know is HOW to do this, even in > > general terms. > > > basically you run about 3 commands. > configure > make > make install > and configure is the command you would pass --enable-dbase to. > windows systems typically have binary installs available; but i really have > never setup php on a windows box. > > anyway, the reason i asked about the distribution to begin with is many > distributions handle php configuration / installation so its quite simple. > on gentoo you just add or remove a use flag and emerge it again if you want > to make a change. on debian there are modules so i believe you can just > apt-get a new module if it isnt already installed. > seems nice, but ill tell you, on gentoo you spend time compiling; on debian > you spend time saying where is that package i want and how do i trick > apt-get to install it correctly. but i digress... :) > > GoDaddy is being very little help (no big surprise there). Their last > > message indicated updating values in the php.ini file would let this work, > > but, as you pointed out, this isn't an option with the php.ini. I looked > > in the phpinfo data and couldn't find anything that matched "dbase" > > anywhere; certainly, no section in the output indicated any dBase settings > > whatsoever. > > > if you havent done so i would just use the search feature of your browser > and type in dbase on that page. you should see something in the configure > directive; either --enable-dbase or --disable-dbase most likely. > > I'm fearing that I'm going to have to find some other way to handle the > > extraction of data from DBF files if I can't get this to work. > > > > are you really committed to godaddy at this point ? > > -nathan > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Can a PHP script process a file on a remote computer?
Hi all: I'm working on a project that requires frequent updating of a large amount of data to a MySQL database. Currently, I'm trying to transfer a CSV file to the server, have it process the file, and then delete it once the processing is complete. Rather than waste the up-front time of having to upload the file (which is currently timing out without fully uploading the file and I have no idea how to resolve this!), is it possible to have the server open the specified file remotely and read it using the fgetcsv() function? If so, how? Any help you can give me will be GREATLY appreciated! Thanks, Jon
Re: [PHP] Can a PHP script process a file on a remote computer?
Thanks, Wolf. I'm going to have to do a lot of long processes -- uploads, queries, and the like -- and I appreciate knowing that I can adjust that. Now if I can just find the PHP.INI file (Stupid GoDaddy-shared-domains; can't find anything on 'em.) Jon - Original Message - From: "Wolf" <[EMAIL PROTECTED]> To: "Jon Westcot" <[EMAIL PROTECTED]> Cc: "PHP General" Sent: Wednesday, October 24, 2007 1:16 PM Subject: Re: [PHP] Can a PHP script process a file on a remote computer? > Go into your php.ini file and increase the script timeout length, which should allow the upload to finish. > > > Jon Westcot <[EMAIL PROTECTED]> wrote: > > Hi all: > > > > I'm working on a project that requires frequent updating of a large amount of data to a MySQL database. Currently, I'm trying to transfer a CSV file to the server, have it process the file, and then delete it once the processing is complete. Rather than waste the up-front time of having to upload the file (which is currently timing out without fully uploading the file and I have no idea how to resolve this!), is it possible to have the server open the specified file remotely and read it using the fgetcsv() function? If so, how? > > > > Any help you can give me will be GREATLY appreciated! > > > > Thanks, > > > > Jon > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Can a PHP script process a file on a remote computer?
Hi Andrew: Thanks for the reply. If I use fopen() to open the remote file, how do I specify its location? Isn't it going to need to know not only the address of the computer I'm using but the path on that computer to the file? Does it get all that from the name I pass over to it? Also, can I adjust the allow_url_fopen setting from within a PHP script, or do I need to manually set it in the PHP.INI file? The server I'm using -- not my own; this is for a friend -- is a shared server with GoDaddy and they make it so incredibly hard to find anything. Thanks! Jon - Original Message - From: "Andrew Ballard" <[EMAIL PROTECTED]> To: "PHP General" Sent: Wednesday, October 24, 2007 1:15 PM Subject: Re: [PHP] Can a PHP script process a file on a remote computer? > How big is the file? fgetcsv() will read any file that you can get a > handle to using fopen() or fsockopen(), including remote files. > However, if allow_url_fopen is not enabled, you can't use fopen(). > Also if the file is so large that your script is timing out while > waiting for it to be uploaded, it could also timeout just trying to > read the file remotely as well. > > Andrew > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Can a PHP script process a file on a remote computer?
Hi again, Andrew: You had asked some questions that I forgot to address with my previous reply, so here goes: The test file that I'm using -- which probably is a small version of the normal file -- is around 60 MB in size (just under 30,000 records). I checked the phpinfo info and it appears that the allow_url_fopen setting is set to "On," which I take is a good thing. How, then, do I specify the filename at the server so that it points back to the file that is on my computer (or whatever computer is trying to initiate the process)? Thanks again for your help, Jon - Original Message ----- From: "Jon Westcot" <[EMAIL PROTECTED]> To: "PHP General" Sent: Wednesday, October 24, 2007 1:22 PM Subject: Re: [PHP] Can a PHP script process a file on a remote computer? > Hi Andrew: > > Thanks for the reply. > > If I use fopen() to open the remote file, how do I specify its location? > Isn't it going to need to know not only the address of the computer I'm > using but the path on that computer to the file? Does it get all that from > the name I pass over to it? > > Also, can I adjust the allow_url_fopen setting from within a PHP script, > or do I need to manually set it in the PHP.INI file? The server I'm > using -- not my own; this is for a friend -- is a shared server with GoDaddy > and they make it so incredibly hard to find anything. > > Thanks! > > Jon > > > - Original Message - > From: "Andrew Ballard" <[EMAIL PROTECTED]> > To: "PHP General" > Sent: Wednesday, October 24, 2007 1:15 PM > Subject: Re: [PHP] Can a PHP script process a file on a remote computer? > > > > How big is the file? fgetcsv() will read any file that you can get a > > handle to using fopen() or fsockopen(), including remote files. > > However, if allow_url_fopen is not enabled, you can't use fopen(). > > Also if the file is so large that your script is timing out while > > waiting for it to be uploaded, it could also timeout just trying to > > read the file remotely as well. > > > > Andrew > > > > -- > > PHP General Mailing List (http://www.php.net/) > > To unsubscribe, visit: http://www.php.net/unsub.php > > > > > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] How do I specify a local file for fopen()?
Hi all: I've been beating my head against a brick wall trying to figure this out and I'm still no closer than I was two weeks ago. How do I specify a local file on my computer to use with fopen() on the server? I've checked and the allow_url_fopen setting is set to On. I use the html to let me browse to the file. This, however, forces me to also POST the entire file to the server, which I DO NOT WANT it to do. I just wanted to be able to use the button to get to the file name. But, even when I do this, the file name returned in the $_FILES array doesn't give me a file name that fopen() will actually open. Do I somehow have to get the server to recognize my computer as an http-based address? If so, how do I do this? The computer that has the file to be opened is a Windows-based computer (running WinXP or Vista), and it obviously has an Internet connection. Do I need to retrieve, from the server, my computer's IP address and use that, in whole or in part, to reference the file to be opened? If so, how? While I'm asking questions, does anyone know how to keep the file referenced in the setup from actually being sent? All I think I really need is the NAME of the file, not its actual contents, since I'm hoping to use fopen() to open the file and then to use fgetcsv() to retrieve the contents. ANY help you all can send my way will be greatly appreciated! Thanks in advance, Jon
[PHP] Question about php.ini file
Hi all: I've learned, much to my pleasure, that I can place a php.ini file in the root directory of my shared domain to allow some customization of the php settings. However, when I do this, I keep getting weird results. Some of the values that I change appear as I've asked them to be. Some appear to not change at all. And some have been changing to what appears to be default values for php. I'm wondering a few things here. First, my computer is Windows-based, but the server is Linux-based. Do I need to somehow convert the Windows text file line termination characters to their Linux variants? If so, how? Is it possible that I'm trying to change values beyond some upper limit which is causing php to use the default value? The main values that I'm trying to change are: post_max_size upload_max_filesize memory_limit max_execution_time max_input_time If anyone knows of upper limits for these, I'd appreciate learning about them. As always, any help you can send my way will be greatly appreciated! Thanks, Jon
Re: [PHP] Question about php.ini file
Hi Cristian: Thanks for the replies. In this case, what I've been noticing is that most of the values have gone DOWN (i.e., gotten smaller) after I placed my own php.ini file in place. For example, upload_max_filesize was 8M before I placed my file, but, afterwards, it was 2M! I'm trying to increase this size, not decrease it. Could it be the line-ending characters that are causing the problem? Could PHP be seeing the new file but not realizing how to process it, which would cause the values to flop over to their defaults? Thanks, Jon - Original Message - > You must take in count that before php even gets to receive something, > it all passes through the HTTP server (apache or something). for most of > these settings there are equivalents in the config file of the server > that you need to change accordingly. > this might be the cause of the weird results. for example, if your limit > for uploaded files is 10M in php but the server is set to accept max 5M, > 5M will be the limit > > Jon Westcot wrote: > > Hi all: > > > > I've learned, much to my pleasure, that I can place a php.ini file in the root directory of my shared domain to allow some customization of the php settings. However, when I do this, I keep getting weird results. Some of the values that I change appear as I've asked them to be. Some appear to not change at all. And some have been changing to what appears to be default values for php. > > > > I'm wondering a few things here. First, my computer is Windows-based, but the server is Linux-based. Do I need to somehow convert the Windows text file line termination characters to their Linux variants? If so, how? Is it possible that I'm trying to change values beyond some upper limit which is causing php to use the default value? > > > > The main values that I'm trying to change are: > > > > post_max_size > > upload_max_filesize > > memory_limit > > max_execution_time > > max_input_time > > > > If anyone knows of upper limits for these, I'd appreciate learning about them. > > > > As always, any help you can send my way will be greatly appreciated! > > > > Thanks, > > > > Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Question about php.ini file
Hi Nathan: I'm not certain if I have all of the items you've mentioned, but I'll look into it. And thanks for the link; I'll check that out, too. Jon - Original Message - From: Nathan Nobbe To: Jon Westcot Cc: PHP General Sent: Sunday, November 04, 2007 5:05 PM Subject: Re: [PHP] Question about php.ini file On 11/4/07, Jon Westcot <[EMAIL PROTECTED]> wrote: Hi Cristian: Thanks for the replies. In this case, what I've been noticing is that most of the values have gone DOWN (i.e., gotten smaller) after I placed my own php.ini file in place. For example, upload_max_filesize was 8M before I placed my file, but, afterwards, it was 2M! I'm trying to increase this size, not decrease it. Could it be the line-ending characters that are causing the problem? Could PHP be seeing the new file but not realizing how to process it, which would cause the values to flop over to their defaults? do you have ssh access to the server? i recommend you start out by creating the simplest file you can by editing it directly on the linux server, via vim or w/e. then using a phpinfo() script, you should see your overridden value in the left column (those are for the locally overridden values). some other things to note are; 1.. apache must be properly configured to allow the .htaccess overrides. 2.. you cant override all the values in php.ini; see the documentation regarding which ones can be overriden and where here is an article regarding the process, http://gentoo-wiki.com/HOWTO_php.ini_overrides_w/_.htaccess -nathan
[PHP] Looking for ways to prevent timeout
Hi all: I'm hoping to find a solution to the problem I'm having with my script timing out while inserting records into a table. Overall, the process is pretty fast, which is impressive, but when it gets to the 22,000 to 23,000 record mark, it seems to time out. I've had it get up over 26,000 so far, but nothing better than that. And I only need to process around 30,000 right now. I've tried setting max_execution_time to 1800; no improvement. The value for max_input_time is -1, which, if I understood it correcctly, is the same as saying no limit. And I've tried calling set_time_limit() with both 0 and with 360, none of which seemed to help. Is there ANY WAY to increase the amount of time I can use when running a script that will work? I've tried everything I can find in the PHP manual. Any help you can provide will be greatly appreciated! Jon
Re: [PHP] Question about php.ini file
Hi Nathan: Another quick question regarding the use of the .htaccess file: If I'm trying to set something like post_max_size, which takes a shorthand value such as 16M in place of the full value, will using the php_value type of statement also accept the shorthand value, or do I need to place in the full value? Thanks again; this solution may be exactly what we need! Jon - Original Message - From: Nathan Nobbe To: Jon Westcot Cc: PHP General Sent: Sunday, November 04, 2007 5:05 PM Subject: Re: [PHP] Question about php.ini file On 11/4/07, Jon Westcot <[EMAIL PROTECTED]> wrote: Hi Cristian: Thanks for the replies. In this case, what I've been noticing is that most of the values have gone DOWN (i.e., gotten smaller) after I placed my own php.ini file in place. For example, upload_max_filesize was 8M before I placed my file, but, afterwards, it was 2M! I'm trying to increase this size, not decrease it. Could it be the line-ending characters that are causing the problem? Could PHP be seeing the new file but not realizing how to process it, which would cause the values to flop over to their defaults? do you have ssh access to the server? i recommend you start out by creating the simplest file you can by editing it directly on the linux server, via vim or w/e. then using a phpinfo() script, you should see your overridden value in the left column (those are for the locally overridden values). some other things to note are; 1.. apache must be properly configured to allow the .htaccess overrides. 2.. you cant override all the values in php.ini; see the documentation regarding which ones can be overriden and where here is an article regarding the process, http://gentoo-wiki.com/HOWTO_php.ini_overrides_w/_.htaccess -nathan
Re: [PHP] Looking for ways to prevent timeout
Hi Nathan: No, I'm not familiar with Ajax. Where can I read up on it? More important, how can I find out if Ajax is implemented on the server? Or is it something I can add myself? Thanks again, Jon - Original Message - From: "Nathan Nobbe" <[EMAIL PROTECTED]> To: "Jon Westcot" <[EMAIL PROTECTED]> Cc: "PHP General" Sent: Sunday, November 04, 2007 7:53 PM Subject: Re: [PHP] Looking for ways to prevent timeout > On 11/4/07, Jon Westcot <[EMAIL PROTECTED]> wrote: > > > > Hi all: > > > > I'm hoping to find a solution to the problem I'm having with my script > > timing out while inserting records into a table. > > > > Overall, the process is pretty fast, which is impressive, but when it > > gets to the 22,000 to 23,000 record mark, it seems to time out. I've had it > > get up over 26,000 so far, but nothing better than that. And I only need to > > process around 30,000 right now. > > > > I've tried setting max_execution_time to 1800; no improvement. The > > value for max_input_time is -1, which, if I understood it correcctly, is the > > same as saying no limit. And I've tried calling set_time_limit() with both > > 0 and with 360, none of which seemed to help. > > > > Is there ANY WAY to increase the amount of time I can use when running > > a script that will work? I've tried everything I can find in the PHP > > manual. > > > > Any help you can provide will be greatly appreciated! > > > are you familiar with ajax ? > i would build some client side tool that would split the job into a series > of requests. > say, you input 20,000; then the script fires off 5 requests, to do 4000 > inserts a piece. > you could easily implement a status bar and it would be guaranteed to work. > also, it would scale just about as high as you can imagine. > > -nathan > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Looking for ways to prevent timeout
Hi Jochem: Thanks for the suggestion. Not to sound more dense than I already seem, but how do I do this? How do I tell the browser that something is still running? I'm issuing a flush() after every 1000 records along with echoing a textual status update. Should I do it more frequently, say every 100 records? I'm really struggling with this concept here, and I appreciate the help that everyone is giving me! Jon - Original Message - From: "Jochem Maas" <[EMAIL PROTECTED]> To: "Jon Westcot" <[EMAIL PROTECTED]> Cc: "PHP General" Sent: Sunday, November 04, 2007 7:28 PM Subject: Re: [PHP] Looking for ways to prevent timeout > Jon Westcot wrote: > > Hi all: > > > > I'm hoping to find a solution to the problem I'm having with my script timing out while inserting records into a table. > > > > Overall, the process is pretty fast, which is impressive, but when it gets to the 22,000 to 23,000 record mark, it seems to time out. I've had it get up over 26,000 so far, but nothing better than that. And I only need to process around 30,000 right now. > > > > I've tried setting max_execution_time to 1800; no improvement. The value for max_input_time is -1, which, if I understood it correcctly, is the same as saying no limit. And I've tried calling set_time_limit() with both 0 and with 360, none of which seemed to help. > > > > Is there ANY WAY to increase the amount of time I can use when running a script that will work? I've tried everything I can find in the PHP manual. > > > > Any help you can provide will be greatly appreciated! > > http://php.net/ignore_user_abort will help, but nothing will stop you hitting a max execution time. > but my guess is your not hitting the max but rather the browser is killing the connection because it's > had no response fom your script and as a result apache is killing your script as it thinks it's no longer > needed (i.e. the browser no longer wants the response). > > > > > Jon > > > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] More info on timeout problem
Hi all: First, thanks for the multiple suggestions. I'm pretty new at PHP programming, so all suggestions are great learning opportunities. Now, some observations: I've tried placing the "ignore_user_abort(TRUE);" in the code. It seems to have made little, if any, impact -- the page still appears to time out. I've also tried placing "set_time_limit(0);" both before and after the "ignore_user_abort(TRUE);" call. Still no improvement. I'm now wondering if some error is occurring that, for some reason, is silently ending the routine. I'm building what may be a very long SQL INSERT statement for each line in the CSV file that I'm reading; could I be hitting some upper limit for the length of the SQL code? I'd think that an error would be presented in this case, but maybe I have to do something explicitly to force all errors to display? Even warnings? Another thing I've noticed is that the "timeout" (I'm not even certain the problem IS a timeout any longer, hence the quotation marks) doesn't happen at the same record every time. That's why I thought it was a timeout problem at first, and assumed that the varying load on the server would account for the different record numbers processed. If I were hitting some problem with the SQL statement, I'd expect it to stop at the same record every time. Or is that misguided thinking, too? More info: I've activated a session (have to be logged into the application to update the data), so is it possible that something with the session module could be causing the problem? I have adjusted the session.gc_maxlifetime value (per an example I saw in the PHP manual comments elsewhere), but is there some other value I should adjust, too? Thanks for all of your continued help and assistance! And please don't argue over "best" ways to solve my problem -- ALL suggestions are welcome! (Even if I don't know thing one about CLI or even how to access it. ) Jon
Re: [PHP] More info on timeout problem
Hi Instruct ICC: > > I'm now wondering if some error is occurring that, for some reason, is > > silently ending the routine. I'm building what may be a very long SQL > > INSERT statement for each line in the CSV file that I'm reading; could > > I be hitting some upper limit for the length of the SQL code? I'd think > > that an error would be presented in this case, but maybe I have to do > > something explicitly to force all errors to display? Even warnings? > > > > Another thing I've noticed is that the "timeout" (I'm not even certain > > the problem IS a timeout any longer, hence the quotation marks) doesn't > > happen at the same record every time. That's why I thought it was a > > timeout problem at first, and assumed that the varying load on the server > > would account for the different record numbers processed. If I were > > hitting some problem with the SQL statement, I'd expect it to stop at > > the same record every time. Or is that misguided thinking, too? > > 1) When you say, "doesn't happen at the same record every time" are you > using the same dataset and speaking about the same line number? Or are > you using different datasets and noticing that the line number varies? If it's > the same dataset, it sounds like "fun" -- as in "a pain in the assets". Yup, same dataset. It took me forever to upload it, so I'm trying to keep it there until I know it's been successfully loaded. It's got about 30,000 records in it, and each one has 240 fields. > 2) I'm writing something similar; letting a user upload a CSV file via a > webpage, then creating an SQL query with many records. So now I'll > be watching your thread. For debugging purposes, create your SQL > statement and print it out on the webpage (or save it somewhere -- > maybe a file). Don't have your webpage script execute the query. > Then see if you get the complete query you expect. Then copy that > query into a database tool like phpmyadmin and see if you get errors > when executing the query. Sounds much like what I'm trying to do. I have had to give up, for the time being, on using PHP to upload the datafile; it's about 56 MB in size and nothing I do seems to let me upload anything larger than a 2MB file. :( How do I save the individual query statements to a file? That may give me a good option for checking a "log" of activity when the process fails again. Thanks for your suggestions! Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] More info on timeout problem
Hi Wolf: Thanks for the suggestion. I've tried setting these in a php.ini file, but that file seems to be constantly ignored, other than the fact that its presence seems to cause every value to take on its default settings. ::sigh:: I am going to try and put the values into a .htaccess file instead; at least I seem to have some control over that file. I'm really starting to hate shared servers. Jon - Original Message - From: "Wolf" <[EMAIL PROTECTED]> To: "Jon Westcot" <[EMAIL PROTECTED]> Cc: "PHP General" Sent: Monday, November 05, 2007 2:20 PM Subject: Re: [PHP] More info on timeout problem > One thing to note, if you have not upped the max file size to be over what you are trying to load, the server will hang. > > ;;; > ; Resource Limits ; > ;;; > > max_execution_time = 7200 ; Maximum execution time of each script, in seconds > max_input_time = 7200 ; Maximum amount of time each script may spend parsing request data > memory_limit = 2G ; Maximum amount of memory a script may consume > > > ; Maximum size of POST data that PHP will accept. > post_max_size = 8M // CHANGE THIS!! > > ; Maximum allowed size for uploaded files. > upload_max_filesize = 2M // CHANGE THIS!! > > Also look in any php.ini files in apache's conf.d directory for files that set it back to these default limits > > You'll notice, I have increased my max execution times, input times, and memory limit but not my upload sizes, but that is only due to the server I snagged it from not doing uploads. I have another server which has a 879M upload limit and has no problems with large files getting to it. > > Wolf > > Jon Westcot <[EMAIL PROTECTED]> wrote: > > Hi Instruct ICC: > > > > > > I'm now wondering if some error is occurring that, for some reason, > > is > > > > silently ending the routine. I'm building what may be a very long SQL > > > > INSERT statement for each line in the CSV file that I'm reading; could > > > > I be hitting some upper limit for the length of the SQL code? I'd think > > > > that an error would be presented in this case, but maybe I have to do > > > > something explicitly to force all errors to display? Even warnings? > > > > > > > > Another thing I've noticed is that the "timeout" (I'm not even > > certain > > > > the problem IS a timeout any longer, hence the quotation marks) doesn't > > > > happen at the same record every time. That's why I thought it was a > > > > timeout problem at first, and assumed that the varying load on the > > server > > > > would account for the different record numbers processed. If I were > > > > hitting some problem with the SQL statement, I'd expect it to stop at > > > > the same record every time. Or is that misguided thinking, too? > > > > > > 1) When you say, "doesn't happen at the same record every time" are you > > > using the same dataset and speaking about the same line number? Or are > > > you using different datasets and noticing that the line number varies? If > > it's > > > the same dataset, it sounds like "fun" -- as in "a pain in the assets". > > > > Yup, same dataset. It took me forever to upload it, so I'm trying to > > keep it there until I know it's been successfully loaded. It's got about > > 30,000 records in it, and each one has 240 fields. > > > > > 2) I'm writing something similar; letting a user upload a CSV file via a > > > webpage, then creating an SQL query with many records. So now I'll > > > be watching your thread. For debugging purposes, create your SQL > > > statement and print it out on the webpage (or save it somewhere -- > > > maybe a file). Don't have your webpage script execute the query. > > > Then see if you get the complete query you expect. Then copy that > > > query into a database tool like phpmyadmin and see if you get errors > > > when executing the query. > > > > Sounds much like what I'm trying to do. I have had to give up, for the > > time being, on using PHP to upload the datafile; it's about 56 MB in size > > and nothing I do seems to let me upload anything larger than a 2MB file. :( > > > > How do I save the individual query statements to a file? That may give > > me a good option for checking a "log" of activity when the process fails > > again. > > > > Thanks for your suggestions! > > > > Jon > > > > -- > > PHP General Mailing List (http://www.php.net/) > > To unsubscribe, visit: http://www.php.net/unsub.php > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] More info on timeout problem
Hi Instruct ICC: > > How do I save the individual query statements to a file? That may give > > me a good option for checking a "log" of activity when the process fails > > again. > > I'm assuming you are building an $sql variable, so you would write that to > a file instead of executing it in a query. Look at an example here > http://php.he.net/manual/en/function.fwrite.php to write data to a file. Thanks for the link; I should have guessed that fwrite() was the command, but I'm starting to get so cross-eyed and synapse-tied here that my thinking is really muzzy. But get this -- I tried your suggestion and am writing the SQL query text to a file and am not doing anything with updating the data in MySQL at all. And it still "times" out, in constantly random areas. If it happened in the exact same place every time, I could live with that, but it's not. It's entirely random (or at least I haven't seen the pattern yet), and that just makes me ill. Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] More info on timeout problem
Hi Kristen: > I'm jumping in here late, so I haven't seen previous posts. Another > possible place I have seen limiting post/upload sizes: > > There is an Apache directive called LimitRequestSize or somesuch which will > take precedence (silently) over any PHP max post size you set. I found this > set once before in an /conf.d/php.conf file that I eventually > found. I just conducted another test of my process, this time not even creating the SQL INSERT string in the program, but just trying to read every record from the input file and just writing a count of records to the file. It worked, much to my surprise and pleasure. But then, when I put back the code to build the string, it timed out again. I'm starting to believe that I'm either using up some resource that isn't being released, or that the directive you mentioned, LimitRequestSize, is being trounced upon. Any ideas on how to find out the value currently set for this directive? Or how I can override it? Can I override it within my own .htaccess file? Thanks for the suggestion! Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] More info on timeout problem, with code
Hi all:
As requested, here's the code:
Session.gc_maxlifetime: " . ini_get("session.gc_maxlifetime") .
"\n"; // shows 1800
echo "Max execution time: " . ini_get("max_execution_time") .
"\n"; // shows 1800
echo "Max input time: " . ini_get("max_input_time") . "\n";
// shows -1
ignore_user_abort(TRUE);
set_time_limit(0);
$query = mysql_query("TRUNCATE evall;");
echo "Results of Truncate: $query\n";
$myfile_replace = "uploads/evall.csv";
$handle = @fopen($myfile_replace, 'rb');
$save_handle = @fopen("tmp/sql_calls.txt", "wb");
$process_count = 0;
if(!$handle) {
echo "The file ($myfile_replace) could not be opened.\n";
flush();
} else {
echo "The file ($myfile_replace) opened correctly.\n";
flush();
$headings = fgetcsv($handle, 1, ","); // Just ignore the first
row returned.
$row = 0;
while (($data = fgetcsv($handle, 1, ",")) !== FALSE) {
$row++;
$num = count($data);
$insert_query = "INSERT INTO evall VALUES(";
for ($c=0; $c < $num; $c++) {
if($c > 0) {
$insert_query .= ",";
}
$insert_query .= '"' . $data[$c] . '"';
}
$insert_query .= ");";
if(fwrite($save_handle, $row . "; " . strlen($insert_query) . ":
" . "\n") === FALSE) {
echo "The query could not be written.\n";
} else {
$process_count++;
}
if($row % 1000 == 0) {
echo "$row records processed so far\n";
flush();
}
}
echo "File import completed. $row records read; $process_count
records added.\n";
flush();
fclose($save_handle);
fclose($handle);
}
ini_set("session.gc_maxlifetime",$old_session_gc_maxlifetime);
ini_set("max_execution_time",$old_max_execution_time);
ini_set("max_input_time",$old_max_input_time);
}
?>
Version 1.9 -- The file uploading process presupposes that the user
has uploaded the EVAll.CSV
file to the "uploads" folder. If this has been done and you're
ready to process
it, click the button.
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] More info on timeout problem, with code
Hi David, et al.: Thanks for the comment. I removed the trailing semi-colon in the two areas where it was being sent to mysql_query() and tried the code again. I'm still getting the same basic problem -- it silently aborts somewhere around 22,000 to 26,000 records being processed out of just under 30,000. When I don't build the $insert_query string, I am able to read through the CSV file completely. As odd as this sounds, should I put in some type of delay? Could the system be thinking it's getting flooded by all of the inserts? Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] More info on timeout problem, with code
Hi Chris:
> What indexes are on this table?
On the import table, there is only one index. And I probably don't even
need an index on it since it will be processed sequentially into other
tables after it's been imported.
> When you do an insert, each one has to update the index
> as well as the data, so maybe that's where all the time is
> being spent in the database (doubt it's the problem but
> try dropping all of the indexes on the table).
I will try this.
> Are you commenting out this whole section?
>
> $insert_query = "INSERT INTO evall VALUES(";
> for ($c=0; $c < $num; $c++) {
> if($c > 0) {
> $insert_query .= ",";
> }
> $insert_query .= '"' . $data[$c] . '"';
> }
> $insert_query .= ");";
Only for the point of testing; normally, that code would need to be
included to generate the INSERT query.
> Try
>
> $insert_query = "INSERT INTO evall values ('" . implode('\'', $data) .
"')";
>
> so you're not doing a for loop when you don't need to.
Thanks for the suggestion. But, won't the "glue" part actually need to
be something like:
'\',\''
That is, I need to close the single quote, place a comma after the
field, and then add in another opening quote. One other thing: I suspect I
need to use addslashes() to the elements in $data -- is there a way to do
this with one statement as you've done above?
> Also as someone else suggested if this is a csv file you
> can use LOAD DATA INFILE to directly import it instead
> of having to create a bunch of insert queries.
>
> See http://dev.mysql.com/doc/refman/5.1/en/load-data.html
>
> Only do this if you are 100% sure of the data (ie it it sanitized in
> some other step).
Trying to use LOAD DATA INFILE was my initial plan, but that simply
didn't work. Turns out that it's because of being on a shared server. I've
talked with The Powers That Be about this and they're going to move to a
dedicated virtual server; that should give me much more flexibility and
control over what I need to update in the various settings and files and
things to get past all of the heartburn I've had over the last weeks.
> > As odd as this sounds, should I put in some type of
> > delay? Could the system be thinking it's getting flooded
> > by all of the inserts?
>
> Doubt it.
Surprisingly enough, when I've used something like usleep(15000), I seem
to be able to process more records than when I don't have it at all.
::shrug:: Just clutching at straws.
Thanks again for your comments. I appreciate them all!
Jon
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] What to do when flush() doesn't?
Hi all: I am trying to get information from a rather long-running PHP script to send out messages to the client as things are being processed. In reading the manual, it seemed that using flush() was the ideal command for this. Well, it doesn't seem to be working for me. I've even tried sending over a large number of blanks (like 4096 of 'em) before doing anything, but still no go. The flush() activity is supposed to occur on a page that has POSTed back to itself, so perhaps that's causing the heartburn? Any suggestions on how I can accomplish this type of user mollification? Thanks, Jon
[PHP] Problem retrieving $_POST values
Hi all:
I'm having problems retrieving an array value that should be in a $_POST
setting. In fact, I'm having trouble with the code hanging with no warnings or
errors whatsoever, so I've got no clue why it's not working.
Working code:
if(isset($_POST["mls"])) {
if(strlen(trim($_POST["mls"])) > 0) {
$query = "mls=" . $_POST["mls"];
}
}
Non-working code:
if(isset($_POST['city_arr'])) {
// This space intentionally left blank.
}
The only thing I know that is different is that "mls" is a single value and
that city_arr is an array. I have tried expressing it as "city_arr[]" with no
success.
Does anyone have an example of how to reference the $_POST value when said
value is an array? I can't seem to find a sample of this anywhere in the
manual.
While I'm on the subject, how can I get a php page to tell me where a
syntax error exists if one does instead of just giving me a blank page? Is
there some (free ) way to test a php script outside of a web page?
Thanks!
Jon
Re: [PHP] Problem retrieving $_POST values
Hi Chris: > Exactly as you have there. > > print_r($_POST); > > will show you everything. Here's a section of what comes back when I do this: [mls] => 1234567 [property_address] => Main St [city_arr] => Array ( [0] => CHNDHT [1] => CHNDLR [2] => GILBER [3] => HIGLEY [4] => MESA [5] => TEMPE ) > How are you creating the field in your form? The three fields mentioned above on the form are specified thus: MLS#: Address: City: CHNDHT CHNDLR GILBER HIGLEY MESA TEMPE > > Is there some (free ) way to test a php script outside of a web page? > > cmd line you can do: > > php -l /path/to/file.php Is this something I can access through SSH? And will it show me errors as the script executes? Thanks for the suggestions! Jon
Re: [PHP] Problem retrieving $_POST values
Hi Chris, et al.: > > Here's a section of what comes back when I do this: > > > > [mls] => 1234567 > > [property_address] => Main St > > [city_arr] => Array > > ( > > [0] => CHNDHT > > [1] => CHNDLR > > [2] => GILBER > > [3] => HIGLEY > > [4] => MESA > > [5] => TEMPE > > ) > > So it is there :) > > print_r($_POST['city_arr']); After much wringing of hands and gnashing of teeth, I found the problem. Typo. Of course. I'd only read over the code about a hundred times before I asked for help, if that's any consolation. > > > > Is there some (free ) way to test a php script outside of a web page? > > > > > > cmd line you can do: > > > > > > php -l /path/to/file.php > > > > Is this something I can access through SSH? And will it show me errors as the script executes? > > Oops, I thought you meant for parse errors. That won't tell you about > anything else unfortunately. Ah, don't "oops" anything! That tip was worth its weight in gold to me! In fact, it told me where one of the typos was that was causing a syntax error during the initial parsing. The other typo was more insidious (trying to use count() as $count()), but just as deadly to the output. Thanks for the great tips! Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Quick questions -- how do I place the cursor (i.e., set focus) to a field on page load?
Hi all:
This may or may not be a PHP-related question, but, since I'm coding with
PHP, I thought I'd ask:
When a form loads, such as a login form, I'd like to have the cursor placed
automatically in the Username field. Right now, it requires me to click it
with the mouse or to tab into it. I know I've seen this type of cursor control
accomplished on countless web pages, but I can't find any hints on how to
accomplish this on my own. Can one of you wonderful code wizards point me in
the right direction?
Also, while I'm asking questions, let me ask another one: I've seen several
pages (frequently, pages that come up after a login page has been filled out)
that provide some user mollification (you know, the typical "Please be patient,
don't get your nylons in a knot, we're typing as fast as we can" message),
often with some type of moving text or graphics (probably an animated GIF) that
then goes away after a certain time and a third page is displayed. I know that
the headers("Location") code can do this, but I was under the (probably
mistaken) impression that this will NOT work if information has already been
displayed on the page. If that's the case, how are these other pages doing
this? Knowing how to move from page to page this way would really be helpful.
Thanks so much.
Jon
Re: [PHP] Quick questions -- how do I place the cursor (i.e., set focus) to a field on page load?
Hi Mike:
> you want javascript.
>
> i'd recommend using jquery (jquery.com)
I'll look into this. Thanks.
> put an id="username" on the username box.
>
> then do
>
>
> $(document).ready(function() {
>$("#username").focus();
> });
>
>
> voila
Hmmm. It didn't work for me. Do I need to change any of the code you
provided or should it work as you wrote it?
Thanks, though, for the code snippet. I'll look up its pieces to see if
I can figure out what isn't working for me.
Jon
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Quick questions -- how do I place the cursor (i.e., set focus) to a field on page load?
Hi Mike, Chris, et al.: Thanks for the info. I'm not certain if I'll be needing jquery, or what benefits it might afford me with my meager little project, but I will check it out. In the meantime, Chris's snippet worked like a charm! Thanks! Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Quick questions -- how do I place the cursor (i.e., set focus) to a field on page load?
Hi Tedd:
> > Also, while I'm asking questions, let me ask another one: I've
> >seen several pages (frequently, pages that come up after a login
> >page has been filled out) that provide some user mollification (you
> >know, the typical "Please be patient, don't get your nylons in a
> >knot, we're typing as fast as we can" message), often with some type
> >of moving text or graphics (probably an animated GIF) that then goes
> >away after a certain time and a third page is displayed.
>
> Here's a collection of animated wait gifs:
>
> http://webbytedd.com/bb/wait/
>
> Steal as you need -- I did.
Thanks. I'll check that out. I'm still wondering, though, how the
redirection works if it is supposed to occur only before any text appears in
the .
> >I know that the headers("Location") code can do this, but I was
> >under the (probably mistaken) impression that this will NOT work if
> >information has already been displayed on the page. If that's the
> >case, how are these other pages doing this? Knowing how to move
> >from page to page this way would really be helpful.
>
> Check out php sessions for passing data between pages.
>
> http://www.php.net/session
> http://www.tizag.com/phpT/phpsessions.php
I guess these will help explain things. Thanks.
Greatly appreciated!
Jon
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Newbie question - current date - time
Hi:
> > Hi Folks,
> >
> > Newbie question :
> >
> > - how do I get and display the current date?
> > - how do I get and display the current time?
> >
> > I see the getdate function - but I'm not sure if this is the right
> > function
> > or how to display it
> >
> > http://www.php.net/manual/en/function.getdate.php
>
> Give this a shot:
>
> $Date = date("y/m/d h:m:i");
> echo $Date;
>
> You can format it how ever you want. Lots of different options. Check
> out: HTTP://www.php.net/date for lots more info.
Um, not to pick nits, and being a newbie myself I'm really risking a lot
here by potentially exposing my ignorance, but was the purpose of the code
above to get the user to read the manual section? It sure worked for me
when I saw the output of the date command and couldn't believe that the
process I was running completed in just six seconds. It all boiled down to
changing (dare I say correcting?) "h:m:i" to "h:i:s" -- the "m" in question
related to the month in the date, not minutes in the time.
And thanks to everyone for being so willing to help and generous with
their time. I really appreciate it!
Jon
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] I need help handling form posting
Hi all: In my quest to make things seem easier for the user to understand, I'm trying to retrieve the time that I start posting a very large file via a form post. The form action sends it back to itself, which is fine. It took me some time to figure out that NOTHING happens on the client until the form has been completely transmitted to the server. In my particular case, this can be a long time, anywhere from 15 to 30 minutes. I'd like to be able to notify the user of when the file upload actually began. Is there a way that I can intercept the click of the "Upload" button, have it update a field (probably a hidden one) with a date/time stamp, and then have that value included in the $_POSTed values? I'm thinking (fearing, rather) that this is probably all off-topic and probably is best addressed with some type of JavaScript solution, but it's late, I'm not thinking clearly, and I've been doing data conversions all day, which is a sure-fire way to get me emulating some Romero zombies. Any help you can spare will be greatly appreciated. Jon
[PHP] Looking for a navigation recommendation
Hi all: I'm working on a project wherein I need to be able to navigate to previous and next sections of data. I'm wondering what the best way to code for this would be. When I enter the page, I'd like it to use something in the MySQL SELECT such as "LIMIT 0,25", but the "0" portion needs to change with the appropriate selections of "Next" or "Prev". I thought that setting Next or Prev as an anchor back to the same page would let me pass along data, but it doesn't seem to be happening, at least, not in the $_POST variables. Any suggestions for me? Or maybe a recommendation for a similar page I could view and learn from? Thanks so much for any help you can send my way. Jon
Re: [PHP] Looking for a navigation recommendation
Hi Tedd: > >Hi all: > > > > I'm working on a project wherein I need to be able to navigate > >to previous and next sections of data. I'm wondering what the best > >way to code for this would be. > > > > When I enter the page, I'd like it to use something in the MySQL > >SELECT such as "LIMIT 0,25", but the "0" portion needs to change > >with the appropriate selections of "Next" or "Prev". I thought that > >setting Next or Prev as an anchor back to the same page would let me > >pass along data, but it doesn't seem to be happening, at least, not > >in the $_POST variables. > > > > Any suggestions for me? Or maybe a recommendation for a similar > >page I could view and learn from? > > > > Thanks so much for any help you can send my way. > > > > Jon > > Jon: > > Do you mean something like this: > > http://webbytedd.com/bbb/proof > > Please note the page thing at the bottom. If so, I'll work up a demo for you. > > Cheers, > > tedd YES! Exactly! Something like that would be very nice to have. MUCH appreciated! Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Looking for a navigation recommendation (SOLVED)
Hi Tedd: > > YES! Exactly! Something like that would be very nice to have. > > > > MUCH appreciated! > > > > Jon > > Jon: > > Here it is: > > http://webbytedd.com/bbb/paging > > I'm assuming that you know how to establish communication with your > database; how to set up your database; and how to use css and the > images as shown there. > > If you get in trouble, I'm available for hire. :-) > > tedd Thanks for the code sample. It pointed me in exactly the right direction to solve what I was trying to accomplish. Appreciatively, Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Performance question for table updating
Hi all:
For those who've been following the saga, I'm working on an application
that needs to load a data file consisting of approximately 29,000 to 35,000
records in it (and not short ones, either) into several tables. I'm using
MySQL as the database.
I've noticed a really horrible performance difference between INSERTing
rows into the table and UPDATEing rows with new data when they already exist in
the table. For example, when I first start with an empty table, the
application inserts around 29,600 records in something less than 6 minutes.
But, when I use the second file, updating that same table takes over 90 minutes.
Here's my question: I had assumed -- probably wrongly -- that it would be
far more expedient to only update rows where data had actually changed;
moreover, that I should only update the changed fields in the particular rows.
This involves a large number of if statements, i.e.,
if($old_row["field_a"] !== $new_row["field_66"] {
$update_query .= "field_a = '" .
mysql_real_escape_string($new_row["field_66"]) . "',";
}
Eventually, I wind up with a query similar to:
UPDATE table_01 SET field_a = 'New value here', updated=CURDATE() WHERE
primary_key=12345
I thought that, to keep the table updating to a minimum, this approach made
the most sense. However, seeing the two hugely different performance times has
made me question whether or not it would be faster to simply update every field
in the table and eliminate all of these test conditions.
And, before someone comments that indexes on the table can cause
performance hits, I DROP nearly all of the indexes at the start of the
processing, only keeping those indexes necessary to do the original INSERT or
the subsequent UPDATE, and then add all of the extra "steroid" indexes (you
know -- the performance-enhancing ones ) after all of the INSERTs and
UPDATEs have been finished.
So, long story short (oops -- too late!), what's the concensus among the
learned assembly here? Is it faster to just UPDATE the record if it already
exists regardless of the fact that maybe only one or two out of 75 or more
fields changed versus testing each one of those 75 fields to try and figure out
which ones actually changed and then only update those?
I look forward to reading all of your thoughts.
Sincerely,
Jon
Re: [PHP] Performance question for table updating (SOLVED)
Hi Rob, et al.: - Original Message - From: "Andrés Robinet" <[EMAIL PROTECTED]> > > -Original Message- > > From: Jon Westcot [mailto:[EMAIL PROTECTED] > > > > :: gigantic snip here:: > > > > So, long story short (oops -- too late!), what's the concensus > > among the learned assembly here? Is it faster to just UPDATE the > > record if it already exists regardless of the fact that maybe only one > > or two out of 75 or more fields changed versus testing each one of > > those 75 fields to try and figure out which ones actually changed and > > then only update those? > > > > I look forward to reading all of your thoughts. > > > > Sincerely, > > > > Jon > > I don't know about consensus over here because I'm kind of newgie (stands > for new geek, as opposed to newbie which stands for new ball breaker :D :D > ). I don't know of your previous messages but I can tell you one story... > Some time ago I got involved in a project that required geo-distance > calculation (you know distance between two points with latitude and > longitude). Basically I had to take a set of points and calculate the > distance of each of those points to a given (reference) one. The math was > something like the "square root of the sum of a constant times the square > sin of..." well, I can't remember it, but the point is, it was a complicated > formula, which I thought it would allow for some optimizations in PHP. > Accustomed to regular (compiled) programming languages I developed a set of > routines to optimize the task and went ahead and queried the database for > the (say, 1000 records) dataset of points. Then applied the math to the > points and the reference point and got the result... in about 5 minutes to > my (disgusting) surprise. > Then I grabbed the MySQL manual, built a "non-optimized" version of the > formula to put directly in the SQL query and get the "shortest distance" > (which was my goal in the end) calculated by MySQL right away. I thought > "ok, I'll prepare a cup of coffee to wait for MySQL to finish the > calculation". To my surprise the query returned the expected result in less > than 2 seconds. > My logic was (wrongly) the following: PHP is a programming language, SQL is > a data access language; I'll get the data using MySQL and do the math using > PHP. But I forgot PHP is an interpreted language, that a number is more than > a number to PHP, but a ZVAL_ object behind the scenes. I forgot > about the memory and the time required to build those objects when one > retrieves data out of a database server. I forgot about parsing time, and > "support logic and safety checks" in the language that overkill any attempt > to build TDCPL (Too Damn Complex Programming Logic) in PHP. > So, now, when I have to do some logic stuff to the retrieved data, I first > check "how much" I can push into the query itself, to get little or nothing > of programming logic in PHP after retrieving (before storing) the data. > All that said, I'd give a shot to the MySQL REPLACE function (I wouldn't > even branch the code to use INSERT or UPDATE depending on the record already > existing or not, If you have a primary key, all you need is REPLACE). But, > PLEASE LOOK AT THE GOTCHAS (like set col_name=col_name+1). Furthermore, If > those data files were to be uploaded by me (I mean, me, the coder, not the > end user), I'd build (use) a program to convert them to SQL sentences in my > desktop PC where I can use faster programming languages and I can wait for > five minutes of heavy processing (instead of overkilling the server for five > minutes which will slow down every other service in there). > In the end it depends on your requirements and where you get the data from > and if and how you want to automate the task (I didn't get your previous > messages, I got subscribed recently, if you can send me a link to those > ones... great!) > > Rob Thanks for the comments and suggestions. Prior to receiving your note, I went back and did a bit of checking on my code. Turns out that the problem was "hardware" related -- the infamous "Loose Screw Behind the Keyboard." The problem actually boiled down to two quotation marks -- they were present in the search code to see if a record with the specified key existed, but were omitted in the WHERE clause of the UPDATE statement. Said update therefore refused to use the nice little index I'd provided for its use and instead scanned through the entire table to find the record in question. Moral of the story? Two, really. First, ensure you always reference values in the way most appropriate for their type. Second, don't make your idiocy public by asking stupid questions on a public forum. What's the quote (probably attributed to Churchill)? "It is better to be ignorant and silent than to voice one's opinions and remove all doubt." ;) Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Quick question on data formatting
Hi all: Since I'm relatively new to PHP, I'm not familiar with all of the shortcuts and efficient ways one could write a routine that handles converting data from one format into another, so I thought I'd ask here for recommendations on how to solve one particular issue I've got. I'm viewing this as a learning experience for me. I've got some text fields coming in that need to be added to a table as numeric fields. In particular, I've got to process dollar amounts that appear with dollar signs and commas into their strictly numeric forms. For example: $123,456.78 becomes 123456.78 $24,680 becomes 24680 1234B becomes 1234 (That last example is an actual example of something I saw in the data file; I can only assume that the user wanted 1234, as that's the closest value I can assume from what was typed in.) What is the fastest, most efficient way to perform these conversions? I'm coming at PHP from "old school" experience where I'd normally read through the text character by character and return a value built from only the characters I have declared as acceptable (i.e., 0-9 and . and possibly a -), but I've got to believe that there's some technique -- or possibly a deeply buried function -- available to me now that would be faster and easier to implement. Any help you can provide will, as always, be greatly appreciated! Thanks, Jon
Re: [PHP] Quick question on data formatting [SOLVED]
Thanks, Rob! I can see that it's going to be important for me to get familiar with the POSIX regular expressions. Jon - Original Message - > On Wed, 2007-11-28 at 22:19 -0700, Jon Westcot wrote: > > Hi all: > > > > Since I'm relatively new to PHP, I'm not familiar with all of the shortcuts and efficient ways one could write a routine that handles converting data from one format into another, so I thought I'd ask here for recommendations on how to solve one particular issue I've got. I'm viewing this as a learning experience for me. > > > > I've got some text fields coming in that need to be added to a table as numeric fields. In particular, I've got to process dollar amounts that appear with dollar signs and commas into their strictly numeric forms. For example: > > > > $123,456.78 becomes 123456.78 > > $24,680 becomes 24680 > > 1234B becomes 1234 > > $number = ereg_replace( '[^[:digit:].]', '' ); > > Cheers, > Rob. > -- > ... > SwarmBuy.com - http://www.swarmbuy.com > > Leveraging the buying power of the masses! > ... > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] One more data formatting question
Hi all: I'm trying to parse out some HTML code back into "regular" string values and I keep getting tripped up by the non-breaking space value ( ). I see that html_decode_entities() can be used to convert this back to a viewable space, but the documentation tells me that the space value it uses is not the same as a TRIMable space (i.e., ASCII 32). Is there a quick, fast, and easily implemented way to convert any non-breaking space found in a string back to the ASCII 32 space value? I suspect that one of those amazing POSIX expressions could do it, but I'm having trouble wrapping my head around them at this early hour. Any help you all can provide will be extremely appreciated! Thanks, Jon
Re: [PHP] One more data formatting question [SOLVED]
Hi:
Thanks for the answer! That worked exactly as I needed it to work!
Jon
- Original Message -
From: "T.Lensselink" <[EMAIL PROTECTED]>
To: "Jon Westcot" <[EMAIL PROTECTED]>
Cc: "PHP General"
Sent: Thursday, November 29, 2007 8:17 AM
Subject: Re: [PHP] One more data formatting question
On Thu, 29 Nov 2007 07:53:56 -0700, "Jon Westcot" <[EMAIL PROTECTED]> wrote:
> Hi all:
>
> I'm trying to parse out some HTML code back into "regular" string
> values and I keep getting tripped up by the non-breaking space value
> ( ). I see that html_decode_entities() can be used to convert this
> back to a viewable space, but the documentation tells me that the space
> value it uses is not the same as a TRIMable space (i.e., ASCII 32).
>
> Is there a quick, fast, and easily implemented way to convert any
> non-breaking space found in a string back to the ASCII 32 space value? I
> suspect that one of those amazing POSIX expressions could do it, but I'm
> having trouble wrapping my head around them at this early hour.
>
> Any help you all can provide will be extremely appreciated!
>
> Thanks,
>
> Jon
str_replace should do the trick.
str_replace(' ', chr(32), $string);
or
str_replace(' ', ' ', $string);
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Newbie question: need to transfer directory contents from my local machine to my website
Hi all: I'm really new at PHP and will probably embarrass myself many times over asking questions that have been asked gazillions of times before, so let this serve as a blanket apology. Now, to my question. Here's what I'm trying to do. I have a simple database on my website that I wish to populate with information from various directories on my local computer. The website is running Linux; my computer is running Windows XP. Once the data are stored, I want to be able to update the information as things change on my local computer (not in real time, mind you, but at my request). I can set up the database access easily enough, and I know how to both populate and query it. What I don't know is how to obtain the information from my local computer via the website. Initially, I'd like to be able to specify the folder on my local computer to access and whether or not to process any subfolders that are found. After the data have been added, I'd like the web application to be able to access the individual folders without having to specify them again (although I'd still be able to identify new folders to include in subsequent updates). I'm not really asking for anyone to write the code for me, but I am looking for suggestions for PHP functions to use to accomplish the inspection of my local computer's folders. I'd also need to know what additional information I'd need to store in the database so that subsequent updates can be automated (i.e., do I need to somehow store my IP address?). Any help you can send my way will be greatly appreciated! Thanks in advance. Sincerely, Jon
Re: [PHP] Comparing of string
Hi JanBro:
Quick guess: are the strings the same length? I've been bitten many
times by string comparisons that appear to be identical but which fail due
to trailing spaces, other "invisible" (i.e., non-printing) characters, and
the like.
Hope this helps.
Jon
- Original Message -
From: "janbro" <[EMAIL PROTECTED]>
To:
Sent: Tuesday, January 03, 2006 11:22 PM
Subject: [PHP] Comparing of string
> Hello List
>
> I've got the following little code:
>
> $formCheck= $_GET['formCheck'];
> $SollKombination = $_SESSION['zufall'];
>
> echo "$SollKombination$formCheck";
> print gettype($formCheck);
> echo "---";
> print gettype($SollKombination);
>
>
> if ($SollKombination == $formCheck){
>echo "test";
> }
>
> To give you some background: This code is supposed to check if a user has
tried to login via my form.
>
> Which gives me the following output:
>
> ZL0X~TT4PQ%0~R0OXPRUHY7E&!4~W337J71V4WDDI6$GS9480XP0TNP2I$1YX75S
> ZL0X~TT4PQ%0~R0OXPRUHY7E&!4~W337J71V4WDDI6$GS9480XP0TNP2I$1YX75S
> string---string
>
> Everything the way it's supposed to be
>
> What I don't get is, why isn't the if statement true? Shouldn't it show
test as well? Where is my mistake?
> I run PHP 5.1.1 on Windows. On my Win PHP 5.0 this code works proper, but
not here ?!?
>
> thx JanBro
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Newbie question: need to transfer directory contents fr om my local machine to my website
Jochem: My most sincere apologies for not replying to your questions. I've been suffering with a rather dibilitating head cold the last few days and somehow glossed over your questions. > 1. what is the local machine? Just my local workstation. > 2. where does the DB live? (is it a different machine?) Yes, the database -- in MySQL -- lives on a different server. > 3. do you need reporting/information about the local machine > inspection anywhere other than the local machine? No, I only need it locally. At least, for right now. > 4. will there be more than one local machine? No. The only place the data will originate FROM will be my local machine. The data are simply directory entries of PDF or CHM files. > 5. why do you want to do this? For starters, I'm trying to cull out duplicates. Some are easily found by comparing file sizes and then looking at the file names. But, some of the files are damaged, so I need to actually open the files to ensure they open without error before deciding which ones to delete. It seemed to me that it would be easier to open the files from a web browser than from another, scratch-built application. The larger question though, is, "Why use PHP for this?" Because I'm trying to learn more about how to use PHP, and because, once the data have been combed through and duplicates have been culled, I'd like to make the data available to a few friends who would like to see the list of files that I've amassed (currently around 12,600, with approximately 3000 known duplicates). Those able to view the data would not be able to update the data at all -- the form to do the update would have been password protected. What I think I'll do now is create the update routines locally, then export the data, upload the data to the server, and import it into a copy of the database. Yeah, it means more work on my part, but that's okay. The updates after I finally get the initial data clean will be somewhat infrequent. Thanks again for your input and suggestions. Sincerely, Jon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
