Re: [PHP] file locking...
On Sat, 2009-02-28 at 21:46 -0800, bruce wrote: > Hi. > > Got a bit of a question/issue that I'm trying to resolve. I'm asking this of > a few groups so bear with me. > > I'm considering a situation where I have multiple processes running, and > each process is going to access a number of files in a dir. Each process > accesses a unique group of files, and then writes the group of files to > another dir. I can easily handle this by using a form of locking, where I > have the processes lock/read a file and only access the group of files in > the dir based on the open/free status of the lockfile. > > However, the issue with the approach is that it's somewhat synchronous. I'm > looking for something that might be more asynchronous/parallel, in that I'd > like to have multiple processes each access a unique group of files from the > given dir as fast as possible. > > So.. Any thoughts/pointers/comments would be greatly appreciated. Any > pointers to academic research, etc.. would be useful. > > thanks > > > > You could do it one of several ways: 1. Have the files actually written to a subversion/git repository, and let that handle differences. 2. Store the files in a database as blobs 3. Do something clever with filename suffixes to indicate versions of the file Ash www.ashleysheridan.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: "use strict" or similar in PHP?
Stuart wrote:
2009/3/1 Shawn McKenzie
Stuart wrote:
2009/2/28 Shawn McKenzie
Robert Cummings wrote:
On Sat, 2009-02-28 at 00:11 +, Ashley Sheridan wrote:
On Fri, 2009-02-27 at 14:32 -0500, Robert Cummings wrote:
On Sat, 2009-02-28 at 00:02 +0600, 9el wrote:
---
Use FreeOpenSourceSoftwares, Stop piracy, Let the developers live.
Get
a Free CD of Ubuntu mailed to your door without any cost. Visit :
www.ubuntu.com
--
On Fri, Feb 27, 2009 at 11:46 PM, Robert Cummings <
rob...@interjinn.com>wrote:
On Fri, 2009-02-27 at 09:28 -0700, LuKreme wrote:
On Feb 27, 2009, at 6:12, Hans Schultz
wrote:
Hahahah,I was thinking the same thing
The trouble is most people mean "compile a source file to an
executable binary" when they sat compile. By this measure, PHP
does
not compile.
I add the following to the top of my PHP shell scripts:
#!/usr/bin/php -qC
Then I do the following:
chmod 775 script.php
Then I run it as follows:
./script.php
Look... and executable binary :) Don't say it's not binary. All
data
on
a hard disk is binary (although I do know what you mean ;)
Well you are running shell script style execution its not example of
Compiled code or Binary
The data in the file is ASCII or UTF text :)
Which are subsets of binary representation ;)
Compilation happens when its zendOptimized or OpCoded. Its then is
converted
into binary content file.
But one could probably quite easily set up a system whereby
eAccelerator
or APC or Zend Optimizer cache bytecodes are torn from a file run
similarly. As I said in an earlier post... the line between the
definition of interpreted language and compiled language is quite
blurry
these days.
Cheers,
Rob.
--
http://www.interjinn.com
Application and Templating Framework for PHP
ASCII is only a subset of binary in a pedantic, literal sense. When
people say binary file, they mean one that contains characters which
are
outside the normal display spectrum, such as chr(0), etc.
And the argument that PHP is not compiled requires a certain level of
pedantry and it is still an incorrect argument since it most certainly
is compiled to an intermediary virtual machine code.
Cheers,
Rob.
Still, in PHP $compile_time == $run_time.
I haven't been following this thread, but this caught my eye as being
completely wrong. There are distinct compile and execution phases when
PHP
runs a script, and different rules apply to each. If you don't believe me
try defaulting the value of a class variable to the result of a function.
-Stuart
Rob, I wasn't talking about time. I was talking about they're pretty
much the same as in when they execute.
Stuart,
Great, then show the OP how he can enforce strict error checking at
compile time and halt compilation and runtime altogether.
Like I said I haven't been following this thread so have no idea what the
original question was. I was just pointing out that your assertion
that $compile_time == $run_time is incorrect.
As far as error checking goes the PHP engine performs syntactical checks
during compile time but due to the highly flexible nature of the language
it's not possible to do most error checking until runtime.
-Stuart
And I quote:
Hello,
I am beginner with PHP and prior to PHP I have worked with java for some time
and with perl for very short period. I can't help to notice some things that
are little annoyance for me with PHP, but I am sure someone more experienced
can help me :-)
Is there in PHP something like "use strict" from perl? I find it pretty
annoying to need to run script over and over again just to find out that I
made typo in variable name.
Is there some way for PHP to cache some data on the page? I like very much
PHP's speed but it would be even better to be able to cache some frequently
used data from database?
Also regarding databases, I liked a lot java's way of sending data to database
using parameters ("select * from user where username = ?" and then passing
parameter separately with database doing necessary escaping and everything).
Is there something like PHPDBC similar to JDBC?
TIA,
Hans
End Quote;
Jim Lucas
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: "use strict" or similar in PHP?
On Sat, 2009-02-28 at 18:57 -0600, Shawn McKenzie wrote: > Stuart wrote: > > 2009/2/28 Shawn McKenzie > > > >> Robert Cummings wrote: > >>> On Sat, 2009-02-28 at 00:11 +, Ashley Sheridan wrote: > On Fri, 2009-02-27 at 14:32 -0500, Robert Cummings wrote: > > On Sat, 2009-02-28 at 00:02 +0600, 9el wrote: > >> --- > >> Use FreeOpenSourceSoftwares, Stop piracy, Let the developers live. Get > >> a Free CD of Ubuntu mailed to your door without any cost. Visit : > >> www.ubuntu.com > >> -- > >> > >> > >> On Fri, Feb 27, 2009 at 11:46 PM, Robert Cummings < > >> rob...@interjinn.com>wrote: > >>> On Fri, 2009-02-27 at 09:28 -0700, LuKreme wrote: > On Feb 27, 2009, at 6:12, Hans Schultz > >> wrote: > > Hahahah,I was thinking the same thing > The trouble is most people mean "compile a source file to an > executable binary" when they sat compile. By this measure, PHP does > not compile. > >>> I add the following to the top of my PHP shell scripts: > >>> > >>>#!/usr/bin/php -qC > >>> > >>> Then I do the following: > >>> > >>>chmod 775 script.php > >>> > >>> Then I run it as follows: > >>> > >>>./script.php > >>> > >>> Look... and executable binary :) Don't say it's not binary. All data > >> on > >>> a hard disk is binary (although I do know what you mean ;) > >> Well you are running shell script style execution its not example of > >> Compiled code or Binary > >> > >> The data in the file is ASCII or UTF text :) > > Which are subsets of binary representation ;) > > > >> Compilation happens when its zendOptimized or OpCoded. Its then is > >> converted > >> into binary content file. > > But one could probably quite easily set up a system whereby > >> eAccelerator > > or APC or Zend Optimizer cache bytecodes are torn from a file run > > similarly. As I said in an earlier post... the line between the > > definition of interpreted language and compiled language is quite > >> blurry > > these days. > > > > Cheers, > > Rob. > > -- > > http://www.interjinn.com > > Application and Templating Framework for PHP > > > > > ASCII is only a subset of binary in a pedantic, literal sense. When > people say binary file, they mean one that contains characters which are > outside the normal display spectrum, such as chr(0), etc. > >>> And the argument that PHP is not compiled requires a certain level of > >>> pedantry and it is still an incorrect argument since it most certainly > >>> is compiled to an intermediary virtual machine code. > >>> > >>> Cheers, > >>> Rob. > >> Still, in PHP $compile_time == $run_time. > > > > > > I haven't been following this thread, but this caught my eye as being > > completely wrong. There are distinct compile and execution phases when PHP > > runs a script, and different rules apply to each. If you don't believe me > > try defaulting the value of a class variable to the result of a function. > > > > -Stuart > > > Rob, I wasn't talking about time. I was talking about they're pretty > much the same as in when they execute. They're not if you're using a compile cache. Cheers, Rob. -- http://www.interjinn.com Application and Templating Framework for PHP -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] file locking...
On Sat, 2009-02-28 at 21:46 -0800, bruce wrote: > Hi. > > Got a bit of a question/issue that I'm trying to resolve. I'm asking this of > a few groups so bear with me. > > I'm considering a situation where I have multiple processes running, and > each process is going to access a number of files in a dir. Each process > accesses a unique group of files, and then writes the group of files to > another dir. I can easily handle this by using a form of locking, where I > have the processes lock/read a file and only access the group of files in > the dir based on the open/free status of the lockfile. > > However, the issue with the approach is that it's somewhat synchronous. I'm > looking for something that might be more asynchronous/parallel, in that I'd > like to have multiple processes each access a unique group of files from the > given dir as fast as possible. > > So.. Any thoughts/pointers/comments would be greatly appreciated. Any > pointers to academic research, etc.. would be useful. Threads? Or spawn off child processes. Maybe I'm not understanding your issues well enough. Cheers, Rob. -- http://www.interjinn.com Application and Templating Framework for PHP -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] www.soongy.com
Hi guys I need to create file upload progress bar using PHP and AJAX. I found two Pecl extensions that support that may be used: "uploadprogress" and "php-apc". I asked my shared hosting provider to add that extensions, but my request was declined, because "uploadprogress" is still BETA version and "php-apc" dont works fine with Fast-CGI version of PHP. Had you similar problem or do you know how else can I track upload progress? Btw site is www.soongy.com Thanks for help
Re: [PHP] Re: "use strict" or similar in PHP?
The OP asked: Is there in PHP something like "use strict" from perl? I find it pretty annoying to need to run script over and over again just to find out that I made typo in variable name. And I've been waiting for an answer myself, but I haven't seen one. From what I remember, in perl if you use "use strict;" it requires to to define your variables (my) before using them. If you make a variable typo in your code, then you'll trigger an error when you try to run it. From what I've seen of php, even with using strict error reporting, you can do that all day long without generating an error. So the answer appears to be "No, you can't do that in PHP." Is that the answer? Cheers, tedd -- --- http://sperling.com http://ancientstones.com http://earthstones.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] www.soongy.com
At 2:20 PM +0400 3/1/09, Gevorg Harutyunyan wrote: Hi guys I need to create file upload progress bar using PHP and AJAX. I found two Pecl extensions that support that may be used: "uploadprogress" and "php-apc". I asked my shared hosting provider to add that extensions, but my request was declined, because "uploadprogress" is still BETA version and "php-apc" dont works fine with Fast-CGI version of PHP. Had you similar problem or do you know how else can I track upload progress? Gevorg: I'm sure this isn't the answer you want, but I went that route several times and never found a better solution than this: http://webbytedd.com/bb/wait/ Throw up a "wait" graphic and allow the process to take it's course without intervention. If you want to programmatically determine over time how much of the file has been uploaded and update a progress bar, then you're in for quite the learning experience. Cheers, tedd PS: After everything is said and done, you may steal any of the graphics I provide above -- I stole them fair and square myself. Number 39 is not a wait graphic, it just assumed me. -- --- http://sperling.com http://ancientstones.com http://earthstones.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] www.soongy.com
There is no need to go that far, try to google a bit about swfupload. In short, this is a flash & javascript component that give's you the ability to maintain the upload, get the current speed, get the current amount of uploaded data, etc. It is very simple and works like a charm on a dedi server. There are some issues on shared server sometimes, but even these things are not that much complicated and can be easily solved. HTH, Nitsan On Sun, Mar 1, 2009 at 4:09 PM, tedd wrote: > At 2:20 PM +0400 3/1/09, Gevorg Harutyunyan wrote: > >> Hi guys >> >> I need to create file upload progress bar using PHP and AJAX. I found two >> Pecl extensions that support that may be used: "uploadprogress" and >> "php-apc". >> I asked my shared hosting provider to add that extensions, but my request >> was declined, because "uploadprogress" is still BETA version and "php-apc" >> dont works >> fine with Fast-CGI version of PHP. >> >> Had you similar problem or do you know how else can I track upload >> progress? >> > > Gevorg: > > I'm sure this isn't the answer you want, but I went that route several > times and never found a better solution than this: > > http://webbytedd.com/bb/wait/ > > Throw up a "wait" graphic and allow the process to take it's course without > intervention. > > If you want to programmatically determine over time how much of the file > has been uploaded and update a progress bar, then you're in for quite the > learning experience. > > Cheers, > > tedd > > PS: After everything is said and done, you may steal any of the graphics I > provide above -- I stole them fair and square myself. Number 39 is not a > wait graphic, it just assumed me. > > -- > --- > http://sperling.com http://ancientstones.com http://earthstones.com > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > >
Re: [PHP] Re: "use strict" or similar in PHP?
On Sun, 2009-03-01 at 08:36 -0500, tedd wrote:
> The OP asked:
>
> >Is there in PHP something like "use strict" from perl? I find it pretty
> >annoying to need to run script over and over again just to find out that I
> >made typo in variable name.
>
> And I've been waiting for an answer myself, but I haven't seen one.
>
> From what I remember, in perl if you use "use strict;" it requires to
> to define your variables (my) before using them. If you make a
> variable typo in your code, then you'll trigger an error when you try
> to run it.
>
> From what I've seen of php, even with using strict error reporting,
> you can do that all day long without generating an error.
>
> So the answer appears to be "No, you can't do that in PHP." Is that the
> answer?
You can do anything you want... :)
$init )
{
$this->{$prop} = $init;
}
}
else
{
$this->{$props} = null;
}
}
$this->___init = false;
}
function __set( $name, $value )
{
if( !$this->___init )
{
die( 'Attempt to set non-existent property: '
.get_class( $this ).'->'.$name."\n" );
}
$this->{$name} = $value;
}
function __get( $name )
{
die( 'Attempt to get non-existent property: '
.get_class( $this ).'->'.$name."\n" );
}
}
//
// Declare strict properties...
//
$my = new StrictProps( 'foo', 'fee', 'fii' );
print_r( $my );
//
// Declare and intialize strict properties...
//
$my = new StrictProps( array( 'foo' => 'Foo1', 'fee' => 'Fee1' ) );
print_r( $my );
echo $my->foo."\n";
echo $my->fee."\n";
echo $my->blah."\n";
?>
Cheers,
Rob.
--
http://www.interjinn.com
Application and Templating Framework for PHP
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] www.soongy.com
At 4:17 PM +0200 3/1/09, Nitsan Bin-Nun wrote: There is no need to go that far, try to google a bit about swfupload. In short, this is a flash & javascript component that give's you the ability to maintain the upload, get the current speed, get the current amount of uploaded data, etc. It is very simple and works like a charm on a dedi server. There are some issues on shared server sometimes, but even these things are not that much complicated and can be easily solved. HTH, Nitsan Oh yeah, try this: http://swfupload.org/documentation/demonstration and go through the "up" link -- and then try the "See it in action!" link and also try the "Demonstration" link. You can even use the demo.swfupload.org link, which will provide you with this: http://demo.swfupload.org/v220beta5/index.htm All of which is well worth the effort if you're trying to waste your time. If their code is as good as their web site, no thanks -- I'll pass. But if I was to seriously investigate it, I would go directly to Google: http://code.google.com/p/swfupload/ However, I haven't a clue as to if it works or not. Cheers, tedd -- --- http://sperling.com http://ancientstones.com http://earthstones.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: "use strict" or similar in PHP?
At 9:21 AM -0500 3/1/09, Robert Cummings wrote: On Sun, 2009-03-01 at 08:36 -0500, tedd wrote: The OP asked: >Is there in PHP something like "use strict" from perl? I find it pretty >annoying to need to run script over and over again just to find out that I >made typo in variable name. And I've been waiting for an answer myself, but I haven't seen one. From what I remember, in perl if you use "use strict;" it requires to to define your variables (my) before using them. If you make a variable typo in your code, then you'll trigger an error when you try to run it. From what I've seen of php, even with using strict error reporting, you can do that all day long without generating an error. So the answer appears to be "No, you can't do that in PHP." Is that the answer? You can do anything you want... :) Rob: Just to show that your words of wisdom don't go unnoticed, see here: http://php1.net/oop/strict/ That's pretty slick. Now if I only understood WT* you did, I would be a better programmer for it. Thanks, tedd -- --- http://sperling.com http://ancientstones.com http://earthstones.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: "use strict" or similar in PHP?
On Sun, 2009-03-01 at 09:48 -0500, tedd wrote: > At 9:21 AM -0500 3/1/09, Robert Cummings wrote: > >On Sun, 2009-03-01 at 08:36 -0500, tedd wrote: > >> The OP asked: > >> > >> >Is there in PHP something like "use strict" from perl? I find it pretty > >> >annoying to need to run script over and over again just to find out that > >> I > >> >made typo in variable name. > >> > >> And I've been waiting for an answer myself, but I haven't seen one. > >> > >> From what I remember, in perl if you use "use strict;" it requires to > >> to define your variables (my) before using them. If you make a > >> variable typo in your code, then you'll trigger an error when you try > >> to run it. > >> > >> From what I've seen of php, even with using strict error reporting, > >> you can do that all day long without generating an error. > >> > >> So the answer appears to be "No, you can't do that in PHP." Is > >>that the answer? > > > >You can do anything you want... :) > > Rob: > > Just to show that your words of wisdom don't go unnoticed, see here: > > http://php1.net/oop/strict/ > > That's pretty slick. > > Now if I only understood WT* you did, I would be a better programmer for it. I used the magic methods __set() and __get() to facilitate error handling. These methods fire when a property is not accessible. So when the property does not exist these will fire. So first we init the properties when we create an instance (I'll discuss this shortly)... to init them we are setting them, this fires the __set() method... but since we check the value of the private ___init property, we don't fire an error in this instance. Future requests for the properties will not incurr a hit to __set() or __get() as long as the property is defined. If it is not defined then voila, the error fires and we die (we could do anything such as fire an exception but I just die for the example). With respect to initialization I use func_get_args() so that an arbitrary number of parameters can be passed to the constructor. This is useful since we don't know the number of properties. I also check if an argument is an array. If it is an array then I treat it as a property with an initialized value and create and set the property accordingly. That's all there is to it. Cheers, Rob. -- http://www.interjinn.com Application and Templating Framework for PHP -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: "use strict" or similar in PHP?
On Sun, 2009-03-01 at 09:57 -0500, Robert Cummings wrote: > On Sun, 2009-03-01 at 09:48 -0500, tedd wrote: > > At 9:21 AM -0500 3/1/09, Robert Cummings wrote: > > >On Sun, 2009-03-01 at 08:36 -0500, tedd wrote: > > >> The OP asked: > > >> > > >> >Is there in PHP something like "use strict" from perl? I find it pretty > > >> >annoying to need to run script over and over again just to find out > > >> that I > > >> >made typo in variable name. > > >> > > >> And I've been waiting for an answer myself, but I haven't seen one. > > >> > > >> From what I remember, in perl if you use "use strict;" it requires to > > >> to define your variables (my) before using them. If you make a > > >> variable typo in your code, then you'll trigger an error when you try > > >> to run it. > > >> > > >> From what I've seen of php, even with using strict error reporting, > > >> you can do that all day long without generating an error. > > >> > > >> So the answer appears to be "No, you can't do that in PHP." Is > > >>that the answer? > > > > > >You can do anything you want... :) > > > > Rob: > > > > Just to show that your words of wisdom don't go unnoticed, see here: > > > > http://php1.net/oop/strict/ > > > > That's pretty slick. > > > > Now if I only understood WT* you did, I would be a better programmer for it. > > I used the magic methods __set() and __get() to facilitate error > handling. These methods fire when a property is not accessible. So when > the property does not exist these will fire. So first we init the > properties when we create an instance (I'll discuss this shortly)... to > init them we are setting them, this fires the __set() method... but > since we check the value of the private ___init property, we don't fire > an error in this instance. Future requests for the properties will not > incurr a hit to __set() or __get() as long as the property is defined. > If it is not defined then voila, the error fires and we die (we could do > anything such as fire an exception but I just die for the example). > > With respect to initialization I use func_get_args() so that an > arbitrary number of parameters can be passed to the constructor. This is > useful since we don't know the number of properties. I also check if an > argument is an array. If it is an array then I treat it as a property > with an initialized value and create and set the property accordingly. > > That's all there is to it. Oh, I didn't bother, but if someone finds this at all useful, they could use debug_backtrace() to walk the call stack and output the actual source file and line number where the illegal property was requested. Cheers, Rob. -- http://www.interjinn.com Application and Templating Framework for PHP -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] file locking...
hi rob... here's the issue in more detail.. i have multiple processes that are generated/created and run in a simultaneous manner. each process wants to get XX number of files from the same batch of files... assume i have a batch of 50,000 files. my issue is how do i allow each of the processes to get their batch of unique files as fast as possible. (the 50K number is an arbotrary number.. my project will shrink/expand over time... if i dump all the 50K files in the same dir, i can have a lock file that would allow each process to sequentially read/write the lock file, and then access the dir to get the XX files the process is needing. (each process is just looking to get the next batch of files for processing. there's no searching based on text in the name of the files. it's a kind of fifo queing system) this approach could work, but it's basically sequential, and could in theory get into race conditions regarding the lockfile. i could also have the process that creates the files, throw the files in some kind of multiple directory processes, where i split the 50K files into separate dirs and somehow implement logic to allow the cient process to fetch the files from the unique/separate dirs.. but this could get ugly. so my issue is essentially how can i allow as close to simultaneous access by client/child processes to a kind of FIFO of files... whatever logic i create for this process, will also be used for the next iteration of the project, where i get rid of the files.. and i use some sort of database as the informational storage. hopefully this provides a little more clarity. thanks -Original Message- From: Robert Cummings [mailto:rob...@interjinn.com] Sent: Sunday, March 01, 2009 2:50 AM To: bruce Cc: php-general@lists.php.net Subject: Re: [PHP] file locking... On Sat, 2009-02-28 at 21:46 -0800, bruce wrote: > Hi. > > Got a bit of a question/issue that I'm trying to resolve. I'm asking this of > a few groups so bear with me. > > I'm considering a situation where I have multiple processes running, and > each process is going to access a number of files in a dir. Each process > accesses a unique group of files, and then writes the group of files to > another dir. I can easily handle this by using a form of locking, where I > have the processes lock/read a file and only access the group of files in > the dir based on the open/free status of the lockfile. > > However, the issue with the approach is that it's somewhat synchronous. I'm > looking for something that might be more asynchronous/parallel, in that I'd > like to have multiple processes each access a unique group of files from the > given dir as fast as possible. > > So.. Any thoughts/pointers/comments would be greatly appreciated. Any > pointers to academic research, etc.. would be useful. Threads? Or spawn off child processes. Maybe I'm not understanding your issues well enough. Cheers, Rob. -- http://www.interjinn.com Application and Templating Framework for PHP -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] file locking...
On Sun, 2009-03-01 at 09:09 -0800, bruce wrote: > hi rob... > > here's the issue in more detail.. > > i have multiple processes that are generated/created and run in a > simultaneous manner. each process wants to get XX number of files from the > same batch of files... assume i have a batch of 50,000 files. my issue is > how do i allow each of the processes to get their batch of unique files as > fast as possible. (the 50K number is an arbotrary number.. my project will > shrink/expand over time... > > if i dump all the 50K files in the same dir, i can have a lock file that > would allow each process to sequentially read/write the lock file, and then > access the dir to get the XX files the process is needing. (each process is > just looking to get the next batch of files for processing. there's no > searching based on text in the name of the files. it's a kind of fifo queing > system) this approach could work, but it's basically sequential, and could > in theory get into race conditions regarding the lockfile. > > i could also have the process that creates the files, throw the files in > some kind of multiple directory processes, where i split the 50K files into > separate dirs and somehow implement logic to allow the cient process to > fetch the files from the unique/separate dirs.. but this could get ugly. > > so my issue is essentially how can i allow as close to simultaneous access > by client/child processes to a kind of FIFO of files... > > whatever logic i create for this process, will also be used for the next > iteration of the project, where i get rid of the files.. and i use some sort > of database as the informational storage. > > hopefully this provides a little more clarity. Would I be right in assuming that a process grabs X of the oldest available files and then begins to work on them. Then the next process would essentially grab the next X oldest files so on and so forth over and over again? Also is the file discarded once processed? Would I be correct in presuming that processing of the files takes longer than grabbing the files wanted? If so then I would have a single lock upon which all processes wait. Each process grabs the lock when it can and then moves X oldest files to a working directory where it can then process them. So... directory structure: /ROOT /ROOT/queue /ROOT/work Locks... /ROOT/lock So let's say you have 500 files: /ROOT/queue/file_001.dat /ROOT/queue/file_002.dat /ROOT/queue/file_003.dat ... /ROOT/queue/file_499.dat /ROOT/queue/file_500.dat And you have 5 processes... /proc/1 /proc/2 /proc/3 /proc/4 /proc/5 Now to start all processes try to grab the lock at the same time, by virtue of lock mechanics only one process gets the lock... let's say for instance 4 While 4 has the lock all the other processes go to sleep for say... 1 usecs... upon failing to get the lock. So process 4 transfers file_001.dat through to file_050.dat into /ROOT/work. /ROOT/work/file_001.dat /ROOT/work/file_002.dat /ROOT/work/file_003.dat ... /ROOT/work/file_049.dat /ROOT/work/file_050.dat Then it releases the lock and begins processing meanwhile the other processes wake up and try to grab the lock again... this time PID 2 gets it. It does the same... /ROOT/work/file_043.dat /ROOT/work/file_044.dat /ROOT/work/file_045.dat ... /ROOT/work/file_049.dat /ROOT/work/file_100.dat /ROOT/queue/file_101.dat /ROOT/queue/file_102.dat /ROOT/queue/file_103.dat ... /ROOT/queue/file_499.dat /ROOT/queue/file_500.dat Now while it was doing that PID 4 finished and all it's files are now deleted. The first thing it does is try to get the lock so it can get more... but it's still owned by PID 2 so PID 4 goes to sleep. Once PID 2 gets it's files it releases the lock and off it goes and the cycle continued. Now there's still an issue with respect to incoming partially written files. During the incoming process those should be written elsewhere... lets say /ROOT/incoming. Once writing of the file is complete it can be moved to /ROOT/queue. Also if you don't want processes to delete the files you can have yet another directory /ROOT/processed. So with everything considered here's your directory structure: /ROOT /ROOT/incoming /ROOT/processed /ROOT/queue /ROOT/work One last thing to consider is that if there are no available files on which to work then you might have your processes sleep a little longer. Cheers, Rob. -- http://www.interjinn.com Application and Templating Framework for PHP -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] file locking...
hi rob... what you have written is similar to my initial approach... my question, and the reason for posting this to a few different groups.. is to see if someone has pointers/thoughts for something much quicker... this is going to handle processing requests from client apps to a webservice.. the backend of the service has to quickly process the files in the dir as fast as possible to return the data to the web client query... thanks -Original Message- From: Robert Cummings [mailto:rob...@interjinn.com] Sent: Sunday, March 01, 2009 9:54 AM To: bruce Cc: php-general@lists.php.net Subject: RE: [PHP] file locking... On Sun, 2009-03-01 at 09:09 -0800, bruce wrote: > hi rob... > > here's the issue in more detail.. > > i have multiple processes that are generated/created and run in a > simultaneous manner. each process wants to get XX number of files from the > same batch of files... assume i have a batch of 50,000 files. my issue is > how do i allow each of the processes to get their batch of unique files as > fast as possible. (the 50K number is an arbotrary number.. my project will > shrink/expand over time... > > if i dump all the 50K files in the same dir, i can have a lock file that > would allow each process to sequentially read/write the lock file, and then > access the dir to get the XX files the process is needing. (each process is > just looking to get the next batch of files for processing. there's no > searching based on text in the name of the files. it's a kind of fifo queing > system) this approach could work, but it's basically sequential, and could > in theory get into race conditions regarding the lockfile. > > i could also have the process that creates the files, throw the files in > some kind of multiple directory processes, where i split the 50K files into > separate dirs and somehow implement logic to allow the cient process to > fetch the files from the unique/separate dirs.. but this could get ugly. > > so my issue is essentially how can i allow as close to simultaneous access > by client/child processes to a kind of FIFO of files... > > whatever logic i create for this process, will also be used for the next > iteration of the project, where i get rid of the files.. and i use some sort > of database as the informational storage. > > hopefully this provides a little more clarity. Would I be right in assuming that a process grabs X of the oldest available files and then begins to work on them. Then the next process would essentially grab the next X oldest files so on and so forth over and over again? Also is the file discarded once processed? Would I be correct in presuming that processing of the files takes longer than grabbing the files wanted? If so then I would have a single lock upon which all processes wait. Each process grabs the lock when it can and then moves X oldest files to a working directory where it can then process them. So... directory structure: /ROOT /ROOT/queue /ROOT/work Locks... /ROOT/lock So let's say you have 500 files: /ROOT/queue/file_001.dat /ROOT/queue/file_002.dat /ROOT/queue/file_003.dat ... /ROOT/queue/file_499.dat /ROOT/queue/file_500.dat And you have 5 processes... /proc/1 /proc/2 /proc/3 /proc/4 /proc/5 Now to start all processes try to grab the lock at the same time, by virtue of lock mechanics only one process gets the lock... let's say for instance 4 While 4 has the lock all the other processes go to sleep for say... 1 usecs... upon failing to get the lock. So process 4 transfers file_001.dat through to file_050.dat into /ROOT/work. /ROOT/work/file_001.dat /ROOT/work/file_002.dat /ROOT/work/file_003.dat ... /ROOT/work/file_049.dat /ROOT/work/file_050.dat Then it releases the lock and begins processing meanwhile the other processes wake up and try to grab the lock again... this time PID 2 gets it. It does the same... /ROOT/work/file_043.dat /ROOT/work/file_044.dat /ROOT/work/file_045.dat ... /ROOT/work/file_049.dat /ROOT/work/file_100.dat /ROOT/queue/file_101.dat /ROOT/queue/file_102.dat /ROOT/queue/file_103.dat ... /ROOT/queue/file_499.dat /ROOT/queue/file_500.dat Now while it was doing that PID 4 finished and all it's files are now deleted. The first thing it does is try to get the lock so it can get more... but it's still owned by PID 2 so PID 4 goes to sleep. Once PID 2 gets it's files it releases the lock and off it goes and the cycle continued. Now there's still an issue with respect to incoming partially written files. During the incoming process those should be written elsewhere... lets say /ROOT/incoming. Once writing of the file is complete it can be moved to /ROOT/queue. Also if you don't want processes to delete the files you can have yet another directory /ROOT/processed. So with everything considered here's your directory structure: /ROOT /ROOT/incoming
RE: [PHP] file locking...
On Sun, 2009-03-01 at 10:05 -0800, bruce wrote: > hi rob... > > what you have written is similar to my initial approach... my question, and > the reason for posting this to a few different groups.. is to see if someone > has pointers/thoughts for something much quicker... > > this is going to handle processing requests from client apps to a > webservice.. the backend of the service has to quickly process the files in > the dir as fast as possible to return the data to the web client query... Then use a database to process who gets what. DB queries will queue up while a lock is in place so batches will occur on first come first served basis. I had thought this was for a background script. This will save your script from having to browse the filesystem files, sort by age, etc. Instead put an index on the ID of the file and grab the X lowest IDs. Cheers, Rob. -- http://www.interjinn.com Application and Templating Framework for PHP -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] file locking...
2009/3/1 Robert Cummings > On Sun, 2009-03-01 at 10:05 -0800, bruce wrote: > > hi rob... > > > > what you have written is similar to my initial approach... my question, > and > > the reason for posting this to a few different groups.. is to see if > someone > > has pointers/thoughts for something much quicker... > > > > this is going to handle processing requests from client apps to a > > webservice.. the backend of the service has to quickly process the files > in > > the dir as fast as possible to return the data to the web client query... > > Then use a database to process who gets what. DB queries will queue up > while a lock is in place so batches will occur on first come first > served basis. I had thought this was for a background script. This will > save your script from having to browse the filesystem files, sort by > age, etc. Instead put an index on the ID of the file and grab the X > lowest IDs. A database would be the best way to do this, but I've need to handle this situation with files in the past and this is the solution I came up with... 1) Get the next filename to process 2) Try to move it to /tmp/whatever. 3) Check to see if /tmp/whatever. exists, and if it does process it then delete it or move it to an archive directory 4) Repeat until there are no files left to process I have this running on a server that processes several million files a day without any issues. For database-based queues I use a similar system but the move is replaced by an update which sets the pid field of a single row. I then do a select where that pid is my pid and process whatever comes back. I have several queues that use this system and combined they're handling 10's of millions of queue items per day without any problems, with the advantage that I can scale across servers as well as processes. -Stuart -- http://stut.net/
Re: [PHP] www.soongy.com
Also check this one out: google uses it in gmail: http://code.google.com/p/jquery-multifile-plugin/downloads/detail?name=multiple-file-upload.zip&can=2&q= Cheers, Tim Tim-Hinnerk Heuer http://www.ihostnz.com Groucho Marx - "I have had a perfectly wonderful evening, but this wasn't it." 2009/3/2 tedd > At 4:17 PM +0200 3/1/09, Nitsan Bin-Nun wrote: > >> There is no need to go that far, try to google a bit about swfupload. >> >> In short, this is a flash & javascript component that give's you the >> ability to maintain the upload, get the current speed, get the current >> amount of uploaded data, etc. It is very simple and works like a charm on a >> dedi server. There are some issues on shared server sometimes, but even >> these things are not that much complicated and can be easily solved. >> >> HTH, >> > > > Nitsan > > Oh yeah, try this: > > http://swfupload.org/documentation/demonstration > > and go through the "up" link -- and then try the "See it in action!" link > and also try the "Demonstration" link. You can even use the > demo.swfupload.org link, which will provide you with this: > > http://demo.swfupload.org/v220beta5/index.htm > > All of which is well worth the effort if you're trying to waste your time. > If their code is as good as their web site, no thanks -- I'll pass. > > But if I was to seriously investigate it, I would go directly to Google: > > http://code.google.com/p/swfupload/ > > However, I haven't a clue as to if it works or not. > > Cheers, > > tedd > > > -- > --- > http://sperling.com http://ancientstones.com http://earthstones.com > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > >
Re: [PHP] www.soongy.com
you can use gears pretty easily to make a seamless multiple file upload now. it's all javascript too so you can make it look how you want, behave how you want, etc. without having to buy/tweak flash code. On Sun, Mar 1, 2009 at 12:07 PM, German Geek wrote: > Also check this one out: google uses it in gmail: > http://code.google.com/p/jquery-multifile-plugin/downloads/detail?name=multiple-file-upload.zip&can=2&q= > > Cheers, > Tim > Tim-Hinnerk Heuer > > http://www.ihostnz.com > Groucho Marx - "I have had a perfectly wonderful evening, but this wasn't > it." > > 2009/3/2 tedd > >> At 4:17 PM +0200 3/1/09, Nitsan Bin-Nun wrote: >> >>> There is no need to go that far, try to google a bit about swfupload. >>> >>> In short, this is a flash & javascript component that give's you the >>> ability to maintain the upload, get the current speed, get the current >>> amount of uploaded data, etc. It is very simple and works like a charm on a >>> dedi server. There are some issues on shared server sometimes, but even >>> these things are not that much complicated and can be easily solved. >>> >>> HTH, >>> >> >> >> Nitsan >> >> Oh yeah, try this: >> >> http://swfupload.org/documentation/demonstration >> >> and go through the "up" link -- and then try the "See it in action!" link >> and also try the "Demonstration" link. You can even use the >> demo.swfupload.org link, which will provide you with this: >> >> http://demo.swfupload.org/v220beta5/index.htm >> >> All of which is well worth the effort if you're trying to waste your time. >> If their code is as good as their web site, no thanks -- I'll pass. >> >> But if I was to seriously investigate it, I would go directly to Google: >> >> http://code.google.com/p/swfupload/ >> >> However, I haven't a clue as to if it works or not. >> >> Cheers, >> >> tedd >> >> >> -- >> --- >> http://sperling.com http://ancientstones.com http://earthstones.com >> >> -- >> PHP General Mailing List (http://www.php.net/) >> To unsubscribe, visit: http://www.php.net/unsub.php >> >> > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] 500 Internal Server Error
Hello All, What is the situation when we get internal server error 500 on PHP pages? >From Internet I got some info like, you get it when : 1) If friendly urls are not supported by apache.(mod_rewrite) 2) If max_execution_time max's out. and some more related to apache. I got a weird situation where I do PHP eval() and if the parsed string is wrong, I get internal server error 500. So question is I was under the impression that PHP code errors will never result in a http response errors. Am I completely wrong here? Can anyone tell me from their experience what are the specific scenarios we get this error? Thanks, V
Re: [PHP] www.soongy.com
At 12:13 PM -0800 3/1/09, mike wrote: you can use gears pretty easily to make a seamless multiple file upload now. it's all javascript too so you can make it look how you want, behave how you want, etc. without having to buy/tweak flash code. On Sun, Mar 1, 2009 at 12:07 PM, German Geek wrote: Also check this one out: google uses it in gmail: > http://code.google.com/p/jquery-multifile-plugin/downloads/detail?name=multiple-file-upload.zip&can=2&q= Understood, but I don't think either of these do what the OP wanted, whcih was a real time file upload progress bar. Cheers, tedd -- --- http://sperling.com http://ancientstones.com http://earthstones.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] 500 Internal Server Error
--- Use FreeOpenSourceSoftwares, Stop piracy, Let the developers live. Get a Free CD of Ubuntu mailed to your door without any cost. Visit : www.ubuntu.com -- On Mon, Mar 2, 2009 at 2:18 AM, VamVan wrote: > Hello All, > > What is the situation when we get internal server error 500 on PHP pages? > > From Internet I got some info like, you get it when : > > 1) If friendly urls are not supported by apache.(mod_rewrite) > 2) If max_execution_time max's out. > and some more related to apache. > > I got a weird situation where I do PHP eval() and if the parsed string is > wrong, I get internal server error 500. > > So question is I was under the impression that PHP code errors will never > result in a http response errors. Am I completely wrong here? Can anyone > tell me from their experience what are the specific scenarios we get this > error? Understandably 2) for the eval. > > > Thanks, > V >
Re: [PHP] www.soongy.com
gears will allow you to do that, more or less. i have it going... On Sun, Mar 1, 2009 at 12:34 PM, tedd wrote: > At 12:13 PM -0800 3/1/09, mike wrote: >> >> you can use gears pretty easily to make a seamless multiple file >> upload now. it's all javascript too so you can make it look how you >> want, behave how you want, etc. without having to buy/tweak flash >> code. >> >> >> On Sun, Mar 1, 2009 at 12:07 PM, German Geek wrote: >>> >>> Also check this one out: google uses it in gmail: >> >> > >> http://code.google.com/p/jquery-multifile-plugin/downloads/detail?name=multiple-file-upload.zip&can=2&q= > > Understood, but I don't think either of these do what the OP wanted, whcih > was a real time file upload progress bar. > > Cheers, > > tedd > -- > --- > http://sperling.com http://ancientstones.com http://earthstones.com > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] www.soongy.com
Thank you all for your help! I tried to do without being dependent from flash, but as I see there are only two ways for this: "flash" and "loading without tracking percentage". I choose second way :) On Mon, Mar 2, 2009 at 1:58 AM, mike wrote: > gears will allow you to do that, more or less. i have it going... > > On Sun, Mar 1, 2009 at 12:34 PM, tedd wrote: > > At 12:13 PM -0800 3/1/09, mike wrote: > >> > >> you can use gears pretty easily to make a seamless multiple file > >> upload now. it's all javascript too so you can make it look how you > >> want, behave how you want, etc. without having to buy/tweak flash > >> code. > >> > >> > >> On Sun, Mar 1, 2009 at 12:07 PM, German Geek wrote: > >>> > >>> Also check this one out: google uses it in gmail: > >> > >> > > >> > http://code.google.com/p/jquery-multifile-plugin/downloads/detail?name=multiple-file-upload.zip&can=2&q= > > > > Understood, but I don't think either of these do what the OP wanted, > whcih > > was a real time file upload progress bar. > > > > Cheers, > > > > tedd > > -- > > --- > > http://sperling.com http://ancientstones.com http://earthstones.com > > > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- Best Regards, Gevorg Harutyunyan
Re: [PHP] www.soongy.com
I have a basic demo here. The code is not at it's best, you have to hit reload to upload a new file. http://mikehost.com/~mike/tmp/u/ It does show you though with some javascript trickery and some math you can derive estimated time and approximate speed. I did have an example of multiple files too (I don't think this one supports that) At some point I'll publish all the components required and clean it up nicely. I also have an nginx module to complement it that should be available soon, to alleviate the need for the PHP processing piece (even though it does work like a charm) On Sun, Mar 1, 2009 at 10:05 PM, Gevorg Harutyunyan wrote: > Thank you all for your help! > > I tried to do without being dependent from flash, but as I see there are > only two ways for this: "flash" and "loading without tracking percentage". I > choose second way :) > > On Mon, Mar 2, 2009 at 1:58 AM, mike wrote: >> >> gears will allow you to do that, more or less. i have it going... >> >> On Sun, Mar 1, 2009 at 12:34 PM, tedd wrote: >> > At 12:13 PM -0800 3/1/09, mike wrote: >> >> >> >> you can use gears pretty easily to make a seamless multiple file >> >> upload now. it's all javascript too so you can make it look how you >> >> want, behave how you want, etc. without having to buy/tweak flash >> >> code. >> >> >> >> >> >> On Sun, Mar 1, 2009 at 12:07 PM, German Geek wrote: >> >>> >> >>> Also check this one out: google uses it in gmail: >> >> >> >> > >> >> >> >> http://code.google.com/p/jquery-multifile-plugin/downloads/detail?name=multiple-file-upload.zip&can=2&q= >> > >> > Understood, but I don't think either of these do what the OP wanted, >> > whcih >> > was a real time file upload progress bar. >> > >> > Cheers, >> > >> > tedd >> > -- >> > --- >> > http://sperling.com http://ancientstones.com http://earthstones.com >> > >> >> -- >> PHP General Mailing List (http://www.php.net/) >> To unsubscribe, visit: http://www.php.net/unsub.php >> > > > > -- > Best Regards, > Gevorg Harutyunyan > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] PHP-based forum software ideas
I'm looking around and usually wikipedia has a complete list of everything... http://en.wikipedia.org/wiki/Comparison_of_Internet_forum_software_(PHP) Anything missing there? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Re: Browser timeout
40 seconds is not very long execution time for timeout, if you have some loop in the script, try to output a dot character, for example, on every iteration, with flush() immediately after echo (or whatever command you use for sending output). ""? "" pí¹e v diskusním pøíspìvku news:907722000902270633j559478cdi1ff0f1c7fc07...@mail.gmail.com... > Hello, I am running a script that process the user's request, which > usually > takes about 40 seconds. The problem is that on Internet Explorer 6, it > timeouts - probably because no response was sent. > How do I solve this problem? Since I tried some things and header() > messages, nothing worked. > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
