[PHP] curl_exec won't return (any data)

[Tolas Anon]

Hi..

For a set of media import routines, i'm using a
javascript->php_on_apache->windows.bat->php_cli->curl->php_script invocation
method.
It seems longwinded, but it's designed to have different parts of the import
process run on different servers.

I'm stuck at getting curl_exec() to return the data of the final php_script
that does the importing.
The script itself runs fine, and i've recorded good details that curl_exec()
is supposed to catch with file_put_contents("/some/debug.txt",
json_encode($returnArray)), and from those debug-printouts it's just a few
tiny steps towards a cascade of "return" statements, followed by a
echo(json_encode($returnArray)) and a normal end to the php_script at the
end of the call chain.

However, curl_exec() seems to hang completely. I've added over a dozen
"debuginfo -> file on server" statements, and the one that should fire
straight after curl_exec() does not fire.

It does this only with large (1.8gb) video files, a smaller (60mb) video
file doesn't produce this problem and the entire import routines work fine
then.

I'd very much appreciate any tips you might have for me.

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

On Tue, Feb 8, 2011 at 1:35 PM, Ashley Sheridan 
wrote:

> On Tue, 2011-02-08 at 13:24 +0100, Tolas Anon wrote:
>
> > Hi..
> >
> > For a set of media import routines, i'm using a
> > javascript->php_on_apache->windows.bat->php_cli->curl->php_script
> invocation
> > method.
> > It seems longwinded, but it's designed to have different parts of the
> import
> > process run on different servers.
> >
> > I'm stuck at getting curl_exec() to return the data of the final
> php_script
> > that does the importing.
> > The script itself runs fine, and i've recorded good details that
> curl_exec()
> > is supposed to catch with file_put_contents("/some/debug.txt",
> > json_encode($returnArray)), and from those debug-printouts it's just a
> few
> > tiny steps towards a cascade of "return" statements, followed by a
> > echo(json_encode($returnArray)) and a normal end to the php_script at the
> > end of the call chain.
> >
> > However, curl_exec() seems to hang completely. I've added over a dozen
> > "debuginfo -> file on server" statements, and the one that should fire
> > straight after curl_exec() does not fire.
> >
> > It does this only with large (1.8gb) video files, a smaller (60mb) video
> > file doesn't produce this problem and the entire import routines work
> fine
> > then.
> >
> > I'd very much appreciate any tips you might have for me.
>
>
> Let me see if I've got this right.
>
> The windows.bat is processing the media file somehow, then calling a
> php_cli script which makes a cURL call to another web-based PHP script?
> Is this right? The final script I assume is getting sent some info from
> the cURL call and is using it somehow (in a DB maybe?) before some sort
> of message back to your curl call. What is the code then doing with it
> after that?
>
> Thanks,
> Ash
> http://www.ashleysheridan.co.uk
>
>
>
oops, i missed a step (php_daemon_script) in the chain of calls;

it's : javascript -> php_on_apache -> windows.bat -> php_cli ->
php_daemon_script -> curl_exec -> php_script

followed by calls-until-finished from javascript that read the status of
php_script via json files written to the server and thus display the status
to the end-user..

the windows.bat just starts up cli-php with admin privileges, which executes
the php_daemon_script, which uses repetitive curl calls to the import-script
("php_script" at the end of my chain) that does all the work for a single
item in the total upload/import queue; it does video conversion with
exec(/path/to/ffmpeg), photo conversion with imagemagick, and updates the db
with the php adodb library.
php_script at the end of it's work returns a simple and short status array
(json_encode()d) to the php_daemon_script via curl_exec, that dictates if
the php_daemon_script should continue calling the (import) php_script more
times.

it's curl_exec that hangs/freezes, both with using CURLOPT_RETURNTRANSFER=1,
or capturing output with ob_start() and ob_get_clean(). i've gathered that
much from my custom debug logs.

Fwd: [PHP] curl_exec won't return (any data)

[Tolas Anon]

-- Forwarded message --
From: Tolas Anon 
Date: Tue, Feb 8, 2011 at 1:56 PM
Subject: Re: [PHP] curl_exec won't return (any data)
To: a...@ashleysheridan.co.uk




On Tue, Feb 8, 2011 at 1:50 PM, Tolas Anon  wrote:

>
>
> On Tue, Feb 8, 2011 at 1:35 PM, Ashley Sheridan 
> wrote:
>
>> On Tue, 2011-02-08 at 13:24 +0100, Tolas Anon wrote:
>>
>> > Hi..
>> >
>> > For a set of media import routines, i'm using a
>> > javascript->php_on_apache->windows.bat->php_cli->curl->php_script
>> invocation
>> > method.
>> > It seems longwinded, but it's designed to have different parts of the
>> import
>> > process run on different servers.
>> >
>> > I'm stuck at getting curl_exec() to return the data of the final
>> php_script
>> > that does the importing.
>> > The script itself runs fine, and i've recorded good details that
>> curl_exec()
>> > is supposed to catch with file_put_contents("/some/debug.txt",
>> > json_encode($returnArray)), and from those debug-printouts it's just a
>> few
>> > tiny steps towards a cascade of "return" statements, followed by a
>> > echo(json_encode($returnArray)) and a normal end to the php_script at
>> the
>> > end of the call chain.
>> >
>> > However, curl_exec() seems to hang completely. I've added over a dozen
>> > "debuginfo -> file on server" statements, and the one that should fire
>> > straight after curl_exec() does not fire.
>> >
>> > It does this only with large (1.8gb) video files, a smaller (60mb) video
>> > file doesn't produce this problem and the entire import routines work
>> fine
>> > then.
>> >
>> > I'd very much appreciate any tips you might have for me.
>>
>>
>> Let me see if I've got this right.
>>
>> The windows.bat is processing the media file somehow, then calling a
>> php_cli script which makes a cURL call to another web-based PHP script?
>> Is this right? The final script I assume is getting sent some info from
>> the cURL call and is using it somehow (in a DB maybe?) before some sort
>> of message back to your curl call. What is the code then doing with it
>> after that?
>>
>> Thanks,
>> Ash
>> http://www.ashleysheridan.co.uk
>>
>>
>>
> oops, i missed a step (php_daemon_script) in the chain of calls;
>
> it's : javascript -> php_on_apache -> windows.bat -> php_cli ->
> php_daemon_script -> curl_exec -> php_script
>
> followed by calls-until-finished from javascript that read the status of
> php_script via json files written to the server and thus display the status
> to the end-user..
>
> the windows.bat just starts up cli-php with admin privileges, which
> executes the php_daemon_script, which uses repetitive curl calls to the
> import-script ("php_script" at the end of my chain) that does all the work
> for a single item in the total upload/import queue; it does video conversion
> with exec(/path/to/ffmpeg), photo conversion with imagemagick, and updates
> the db with the php adodb library.
> php_script at the end of it's work returns a simple and short status array
> (json_encode()d) to the php_daemon_script via curl_exec, that dictates if
> the php_daemon_script should continue calling the (import) php_script more
> times.
>
> it's curl_exec that hangs/freezes, both with using CURLOPT_RETURNTRANSFER=1,
> or capturing output with ob_start() and ob_get_clean(). i've gathered that
> much from my custom debug logs.
>

both php_daemon_script and php_script should use ini_set
('max_execution_time',0);


but i just saw that php_daemon_script does not!!
that might be my error, i hope.

i'll let you know in about 1 hour when my test completes..

Re: [PHP] Paging and permissions

[Tolas Anon]

On Tue, Feb 8, 2011 at 1:36 PM, Arno Kuhl  wrote:

> I'm hoping some clever php gurus have been here before and are willing to
> share some ideas.
>
> I have a site where articles are assigned to categories in containers. An
> article can be assigned to only one category per container, but one or more
> containers. Access permissions can be set per article, per category and/or
> per container, for one or more users and/or user groups. If an article is
> assigned to 10 categories and only one of those has a permission denying
> access, then the article can't be accessed even if browsing through one of
> the other 9 categories. Currently everything works fine, with article
> titles
> showing when browsing through category or search result lists, and a
> message
> is displayed when the article is clicked if it cannot be viewed because of
> a
> permission.
>
> Now there's a requirement to not display the article title in category
> lists
> and search results if it cannot be viewed. I'm stuck with how to determine
> the number of results for paging at the start of the list or search. The
> site is quite large (20,000+ articles and growing) so reading the entire
> result set and sifting through it with permission rules for each request is
> not an option. But it might be an option if done once at the start of each
> search or list request, and then use that temporary modified result set for
> subsequent requests on the same set. I thought of saving the set to a
> temporary db table or file (not sure about overhead of
> serializing/unserializing large arrays). A sizing exercise based on the
> recordset returned for searches and lists shows a max of about 150MB for
> 20,000 articles and 380MB for 50,000 articles that needs to be saved
> temporarily per search or list request - in the vast majority of cases the
> set will be *much* smaller but it needs to cope with the worst case, and
> still do so a year down the line.
>
> All this extra work because I can't simply get an accurate number of
> results
> for paging, because of permissions!
>
> So my questions are:
> 1. Which is better (performance) for this situation: file or db?
> 2. How do I prepare a potentially very large data set for file or fast
> writing to a new table (ie I obviously don't want to write it record by
> record)
> 3. Are there any other alternatives worth looking at?
>
> TIA
>
> Cheers
> Arno
>

Seems to me you make your setup needlessly complicated and restrictive.

And it's bad form to display articles in search results that aren't allowed
to be viewed..

Tell us more about why you want it to be so restrictive, i just don't
understand it.

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

ok, i've done another run, this time with both php_daemon_script and
php_script allowed to run indefinitely, and i've triple-checked my
results..

- all the calls to convert and import the +-2gb video file in
php_script (the import-worker-script) complete just fine, they update
a debug file on the server with correct results. from there it's just
a few simple steps back to curl_exec(), which i've tripple-checked to
be followed correctly, with more print-to-seperate-debug-file
statements.

- the call to curl_exec() in launched by php_daemon_script never
completes. i've added a print-to-debug-file statement right after
curl_exec() there, and it does NOT update that debug file. previous
experience shows that this is true for curl_setopt
(CURLOPT_RETURNTRANSFER,1) aswell as
ob_start();curl_exec(blah);file_put_contents("/path/to/debug-file-N.txt",ob_get_clean()).

- the php_daemon_script continues to run in my debug window (i now
launch windows.bat manually), it does not crash/end, it freezes. and
from experience i can tell you that it will stay frozen for several
hours, not doing anything anymore.

i've run out of ideas :(

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

for completeness, i'll say that i'm using libcurl-7.21.3, and (again)
that these import routines work without problems for smaller video
files (tested with +-60mb vid file).

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

also: the vid file itself is converted correctly too, and inserted
into my db correctly.
i can even view the converted flv and it has the correct length and everything!

it's just that the frigging import won't continue with the rest of the
files in the queue..
it's maddening! ;-)

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

On Tue, Feb 8, 2011 at 3:39 PM, Tolas Anon  wrote:
> also: the vid file itself is converted correctly too, and inserted
> into my db correctly.
> i can even view the converted flv and it has the correct length and 
> everything!
>
> it's just that the frigging import won't continue with the rest of the
> files in the queue..
> it's maddening! ;-)
>

eh, "view the converted flv" _in_ the cms that does the importing, as
an end-user of the cms would.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

the one thing i can think of is that curl_exec() somehow stops
listening for results and hangs the calling php script
(php_daemon_script in this case) if it does not receive any data for
more than a few minutes (converting the 60mb vid file takes about a
minute, and the 2gb script well over 30 minutes)..

however i haven't been able to find any bugreports via google that
describe this bug..

could it be i found a new bug in libcurl?...

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

Things I've checked in the meanwhile;

curl_setopt($ch,CURLOPT_TIMEOUT, 0); and
curl_setopt($ch,CURLOPT_TIMEOUT, 9); have the same freezing
results as before

I've used a packetsniffer to analyze the dataflow on my port 80, with
all other http apps incl the browser not in memory/running, so i could
get a clear view.
Turns out i do get the traffic my php_daemon_script relies on and
should get from curl_exec() when the 2gb video file has finished
converting & importing, flowing from an 192.xyz.xyz.xyz address (which
is my apache) to a 82.xyz.xyz.xyz address (also resolves to my
apache).

I currently run the entire site, upload and import on the same machine
and instance of apache.
And the import was run from that windows.bat again, which i had
running as a normal dos window.
I made sure the sniffer caught both the initialization of the
offending 2gb-convert call, and it's termination, which i monitored by
monitoring the server filesystem (the size of the converted vid, and
the fact it got moved into it's final directory for serving to the
end-user).

This apache installation is on an adsl modem that does
outside-to-inside port 80 forwarding to my windows apache
installation, and i use the outside-world domain name linked to my
adsl IP for the site and upload and import.

However, this "wanted" traffic is reported by wireshark (the sniffer)
as having an invalid 0x header checksum, which should be
"something else" (a hex value of course).
The packet flow is like this:
192... -> 82... : HTTP/1.1 200 OK (text/html) {wanted data}
192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
192... -> 82... : HTTP > portno [FIN,ACK] seq=... ack=... win=. len=0
192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
192... -> 82... : HTTP > portno [RST,ACK] seq=... ack=... win=0 len=0
After this the http capturing goes silent again.

The wanted packet does not show up in the debug-info-to-file call made
by the php_daemon_script just after it does
$result=curl_exec(valid-settings);

I don't use sniffers often, so i have to ask;

Is this enough evidence to report it as a libcurl bug?
Or do i have to suspect php-cli and cmd.exe as well?



P.S.;
Entire WAMP installed by WampServer2.1d-x64.exe
PHP Version 5.3.4

System  Windows NT NOOT 6.1 build 7600 (Unknow Windows version Home
Premium Edition) AMD64
Build Date  Dec 15 2010 23:40:06
CompilerMSVC9 (Visual C++ 2008)
Architecturex64

libcurl-7.21.3.0 added manually

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

On Tue, Feb 8, 2011 at 7:01 PM, Tolas Anon  wrote:
> Things I've checked in the meanwhile;
>
> curl_setopt($ch,CURLOPT_TIMEOUT, 0); and
> curl_setopt($ch,CURLOPT_TIMEOUT, 9); have the same freezing
> results as before
>
> I've used a packetsniffer to analyze the dataflow on my port 80, with
> all other http apps incl the browser not in memory/running, so i could
> get a clear view.
> Turns out i do get the traffic my php_daemon_script relies on and
> should get from curl_exec() when the 2gb video file has finished
> converting & importing, flowing from an 192.xyz.xyz.xyz address (which
> is my apache) to a 82.xyz.xyz.xyz address (also resolves to my
> apache).
>
> I currently run the entire site, upload and import on the same machine
> and instance of apache.
> And the import was run from that windows.bat again, which i had
> running as a normal dos window.
> I made sure the sniffer caught both the initialization of the
> offending 2gb-convert call, and it's termination, which i monitored by
> monitoring the server filesystem (the size of the converted vid, and
> the fact it got moved into it's final directory for serving to the
> end-user).
>
> This apache installation is on an adsl modem that does
> outside-to-inside port 80 forwarding to my windows apache
> installation, and i use the outside-world domain name linked to my
> adsl IP for the site and upload and import.
>
> However, this "wanted" traffic is reported by wireshark (the sniffer)
> as having an invalid 0x header checksum, which should be
> "something else" (a hex value of course).
> The packet flow is like this:
> 192... -> 82... : HTTP/1.1 200 OK (text/html) {wanted data}
> 192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
> 192... -> 82... : HTTP > portno [FIN,ACK] seq=... ack=... win=. len=0
> 192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
> 192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
> 192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
> 192... -> 82... : RETRANSMISSION HTTP/1.1 200 OK (text/html) {wanted data}
> 192... -> 82... : HTTP > portno [RST,ACK] seq=... ack=... win=0 len=0
> After this the http capturing goes silent again.
>
> The wanted packet does not show up in the debug-info-to-file call made
> by the php_daemon_script just after it does
> $result=curl_exec(valid-settings);
>
> I don't use sniffers often, so i have to ask;
>
> Is this enough evidence to report it as a libcurl bug?
> Or do i have to suspect php-cli and cmd.exe as well?
>
>
>
> P.S.;
> Entire WAMP installed by WampServer2.1d-x64.exe
> PHP Version 5.3.4
>
> System  Windows NT NOOT 6.1 build 7600 (Unknow Windows version Home
> Premium Edition) AMD64
> Build Date      Dec 15 2010 23:40:06
> Compiler        MSVC9 (Visual C++ 2008)
> Architecture    x64
>
> libcurl-7.21.3.0 added manually
>

I've been thinking about that reported bad checksum. libcurl may be
right to reject those packets.
And i guess that apache determines that 0x checksum.
That is, _if_ i can trust the accuracy of that wireshark app.

Bit too many variables at work here for my liking, but this bug just
has to get fixed.

I guess i'll un-install wireshark, install a different sniffer app and
re-do the whole thing, i'll let you know the results when they are in.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

ok, did another run with a different packet analyzer, one that does

not have winPcap under the hood like WireShark does. I chose

microsoft network monitor 3.4 for the task.

the launch of the 2gb vid import (only part by curl_exec):

format:
packetnumber[TAB] time_date_local_adjusted[TAB] time_offset[TAB]

processname[TAB] source[TAB] destination[TAB] protocolname[TAB]

description[TAB] conv_id[TAB] tolas_comment

134 19:50:08 8-2-2011   31.2774633  Unavailable 

WindowsNameOfApacheServer   InternetDomainOfApacheServerHTTP

HTTP:Request, POST /site/cms/php/php_(import_)script.php,

Query:PHPSESSID=o365inhor0ln8mqdr0p8vb3rm7&action=batchNext 

{HTTP:40, TCP:38, IPv4:8}   

135 19:50:08 8-2-2011   31.2777843  Unavailable 

InternetDomainOfApacheServerWindowsNameOfApacheServer   TCP 

TCP:Flags=...A, SrcPort=3573, DstPort=HTTP(80), PayloadLen=0,

Seq=1169302766, Ack=4118332969, Win=260 (scale factor 0x8) = 66560  

{TCP:39, IPv4:8}

136 19:50:08 8-2-2011   31.2781472  Unavailable 

InternetDomainOfApacheServerWindowsNameOfApacheServer   HTTP

HTTP:Request, POST /site/cms/php/php_(import_)script.php,

Query:PHPSESSID=o365inhor0ln8mqdr0p8vb3rm7&action=batchNext 

{HTTP:41, TCP:39, IPv4:8}

138 19:50:08 8-2-2011   31.4807963  Unavailable 

WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 

TCP:Flags=...A, SrcPort=HTTP(80), DstPort=3573, PayloadLen=0,

Seq=4118332969, Ack=1169303037, Win=260 (scale factor 0x8) = 66560  

{TCP:39, IPv4:8}

139 19:50:08 8-2-2011   31.4859769  Unavailable 

InternetDomainOfApacheServerWindowsNameOfApacheServer   TCP 

TCP:Flags=...A, SrcPort=HTTP(80), DstPort=3573, PayloadLen=0,

Seq=4118332969, Ack=1169303037, Win=260 (scale factor 0x8) = 66560  

{TCP:38, IPv4:8}

140 19:50:08 8-2-2011   31.6234279  adsl-router 

WindowsNameOfApacheServer   TCP TCP:Flags=..S.,

SrcPort=2456, DstPort=14013, PayloadLen=0, Seq=1335058890, Ack=0,

Win=5840 ( Negotiating scale factor 0x2 ) = 5840{TCP:42,

IPv4:4}

141 19:50:11 8-2-2011   34.6104085  adsl-router 

WindowsNameOfApacheServer   TCP TCP:[SynReTransmit #140]

Flags=..S., SrcPort=2456, DstPort=14013, PayloadLen=0,

Seq=1335058890, Ack=0, Win=5840 ( Negotiating scale factor 0x2 ) =

5840{TCP:42, IPv4:4}

Packet 137 = adsl-router discovery traffic, not relevant.





Then, after 45 minutes or so, the completion of the conversion and
import, with the wanted data sent back to curl_exec() running from
php-cli, which was started by windows.bat

Some other (network discovery) traffic is likely mixed in, I added it
to be on the safe side.

590120:32:45 8-2-2011   2588.7346959Unavailable 
WindowsNameOfApacheServer
InternetNameOfApacheServer  HTTPHTTP:Response, HTTP/1.1, 
Status: Ok,
URL: /site/cms/php/php_(import_)script.php  {HTTP:41, TCP:39, IPv4:8}
{{CONTAINS THE WANTED DATA}}

590220:32:45 8-2-2011   2588.7354838
InternetNameOfApacheServer  WindowsNameOfApacheServer
ICMPICMP:Destination Unreachable Message, Communication
Administratively Prohibited, 82.161.37.94   {IPv4:8}

590320:32:48 8-2-2011   2591.6092751
FE80:0:0:0:E579:89FF:369D:668B  FF02:0:0:0:0:0:0:C  SSDPSSDP:Request,
M-SEARCH *  {HTTP:3, UDP:2, IPv6:1}

590420:32:48 8-2-2011   2591.7421833Unavailable 
WindowsNameOfApacheServer
InternetNameOfApacheServer  TCP TCP:[ReTransmit 
#5901]Flags=...AP...,
SrcPort=HTTP(80), DstPort=3573, PayloadLen=558, Seq=4118332969 -
4118333527, Ack=1169303037, Win=260 (scale factor 0x8) =
66560   {TCP:39, IPv4:8}

590520:32:48 8-2-2011   2591.7452755
InternetNameOfApacheServer  WindowsNameOfApacheServer
ICMPICMP:Destination Unreachable Message, Communication
Administratively Prohibited, 82.161.37.94   {IPv4:8}

590620:32:49 8-2-2011   2592.4183261192.168.178.1   
WindowsNameOfApacheServer
TCP TCP:Flags=..S., SrcPort=2119, DstPort=14013, PayloadLen=0,
Seq=4038649047, Ack=0, Win=5840 ( Negotiating scale factor 0x2 ) =
5840{TCP:1001, IPv4:4}

590720:32:51 8-2-2011   2594.2343496Unavailable 
WindowsNameOfApacheServer
InternetNameOfApacheServer  TCP TCP:Flags=...A...F, 
SrcPort=HTTP(80),
DstPort=3573, PayloadLen=0, Seq=4118333527, Ack=1169303037, Win=260
(scale factor 0x8) = 66560  {TCP:39, IPv4:8}

590820:32:51 8-2-2011   2594.2349872
InternetNameOfApacheServer  WindowsNameOfApacheServer
ICMPICMP:Destination Unreachable Message, Communication
Administratively Prohibited, 82.161.37.94   {IPv4:8}

590920:32:52 8-2-2011   2595.4057840192.168.178.1   
WindowsNameOfApacheServer
TCP

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

On Tue, Feb 8, 2011 at 8:54 PM, Tolas Anon  wrote:
> But in the meanwhile I found a new idea to try as well;
>
>        curl_setopt($ch, CURLOPT_HTTPHEADER, array(
>                        'Connection: Keep-Alive',
>                        'Keep-Alive: 300'
>        ));
>
> I already checked via phpinfo() that keep-alive is on in
> apache2handler, and no other mentions of "keepalive" or "keep alive"
> in the phpinfo() output.
>
> I'll post the results.
>

ehm

http://www.io.com/~maus/HttpKeepAlive.html :

HTTP/1.0

Under HTTP 1.0, there is no official specification for how keepalive
operates. It was, in essence, tacked on to an existing protocol. If
the browser supports keep-alive, it adds an additional header to the
request:
Connection: Keep-Alive

Then, when the server receives this request and generates a response,
it also adds a header to the response:
Connection: Keep-Alive

Following this, the connection is NOT dropped, but is instead kept
open. When the client sends another request, it uses the same
connection. This will continue until either the client or the server
decides that the conversation is over, and one of them drops the
connection.

-
HTTP/1.1

Under HTTP 1.1, the official keepalive method is different. All
connections are kept alive, unless stated otherwise with the following
header:
Connection: close

The Connection: Keep-Alive header no longer has any meaning because of this.
Additionally, an optional Keep-Alive: header is described, but is so
underspecified as to be meaningless. Avoid it.

-

And of course, my return data packet 5901 uses HTTP1.1, so the test
i'm running now probably won't fix things.. :(((

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

On Tue, Feb 8, 2011 at 9:33 PM, Steve Staples  wrote:
> i've been sorta reading this (as I am sure most maybe stopped after the
> 4th consecutive post)...

yea, i feel i gotta be complete so as to not waste the time of people
who do wanna help.

> but what I am wondering is...
>
> why can't you just write the output of the what you're doing to a file,
> or the db, and then query along the way or when you need/want some
> information on it??   Maybe i just haven't quite figured out, or got the
> gist of what you are trying to accomplish...

That's exactly what i'm doing.
I need curl_exec() to absorb the response to requests that take over
1hr to complete, because i want to be able to run different parts of
the media import process on different servers.

> it also seems to me, that this really isn't a PHP specific issue, so all
> the posts that you're doing, really doesn't pertain to the PHP mailing
> list, so (and sorry to say this) maybe stop posting all the incremental
> updates you're doing, and when there is a major break through, or
> someone has an idea on how to help solve your issue, update us.
>
> Steve.

I wish to have a complete log of this bughunt somewhere online, one of
the many websites with the php mailing list content will do nicely.

I suppose i could've chosen to subscribe to the libcurl mailinglist,
but this seemed a good place because i thought to find many people
that use libcurl in different ways here.
And at the start it could've been the php-cli, or the apache, or the
lib-curl, i just didn't know.

Even though lib-curl atm looks the more likely suspect, i'll continue
this log here.

If you don't like it, don't read it.
With a decent mail reader it is shoved under 1 header anyways.

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

On Tue, Feb 8, 2011 at 11:01 PM, Tolas Anon  wrote:
> Ok, another run, this time after adding the following to
> php_daemon_script;
>
>        curl_setopt($ch, CURLOPT_HTTPHEADER, array(
>                        'Connection: Keep-Alive',
>                        'Keep-Alive: 300'
>        ));
>
> Exec-sum:
> No change whatsoever in behaviour, curl_exec still freezes after
> processing the 2gb file completely. ~maus was right.
>
> Since i create the "folder" in the cms to hold the media-imported
> files before starting the import, i can view the offending 2gb video
> as an end-user of my cms. but files that were behind the 2gb video
> in the queue never get imported because curl_exec (or something
> feeding it) has frozen.
>
> It could be that apache sends the wrong packets, or libcurl rejects
> 'good' packets, or libcurl just froze completely. I might not have
> thought of all possibilities though, that's one of the reasons I
> post these mails.
>
> I've run out of ideas again..
> And since i've been at it since 05:00, i'll get some sleep now..
>
> opening bell of 2gb import, microsoft network monitor 3.4 again:
>
>
> 89268   21:48:23 8-2-2011       314.7693867     Unavailable-4920
>
> InternetDomainNameOfApacheServer        WindowsNameOfApacheServer
>
> HTTP    HTTP:Request, POST /site/cms/php/php_(import_)script.php,
>
> Query:PHPSESSID=bhgnqukbn4v0dbbv25i14ca2o4&action=batchNext
>
> {HTTP:112, TCP:110, IPv4:1} {{ALL GOOD DATA}}
>
>
> And the closing bell (the omissions are there because i let the
>
> sniffer app filter by process id -> ip-address) :
>
> 363     22:31:16 8-2-2011       676.1139552     Unavailable-4920
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    HTTP
> HTTP:Response, HTTP/1.1, Status: Ok, URL:{OMMITTED BECAUSE OF MEMORY
> EXHAUSTION->KILL OF SNIFFER APP}        {HTTP:31, TCP:30, IPv4:29}
> {{CONTAINS THE WANTED DATA}}
>
> 367     22:31:19 8-2-2011       679.1089487     Unavailable-4920
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    TCP
> TCP:[ReTransmit #363]Flags=...AP..., SrcPort=HTTP(80), DstPort=8372,
> PayloadLen=614, Seq=3326586000 - 3326586614, Ack=3291710338, Win=260
> {TCP:30, IPv4:29}
>
> 372     22:31:21 8-2-2011       681.6189851     Unavailable-4920
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    TCP
> TCP:Flags=...A...F, SrcPort=HTTP(80), DstPort=8372, PayloadLen=0,
> Seq=3326586614, Ack=3291710338, Win=260 {TCP:30, IPv4:29}
>
> 376     22:31:25 8-2-2011       685.1089219     Unavailable-4920
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    TCP
> TCP:[ReTransmit #363]Flags=...AP..F, SrcPort=HTTP(80), DstPort=8372,
> PayloadLen=614, Seq=3326586000 - 3326586615, Ack=3291710338, Win=260
> {TCP:30, IPv4:29}
>
> 384     22:31:37 8-2-2011       697.1089168     Unavailable-4920
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    TCP
> TCP:[ReTransmit #363]Flags=...AP..F, SrcPort=HTTP(80), DstPort=8372,
> PayloadLen=614, Seq=3326586000 - 3326586615, Ack=3291710338, Win=260
> {TCP:30, IPv4:29}
>
> 400     22:32:01 8-2-2011       721.1089938     Unavailable-4920
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    TCP
> TCP:[ReTransmit #363]Flags=...AP..F, SrcPort=HTTP(80), DstPort=8372,
> PayloadLen=614, Seq=3326586000 - 3326586615, Ack=3291710338, Win=260
> {TCP:30, IPv4:29}
>
> 431     22:32:49 8-2-2011       769.1090750     Unavailable-4920
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    TCP
> TCP:[ReTransmit #363]Flags=...AP..F, SrcPort=HTTP(80), DstPort=8372,
> PayloadLen=614, Seq=3326586000 - 3326586615, Ack=3291710338, Win=260
> {TCP:30, IPv4:29}
>
> 485     22:33:49 8-2-2011       829.1041446     Unavailable-4920
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    TCP
> TCP:Flags=...A.R.., SrcPort=HTTP(80), DstPort=8372, PayloadLen=0,
> Seq=3326586615, Ack=3291710338, Win=0   {TCP:30, IPv4:29}
>
> 550     22:35:38 8-2-2011       938.7487524     Unavailable-676
> WindowsNameOfApacheServer       InternetDomainOfApacheServer    TCP
> TCP:Flags=...A.R.., SrcPort=8372, DstPort=HTTP(80), PayloadLen=0,
> Seq=3291710338, Ack=3326586000, Win=0   {TCP:59, IPv4:29}
>
> 551     22:35:38 8-2-2011       938.7514250     Unavailable-4920
> InternetDomainOfApacheServer    WindowsNameOfApacheServer       TCP
> TCP:Flags=...A.R.., SrcPort=8372, DstPort=HTTP(80), PayloadLen=0,
> Seq=3291710338, Ack=3326586000, Win=0   {TCP:60, IPv4:29}
>
>
> Please note that packet 550 was apparently from a different cpu
> process.
>

oh, packet number 363 has HTTP/1.1 as the protocol version..

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

Ok, another run, this time after adding the following to
php_daemon_script;

curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Connection: Keep-Alive',
'Keep-Alive: 300'
));

Exec-sum:
No change whatsoever in behaviour, curl_exec still freezes after
processing the 2gb file completely. ~maus was right.

Since i create the "folder" in the cms to hold the media-imported
files before starting the import, i can view the offending 2gb video
as an end-user of my cms. but files that were behind the 2gb video
in the queue never get imported because curl_exec (or something
feeding it) has frozen.

It could be that apache sends the wrong packets, or libcurl rejects
'good' packets, or libcurl just froze completely. I might not have
thought of all possibilities though, that's one of the reasons I
post these mails.

I've run out of ideas again..
And since i've been at it since 05:00, i'll get some sleep now..

opening bell of 2gb import, microsoft network monitor 3.4 again:


89268   21:48:23 8-2-2011   314.7693867 Unavailable-4920

InternetDomainNameOfApacheServerWindowsNameOfApacheServer   

HTTPHTTP:Request, POST /site/cms/php/php_(import_)script.php,

Query:PHPSESSID=bhgnqukbn4v0dbbv25i14ca2o4&action=batchNext 

{HTTP:112, TCP:110, IPv4:1} {{ALL GOOD DATA}}


And the closing bell (the omissions are there because i let the

sniffer app filter by process id -> ip-address) :

363 22:31:16 8-2-2011   676.1139552 Unavailable-4920
WindowsNameOfApacheServer   InternetDomainOfApacheServerHTTP
HTTP:Response, HTTP/1.1, Status: Ok, URL:{OMMITTED BECAUSE OF MEMORY
EXHAUSTION->KILL OF SNIFFER APP}{HTTP:31, TCP:30, IPv4:29}
{{CONTAINS THE WANTED DATA}}

367 22:31:19 8-2-2011   679.1089487 Unavailable-4920
WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 
TCP:[ReTransmit #363]Flags=...AP..., SrcPort=HTTP(80), DstPort=8372,
PayloadLen=614, Seq=3326586000 - 3326586614, Ack=3291710338, Win=260
{TCP:30, IPv4:29}

372 22:31:21 8-2-2011   681.6189851 Unavailable-4920
WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 
TCP:Flags=...A...F, SrcPort=HTTP(80), DstPort=8372, PayloadLen=0,
Seq=3326586614, Ack=3291710338, Win=260 {TCP:30, IPv4:29}

376 22:31:25 8-2-2011   685.1089219 Unavailable-4920
WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 
TCP:[ReTransmit #363]Flags=...AP..F, SrcPort=HTTP(80), DstPort=8372,
PayloadLen=614, Seq=3326586000 - 3326586615, Ack=3291710338, Win=260
{TCP:30, IPv4:29}

384 22:31:37 8-2-2011   697.1089168 Unavailable-4920
WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 
TCP:[ReTransmit #363]Flags=...AP..F, SrcPort=HTTP(80), DstPort=8372,
PayloadLen=614, Seq=3326586000 - 3326586615, Ack=3291710338, Win=260
{TCP:30, IPv4:29}

400 22:32:01 8-2-2011   721.1089938 Unavailable-4920
WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 
TCP:[ReTransmit #363]Flags=...AP..F, SrcPort=HTTP(80), DstPort=8372,
PayloadLen=614, Seq=3326586000 - 3326586615, Ack=3291710338, Win=260
{TCP:30, IPv4:29}

431 22:32:49 8-2-2011   769.1090750 Unavailable-4920
WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 
TCP:[ReTransmit #363]Flags=...AP..F, SrcPort=HTTP(80), DstPort=8372,
PayloadLen=614, Seq=3326586000 - 3326586615, Ack=3291710338, Win=260
{TCP:30, IPv4:29}

485 22:33:49 8-2-2011   829.1041446 Unavailable-4920
WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 
TCP:Flags=...A.R.., SrcPort=HTTP(80), DstPort=8372, PayloadLen=0,
Seq=3326586615, Ack=3291710338, Win=0   {TCP:30, IPv4:29}

550 22:35:38 8-2-2011   938.7487524 Unavailable-676 
WindowsNameOfApacheServer   InternetDomainOfApacheServerTCP 
TCP:Flags=...A.R.., SrcPort=8372, DstPort=HTTP(80), PayloadLen=0,
Seq=3291710338, Ack=3326586000, Win=0   {TCP:59, IPv4:29}

551 22:35:38 8-2-2011   938.7514250 Unavailable-4920
InternetDomainOfApacheServerWindowsNameOfApacheServer   TCP 
TCP:Flags=...A.R.., SrcPort=8372, DstPort=HTTP(80), PayloadLen=0,
Seq=3291710338, Ack=3326586000, Win=0   {TCP:60, IPv4:29}


Please note that packet 550 was apparently from a different cpu
process.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

The one thing that strikes me as odd, before i go, is that i saw no
actual HTTP-KEEPALIVE traffic flowing... I might have missed it, but i
don't think so..

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

exec-sum update :

details continued at http://curl.haxx.se/mail/lib-2011-02/0101.html

it turns out using the internetdomainname of the apache server is the
source of the problem.
the adsl router could be dropping the connection, or something else.

calling curl_exec() to a localhost address will work fine for the time
that it takes.
(no traffic back for 1hr, then a short status msg only)

however my media import routines are useless if i only can use 'm on localhost.

so the idea i will try tommorow is outputting keepalive weird-bytes on
the text/html level
from the php_script import-script, and then stripping them in php_daemon_script.

i would like to know if i can use feof() to check that ffmpeg launched
via popen() has terminated..
i think so, but the docs aren't perfectly clear.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] ob_flush, does it cause data to be sent over the wire immediately?

[Tolas Anon]

Hi..

For my curl_exec problem,
http://readlist.com/lists/lists.php.net/php-general/16/81195.html
http://curl.haxx.se/mail/lib-2011-02/0101.html

I need to know if this simple test will cause data to be sent every 25
seconds _over the wire_.



reason; i just can't get my media import routines that only send
sporadic status messages to work for big video files.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Re: ob_flush, does it cause data to be sent over the wire immediately?

[Tolas Anon]

On Fri, Feb 11, 2011 at 4:05 AM, Tolas Anon  wrote:
> Hi..
>
> For my curl_exec problem,
> http://readlist.com/lists/lists.php.net/php-general/16/81195.html
> http://curl.haxx.se/mail/lib-2011-02/0101.html
>
> I need to know if this simple test will cause data to be sent every 25
> seconds _over the wire_.
>
>  set_time_limit (0);
> error_reporting (E_ALL);
> for ($i=0; $i<((3600+1800)/25); $i++) {
>        echo '.';
>        ob_flush();
>        sleep (25);
> };
> echo "wanted-data";
> ?>
>
> reason; i just can't get my media import routines that only send
> sporadic status messages to work for big video files.
>

well, first off i should be using php:flush() instead of
php:ob_flush() in my simple test.

second;
http://www.php.net/manual/en/function.flush.php :


Several servers, especially on Win32, will still buffer the output
from your script until it terminates before transmitting the results
to the browser.
.


aargh..

well, i'll have to dig deep here.

will inform you of any relevant progress, digest style

--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] [SOLVED] Re: [PHP] curl_exec won't return (any data)

[Tolas Anon]

fromTolas Anon 
to  libcurl development 
dateFri, Feb 11, 2011 at 5:09 PM
subject Re: php curl_exec won't return any data, libcurl-7.21.3.0, php
5.3.4, WampServer2.1d-x64.exe
mailed-by   gmail.com

On Fri, Feb 11, 2011 at 1:32 PM, Daniel Stenberg  wrote:
> On Fri, 11 Feb 2011, Tolas Anon wrote:
>
>>> It is also very easy for an application to enable the options as I've
>>> shown.
>>
>> Just not for the platforms i use, apparently.. :(
>
> Why not? A quick search for "tcp keepalive windows" shows this:
> http://msdn.microsoft.com/en-us/library/ms819735.aspx


BUYAKAA! This fixed my problem in the simple test AND in my application!
No need to send keep-alive bytes on the text/html level either.

I just added the "KeepAliveTime" setting with windows 7 regedit.exe in
HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/services/Tcpip, as a
REG_DWORD value, set it to decimal 25000 (so 25 seconds), rebooted,
and it all works as i want it now..

Note that windows registry entries key names are case-sensitive, wrong
casing and they'll be deleted on restart.

Lots of thanks, Daniel. I would've never solved this on my own..

I'll make an entry in the php.net website "comments" about this.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] session_start() may take 5 seconds :(

[Tolas Anon]

My web-app sometimes takes just over 5 seconds to execute a single
start_session() statement.
No other connections are open to the site when i hit the button that
makes this happen, the session is just 78kb on a local disk, and it's
consistent behavior.
So far, it only happens when i request certain page content via AJAX,
but I'd like to get rid of it asap.
Strangely, some other page content requested under the same conditions
via the same pipeline returns without the delay. It doesn't do
anything differently except the business code, and the session is
opened in app-wide generic code that is executed before the business
code is executed. The very timing with microtime() happens before the
business code executes.

I've already googled, but haven't found anything useful.

It's on a windows box (latest wampserver) and due to a ubuntu firefox
limitation i can't test on linux at the moment, nor run strace on it
:(

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] session_start() may take 5 seconds :(

[Tolas Anon]

On Thu, Mar 31, 2011 at 8:10 AM, Negin Nickparsa  wrote:
> session1.php:
>  session_start();
> $_SESSION['s1']='sth';
> header("location: session2.php?".SID);
> ?>
> session2.php:
>  session_start();
> if(isset($_SESSION['s1']))
> echo $_SESSION['s1'];
> else
> die("No session!");
> ?>
>
>

ok, that executes with success and fast..

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Re: session_start() may take 5 seconds :(

[Tolas Anon]

On Thu, Mar 31, 2011 at 7:58 AM, Tolas Anon  wrote:
> My web-app sometimes takes just over 5 seconds to execute a single
> start_session() statement.
> No other connections are open to the site when i hit the button that
> makes this happen, the session is just 78kb on a local disk, and it's
> consistent behavior.
> So far, it only happens when i request certain page content via AJAX,
> but I'd like to get rid of it asap.
> Strangely, some other page content requested under the same conditions
> via the same pipeline returns without the delay. It doesn't do
> anything differently except the business code, and the session is
> opened in app-wide generic code that is executed before the business
> code is executed. The very timing with microtime() happens before the
> business code executes.
>
> I've already googled, but haven't found anything useful.
>
> It's on a windows box (latest wampserver) and due to a ubuntu firefox
> limitation i can't test on linux at the moment, nor run strace on it
> :(
>

This afternoon, all my business logic had the same delay, due to
session_start().

I bypassed the ubuntu firefox problem and was able to test under
linux; no delay.

So i'll mail the wampserver admin of this problem, and continue under ubuntu.

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

[PHP] Re: session_start() may take 5 seconds :(

[Tolas Anon]

On Thu, Mar 31, 2011 at 4:53 PM, Tolas Anon  wrote:
> On Thu, Mar 31, 2011 at 7:58 AM, Tolas Anon  wrote:
>> My web-app sometimes takes just over 5 seconds to execute a single
>> start_session() statement.
>> No other connections are open to the site when i hit the button that
>> makes this happen, the session is just 78kb on a local disk, and it's
>> consistent behavior.
>> So far, it only happens when i request certain page content via AJAX,
>> but I'd like to get rid of it asap.
>> Strangely, some other page content requested under the same conditions
>> via the same pipeline returns without the delay. It doesn't do
>> anything differently except the business code, and the session is
>> opened in app-wide generic code that is executed before the business
>> code is executed. The very timing with microtime() happens before the
>> business code executes.
>>
>> I've already googled, but haven't found anything useful.
>>
>> It's on a windows box (latest wampserver) and due to a ubuntu firefox
>> limitation i can't test on linux at the moment, nor run strace on it
>> :(
>>
>
> This afternoon, all my business logic had the same delay, due to
> session_start().
>
> I bypassed the ubuntu firefox problem and was able to test under
> linux; no delay.
>
> So i'll mail the wampserver admin of this problem, and continue under ubuntu.
>
http://www.wampserver.com/phorum/read.php?2,72729

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php

Re: [PHP] upload big files

[Tolas Anon]

On Thu, Apr 14, 2011 at 10:36 AM, Ricardo Martinez  wrote:
> Hi List!
>
> Need can upload big files ( around 300MB ) the system must be easy of use (
> cann't use ftp ).
>
> Was thinking in make it, using a form with post, and update the cfg files
> for can upload big files.
> But don't know how it going to work. I think that can give a lot of errors
> or be unestable.
>
> Just asking if anyone had a project like this one ;)
>
> Greetings!
>

I've got good experiences with http://jumploader.com
It's Java, and can do HTTP uploads > 2Gb fine, with the partitioned php uploader

-- 
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php