[R-pkg-devel] socketConnection, delay when reading from
Hi,
I have been working on a R-client for the BaseX XML-database and version
0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
Version 0.3 of RBaseX can be found here
(https://cran.microsoft.com/web/packages/RBaseX/index.html).
The client-server protocol specifies that the communication between the
client and the database is based on a socket. The code (below) shows how
I create that socket.
Writing to the socket works perfect. Reading from the sockets (see
second codeblock) also produces correct results. The problem however is
that the timeout, as specified when initializing the socket, causes a 1
second delay for every read-operation.
I have experimented a lot with different settings and have been
searching a lot on internet, but I can't find any method to get rid of
that delay. (In C or C++ that should be easier but I have never before
had any need to use those languages).
The very first version of my client used a block-size of 1 when reading.
That gave acceptable response times for small query-results but reading
large responses from the database took very long time.
Do you have any suggestions on how to cope with this problem?
Ben Engbers
-
CreateSocket = function(host, port = 1984L, username, password) {
tryCatch(
{conn <- private$conn <- socketConnection(
host = "localhost", port,
open = "w+b", server = FALSE, blocking = TRUE, encoding =
"UTF-8", timeout = 1)
}, error = function(e) {
stop("Cannot open the connection")
}
)
-
readBin_ <- function(conn) {
chars_read <- raw(0)
rd <- readBin(conn, what = "raw", 1024)
while(length(rd) == 1024) {
chars_read <- c(chars_read, rd)
rd <- readBin(conn, "raw", 1024)
}
if (length(rd) > 0) chars_read <- c(chars_read, rd)
return(chars_read)
}
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
On 27/11/2021 4:09 a.m., Ben Engbers wrote:
Hi,
I have been working on a R-client for the BaseX XML-database and version
0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
Version 0.3 of RBaseX can be found here
(https://cran.microsoft.com/web/packages/RBaseX/index.html).
The client-server protocol specifies that the communication between the
client and the database is based on a socket. The code (below) shows how
I create that socket.
Writing to the socket works perfect. Reading from the sockets (see
second codeblock) also produces correct results. The problem however is
that the timeout, as specified when initializing the socket, causes a 1
second delay for every read-operation.
I have experimented a lot with different settings and have been
searching a lot on internet, but I can't find any method to get rid of
that delay. (In C or C++ that should be easier but I have never before
had any need to use those languages).
The very first version of my client used a block-size of 1 when reading.
That gave acceptable response times for small query-results but reading
large responses from the database took very long time.
Do you have any suggestions on how to cope with this problem?
You are attempting to read 1024 bytes. If the socket returns fewer, the
code presumably assumes it should wait for more. You should specify the
number to read to be no larger than you'll receive. That's why reading
a byte at a time works well for small reads.
To handle larger ones, you need to know what size blocks you are going
to receive with each read. If that varies unpredictably, I doubt if
there's much you can do other than reducing the timeout.
Duncan Murdoch
Ben Engbers
-
CreateSocket = function(host, port = 1984L, username, password) {
tryCatch(
{conn <- private$conn <- socketConnection(
host = "localhost", port,
open = "w+b", server = FALSE, blocking = TRUE, encoding =
"UTF-8", timeout = 1)
}, error = function(e) {
stop("Cannot open the connection")
}
)
-
readBin_ <- function(conn) {
chars_read <- raw(0)
rd <- readBin(conn, what = "raw", 1024)
while(length(rd) == 1024) {
chars_read <- c(chars_read, rd)
rd <- readBin(conn, "raw", 1024)
}
if (length(rd) > 0) chars_read <- c(chars_read, rd)
return(chars_read)
}
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
That's what I figured out also😉. The problem is that even when I reduce the
size to 2 bytes and there is only 1byte available in the socket, the code will
wait for 1 second which is the minimum time-out😔.BenVerzonden vanaf mijn Galaxy
Oorspronkelijk bericht Van: Duncan Murdoch
Datum: 27-11-2021 12:02 (GMT+01:00) Aan:
ben.engb...@be-logical.nl, r-package-devel@r-project.org Onderwerp: Re:
[R-pkg-devel] socketConnection, delay when reading from On 27/11/2021 4:09
a.m., Ben Engbers wrote:> > Hi,> > I have been working on a R-client for the
BaseX XML-database and version> 0.9.2 is nearly finished (submitting version
0.9.0 was rejected by CRAN).> Version 0.3 of RBaseX can be found here>
(https://cran.microsoft.com/web/packages/RBaseX/index.html).> > The
client-server protocol specifies that the communication between the> client and
the database is based on a socket. The code (below) shows how> I create that
socket.> > Writing to the socket works perfect. Reading from the sockets (see>
second codeblock) also produces correct results. The problem however is> that
the timeout, as specified when initializing the socket, causes a 1> second
delay for every read-operation.> I have experimented a lot with different
settings and have been> searching a lot on internet, but I can't find any
method to get rid of> that delay. (In C or C++ that should be easier but I have
never before> had any need to use those languages).> The very first version of
my client used a block-size of 1 when reading.> That gave acceptable response
times for small query-results but reading> large responses from the database
took very long time.> > Do you have any suggestions on how to cope with this
problem?You are attempting to read 1024 bytes. If the socket returns fewer,
the code presumably assumes it should wait for more. You should specify the
number to read to be no larger than you'll receive. That's why reading a byte
at a time works well for small reads.To handle larger ones, you need to know
what size blocks you are going to receive with each read. If that varies
unpredictably, I doubt if there's much you can do other than reducing the
timeout.Duncan Murdoch> > Ben Engbers> > ->Â Â
CreateSocket = function(host, port = 1984L, username, password) {>
tryCatch(>Â Â {conn <- private$conn <- socketConnection(>
host = "localhost", port,> open = "w+b", server = FALSE, blocking =
TRUE, encoding => "UTF-8", timeout = 1)>Â Â }, error = function(e) {>
stop("Cannot open the connection")>Â Â }> )> >
-> > readBin_ <- function(conn) {> chars_read
<- raw(0)> rd <- readBin(conn, what = "raw", 1024)> while(length(rd) ==
1024) {>Â Â chars_read <- c(chars_read, rd)>Â Â rd <- readBin(conn,
"raw", 1024)>Â Â }> if (length(rd) > 0) chars_read <- c(chars_read, rd)>Â
   return(chars_read)> }> > __>
R-package-devel@r-project.org mailing list>
https://stat.ethz.ch/mailman/listinfo/r-package-devel>
[[alternative HTML version deleted]]
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
Does the message start with a length or a command whose argument length is known
depending on the particular command?
If so first read the length or command and from that the length of the
remainder of
the message can be determined.
On Sat, Nov 27, 2021 at 4:09 AM Ben Engbers wrote:
>
>
> Hi,
>
> I have been working on a R-client for the BaseX XML-database and version
> 0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
> Version 0.3 of RBaseX can be found here
> (https://cran.microsoft.com/web/packages/RBaseX/index.html).
>
> The client-server protocol specifies that the communication between the
> client and the database is based on a socket. The code (below) shows how
> I create that socket.
>
> Writing to the socket works perfect. Reading from the sockets (see
> second codeblock) also produces correct results. The problem however is
> that the timeout, as specified when initializing the socket, causes a 1
> second delay for every read-operation.
> I have experimented a lot with different settings and have been
> searching a lot on internet, but I can't find any method to get rid of
> that delay. (In C or C++ that should be easier but I have never before
> had any need to use those languages).
> The very first version of my client used a block-size of 1 when reading.
> That gave acceptable response times for small query-results but reading
> large responses from the database took very long time.
>
> Do you have any suggestions on how to cope with this problem?
>
> Ben Engbers
>
> -
> CreateSocket = function(host, port = 1984L, username, password) {
>tryCatch(
> {conn <- private$conn <- socketConnection(
>host = "localhost", port,
>open = "w+b", server = FALSE, blocking = TRUE, encoding =
> "UTF-8", timeout = 1)
> }, error = function(e) {
>stop("Cannot open the connection")
> }
>)
>
> -
>
> readBin_ <- function(conn) {
>chars_read <- raw(0)
>rd <- readBin(conn, what = "raw", 1024)
>while(length(rd) == 1024) {
> chars_read <- c(chars_read, rd)
> rd <- readBin(conn, "raw", 1024)
> }
>if (length(rd) > 0) chars_read <- c(chars_read, rd)
>return(chars_read)
> }
>
> __
> R-package-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
--
Statistics & Software Consulting
GKX Group, GKX Associates Inc.
tel: 1-877-GKX-GROUP
email: ggrothendieck at gmail.com
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
No, according to the specification the minimal number of bytes that is
returned is 2. There is no maximum. (When querying a database you never
know on forehand how many records match the query so by definition you
can't calculate the size of the message).
In some C, C++ or Java-code I found on internet, it was possible to
change the timeout settings so that there would be no delay. Of course
this would have as consequence that in your code you have to deal with
the possibility that the message has not been completely returned.
In R you can set the timeout to 0 but that results in errors (at least
on Windows)
Op 27-11-2021 om 14:57 schreef Gabor Grothendieck:
Does the message start with a length or a command whose argument length is known
depending on the particular command?
If so first read the length or command and from that the length of the
remainder of
the message can be determined.
On Sat, Nov 27, 2021 at 4:09 AM Ben Engbers wrote:
Hi,
I have been working on a R-client for the BaseX XML-database and version
0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
Version 0.3 of RBaseX can be found here
(https://cran.microsoft.com/web/packages/RBaseX/index.html).
The client-server protocol specifies that the communication between the
client and the database is based on a socket. The code (below) shows how
I create that socket.
Writing to the socket works perfect. Reading from the sockets (see
second codeblock) also produces correct results. The problem however is
that the timeout, as specified when initializing the socket, causes a 1
second delay for every read-operation.
I have experimented a lot with different settings and have been
searching a lot on internet, but I can't find any method to get rid of
that delay. (In C or C++ that should be easier but I have never before
had any need to use those languages).
The very first version of my client used a block-size of 1 when reading.
That gave acceptable response times for small query-results but reading
large responses from the database took very long time.
Do you have any suggestions on how to cope with this problem?
Ben Engbers
-
CreateSocket = function(host, port = 1984L, username, password) {
tryCatch(
{conn <- private$conn <- socketConnection(
host = "localhost", port,
open = "w+b", server = FALSE, blocking = TRUE, encoding =
"UTF-8", timeout = 1)
}, error = function(e) {
stop("Cannot open the connection")
}
)
-
readBin_ <- function(conn) {
chars_read <- raw(0)
rd <- readBin(conn, what = "raw", 1024)
while(length(rd) == 1024) {
chars_read <- c(chars_read, rd)
rd <- readBin(conn, "raw", 1024)
}
if (length(rd) > 0) chars_read <- c(chars_read, rd)
return(chars_read)
}
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
Whether the length is variable or not isn't relevant. The point is
whether the message is prefaced by a length or command from which the
length can be derived. Maybe it is not and you will have to rely on
inefficient methods but in many cases protocols are designed to avoid
such problems.
On Sat, Nov 27, 2021 at 9:40 AM Ben Engbers wrote:
>
> No, according to the specification the minimal number of bytes that is
> returned is 2. There is no maximum. (When querying a database you never
> know on forehand how many records match the query so by definition you
> can't calculate the size of the message).
>
> In some C, C++ or Java-code I found on internet, it was possible to
> change the timeout settings so that there would be no delay. Of course
> this would have as consequence that in your code you have to deal with
> the possibility that the message has not been completely returned.
>
> In R you can set the timeout to 0 but that results in errors (at least
> on Windows)
>
> Op 27-11-2021 om 14:57 schreef Gabor Grothendieck:
> > Does the message start with a length or a command whose argument length is
> > known
> > depending on the particular command?
> > If so first read the length or command and from that the length of the
> > remainder of
> > the message can be determined.
> >
> > On Sat, Nov 27, 2021 at 4:09 AM Ben Engbers
> > wrote:
> >>
> >>
> >> Hi,
> >>
> >> I have been working on a R-client for the BaseX XML-database and version
> >> 0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
> >> Version 0.3 of RBaseX can be found here
> >> (https://cran.microsoft.com/web/packages/RBaseX/index.html).
> >>
> >> The client-server protocol specifies that the communication between the
> >> client and the database is based on a socket. The code (below) shows how
> >> I create that socket.
> >>
> >> Writing to the socket works perfect. Reading from the sockets (see
> >> second codeblock) also produces correct results. The problem however is
> >> that the timeout, as specified when initializing the socket, causes a 1
> >> second delay for every read-operation.
> >> I have experimented a lot with different settings and have been
> >> searching a lot on internet, but I can't find any method to get rid of
> >> that delay. (In C or C++ that should be easier but I have never before
> >> had any need to use those languages).
> >> The very first version of my client used a block-size of 1 when reading.
> >> That gave acceptable response times for small query-results but reading
> >> large responses from the database took very long time.
> >>
> >> Do you have any suggestions on how to cope with this problem?
> >>
> >> Ben Engbers
> >>
> >> -
> >> CreateSocket = function(host, port = 1984L, username, password) {
> >> tryCatch(
> >> {conn <- private$conn <- socketConnection(
> >> host = "localhost", port,
> >> open = "w+b", server = FALSE, blocking = TRUE, encoding =
> >> "UTF-8", timeout = 1)
> >> }, error = function(e) {
> >> stop("Cannot open the connection")
> >> }
> >> )
> >>
> >> -
> >>
> >> readBin_ <- function(conn) {
> >> chars_read <- raw(0)
> >> rd <- readBin(conn, what = "raw", 1024)
> >> while(length(rd) == 1024) {
> >> chars_read <- c(chars_read, rd)
> >> rd <- readBin(conn, "raw", 1024)
> >> }
> >> if (length(rd) > 0) chars_read <- c(chars_read, rd)
> >> return(chars_read)
> >> }
> >>
> >> __
> >> R-package-devel@r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> >
> >
> >
>
--
Statistics & Software Consulting
GKX Group, GKX Associates Inc.
tel: 1-877-GKX-GROUP
email: ggrothendieck at gmail.com
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
This is a null-terminated message protocol [1]. It has to be processed one byte
at a time.
[1] https://docs.basex.org/wiki/Server_Protocol
On November 27, 2021 7:45:31 AM PST, Gabor Grothendieck
wrote:
>Whether the length is variable or not isn't relevant. The point is
>whether the message is prefaced by a length or command from which the
>length can be derived. Maybe it is not and you will have to rely on
>inefficient methods but in many cases protocols are designed to avoid
>such problems.
>
>On Sat, Nov 27, 2021 at 9:40 AM Ben Engbers wrote:
>>
>> No, according to the specification the minimal number of bytes that is
>> returned is 2. There is no maximum. (When querying a database you never
>> know on forehand how many records match the query so by definition you
>> can't calculate the size of the message).
>>
>> In some C, C++ or Java-code I found on internet, it was possible to
>> change the timeout settings so that there would be no delay. Of course
>> this would have as consequence that in your code you have to deal with
>> the possibility that the message has not been completely returned.
>>
>> In R you can set the timeout to 0 but that results in errors (at least
>> on Windows)
>>
>> Op 27-11-2021 om 14:57 schreef Gabor Grothendieck:
>> > Does the message start with a length or a command whose argument length is
>> > known
>> > depending on the particular command?
>> > If so first read the length or command and from that the length of the
>> > remainder of
>> > the message can be determined.
>> >
>> > On Sat, Nov 27, 2021 at 4:09 AM Ben Engbers
>> > wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I have been working on a R-client for the BaseX XML-database and version
>> >> 0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
>> >> Version 0.3 of RBaseX can be found here
>> >> (https://cran.microsoft.com/web/packages/RBaseX/index.html).
>> >>
>> >> The client-server protocol specifies that the communication between the
>> >> client and the database is based on a socket. The code (below) shows how
>> >> I create that socket.
>> >>
>> >> Writing to the socket works perfect. Reading from the sockets (see
>> >> second codeblock) also produces correct results. The problem however is
>> >> that the timeout, as specified when initializing the socket, causes a 1
>> >> second delay for every read-operation.
>> >> I have experimented a lot with different settings and have been
>> >> searching a lot on internet, but I can't find any method to get rid of
>> >> that delay. (In C or C++ that should be easier but I have never before
>> >> had any need to use those languages).
>> >> The very first version of my client used a block-size of 1 when reading.
>> >> That gave acceptable response times for small query-results but reading
>> >> large responses from the database took very long time.
>> >>
>> >> Do you have any suggestions on how to cope with this problem?
>> >>
>> >> Ben Engbers
>> >>
>> >> -
>> >> CreateSocket = function(host, port = 1984L, username, password) {
>> >> tryCatch(
>> >> {conn <- private$conn <- socketConnection(
>> >> host = "localhost", port,
>> >> open = "w+b", server = FALSE, blocking = TRUE, encoding =
>> >> "UTF-8", timeout = 1)
>> >> }, error = function(e) {
>> >> stop("Cannot open the connection")
>> >> }
>> >> )
>> >>
>> >> -
>> >>
>> >> readBin_ <- function(conn) {
>> >> chars_read <- raw(0)
>> >> rd <- readBin(conn, what = "raw", 1024)
>> >> while(length(rd) == 1024) {
>> >> chars_read <- c(chars_read, rd)
>> >> rd <- readBin(conn, "raw", 1024)
>> >> }
>> >> if (length(rd) > 0) chars_read <- c(chars_read, rd)
>> >> return(chars_read)
>> >> }
>> >>
>> >> __
>> >> R-package-devel@r-project.org mailing list
>> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>> >
>> >
>> >
>>
>
>
--
Sent from my phone. Please excuse my brevity.
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
Op 27-11-2021 om 17:03 schreef Jeff Newmiller: This is a null-terminated message protocol [1]. It has to be processed one byte at a time. [1] https://docs.basex.org/wiki/Server_Protocol The message may contain embedded 0x00's. To distinguish these embedded 0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's and 0xFFare prefixed with a 0xFF byte. This means that when you process one byte at a time you have to perform a check on every byte. This results in totally unacceptable response times (My first version of this client was based on this approach) The only alternative solution I can think off is to use C++ to create a socket and a function that reads from the socket. But since I have hardly any experience with C++ programming nor using the rcpp package Ben __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
Maybe time to learn it. At least to assemble complete messages. That said, the design of this protocol is intrinsically inefficient. Maybe they will upgrade if the software gets popular. On November 27, 2021 8:24:36 AM PST, Ben Engbers wrote: >Op 27-11-2021 om 17:03 schreef Jeff Newmiller: >> This is a null-terminated message protocol [1]. It has to be processed one >> byte at a time. >> >> [1] https://docs.basex.org/wiki/Server_Protocol >> >The message may contain embedded 0x00's. To distinguish these embedded >0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's and 0xFFare >prefixed with a 0xFF byte. This means that when you process one byte at >a time you have to perform a check on every byte. This results in >totally unacceptable response times (My first version of this client was >based on this approach) > >The only alternative solution I can think off is to use C++ to create a >socket and a function that reads from the socket. But since I have >hardly any experience with C++ programming nor using the rcpp package > >Ben -- Sent from my phone. Please excuse my brevity. __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
On 11/27/21 5:24 PM, Ben Engbers wrote:
Op 27-11-2021 om 17:03 schreef Jeff Newmiller:
This is a null-terminated message protocol [1]. It has to be
processed one byte at a time.
[1] https://docs.basex.org/wiki/Server_Protocol
The message may contain embedded 0x00's. To distinguish these embedded
0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's and
0xFFare prefixed with a 0xFF byte. This means that when you process
one byte at a time you have to perform a check on every byte. This
results in totally unacceptable response times (My first version of
this client was based on this approach)
The only alternative solution I can think off is to use C++ to create
a socket and a function that reads from the socket. But since I have
hardly any experience with C++ programming nor using the rcpp package
I think you could use non-blocking read. Read what is available in
chunks (e.g. up to 1024 bytes long). And based on what is read already,
figure out whether it is all data or not. Something along the lines as
the demo below. The demo, though, is polling too much (re-running
readBin to only find out no data is available yet). It should be
possible to improve with socketSelect().
Best
Tomas
# the client
con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0
done <- function(n) {
 n >= 1e8
}
while(!done(total)) {
  r <- readBin(con2, "raw", 1024)
  total <- total + length(r)
  cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)
# the server
n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE,
open="a+b")
cat("Connected...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)
Ben
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
On 11/27/21 8:05 PM, Tomas Kalibera wrote:
On 11/27/21 5:24 PM, Ben Engbers wrote:
Op 27-11-2021 om 17:03 schreef Jeff Newmiller:
This is a null-terminated message protocol [1]. It has to be
processed one byte at a time.
[1] https://docs.basex.org/wiki/Server_Protocol
The message may contain embedded 0x00's. To distinguish these
embedded 0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's
and 0xFFare prefixed with a 0xFF byte. This means that when you
process one byte at a time you have to perform a check on every byte.
This results in totally unacceptable response times (My first version
of this client was based on this approach)
The only alternative solution I can think off is to use C++ to create
a socket and a function that reads from the socket. But since I have
hardly any experience with C++ programming nor using the rcpp
package
I think you could use non-blocking read. Read what is available in
chunks (e.g. up to 1024 bytes long). And based on what is read
already, figure out whether it is all data or not. Something along the
lines as the demo below. The demo, though, is polling too much
(re-running readBin to only find out no data is available yet). It
should be possible to improve with socketSelect().
This is an extended demo with socketSelect() used to wait on the client
for some data to be available, to avoid consuming too much CPU by
polling. To be pasted into two R sessions running on the same computer.
You would have to replace the function done() with something figuring
out from the data whether it is complete or not, based on the protocol.
Best
Tomas
# the client
con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0
done <- function(n) {
 n >= 2e8
}
while(!done(total)) {
  cat("Waiting for data to become ready...\n")
  socketSelect(list(con2))
  cat("Reading data...\n")
  r <- readBin(con2, "raw", 1024)
  total <- total + length(r)
  cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)
# the server
n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE,
open="a+b")
cat("Connected...\n")
writeBin(w, con1)
cat("Sent data the first time, sleeping...\n")
Sys.sleep(10)
cat("Sending data the second time...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)
Best
Tomas
# the client
con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0
done <- function(n) {
 n >= 1e8
}
while(!done(total)) {
  r <- readBin(con2, "raw", 1024)
  total <- total + length(r)
  cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)
# the server
n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE,
open="a+b")
cat("Connected...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)
Ben
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
I'll first finish the current development but then I will see if your
suggestion is usefull.
Ben
Op 27-11-2021 om 20:19 schreef Tomas Kalibera:
On 11/27/21 8:05 PM, Tomas Kalibera wrote:
On 11/27/21 5:24 PM, Ben Engbers wrote:
Op 27-11-2021 om 17:03 schreef Jeff Newmiller:
This is a null-terminated message protocol [1]. It has to be
processed one byte at a time.
[1] https://docs.basex.org/wiki/Server_Protocol
The message may contain embedded 0x00's. To distinguish these
embedded 0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's
and 0xFFare prefixed with a 0xFF byte. This means that when you
process one byte at a time you have to perform a check on every byte.
This results in totally unacceptable response times (My first version
of this client was based on this approach)
The only alternative solution I can think off is to use C++ to create
a socket and a function that reads from the socket. But since I have
hardly any experience with C++ programming nor using the rcpp
package
I think you could use non-blocking read. Read what is available in
chunks (e.g. up to 1024 bytes long). And based on what is read
already, figure out whether it is all data or not. Something along the
lines as the demo below. The demo, though, is polling too much
(re-running readBin to only find out no data is available yet). It
should be possible to improve with socketSelect().
This is an extended demo with socketSelect() used to wait on the client
for some data to be available, to avoid consuming too much CPU by
polling. To be pasted into two R sessions running on the same computer.
You would have to replace the function done() with something figuring
out from the data whether it is complete or not, based on the protocol.
Best
Tomas
# the client
con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0
done <- function(n) {
 n >= 2e8
}
while(!done(total)) {
  cat("Waiting for data to become ready...\n")
  socketSelect(list(con2))
  cat("Reading data...\n")
  r <- readBin(con2, "raw", 1024)
  total <- total + length(r)
  cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)
# the server
n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE,
open="a+b")
cat("Connected...\n")
writeBin(w, con1)
cat("Sent data the first time, sleeping...\n")
Sys.sleep(10)
cat("Sending data the second time...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)
Best
Tomas
# the client
con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0
done <- function(n) {
 n >= 1e8
}
while(!done(total)) {
  r <- readBin(con2, "raw", 1024)
  total <- total + length(r)
  cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)
# the server
n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE,
open="a+b")
cat("Connected...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)
Ben
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] socketConnection, delay when reading from
Google?
https://developer.r-project.org/Blog/public/2020/03/17/socket-connections-update/
https://www.digitalocean.com/community/tutorials/understanding-sockets
https://developer.ibm.com/tutorials/l-sockpit/
On November 27, 2021 2:36:48 PM PST, Ben Engbers wrote:
>Hi,
>
>Looks promising! Where can I find more information or tutorials about
>psock or parallel computing?
>
>Best,
>Ben
>
>Op 27-11-2021 om 20:19 schreef Tomas Kalibera:
>> On 11/27/21 8:05 PM, Tomas Kalibera wrote:
>>
>> This is an extended demo with socketSelect() used to wait on the client
>> for some data to be available, to avoid consuming too much CPU by
>> polling. To be pasted into two R sessions running on the same computer.
>> You would have to replace the function done() with something figuring
>> out from the data whether it is complete or not, based on the protocol.
>>
>> Best
>> Tomas
>>
>>
>> # the client
>>
>> con2 <- socketConnection("localhost", port = 6011, open = "rb")
>> cat("Connected...\n")
>> total <- 0
>>
>> done <- function(n) {
>> Â n >= 2e8
>> }
>>
>> while(!done(total)) {
>> Â Â cat("Waiting for data to become ready...\n")
>> Â Â socketSelect(list(con2))
>> Â Â cat("Reading data...\n")
>> Â Â r <- readBin(con2, "raw", 1024)
>> Â Â total <- total + length(r)
>> Â Â cat("Read", length(r), "bytes (total ", total, ").\n")
>> }
>> close(con2)
>>
>> # the server
>>
>> n <- 1e8
>> w <- as.raw(runif(n, 0, 255))
>> con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE,
>> open="a+b")
>> cat("Connected...\n")
>> writeBin(w, con1)
>> cat("Sent data the first time, sleeping...\n")
>> Sys.sleep(10)
>> cat("Sending data the second time...\n")
>> writeBin(w, con1)
>> cat("Data sent to client...\n")
>> close(con1)
>>
>>>
>>> Best
>>> Tomas
>>>
>>> # the client
>>>
>>> con2 <- socketConnection("localhost", port = 6011, open = "rb")
>>> cat("Connected...\n")
>>> total <- 0
>>>
>>> done <- function(n) {
>>> Â n >= 1e8
>>> }
>>>
>>> while(!done(total)) {
>>> Â Â r <- readBin(con2, "raw", 1024)
>>> Â Â total <- total + length(r)
>>> Â Â cat("Read", length(r), "bytes (total ", total, ").\n")
>>> }
>>> close(con2)
>>>
>>> # the server
>>>
>>> n <- 1e8
>>> w <- as.raw(runif(n, 0, 255))
>>> con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE,
>>> open="a+b")
>>> cat("Connected...\n")
>>> writeBin(w, con1)
>>> cat("Data sent to client...\n")
>>> close(con1)
>>>
Ben
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
--
Sent from my phone. Please excuse my brevity.
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
