On Mon, Jan 8, 2018 at 4:53 AM, Carsten Haitzler <[email protected]> wrote: > On Fri, 5 Jan 2018 11:26:14 -0200 Gustavo Sverzut Barbieri > <[email protected]> ... >> > see https://phab.enlightenment.org/T5522 >> > >> > it has a lot more details in the todo now. >> > >> > btw i was wanting to talk about the whole reader/writer stuff using in >> > efl.net as i am going to have to have a similar io interface from parent to >> > child loop and back (and also between loops object siblings etc.) >> > >> > the whole buffered binbuf slice etc. >> >> clarify :-) class name? method? > > well read_changed() callbacks... i have to do: > > if (efl_io_reader_can_read_get(event->object)) { > Eina_Slice slice = efl_io_buffered_stream_slice_get(event->object); > eina_binbuf_append_slice(buf, slice); // store the data somewhere > } > > as opposed to just a simple: > > Efl_Data *dat = event->info; > eina_binbuf_append(buf, dat->data, dat->size); // store the data somewhere > > it's a lot more code to basically "deal with some new data that arrived".
there is a key difference here: efl.io.reader is like read(2) + select(2) wrapper. You should b consuming data using read(2). buffered stream is like fdopen(), that provides buffered fwrite() and fread() on top of basic syscalls. The buffered stream allows you to peek at contents (slice_get) without consuming it. Main use cases: - peek to check if all data is there prior to consume (ie: length + payload messages), let the buffered stream get more data prior to consuming it all at once; - when auxiliary data must be constructed, ie: objects, then instead of simply consuming (copy) data, you read from internal buffer and create the matching object from slices (or subslices). This is usually paired with "discard()" calls, to "mark as consumed" without the matching memcpy() to an application owned buffer. then providing the "Efl_Data" is not good, as it wasn't good with old API, as it doesn't allow the first case, only the second (zero copy, since it was already copied). If you had a partial message, you had to create your own buffer, copy and keep that around. this isn't needed with the existing API: don't efl_io_reader_read() or efl_io_buffered_stream_discard() and data will be buffered for you. >> > thing just seems like overly complex >> > work/effort for just sending a buffer and getting a buffer of bytes to/from >> > a target object. :( >> > >> > what do you think? >> >> efl.io.reader.read == read(2) >> efl.io.writer.write == write(2) >> efl.io.closer.close == close(2) >> >> efl.io.reader.can_read == select(2) + readfds >> efl.io.writer.can_write == select(2) + writefds >> >> they return up-to that amount of bytes, what's there to be read or >> what can be written without blocking. no buffering done, which makes >> everything simpler (the implementations are almost 1:1 on unix >> syscalls). >> >> the buffering that was previously replicated in ecore_exe, ecore_con, >> ecore_ipc... is now moved to outside and can be reused everywhere... >> as it's based on the reader/writer interfaces. Then ecore_exe in eo >> will be much, much simpler... and code using ecore_exe in eo will >> looks very close to efl.net or other efl.io, allows to replace the >> source or destination with various objects, from different domains >> (network, file, memory, process...) and everything else will copy. > > actually you hit my point. i wast the code to be very similar. be it efl.net > ipc stuff to/from a server or a client, or it's ipc to/from an exe or to/from > a > thread... i'd like code to be re-usable with minimal or even almost zero > changes. > > what i am also wondering is how we should have parsers built on top of this? > like xml/json/newline delimited strings/etc. etc. ... common data format > parsers... should they be an object that you register another object with as > long its of the class efl.io.reader ? or should it be buffered? or should you > inherit the base class and wrap it with a parser that eats the reader data and > produced "parsed data" in other events on the same object? (inherit efl.exe > and > then do this?). multiple inheritance here could do the job... if the object > listens to its own events. for simplicity I'd build sax-like parsers on top of buffered streams, then you need no buffering handled at parser side to deal with partial messages. parsing json/xml: wait for a delimiter to be in the slice (ie: "," or '"'), using slice_get() to peek at contents. When it's there, parse from the slice and discard, or efl_io_reader_read(), may be useful if you're dealing with functions that need zero-terminator (\0), so you copy to your buffer and ensure it's there. once element is fully there, callback the user to construct something, like an eina_hash, or populate a structure. > the problem with the above design is you really can > have only a single reader for the raw read/buffered data because once read... > it's gone. this kind of disturbs me a bit given thinking about these parsers. I'm still to see use cases you need the duplication behavior, but can be done, similarly to gstreamer you could create multiple consumers (each at their own pace) or simply transform the can_read + read into an event, like old API did. Never found that to be very useful to me as you need to handle buffering yourself. If you handle multiple consumers for the same data, at least the buffering is unified, just keep an array of indexes, once all indexes are > 0, remove that amount from the beginning of your binbuf and subtract all indexes so they become relative to zero again. >> see efl_io_copier_example.c, it's like a netcat-on-steroids... >> >> if all you want is to have a buffered input/ouput, you wrap the >> unbuffered i/o with efl.io.buffered_stream, giving the unbuffered i/o >> as "inner_io". Then all your reads/writes will be buffered... this is >> similar to fdopen() + fwrite() and fread(). > > this is what i want... but only being able to have a single reader (eg for the > raw data) disturbs me (if we then want to also have parsers too). imagine > hooking up 2 parsers (via inheritance or via object linking/registering) ... > you can't. they are mutually exclusive. not sure why you'd want 2 parsers... > maybe one produces a superset of the other and is easier to deal with? e.g. > newline delimited text vs csv parser (which is a subset of newline ...) ? then you chain, not consume the same data. It's like if you use an efl.io.file (which implements the efl.io.reader) and you use an efl.io.buffered_stream or efl.io.copier... these offer line-based events, which you can use to create csv. not on the efl.io.file directly!) for sax-based parsers, this is very easy: provide events with the full chunk (ie: open tag, close tag, tag contents...), but only that small chunk, not a complex object. >> internally efl.io.buffered_stream keeps binbufs for reads and writes >> (as fread, fwrite)... you can consume them using efl.io.reader.read(), >> which will pop stuff from the read buffer... efl.io.writer.write() >> will push stuff to write buffer. >> >> however, since many people want to "peek" at buffer, without really >> reading from it, you can get the read-only slice, example: >> >> - protocol: 4 bytes for integer length, followed by [length] bytes message >> >> you can keep the message and check if the 4 bytes are there... and >> then peek if the rest of the message is there. If it's not, just do >> nothing... wait it to fill a bit more. > > eh? what is this above? ... ? you mean add something like this? many network protocols work like that: packet size, followed by packet payload. others are looking for delimiters (ie: xml). in any case, until you have the full message, you can't process unless you keep partial message handling yourself. if you do have a lib/code that handles partial message, then there is no need for buffered_stream... just feed directly from efl.io.reader "can_read". Like jpeg parsers and some xml parsers. if you just have lib/code that handles full message, then use buffered_stream and keep peeking (slice_get) without consuming until you have the full message. basically it's your choice to handle buffer manually or not. Before you had to deal with manual buffering, UNLESS the efl lib provided that for you (ecore_ipc did, ecore_exe didn't, etc...) -- Gustavo Sverzut Barbieri -------------------------------------- Mobile: +55 (16) 99354-9890 ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
