[
https://issues.apache.org/jira/browse/HADOOP-10047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798124#comment-13798124
]
Gopal V commented on HADOOP-10047:
----------------------------------
This was not meant to be a final patch, just a checkpoint for people to
actually run & test instead of just read - I will remove the bad import from
my git branches.
bq. I don't think we should accept indirect buffers.
I did that because I wanted to extend the compressor with as much compatibility
as possible. There is no JNI code for indirect buffers here, it copies into a
direct buffer - as the codecs have always done. But it does add complexity
which is unnecessary.
bq. we need to see a patch that uses this interface (for more than just a test)
before we can agree that it's the right one.
I posted the API docs hoping for some commentary on people who use the
CompressionCodecs. From my side, I modeled it around the API needs of ORC when
I designed it and extended it to add the ability to decompress zero-copy reads
which cross block boundaries (in zlib+bzip2 at least - unlikely for SNAPPY/LZ4
native APIs).
In that angle, I only need the Decompressor APIs and I do not think I intend to
use the Compressor APIs at all. That was done to round it off cleanly.
I could drop that and still move ahead with my work.
bq. I think it might actually be simpler to have ZlibDirectDecompressor, etc.
be separate classes from ZlibDecompressor.
It would be much simpler to create a separate class for this, but that means
adding more native code to the system, mainly because of how the current
inflateDirect()/deflateDirect() does JNI getField/setField calls instead of
moving around the buffer's limit/position fields.
If you have an idea on adding it without adding more derivatives of CodecPool,
CompressionCodec and CompressionCodecFactory, I will be interested in hearing
that. The down-casting seemed like the easiest way to check for the API's
presence. But that was just the path of least resistance for me.
bq. If the compressors and decompressors are buffering at all, issues like
these can become problematic.
As for the buffering issues, you will notice that the Zlib implementation
handles buffering cleanly. It is only operating on the pure native fast-path if
the new API is the only one used, with a clean output buffer and the local
buffers are empty.
I think bzip2 follows in zlib's footsteps in this matter.
lz4 and SNAPPY require the entire buffer to compress in one go and only lz4 has
a window decompression function for output sizes (the _partial one). I could
probably write a SNAPPY decompressor which handles partial output, but the
default API doesn't do that.
The best I could tell was that buffering is broken in both of them already -
you need to initialize them with the a directBuffersize > the largest buffer
you expect. Otherwise they throw exceptions anyway.
> Allow Compressor/Decompressor APIs to expose a Direct ByteBuffer API
> --------------------------------------------------------------------
>
> Key: HADOOP-10047
> URL: https://issues.apache.org/jira/browse/HADOOP-10047
> Project: Hadoop Common
> Issue Type: New Feature
> Components: io
> Reporter: Gopal V
> Assignee: Gopal V
> Labels: compression
> Attachments: DirectCompressor.html, DirectDecompressor.html,
> HADOOP-10047-WIP.patch, HADOOP-10047-with-tests.patch
>
>
> With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform
> all I/O operations without copying data into byte[] buffers or other buffers
> which wrap over them.
> This is a proposal for adding new DirectCompressor and DirectDecompressor
> interfaces to the io.compress, to indicate codecs which want to surface the
> direct buffer layer upwards.
> The implementation may or may not copy the buffers passed in, but should work
> with direct heap/mmap buffers and cannot assume .array() availability.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
