asfimport opened a new issue, #404:
URL: https://github.com/apache/arrow-java/issues/404
It would be useful for applications which need convert Avro data to Arrow
data.
This is an adapter which convert data with existing API (like JDBC adapter)
rather than a native reader (like orc).
We implement this function through Avro java project, receiving param like
Decoder/Schema/DatumReader of Avro and return VectorSchemaRoot. For each data
type we have a consumer class as below to get Avro data and write it into
vector to avoid boxing/unboxing (e.g. GenericRecord#get returns Object)
```java
public class AvroIntConsumer implements Consumer {
private final IntWriter writer;
public AvroIntConsumer(IntVector vector)
{ this.writer = new IntWriterImpl(vector); }
@Override
public void consume(Decoder decoder) throws IOException
{ writer.writeInt(decoder.readInt());
writer.setPosition(writer.getPosition() + 1); }
```
We intended to support primitive and complex types (null value represented
via unions type with null type), size limit and field selection could be
optional for users.
**Reporter**: [Ji Liu](https://issues.apache.org/jira/browse/ARROW-5845) /
@tianchen92
#### Subtasks:
- [X] [[Java] Create Avro adapter module and add
dependencies](https://github.com/apache/arrow/issues/22265)
- [X] [[Java] Initial implement to convert Avro record with primitive
types](https://github.com/apache/arrow/issues/22278)
- [X] [[Java] Avro adapter implement simple Record type
](https://github.com/apache/arrow/issues/22396)
- [X] [[Java] Avro adapter support convert nullable
value](https://github.com/apache/arrow/issues/22439)
- [X] [[Java] Avro adapter implement unions
type](https://github.com/apache/arrow/issues/22496)
- [X] [[Java] Avro adapter avoid potential resource
leak.](https://github.com/apache/arrow/issues/22590)
- [X] [[Java] Add API to avro adapter to limit number of rows returned at a
time.](https://github.com/apache/arrow/issues/22608)
- [X] [[Java] Avro adapter implement Array/Map/Fixed
type](https://github.com/apache/arrow/issues/22649)
- [X] [[Java] Avro adapter implement Enum type and nested Record
type](https://github.com/apache/arrow/issues/16862)
- [X] [[Java] Add benchmark and large fake data UT for avro
adapter](https://github.com/apache/arrow/issues/22830)
- [X] [[Java] Add support for skipping decoding of columns/field in Avro
converter](https://github.com/apache/arrow/issues/22949)
- [ ] [[Java] Experiment with performance difference of avoiding the use of
Avro Decoder](https://github.com/apache/arrow-java/issues/281)
- [X] [[Java] Support logical type encodings from
Avro](https://github.com/apache/arrow/issues/22950)
- [ ] [[Java] Avro - Experiment with "compiled" consumer delegates for
performance.](https://github.com/apache/arrow-java/issues/280)
- [X] [[JAVA] Avro adapter benchmark only runs once in
JMH](https://github.com/apache/arrow/issues/23064)
- [X] [[Java] Extract a common base class for avro converter
consumers](https://github.com/apache/arrow/issues/23236)
- [X] [[Java] Avro converter should convert attributes and props to
FieldType metadata](https://github.com/apache/arrow/issues/23755)
<sub>**Note**: *This issue was originally created as
[ARROW-5845](https://issues.apache.org/jira/browse/ARROW-5845). Please see the
[migration documentation](https://github.com/apache/arrow/issues/14542) for
further details.*</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
