Hi Joice:
Thanks your reporting.
It seems the root cause of this issue is missing "_orc_acid_version" file.
>From Hive version >= 3.0, delta/base files will always have file
'_orc_acid_version' with value >= '2'.
Maybe the hive3 of HDP has similar issue?
https://issues.apache.org/jira/browse/HIVE-16964
A workaround is try to create table without transactional props.
TBLPROPERTIES("transactional"="true")
pls try it, thanks.
Best regards,
Kaka
Joice Jacob <joicejacob1...@gmail.com> 于2023年10月6日周五 22:58写道：
> Hi,
> I've checked the Hive data directory, and I couldn't find the
> "_orc_acid_version" file. I have attached screenshots for your reference.
>
> Are there any recommended workarounds or alternative approaches that I can
> consider to resolve this issue? I'm open to exploring different solutions
> to ensure the successful integration of Doris with Hive in CDP. Any
> guidance or suggestions would be greatly appreciated.
>
> Screenshots attached for your reference.
>
> Best regards,
> Joice
>
> [image: image.png]
>
> On Fri, Oct 6, 2023 at 6:28 PM Mingyu Chen <morning...@163.com> wrote:
>
>> Could you list your hive data dir, to see what files exist? like:
>> _orc_acid_version
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Best Regards
>> Mingyu Chen
>>
>> Email:
>> morning...@apache.org
>>
>>
>>
>>
>>
>> At 2023-10-05 22:31:58, "Mingyu Chen" <morning...@163.com> wrote:
>> >Oh, I see, let me check it again.
>> >
>> >
>> >
>> >
>> >--
>> >
>> >Best Regards
>> >Mingyu Chen
>> >
>> >Email:
>> >morning...@apache.org
>> >
>> >
>> >
>> >
>> >
>> >At 2023-10-05 21:49:28, "Joice Jacob" <joicejacob1...@gmail.com> wrote:
>> >>Hi,
>> >>I wanted to share some important information regarding the CDP Hive
>> version
>> >>that I am currently using, which is Hive 3.1.3000.7.1.7.0-551.
>> >>
>> >>In CDP Hive version 3.x, a significant change has been introduced
>> regarding
>> >>managed tables. By default, managed tables in Hive 3.x are considered
>> >>transactional. This means that if you create a managed table explicitly
>> >>specifying it as transactional with a value of "false," it will be
>> treated
>> >>as an external table instead.
>> >>Thank you for your attention to this matter, and I appreciate your
>> >>continued support.
>> >>
>> https://www.thecodersstop.com/hadoop/apache-hive-3-changes-in-cdp-upgrade-part-1/
>> >>Best regards,
>> >>Joice
>> >>
>> >>On Thu, Oct 5, 2023 at 7:01 PM Mingyu Chen <morning...@163.com> wrote:
>> >>
>> >>> I saw that your hive table is with property "transactional" = "true",
>> >>> And Doris only support ACID table with Hive 3, not support with Hive
>> 2.x.
>> >>> So you may need to create non-transactional hive table for Doris to
>> visit.
>> >>>
>> >>>
>> >>> --
>> >>> Best Regards
>> >>> Mingyu Chen
>> >>>
>> >>> Email:
>> >>> morning...@apache.org
>> >>>
>> >>>
>> >>> 在 2023-10-05 13:30:25，"Joice Jacob" <joicejacob1...@gmail.com> 写道：
>> >>>
>> >>> Dear Doris Community & Jiafeng.Zhang
>> >>> ,
>> >>> Thank you for your prompt response and your willingness to assist
>> with the
>> >>> issue I've been encountering with the integration of Doris and
>> Cloudera's
>> >>> Hive.
>> >>>
>> >>> To provide you with the necessary information, here are the details
>> you
>> >>> requested:
>> >>>
>> >>> *1. Hive Catalog Creation Statement:*
>> >>>
>> >>> My Hive catalog creation statement is as follows:
>> >>>
>> >>> CREATE CATALOG hive PROPERTIES (
>> >>> 'type'='hms',
>> >>> 'hive.metastore.uris' = 'thrift://10.128.0.4:9083',
>> >>> 'hive.version' = '3.1.3',
>> >>> 'hive.metastore.sasl.enabled' = 'true',
>> >>> 'hive.metastore.kerberos.principal' = 'hive/instanc...@hadoop.com
>> ',
>> >>> 'hadoop.security.authentication' = 'kerberos',
>> >>> 'hadoop.kerberos.keytab' = '/home/techuser/doris/hive.keytab',
>> >>> 'hadoop.kerberos.principal' = 'hive/instanc...@hadoop.com',
>> >>> 'yarn.resourcemanager.principal' = 'yarn/instanc...@hadoop.com'
>> >>> );
>> >>>
>> >>> *2. CDP Hive Version:*
>> >>>
>> >>> The CDP Hive version I am using is: Hive 3.1.3000.7.1.7.0-551
>> >>>
>> >>> *3. Configuration Attempt:*
>> >>>
>> >>> I have tried specifying the Hive version in my catalog configuration
>> with
>> >>> the following statement:
>> >>> "hive.version=2.1.0"
>> >>> Despite attempting to set the Hive version to 2.1.0, I continue to
>> >>> experience the same issue, which is detailed in my previous emails.
>> >>>
>> >>> *4. Logs: *
>> >>> I will attach both the fe.log and fe.warn logs to this email for your
>> >>> reference. These logs should provide additional context regarding the
>> issue
>> >>> I'm facing.
>> >>>
>> >>> Once again, I want to express my gratitude for your assistance and
>> support
>> >>> in resolving this matter. I look forward to your insights and
>> >>> recommendations based on the provided logs and catalog configuration.
>> >>>
>> >>> Please feel free to let me know if you require any further
>> information or
>> >>> if there are additional steps I should take to assist in diagnosing
>> and
>> >>> resolving the issue.
>> >>>
>> >>> [image: doris_error.JPG]
>> >>> fe.log log at hive catlog creation time
>> >>> [image: fe_log_at_hivecatalog.JPG]
>> >>> Hive managed table script
>> >>> [image: image.png]
>> >>>
>> >>>
>> >>> Best regards,
>> >>> Joice
>> >>>
>> >>> On Thu, Oct 5, 2023 at 8:10 AM Jiafeng.Zhang <zhang...@gmail.com>
>> wrote:
>> >>>
>> >>>> Can you provide your hive catalog creation statement, your fe.log
>> log at
>> >>>> that time, and your cdp hive version? This will help us locate the
>> >>>> problem,
>> >>>> thank you.
>> >>>> You can also try specifying your hive version in your catalog
>> statement:
>> >>>> "hive.version=2.1.0"
>> >>>>
>> >>>> Joice Jacob <joicejacob1...@gmail.com> 于2023年10月5日周四 00:45写道：
>> >>>>
>> >>>> > I am reaching out once again to seek assistance and share a
>> specific
>> >>>> issue
>> >>>> > I've encountered while integrating Doris with Cloudera-flavored
>> Hive.
>> >>>> The
>> >>>> > error message I'm facing is as follows:
>> >>>> > detailMessage = get file split failed for table: baby1, err:
>> >>>> > java.lang.Exception: Hive 2.x versioned full-acid tables need to
>> run
>> >>>> major
>> >>>> > compaction.
>> >>>> >
>> >>>> > This error message appears when I attempt to query a Hive table
>> from
>> >>>> > Doris, and it seems to be related to Hive 2.x versioned full-ACID
>> tables
>> >>>> > requiring a major compaction.
>> >>>> > I would like to ask the Doris community for guidance on how to
>> handle
>> >>>> this
>> >>>> > issue effectively. Specifically, I am interested in understanding
>> the
>> >>>> best
>> >>>> > practices and steps to follow when dealing with Cloudera-flavored
>> Hive
>> >>>> > tables that require major compaction for Doris integration.
>> >>>> >
>> >>>> > If anyone in the community has successfully addressed this issue
>> or can
>> >>>> > provide insights into how to configure and manage
>> Cloudera-flavored Hive
>> >>>> > tables for integration with Doris, your expertise would be highly
>> >>>> > appreciated.
>> >>>> >
>> >>>> > Thank you for your time and support, and I look forward to
>> receiving
>> >>>> your
>> >>>> > valuable input.
>> >>>> > Joice
>> >>>> >
>> >>>> > On Wed, Oct 4, 2023 at 9:23 PM Joice Jacob <
>> joicejacob1...@gmail.com>
>> >>>> > wrote:
>> >>>> >
>> >>>> >> Dear Doris Community,
>> >>>> >> I have an update on the issue I previously mentioned regarding the
>> >>>> >> integration of Hive on a CDP distribution with Doris 2.0.1.1.
>> >>>> >>
>> >>>> >> After further investigation, I have identified that the issue is
>> >>>> related
>> >>>> >> to Hive managed tables being transactional by default. This
>> appears to
>> >>>> be
>> >>>> >> causing the error I encountered earlier.
>> >>>> >>
>> >>>> >> To address this issue, I am seeking guidance from the community
>> on any
>> >>>> >> specific configurations or settings that need to be adjusted for
>> Hive
>> >>>> >> transactional tables when using Hive as the catalog in Doris. Are
>> >>>> there any
>> >>>> >> recommended configurations or best practices that I should follow
>> to
>> >>>> ensure
>> >>>> >> smooth integration and query execution?
>> >>>> >>
>> >>>> >> Any insights or recommendations from the Doris community would be
>> >>>> greatly
>> >>>> >> appreciated. Your expertise and guidance will be instrumental in
>> >>>> helping me
>> >>>> >> resolve this challenge.
>> >>>> >>
>> >>>> >> Thank you for your continued support, and I look forward to your
>> >>>> valuable
>> >>>> >> input.
>> >>>> >>
>> >>>> >> Best regards,
>> >>>> >> Joice
>> >>>> >>
>> >>>> >> On Wed, Oct 4, 2023 at 6:54 PM Joice Jacob <
>> joicejacob1...@gmail.com>
>> >>>> >> wrote:
>> >>>> >>
>> >>>> >>> Dear Doris Community,
>> >>>> >>>
>> >>>> >>> I am reaching out to the community to seek assistance with an
>> >>>> >>> integration issue I've encountered while trying to use Hive on a
>> CDP
>> >>>> >>> distribution with Doris 2.0.1.1.
>> >>>> >>>
>> >>>> >>> Here are the details of my setup:
>> >>>> >>>
>> >>>> >>> Doris Version: 2.0.1.1
>> >>>> >>> Hive Version: 3.1.3
>> >>>> >>> Cluster Security: Kerberized
>> >>>> >>>
>> >>>> >>> I have successfully created a Hive catalog in Doris and have
>> been able
>> >>>> >>> to set up the integration between Hive and Doris. However, when I
>> >>>> attempt
>> >>>> >>> to query a Hive table using Doris, I encounter the following
>> error:
>> >>>> >>>
>> >>>> >>> ERROR 1105 (HY000): errCode = 2, detailMessage = get file split
>> failed
>> >>>> >>> for table: baby1, err: java.lang.Exception: Hive 2.x versioned
>> >>>> full-acid
>> >>>> >>> tables need to run major compaction.
>> >>>> >>>
>> >>>> >>> I have already performed a major compaction as recommended, but
>> I am
>> >>>> >>> still encountering the same error.
>> >>>> >>>
>> >>>> >>> I would greatly appreciate any insights, guidance, or solutions
>> that
>> >>>> the
>> >>>> >>> Doris community can offer to help me resolve this issue. If
>> anyone has
>> >>>> >>> encountered a similar problem or has expertise in integrating
>> Hive
>> >>>> with
>> >>>> >>> Doris, your assistance would be invaluable.
>> >>>> >>>
>> >>>> >>> Thank you in advance for your time and support. I look forward to
>> >>>> >>> hearing from the community and working together to find a
>> solution to
>> >>>> this
>> >>>> >>> challenge.
>> >>>> >>>
>> >>>> >>> [image: baby_table.JPG]
>> >>>> >>> [image: doris_hive_catlog_result.JPG]
>> >>>> >>>
>> >>>> >>> [image: doris_fe_log.JPG]
>> >>>> >>>
>> >>>> >>> [image: hive_metastore_error.JPG]
>> >>>> >>>
>> >>>> >>> Thanks
>> >>>> >>> Joice
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>>
>> >>>> --
>> >>>> 张家峰
>> >>>>
>> >>>
>>
>
