[ANNOUNCE] Apache Doris Shade 1.0.2 Release
Hi All, We are pleased to announce the release of Apache Doris Shade 1.0.2. The release is available at: https://dist.apache.org/repos/dist/release/doris/doris-shade/1.0.2/ Thanks to everyone who has contributed to this release, and the release note can be found here: https://github.com/apache/doris-shade/releases/tag/doris-shade-1.0.2 -- Brs, di.wu
Re: Re:Re: Re: Issue Integrating Hive on CDP Distribution with Doris 2.0.1.1
Hi Joice,
Got it, thanks very much. We will setup a cdp env and do some compatibility
work.
Best regards,
Kaka
Joice Jacob 于2023年10月7日周六 14:58写道：
> Hi,Dear Doris Community,
> I wanted to provide an update on my recent experience with integrating
> Doris with both Apache Hive and CDP Hive, as well as seek insights into a
> particular observation.
>
> In my integration with Apache Hive, I noticed that transaction tables
> retain the "_orc_acid_version" file, which aligns with the expected
> behavior. However, when I integrated with CDP Hive, I observed that this
> "_orc_acid_version" file was conspicuously absent.
>
> This discrepancy in behavior between the two environments has left me
> somewhat perplexed, and I'm keen to understand the underlying reasons
> behind it. It appears that in CDP Hive, *managed tables are set to be
> transactional by default*. Interestingly, when we create managed tables
> with the TBLPROPERTIES("transactional"="false") option, they are seemingly
> *treated
> as external tables*.
>
> I've attached screenshots to illustrate this scenario for further clarity.
> CREATE TABLE BABY2(
> id int,
> FNAME VARCHAR(50),
> GENDER VARCHAR(2),
> TOTCOUNT INT
> )
> PARTITIONED BY(DATA_ID INT)
> TBLPROPERTIES (
> 'transactional'='false',
> 'orc.compress'='snappy');
>
> if we execute above statement table create like this.
> [image: image.png]
>
> I would greatly appreciate any insights, suggestions, or explanations that
> the Doris community may have regarding this behavior. Your expertise will
> be invaluable in helping me navigate this integration and address any
> associated challenges.
>
> Thank you in advance for your time and support. I look forward to hearing
> from you and to collaborating with the community to better understand and
> resolve this matter.
>
> Best regards,
> joice
>
> On Sat, Oct 7, 2023 at 11:46 AM kaka chen wrote:
>
>> Hi Joice:
>> Thanks your reporting.
>>
>> It seems the root cause of this issue is missing "_orc_acid_version" file.
>> From Hive version >= 3.0, delta/base files will always have file
>> '_orc_acid_version' with value >= '2'.
>> Maybe the hive3 of HDP has similar issue?
>> https://issues.apache.org/jira/browse/HIVE-16964
>>
>> A workaround is try to create table without transactional props.
>> TBLPROPERTIES("transactional"="true")
>> pls try it, thanks.
>>
>> Best regards,
>> Kaka
>>
>>
>> Joice Jacob 于2023年10月6日周五 22:58写道：
>>
>> > Hi,
>> > I've checked the Hive data directory, and I couldn't find the
>> > "_orc_acid_version" file. I have attached screenshots for your
>> reference.
>> >
>> > Are there any recommended workarounds or alternative approaches that I
>> can
>> > consider to resolve this issue? I'm open to exploring different
>> solutions
>> > to ensure the successful integration of Doris with Hive in CDP. Any
>> > guidance or suggestions would be greatly appreciated.
>> >
>> > Screenshots attached for your reference.
>> >
>> > Best regards,
>> > Joice
>> >
>> > [image: image.png]
>> >
>> > On Fri, Oct 6, 2023 at 6:28 PM Mingyu Chen wrote:
>> >
>> >> Could you list your hive data dir, to see what files exist? like:
>> >> _orc_acid_version
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Best Regards
>> >> Mingyu Chen
>> >>
>> >> Email:
>> >> morning...@apache.org
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> At 2023-10-05 22:31:58, "Mingyu Chen" wrote:
>> >> >Oh, I see, let me check it again.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >--
>> >> >
>> >> >Best Regards
>> >> >Mingyu Chen
>> >> >
>> >> >Email:
>> >> >morning...@apache.org
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >At 2023-10-05 21:49:28, "Joice Jacob"
>> wrote:
>> >> >>Hi,
>> >> >>I wanted to share some important information regarding the CDP Hive
>> >> version
>> >> >>that I am currently using, which is Hive 3.1.3000.7.1.7.0-551.
>> >> >>
>> >> >>In CDP Hive version 3.x, a significant change has been introduced
>> >> regarding
>> >> >>managed tables. By default, managed tables in Hive 3.x are considered
>> >> >>transactional. This means that if you create a managed table
>> explicitly
>> >> >>specifying it as transactional with a value of "false," it will be
>> >> treated
>> >> >>as an external table instead.
>> >> >>Thank you for your attention to this matter, and I appreciate your
>> >> >>continued support.
>> >> >>
>> >>
>> https://www.thecodersstop.com/hadoop/apache-hive-3-changes-in-cdp-upgrade-part-1/
>> >> >>Best regards,
>> >> >>Joice
>> >> >>
>> >> >>On Thu, Oct 5, 2023 at 7:01 PM Mingyu Chen
>> wrote:
>> >> >>
>> >> >>> I saw that your hive table is with property "transactional" =
>> "true",
>> >> >>> And Doris only support ACID table with Hive 3, not support with
>> Hive
>> >> 2.x.
>> >> >>> So you may need to create non-transactional hive table for Doris to
>> >> visit.
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Best Regards
>> >> >>> Mingyu Chen
>> >> >>>
>> >> >>> Email:
>> >> >>> morning...@apache.org
>> >> >>>
>> >> >>>
>> >> >>> 在 2023-10-
Re: Re:Re: Re: Issue Integrating Hive on CDP Distribution with Doris 2.0.1.1
Hi Joice,
BTW，could you pls try to query it by Doris after putting the
"_orc_acid_version"
file to the hdfs directory of hive manually? Thanks.
Best regards,
Kaka
kaka chen 于2023年10月8日周日 10:10写道：
> Hi Joice,
>
> Got it, thanks very much. We will setup a cdp env and do some
> compatibility work.
>
> Best regards,
> Kaka
>
> Joice Jacob 于2023年10月7日周六 14:58写道：
>
>> Hi,Dear Doris Community,
>> I wanted to provide an update on my recent experience with integrating
>> Doris with both Apache Hive and CDP Hive, as well as seek insights into a
>> particular observation.
>>
>> In my integration with Apache Hive, I noticed that transaction tables
>> retain the "_orc_acid_version" file, which aligns with the expected
>> behavior. However, when I integrated with CDP Hive, I observed that this
>> "_orc_acid_version" file was conspicuously absent.
>>
>> This discrepancy in behavior between the two environments has left me
>> somewhat perplexed, and I'm keen to understand the underlying reasons
>> behind it. It appears that in CDP Hive, *managed tables are set to be
>> transactional by default*. Interestingly, when we create managed tables
>> with the TBLPROPERTIES("transactional"="false") option, they are seemingly
>> *treated
>> as external tables*.
>>
>> I've attached screenshots to illustrate this scenario for further clarity.
>> CREATE TABLE BABY2(
>> id int,
>> FNAME VARCHAR(50),
>> GENDER VARCHAR(2),
>> TOTCOUNT INT
>> )
>> PARTITIONED BY(DATA_ID INT)
>> TBLPROPERTIES (
>> 'transactional'='false',
>> 'orc.compress'='snappy');
>>
>> if we execute above statement table create like this.
>> [image: image.png]
>>
>> I would greatly appreciate any insights, suggestions, or explanations
>> that the Doris community may have regarding this behavior. Your expertise
>> will be invaluable in helping me navigate this integration and address any
>> associated challenges.
>>
>> Thank you in advance for your time and support. I look forward to hearing
>> from you and to collaborating with the community to better understand and
>> resolve this matter.
>>
>> Best regards,
>> joice
>>
>> On Sat, Oct 7, 2023 at 11:46 AM kaka chen wrote:
>>
>>> Hi Joice:
>>> Thanks your reporting.
>>>
>>> It seems the root cause of this issue is missing "_orc_acid_version"
>>> file.
>>> From Hive version >= 3.0, delta/base files will always have file
>>> '_orc_acid_version' with value >= '2'.
>>> Maybe the hive3 of HDP has similar issue?
>>> https://issues.apache.org/jira/browse/HIVE-16964
>>>
>>> A workaround is try to create table without transactional props.
>>> TBLPROPERTIES("transactional"="true")
>>> pls try it, thanks.
>>>
>>> Best regards,
>>> Kaka
>>>
>>>
>>> Joice Jacob 于2023年10月6日周五 22:58写道：
>>>
>>> > Hi,
>>> > I've checked the Hive data directory, and I couldn't find the
>>> > "_orc_acid_version" file. I have attached screenshots for your
>>> reference.
>>> >
>>> > Are there any recommended workarounds or alternative approaches that I
>>> can
>>> > consider to resolve this issue? I'm open to exploring different
>>> solutions
>>> > to ensure the successful integration of Doris with Hive in CDP. Any
>>> > guidance or suggestions would be greatly appreciated.
>>> >
>>> > Screenshots attached for your reference.
>>> >
>>> > Best regards,
>>> > Joice
>>> >
>>> > [image: image.png]
>>> >
>>> > On Fri, Oct 6, 2023 at 6:28 PM Mingyu Chen wrote:
>>> >
>>> >> Could you list your hive data dir, to see what files exist? like:
>>> >> _orc_acid_version
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> Best Regards
>>> >> Mingyu Chen
>>> >>
>>> >> Email:
>>> >> morning...@apache.org
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> At 2023-10-05 22:31:58, "Mingyu Chen" wrote:
>>> >> >Oh, I see, let me check it again.
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >--
>>> >> >
>>> >> >Best Regards
>>> >> >Mingyu Chen
>>> >> >
>>> >> >Email:
>>> >> >morning...@apache.org
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >At 2023-10-05 21:49:28, "Joice Jacob"
>>> wrote:
>>> >> >>Hi,
>>> >> >>I wanted to share some important information regarding the CDP Hive
>>> >> version
>>> >> >>that I am currently using, which is Hive 3.1.3000.7.1.7.0-551.
>>> >> >>
>>> >> >>In CDP Hive version 3.x, a significant change has been introduced
>>> >> regarding
>>> >> >>managed tables. By default, managed tables in Hive 3.x are
>>> considered
>>> >> >>transactional. This means that if you create a managed table
>>> explicitly
>>> >> >>specifying it as transactional with a value of "false," it will be
>>> >> treated
>>> >> >>as an external table instead.
>>> >> >>Thank you for your attention to this matter, and I appreciate your
>>> >> >>continued support.
>>> >> >>
>>> >>
>>> https://www.thecodersstop.com/hadoop/apache-hive-3-changes-in-cdp-upgrade-part-1/
>>> >> >>Best regards,
>>> >> >>Joice
>>> >> >>
>>> >> >>On Thu, Oct 5, 2023 at 7:01 PM Mingyu Chen
>>> wrote:
>>> >> >>
>>> >> >>> I saw that your hive table is with property "transactio
Re: Re:Re: Re: Issue Integrating Hive on CDP Distribution with Doris 2.0.1.1
Yes, I will check. I will update you by EOD today.
Thanks for support.
Thanks & Regards
Joice
On Sun, Oct 8, 2023, 7:48 AM kaka chen wrote:
> Hi Joice,
>
> BTW，could you pls try to query it by Doris after putting the
> "_orc_acid_version"
> file to the hdfs directory of hive manually? Thanks.
>
> Best regards,
> Kaka
>
> kaka chen 于2023年10月8日周日 10:10写道：
>
>> Hi Joice,
>>
>> Got it, thanks very much. We will setup a cdp env and do some
>> compatibility work.
>>
>> Best regards,
>> Kaka
>>
>> Joice Jacob 于2023年10月7日周六 14:58写道：
>>
>>> Hi,Dear Doris Community,
>>> I wanted to provide an update on my recent experience with integrating
>>> Doris with both Apache Hive and CDP Hive, as well as seek insights into a
>>> particular observation.
>>>
>>> In my integration with Apache Hive, I noticed that transaction tables
>>> retain the "_orc_acid_version" file, which aligns with the expected
>>> behavior. However, when I integrated with CDP Hive, I observed that this
>>> "_orc_acid_version" file was conspicuously absent.
>>>
>>> This discrepancy in behavior between the two environments has left me
>>> somewhat perplexed, and I'm keen to understand the underlying reasons
>>> behind it. It appears that in CDP Hive, *managed tables are set to be
>>> transactional by default*. Interestingly, when we create managed tables
>>> with the TBLPROPERTIES("transactional"="false") option, they are seemingly
>>> *treated
>>> as external tables*.
>>>
>>> I've attached screenshots to illustrate this scenario for further
>>> clarity.
>>> CREATE TABLE BABY2(
>>> id int,
>>> FNAME VARCHAR(50),
>>> GENDER VARCHAR(2),
>>> TOTCOUNT INT
>>> )
>>> PARTITIONED BY(DATA_ID INT)
>>> TBLPROPERTIES (
>>> 'transactional'='false',
>>> 'orc.compress'='snappy');
>>>
>>> if we execute above statement table create like this.
>>> [image: image.png]
>>>
>>> I would greatly appreciate any insights, suggestions, or explanations
>>> that the Doris community may have regarding this behavior. Your expertise
>>> will be invaluable in helping me navigate this integration and address any
>>> associated challenges.
>>>
>>> Thank you in advance for your time and support. I look forward to
>>> hearing from you and to collaborating with the community to better
>>> understand and resolve this matter.
>>>
>>> Best regards,
>>> joice
>>>
>>> On Sat, Oct 7, 2023 at 11:46 AM kaka chen wrote:
>>>
Hi Joice:
Thanks your reporting.
It seems the root cause of this issue is missing "_orc_acid_version"
file.
From Hive version >= 3.0, delta/base files will always have file
'_orc_acid_version' with value >= '2'.
Maybe the hive3 of HDP has similar issue?
https://issues.apache.org/jira/browse/HIVE-16964
A workaround is try to create table without transactional props.
TBLPROPERTIES("transactional"="true")
pls try it, thanks.
Best regards,
Kaka
Joice Jacob 于2023年10月6日周五 22:58写道：
> Hi,
> I've checked the Hive data directory, and I couldn't find the
> "_orc_acid_version" file. I have attached screenshots for your
reference.
>
> Are there any recommended workarounds or alternative approaches that
I can
> consider to resolve this issue? I'm open to exploring different
solutions
> to ensure the successful integration of Doris with Hive in CDP. Any
> guidance or suggestions would be greatly appreciated.
>
> Screenshots attached for your reference.
>
> Best regards,
> Joice
>
> [image: image.png]
>
> On Fri, Oct 6, 2023 at 6:28 PM Mingyu Chen
wrote:
>
>> Could you list your hive data dir, to see what files exist? like:
>> _orc_acid_version
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Best Regards
>> Mingyu Chen
>>
>> Email:
>> morning...@apache.org
>>
>>
>>
>>
>>
>> At 2023-10-05 22:31:58, "Mingyu Chen" wrote:
>> >Oh, I see, let me check it again.
>> >
>> >
>> >
>> >
>> >--
>> >
>> >Best Regards
>> >Mingyu Chen
>> >
>> >Email:
>> >morning...@apache.org
>> >
>> >
>> >
>> >
>> >
>> >At 2023-10-05 21:49:28, "Joice Jacob"
wrote:
>> >>Hi,
>> >>I wanted to share some important information regarding the CDP Hive
>> version
>> >>that I am currently using, which is Hive 3.1.3000.7.1.7.0-551.
>> >>
>> >>In CDP Hive version 3.x, a significant change has been introduced
>> regarding
>> >>managed tables. By default, managed tables in Hive 3.x are
considered
>> >>transactional. This means that if you create a managed table
explicitly
>> >>specifying it as transactional with a value of "false," it will be
>> treated
>> >>as an external table instead.
>> >>Thank you for your attention to this matter, and I appreciat
