[
https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16827711#comment-16827711
]
ASF GitHub Bot commented on AIRFLOW-3720:
-----------------------------------------
potiuk commented on pull request #4766: [AIRFLOW-3720] Add prefix to file match
in GCS_TO_S3 operator to avoid missmatch
URL: https://github.com/apache/airflow/pull/4766
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> GoogleCloudStorageToS3Operator - incorrect folder compare
> ----------------------------------------------------------
>
> Key: AIRFLOW-3720
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3720
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
> Affects Versions: 1.10.0
> Reporter: Chaim
> Assignee: Chaim
> Priority: Major
> Fix For: 2.0.0
>
>
> the code that compares folders from gcp to s3 is incorrect.
> the code is:
> files = set(files) - set(existing_files)
> but the list from gcp has a "/" to the name, for example: "myfolder/", while
> in s3 it does not have "/" so the folder is "myfolder"
> the result is that the code tries to recopy the folder name but fails since
> it already exists
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
