saurabhd336 opened a new pull request, #11655:
URL: https://github.com/apache/pinot/pull/11655
When GcsPinotFS is initialised with credentials that only have a folder
level access, most operations (eg: copyToLocalFile) fail with
```
java.io.IOException: com.google.cloud.storage.StorageException: <account>
does not have storage.buckets.get access to the Google Cloud Storage bucket.
Permission 'storage.buckets.get' denied on resource (or it may not exist).
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.getBlob(GcsPinotFS.java:279)
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.copyToLocalFile(GcsPinotFS.java:216)
....
Caused by: com.google.cloud.storage.StorageException:
startree-...@production-267908.iam.gserviceaccount.com does not have
storage.buckets.get access to the Google Cloud Storage bucket. Permission
'storage.buckets.get' denied on resource (or it may not exist).
at
com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:231)
at
com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:423)
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:297)
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:294)
at
com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:293)
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.getBucket(GcsPinotFS.java:271)
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.getBlob(GcsPinotFS.java:277)
... 16 more
Caused by:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
GET
https://storage.googleapis.com/storage/v1/b/platform-poc-data?projection=full
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "<account> does not have storage.buckets.get access to the
Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on
resource (or it may not exist).",
"reason" : "forbidden"
} ],
"message" : "<account> does not have storage.buckets.get access to the
Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on
resource (or it may not exist)."
}
at
com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:149)
at
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:112)
at
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:39)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:443)
at
com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
at
com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:420)
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:297)
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:294)
at
com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:293)
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.getBucket(GcsPinotFS.java:271)
at
org.apache.pinot.plugin.filesystem.GcsPinotFS.getBlob(GcsPinotFS.java:277)
... 16 more
```
This is due to each file access trying to first get `Bucket` object when
downloading the file. We need to support cases where we may not have bucket
level access but its still possible to download the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
