apoorvmintri commented on issue #1592: URL: https://github.com/apache/camel-kafka-connector/issues/1592#issuecomment-1905444125
@oscerd I believe I have found the issue/workaround. I will paste images here rather to make things more readable. FYI, I am using redpanda console to make life easier. ## Using v3.18.2 Using the connector config described in the problem above. I get the following info in my Kafka - #### Value: <img width="295" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/11086040/293e4fae-79ce-49b3-a2f0-a409590c0202";> #### Header: <img width="605" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/11086040/c1fd1bb5-a194-4d62-97fe-d4628a9c44e0";> #### Synopsis: As you can see RecordId does not exist in the value or the header. Please note the "Account Name" I provided in Salesforce was "Using 3.18.2 without rawPayload" - I will come back to this. ## Using 3.18.2 with rawPayload=true I tried setting `rawPayload=true` through both `camel.component.salesforce` & `camel.source.endpoint`, but that didnt work. With component.salesforce, I also had to provide loginUrl, clientId etc as well on top of the kamelet requirement as well - otherwise it would give an exception. ## Using 0.11.5 I decided to use this version to avoid kamelet altogether. #### Connector Config: <img width="421" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/11086040/657c8b3b-0d84-4aae-b09b-d3ade57e4e50";> Please note the above image has `"camel.component.salesforce.rawPayload": "true"`. **I did try both true and false (default) for this property and this was the result.** #### Value (rawPayload=false): <img width="292" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/11086040/d2ee3111-49b2-49da-8575-3fc14e94b911";> #### Value (rawPayload=true): <img width="497" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/11086040/246739d3-49e3-40f9-b21a-e98c55c08c3f";> The header is the same as before. But as you can see recordIds come with the value payload when rawPayload is set to true. Please also note I had to use the `ByteArrayConverter `with `marshal=json-jackson` ## Workaround in v3.18.2 There is no way I could find to set `rawPayload=true` via kamelet or any other property as mentioned above. Hence I decided to force this property into the endpoint via the topic name - `"camel.kamelet.salesforce-source.topicName": "subscribe:/data/AccountChangeEvent?rawPayload=true&"` **NOTE:** The `&` is very important at the end of the topicName otherwise when the endpoint is concatenated in the end with other properties, it adds `?` again, which will raise an exception. By including this property via the topicName, I was able to receive the value as below. #### Value (enforced rawPayload=true)  ## Other Issues with v3.18.2 Workaround The final workaround does not give a proper json rather a string with `\` escape character. I was unable to use marshal as it would throw an exception with v3.18.2. This could be managed via the pipeline / Kafka consumer. I also tried using SMT as follows, as mentioned by you in another github issue related to telegram I believe - <img width="398" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/11086040/8c0752b1-e3ee-44fc-8a84-ef410e63584f";> Unfortunately, this converts the value payload to a JWT style, which then needs to be decoded. <img width="381" alt="image" src="https://github.com/apache/camel-kafka-connector/assets/11086040/27e778b6-ca92-4bc8-b7fd-b63c555d00a6";> ## SUMMARY If developers want to use v3.18.2 the above workaround will do the job as intended but not ideal i.e. not clean. Otherwise we need to revert to v0.11.5 for now. Overall, I believe `kamelet` is not ready for salesforce and required properties like `rawPayload` is missing. FYI, these issues are pertaining to our use case and we aren't using all the potential properties. There might be other issues with kamelet when used with salesforce as source. **Also, we haven't tested salesforce as sink with v3.18.2 yet.** Which might reveal other issues. Currently, I need to discuss with my team on testing sink versions for smooth sailing before finalizing the actual connector versions. I'll try to keep you posted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@camel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
