Re: Implementing foreign data wrappers and avoiding n+1 querying

2022-12-23 Thread Brad White
We had a similar situation in a completely different context. Our eventual solution was to fire off a request as soon as one came in. Then we batched further requests until the first returned. Whenever a request returned, we sent any pending requests. Any single request not sent immediately was slo

Re: Implementing foreign data wrappers and avoiding n+1 querying

2022-12-22 Thread David Gilman
I apologize that my post was not super clear, I am thinking about implementing a fdw from scratch, and the target database is one of those NoSQL databases where you have to send JSON over a HTTP connection for each query. I have reviewed the postgres fdw code to see how it works and to see what's

Re: Implementing foreign data wrappers and avoiding n+1 querying

2022-12-21 Thread David Rowley
On Thu, 22 Dec 2022 at 13:31, David Gilman wrote: > > When a fdw table participates in query planning and finds itself as > part of a join it can output a parameterized path. If chosen, Postgres > will dutifully call the fdw over and over via IterateForeignScan to > fetch matching tuples. Many fdw

Implementing foreign data wrappers and avoiding n+1 querying

2022-12-21 Thread David Gilman
When a fdw table participates in query planning and finds itself as part of a join it can output a parameterized path. If chosen, Postgres will dutifully call the fdw over and over via IterateForeignScan to fetch matching tuples. Many fdw extensions do network traffic, though, and it would be benef