djanand commented on code in PR #1883:
URL:
https://github.com/apache/datafusion-ballista/pull/1883#discussion_r3584708195
##########
python/python/ballista/jupyter.py:
##########
@@ -108,8 +108,19 @@ def decorator(func):
return decorator
+from datafusion.dataframe_formatter import (
+ configure_formatter,
+ get_formatter,
+ set_formatter,
+)
+
from .extension import BallistaSessionContext, DistributedDataFrame
+# Default number of rows rendered for a ``%%sql`` cell when ``--limit`` is not
+# given. This caps only the display (via datafusion's HTML formatter); the
+# underlying result keeps all of its rows.
+DEFAULT_DISPLAY_LIMIT = 50
Review Comment:
The `--limit` is a display cap atm. It only changes how many rows render. It
does not reduce compute or collection i.e the full result is still
materialized. On datafusion side it seems we already have ```df.limit(n,
offset), df.head(n)``` which pushes the limit into the plan. I think you mean
that setting `%%sql --limit x` should push it to the execution layer rather
than just display side. right ? I can look into that side lmk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
