[
https://issues.apache.org/jira/browse/SPARK-53317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-53317:
-----------------------------------
Labels: pull-request-available (was: )
> [spark-connect-rust] Registry a DataFrame from Polars
> -----------------------------------------------------
>
> Key: SPARK-53317
> URL: https://issues.apache.org/jira/browse/SPARK-53317
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-0.1.0
> Reporter: Steve Russo
> Priority: Major
> Labels: pull-request-available
>
> The rust client only supports the ability to `create_dataframe` from
> ArrowBatch. To increase the usability of the client an additional method
> should be create to allow for the creation of a dataframe from a polars
> dataframe.
> Example provided below.
>
> {code:java}
> use polars::prelude::*;
> use spark_connect_rs::{SparkSession, SparkSessionBuilder};
> #[tokio::main]
> async fn main() -> Result<(), Box<dyn std::error::Error>> {
> let spark: SparkSession =
> SparkSessionBuilder::remote("sc://127.0.0.1:15002/")
> .build()
> .await?;
> let s1 = Series::new("Name", &["Alice", "Bob", "Charlie"]);
> let s2 = Series::new("Age", &[30, 24, 35]);
> let s3 = Series::new("City", &["New York", "London", "Paris"]);
> let df: DataFrame = DataFrame::new(vec![s1, s2, s3])?;
> // from_polars
> let spark_df = spark.from_polars(df)
> spark_df.show(Some(4), None, None).await?;
> Ok(())
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
