Question 1mediummulti select
Read the full Using Spark Connect explanation →Databricks-Spark-Assoc Using Spark Connect • Complete Question Bank
Complete Databricks-Spark-Assoc Using Spark Connect question bank — all 0 questions with answers and detailed explanations.
spark = SparkSession.builder.remote("sc://my-workspace.cloud.databricks.com:443/;token=dapi12345;clusterId=0101-123456-abc789").getOrCreate()
df = spark.read.table("default.sales")
def my_transform(batch_df):
return batch_df.filter(batch_df.amount > 100)
# Operation fails here
df.foreachBatch(my_transform)Refer to the exhibit.
Traceback (most recent call last): File "app.py", line 12, in <module> df = spark.read.table("default.sales") File "/opt/spark/python/pyspark/sql/session.py", line 314, in table
return DataFrame(self._client.execute_plan(parser.parse_table(name))))
File "/opt/spark/python/pyspark/sql/connect/client/core.py", line 112, in execute_plan(y+"sessionID"), grpc.RpcError: StatusCode.UNAVAILABLE
An engineer attempts to run a PySpark script using Spark Connect but encounters the traceback shown above. What is the most likely root cause of this execution failure?