(TIL) Spark: Orderby Partitioning

less than 1 minute read

Remember that orderBy uses the number of partitions specified by spark.conf.get("spark.sql.shuffle.partitions"). The default for this is 200. Can change manually to say 8 by using:

spark.conf.set("spark.sql.shuffle.partitions", "8")`

Tags: ,

Categories:

Updated:

Comments