(TIL) Spark: Orderby Partitioning
Remember that orderBy uses the number of partitions specified by spark.conf.get("spark.sql.shuffle.partitions"). The default for this is 200. Can change manu...
Recent Posts
SHAttered
Awesome work to demonstrate how to deliberately cause a SHA-1 collision.
(TIL) Spark: Materializing and Unpersisting Cache
If you want to reuse a dataframe df without having to recreate it, you can use df.cache() to tell Spark to keep the dataframe in memory.
Unlearning descriptive statistics
Top tips on better descriptive statistics:
The legends of mathematics that almost never were
It’s always made me sad when people tell me they dislike mathematics. I always wonder if it was the subject or their teachers that they disliked…