Blog post about processing larger than memory queries

[alamb]

Is your feature request related to a problem or challenge?

• part of [Epic] DataFusion Blogs #14836

As part of trying to help people understand how to tune memory related settings in #17069, @2010YOUY01 pointed out #17069 (comment)

Let's make it concise now, I think adding a few more sentences of explanation might actually confuse those without the background knowledge. A tutorial-style doc is still needed to describe the full picture.

I agree and I think a blog post explaining how DataFusion processes larger than memory data sets would be really helpful to give people the broader context

Describe the solution you'd like

I suggest a blog on https://datafusion.apache.org/blog/ (make a PR to https://github.com/alamb/datafusion-site)

Describe alternatives you've considered

Some ideas

1. Background on memory usage in DataFusion (the memory manager model and what takes large amounts of memory) -- can probably copy/paste from https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html#memory-management-overview
2. Memory Consumers Describe at a high level what consumes most memory in a plan (group by hash table, sorting, etc)
3. Optimizations that DataFusion does to try and avoid needing memory (e.g. take advantage of pre-xisting sort orders, topk, etc)
4. Spilling Sort: Provide an overview of the main spilling sort algorithm (sort in memory, spill to disk, merge pre-sorted runs, etc)
5. Using Spilling Sort in Group By: Explain that the grouping operation uses the same underlying building block
6. Call for help: 🎣 for people to figure out how to add spilling for joins and make the spilling hash faster

Additional context

I am very happy to help write this

[alamb]

@ding-young maybe you want to try out writing blogs as part of GSOC ?

cc @2010YOUY01

[2010YOUY01]

Thank you for posting the issue for blog idea!

I wish this blog to be more of explaining how spilling execution work internally, and add more example codes to help potential users get started.
Although @rluvaton @ding-young and others has done a lot to fix existing bugs, we still have a long way to go to stabilize this feature (i.e. less buggy, make it faster), we should definitely admit that and 🎣 for help in this 1st blog.

I plan to work with @ding-young for this blog post.

[alamb]

In my opinion the best blog posts follow the following formula, for your consideration

1. Explain why the feature is relevant (so the target audience can easily tell the blog is for them)
2. Teach something interesting about database technology in general
3. Teach something specific about DataFusion

That results in a wider audience than something that only focuses on DataFusion (and therefore also brings more people into the DataFusion comunity)