Refactor Guides Docs

[scott-routledge2]

Changes included in this PR

Adds Bodo DataFrames Guide to docs/guides page and notebook to examples/#Tutorials

Refactors guides to be in docs/guides folders. Moves all JIT guides to a subdirectory of guides, fixes links etc.

The important changes are in docs/docs/guides/dataframes/dataframes_intro.md and mkdocs.yaml for the updated side bar.

Testing strategy

Ran notebook in pixi env

User facing changes

Docs

Checklist

• Pipelines passed before requesting review. To run CI you must include [run CI] in your commit message.
• I am familiar with the Contributing Guide
• I have installed + ran pre-commit hooks.

[DrTodd13]

I wonder if we should mention the ability to execute an expensive plan resulting in little data that we know will be reused later?

[scott-routledge2]

I think this page was unreachable. Is there any info here we want to add to another page/section?

[scott-routledge2]

The #'s in the link were leading to the wrong page.

[scott-routledge2]

I wonder if we should mention the ability to execute an expensive plan resulting in little data that we know will be reused later?

Is this referring to CTE? I think that would be nice to have. Do you have a good example?

[DrTodd13]

I wonder if we should mention the ability to execute an expensive plan resulting in little data that we know will be reused later?

Is this referring to CTE? I think that would be nice to have. Do you have a good example?

We can detect and optimize CTE within one query. The issue is when you have repeated computations across queries and so our CTE optimization can do nothing in that case. This is the reason for persist in dask. So, something like this:

import bodo.pandas as bd

df = bd.read_parquet("bigdata.parquet")

# Expensive transformation (e.g. multi-join + groupby)
expensive = df.merge(other, on="key").groupby("category").agg({"value": "sum"})

expensive.execute_plan()

# Without execute_plan above, this would trigger execution of expensive.
print(expensive + 7)

# Without execute_plan above, this would also trigger a second execution of expensive.
result = print(some_other_computation(expensive[expensive["value"] > 1000]))

[ehsantn]

Thanks @scott-routledge2! This is a big improvement. See minor comments below.

[DrTodd13]

Thanks for the detailed work. Looks good.

[DrTodd13]

I tend to do bodo.pandas.DataFrame(pandas_dataframe) for conversion. Do we want to mention this option?

[scott-routledge2]

I thought bodo.pandas.DataFrame wasn't stable but if it is just a wrapper around from_pandas then maybe we should. @ehsantn thoughts?

[scott-routledge2]

It might be simpler to just use bodo.pandas.DataFrame in some of the examples that construct dataframes, so I'll include it.

[ehsantn]

It just calls from_pandas so whichever looks better to you should be fine.