Displaying dataframes with many columns can crash the frontend

[akshayka]

Describe the bug

Rendering a dataframe with on the order of 1,000 columns makes my browser tab sluggish, and 10,000 columns crashes the browser tab, regardless of how many rows the dataframe has.

It may make sense to reintroduce a limit on the number of columns that ui.table can display.

Related to: #2899 (comment)

Environment

0.9.27, Chrome

Code to reproduce

# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "marimo",
#     "numpy==2.1.3",
#     "pandas==2.2.3",
# ]
# ///

import marimo

__generated_with = "0.9.27"
app = marimo.App(width="medium")


@app.cell
def __():
    import pandas as pd
    return (pd,)


@app.cell
def __():
    import numpy as np
    return (np,)


@app.cell
def __():
    import marimo as mo
    return (mo,)


@app.cell(hide_code=True)
def __(mo):
    mo.md("""Set cols to 10000 to crash the frontend. Number of rows appears to have no effect.""")
    return


@app.cell
def __(mo):
    rows = mo.ui.number(start=1, value=10, label="rows")
    columns = mo.ui.number(start=1, value=10, label="cols")

    mo.hstack([rows, columns], justify="start")
    return columns, rows


@app.cell
def __(columns, np, rows):
    data = np.zeros((rows.value, columns.value))
    return (data,)


@app.cell
def __(data, pd):
    df = pd.DataFrame(data, columns=[str(i) for i in range(data.shape[1])])
    return (df,)


@app.cell
def __(df):
    df
    return


if __name__ == "__main__":
    app.run()

[mscolnick]

do you experience this when you turn off column summaries?

[akshayka]

Yea, crashes with and without column summaries. Here's without:

[mscolnick]

We can:

1. fail above certain column count
2. cap at a limit,e.g. 100, but can be configurable by args
3. or virtualize columns (https://tanstack.com/table/v8/docs/framework/react/examples/virtualized-rows)

My preference is to start with (2).

[akshayka]

We used to do (2), without configuration, but removed the cap at the request of #2662

I am fine with reintroducing a cap. Instead of adding an arg to ui.table which we may remove once we do (3), I'd prefer to have the override through an environment variable (similar to MARIMO_OUTPUT_MAX_BYTES, MARIMO_TABLE_MAX_COLUMNS)

[alexscordellis]

I've tried to reproduce on my home machine using variants on your example above (e.g. I tried with random floats so the csvs were not the minimal size made up of repeated 0,) and I get similar behaviour to you: 1000s of columns becomes problematic

I noticed marimo already (0.9.27) chooses not to send the full dataframe and do the histograms if > 20k rows or > 40 columns so I tried hitting the worst case of 20k rows and 40 columns. This gives about 15MB of csv on the wire, which seems pretty reasonable, but rendering the histograms is very sluggish and has been occasionally enough to give me the "Not enough memory to open this page" chrome error. It seems to be OK the first time I render one such dataframe in a tab, but sometimes crashes later. I haven't been able to pin down exactly when it does. One possible trigger is switching tabs (in Chrome) away and back again. I speculate that the problematic notebooks that I and colleagues have seen at work are ones with this kind of size dataframe, perhaps with multiple such rendered in one notebook. Sorry this is a bit vague and I can't yet give you a deterministic repro.

[mscolnick]

@alexscordellis thanks for the notes. I'll be clamping columns at 50, but will be configurable. If there is a use-case for showing all of them, I can virtualized the table as well.

[akshayka]

Thanks for the additional context; I'll make a separate issue for the 20k rows and 40 columns, since it's possible it's caused by another issue (the column summaries).