Skip to content

fix: Ensure only tables or aliases that exist are projected #52

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 29, 2024
Merged

fix: Ensure only tables or aliases that exist are projected #52

merged 1 commit into from
Oct 29, 2024

Conversation

@peasee
Copy link

@peasee peasee commented Oct 28, 2024

Rationale for this change

Ensures that projections only contain table references to tables or subqueries that actually exist in the query.

E.g. SELECT ta.j1_id FROM (SELECT * FROM j1 ta) is rewritten to SELECT j1_id FROM (SELECT * FROM j1 ta) because ta does not exist in the outer projection (outside of the subquery).

sgrebnov
sgrebnov reviewed Oct 29, 2024
View reviewed changes
sgrebnov
sgrebnov reviewed Oct 29, 2024
View reviewed changes
@sgrebnov
Copy link

sgrebnov commented Oct 29, 2024

I think we should collaborate with datafusion community on this so propose to merge this as-is and raise upstream PR where we will adjust/improve approach if needed and switch during upgrade.

@peasee peasee merged commit 60ca421 into spiceai-42 Oct 29, 2024
@peasee peasee deleted the fix/invalid-table-reference branch October 29, 2024 05:58
phillipleblanc pushed a commit that referenced this pull request Nov 12, 2024
@peasee @phillipleblanc
fix: Ensure only tables or aliases that exist are projected (#52)
5e55154
phillipleblanc pushed a commit that referenced this pull request Nov 12, 2024
@peasee @phillipleblanc
fix: Ensure only tables or aliases that exist are projected (#52)
a29d4d0
phillipleblanc pushed a commit that referenced this pull request Apr 8, 2025
@peasee @phillipleblanc
fix: Ensure only tables or aliases that exist are projected (#52)
e151863 
fix: More dangling references (#54)

* fix: More dangling references

* test: Add tests for remove_dangling_identifiers

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.
phillipleblanc pushed a commit that referenced this pull request Apr 25, 2025
@peasee @phillipleblanc
fix: Ensure only tables or aliases that exist are projected (#52)
b9e9c27 
fix: More dangling references (#54)

* fix: More dangling references

* test: Add tests for remove_dangling_identifiers

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.
sgrebnov pushed a commit that referenced this pull request May 22, 2025
@peasee @sgrebnov
fix: Ensure only tables or aliases that exist are projected (#52)
de4db8c 
fix: More dangling references (#54)

* fix: More dangling references

* test: Add tests for remove_dangling_identifiers

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.

# Conflicts:
#	datafusion/sql/src/unparser/ast.rs
#	datafusion/sql/tests/cases/plan_to_sql.rs
sgrebnov pushed a commit that referenced this pull request May 26, 2025
@peasee @sgrebnov
fix: Ensure only tables or aliases that exist are projected (#52)
2fb089e 
fix: More dangling references (#54)

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.
kczimm pushed a commit that referenced this pull request Aug 19, 2025
@peasee @kczimm
fix: Ensure only tables or aliases that exist are projected (#52)
50e46d9 
fix: More dangling references (#54)

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.
kczimm pushed a commit that referenced this pull request Aug 21, 2025
@peasee @kczimm
fix: Ensure only tables or aliases that exist are projected (#52)
7151f4d 
fix: More dangling references (#54)

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.
Jeadie pushed a commit that referenced this pull request Sep 9, 2025
@peasee @Jeadie
fix: Ensure only tables or aliases that exist are projected (#52)
2836e04 
fix: More dangling references (#54)

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.
Jeadie pushed a commit that referenced this pull request Sep 12, 2025
@peasee @Jeadie
fix: Ensure only tables or aliases that exist are projected (#52)
ce559df 
fix: More dangling references (#54)

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.
@Jeadie Jeadie mentioned this pull request Sep 25, 2025
Closed
28 tasks
peasee added a commit that referenced this pull request Oct 27, 2025
@peasee
fix: Ensure only tables or aliases that exist are projected (#52)
8169342 
fix: More dangling references (#54)

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.
peasee added a commit that referenced this pull request Oct 27, 2025
@peasee @phillipleblanc
@kczimm @sgrebnov @Jeadie @krinart @mach-kernel
feat: Update for Datafusion v50 (#109)
cd6b2f8 
* fix: Ensure only tables or aliases that exist are projected (#52)
fix: More dangling references (#54)

UPSTREAM NOTE: This PR was attempted to be upstreamed in apache#13405 - but it was not accepted due to the complexity it brought. Phillip needs to figure out what a good solution that solves our problem and can be upstreamed is.

* Support for metadata columns (`location`, `size`, `last_modified`)  in ListingTableProvider (#74)

UPSTREAM NOTE: This PR was attempted to be upstreamed but was not accepted. Needs to be applied manually
apache#15181

* Infer placeholder datatype for `Expr::InSubquery` (#80)

UPSTREAM NOTE: Upstream PR has been created but not merged yet. Should be available in DF49
apache#15980

* Infer placeholder datatype after `LIMIT` clause as `DataType::Int64` (#81)

UPSTREAM NOTE: Upstream PR has been created but not merged yet. Should be available in DF49
apache#15980

* Do not double alias Exprs

UPSTREAM NOTE: This was attempted to be fixed with
apache#15008 but was closed

This is the tracking issue on DataFusion:
apache#14895
Do not double alias Exprs

* Add prefix to location metadata column (#82)

UPSTREAM NOTE: This will not be upstreamed as is.

* Infer placeholder types for CASE expressions (#87)

UPSTREAM NOTE: This has not been submitted upstream yet.

* Expand `infer_placeholder_types` to infer all possible placeholder types based on their expression (#88)

UPSTREAM NOTE: This has not been submitted upstream yet.

* Fix `Expr::infer_placeholder_types` inference to not fail (#89)

UPSTREAM NOTE: This has not been submitted upstream yet.

* cherry-pick parquet patch (#94)

* Fix array types coercion: preserve child element nullability for list types (#96)

UPSTREAM NOTE: This was submitted upstream and should be available in DF50

apache#17306

* Expand `infer_placeholder_types` to infer all possible placeholder types based on their expression (#88)

UPSTREAM NOTE: This has not been submitted upstream yet.

* do not enforce type guarantees on all Expr traversed in infer_placeholder_types (#97)

* Use UDTF function args in `LogicalPlan::TableScan` name (#98)

* use UDTF function args in LogicalPlan::TableScan name

* update test snapshots

* Implement timestamp_cast_dtype for SqliteDialect (#99)

* Use text for sqlite timestamp

* Add test

* Custom timestamp format for DuckDB (#102)

* Revert "cherry-pick parquet patch (#94)"

This reverts commit d780cc2.

* Support ExprNamed arguments to Scalar UDFs (#104)

* support ExprNamed until 17379 ships

* add same exprnamed lifting to udtf

* resolve projection against `ListingTable` table_schema incl. partition columns (#106)

* fix: Ensure ListingTable partitions are pruned when filters are not used (#108)

* fix: Prune partitions when no filters are defined

* fix: Backport for DF49:

* review: Address comments

* FileScanConfig: Preserve schema metadata across serde boundary (#107)

* FileScanConfig: preserve schema metadata across serde boundary

* add test

* Merge conflict fixes

UPSTREAM NOTE: this should not be upstreamed. This contains conflict fixes from various cherry-picks and differences in v50.

* update arrow-rs fork

UPSTREAM NOTE: this should not be upstreamed

---------

Co-authored-by: Phillip LeBlanc <[email protected]>
Co-authored-by: Kevin Zimmerman <[email protected]>
Co-authored-by: sgrebnov <[email protected]>
Co-authored-by: jeadie <[email protected]>
Co-authored-by: Jack Eadie <[email protected]>
Co-authored-by: Viktor Yershov <[email protected]>
Co-authored-by: Viktor Yershov <[email protected]>
Co-authored-by: David Stancu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sgrebnov sgrebnov sgrebnov approved these changes

Assignees

No one assigned

Labels

needs_upstream

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@peasee @sgrebnov