Enhance Python bindings for ArcadeDB

[tae898]

Bring the latest changes to the python bindings to upstream main

[gemini-code-assist]

Summary of Changes

Hello @tae898, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on enhancing the Python bindings for ArcadeDB by expanding platform compatibility, significantly reducing package size, and improving the flexibility of JVM configuration. It also introduces new query language examples and robust dataset management tools, making the Python integration more powerful and user-friendly for a wider range of use cases and environments.

Highlights

• Expanded Platform Support: Added official support for Windows x86_64, increasing the number of supported platforms from 3 to 4. This is reflected in the README and build scripts.
• Significant Package Size Reduction: Optimized the bundled JRE size from ~249MB to ~60MB and the compressed wheel size from ~116MB to ~68MB, leading to a much smaller overall package footprint.
• Enhanced JVM Argument Management: Introduced programmatic control for JVM heap size and other arguments, allowing for more flexible configuration directly from Python code. Improved out-of-memory error messages to guide users on increasing heap size via jvm_kwargs.
• New Gremlin Query Examples: Added a comprehensive demonstrate_gremlin_queries function to the 02_social_network_graph.py example, showcasing various Gremlin traversals.
• Advanced Dataset Management and Generation: The download_data.py script was significantly expanded to support new dataset types (TPC-H, LDBC SNB, MSMARCO embeddings) and features like Stack Overflow vector generation, subset creation, and schema inference.
• Improved Vector Index Control: Added a build_graph_now parameter to create_vector_index to control eager graph building, providing more flexibility for performance tuning after bulk operations.
• Removed Server Mode Example: The 08_server_mode_rest_api.py example, which demonstrated running ArcadeDB as a server with concurrent HTTP clients, has been removed.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• bindings/python/.gitignore
  • Added local-jars/ to the ignore list for local build artifacts.
• bindings/python/Dockerfile.build
  • Updated the base image for the JRE builder from eclipse-temurin:25-jdk-jammy to amazoncorretto:25.
  • Installed findutils and binutils for necessary tooling in the JRE builder stage.
• bindings/python/README.md
  • Updated the test count from 258 to 260 passing tests.
  • Increased the reported number of supported platforms from 3 to 4, explicitly listing Windows x86_64.
  • Reduced the reported bundled JRE size from ~249MB to ~60MB.
  • Reduced the reported compressed wheel size from ~116MB to ~68MB.
  • Reduced the reported total uncompressed size from ~281MB to ~95MB.
  • Updated the 'Building from Source' section to include Windows in native build instructions.
• bindings/python/build.sh
  • Updated the Maven Docker image from maven:3.9-eclipse-temurin-25 to maven:3.9-amazoncorretto-25.
  • Added windows/amd64 to the list of supported platforms in the usage instructions.
  • Added a description for the optional JAR_LIB_DIR argument in the usage instructions.
  • Updated the package features to reflect support for 4 platforms instead of 3.
• bindings/python/examples/02_social_network_graph.py
  • Added a call to demonstrate_gremlin_queries(db) within the demonstrate_graph_queries function.
  • Implemented demonstrate_gremlin_queries function, showcasing various Gremlin graph traversal examples.
• bindings/python/examples/04_csv_import_documents.py
  • Updated documentation to recommend using the --heap-size argument instead of the ARCADEDB_JVM_ARGS environment variable for setting JVM heap.
  • Added a --heap-size command-line argument to the script's argument parser.
  • Modified the JVM heap configuration check to prioritize the --heap-size argument.
  • Passed jvm_kwargs to the arcadedb.create_database call to allow programmatic heap size configuration.
• bindings/python/examples/06_vector_search_recommendations.py
  • Updated documentation to recommend using the --heap-size argument for large datasets.
  • Modified import_from_jsonl function to accept an optional jvm_kwargs parameter.
  • Modified generate_embeddings function to accept an optional force_embed parameter.
  • Added a --heap-size command-line argument to the script's argument parser.
  • Passed jvm_kwargs to arcadedb.create_database and arcadedb.open_database calls.
  • Passed force_embed argument to generate_embeddings calls.
• bindings/python/examples/08_server_mode_rest_api.py
  • Removed the entire file, discontinuing the 08_server_mode_rest_api.py example.
• bindings/python/examples/README.md
  • Added a link to the download_data.md guide.
  • Removed an extraneous empty line.
  • Added a note for 04_csv_import_documents.py about downloading the MovieLens dataset.
  • Removed the entries for 07_stackoverflow_multimodel.py and 08_server_mode_rest_api.py examples.
• bindings/python/examples/download_data.py
  • Expanded the list of available datasets to include stackoverflow-tiny, stackoverflow-xlarge, stackoverflow-full (renamed from large), TPC-H benchmarks (tpch-sf1, sf10, sf100), LDBC SNB benchmarks (ldbc-snb-sf1, sf10, sf100), and MSMARCO v2.1 embeddings (msmarco-1m, 5m, 10m).
  • Added utility functions for cleaning directories (ensure_clean_dir), parsing TPC-H DDL (parse_tpch_ddl), iterating TPC-H records (_iter_tpch_records), converting TPC-H .tbl to CSV (convert_tpch_tbl_to_csv), and generating TPC-H CSV and schema (generate_tpch_csv_and_schema).
  • Added functions for inferring and merging data types (_infer_value_type, _merge_types), detecting CSV headers (_looks_like_header), and applying LDBC overrides (_apply_ldbc_overrides).
  • Implemented generate_ldbc_snb_schema for LDBC SNB datasets.
  • Modified download_movielens to use ensure_clean_dir for better cleanup.
  • Updated download_stackoverflow to support new dataset sizes (tiny, large, xlarge, full) and utilize ensure_clean_dir.
  • Added helper functions _write_stackoverflow_subset and _write_stackoverflow_subset_by_bytes for creating Stack Overflow dataset subsets.
  • Introduced functions for Stack Overflow vector generation: _iter_stackoverflow_rows, _clean_stackoverflow_text, _VectorShardWriter, and embed_stackoverflow_vectors.
  • Added download_tpch and download_ldbc_snb functions for generating benchmark datasets via Docker.
  • Implemented convert_msmarco_parquet_to_shards and download_msmarco for handling MSMARCO embedding datasets.
  • Added _count_xml_rows_fast and emit_stackoverflow_entity_counts for faster XML row counting and entity count reporting.
  • Updated the main function to handle the new dataset choices and associated download/generation logic.
  • Added new command-line arguments for Stack Overflow vector generation: --no-vectors, --vector-model, --vector-batch-size, --vector-shard-size, --vector-max-rows, --vector-gt-queries, and --vector-gt-topk.
  • Adjusted the sample_size logic for Stack Overflow verification to include xlarge and full datasets.
• bindings/python/pyproject.toml
  • Updated a comment for the py7zr dependency, removing specific example reference.
• bindings/python/src/arcadedb_embedded/core.py
  • Added build_graph_now parameter to create_vector_index to control eager graph construction, defaulting to True.
  • Modified create_vector_index to conditionally call index.build_graph_now() based on the new parameter.
  • Modified DatabaseFactory.__init__ to accept an optional jvm_kwargs dictionary, which is passed to start_jvm.
  • Updated create_database and open_database convenience functions to accept an optional jvm_kwargs parameter.
• bindings/python/src/arcadedb_embedded/importer.py
  • Imported the jpype module.
  • Modified Importer.__init__ to accept an optional jvm_kwargs dictionary, ensuring JVM is started with these parameters if not already configured.
  • Improved the out-of-memory error message in _import_using_java to suggest using jvm_kwargs for heap size configuration.
  • Modified import_xml function to accept an optional jvm_kwargs parameter, which is passed to the Importer constructor.
• bindings/python/src/arcadedb_embedded/jvm.py
  • Introduced a global _JVM_CONFIG variable to track the JVM's initial configuration.
  • Modified start_jvm to accept heap_size, disable_xml_limits, and jvm_args parameters for programmatic JVM configuration.
  • Added logic to prevent re-configuring the JVM if it's already started with different parameters, raising an ArcadeDBError.
  • Implemented _normalize_jvm_args to standardize JVM argument input.
  • Implemented _parse_memory_size to convert JVM memory strings (e.g., '8g') to bytes.
  • Implemented _dedupe_max_heap to handle multiple -Xmx arguments, retaining the largest valid value.
  • Updated _build_jvm_args to integrate the new programmatic parameters, handle XML entity limits, and ensure robust heap size management.
• bindings/python/src/arcadedb_embedded/server.py
  • Modified ArcadeDBServer.__init__ to accept an optional jvm_kwargs dictionary, which is passed to start_jvm.
• bindings/python/src/arcadedb_embedded/vector.py
  • Changed the default overquery_factor in find_nearest_approximate from 1 to 4 for potentially better recall.
  • Updated the docstring for build_graph_now for clarity.
• bindings/python/tests/README.md
  • Updated the reported test count from 258 to 260 tests.
• bindings/python/tests/test_concurrency.py
  • Imported mean from the statistics module and ArcadeDBError from arcadedb_embedded.exceptions.
  • Added test_oltp_mixed_workload_threads to test multi-threaded OLTP-style mixed read/write workloads within a single process.
• bindings/python/tests/test_importer.py
  • Added test_csv_complex_data_types to verify CSV import with various data types and edge cases.
  • Added test_csv_null_and_empty_values to test CSV handling of NULL, empty strings, and missing values.
  • Added test_csv_unicode_and_special_chars to ensure correct import of Unicode and special characters.
  • Reordered test_large_dataset_performance to appear after the new CSV tests.
• bindings/python/tests/test_jvm_args.py
  • Updated calls to _build_jvm_args to pass the new heap_size, disable_xml_limits, and jvm_args parameters.
  • Added test_dedupe_heap_keeps_max to verify that multiple -Xmx arguments correctly result in the maximum heap size being used.
• bindings/python/tests/test_vector.py
  • Added test_create_vector_index_build_graph_now_default_true to confirm eager graph building by default.
  • Added test_create_vector_index_build_graph_now_can_be_disabled to verify that eager graph building can be explicitly disabled.
  • Marked several approximate search tests (test_lsm_vector_search_approximate_product, test_lsm_vector_search_approximate_typeindex, test_lsm_vector_search_approximate_persistence) with @pytest.mark.skip.
  • Modified the filler vector generation in test_lsm_vector_search_approximate_overquery to ensure unique vectors and better test conditions.
  • Adjusted the assertion threshold in test_lsm_vector_search_approximate_overquery from 0.9 to 0.899.

Ignored Files

• Ignored by pattern: .github/workflows/** (2)
  • .github/workflows/test-python-bindings.yml
  • .github/workflows/test-python-examples.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 67056af.

🟢 All jobs passed!

But CI Insights is watching 👀

[gemini-code-assist]

Code Review

The pull request significantly enhances the ArcadeDB Python bindings by adding support for Windows, expanding dataset options in examples, and providing more robust JVM configuration options. The introduction of programmatic heap size control and improved XML limit handling are valuable additions. However, there are some copy-paste errors in the new Gremlin example and potential issues with JVM argument parsing when spaces are involved.

[tae898]

@robfrank

I removed a couple of python examples, since they were kinda messy. I'm adding cleaner examples and will soon make a new PR for them.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.86%. Comparing base (fcd6a39) to head (67056af).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3466      +/-   ##
==========================================
- Coverage   62.86%   62.86%   -0.01%     
==========================================
  Files        1296     1296              
  Lines       85831    85831              
  Branches    17641    17641              
==========================================
- Hits        53960    53956       -4     
+ Misses      24181    24178       -3     
- Partials     7690     7697       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[robfrank]

good job @tae898 , thanks a lot!