Update dependency datasets to v2.21.0 #29

konflux-internal-p02 · 2024-11-23T12:08:57Z

This PR contains the following updates:

Package	Update	Change
datasets	minor	`==2.15.0` -> `==2.21.0`

Warning

Some dependencies could not be looked up. Check the warning logs for more information.

Release Notes

huggingface/datasets (datasets)

`v2.21.0`

Compare Source

Features

Support pyarrow large_list by @albertvillanova in https://github.com/huggingface/datasets/pull/7019

Support Polars round trip:

import polars as pl
from datasets import Dataset

df1 = pl.from_dict({"col_1": [[1, 2], [3, 4]]}
df2 = Dataset.from_polars(df).to_polars()
assert df1.equals(df2)

Important

Remove default trust_remote_code=True by @lhoestq in https://github.com/huggingface/datasets/pull/6954
- datasets with a python loading script now require passing trust_remote_code=True to be used

Datasets features

[Resumable IterableDataset] Add IterableDataset state_dict by @lhoestq in https://github.com/huggingface/datasets/pull/6658

checkpoint and resume an iterable dataset (e.g. when streaming):

>>> iterable_dataset = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)
>>> for idx, example in enumerate(iterable_dataset):
...     print(example)
...     if idx == 2:
...         state_dict = iterable_dataset.state_dict()
...         print("checkpoint")
...         break
>>> iterable_dataset.load_state_dict(state_dict)
>>> print(f"restart from checkpoint")
>>> for example in iterable_dataset:
...     print(example)

Returns:

{'a': 0}
{'a': 1}
{'a': 2}
checkpoint
restart from checkpoint
{'a': 3}
{'a': 4}
{'a': 5}

General improvements and bug fixes

Add docs about the CLI by @albertvillanova in https://github.com/huggingface/datasets/pull/6831
Remove token arg from CLI examples by @albertvillanova in https://github.com/huggingface/datasets/pull/6839
Allow deleting a subset/config from a no-script dataset by @albertvillanova in https://github.com/huggingface/datasets/pull/6820
Fix line-endings in tests on Windows by @albertvillanova in https://github.com/huggingface/datasets/pull/6857
Fix CI by temporarily pinning huggingface-hub < 0.23.0 by @albertvillanova in https://github.com/huggingface/datasets/pull/6861
Fix dataset name for community Hub script-datasets by @albertvillanova in https://github.com/huggingface/datasets/pull/6855
Update tqdm >= 4.66.3 to fix vulnerability by @albertvillanova in https://github.com/huggingface/datasets/pull/6870
Fix download for dict of dicts of URLs by @albertvillanova in https://github.com/huggingface/datasets/pull/6871
Set dev version by @albertvillanova in https://github.com/huggingface/datasets/pull/6873
Shorten long logs by @lhoestq in https://github.com/huggingface/datasets/pull/6875
Support jax 0.4.27 in CI tests by @albertvillanova in https://github.com/huggingface/datasets/pull/6885
Close gzipped files properly by @lhoestq in https://github.com/huggingface/datasets/pull/6893
Make CLI convert_to_parquet not raise error if no rights to create script branch by @albertvillanova in https://github.com/huggingface/datasets/pull/6902
Fix YAML error in README files appearing on GitHub by @albertvillanova in https://github.com/huggingface/datasets/pull/6898
Document that to_json defaults to JSON Lines by @albertvillanova in https://github.com/huggingface/datasets/pull/6895
Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset by @albertvillanova in https://github.com/huggingface/datasets/pull/6883
Create function to convert to parquet by @albertvillanova in https://github.com/huggingface/datasets/pull/6878
Update features.py to avoid bfloat16 unsupported error by @skaulintel in https://github.com/huggingface/datasets/pull/6607
Fix decoding multi part extension by @lhoestq in https://github.com/huggingface/datasets/pull/6904
Use pandas ujson in JSON loader to improve performance by @albertvillanova in https://github.com/huggingface/datasets/pull/6874
Update requests >=2.32.1 to fix vulnerability by @albertvillanova in https://github.com/huggingface/datasets/pull/6909
Fix wrong type hints in data_files by @albertvillanova in https://github.com/huggingface/datasets/pull/6910
Remove dead code for non-dict data_files from packaged modules by @albertvillanova in https://github.com/huggingface/datasets/pull/6911
Support fsspec 2024.5.0 by @albertvillanova in https://github.com/huggingface/datasets/pull/6921
Remove torchaudio remnants from code by @albertvillanova in https://github.com/huggingface/datasets/pull/6922
[WebDataset] Add .pth support for torch tensors by @lhoestq in https://github.com/huggingface/datasets/pull/6920
Unpin hfh by @lhoestq in https://github.com/huggingface/datasets/pull/6876
Preserve JSON column order and support list of strings field by @albertvillanova in https://github.com/huggingface/datasets/pull/6914
[WebDataset] Support compressed files by @lhoestq in https://github.com/huggingface/datasets/pull/6931
update ci user by @lhoestq in https://github.com/huggingface/datasets/pull/6933
Revert ci user by @lhoestq in https://github.com/huggingface/datasets/pull/6934
Fix NonMatchingSplitsSizesError/ExpectedMoreSplits when passing data_dir/data_files in no-code Hub datasets by @albertvillanova in https://github.com/huggingface/datasets/pull/6925
Set dev version by @albertvillanova in https://github.com/huggingface/datasets/pull/6944
Update yanked version of minimum requests requirement by @albertvillanova in https://github.com/huggingface/datasets/pull/6945
Re-enable import sorting disabled by flake8:noqa directive when using ruff linter by @albertvillanova in https://github.com/huggingface/datasets/pull/6946
Update dataset_dict.py by @Arunprakash-A in https://github.com/huggingface/datasets/pull/6932
Update process.mdx: Code Listings Fixes by @FadyMorris in https://github.com/huggingface/datasets/pull/6928
Fix small typo by @marcenacp in https://github.com/huggingface/datasets/pull/6955
update docs on N-dim arrays by @lhoestq in https://github.com/huggingface/datasets/pull/6956
Fix typos in docs by @albertvillanova in https://github.com/huggingface/datasets/pull/6957
Validate config name and data_files in packaged modules by @albertvillanova in https://github.com/huggingface/datasets/pull/6915
Add support for categorical/dictionary types by @EthanSteinberg in https://github.com/huggingface/datasets/pull/6892
feat(ci): add trufflehog secrets detection by @McPatate in https://github.com/huggingface/datasets/pull/6960
Better error handling in dataset_module_factory by @Wauplin in https://github.com/huggingface/datasets/pull/6959
Move info_utils errors to exceptions module by @albertvillanova in https://github.com/huggingface/datasets/pull/6952
fix(ci): remove unnecessary permissions by @McPatate in https://github.com/huggingface/datasets/pull/6962

New Contributors

@skaulintel made their first contribution in https://github.com/huggingface/datasets/pull/6607
@Arunprakash-A made their first contribution in https://github.com/huggingface/datasets/pull/6932
@FadyMorris made their first contribution in https://github.com/huggingface/datasets/pull/6928
@marcenacp made their first contribution in https://github.com/huggingface/datasets/pull/6955
@EthanSteinberg made their first contribution in https://github.com/huggingface/datasets/pull/6892
@McPatate made their first contribution in https://github.com/huggingface/datasets/pull/6960

Full Changelog: huggingface/datasets@2.19.0...2.20.0

`v2.19.2`

Compare Source

Bug fixes

Make CLI convert_to_parquet not raise error if no rights to create script branch by @albertvillanova in https://github.com/huggingface/datasets/pull/6902
Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset by @albertvillanova in https://github.com/huggingface/datasets/pull/6883
Update requests >=2.32.1 to fix vulnerability by @albertvillanova in https://github.com/huggingface/datasets/pull/6909
Fix NonMatchingSplitsSizesError/ExpectedMoreSplits when passing data_dir/data_files in no-code Hub datasets by @albertvillanova in https://github.com/huggingface/datasets/pull/6925

Full Changelog: huggingface/datasets@2.19.1...2.19.2

`v2.19.1`

Compare Source

Bug fixes

Fix download for dict of dicts of URLs by @albertvillanova in https://github.com/huggingface/datasets/pull/6871

Full Changelog: huggingface/datasets@2.19.0...2.19.1

`v2.19.0`

Compare Source

Dataset Features

Add Polars compatibility by @psmyth94 in https://github.com/huggingface/datasets/pull/6531

convert to a Polars dataframe using .to_polars();

import polars as pl
from datasets import load_dataset
ds = load_dataset("DIBT/10k_prompts_ranked", split="train")
ds.to_polars() \
    .groupby("topic") \
    .agg(pl.len(), pl.first()) \
    .sort("len", descending=True)

Use Polars formatting to return Polars objects when accessing a dataset:
```
ds = ds.with_format("polars")
ds[:10].group_by("kind").len()
```

Add fsspec support for to_json, to_csv, and to_parquet by @alvarobartt in https://github.com/huggingface/datasets/pull/6096

Save on HF in any file format:

ds.to_json("hf://datasets/username/my_json_dataset/data.jsonl")
ds.to_csv("hf://datasets/username/my_csv_dataset/data.csv")
ds.to_parquet("hf://datasets/username/my_parquet_dataset/data.parquet")

Add mode parameter to Image feature by @mariosasko in https://github.com/huggingface/datasets/pull/6735
- Set images to be read in a certain mode like "RGB"
```
dataset = dataset.cast_column("image", Image(mode="RGB"))
```
Add CLI function to convert script-dataset to Parquet by @albertvillanova in https://github.com/huggingface/datasets/pull/6795
- run command to open a PR in script-based dataset to convert it to Parquet:
  datasets-cli convert_to_parquet <dataset_id>
Add Dataset.take and Dataset.skip by @lhoestq in https://github.com/huggingface/datasets/pull/6813
- same as IterableDataset.take and IterableDataset.skip
```
ds = ds.take(10)  # take only the first 10 examples
```

General improvements and bug fixes

Bump huggingface-hub lower version to 0.21.2 by @albertvillanova in https://github.com/huggingface/datasets/pull/6713
fix CastError pickling by @lhoestq in https://github.com/huggingface/datasets/pull/6712
Expand no-code dataset info with datasets-server info by @mariosasko in https://github.com/huggingface/datasets/pull/6714
Fix sliced ConcatenationTable pickling with mixed schemas vertically by @lhoestq in https://github.com/huggingface/datasets/pull/6715
Fix concurrent script loading with force_redownload by @lhoestq in https://github.com/huggingface/datasets/pull/6718
get_dataset_default_config_name docstring by @lhoestq in https://github.com/huggingface/datasets/pull/6723
Deprecate Beam API and download from HF GCS bucket by @mariosasko in https://github.com/huggingface/datasets/pull/6474
Deprecate Pandas builder by @mariosasko in https://github.com/huggingface/datasets/pull/6730
Using a registry instead of calling globals for fetching feature types by @psmyth94 in https://github.com/huggingface/datasets/pull/6727
Update torch_formatter.py by @VarunNSrivastava in https://github.com/huggingface/datasets/pull/6402
Improve default patterns resolution by @mariosasko in https://github.com/huggingface/datasets/pull/6704
Transpose images with EXIF Orientation tag by @mariosasko in https://github.com/huggingface/datasets/pull/6739
Fix missing download_config in get_data_patterns by @lhoestq in https://github.com/huggingface/datasets/pull/6742
Allow null values in dict columns by @mariosasko in https://github.com/huggingface/datasets/pull/6743
Fix fsspec tqdm callback by @lhoestq in https://github.com/huggingface/datasets/pull/6749
chore(deps): bump fsspec by @shcheklein in https://github.com/huggingface/datasets/pull/6747
Fix offline mode with single config by @lhoestq in https://github.com/huggingface/datasets/pull/6741
Remove deprecated code by @Wauplin in https://github.com/huggingface/datasets/pull/6761
fixing the issue 6755(small typo) by @JINO-ROHIT in https://github.com/huggingface/datasets/pull/6767
remove_columns/rename_columns doc fixes by @mariosasko in https://github.com/huggingface/datasets/pull/6772
Fix CI by @mariosasko in https://github.com/huggingface/datasets/pull/6780
rename datasets-server to dataset-viewer by @severo in https://github.com/huggingface/datasets/pull/6785
Install dependencies with uv in CI by @mariosasko in https://github.com/huggingface/datasets/pull/6779
Fix cache conflict in _check_legacy_cache2 by @lhoestq in https://github.com/huggingface/datasets/pull/6792
Fix typo in docs (upload CLI) by @Wauplin in https://github.com/huggingface/datasets/pull/6802
fix DatasetBuilder._split_generators incomplete type annotation by @JonasLoos in https://github.com/huggingface/datasets/pull/6799
#6791 Improve type checking around FAISS by @Dref360 in https://github.com/huggingface/datasets/pull/6803
Fix --repo-type order in cli upload docs by @lhoestq in https://github.com/huggingface/datasets/pull/6804
Fix hf-internal-testing/dataset_with_script commit SHA in CI test by @albertvillanova in https://github.com/huggingface/datasets/pull/6806
Fix cache path to snakecase for CachedDatasetModuleFactory and Cache by @izhx in https://github.com/huggingface/datasets/pull/6754
Multithreaded downloads by @lhoestq in https://github.com/huggingface/datasets/pull/6794
Remove os.path.relpath in resolve_patterns by @mariosasko in https://github.com/huggingface/datasets/pull/6815
Extract data on the fly in packaged builders by @mariosasko in https://github.com/huggingface/datasets/pull/6784
add allow_primitive_to_str and allow_decimal_to_str instead of allow_number_to_str by @Modexus in https://github.com/huggingface/datasets/pull/6811
Support indexable objects in Dataset.__getitem__ by @mariosasko in https://github.com/huggingface/datasets/pull/6817
Make convert_to_parquet CLI command create script branch by @albertvillanova in https://github.com/huggingface/datasets/pull/6809
Fix parquet export infos by @lhoestq in https://github.com/huggingface/datasets/pull/6822

New Contributors

@VarunNSrivastava made their first contribution in https://github.com/huggingface/datasets/pull/6402
@shcheklein made their first contribution in https://github.com/huggingface/datasets/pull/6747
@JINO-ROHIT made their first contribution in https://github.com/huggingface/datasets/pull/6767
@JonasLoos made their first contribution in https://github.com/huggingface/datasets/pull/6799
@izhx made their first contribution in https://github.com/huggingface/datasets/pull/6754
@Modexus made their first contribution in https://github.com/huggingface/datasets/pull/6811

Full Changelog: huggingface/datasets@2.18.0...2.19.0

`v2.18.0`

Compare Source

Dataset features

Make JSON builder support an array of strings by @albertvillanova in https://github.com/huggingface/datasets/pull/6696
Base parquet batch_size on parquet row group size by @lhoestq in https://github.com/huggingface/datasets/pull/6701
- Faster cold start for streaming
Change default compression argument for JsonDatasetWriter by @Rexhaif in https://github.com/huggingface/datasets/pull/6659
Automatic Conversion for uint16/uint32 to Compatible PyTorch Dtypes by @mohalisad in https://github.com/huggingface/datasets/pull/6660
fsspec: support fsspec>=2023.12.0 glob changes by @pmrowla in https://github.com/huggingface/datasets/pull/6687
- Support latest fsspec up to 2024.2.0

General improvements and bug fixes

Fix for Incorrect ex_iterable used with multi num_worker by @kq-chen in https://github.com/huggingface/datasets/pull/6582
- Previously using PyTorch DDP and num_workers could lead to incorrect shards assignments to workers and cause errors
Fix imagefolder dataset url by @mariosasko in https://github.com/huggingface/datasets/pull/6683
Improve error message for gated datasets on load by @lewtun in https://github.com/huggingface/datasets/pull/6684
Updated Quickstart Notebook link by @Codeblockz in https://github.com/huggingface/datasets/pull/6685
Update the print message for chunked_dataset in process.mdx by @gzbfgjf2 in https://github.com/huggingface/datasets/pull/6693
Faster xlistdir by @mariosasko in https://github.com/huggingface/datasets/pull/6698
Update GitHub Actions to Node 20 by @albertvillanova in https://github.com/huggingface/datasets/pull/6682
Update release instructions by @albertvillanova in https://github.com/huggingface/datasets/pull/6681
Pass through information about location of cache directory. by @stridge-cruxml in https://github.com/huggingface/datasets/pull/6677
Allow SplitDict setitem to replace existing SplitInfo by @lhoestq in https://github.com/huggingface/datasets/pull/6665
Update ruff by @lhoestq in https://github.com/huggingface/datasets/pull/6706
Silence ruff deprecation messages by @mariosasko in https://github.com/huggingface/datasets/pull/6707
fix: show correct package name to install biopython by @BioGeek in https://github.com/huggingface/datasets/pull/6662
Fix data_files when passing data_dir by @lhoestq in https://github.com/huggingface/datasets/pull/6705
Release: 2.18.0 by @lhoestq in https://github.com/huggingface/datasets/pull/6708

New Contributors

@Codeblockz made their first contribution in https://github.com/huggingface/datasets/pull/6685
@gzbfgjf2 made their first contribution in https://github.com/huggingface/datasets/pull/6693
@stridge-cruxml made their first contribution in https://github.com/huggingface/datasets/pull/6677
@pmrowla made their first contribution in https://github.com/huggingface/datasets/pull/6687
@BioGeek made their first contribution in https://github.com/huggingface/datasets/pull/6662
@Rexhaif made their first contribution in https://github.com/huggingface/datasets/pull/6659
@mohalisad made their first contribution in https://github.com/huggingface/datasets/pull/6660
@kq-chen made their first contribution in https://github.com/huggingface/datasets/pull/6582

Full Changelog: huggingface/datasets@2.17.1...2.18.0

`v2.17.1`

Compare Source

Bug Fixes

Revert the changes in arrow_writer.py from #6636 by @bryant1410 in https://github.com/huggingface/datasets/pull/6664
Remove deprecated verbose parameter from CSV builder by @albertvillanova in https://github.com/huggingface/datasets/pull/6672

Full Changelog: huggingface/datasets@2.17.0...2.17.1

`v2.17.0`

Compare Source

Dataset Features

[WebDataset] Audio support and bug fixes by @lhoestq in https://github.com/huggingface/datasets/pull/6573
Add concurrent loading of shards to datasets.load_from_disk by @kkoutini in https://github.com/huggingface/datasets/pull/6464
Support data_dir parameter in push_to_hub by @albertvillanova in https://github.com/huggingface/datasets/pull/6634
Support push_to_hub without org/user to default to logged-in user by @albertvillanova in https://github.com/huggingface/datasets/pull/6629
Allow concatenation of datasets with mixed structs by @Dref360 in https://github.com/huggingface/datasets/pull/6587

General improvements and bug fixes

Fix parallel downloads for datasets without scripts by @lhoestq in https://github.com/huggingface/datasets/pull/6551
Fix imagefolder with one image by @lhoestq in https://github.com/huggingface/datasets/pull/6556
Fix tests based on datasets that used to have scripts by @lhoestq in https://github.com/huggingface/datasets/pull/6574
remove eli5 test by @lhoestq in https://github.com/huggingface/datasets/pull/6583
[IterableDataset] Fix drop_last_batchin map after shuffling or sharding by @lhoestq in https://github.com/huggingface/datasets/pull/6575
Support standalone yaml by @lhoestq in https://github.com/huggingface/datasets/pull/6557
Drop redundant None guard. by @xkszltl in https://github.com/huggingface/datasets/pull/6596
fix os.listdir return name is empty string by @d710055071 in https://github.com/huggingface/datasets/pull/6581
Fix CI: pyarrow 15, pandas 2.2 and sqlachemy by @lhoestq in https://github.com/huggingface/datasets/pull/6617
Dedicated RNG object for fingerprinting by @mariosasko in https://github.com/huggingface/datasets/pull/6606
Migrate from setup.cfg to pyproject.toml by @mariosasko in https://github.com/huggingface/datasets/pull/6619
keep more info in DatasetInfo.from_merge #6585 by @JochenSiegWork in https://github.com/huggingface/datasets/pull/6586
Read GeoParquet files using parquet reader by @weiji14 in https://github.com/huggingface/datasets/pull/6508
Use schema metadata only if it matches features by @lhoestq in https://github.com/huggingface/datasets/pull/6616
Raise error on bad split name by @lhoestq in https://github.com/huggingface/datasets/pull/6626
Disable tqdm bars in non-interactive environments by @mariosasko in https://github.com/huggingface/datasets/pull/6627
Add with_rank param to Dataset.filter by @mariosasko in https://github.com/huggingface/datasets/pull/6608
Bump max range of dill to 0.3.8 by @ringohoffman in https://github.com/huggingface/datasets/pull/6630
Fix filelock: use current umask for filelock >= 3.10 by @lhoestq in https://github.com/huggingface/datasets/pull/6631
Faster webdataset streaming by @lhoestq in https://github.com/huggingface/datasets/pull/6578
Multi gpu docs by @lhoestq in https://github.com/huggingface/datasets/pull/6550
dataset viewer requires no-script by @severo in https://github.com/huggingface/datasets/pull/6633
Make split slicing consistent with list slicing by @mariosasko in https://github.com/huggingface/datasets/pull/5891
Do not use Parquet exports if revision is passed by @albertvillanova in https://github.com/huggingface/datasets/pull/6555
Make CLI test support multi-processing by @albertvillanova in https://github.com/huggingface/datasets/pull/6628
Fix reload cache with data dir by @lhoestq in https://github.com/huggingface/datasets/pull/6632
Fix array cast/embed with null values by @mariosasko in https://github.com/huggingface/datasets/pull/6283
Faster column validation and reordering by @psmyth94 in https://github.com/huggingface/datasets/pull/6636
Better multi-gpu example by @lhoestq in https://github.com/huggingface/datasets/pull/6646
Fix missing info when loading some datasets from Parquet export by @lhoestq in https://github.com/huggingface/datasets/pull/6635
Minor multi gpu doc improvement by @lhoestq in https://github.com/huggingface/datasets/pull/6649
Document usage of hfh cli instead of git by @lhoestq in https://github.com/huggingface/datasets/pull/6648

New Contributors

@xkszltl made their first contribution in https://github.com/huggingface/datasets/pull/6596
@kkoutini made their first contribution in https://github.com/huggingface/datasets/pull/6464
@JochenSiegWork made their first contribution in https://github.com/huggingface/datasets/pull/6586
@weiji14 made their first contribution in https://github.com/huggingface/datasets/pull/6508
@ringohoffman made their first contribution in https://github.com/huggingface/datasets/pull/6630
@psmyth94 made their first contribution in https://github.com/huggingface/datasets/pull/6636

Full Changelog: huggingface/datasets@2.16.1...2.17.0

`v2.16.1`

Compare Source

Bug fixes

Fix dl_manager.extract returning FileNotFoundError by @lhoestq in https://github.com/huggingface/datasets/pull/6543
- Fix bug causing FileNotFoundError when passing a relative directory as cache_dir to load_dataset
Fix custom configs from script by @lhoestq in https://github.com/huggingface/datasets/pull/6544
- Fix bug when loading a dataset with a loading script using custom arguments would fail
- e.g. load_dataset("ted_talks_iwslt", language_pair=("ja", "en"), year="2015")

Full Changelog: huggingface/datasets@2.16.0...2.16.1

`v2.16.0`

Compare Source

Security features

Add trust_remote_code argument by @lhoestq in https://github.com/huggingface/datasets/pull/6429
- Some Hugging Face datasets contain custom code which must be executed to correctly load the dataset. The code can be inspected in the repository content at https://hf.co/datasets/<repo_id>. A warning is shown to let the user know about the custom code, and they can avoid this message in future by passing the argument trust_remote_code=True.
- Passing trust_remote_code=True will be mandatory to load these datasets from the next major release of datasets.
- Using the environment variable HF_DATASETS_TRUST_REMOTE_CODE=0 you can already disable custom code by default without waiting for the next release of datasets
Use parquet export if possible by @lhoestq in https://github.com/huggingface/datasets/pull/6448
- This allows loading most old datasets based on custom code by downloading the Parquet export provided by Hugging Face
- You can see a dataset's Parquet export at https://hf.co/datasets/<repo_id>/tree/refs%2Fconvert%2Fparquet

Features

Webdataset dataset builder by @lhoestq in https://github.com/huggingface/datasets/pull/6391
Implement get dataset default config name by @albertvillanova in [https://github.com/Implement get dataset default config name huggingface/datasets#6511](http

Configuration

📅 Schedule: Branch creation - "after 5am on saturday" (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Signed-off-by: konflux-internal-p02 <126015336+konflux-internal-p02[bot]@users.noreply.github.com>

konflux-internal-p02 · 2024-11-23T16:08:32Z

Edited/Blocked Notification

Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.

You can manually request rebase by checking the rebase/retry box above.

⚠️ Warning: custom changes will be lost.

Update dependency datasets to v2.21.0

a83121c

Signed-off-by: konflux-internal-p02 <126015336+konflux-internal-p02[bot]@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Update dependency datasets to v2.21.0 #29

Update dependency datasets to v2.21.0 #29

Uh oh!

konflux-internal-p02 bot commented Nov 23, 2024

Uh oh!

konflux-internal-p02 bot commented Nov 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

Update dependency datasets to v2.21.0 #29

Are you sure you want to change the base?

Update dependency datasets to v2.21.0 #29

Uh oh!

Conversation

konflux-internal-p02 bot commented Nov 23, 2024

Release Notes

Features

What's Changed

New Contributors

Important

Datasets features

General improvements and bug fixes

New Contributors

Bug fixes

Bug fixes

Dataset Features

General improvements and bug fixes

New Contributors

Dataset features

General improvements and bug fixes

New Contributors

Bug Fixes

Dataset Features

General improvements and bug fixes

New Contributors

Bug fixes

Security features

Features

Configuration

Uh oh!

konflux-internal-p02 bot commented Nov 23, 2024

Edited/Blocked Notification

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants