Skip to content

Conversation

@istranic
Copy link
Contributor

@istranic istranic commented Jan 8, 2023

🚀 🚀 Pull Request

Checklist:

  • My code follows the style guidelines of this project and the Contributing document
  • I have commented my code, particularly in hard-to-understand areas
  • I have kept the coverage-rate up
  • I have performed a self-review of my own code and resolved any problems
  • I have checked to ensure there aren't any other open Pull Requests for the same change
  • I have described and made corresponding changes to the relevant documentation
  • New and existing unit tests pass locally with my changes

Changes

  • Added support for ingesting yolo data
  • Added ability to automatically connect datasets in ingest_coco and ingest_yolo
  • Fixed other ingestion functions by adding reporting and making sure that Deep Lake dataset are created only after data is parsed.
  • Fixed a bug in deeplake.list

@CLAassistant
Copy link

CLAassistant commented Jan 8, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ istranic
❌ Ivo Stranic


Ivo Stranic seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@istranic istranic marked this pull request as draft January 8, 2023 20:14
@codecov
Copy link

codecov bot commented Jan 8, 2023

Codecov Report

Base: 85.21% // Head: 85.34% // Increases project coverage by +0.13% 🎉

Coverage data is based on head (c39b879) compared to base (6e53216).
Patch coverage: 94.87% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2099      +/-   ##
==========================================
+ Coverage   85.21%   85.34%   +0.13%     
==========================================
  Files         283      287       +4     
  Lines       30645    30947     +302     
==========================================
+ Hits        26113    26413     +300     
- Misses       4532     4534       +2     
Flag Coverage Δ
unittests 85.34% <94.87%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
deeplake/auto/tests/test_coco_template.py 100.00% <ø> (ø)
deeplake/auto/unstructured/util.py 82.60% <ø> (-0.25%) ⬇️
deeplake/auto/unstructured/yolo/yolo.py 90.76% <90.76%> (ø)
deeplake/api/dataset.py 90.59% <95.65%> (+0.19%) ⬆️
deeplake/auto/unstructured/yolo/utils.py 96.42% <96.42%> (ø)
deeplake/__init__.py 95.00% <100.00%> (+0.08%) ⬆️
deeplake/auto/tests/test_yolo_template.py 100.00% <100.00%> (ø)
deeplake/auto/unstructured/coco/coco.py 94.54% <100.00%> (ø)
deeplake/auto/unstructured/yolo/constants.py 100.00% <100.00%> (ø)
deeplake/tests/path_fixtures.py 89.02% <100.00%> (+1.05%) ⬆️
... and 6 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@istranic istranic marked this pull request as ready for review January 10, 2023 20:02
@istranic istranic requested a review from ProgerDav January 10, 2023 20:02
tensor_meta["images"].is_link,
self.image_creds_key,
),
self.label_params["name"]: yolo_labels.astype(
Copy link
Contributor

@ProgerDav ProgerDav Jan 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, the astype conversion results in coping of the array, although its small.
Can be solved by passing the type from the tensor upfront both to np.zeros(4, dtype=...) and the read_yolo_coordinates, then to the array creation like np.array(..., dtype=dtype).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @ProgerDav Not sure I 100% get it. I want to cast at the end using astype, because I don't want to pass the dtype to read_yolo_coordinates. I want to assume that I don't know the dtype returned by read_yolo_coordinates, and then cast at the end. I see how I could populate the zeros with the correct dtype in advance, but I don't think I can do that with the output of read_yolo_coordinates, no?

self.coordinates_params["name"] = coordinates_name

def _initialize_params(self, image_params, label_params, coordinates_params):
image_params_updated = DEFAULT_IMAGE_TENSOR_PARAMS.copy()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These parts may be simplified like self.image_params = {**DEFAULT_IMAGE_TENSOR_PARAMS, **image_params}. Effect if the same, only items present in the second dictionary will be overwritten over the default ones.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed


if coordinates.size % 2 != 0:
raise IngestionError(
f"Error ih annotation {file_name}. Polygons must have an even number of points."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@ProgerDav ProgerDav merged commit 807f312 into main Jan 20, 2023
@ProgerDav ProgerDav deleted the feature/ingest_yolo branch January 20, 2023 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants