-
Notifications
You must be signed in to change notification settings - Fork 882
doc(katib): update push-based metrics collector. #3844
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
doc(katib): update push-based metrics collector. #3844
Conversation
|
Thanks for adding this @Electronic-Waste! /assign @hbelmiro @StefanoFioravanzo |
Electronic-Waste
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left some explanations for the fields in tune function. If you have any questions, don't hesitate to contact me.
content/en/docs/components/katib/user-guides/metrics-collector.md
Outdated
Show resolved
Hide resolved
content/en/docs/components/katib/user-guides/metrics-collector.md
Outdated
Show resolved
Hide resolved
hbelmiro
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
andreyvelich
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this. I left a few comments.
/assign @kubeflow/wg-training-leads
content/en/docs/components/katib/user-guides/metrics-collector.md
Outdated
Show resolved
Hide resolved
content/en/docs/components/katib/user-guides/metrics-collector.md
Outdated
Show resolved
Hide resolved
| Use tune function and specify the `metrics_collector_config` field. You can reference to the following example: | ||
|
|
||
| ``` | ||
| import kubeflow.katib as katib |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use simple example here and remove all unnecessary parameters and function calls.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andreyvelich Sorry, I may not fully understand what you mean "unnecessary parameters and function calls".
I copied the example in the get-started chapter and replaced print with report_metrics(). Do you mean that I don't need to call the tune() function and just define a main function in the example?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think, we should just show example how user can use tune API with push-based metrics collector without any additional parameters they should set. For example:
import kubeflow.katib as katib
def objective(parameters):
import time
import kubeflow.katib as katib
time.sleep(5)
result = 4 * int(parameters["a"])
# Push metrics to Katib DB.
katib.report_metrics({"result": result})
katib.KatibClient().tune(
name="push-metrics-exp",
objective=objective,
parameters= {"a": katib.search.int(min=10, max=20)}
objective_metric_name="result",
max_trial_count=2,
metrics_collector_config={"kind": "Push"},
# When SDK is released, replace it with packages_to_install=["kubeflow-katib==0.18.0"]
packages_to_install=["git+https://github.com/kubeflow/katib.git@master#subdirectory=sdk/python/v1beta1"],
)That should allow user to focus on important changes they need to make to try this out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andreyvelich Thanks for your clarification! I'll update the blog.
|
|
||
| - [`tune`](https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py#L166) function | ||
|
|
||
| Use tune function and specify the `metrics_collector_config` field. You can reference to the following example: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You might need to explain how report_metrics() functions works separately. E.g. user needs to install kubeflow-katib SDK in their environment, and the report_metrics() will automatically add the timestamp for metrics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SGTM. I'll add the explanation to report_metrics().
Signed-off-by: Electronic-Waste <[email protected]>
Signed-off-by: Electronic-Waste <[email protected]>
05975ad to
0088100
Compare
|
@kubeflow/wg-automl-leads I've made some changes to the PR as you request. PTAL👀 if you have time. |
hbelmiro
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
|
We can merge it. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andreyvelich The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This PR aims at guiding users using push-based metrics collection: kubeflow/katib#2340.
I rewrote the metrics-collector chapter and divided it into three parts:
NoneMC.PushMC, with an example attached.cc👀 @kubeflow/wg-automl-leads . Please let me know if you have any suggestions.