feat: add ca end-to-end tests #409
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bassmang
changed the title
olgavrou
reviewed
Aug 27, 2021
olgavrou
reviewed
Aug 27, 2021
| "Number of deferred action for interaction events. Set the " | ||
| "deferred_action flag to true for first deferred_action_count number of " | ||
| "actions")("no_loop_actions", | ||
| "Flag to disable actions being taken for all outcome events"); |
Collaborator
There was a problem hiding this comment.
I am not sure what this argument does, do you want to not send activations for some deferred events?
Member
Author
There was a problem hiding this comment.
Yes the purpose of this is to ensure that skiplearn is included in all examples, to test that its being set in the dsjson conversion. It may be useful to change this option to a float instead, and use it to determine the % of events to activate. That way I could set it to 0 for this case, and add some more flexibility to example_gen for other loops
Member
Author
There was a problem hiding this comment.
@olgavrou I just removed the additional flag to simplify the PR
Collaborator
|
Not all deferred actions are activated the behaviour already exists because we activate based on a random number so not sure we need the extra arg
|
olgavrou
approved these changes
Aug 30, 2021
zwd-ms
added a commit
that referenced
this pull request
Oct 28, 2021
* [Binary parser] reward functions for ccb format (#361) * [Binary parser] refactoring rewards (#366) * Example gen add ccb loop for e2e testing (#368) * [Binary parser] add e2e ccb test, compare dsjson and fb logged files create the same model (#369) * minor binary parser cleanup (#370) * [Binary parser] add external parser test for apprentice mode cb (#374) * CCB apprentice reward (#373) * [Binary parser] add metrics for cb (#371) * [Binary parser] don't log when skip learn, more tests, skip over unknown msg type (#375) * [binary parser] ccb skip learn (#376) * refactor: add error message to fix config file (#377) * Fix CI's after flatbuffer version update to 2.0 (#390) * try set fb span minimal * add to preprocessor definitions * add to unit_test project file * Revert "mac ci: continue on error true (#327)" (#385) * Fix python build path on windows, and formatting. (#383) * Update build_docs.yml (#391) * only convert timestamp to string before exiting (#382) * ntohl is a define on osx, rename the function. (#386) * Add bunch of nice to haves CLI options and fix FB 2.0 compat. (#387) * our build requires CMP0074 due to usage of PackageName_ROOT variables. (#393) * our build requires CMP0074 due to usage of PackageName_ROOT variables. * try to use cmake_policy * Activations in multistep: first PR with schema changes only (#392) * deferred action to multistep schema * Multistep to problem type * try to set cmake policy for CMP0074 * OLD -> NEW * try default policy for cmp0074 * example_gen enhancements (#380) * [Binary parser] tests for CCB apprentice and skip learn (#379) * Switch to SAX parsing get the perf we need. (#388) * fix cli override to work with converter. * Switch from DOM to SAX and make it significantly faster. On my machine, using our standard gen path: With DOM: 76Mbps. With SAX: 103Mbps. * use RawValue instead of doing a SAX read/write. * disable example parsing. * cleanp and std::replace eol with space. * update * Fix disable example parsing for converted. * Add pdrop field for cb log converter (#396) * pdrop fix in binary parser (#351) * pdrop fix in binary parser * int -> float * main function for log_gen * verbose mode for parser * log gen fixes/enhancements * test generation notebook * simplified test data * cleanup * pdrop==0.5 test * test_gen + results update * explicit skip of examples with pdrop out of [0, 1) * pdrop==1 test * regenerated test sets (with pdrop == 0) * rcrfrmr -> rcrrmr * pdrop tests temporary removal * build fix * unit tests restore * wrong change revert * readme update * ipynb -> py * verbose print fix * wrong pdrop fix * log error for pdrop==1 * feat: Add binary parser for cats (#394) * feat: Add binary parser for cats * move interaction_data struct out of VW * commented changes and added learning_mode to cats * formatting * add reward calc tests * formatting * joined_event function update * check example length * finn_in_label bool Co-authored-by: olgavrou <olgavrou@gmail.com> * ccb log converter (#381) * Multistep joiner test + activations for multistep (#398) * pdrop fix in binary parser * int -> float * main function for log_gen * verbose mode for parser * log gen fixes/enhancements * test generation notebook * simplified test data * cleanup * pdrop==0.5 test * test_gen + results update * explicit skip of examples with pdrop out of [0, 1) * pdrop==1 test * regenerated test sets (with pdrop == 0) * rcrfrmr -> rcrrmr * pdrop tests temporary removal * build fix * unit tests restore * wrong change revert * readme update * ipynb -> py * verbose print fix * wrong pdrop fix * test_gen cleanup * log_gen: multistep event * log_gen: multistep_event / outcome * multistep log_gen fixes * multistep joiner test data gen * multistep log_gen: previous_id * log_gen; reward sign fix * test_gen: indices fix * multistep_2_episodes test * multistep deferred episodes test data * multistep activations failing test * log error for pdrop==1 * get rid of mk_cb_outcome * multistep activations test fix * ipython directives cleanup * feat: Add skip_learn and metrics tests for cats binary parser (#399) * feat: Add skip_learn tests for cats binary parser * add metrics tests * Update test_metrics.cc * feat: cats log converter (#400) * feat: cats log converter * skip_learn and formatting * Update flatbuffers version in readme (#401) * [Binary parser] set weight on all examples (#402) * Log gen cleanup (#406) * log_gen cleanup * deterministic client time generation * deterministic timestamp generation 2 * spacing cleanup * [Binary parser] enable client time utc (#403) * [Binary parser] Enable binary parser to handle slates payload (#404) * Slates parsing and reward calculation * Combine fill_in_label and set_cost * refactor skip learn + default reward * Slates log converter * set policy if exists in cmake version available (#410) * set policy if exists in cmake version available * add if policy to external_parser * Topological sort within episode (#408) * log_gen cleanup * deterministic client time generation * deterministic timestamp generation 2 * spacing cleanup * unordered episode testgn + test file * unordered episode joiner test (failing) * test_gen timestamps fix * unordered episode test fix * legacy comment cleanup * reference fix * comment * Slates e2e tests (#411) * feat: adding the original label cost to the binary parsers metrics (#412) * adding the original label cost to the binary parsers metrics * Update external_parser/joiners/example_joiner.cc Fix summation bug Co-authored-by: olgavrou <olgavrou@gmail.com> * no summation for skip learn events. Fix tests Co-authored-by: olgavrou <olgavrou@gmail.com> * feat: add ca end-to-end tests (#409) * feat: add ca-loop to example_gen * add test for ca-loop * formatting * e2e tests * formatting * fix test * test VW model for dsjson vs binary * formatting * commented changes * fix bug * revert activation_ratio arg Co-authored-by: cheng-tan <chengtan2013@gmail.com> Co-authored-by: olgavrou <olgavrou@gmail.com> * ci: dont update homebrew in MacOS CI (#418) * ci: upgrade Ubuntu16.04 workflows to Ubuntu 18.04 (#417) * log_parser 2 pandas (+ usage example) (#425) * Add apim authorization (#414) * initial commit * transportclient * fix spaces * fix spaces * add file references to project * fix pipeline build failures * unit test build failure * unit test cmake failures * fixes api key + add some comments * fix compile flag for build failure * rename function get to something more meaningful refactor apim create sender code * fix references in filter proj * causing mac os ci build failure * addresses comments * fix build error * restore.sh * build, transport, cmake * fix build error windows * revert http_auth * http_auth to event hub rename * eventhub changes back in * addresses comments * minor space clean up * fix more spaces * extra spaces * adds internal_constants.h and check for apikey * Move http_transport_client template implementation code to .h file + rename internal constants namespace (#427) * initial commit move template implementation code to .h file, rename internal constants variable * rename internal constants * trigger build * add internal_constants.h to cmake (#428) * add internal_constants.h to cmake * Trigger Build * Multistep rewards (#426) * suffix reward v0 * multistep reward parameter * multistep reward functions implementation * unit tests fix * multistep rewards unit tests * cleanup + comment * minor cleanups * tests cleanup * get rid of dynamic_cast * vector<pair> -> map * windows build fix * win build fix 2 * build fix * doc: update readme docs to make building clearer especially with regards to cmake (#432) * doc: update readme docs to make building clearer especially with regards to cmake * remove targets from readme * Update vw submodule (#433) * feat: Adds new metric DsjsonSumOriginalCostFirstSlot that aggregrates… (#429) * fix: emit dsjson_sum_cost_original metric (#434) * feat: Enable subsampling of events via configuration (#430) * Enable subsampling of events via configuration * use function to check is_equal * Update unit_test/async_batcher_test.cc Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> * PR comments * constantize a magic number * remove unnecessary comment * address PR comments Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> Co-authored-by: olgavrou <olgavrou@gmail.com> * feat: add new metric dsjson_sum_cost_original_baseline (#435) * Fix cmake for conditional python include (#436) * Benchmarks and github action to run them in CI (#405) * Fix sending to episode eventhub after merging. * Fix merge and build warnings. Co-authored-by: cheng-tan <chengtan2013@gmail.com> Co-authored-by: olgavrou <olgavrou@gmail.com> Co-authored-by: Griffin Bassman <griffinbassman@gmail.com> Co-authored-by: Eduardo Salinas <edus@microsoft.com> Co-authored-by: Rodrigo Kumpera <kumpera@users.noreply.github.com> Co-authored-by: Alexey Taymanov <41013086+ataymano@users.noreply.github.com> Co-authored-by: peterychang <49209570+peterychang@users.noreply.github.com> Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com> Co-authored-by: Sheetal Lahabar <lamadh@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.