Skip to content

Commit ee8fe0a

Browse files
tanwarshdependabot[bot]MasterSkepticistapayalchaSitu Singh
authored
Federated Analytics and Histogram Taskrunner Workspace (#1493)
* Bump matplotlib from 3.10.0 to 3.10.1 (#1416) Bumps [matplotlib](https://github.com/matplotlib/matplotlib) from 3.10.0 to 3.10.1. - [Release notes](https://github.com/matplotlib/matplotlib/releases) - [Commits](https://github.com/matplotlib/matplotlib/compare/v3.10.0...v3.10.1) --- updated-dependencies: - dependency-name: matplotlib dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: yes <[email protected]> * Remove implicit package installs (#1413) * Remove implicit package installs Signed-off-by: Shah, Karan <[email protected]> * Update CI Signed-off-by: Shah, Karan <[email protected]> * Install pkg cmd for tests Signed-off-by: Shah, Karan <[email protected]> * Patch gandlf test Signed-off-by: Shah, Karan <[email protected]> --------- Signed-off-by: Shah, Karan <[email protected]> Signed-off-by: yes <[email protected]> * code chages Signed-off-by: yes <[email protected]> * Adding Secure Aggregation Test (#1419) * Secure Aggregation Tests Signed-off-by: Chaurasiya, Payal <[email protected]> * Secure Aggregation Tests Signed-off-by: Chaurasiya, Payal <[email protected]> * Handle success and exception messages Signed-off-by: Chaurasiya, Payal <[email protected]> * Handle success and exception messages Signed-off-by: Chaurasiya, Payal <[email protected]> * Format issue Signed-off-by: Chaurasiya, Payal <[email protected]> * Put Secure Agg Signed-off-by: Chaurasiya, Payal <[email protected]> * Review comments Signed-off-by: Chaurasiya, Payal <[email protected]> --------- Signed-off-by: Chaurasiya, Payal <[email protected]> Signed-off-by: yes <[email protected]> * [tutorial] Fix `KeyError: 'w_old'` (#1425) Signed-off-by: Situ Singh <[email protected]> Signed-off-by: yes <[email protected]> * Moving GaNDLF test to pytest framework (#1423) * Upload artifacts for GanDLF Signed-off-by: noopur <[email protected]> * Moving GaNDLF to pytest Signed-off-by: noopur <[email protected]> * Move papermill import isndie relevant function Signed-off-by: noopur <[email protected]> * Add paramiko install in the step Signed-off-by: noopur <[email protected]> * Use model_name with request.config Signed-off-by: noopur <[email protected]> * Modified Signed-off-by: noopur <[email protected]> * Corrected the marker Signed-off-by: noopur <[email protected]> * Corrected the range loop for collab data Signed-off-by: noopur <[email protected]> * Export workspace after certify steps Signed-off-by: noopur <[email protected]> * Make GaNDLF workflow as E2E Signed-off-by: noopur <[email protected]> * Correction in conditions Signed-off-by: noopur <[email protected]> * More checks at pytest level Signed-off-by: noopur <[email protected]> * Added missing import Signed-off-by: noopur <[email protected]> * Debug stmts Signed-off-by: noopur <[email protected]> * Corrected path for config seg file Signed-off-by: noopur <[email protected]> * Corrected curr work dir Signed-off-by: noopur <[email protected]> * Correction Signed-off-by: noopur <[email protected]> * Correction due to code rebase Signed-off-by: noopur <[email protected]> * Review comments incorporated Signed-off-by: noopur <[email protected]> * Correction in checkout of gandlf step Signed-off-by: noopur <[email protected]> * Final changes in gandlf.yml Signed-off-by: noopur <[email protected]> * Review comments incorp Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Cleanup (#1422) * DCE, rename variable Signed-off-by: Shah, Karan <[email protected]> * Move to module-level loggers Signed-off-by: Shah, Karan <[email protected]> * Plan.logger to logger Signed-off-by: Shah, Karan <[email protected]> * Fixup Signed-off-by: Shah, Karan <[email protected]> * Also for workspaces Signed-off-by: Shah, Karan <[email protected]> * Fix again for logger Signed-off-by: Shah, Karan <[email protected]> * Remove run_simulation tests Signed-off-by: Shah, Karan <[email protected]> * Header creation is stateless and common across both Signed-off-by: Shah, Karan <[email protected]> * Remove TASK_REGISTRY code; deprecated with interactive Signed-off-by: Shah, Karan <[email protected]> * More use_delta_updates changes Signed-off-by: Shah, Karan <[email protected]> * lets see if this revert works Signed-off-by: Shah, Karan <[email protected]> * Remove openfl install in notebook Signed-off-by: Shah, Karan <[email protected]> * revert few flags Signed-off-by: Shah, Karan <[email protected]> * please please please this should work Signed-off-by: Shah, Karan <[email protected]> * Migrate stateless function to common Signed-off-by: Shah, Karan <[email protected]> * Cleanup outdated docstrings Signed-off-by: Shah, Karan <[email protected]> * Remove Interactive API RPCs Signed-off-by: Shah, Karan <[email protected]> * Remove set devices call; not used anywhere Signed-off-by: Shah, Karan <[email protected]> * Replace checks with inline assertions Signed-off-by: Shah, Karan <[email protected]> * Unfold one-liner functions Signed-off-by: Shah, Karan <[email protected]> * Remove moot tests Signed-off-by: Shah, Karan <[email protected]> --------- Signed-off-by: Shah, Karan <[email protected]> Signed-off-by: yes <[email protected]> * B413 bandit issue ignored (#1427) * B413 bandit issue ignored Signed-off-by: yes <[email protected]> * code formatted Signed-off-by: yes <[email protected]> --------- Signed-off-by: yes <[email protected]> * Bandit B108 medium issue(Probable insecure usage of temp file/directory) fix (#1424) * Bandit B108 medium issue(Probable insecure usage of temp file/directory.) Signed-off-by: yes <[email protected]> * Bandit B108 medium issue(Probable insecure usage of temp file/directory.) Signed-off-by: yes <[email protected]> * Bandit B108 medium issue(Probable insecure usage of temp file/directory.) Signed-off-by: yes <[email protected]> * Bandit B108 medium issue(Probable insecure usage of temp file/directory.) Signed-off-by: yes <[email protected]> * dummy commit Signed-off-by: yes <[email protected]> * dummy commit Signed-off-by: yes <[email protected]> --------- Signed-off-by: yes <[email protected]> * fix(plan reading): use yaml safe_load (#1432) Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: yes <[email protected]> * [bandit] Ignore B614: Use of unsafe `torch.load` (#1426) * B614 Use of unsafe PyTorch load fix Signed-off-by: yes <[email protected]> * added safetensors==0.5.3 Signed-off-by: yes <[email protected]> * added safetensors==0.5.3 Signed-off-by: yes <[email protected]> * testing changes Signed-off-by: yes <[email protected]> * B614 Use of unsafe PyTorch load fix Signed-off-by: yes <[email protected]> * B614 Use of unsafe PyTorch load fix Signed-off-by: yes <[email protected]> * revert changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * formatted code Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * revert changes Signed-off-by: yes <[email protected]> * revert changes Signed-off-by: yes <[email protected]> --------- Signed-off-by: yes <[email protected]> * Use Single Function For Logging With Rich Text Handler (#1435) * Use rich handler for e2e logger Signed-off-by: noopur <[email protected]> * Use rich handler for e2e logger Signed-off-by: noopur <[email protected]> * More params to Rich constructor Signed-off-by: noopur <[email protected]> * Do not run straggler Signed-off-by: noopur <[email protected]> * Use formatter for rich Signed-off-by: noopur <[email protected]> * Missed a comma Signed-off-by: noopur <[email protected]> * More changes Signed-off-by: noopur <[email protected]> * E2E: use rich text handler Signed-off-by: noopur <[email protected]> * Keep formatter with message only Signed-off-by: noopur <[email protected]> * Use logs file Signed-off-by: noopur <[email protected]> * More changes Signed-off-by: noopur <[email protected]> * Retain same function name Signed-off-by: noopur <[email protected]> * Code formatting fixed Signed-off-by: noopur <[email protected]> * Review comments incorporated Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Add collaborator name attribute to `AggregatorGRPCClient` (#1436) * Remove explicit collaborator name arg within RPC calls Signed-off-by: Shah, Karan <[email protected]> * Update tests Signed-off-by: Shah, Karan <[email protected]> --------- Signed-off-by: Shah, Karan <[email protected]> Signed-off-by: yes <[email protected]> * Bump keras from 3.8.0 to 3.9.0 in /openfl-workspace/keras/2dunet (#1439) Bumps [keras](https://github.com/keras-team/keras) from 3.8.0 to 3.9.0. - [Release notes](https://github.com/keras-team/keras/releases) - [Commits](https://github.com/keras-team/keras/compare/v3.8.0...v3.9.0) --- updated-dependencies: - dependency-name: keras dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: yes <[email protected]> * Deleting AggregatorBasedWorkflow tutorials and examples (#1438) Signed-off-by: Teodor Parvanov <[email protected]> Signed-off-by: yes <[email protected]> * updated keras (#1447) Signed-off-by: yes <[email protected]> * updated keras (#1446) Signed-off-by: yes <[email protected]> * updated keras (#1445) Signed-off-by: yes <[email protected]> * Bump keras in /openfl-workspace/keras/tensorflow/mnist (#1444) Bumps [keras](https://github.com/keras-team/keras) from 3.8.0 to 3.9.0. - [Release notes](https://github.com/keras-team/keras/releases) - [Commits](https://github.com/keras-team/keras/compare/v3.8.0...v3.9.0) --- updated-dependencies: - dependency-name: keras dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shailesh Tanwar <[email protected]> Signed-off-by: yes <[email protected]> * Remove the usage of `shell=True` when running commands using `subprocess` (#1443) * fix(bandit b602): remove the usage of shell=True when running commands using subprocess Signed-off-by: Pant, Akshay <[email protected]> * fix(execute): keyword argument name Signed-off-by: Pant, Akshay <[email protected]> * fix(execute): value type in dict Signed-off-by: Pant, Akshay <[email protected]> * fix(dockerize): command args Signed-off-by: Pant, Akshay <[email protected]> --------- Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: yes <[email protected]> * Fix typos (#1449) Signed-off-by: Yaroslav Halchenko <[email protected]> Signed-off-by: yes <[email protected]> * Upgrading nbdev package version from 2.3.12 to 2.3.37 to fix ImportError (#1456) * Taking latest nbdev as previous giving error Signed-off-by: Chaurasiya, Payal <[email protected]> * Taking latest nbdev as previous giving error Signed-off-by: Chaurasiya, Payal <[email protected]> --------- Signed-off-by: Chaurasiya, Payal <[email protected]> Signed-off-by: yes <[email protected]> * TaskRunner E2E: Remove ANSI escape codes and unwanted patterns from participant log files with LOG_FILE usage (#1455) * Use LOG_FILE for logging to participant log files Signed-off-by: noopur <[email protected]> * Append & for background processes Signed-off-by: noopur <[email protected]> * Do not show date in log files Signed-off-by: noopur <[email protected]> * Param correction Signed-off-by: noopur <[email protected]> * More changes Signed-off-by: noopur <[email protected]> * Using & Signed-off-by: noopur <[email protected]> * Keep bg file running Signed-off-by: noopur <[email protected]> * Upgrade nbdev from 2.3.12 to 2.3.37 Signed-off-by: noopur <[email protected]> * Skip csv from artifact upload Signed-off-by: noopur <[email protected]> * Variable moved up Signed-off-by: noopur <[email protected]> * Move time.sleep before process id check Signed-off-by: noopur <[email protected]> * Remove current round logging Signed-off-by: noopur <[email protected]> * Do not force rich text to terminal Signed-off-by: noopur <[email protected]> * Specific console for conftest Signed-off-by: noopur <[email protected]> * Use single rich handler in conftest Signed-off-by: noopur <[email protected]> * Change log file path for dockerized workspace Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Update `installation.rst` (#1452) * Documentation update to specify: - requirements for python versions compatible with OpenFL - requirements for platforms compatible with the OpenFL docker image Signed-off-by: Lior Malka <[email protected]> * Responding to PR comments Signed-off-by: Lior Malka <[email protected]> * Paraphrasing python supported versions Signed-off-by: Lior Malka <[email protected]> --------- Signed-off-by: Lior Malka <[email protected]> Signed-off-by: yes <[email protected]> * dependency(ray): update to 2.43.0 from 2.9.2 (#1448) Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * Introducing Task Runner workspace to enable interoperability with Flower workloads (#1433) * enable flwr workspace Signed-off-by: kta-intel <[email protected]> * add tmp dir to minimize how much patching is needed Signed-off-by: kta-intel <[email protected]> * add hash verification to setup data Signed-off-by: kta-intel <[email protected]> * remove patch Signed-off-by: kta-intel <[email protected]> * fix save location Signed-off-by: kta-intel <[email protected]> * docstring Signed-off-by: kta-intel <[email protected]> * remove superfluous edit Signed-off-by: kta-intel <[email protected]> * update readme Signed-off-by: kta-intel <[email protected]> * remove debugger Signed-off-by: kta-intel <[email protected]> * update aggregator client Signed-off-by: kta-intel <[email protected]> * formatting Signed-off-by: kta-intel <[email protected]> * remove todo Signed-off-by: kta-intel <[email protected]> * minor functionality fixes Signed-off-by: kta-intel <[email protected]> * update docstrings Signed-off-by: kta-intel <[email protected]> * update docstrings Signed-off-by: kta-intel <[email protected]> * update README instructions Signed-off-by: kta-intel <[email protected]> * do not add connector to settings unless connector exists Signed-off-by: kta-intel <[email protected]> * check attribute for connector availability Signed-off-by: kta-intel <[email protected]> * formatting Signed-off-by: kta-intel <[email protected]> * formatting Signed-off-by: kta-intel <[email protected]> * code cleanup Signed-off-by: kta-intel <[email protected]> * fix readme Signed-off-by: kta-intel <[email protected]> * update plan to reflect refactoring Signed-off-by: kta-intel <[email protected]> * grammar fix Signed-off-by: kta-intel <[email protected]> * remove __all__ Signed-off-by: kta-intel <[email protected]> * remove superfluous init Signed-off-by: kta-intel <[email protected]> * change flwr home dir Signed-off-by: kta-intel <[email protected]> * remove patch comment, .sort() added in flwr 1.16 Signed-off-by: kta-intel <[email protected]> * update name for grpc protocols and components Signed-off-by: kta-intel <[email protected]> * remove Connector abc Signed-off-by: kta-intel <[email protected]> * refactoring local grpc to interop Signed-off-by: kta-intel <[email protected]> * import fixes Signed-off-by: kta-intel <[email protected]> * remove duplicate self.callback Signed-off-by: kta-intel <[email protected]> * fix hashes Signed-off-by: kta-intel <[email protected]> * move into workspace for relative path installation Signed-off-by: kta-intel <[email protected]> * update readme to remove connector ABC Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: kta-intel <[email protected]> Signed-off-by: yes <[email protected]> * Introduce a default `WaitForAllPolicy` for straggler handling (#1461) * feat(straggler handling): add NoPolicy Signed-off-by: Pant, Akshay <[email protected]> * feat(straggler handling): use NoPolicy as the default Signed-off-by: Pant, Akshay <[email protected]> * restructure(straggler handling): rename NoPolicy to WaitForAllPolicy Signed-off-by: Pant, Akshay <[email protected]> --------- Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: yes <[email protected]> * Get current round for non FedEval scenarios (#1463) Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * [taskrunner] Secure Aggregation cleanup (#1420) * fix(dependency): remove pycryptodome from openfl setup Signed-off-by: Pant, Akshay <[email protected]> * restructure(aggregator secagg): removed bootstrap steps from aggregator to helper class Signed-off-by: Pant, Akshay <[email protected]> * restructure(collaborator secagg): move masking step to utility function Signed-off-by: Pant, Akshay <[email protected]> * restructure(secagg): import utility functions only when used Signed-off-by: Pant, Akshay <[email protected]> * fix(secagg): add check for pycrptodome installation when importing Signed-off-by: Pant, Akshay <[email protected]> * docs(secagg): add future enhancements that are not supported Signed-off-by: Pant, Akshay <[email protected]> * fix(secagg): bootstrap class e2e Signed-off-by: Pant, Akshay <[email protected]> * fix(secagg): update bootstrap class usage and not masking metrics Signed-off-by: Pant, Akshay <[email protected]> * fix(secagg): move masked input vector calculation to collaborator class and remove masking from metrics Signed-off-by: Pant, Akshay <[email protected]> * restructure(secagg): rename methods more appropriately Signed-off-by: Pant, Akshay <[email protected]> * fix(aggregator): remove duplicate callbacklist initialisation Signed-off-by: Pant, Akshay <[email protected]> * restructure(aggregator secagg setup): remove unnecesary else conditional block Signed-off-by: Pant, Akshay <[email protected]> * docs(secagg): add missing word Signed-off-by: Pant, Akshay <[email protected]> * fix(secagg): use importlib instead of pkg_resources to check module installation Signed-off-by: Pant, Akshay <[email protected]> * fix(secagg): check pycrytodome install Signed-off-by: Pant, Akshay <[email protected]> * restructure(aggregator): agg function slection Signed-off-by: Pant, Akshay <[email protected]> * fix: remove repeated line Signed-off-by: Pant, Akshay <[email protected]> * feat(plan): add plan verify to check for incompatible features Signed-off-by: Pant, Akshay <[email protected]> * docs(secagg): add explicit mention for using WaitForAllPolicy with secagg Signed-off-by: Pant, Akshay <[email protected]> * fix(plan): change verify to non-static method Signed-off-by: Pant, Akshay <[email protected]> --------- Signed-off-by: Pant, Akshay <[email protected]> Co-authored-by: Payal Chaurasiya <[email protected]> Signed-off-by: yes <[email protected]> * Added notes for non-TLS in Quickstart documentation (#1462) * Separate section for non-TLS Signed-off-by: noopur <[email protected]> * Correction Signed-off-by: noopur <[email protected]> * Correction Signed-off-by: noopur <[email protected]> * Keep non TLS at the top Signed-off-by: noopur <[email protected]> * Optional step Signed-off-by: noopur <[email protected]> * Review comment incorp Signed-off-by: noopur <[email protected]> * Forgot to save the changes earlier Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Security TLS certificate report tests (#1460) * Taking latest nbdev as previous giving error Signed-off-by: Chaurasiya, Payal <[email protected]> * Taking latest nbdev as previous giving error Signed-off-by: Chaurasiya, Payal <[email protected]> * Security TLS certificate report tests Signed-off-by: Chaurasiya, Payal <[email protected]> * Security TLS certificate report tests Signed-off-by: Chaurasiya, Payal <[email protected]> * FedEval fix Signed-off-by: Chaurasiya, Payal <[email protected]> * Small change Signed-off-by: Chaurasiya, Payal <[email protected]> * Logs correction Signed-off-by: Chaurasiya, Payal <[email protected]> * Format changes Signed-off-by: Chaurasiya, Payal <[email protected]> --------- Signed-off-by: Chaurasiya, Payal <[email protected]> Signed-off-by: yes <[email protected]> * formatting fix Signed-off-by: yes <[email protected]> * formatting fix Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * fix(secagg): correct module name to import module (#1466) Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * Reduce Aggregator Memory Usage (#1459) * Changes to reduce unnecessary allocations Signed-off-by: Shah, Karan <[email protected]> * Remove compression/decompression for _prepare_trained Signed-off-by: Shah, Karan <[email protected]> * Execute SendLocalTaskResults in a queued call Signed-off-by: Shah, Karan <[email protected]> * Update test with the remove col_name argument Signed-off-by: Shah, Karan <[email protected]> * Use allclose instead of equal Signed-off-by: Shah, Karan <[email protected]> * Revert "Remove compression/decompression for _prepare_trained" This reverts commit cb0adebcf80bc95f47dc2f28c416d2a622bfc039. Signed-off-by: Shah, Karan <[email protected]> * Move lock decorator to common Signed-off-by: Shah, Karan <[email protected]> --------- Signed-off-by: Shah, Karan <[email protected]> Signed-off-by: yes <[email protected]> * code chnages Signed-off-by: yes <[email protected]> * code changes as per comments Signed-off-by: yes <[email protected]> * reverted to torch Signed-off-by: yes <[email protected]> * Skipping GaNDLF run from PR pipeline (#1482) * setuptools upgrade pr Signed-off-by: payalcha <[email protected]> * setuptools upgrade pr Signed-off-by: payalcha <[email protected]> --------- Signed-off-by: payalcha <[email protected]> Signed-off-by: yes <[email protected]> * To streamline Collaborator yaml for torch/mnist with all other cols.yaml file (#1479) * Taking latest nbdev as previous giving error Signed-off-by: Chaurasiya, Payal <[email protected]> * Taking latest nbdev as previous giving error Signed-off-by: Chaurasiya, Payal <[email protected]> * Security TLS certificate report tests Signed-off-by: Chaurasiya, Payal <[email protected]> * Security TLS certificate report tests Signed-off-by: Chaurasiya, Payal <[email protected]> * FedEval fix Signed-off-by: Chaurasiya, Payal <[email protected]> * Small change Signed-off-by: Chaurasiya, Payal <[email protected]> * Logs correction Signed-off-by: Chaurasiya, Payal <[email protected]> * Format changes Signed-off-by: Chaurasiya, Payal <[email protected]> * To streamline Collaborator yaml for torch/mnist with all other cols.yaml file Signed-off-by: Chaurasiya, Payal <[email protected]> * change tp fpdf2 from fpdf Signed-off-by: Chaurasiya, Payal <[email protected]> * change tp fpdf2 from fpdf Signed-off-by: Chaurasiya, Payal <[email protected]> * review comments Signed-off-by: Chaurasiya, Payal <[email protected]> * Update taskrunner tutorial Signed-off-by: payalcha <[email protected]> --------- Signed-off-by: Chaurasiya, Payal <[email protected]> Signed-off-by: payalcha <[email protected]> Co-authored-by: Noopur <[email protected]> Signed-off-by: yes <[email protected]> * reverted to torch Signed-off-by: yes <[email protected]> * update README.md and plan.yaml for `flower-app-pytorch` workspace (#1470) * chore: update openfl version to 1.8 (#1464) Signed-off-by: Pant, Akshay <[email protected]> * fix(secagg): correct module name to import module (#1467) Signed-off-by: Pant, Akshay <[email protected]> * fix(secagg): correct module name to import module (#1466) Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: kta-intel <[email protected]> * update README.md Signed-off-by: kta-intel <[email protected]> * 1.8: Raise error if data is not found when running flower-app-pytorch workspace (#1473) * fix(secagg): correct module name to import module (#1466) Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: kta-intel <[email protected]> * raise error if data is not found Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: kta-intel <[email protected]> Co-authored-by: Akshay Pant <[email protected]> * add fqdn flag to aggregator commands, fix auto_shutdown key, add additional notes to experiment configuration Signed-off-by: kta-intel <[email protected]> * restructure: remove trailing whitespace (#1483) Signed-off-by: Pant, Akshay <[email protected]> --------- Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: kta-intel <[email protected]> Signed-off-by: Pant, Akshay <[email protected]> Co-authored-by: Akshay Pant <[email protected]> Co-authored-by: Noopur <[email protected]> Signed-off-by: yes <[email protected]> * Task Runner E2E: Remove additional folder permissioning (#1485) Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Issue with Gandlf solved (#1486) Signed-off-by: payalcha <[email protected]> Signed-off-by: yes <[email protected]> * OpenFL v1.8 release notes (#1487) Signed-off-by: Teodor Parvanov <[email protected]> Signed-off-by: yes <[email protected]> * revert core components changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * E2E automation for `flower-app-pytorch` workspace (#1494) * Test Flower automation Signed-off-by: noopur <[email protected]> * Run only once Signed-off-by: noopur <[email protected]> * Added Flower as separate job Signed-off-by: noopur <[email protected]> * Extra closing bracket Signed-off-by: noopur <[email protected]> * Removed dependencies installation step Signed-off-by: noopur <[email protected]> * Correction of step name and logging Signed-off-by: noopur <[email protected]> * New marker for flower model Signed-off-by: noopur <[email protected]> * Separate workflow for Flower Signed-off-by: noopur <[email protected]> * Changes for dockerized ws Signed-off-by: noopur <[email protected]> * Modified test name to reflect flower model Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * 1.8: Address Coverity issues related to `flower-app-pytorch` workspace (#1488) (#1490) * bind socket to a specific interface * add input validation and allow list * address filepath coverity issue by creating a function to check path safety using set of allowed characters * pin upper bound to flwr * formatting --------- Signed-off-by: kta-intel <[email protected]> Signed-off-by: yes <[email protected]> * Hotfix for PQ pipeline (#1497) Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Add trufflehog to scan secrets in repo and logs file (#1495) * Add trufflehog to scan secrets in repo and logs file Signed-off-by: payalcha <[email protected]> * title fix Signed-off-by: payalcha <[email protected]> * Review comments Signed-off-by: payalcha <[email protected]> --------- Signed-off-by: payalcha <[email protected]> Signed-off-by: yes <[email protected]> * Change openfl version from 1.8 to 1.9.0.dev (#1503) * Change openfl version from 1.8 to 1.9.0-dev Signed-off-by: noopur <[email protected]> * Change openfl version from 1.8 to 1.9.0.dev Signed-off-by: noopur <[email protected]> * Modified version file as well Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * PQ Pipeline: Ensure usage of same commit id across workflow jobs (#1501) * Initial changes for testing Signed-off-by: noopur <[email protected]> * Initial changes for testing Signed-off-by: noopur <[email protected]> * All workflows modified Signed-off-by: noopur <[email protected]> * All workflows modified Signed-off-by: noopur <[email protected]> * All workflows modified Signed-off-by: noopur <[email protected]> * Correction of job name Signed-off-by: noopur <[email protected]> * Correction of job name Signed-off-by: noopur <[email protected]> * Testing one workflow Signed-off-by: noopur <[email protected]> * Final set of changes Signed-off-by: noopur <[email protected]> * Correct the repo URL Signed-off-by: noopur <[email protected]> * Only commit related changes Signed-off-by: noopur <[email protected]> * Correction in trufflehog job name Signed-off-by: noopur <[email protected]> * Added commit_id to SSL wf after rebase Signed-off-by: noopur <[email protected]> * Modified version file as well Signed-off-by: noopur <[email protected]> * Make commit_id optional input Signed-off-by: noopur <[email protected]> * Job name added for Trufflehog Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Co-authored-by: Payal Chaurasiya <[email protected]> Signed-off-by: yes <[email protected]> * changes as per comments Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * plan updated Signed-off-by: yes <[email protected]> * Added conda installation instructions (#1508) * added publications Signed-off-by: sarthakpati <[email protected]> * added conda installation instructions Signed-off-by: sarthakpati <[email protected]> --------- Signed-off-by: sarthakpati <[email protected]> Signed-off-by: yes <[email protected]> * Bump pytest-asyncio from 0.25.3 to 0.26.0 (#1509) Bumps [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) from 0.25.3 to 0.26.0. - [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases) - [Commits](https://github.com/pytest-dev/pytest-asyncio/compare/v0.25.3...v0.26.0) --- updated-dependencies: - dependency-name: pytest-asyncio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Payal Chaurasiya <[email protected]> Signed-off-by: yes <[email protected]> * add hippmapp3r (#1498) * add hippmapp3r Signed-off-by: porteratzo <[email protected]> * addresed comments Signed-off-by: porteratzo <[email protected]> * add reference Signed-off-by: porteratzo <[email protected]> --------- Signed-off-by: porteratzo <[email protected]> Signed-off-by: yes <[email protected]> * Bump ruff from 0.9.9 to 0.11.2 (#1515) Signed-off-by: payalcha <[email protected]> Signed-off-by: yes <[email protected]> * Task Runner E2E: Simplify the components' `start` and `stop` processes (#1514) * Run resiliency 3 times Signed-off-by: noopur <[email protected]> * Run 50 rounds Signed-off-by: noopur <[email protected]> * Extra debugging Signed-off-by: noopur <[email protected]> * Extra debugging Signed-off-by: noopur <[email protected]> * Testing Signed-off-by: noopur <[email protected]> * Testing Signed-off-by: noopur <[email protected]> * Modified Signed-off-by: noopur <[email protected]> * Remove extra logging Signed-off-by: noopur <[email protected]> * Remove extra logging Signed-off-by: noopur <[email protected]> * Kill process as a one-liner Signed-off-by: noopur <[email protected]> * Kill process as a one-liner Signed-off-by: noopur <[email protected]> * Upgraded pytest from 8.3.4 to 8.3.5 Signed-off-by: noopur <[email protected]> * Specific functions to fetch pids and kill processes Signed-off-by: noopur <[email protected]> * Multiple changes done Signed-off-by: noopur <[email protected]> * File changes as part of lint fixing Signed-off-by: noopur <[email protected]> * Minor logging fixes and copyright year correction Signed-off-by: noopur <[email protected]> * Review comments addressed Signed-off-by: noopur <[email protected]> * Revert pytest version upgrade change Signed-off-by: noopur <[email protected]> * Check if start_process is present Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Bump pytest from 8.3.4 to 8.3.5 (#1510) Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.3.4 to 8.3.5. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest/compare/8.3.4...8.3.5) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Payal Chaurasiya <[email protected]> Co-authored-by: Noopur <[email protected]> Signed-off-by: yes <[email protected]> * Bump aquasecurity/trivy-action in the github-actions group (#1512) Bumps the github-actions group with 1 update: [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action). Updates `aquasecurity/trivy-action` from 0.29.0 to 0.30.0 - [Release notes](https://github.com/aquasecurity/trivy-action/releases) - [Commits](https://github.com/aquasecurity/trivy-action/compare/0.29.0...0.30.0) --- updated-dependencies: - dependency-name: aquasecurity/trivy-action dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Payal Chaurasiya <[email protected]> Co-authored-by: Akshay Pant <[email protected]> Signed-off-by: yes <[email protected]> * Change to add time taken for each round in metrics (#1517) * Change to add time taken for each round in metrics Signed-off-by: payalcha <[email protected]> * Review comments Signed-off-by: payalcha <[email protected]> --------- Signed-off-by: payalcha <[email protected]> Signed-off-by: yes <[email protected]> * [Workflow Interface] Rename and Refactor WorkspaceExport into NotebookTools (#1364) * Experimental-export-module-refactoring Signed-off-by: refai06 <[email protected]> * Incorporated comments Signed-off-by: refai06 <[email protected]> * Improvements added Signed-off-by: refai06 <[email protected]> * Added testcase Signed-off-by: refai06 <[email protected]> * Code Enhancement Signed-off-by: refai06 <[email protected]> * Docstring & loggers update Signed-off-by: refai06 <[email protected]> * Added NotebookTools module Testcase Signed-off-by: refai06 <[email protected]> * Testcase update Signed-off-by: refai06 <[email protected]> * Review_comments_incorporated Signed-off-by: refai06 <[email protected]> * Fix testcase Signed-off-by: refai06 <[email protected]> * Added review comments and testcase update Signed-off-by: refai06 <[email protected]> * Review comments Signed-off-by: refai06 <[email protected]> * Remove testcase Signed-off-by: refai06 <[email protected]> * Enhance code and incorporate review comments Signed-off-by: refai06 <[email protected]> * Code changes Signed-off-by: refai06 <[email protected]> * revert tutorial change Signed-off-by: refai06 <[email protected]> * Incorporate comments Signed-off-by: refai06 <[email protected]> * Docstring update Signed-off-by: refai06 <[email protected]> --------- Signed-off-by: refai06 <[email protected]> Signed-off-by: yes <[email protected]> * PQ Pipeline: Publish `openfl-nightly` package to `pypi` on daily basis (#1504) * PQ Pipeline: Publish openfl-nightly package to pypi on daily basis Signed-off-by: noopur <[email protected]> * Add trufflehog to scan secrets in repo and logs file (#1495) * Add trufflehog to scan secrets in repo and logs file Signed-off-by: payalcha <[email protected]> * title fix Signed-off-by: payalcha <[email protected]> * Review comments Signed-off-by: payalcha <[email protected]> --------- Signed-off-by: payalcha <[email protected]> Signed-off-by: noopur <[email protected]> * Correction Signed-off-by: noopur <[email protected]> * Change openfl version from 1.8 to 1.9.0.dev (#1503) * Change openfl version from 1.8 to 1.9.0-dev Signed-off-by: noopur <[email protected]> * Change openfl version from 1.8 to 1.9.0.dev Signed-off-by: noopur <[email protected]> * Modified version file as well Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> * PQ Pipeline: Ensure usage of same commit id across workflow jobs (#1501) * Initial changes for testing Signed-off-by: noopur <[email protected]> * Initial changes for testing Signed-off-by: noopur <[email protected]> * All workflows modified Signed-off-by: noopur <[email protected]> * All workflows modified Signed-off-by: noopur <[email protected]> * All workflows modified Signed-off-by: noopur <[email protected]> * Correction of job name Signed-off-by: noopur <[email protected]> * Correction of job name Signed-off-by: noopur <[email protected]> * Testing one workflow Signed-off-by: noopur <[email protected]> * Final set of changes Signed-off-by: noopur <[email protected]> * Correct the repo URL Signed-off-by: noopur <[email protected]> * Only commit related changes Signed-off-by: noopur <[email protected]> * Correction in trufflehog job name Signed-off-by: noopur <[email protected]> * Added commit_id to SSL wf after rebase Signed-off-by: noopur <[email protected]> * Modified version file as well Signed-off-by: noopur <[email protected]> * Make commit_id optional input Signed-off-by: noopur <[email protected]> * Job name added for Trufflehog Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Co-authored-by: Payal Chaurasiya <[email protected]> Signed-off-by: noopur <[email protected]> * Run Trivy and Bandit on PQ scheduled run Signed-off-by: noopur <[email protected]> * Syntax correction Signed-off-by: noopur <[email protected]> * Added required permissions for Trivy and Bandit Signed-off-by: noopur <[email protected]> * Remove generic permissions Signed-off-by: noopur <[email protected]> * PQ pipeline - run trivy and bandit for workflow_dispatch as well Signed-off-by: noopur <[email protected]> * Use version from setup.pt Signed-off-by: noopur <[email protected]> * Extra files for lint related fixes Signed-off-by: noopur <[email protected]> * Call publish wf from pq with specific commit id Signed-off-by: noopur <[email protected]> * All jobs to run for scheduler from main branch or manual trigger only Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: payalcha <[email protected]> Co-authored-by: Payal Chaurasiya <[email protected]> Signed-off-by: yes <[email protected]> * Changes in PQ Pipeline & Publish Package workflows for PYPI_API_TOKEN (#1520) * Added dev environment Signed-off-by: noopur <[email protected]> * Added dev environment Signed-off-by: noopur <[email protected]> * Do not continue on error Signed-off-by: noopur <[email protected]> * Added permission for token read write Signed-off-by: noopur <[email protected]> * Use shell commands Signed-off-by: noopur <[email protected]> * Testing Signed-off-by: noopur <[email protected]> * Testing Signed-off-by: noopur <[email protected]> * Final changes Signed-off-by: noopur <[email protected]> * Changes in PQ pipeline as well Signed-off-by: noopur <[email protected]> * Comment out jobs fopr quick testing Signed-off-by: noopur <[email protected]> * Modified Signed-off-by: noopur <[email protected]> * Modified Signed-off-by: noopur <[email protected]> * Modified Signed-off-by: noopur <[email protected]> * Pass on token Signed-off-by: noopur <[email protected]> * Inherit secret Signed-off-by: noopur <[email protected]> * Do not pass env from called wf Signed-off-by: noopur <[email protected]> * Do not pass env from called wf Signed-off-by: noopur <[email protected]> * Revert the testing changes Signed-off-by: noopur <[email protected]> * Renmote printing job Signed-off-by: noopur <[email protected]> * Run publish step for manual trigger as well Signed-off-by: noopur <[email protected]> * Ignore case for Aggregated model validation score Signed-off-by: noopur <[email protected]> * Read 10 lines Signed-off-by: noopur <[email protected]> * For trusted publishing Signed-off-by: noopur <[email protected]> * Revert previous change Signed-off-by: noopur <[email protected]> * Search string in lowercase Signed-off-by: noopur <[email protected]> * Review comments incorp Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Collaborator CLI command for pinging the aggregator (#1516) * Adding a collaborator command for pinging the aggregator, without starting any tasks Signed-off-by: Teodor Parvanov <[email protected]> * Addressing review comments Signed-off-by: Teodor Parvanov <[email protected]> * Additional logging for the "fx collaborator ping" command Signed-off-by: Teodor Parvanov <[email protected]> * Additional documentation on the usage of fx collaborator ping Signed-off-by: Teodor Parvanov <[email protected]> --------- Signed-off-by: Teodor Parvanov <[email protected]> Signed-off-by: yes <[email protected]> * Handling for .proto files in openfl-nightly package (#1523) * Taking latest nbdev as previous giving error Signed-off-by: Chaurasiya, Payal <[email protected]> * Taking latest nbdev as previous giving error Signed-off-by: Chaurasiya, Payal <[email protected]> * Security TLS certificate report tests Signed-off-by: Chaurasiya, Payal <[email protected]> * Security TLS certificate report tests Signed-off-by: Chaurasiya, Payal <[email protected]> * FedEval fix Signed-off-by: Chaurasiya, Payal <[email protected]> * Small change Signed-off-by: Chaurasiya, Payal <[email protected]> * Logs correction Signed-off-by: Chaurasiya, Payal <[email protected]> * Format changes Signed-off-by: Chaurasiya, Payal <[email protected]> * To streamline Collaborator yaml for torch/mnist with all other cols.yaml file Signed-off-by: Chaurasiya, Payal <[email protected]> * change tp fpdf2 from fpdf Signed-off-by: Chaurasiya, Payal <[email protected]> * change tp fpdf2 from fpdf Signed-off-by: Chaurasiya, Payal <[email protected]> * review comments Signed-off-by: Chaurasiya, Payal <[email protected]> * Fix for openfl-nightly package Signed-off-by: Chaurasiya, Payal <[email protected]> * Revert changes from pre-test Signed-off-by: Chaurasiya, Payal <[email protected]> --------- Signed-off-by: Chaurasiya, Payal <[email protected]> Signed-off-by: yes <[email protected]> * Run an E2E test before publishing openfl-nightly package (#1525) * Check wheel file name Signed-off-by: noopur <[email protected]> * Added test case run Signed-off-by: noopur <[email protected]> * Added test case run Signed-off-by: noopur <[email protected]> * Minor corrections Signed-off-by: noopur <[email protected]> * Github summary step Signed-off-by: noopur <[email protected]> * Modified Signed-off-by: noopur <[email protected]> * Final changes Signed-off-by: noopur <[email protected]> * Remove extra WHL_FILE assignment Signed-off-by: noopur <[email protected]> * Final changes Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Verifiable Dataset (#1434) * Verifiable Dataset Signed-off-by: Dar, Efrat <[email protected]> * Pytorch classes for verifaiable dataset loading and verify Signed-off-by: Dar, Efrat <[email protected]> * Move code to openfl.federated.data.sources Signed-off-by: Dar, Efrat <[email protected]> * New validation method (comb of concise and verbose): When create hash, save in memory both root_hash (concise) and all_hashes (verbose) Then when validate the whole dataset, verify against root_hash only, and when validate a single object, verify against relevant hash from all_hashes. When serialize to json, save only root_hash. Signed-off-by: Dar, Efrat <[email protected]> * Fixes Signed-off-by: Dar, Efrat <[email protected]> * Fix: move base_path to LocalDataSource Signed-off-by: Dar, Efrat <[email protected]> * Fixes Signed-off-by: Dar, Efrat <[email protected]> --------- Signed-off-by: Dar, Efrat <[email protected]> Co-authored-by: teoparvanov <[email protected]> Signed-off-by: yes <[email protected]> * Update HippMapp3r plan, requirments and readme (#1524) * fixes Signed-off-by: porteratzo <[email protected]> * Update openfl-workspace/keras/hippmapp3r/README.md Co-authored-by: Patrick Foley <[email protected]> Signed-off-by: porteratzo <[email protected]> * Update openfl-workspace/keras/hippmapp3r/README.md Co-authored-by: Patrick Foley <[email protected]> Signed-off-by: porteratzo <[email protected]> --------- Signed-off-by: porteratzo <[email protected]> Co-authored-by: Patrick Foley <[email protected]> Signed-off-by: yes <[email protected]> * Task callback enhancement / Bugfix (#1469) * task callback support * adjust save model to reflect right tensor key * bump round number after saving model so tasks does not use advanced round number until model is updated * set extra params on the callback object and not function for flexibility based on CR comments * use round as is from persistent store * attempt for lambda filtering to avoid duplicate iterations --------- Signed-off-by: Balakrishnan, Senthil <[email protected]> Signed-off-by: yes <[email protected]> * Removed keras/jax/mnist from dockerized ws run (#1532) Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * No-op workspace for TaskRunner API (#1527) * Initializing a no-op workspace Signed-off-by: Teodor Parvanov <[email protected]> * Refactor no-op task runner and data loader to workspace Signed-off-by: Teodor Parvanov <[email protected]> * Fixed copyright headers Signed-off-by: Teodor Parvanov <[email protected]> * Workaround for model initialization for the no-op workspace Signed-off-by: Teodor Parvanov <[email protected]> * Additionally standardizing the copyright headers Signed-off-by: Teodor Parvanov <[email protected]> * Adding a README.md Signed-off-by: Teodor Parvanov <[email protected]> --------- Signed-off-by: Teodor Parvanov <[email protected]> Co-authored-by: Payal Chaurasiya <[email protected]> Signed-off-by: yes <[email protected]> * Add empty requirements.txt to the no-op workspace template (#1536) Signed-off-by: Teodor Parvanov <[email protected]> Signed-off-by: yes <[email protected]> * Removing the references of federated-evaluation/aggregator.yaml (#1535) Signed-off-by: yes <[email protected]> * undo round number updates (#1537) Signed-off-by: Balakrishnan, Senthil <[email protected]> Signed-off-by: yes <[email protected]> * Adding workflow for creating openfl Dependency list (#1530) * create ospdt dependency workflow * dependency list * fixed code format * create ospdt dependency workflow * dependency list * fixed code format * modified filename as release_version-date.xlsx * added format * resolved comments * resolved comments --------- Co-authored-by: Rajith <[email protected]> Signed-off-by: yes <[email protected]> * Run a FedEval testcase in PR pipeline (#1539) * Run FedEval test case in PR pipeline Signed-off-by: noopur <[email protected]> * Added condition for non-draft PR as well Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Fixing bandit issue (#1540) Signed-off-by: Chaurasiya, Payal <[email protected]> Signed-off-by: yes <[email protected]> * return from save_model immediately if federation evaluation mode is enabled (#1538) * return from save_model immediately if federation evaluation mode is enabled Signed-off-by: Rahul Garg <[email protected]> * fix the format issue Signed-off-by: Rahul Garg <[email protected]> * Format issue Signed-off-by: Rahul Garg <[email protected]> * Fix the test case and move the location to skip the model save Signed-off-by: Rahul Garg <[email protected]> * remove unnecessary code Signed-off-by: Rahul Garg <[email protected]> * formatting issue fix Signed-off-by: Rahul Garg <[email protected]> * fix linting issue Signed-off-by: Rahul Garg <[email protected]> * addressing comments Signed-off-by: Rahul Garg <[email protected]> --------- Signed-off-by: Rahul Garg <[email protected]> Co-authored-by: Payal Chaurasiya <[email protected]> Signed-off-by: yes <[email protected]> * TaskRunner FedEval - fetch model score from metric file instead of tensor db (#1543) * Read accuracy from metric file for FedEval Signed-off-by: noopur <[email protected]> * Read accuracy from metric file for FedEval Signed-off-by: noopur <[email protected]> * Variable name correction Signed-off-by: noopur <[email protected]> * Consider only numeric value of the line Signed-off-by: noopur <[email protected]> * File converted to json and some lint fixes Signed-off-by: noopur <[email protected]> * Explicit run of test-requirements.txt in GaNDLF Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Load native model with fx plan initialize (#1528) * enable loading of model path in native format Signed-off-by: kta-intel <[email protected]> * add load_native to runner_xgb.py Signed-off-by: kta-intel <[email protected]> * formatting Signed-off-by: kta-intel <[email protected]> * check pbuf file extension Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: kta-intel <[email protected]> Co-authored-by: teoparvanov <[email protected]> Signed-off-by: yes <[email protected]> * Minor bug fix to `runner_xgb.py` to properly rebuild model during aggregated model validation (#1529) * fix conditional to rebuild model only during aggregated validation Signed-off-by: kta-intel <[email protected]> * remove print statement Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: kta-intel <[email protected]> Co-authored-by: Payal Chaurasiya <[email protected]> Co-authored-by: teoparvanov <[email protected]> Signed-off-by: yes <[email protected]> * TaskRunner E2E - Added missing `needs` for straggler and eden jobs (#1545) * Sort collaborators while displaying Signed-off-by: noopur <[email protected]> * Added needs in straggler and eden jobs Signed-off-by: noopur <[email protected]> * Input selection fix Signed-off-by: noopur <[email protected]> * Syntax fix Signed-off-by: noopur <[email protected]> * Check last few lines based on no of collabs Signed-off-by: noopur <[email protected]> * Check last few lines based on no of collabs Signed-off-by: noopur <[email protected]> * Revert sorting logic Signed-off-by: noopur <[email protected]> * Simplified the no of lines logic Signed-off-by: noopur <[email protected]> * Consider all lines if lesser than the index Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * Adding Federation Runtime to Federated Evaluation workflow (#1519) * Adding Federation Runtime to Federated Evaulation workflow * removed MNIST_FedEvaluation * Adding Federation Runtime to Federated Evaulation workflow * removed MNIST_FedEvaluation * Update openfl-tutorials/experimental/workflow/FederatedEvaluation/workspace/MNIST_FederatedEvaluation.ipynb Co-authored-by: Copilot <[email protected]> * added doc string --------- Co-authored-by: Rajith <[email protected]> Co-authored-by: Copilot <[email protected]> Signed-off-by: yes <[email protected]> * Documentation: minor correction in `MNIST Example: Workflow API` (#1549) * Minor correction in Workflow API doc Signed-off-by: noopur <[email protected]> * Revert some unintended changes Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]> Signed-off-by: yes <[email protected]> * rename analysis to analytics, moved code to src folder Signed-off-by: yes <[email protected]> * changes to remove keras model Signed-off-by: yes <[email protected]> * changes to remove keras model Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * added callback on_round_end for FA Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> * code changes Signed-off-by: yes <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: yes <[email protected]> Signed-off-by: Shah, Karan <[email protected]> Signed-off-by: Chaurasiya, Payal <[email protected]> Signed-off-by: Situ Singh <[email protected]> Signed-off-by: noopur <[email protected]> Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: Teodor Parvanov <[email protected]> Signed-off-by: Pant, Akshay <[email protected]> Signed-off-by: Yaroslav Halchenko <[email protected]> Signed-off-by: Lior Malka <[email protected]> Signed-off-by: kta-intel <[email protected]> Signed-off-by: payalcha <[email protected]> Signed-off-by: sarthakpati <[email protected]> Signed-off-by: porteratzo <[email protected]> Signed-off-by: refai06 <[email protected]> Signed-off-by: Dar, Efrat <[email protected]> Signed-off-by: Balakrishnan, Senthil <[email protected]> Signed-off-by: Rahul Garg <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Karan Shah <[email protected]> Co-authored-by: Payal Chaurasiya <[email protected]> Co-authored-by: Situ Singh <[email protected]> Co-authored-by: Noopur <[email protected]> Co-authored-by: Akshay Pant <[email protected]> Co-authored-by: teoparvanov <[email protected]> Co-authored-by: Yaroslav Halchenko <[email protected]> Co-authored-by: Lior Malka <[email protected]> Co-authored-by: Kevin Ta <[email protected]> Co-authored-by: Sarthak Pati <[email protected]> Co-authored-by: porteratzo <[email protected]> Co-authored-by: refai06 <[email protected]> Co-authored-by: Efrat1 <[email protected]> Co-authored-by: Patrick Foley <[email protected]> Co-authored-by: senthil <[email protected]> Co-authored-by: rahulga1 <[email protected]> Co-authored-by: rajithkrishnegowda <[email protected]> Co-authored-by: Rajith <[email protected]> Co-authored-by: Copilot <[email protected]>
1 parent 79fb0ca commit ee8fe0a

File tree

12 files changed

+419
-2
lines changed

12 files changed

+419
-2
lines changed

‎openfl-workspace/federated_analytics/histogram/.workspace‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
current_plan_name: default
2+

‎openfl-workspace/federated_analytics/histogram/plan/cols.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Copyright (C) 2025 Intel Corporation
2+
# Licensed subject to the terms of the separately executed evaluation license agreement between Intel Corporation and you.
3+
4+
collaborators:

‎openfl-workspace/federated_analytics/histogram/plan/data.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Copyright (C) 2025 Intel Corporation
2+
# Licensed subject to the terms of the separately executed evaluation license agreement between Intel Corporation and you.
3+
4+
# collaborator_name,data_directory_path
5+
one,1

‎openfl-workspace/federated_analytics/histogram/plan/defaults‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
../../workspace/plan/defaults
2+

‎openfl-workspace/federated_analytics/histogram/plan/plan.yaml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Copyright (C) 2025 Intel Corporation
2+
# Licensed subject to the terms of the separately executed evaluation license agreement between Intel Corporation and you.
3+
4+
aggregator :
5+
defaults : plan/defaults/aggregator.yaml
6+
template : openfl.component.Aggregator
7+
settings :
8+
last_state_path : save/result.json
9+
rounds_to_train : 1 # Number of training rounds (set to 1 for Federated Analytics).
10+
11+
collaborator :
12+
defaults : plan/defaults/collaborator.yaml
13+
template : openfl.component.Collaborator
14+
settings :
15+
use_delta_updates : false
16+
opt_treatment : RESET
17+
18+
data_loader :
19+
defaults : plan/defaults/data_loader.yaml
20+
template : src.dataloader.IRISInMemory
21+
settings :
22+
collaborator_count : 2
23+
data_group_name : iris
24+
batch_size : 150
25+
26+
task_runner :
27+
defaults : plan/defaults/task_runner.yaml
28+
template : src.taskrunner.IrisHistogram
29+
30+
network :
31+
defaults : plan/defaults/network.yaml
32+
33+
assigner :
34+
template : openfl.component.RandomGroupedAssigner
35+
settings :
36+
task_groups :
37+
- name : analytics
38+
percentage : 1.0
39+
tasks :
40+
- analytics
41+
42+
tasks :
43+
analytics:
44+
function : analytics
45+
aggregation_type:
46+
template: src.aggregatehistogram.AggregateHistogram
47+
48+
kwargs :
49+
columns: ['sepal length (cm)', 'sepal width (cm)']
50+

‎openfl-workspace/federated_analytics/histogram/requirements.txt‎

Whitespace-only changes.

‎openfl-workspace/federated_analytics/histogram/src/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Copyright (C) 2025 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
"""You may copy this file as the starting point of your own model."""

‎openfl-workspace/federated_analytics/histogram/src/aggregatehistogram.py‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Copyright (C) 2025 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
5+
"""Histogram module."""
6+
7+
import numpy as np
8+
9+
from openfl.interface.aggregation_functions.core import AggregationFunction
10+
11+
12+
class AggregateHistogram(AggregationFunction):
13+
"""Histogram aggregation."""
14+
15+
def call(self, local_tensors, *_) -> np.ndarray:
16+
"""
17+
Aggregates a list of local histograms into a single global histogram.
18+
19+
This method takes a list of objects, each containing a `tensor` attribute
20+
(which represents a local histogram as a numpy array), and combines them
21+
into a single aggregated histogram. A histogram is a representation of
22+
the distribution of data, typically used to count occurrences of values
23+
within specified ranges (bins). Aggregating histograms involves summing
24+
up the corresponding bins across all local histograms to produce a global
25+
view of the data distribution.
26+
27+
local_tensors (list): A list of objects, where each object has a
28+
`tensor` attribute that is a numpy array representing a local
29+
histogram.
30+
31+
np.ndarray: The aggregated histogram as a numpy array. Each bin in
32+
the resulting histogram is the sum of the corresponding bins from
33+
all input histograms. If the input list is empty, raises a
34+
ValueError indicating that the result is empty.
35+
36+
Raises:
37+
ValueError: If the input list `local_tensors` is empty, indicating
38+
that there are no histograms to aggregate.
39+
"""
40+
41+
if not local_tensors:
42+
raise ValueError("Histogram result is empty.")
43+
44+
agg_histogram = np.zeros_like(local_tensors[0].tensor)
45+
for local_tensor in local_tensors:
46+
agg_histogram += local_tensor.tensor
47+
return agg_histogram

‎openfl-workspace/federated_analytics/histogram/src/dataloader.py‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Copyright (C) 2025 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
from openfl.federated.data.loader import DataLoader
4+
from sklearn.datasets import load_iris
5+
import pandas as pd
6+
7+
8+
class IRISInMemory(DataLoader):
9+
"""Data Loader for IRIS Dataset."""
10+
11+
def __init__(self, batch_size, data_path, **kwargs):
12+
super().__init__(**kwargs)
13+
14+
# download data
15+
self._download_raw_data()
16+
# create shards
17+
self.data_shard = self.load_data_shard(
18+
shard_num=int(data_path), **kwargs
19+
)
20+
21+
def _download_raw_data(self):
22+
"""
23+
Downloads the raw Iris dataset and saves it as a CSV file.
24+
This method loads the Iris dataset using the `load_iris` function from
25+
the `sklearn.datasets` module. The dataset is then converted to a
26+
pandas DataFrame and saved as a CSV file named 'client.csv' in the
27+
'./data/' directory.
28+
Returns:
29+
None
30+
"""
31+
32+
iris = load_iris(as_frame=True)
33+
data = iris['data']
34+
data.to_csv('./data/client.csv', index=False)
35+
36+
def _load_data(self):
37+
"""
38+
Loads data from a CSV file.
39+
This method reads the data from a CSV file located at './data/client.csv'
40+
and returns it as a pandas DataFrame.
41+
Returns:
42+
pd.DataFrame: The data loaded from the CSV file.
43+
"""
44+
45+
return pd.read_csv('./data/client.csv')
46+
47+
def load_data_shard(self, shard_num, collaborator_count, **kwargs):
48+
"""
49+
Load a specific shard of dataset for a given collaborator.
50+
Args:
51+
shard_num (int): The shard number to load.
52+
collaborator_count (int): The total number of collaborators.
53+
**kwargs: Additional keyword arguments.
54+
Returns:
55+
pandas.DataFrame: The shard of dataset corresponding to the given shard number.
56+
"""
57+
58+
return self._load_data().iloc[shard_num-1::collaborator_count]
59+
60+
def query(self, columns, **kwargs):
61+
"""
62+
Query the data shard for the specified columns.
63+
Args:
64+
columns (list): A list of column names to query from the data shard.
65+
**kwargs: Additional keyword arguments (currently not used).
66+
Returns:
67+
DataFrame: A DataFrame containing the data for the specified columns.
68+
Raises:
69+
ValueError: If the columns parameter is not a list.
70+
"""
71+
if not isinstance(columns, list):
72+
raise ValueError("Columns parameter must be a list")
73+
return self.data_shard[columns]
74+
75+
def get_feature_shape(self):
76+
"""
77+
This function is not required and is kept for compatibility.
78+
79+
Returns:
80+
None
81+
"""
82+
pass

‎openfl-workspace/federated_analytics/histogram/src/runner_fa.py‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Copyright (C) 2025 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
5+
"""
6+
Base classes for Federated Analytics.
7+
8+
This file can serve as a template for creating your own Federated Analytics experiments.
9+
"""
10+
11+
from openfl.federated.task.runner import TaskRunner
12+
from openfl.utilities import TensorKey
13+
from openfl.utilities.split import split_tensor_dict_for_holdouts
14+
15+
import logging
16+
import numpy as np
17+
18+
logger = logging.getLogger(__name__)
19+
20+
21+
class FederatedAnalyticsTaskRunner(TaskRunner):
22+
"""The base class for Federated Analytics Task Runner."""
23+
24+
def __init__(self, **kwargs):
25+
"""Initializes the FederatedAnalyticsTaskRunner instance.
26+
27+
Args:
28+
**kwargs: Additional parameters to pass to the function
29+
"""
30+
super().__init__(**kwargs)
31+
32+
# Dummy model initialization. Dummy models and weights are used here as placeholders
33+
# to ensure compatibility with the core OpenFL framework, which currently assumes
34+
# the presence of a model for federated learning tasks.
35+
#
36+
# This approach is necessary to support Federated Analytics use cases, which do not
37+
# involve traditional model training, until OpenFL is refactored to accommodate
38+
# broader use cases beyond learning.
39+
#
40+
# For more details, refer to the discussion at:
41+
# https://github.com/securefederatedai/openfl/discussions/1385#discussioncomment-13009961.
42+
self.model = None
43+
44+
self.model_tensor_names = []
45+
self.required_tensorkeys_for_function = {}
46+
47+
def analytics(self, col_name, round_num, **kwargs):
48+
"""
49+
Return analytics result as tensors.
50+
51+
Args:
52+
col_name (str): collaborator name.
53+
round_num (int): The current round number.
54+
**kwargs: Additional parameters for analysis.
55+
56+
Returns:
57+
dict: A dictionary of analysis results.
58+
"""
59+
results = self.analytics_task(**kwargs)
60+
tags = ("analytics",)
61+
origin = col_name
62+
output_metric_dict = {
63+
TensorKey(metric_name, origin, round_num, False, tags): metric_value
64+
for metric_name, metric_value in results.items()
65+
}
66+
return output_metric_dict, output_metric_dict
67+
68+
def analytics_task(self, **kwargs):
69+
"""
70+
Perform analytics on the provided data.
71+
This method should be implemented by subclasses to perform specific analysis tasks.
72+
Args:
73+
**kwargs: Arbitrary keyword arguments that can be used for analysis.
74+
Raises:
75+
NotImplementedError: If the method is not implemented by a subclass.
76+
"""
77+
raise NotImplementedError
78+
79+
def get_tensor_dict(self, with_opt_vars, suffix=""):
80+
"""
81+
Get the model weights as a tensor dictionary.
82+
83+
Args:
84+
with_opt_vars (bool): If we should include the optimizer's status.
85+
suffix (str): Universally.
86+
87+
Returns:
88+
model_weights (dict): The tensor dictionary.
89+
"""
90+
return {'dummy_tensor': np.float32(1)}
91+
92+
def get_required_tensorkeys_for_function(self, func_name, **kwargs):
93+
"""Get the required tensors for specified function that could be called
94+
as part of a task.
95+
96+
By default, this is just all of the layers and optimizer of the dummy model.
97+
98+
Args:
99+
func_name (str): The function name.
100+
**kwargs: Any function arguments.
101+
102+
Returns:
103+
list: List of TensorKey objects.
104+
"""
105+
return []
106+
107+
def initialize_tensorkeys_for_functions(self, with_opt_vars=False):
108+
"""
109+
This function is not required and is kept for compatibility.
110+
111+
Returns:
112+
None
113+
"""
114+
pass

0 commit comments

Comments
 (0)