Validation for Model Configs #1306

rishi-aga · 2020-05-06T23:33:53Z

Authored-by: rishi-aga

One of the multiple PRs that will resolve #1178

Description

Validation/Dry Run for Dynamic Config

Motivation and Context

Please refer #1178

How Has This Been Tested?

Unit tests have been included.

License

I confirm that this contribution is made under an Apache 2.0 license and that I have the authority necessary to make this contribution on behalf of its copyright owner.

moizarafat · 2020-05-12T01:47:52Z

elide-contrib/elide-dynamic-config-helpers/README.md

+Usage: java -cp <Jar File Name> com.yahoo.elide.contrib.dynamicconfighelpers.validator.DynamicConfigValidator <Path for Model Configs Directory>
+```
+Expected Model Configs Directory Structure:
+```java


is any thing other than Java be used here?

changed to text

moizarafat · 2020-05-12T01:49:04Z

elide-contrib/elide-dynamic-config-helpers/README.md

+Validate the config files in local before deployment.
+
+Sample command:
+```java


lets include the mvn command to generate the jar too.

or mention downloading the jar from maven.

added mvn install steps

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

moizarafat · 2020-05-18T23:07:31Z

elide-contrib/elide-dynamic-config-helpers/src/test/resources/validator/valid/security.hjson

+    rules: [
+        {
+            type: filter
+            filter: company_id=in=${principal.companies}


as per the latest schema, this value will not work.

changed to filter enum

moizarafat · 2020-05-18T23:07:45Z

elide-contrib/elide-dynamic-config-helpers/src/test/resources/validator/valid/security.hjson

+            name: User belongs to company <% bar%>
+        },
+        {
+            filter: id==${principal.id}


as per the latest schema, this value will not work.

changed to filter enum

moizarafat · 2020-05-18T23:07:58Z

...elide-dynamic-config-helpers/src/test/resources/validator/undefined_handlebar/security.hjson

+    rules: [
+        {
+            type: filter
+            filter: company_id=in=${principal.companies}


as per the latest schema, this value will not work.

changed to filter enum

moizarafat · 2020-05-18T23:08:03Z

...elide-dynamic-config-helpers/src/test/resources/validator/undefined_handlebar/security.hjson

+            name: User belongs to company
+        },
+        {
+            filter: id==${principal.id}


as per the latest schema, this value will not work.

changed to filter enum

aklish · 2020-05-19T17:46:33Z

...helpers/src/main/java/com/yahoo/elide/contrib/dynamicconfighelpers/DynamicConfigHelpers.java

-    private static final String SECURITY_CONFIG_PATH = "security.hjson";
-    private static final String VARIABLE_CONFIG_PATH = "variables.hjson";
-    private static final String NEW_LINE = "\n";
+    public static final String TABLE_CONFIG_PATH = "tables/";


Use File.separator.

aklish · 2020-05-19T17:52:51Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+    private static ElideTableConfig elideTableConfig;
+    private static ElideSecurityConfig elideSecurityConfig;
+    private static Map<String, Object> variables;
+    private static final String[] SQL_DISALLOWED_WORDS = new String[] { "DROP ", "TRUNCATE ", "DELETE ", "INSERT ",


There are more DDL and DCL commands than this:
ALTER, COMMENT, CREATE, DESCRIBE, SHOW, USE, GRANT, REVOKE, CONNECT, LOCK, EXPLAIN, CALL, MERGE, RENAME

We should also add a semicolon to the list.

Added additional keywords.
Added explicit check that sql should not contain semicolon.
Changed to split SQL on whitespaces and match exact keywords now

aklish · 2020-05-19T17:54:13Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+            throw new IllegalStateException("No Arguments provided!");
+        }
+
+        if (args.length > 1 || DynamicConfigHelpers.isNullOrEmpty(args[0])) {


Use an argument parser like org.apache.commons.cli.Options

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

aklish · 2020-05-19T19:21:57Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+    }
+
+    private static boolean readSecurityConfig(String absoluteBasePath) {
+        boolean isSecurityConfig = exists(absoluteBasePath + DynamicConfigHelpers.SECURITY_CONFIG_PATH);


Build the full path once in this function.

aklish · 2020-05-19T19:22:36Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+            validateConfigForMissingVariables(securityConfigContent, variables);
+            try {
+                elideSecurityConfig = DynamicConfigHelpers.getElideSecurityPojo(absoluteBasePath, variables);
+            } catch (Exception e) {


As a general rule, most code should not throw or catch Exception. Please be explicit.

Only checked exceptions need to be mapped to IllegalStateException. Runtime exceptions don't need to be swallowed here.

aklish · 2020-05-19T19:24:05Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+    }
+
+    private static boolean readTableConfig(String absoluteBasePath) {
+        boolean isTableConfig = exists(absoluteBasePath + DynamicConfigHelpers.TABLE_CONFIG_PATH);


Build the file path once.

aklish · 2020-05-19T19:25:07Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+            }
+            try {
+                elideTableConfig = DynamicConfigHelpers.getElideTablePojo(absoluteBasePath, variables);
+            } catch (Exception e) {


Don't catch or throw Exception.

aklish · 2020-05-19T19:31:58Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+    }
+
+    private static boolean containsDisallowedWords(String str, String[] keywords) {
+        return Arrays.stream(keywords).parallel().anyMatch(str.toUpperCase(Locale.ENGLISH)::contains);


Probably faster not to use parallel here. We are talking about a dozen or so keywords and a small set of tables.

removed parallel

aklish · 2020-05-19T19:32:14Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+    }
+
+    private static void usage() {
+        log.info("Usage: java -cp <Jar File Name>"


use cli.options.

moizarafat · 2020-05-21T02:46:33Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+
+    public static void main(String[] args) throws IOException, ParseException {
+
+        Options options = new Options();


I think Options options can be a static final variable cos it does not change.

private static final Options options = prepareOptions()
private static Options prepareOptions() { Options options = new Options(); defineArgs(options); }

moizarafat · 2020-05-21T02:47:38Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+                + "./tables/table2.hjson\n"
+                + "./tables/tableN.hjson\n");
+        configDirOption.setRequired(true);
+        options.addOption(configDirOption);


We should include another -h/--help for printing the usage

added option for help. Had to make c/configDir optional now and added explicit check for that if args h/help is not passed

moizarafat · 2020-05-21T02:48:51Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+
+        Options options = new Options();
+        defineArgs(options);
+        usage(options);


instead of printing usage always, lets do -h/--help option for printing the usage on demand.

moizarafat · 2020-05-21T02:50:03Z

.../java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidatorTest.java

+    public void testMissingTableDir() {
+        Exception e = assertThrows(IllegalStateException.class, () -> DynamicConfigValidator
+                .main(new String[] { "--configDir", "src/test/resources/validator/missing_table_dir" }));
+        assertTrue(e.getMessage().startsWith("Table Configs Directory doesn't exists at location"));


The message is Model Configs Directory doesn't exists

one empty directory didnt get check in, checked in now to ensure it runs fine.

moizarafat · 2020-05-21T02:55:32Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+        if (!DynamicConfigHelpers.isNullOrEmpty(sqlDefinition) && (sqlDefinition.contains(SEMI_COLON)
+                || containsDisallowedWords(sqlDefinition, SQL_SPLIT_REGEX, SQL_DISALLOWED_WORDS))) {
+            throw new IllegalStateException(
+                    "sql/definition provided in table config contain either ';' or one of these words: "


';' can be replaced by SEMI_COLON

moizarafat · 2020-05-21T02:57:36Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+    /**
+     * Print Usage.
+     */
+    private static void usage(Options options) {


I think if options is made static final, this method will not need arguments.

coveralls · 2020-05-22T19:19:05Z

Coverage increased (+0.1%) to 80.584% when pulling 82abef2 on elide-5.x-dynamic-config-validator into 3dc187c on elide-5.x.

Co-authored-by: rishi-aga <[email protected]>

aklish · 2020-05-29T14:03:57Z

elide-contrib/elide-dynamic-config-helpers/README.md

+To build and run:
+```text
+1. mvn clean install
+2. Look for Jar File under elide-contrib/elide-dynamic-config-helpers/target directory with name matching "elide-dynamic-config-*-jar-with-dependencies.jar"


You can skip step 2 and provide the relative path (listed above) in step 3a and 3b.

aklish · 2020-05-29T14:16:03Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+     * @return boolean true if variable config file exists else false
+     * @throws JsonProcessingException
+     */
+    private boolean readVariableConfig() throws JsonProcessingException {


Does the dynamic logic read the files twice? Once to build the config for compilation and once to validate? We really ought to read the config once to make sure we are validating the same configuration we are running with.

The main function however can separate read and validate - but the validator needs to be hooked into two places - the utility main function and the normal path when the service launches and compiles the configs.

@aklish @moizarafat

Does the dynamic logic read the files twice? Once to build the config for compilation and once to validate? We really ought to read the config once to make sure we are validating the same configuration we are running with.

Splitted getElideTablePojo & getElideSecurityPojo methods of DynamicConfigHelpers class such that we read the configs only once.

The main function however can separate read and validate - but the validator needs to be hooked into two places - the utility main function and the normal path when the service launches and compiles the configs.

Created a separate class method to read and validate config files which can be called outside of Main function also. Please check if this is what we wanted.

aklish · 2020-05-29T14:21:32Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+    private static final String SQL_SPLIT_REGEX = "\\s+";
+    private static final String SEMI_COLON = ";";
+    private static final Pattern HANDLEBAR_REGEX = Pattern.compile("<%(.*?)%>");
+    private static final Options OPTIONS = prepareOptions();


Let's make this a private member of the main function rather than a static variable.

aklish · 2020-05-29T14:22:51Z

...main/java/com/yahoo/elide/contrib/dynamicconfighelpers/validator/DynamicConfigValidator.java

+     * @param elideTableConfig ElideTableConfig
+     * @return boolean true if all sql/definition passes validation
+     */
+    private static boolean validateSqlInTableConfig(ElideTableConfig elideTableConfig) {


Nice functional breakdown for the validation logic.

* Dynamic Config Validator Co-authored-by: rishi-aga <[email protected]> * Review Comments * Review Comments * Use DynamicConfigValidator instead of ElideConfigParser Co-authored-by: moiz arafat <[email protected]>

* parent a14ccfbb7b558d27799b4e7b1916850519639708 author Jack (정환) <[email protected]> 1559678988 -0700 committer Aaron Klish <[email protected]> 1589583449 -0500 Create AggregationDataStore module (#845) * Create AggregationDataStore module * Address Aaron's comments * Fix build failure AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron Define QueryEngine Contract (#867) Fixed rebase on master SQL Query Engine (#878) * AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron Added calcite as a dependency. Merged in changes for QueryEngine interface Fixed checkstyle issues Added basic H2 DB test harness Started breaking out projections Moved getValue and setValue from PersistentResource to EntityDictionary Added basic logic to hydrate entities Added FromTable and FromSubquery annotations. Add explicit exclusion of entity relationship hydration Minor cleanup Refactored HQLFilterOperation to take an alias generator Added test support for RSQL filter generation. Some cleanup Added basic support for WHERE clause filtering on the fact table Added working test for subquery SQL Added basic join logic for filters Added a test with a subquery and a filter join Refactored Schema classes and Query to support metric aggregation SQL expansion Added group by support Added logic for ID generation Added sorting logic and test Added pagination support and testing All column references use proper name now for SQL Removed calcite as a query engine Refactored HQLFilterOperation so it can be used for Having and Where clause generaiton Added HAVING clause support Changed Query to take schema instead of entityClass First pass at cleanup Fixed checkstyles Cleanup Cleanup Added a complex SQL expression test and fixed bugs Fixed merge issues. Added another test. Added better logging Fixed bug in pagination SQL generation * Build is working * Inspection rework Add EntityProjection plumbing (#949) * AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron * Initial sketch PersistentResourceTest now passes LifeCycleTest tests now pass More API changes for data store transaction. Also fixed createObject in persistent resource to take the correct projection Started to refactor tests IncludedProcessorTest refactored Refactored LifeCycleTest Started refactor on PersistentResourceTest More refactoring. Fixed a bug in Resource.toPersistentResource Only one test failing in PersistentResourceTest PersistentResourceTests now pass UpdateOnCreateTests now pass Added skeleton for translating JSON-API URL path into an EntityProjection Basic EntityProjectionMaker almost complete Added ability to merge entity projections by relationship Added first test for EntityProjectionMaker Non-working veresion (but clean) Tests now pass All EntityProjectioNMaker tests pass Elide-Core now builds Added handling of sparse attributes and relationships Expanding attributes for included entities Fixed a number of bugs found in IT tests Fixed some of the EntityProjectionMaker tests Fixed unit tests Made temporary modifications to exclude GraphQL (Build now passes) Added sparse field unit tests for EntityProjectionMaker * Fixed build issues after rebase * Removed duplicated Schema class from rebase * Entity projection with aliases (#963) * Hacked up PersistentResource with new design * Core now compiles (and tests can run * EntityProjectionMaker tests pass * Build now passes (major cleanup still needed * Wire in entity projection4 json api (#964) * Fixed DataStore API. Fixed a lot of the core unit tests * Checkstyles and more fixes * Hibernate 5 Tests Pass * Full build passes * Wire in entity projection4 json api (#965) * Initial concept. No testing changed. * Core compiles and EntityProjectionMaker tests (original ones) now pass * Minor edits to TestRequestScope * Full build passes now * removed entity dictionary from entity projection * Pre-inspection cleanup * minor inspection fixup Hydrate Relationship (#987) * AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron * Added basic H2 DB test harness * Started breaking out projections * Moved getValue and setValue from PersistentResource to EntityDictionary * Added basic logic to hydrate entities * Added FromTable and FromSubquery annotations. Add explicit exclusion of entity relationship hydration * Refactored HQLFilterOperation to take an alias generator * Added test support for RSQL filter generation. Some cleanup * Added basic support for WHERE clause filtering on the fact table * Added working test for subquery SQL * Added basic join logic for filters * Added a test with a subquery and a filter join * Refactored Schema classes and Query to support metric aggregation SQL expansion * Added group by support * Added logic for ID generation * Added sorting logic and test * Added pagination support and testing * All column references use proper name now for SQL * Removed calcite as a query engine * Refactored HQLFilterOperation so it can be used for Having and Where clause generaiton * Added HAVING clause support * Changed Query to take schema instead of entityClass * First pass at cleanup * Fixed checkstyles * Cleanup * Hydrate Relationship * Cleanup * Added a complex SQL expression test and fixed bugs * Fixed merge issues. Added another test. Added better logging * Self-review * Self-review * Self-review * Self-review * Self-review * Address comments from @aklish * Refactor EntityHydrator (#893) * rebase * keep Jiaqi's changes * fix id * fix maven verify * Remove HQLFilterOperation * fix dictionary * fix SqlEngineTest * remove unused part * make codacy happy * should use getParametrizedType * address comments Implement GraphQLEntityProjectionMaker (#986) * AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron * Initial sketch PersistentResourceTest now passes LifeCycleTest tests now pass More API changes for data store transaction. Also fixed createObject in persistent resource to take the correct projection Started to refactor tests IncludedProcessorTest refactored Refactored LifeCycleTest Started refactor on PersistentResourceTest More refactoring. Fixed a bug in Resource.toPersistentResource Only one test failing in PersistentResourceTest PersistentResourceTests now pass UpdateOnCreateTests now pass Added skeleton for translating JSON-API URL path into an EntityProjection Basic EntityProjectionMaker almost complete Added ability to merge entity projections by relationship Added first test for EntityProjectionMaker Non-working veresion (but clean) Tests now pass All EntityProjectioNMaker tests pass Elide-Core now builds Added handling of sparse attributes and relationships Expanding attributes for included entities Fixed a number of bugs found in IT tests Fixed some of the EntityProjectionMaker tests Fixed unit tests Made temporary modifications to exclude GraphQL (Build now passes) Added sparse field unit tests for EntityProjectionMaker * Fixed build issues after rebase * GraphQL projection maker using document * Argument handling and fragment check * Add comments * Add fragment resolver * fix typo * break code into more methods * remove pagination and sorting * Removed duplicated Schema class from rebase * re-arrange keywords * Address comment * Add arguments for attribute fields * Handle arguments * support partial query, update edges/node logic * Entity projection with aliases * Entity projection with aliases (#963) * Hacked up PersistentResource with new design * Core now compiles (and tests can run * EntityProjectionMaker tests pass * Build now passes (major cleanup still needed * fix create relationship object using entity * Add tests passed * code clean up * refactor fatcher, fix test cases * rename keywords * rebase branch (#12) * rebased * Graphql projection refactor (#13) * fix fragment resolver * Fix variable resolver * Wire in entity projection4 json api (#964) * Fixed DataStore API. Fixed a lot of the core unit tests * Checkstyles and more fixes * Hibernate 5 Tests Pass * Full build passes * Wire in entity projection4 json api (#965) * Initial concept. No testing changed. * Core compiles and EntityProjectionMaker tests (original ones) now pass * Minor edits to TestRequestScope * Full build passes now * removed entity dictionary from entity projection * Pre-inspection cleanup * minor inspection fixup * rebase * Rebased on AggregationDataStore * clean up extra new lines * address comments * Builder pattern * update comments * remove projection in entity * fix jackson * Hydrate Relationship (#987) (#15) * Address some codecy comments * Add comment for partial query * Reenable tests * Address comments, refactor alias * Add test for alias * swapped test case * fix get type Added AggregationDataStore Code (#991) * Adding testing for aggregation data store * Debugging integration tests * Continuing testing work * AggregationDataStore * AggregationDataStore testing * Added more tests * Aggregation Data Store * Cleaned up testing code * Cleaned up code, fixed helper for AggregationDataStore * end * Fixed checkstyle, other minor fixes * fixed comment * Minor fixes * Fixed id type issue, added exception for queries with no metrics Fixed build (#993) Making TimeDimension an interface (#992) [maven-release-plugin] prepare release 5.0.0-pr1 [maven-release-plugin] prepare for next development iteration Renamed graphQL file to match test (#1002) [maven-release-plugin] prepare release 5.0.0-pr2 [maven-release-plugin] prepare for next development iteration Add JoinTo annotation (#1006) * Added JoinTo Annotation * Added working test * Added TODO comment for next PR * Added TODO comment for next PR * Added Sorting and Filtering support for JoinTo Columns * Fixed IT tests for Aggregation Data Store * Moved entityManager creation to happen separately for each query (#1008) * Moved entityManager creation to happen separately for each query * Closing EntityManager after each query * Inspection rework Column annotation (#1017) * Solved column issue and added QueryEngineFactory * Caching query engine in AggregationDataStore * Fixed column description * Update SQLQueryEngine.java (#1019) * Add SQLMetrix, rearrange packages (#1020) * Add SQLMetrix, rearrange packages * address comment Manager transacton manually (#1021) * Manager transacton manually * Add readonly Hydrate GraphQL Schema with parameterized attributes (#1018) * GraphQL schema expose expected argument name and its type for each attribute * Change empty arguments to unmodifable set AggregationStore: Add multiple time grain definitions to schema (#1022) * Fixed checkstyle warnings and errors. Separated the Query dimension interface from the Schema dimension interface * Added skeleton code to convert entity projection arguments into time grains * Cleanup * Class renames per inspection comments * Inspection comments Refactor time dimension logic (#1028) * Manager transacton manually * Add readonly * some rework * use getTimeDimension() * change exception ISSUE-1026 Add support for @Subselect (#1038) * Manager transacton manually * Add readonly * some rework * use getTimeDimension() * change exception * ISSUE-1026 Add support for @Subselect * Address comments ISSUE-1027 Support join for having clause (#1039) * Manager transacton manually * Add readonly * some rework * use getTimeDimension() * change exception * ISSUE-1027 Support join for having clause function name fixed to enableISO8601Dates (#1052) Support for multiple queries at root is added (#1044) * Support for multiple queries at root is added * Added test with alias * comments resolved Add time grain to GraphQL schema (#1042) * Added basic plumbing to push attributes from the entity projection down to the QueryEngine * Added logic to expand SQL time expression in SQLQueryEngine * Added SQLQueryEngine tests * Added IT tests * The AggregationStore now adds graphql parameters for parameterized columns * Minor refactor * Inspection rework * Minor fix Support multiple query of same entity with different alias (#1055) * Support multiple query of same entity with different alias * add static method to generate keyname for GraphQLProjectionInfo projections * Remove aliasPartialQuerySameAttribute MetadataStore Models (#1068) * Manager transacton manually * Add readonly * some rework * use getTimeDimension() * change exception * Metadatastore models * Address comments * address comments * move root * fix style check SQLQueryTemplate Model (#1073) * SQLQueryTemplate * SQLTables * refactor * update sql dimension projection * update sql dimension projection * clean up dimension projection * refactor sql components * aggregatable field rework * add comments * rearrange packages * Add dimension projection back * fix checkstyle * Add dictionary * Simplify MetricFunction and SQLQueryTemplate * Address comments Integrate Metadata Model and SQLQueryTemplate Model (#1083) * Integrate Metadata Model and SQLQueryTemplate Model * remove AggregationDictionary and AggregationManager * Add timezone * Can only query analyticView Fixed issues with rebase Add auto configuration for aggregation store (#1087) * Added autoconfiguration for QueryEngineFactory * Unified class scanning. Started cleaning up datastores so they only register the entities they manage * Full build passes * Minor cleanup * Minor refactoring * Added EntityManagerFactory bean configuratino * Refactored class scanning for Elide standalone * Updated spring boot starter pom * Removed @Entity from all metadata models. Started cleaning up entity dictionary entity registration * Broken implementation. Just checking in so I can revert if needed. * All tests pass * Added unit tests * Minor cleanup * One more fix * Fixed broken tests * Added package include support back * Class scanning for annotations ignores inherited * Added a test based on inspection comments * Inspection comment fix * Changed initalization of MetadataStore * More inspection rework * Turned back on OWASP scanning * More rework remove @Inherited (#1092) Support Non JPA Entity in AggregationDataStore (#1051) * Create AggregationDataStore module (#845) * Create AggregationDataStore module * Address Aaron's comments * Fix build failure AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron Define QueryEngine Contract (#867) Fixed rebase on master SQL Query Engine (#878) * AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron Added calcite as a dependency. Merged in changes for QueryEngine interface Fixed checkstyle issues Added basic H2 DB test harness Started breaking out projections Moved getValue and setValue from PersistentResource to EntityDictionary Added basic logic to hydrate entities Added FromTable and FromSubquery annotations. Add explicit exclusion of entity relationship hydration Minor cleanup Refactored HQLFilterOperation to take an alias generator Added test support for RSQL filter generation. Some cleanup Added basic support for WHERE clause filtering on the fact table Added working test for subquery SQL Added basic join logic for filters Added a test with a subquery and a filter join Refactored Schema classes and Query to support metric aggregation SQL expansion Added group by support Added logic for ID generation Added sorting logic and test Added pagination support and testing All column references use proper name now for SQL Removed calcite as a query engine Refactored HQLFilterOperation so it can be used for Having and Where clause generaiton Added HAVING clause support Changed Query to take schema instead of entityClass First pass at cleanup Fixed checkstyles Cleanup Cleanup Added a complex SQL expression test and fixed bugs Fixed merge issues. Added another test. Added better logging Fixed bug in pagination SQL generation * Build is working * Inspection rework Add EntityProjection plumbing (#949) * AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron * Initial sketch PersistentResourceTest now passes LifeCycleTest tests now pass More API changes for data store transaction. Also fixed createObject in persistent resource to take the correct projection Started to refactor tests IncludedProcessorTest refactored Refactored LifeCycleTest Started refactor on PersistentResourceTest More refactoring. Fixed a bug in Resource.toPersistentResource Only one test failing in PersistentResourceTest PersistentResourceTests now pass UpdateOnCreateTests now pass Added skeleton for translating JSON-API URL path into an EntityProjection Basic EntityProjectionMaker almost complete Added ability to merge entity projections by relationship Added first test for EntityProjectionMaker Non-working veresion (but clean) Tests now pass All EntityProjectioNMaker tests pass Elide-Core now builds Added handling of sparse attributes and relationships Expanding attributes for included entities Fixed a number of bugs found in IT tests Fixed some of the EntityProjectionMaker tests Fixed unit tests Made temporary modifications to exclude GraphQL (Build now passes) Added sparse field unit tests for EntityProjectionMaker * Fixed build issues after rebase * Removed duplicated Schema class from rebase * Entity projection with aliases (#963) * Hacked up PersistentResource with new design * Core now compiles (and tests can run * EntityProjectionMaker tests pass * Build now passes (major cleanup still needed * Wire in entity projection4 json api (#964) * Fixed DataStore API. Fixed a lot of the core unit tests * Checkstyles and more fixes * Hibernate 5 Tests Pass * Full build passes * Wire in entity projection4 json api (#965) * Initial concept. No testing changed. * Core compiles and EntityProjectionMaker tests (original ones) now pass * Minor edits to TestRequestScope * Full build passes now * removed entity dictionary from entity projection * Pre-inspection cleanup * minor inspection fixup Hydrate Relationship (#987) * AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron * Added basic H2 DB test harness * Started breaking out projections * Moved getValue and setValue from PersistentResource to EntityDictionary * Added basic logic to hydrate entities * Added FromTable and FromSubquery annotations. Add explicit exclusion of entity relationship hydration * Refactored HQLFilterOperation to take an alias generator * Added test support for RSQL filter generation. Some cleanup * Added basic support for WHERE clause filtering on the fact table * Added working test for subquery SQL * Added basic join logic for filters * Added a test with a subquery and a filter join * Refactored Schema classes and Query to support metric aggregation SQL expansion * Added group by support * Added logic for ID generation * Added sorting logic and test * Added pagination support and testing * All column references use proper name now for SQL * Removed calcite as a query engine * Refactored HQLFilterOperation so it can be used for Having and Where clause generaiton * Added HAVING clause support * Changed Query to take schema instead of entityClass * First pass at cleanup * Fixed checkstyles * Cleanup * Hydrate Relationship * Cleanup * Added a complex SQL expression test and fixed bugs * Fixed merge issues. Added another test. Added better logging * Self-review * Self-review * Self-review * Self-review * Self-review * Address comments from @aklish * Refactor EntityHydrator (#893) * rebase * keep Jiaqi's changes * fix id * fix maven verify * Remove HQLFilterOperation * fix dictionary * fix SqlEngineTest * remove unused part * make codacy happy * should use getParametrizedType * address comments Implement GraphQLEntityProjectionMaker (#986) * AggregationDataStore: Schema (#846) * AggregationDataStore: Static Attribute Aggregation * Address comments * Implement TimeDimension and all its supporting components * refactor * Address comments from @aklish * Address comments from @aklish && Tests & Javadoc * Address comments from @aklish * Address comments from @aklish and @hellohanchen * Address comments from Aaron * ToMany is not supported * Address comments from Aaron * Initial sketch PersistentResourceTest now passes LifeCycleTest tests now pass More API changes for data store transaction. Also fixed createObject in persistent resource to take the correct projection Started to refactor tests IncludedProcessorTest refactored Refactored LifeCycleTest Started refactor on PersistentResourceTest More refactoring. Fixed a bug in Resource.toPersistentResource Only one test failing in PersistentResourceTest PersistentResourceTests now pass UpdateOnCreateTests now pass Added skeleton for translating JSON-API URL path into an EntityProjection Basic EntityProjectionMaker almost complete Added ability to merge entity projections by relationship Added first test for EntityProjectionMaker Non-working veresion (but clean) Tests now pass All EntityProjectioNMaker tests pass Elide-Core now builds Added handling of sparse attributes and relationships Expanding attributes for included entities Fixed a number of bugs found in IT tests Fixed some of the EntityProjectionMaker tests Fixed unit tests Made temporary modifications to exclude GraphQL (Build now passes) Added sparse field unit tests for EntityProjectionMaker * Fixed build issues after rebase * GraphQL projection maker using document * Argument handling and fragment check * Add comments * Add fragment resolver * fix typo * break code into more methods * remove pagination and sorting * Removed duplicated Schema class from rebase * re-arrange keywords * Address comment * Add arguments for attribute fields * Handle arguments * support partial query, update edges/node logic * Entity projection with aliases * Entity projection with aliases (#963) * Hacked up PersistentResource with new design * Core now compiles (and tests can run * EntityProjectionMaker tests pass * Build now passes (major cleanup still needed * fix create relationship object using entity * Add tests passed * code clean up * refactor fatcher, fix test cases * rename keywords * rebase branch (#12) * rebased * Graphql projection refactor (#13) * fix fragment resolver * Fix variable resolver * Wire in entity projection4 json api (#964) * Fixed DataStore API. Fixed a lot of the core unit tests * Checkstyles and more fixes * Hibernate 5 Tests Pass * Full build passes * Wire in entity projection4 json api (#965) * Initial concept. No testing changed. * Core compiles and EntityProjectionMaker tests (original ones) now pass * Minor edits to TestRequestScope * Full build passes now * removed entity dictionary from entity projection * Pre-inspection cleanup * minor inspection fixup * rebase * Rebased on AggregationDataStore * clean up extra new lines * address comments * Builder pattern * update comments * remove projection in entity * fix jackson * Hydrate Relationship (#987) (#15) * Address some codecy comments * Add comment for partial query * Reenable tests * Address comments, refactor alias * Add test for alias * swapped test case * fix get type Added AggregationDataStore Code (#991) * Adding testing for aggregation data store * Debugging integration tests * Continuing testing work * AggregationDataStore * AggregationDataStore testing * Added more tests * Aggregation Data Store * Cleaned up testing code * Cleaned up code, fixed helper for AggregationDataStore * end * Fixed checkstyle, other minor fixes * fixed comment * Minor fixes * Fixed id type issue, added exception for queries with no metrics Fixed build (#993) Making TimeDimension an interface (#992) * [maven-release-plugin] prepare release 5.0.0-pr1 * [maven-release-plugin] prepare for next development iteration * Renamed graphQL file to match test (#1002) * [maven-release-plugin] prepare release 5.0.0-pr2 * [maven-release-plugin] prepare for next development iteration * Add JoinTo annotation (#1006) * Added JoinTo Annotation * Added working test * Added TODO comment for next PR * Added TODO comment for next PR * Added Sorting and Filtering support for JoinTo Columns * Fixed IT tests for Aggregation Data Store * Moved entityManager creation to happen separately for each query (#1008) * Moved entityManager creation to happen separately for each query * Closing EntityManager after each query * Inspection rework * Column annotation (#1017) * Solved column issue and added QueryEngineFactory * Caching query engine in AggregationDataStore * Fixed column description * Update SQLQueryEngine.java (#1019) * Add SQLMetrix, rearrange packages (#1020) * Add SQLMetrix, rearrange packages * address comment * Manager transacton manually * Add readonly * Manager transacton manually (#1021) * Manager transacton manually * Add readonly * Hydrate GraphQL Schema with parameterized attributes (#1018) * GraphQL schema expose expected argument name and its type for each attribute * Change empty arguments to unmodifable set * AggregationStore: Add multiple time grain definitions to schema (#1022) * Fixed checkstyle warnings and errors. Separated the Query dimension interface from the Schema dimension interface * Added skeleton code to convert entity projection arguments into time grains * Cleanup * Class renames per inspection comments * Inspection comments * some rework * use getTimeDimension() * change exception * Refactor time dimension logic (#1028) * Manager transacton manually * Add readonly * some rework * use getTimeDimension() * change exception * ISSUE-1026 Add support for @Subselect (#1038) * Manager transacton manually * Add readonly * some rework * use getTimeDimension() * change exception * ISSUE-1026 Add support for @Subselect * Address comments * ISSUE-1027 Support join for having clause (#1039) * Manager transacton manually * Add readonly * some rework * use getTimeDimension() * change exception * ISSUE-1027 Support join for having clause * View Design * Add tests and cleanup * rename annotation * function name fixed to enableISO8601Dates (#1052) * fix bugs * Support for multiple queries at root is added (#1044) * Support for multiple queries at root is added * Added test with alias * comments resolved * merge annotations * don't group by view relationship * Add time grain to GraphQL schema (#1042) * Added basic plumbing to push attributes from the entity projection down to the QueryEngine * Added logic to expand SQL time expression in SQLQueryEngine * Added SQLQueryEngine tests * Added IT tests * The AggregationStore now adds graphql parameters for parameterized columns * Minor refactor * Inspection rework * Minor fix * Support multiple query of same entity with different alias (#1055) * Support multiple query of same entity with different alias * add static method to generate keyname for GraphQLProjectionInfo projections * Remove aliasPartialQuerySameAttribute * MetadataStore Models (#1068) * Manager transacton manually * Add readonly * some rework * use getTimeDimension() * change exception * Metadatastore models * Address comments * address comments * move root * fix style check * SQLQueryTemplate Model (#1073) * SQLQueryTemplate * SQLTables * refactor * update sql dimension projection * update sql dimension projection * clean up dimension projection * refactor sql components * aggregatable field rework * add comments * rearrange packages * Add dimension projection back * fix checkstyle * Add dictionary * Simplify MetricFunction and SQLQueryTemplate * Address comments * Integrate Metadata Model and SQLQueryTemplate Model (#1083) * Integrate Metadata Model and SQLQueryTemplate Model * remove AggregationDictionary and AggregationManager * Add timezone * Can only query analyticView * integrate view with aggregation and metadata * remove includeField * remove @view * Use NonEntityDictionary * remove id * revert access changes * fix JPA entity check * remove @Entity from analyticViews * use table name as relationship type id * revert NonEntitydictinoary * tiny rework * Integration tests * Add jsonapi ittest * aggregation data store doesn't manage jpa entities * address comments fix integration dependencies (#1093) [maven-release-plugin] prepare release 5.0.0-pr3 [maven-release-plugin] prepare for next development iteration Fixed elide standalone pom from rebase Fixed minor bug in rebase Fixed rebase Improving class scanning performance for MetadataStore (#1117) Enable elide5 travis builds (#1129) * Move repeated @Sql annotations to class level (#1119) * Turning on travis builds with code coverage for Elide 5.x * Fixing security issue in spring-boot-web Co-authored-by: Brutus5000 <[email protected]> Fix sorting and ambiguous join issue (#1127) * Added sorting on aggregated metric based on latest elide-5.x * Fix ambiguity problem * update comments * fix codacy * refactor generateColumnReference * update comment * address comments * test cleanup * update unittest * fix elide core alias * QueryValidatorTest * EntityProjectionTranslatorTest * go joinFragment approach * delete jointrienode Support no metric query (#1137) [maven-release-plugin] prepare release 5.0.0-pr4 [maven-release-plugin] prepare for next development iteration Check dependency injection (#1138) * Move repeated @Sql annotations to class level (#1119) * Fixing OWASP security warning for Tomcat dependency in Spring Web (#1132) * Adding support for dependency injection of Checks. Added test injection classes * Unit tests pass * Tests pass * Removed Initializer Concept Co-authored-by: Brutus5000 <[email protected]> Fix travis log length (#1140) * Move repeated @Sql annotations to class level (#1119) * Fixing OWASP security warning for Tomcat dependency in Spring Web (#1132) * Removed unnecessary request/response logging (to shorten travis logs) * Address inspection comments * Address inspection comments * Address inspection comments * Removed logging of graphQL model building to shorten length * Fixed compilation error Co-authored-by: Brutus5000 <[email protected]> [maven-release-plugin] prepare release 5.0.0-pr5 [maven-release-plugin] prepare for next development iteration Refactoring Elide Security Checks (#1144) Removed UpdateOnCreate. Refactored AuditLogger, Pagination, & Sorting (#1146) * Removed UpdateOnCreate. Refactored AuditLogger * Refactored Sorting * Refactored Pagination * Refactored Pagination * Pagination refactor builds and tests pass * Codacy fixes * Inspection rework * Fixes build * More inspection rework * Fix build Refactor share permission (#1154) * Refactored Sorting * Refactored Pagination * Refactored Pagination * Pagination refactor builds and tests pass * Codacy fixes * Refactored SharePermission to NonTransferable * Fixed build * Fixed startup bug * Fixed codacy and inspection comments * Update elide-core/src/main/java/com/yahoo/elide/core/EntityDictionary.java Co-Authored-By: Jon Kilroy <[email protected]> * Inspection rework Co-authored-by: Jon Kilroy <[email protected]> metadata refactor (#1179) * metadata refactor * merge table and analyticView * fix reflection package * Make Table constrcut its own columns * table json type alias * add comment @Join and JoinPath add comment hide non-jpd entities in grpahql hide joins refactor hidden remove ant remove relationshp, update model (#1186) Rebased on master [maven-release-plugin] prepare release 5.0.0-pr6 [maven-release-plugin] prepare for next development iteration sourceColumn (#1196) * sourceColumn * address comment * change to sourcePath Cleaning up ElideSettings Added more test fixes Fixed a number of broken tests Build completes Fixed JSON-API Patch Response Fixed GraphQL errors. Added better errors for Forbidden Access Exception. Minor fix Added setId to EntityDictionary Initial version All test pass except for spring Build passes Adding checkstyle comments for classes Inspection comments Fixed checkstyle issues Change the way types are named in the GraphQL schema (#1215) * name utils * ready for review * fix: AggregationDataStoreIntegrationTest#testGraphQLSchema Label Resolver for Dimension Formula (#1208) * sql expression to dimension formula * Metric formula * Add unit tests * refacot formula references * fix comment * fix comment * resolve physical column * refactor dimensionFormula * cleanup * label resolver * cleanup * refactor * cleanup * move code * labelStore * labelStore 2 * remove generator * refactor metric formula * address comments * move symbol table into sql query engine * remove sourceColumn * update reference expression * visitor design * add comments * add unit join path test * fix timeDimensionProjection * fix null value number * address comments * fix codacy Co-authored-by: hchen04 <[email protected]> Fixing broken javadoc ColumnProjection Refactor (#1239) * unify projections * remove getFunction() * add table into query template Co-authored-by: hchen04 <[email protected]> Refactored metric SQL expansion to occur dynamically at query time ra… (#1270) * Refactored metric SQL expansion to occur dynamically at query time rather than statically during service initialization * Update elide-datastore/elide-datastore-aggregation/src/main/java/com/yahoo/elide/datastores/aggregation/queryengines/sql/query/SQLMetricProjection.java Co-Authored-By: Jon Kilroy <[email protected]> Co-authored-by: Aaron Klish <[email protected]> Co-authored-by: Jon Kilroy <[email protected]> Remove path logic from aggregation store (#1271) * Extended query validation to ensure where clause and sorting clauses don't traverse relationships * Added error check for relationship traversal for Having clauses * Hacked up logic to remove reference Table resolve references that take a path * Minor refactoring * Removed logic to extend join path * Refactored column projections to use generics * Removed reference functions from MetricProjection base class * Refactored so that all SQL generation is done inside the ColumnProjection * Refactored so all projection happens through projections * Refactored column projection creation * Removed unnecessary code * Added templateQuery to arguments required to generate SQL in column projections * Fixed codacy issues Co-authored-by: Aaron Klish <[email protected]> Added InMemoryStore to list of datastores that run IT tests (#1225) Co-authored-by: Aaron Klish <[email protected]> Adding TimeDimensions to Table (#1273) Elide 5.x async (#1203) * Adding Async Entity Models Co-authored-by: moizarafat <[email protected]> * Adding async module and fixing parent pom version Co-authored-by: moizarafat <[email protected]> * Adding async service classes, security and cleanup services Co-authored-by: moizarafat <[email protected]> * Adding Copyright License Header to Async module classes Co-authored-by: moizarafat <[email protected]> * Using new request scope for datastore transactions Co-authored-by: moizarafat <[email protected]> * Remove thread sleep used in testing Co-authored-by: moizarafat <[email protected]> * Fixing based on code quality review from codacy Co-authored-by: moizarafat <[email protected]> * Fixing additional issues based on code quality review from codacy Co-authored-by: moizarafat <[email protected]> * Adding getter for AsyncQueryResult Co-authored-by: moizarafat <[email protected]> * Reformatting. Co-authored-by: Abhino <[email protected]> * Adding mappedBy. Co-authored-by: Abhino <[email protected]> * Fixing several review comments - pom, lombok, thread exec Co-authored-by: moizarafat <[email protected]> * Fixing codacity errors Co-authored-by: moizarafat <[email protected]> * Moving all the DB ORM logic to a utility class for Async Co-authored-by: moizarafat <[email protected]> * Remove dependency on number of hosts, delete permission, separate method for interrupttime calculation Co-authored-by: Abhino <[email protected]> * Variable Naming convention. Co-authored-by: Abhino <[email protected]> * Variable Naming convention. Co-authored-by: Abhino <[email protected]> * Removing unwanted params. Co-authored-by: Abhino <[email protected]> * Making cleaner a separate service. Co-authored-by: Abhino <[email protected]> * Remove unwanted params. Co-authored-by: Abhino <[email protected]> * Remove unwanted tabs. Co-authored-by: Abhino <[email protected]> * Updating order of modules. Co-authored-by: Abhino <[email protected]> * Adding cleanup sql. Co-authored-by: Abhino <[email protected]> * Adding delete logic. Co-authored-by: Abhino <[email protected]> * Adding query result retention. Co-authored-by: Abhino <[email protected]> * Resolving build error. Co-authored-by: Abhino <[email protected]> * Changing AsyncDbUtil to use Functional Interface Co-authored-by: moizarafat <[email protected]> * Removing sleep Co-authored-by: moizarafat <[email protected]> * Adding functional interface logic for executeInTransaction Co-authored-by: moizarafat <[email protected]> * Adding debug statements for AsyncDbUtil Co-authored-by: moizarafat <[email protected]> * Codacity errors, exception handling Co-authored-by: moizarafat <[email protected]> * Adding Async Entity Models Co-authored-by: moizarafat <[email protected]> * Adding async module and fixing parent pom version Co-authored-by: moizarafat <[email protected]> * Adding async service classes, security and cleanup services Co-authored-by: moizarafat <[email protected]> * Adding Copyright License Header to Async module classes Co-authored-by: moizarafat <[email protected]> * Using new request scope for datastore transactions Co-authored-by: moizarafat <[email protected]> * Remove thread sleep used in testing Co-authored-by: moizarafat <[email protected]> * Fixing based on code quality review from codacy Co-authored-by: moizarafat <[email protected]> * Fixing additional issues based on code quality review from codacy Co-authored-by: moizarafat <[email protected]> * Adding getter for AsyncQueryResult Co-authored-by: moizarafat <[email protected]> * Reformatting. Co-authored-by: Abhino <[email protected]> * Adding mappedBy. Co-authored-by: Abhino <[email protected]> * Fixing several review comments - pom, lombok, thread exec Co-authored-by: moizarafat <[email protected]> * Fixing codacity errors Co-authored-by: moizarafat <[email protected]> * Moving all the DB ORM logic to a utility class for Async Co-authored-by: moizarafat <[email protected]> * Remove dependency on number of hosts, delete permission, separate method for interrupttime calculation Co-authored-by: Abhino <[email protected]> * Variable Naming convention. Co-authored-by: Abhino <[email protected]> * Variable Naming convention. Co-authored-by: Abhino <[email protected]> * Removing unwanted params. Co-authored-by: Abhino <[email protected]> * Making cleaner a separate service. Co-authored-by: Abhino <[email protected]> * Remove unwanted params. Co-authored-by: Abhino <[email protected]> * Remove unwanted tabs. Co-authored-by: Abhino <[email protected]> * Updating order of modules. Co-authored-by: Abhino <[email protected]> * Adding cleanup sql. Co-authored-by: Abhino <[email protected]> * Adding delete logic. Co-authored-by: Abhino <[email protected]> * Adding query result retention. Co-authored-by: Abhino <[email protected]> * Resolving build error. Co-authored-by: Abhino <[email protected]> * Changing AsyncDbUtil to use Functional Interface Co-authored-by: moizarafat <[email protected]> * Removing sleep Co-authored-by: moizarafat <[email protected]> * Adding functional interface logic for executeInTransaction Co-authored-by: moizarafat <[email protected]> * Adding debug statements for AsyncDbUtil Co-authored-by: moizarafat <[email protected]> * Codacity errors, exception handling Co-authored-by: moizarafat <[email protected]> * Rebased with elide-5.x and changed User principal logic Co-authored-by: moizarafat <[email protected]> * Rebase Co-authored-by: Abhino <[email protected]> * Changing CleanerThread to use ExecuteInTransaction Co-authored-by: moizarafat <[email protected]> * Removing unused imports Co-authored-by: moizarafat <[email protected]> * Delete Method changes Co-authored-by: Abhino <[email protected]> * Overriding hashCode and equals Co-authored-by: Abhino <[email protected]> * Overriding hashCode and equals Co-authored-by: Abhino <[email protected]> * Removing unused method Co-authored-by: Abhino <[email protected]> * Changing response logic for AsyncQuery Co-authored-by: moizarafat <[email protected]> * Adding return to delete method * Resolving Review Comments Co-authored-by: Abhino <[email protected]> * Resolve some review comments Co-authored-by: moizarafat <[email protected]> * Changing the DAO contract and updating references to DAO Co-authored-by: moizarafat <[email protected]> * Removing singletons and extra constructors, simplyfing logic Co-authored-by: moizarafat <[email protected]> * Adding default constructor and setters for DefaultDAO Co-authored-by: moizarafat <[email protected]> * Fixing codacy error Co-authored-by: moizarafat <[email protected]> * Changing Base to use correct obj Co-authored-by: moizarafat <[email protected]> * Fixing review comments Co-authored-by: moizarafat <[email protected]> * Resolving review comments Co-authored-by: moizarafat <[email protected]> * Review comments Co-authored-by: Abhino <[email protected]> * Elide 5.x elide-async integration in elide-standalone (#1205) * Integrating elide-async. Co-authored-by: Abhino <[email protected]> * Default enableAsync to False. Co-authored-by: Abhino <[email protected]> * Default enableAsync to False. Co-authored-by: Abhino <[email protected]> * Integrating elide-async. Co-authored-by: Abhino <[email protected]> * Default enableAsync to False. Co-authored-by: Abhino <[email protected]> * Default enableAsync to False. Co-authored-by: Abhino <[email protected]> * Resolving review comments. Co-authored-by: Abhino <[email protected]> * Resolving review comments. Co-authored-by: Abhino <[email protected]> * Removing unwanted params. Co-authored-by: Abhino <[email protected]> * Binding Cleaner service. Co-authored-by: Abhino <[email protected]> Binding Cleaner service. Co-authored-by: Abhino <[email protected]> * Adding query result retention. Co-authored-by: Abhino <[email protected]> * Integrating elide-async. Co-authored-by: Abhino <[email protected]> * Default enableAsync to False. Co-authored-by: Abhino <[email protected]> * Default enableAsync to False. Co-authored-by: Abhino <[email protected]> * Integrating elide-async. Co-authored-by: Abhino <[email protected]> * Resolving review comments. Co-authored-by: Abhino <[email protected]> * Resolving review comments. Co-authored-by: Abhino <[email protected]> * Removing unwanted params. Co-authored-by: Abhino <[email protected]> * Binding Cleaner service. Co-authored-by: Abhino <[email protected]> Binding Cleaner service. Co-authored-by: Abhino <[email protected]> * Adding query result retention. Co-authored-by: Abhino <[email protected]> * Making AsyncQueryDAO configurable Co-authored-by: Abhino <[email protected]> * Making AsyncQueryDAO configurable Co-authored-by: Abhino <[email protected]> * Review comments Co-authored-by: Abhino <[email protected]> * Review comments Co-authored-by: abhino <[email protected]> * Changing call to DefaultAsyncQueryDAO * Review comments Co-authored-by: Abhino <[email protected]> Co-authored-by: Abhino <[email protected]> * Elide 5.x elide-async integration in elide-spring (#1204) * Integrating elide-async. Co-authored-by: Abhino <[email protected]> Integrating elide-async. Co-authored-by: Abhino <[email protected]> * Removing unwanted params. Co-authored-by: Abhino <[email protected]> * Binding Cleaner service. Co-authored-by: Abhino <[email protected]> * Adding query result retention. Co-authored-by: Abhino <[email protected]> * Adding DAO configuration Co-authored-by: Abhino <[email protected]> * Checkstyle error Co-authored-by: Abhino <[email protected]> * Review comments Co-authored-by: Abhino <[email protected]> * Review comments Co-authored-by: Abhino <[email protected]> * Review comments Co-authored-by: Abhino <[email protected]> * Codacy Error Co-authored-by: Abhino <[email protected]> * Changing include type to asyncQuery to avoid conflict Co-authored-by: moizarafat <[email protected]> * Sync Co-authored-by: Abhino <[email protected]> * Singleton for Cleaner Service Co-authored-by: Abhino <[email protected]> * Singleton for Cleaner Service Co-authored-by: Abhino <[email protected]> * Singleton for Cleaner Service Co-authored-by: Abhino <[email protected]> * Singleton for Executor Service Co-authored-by: Abhino <[email protected]> * Sync with Standalone Value * Changes to change UUID columns type as varchar(36) * Review Comments * Review Comments * Fix review comments Co-authored-by: Abhino <[email protected]> * Updating per review comments * Remove unused import * Removing status change to Queued * removing unused import * prepersist for status Co-authored-by: moizarafat <[email protected]> Co-authored-by: avijay <[email protected]> Co-authored-by: moiz arafat <[email protected]> Co-authored-by: Moiz Arafat <[email protected]> Co-authored-by: moizarafat <[email protected]> Simplify life cycle hooks (#1224) * Initial version * Finished code. Starting to flesh out tests * Elide core compiles * Removed CRUDEvent.CrudAction * Added new test structure for LifeCycleTest * Fixed bug in Persistent Resource where relationship reads did not tigger a lifecycle event * Added several LifeCycle Tests * Added create test for persistent resource * Added Elide Persistent Resource Update Test * Added update with change spec test * Added Relationship Edit test * Added relationship test * Fixed checkstyle and compilation errors * Added remove from collection test * Added exception tests * Added delete test * Added read test * Fixed checkstyles * Elide core builds and tests pass * Fixed graphQL tests * Full build now passes * Removed old life cycle annotations * Minor cleanup * Fixed codacy issues * Update README.md Co-Authored-By: Jon Kilroy <[email protected]> Co-authored-by: Aaron Klish <[email protected]> Co-authored-by: Jon Kilroy <[email protected]> Fixed elide-5.x build Fixed rebase issues Turning off retireJS for dependency check Better errors for missing IDs in Patch Extension Request. (#1278) (#1281) * Better errors for missing IDs in Patch Extension Request. (#1278) * Return a better error when handling invalid patch extension requests that are missing IDs * Added tests Co-authored-by: Aaron Klish <[email protected]> * Expect encoded response as array Co-authored-by: Aaron Klish <[email protected]> Co-authored-by: Aaron Klish <[email protected]> Co-authored-by: wcekan <[email protected]> [maven-release-plugin] prepare release 5.0.0-pr8 [maven-release-plugin] prepare for next development iteration Patch Extension Lifecycle tests (#1280) * Patch Extension Lifecycle tests * Missing check * Update error body Co-authored-by: wcekan <[email protected]> Add version support (#1295) * Initial non working version * More changes * Build passes * Expanded version param outside entity dictionary * Plumbed API version up through RequestScope * Plumbed API version up to controllers and endpoints * All code minus async written and working * Code complete * Build passes * All tests are passing * Changed aggregation store Table id to include version. Metadata will now surface table versions * Added a swagger IT test with API versions * Replace empty version string with constant. Fixed a few bugs in JSON-API parser * Added graphql type introspection test * Added graphql type introspection test * Added spring controller test. Fixed issue with version header parsing * Added spring controller test for graphql * Added swagger controller tests * Added more happy path versioned tests * Fixed bug in GraphQLIT * Added graphQL test for invalid API version * ADded invalid API version graphql controller test * Fixed checkstyles * Added standalone and patch extension tests * Fixed build errors and codacy errors Co-authored-by: Aaron Klish <[email protected]> Migrated spring controllers to use async (#1296) Co-authored-by: Aaron Klish <[email protected]> fixes for Async Models lifecycle hooks failing (#1294) * elide standalone fixes for lifecycle hooks failing * Manual Binding of lifecycle hooks * Manual binding of lifecycle in elide spring * Refactor * Review Comments * elide standalone fixes for lifecycle hooks failing * Manual Binding of lifecycle hooks * Manual binding of lifecycle in elide spring * Refactor * Review Comments * Rebase with 5.x Co-authored-by: moiz arafat <[email protected]> Implement equals/hashCode and immutability where needed (#1297) * ColumnProjection must implement equals/hashCode * Use @Value for SQLTimeDimensionProjection * Use @Value for Argument * Use @Value for TimeDimensionGrain * Use @Value for TimeDimension * Make Column/Metric immutable * Table can be immutable * Use @Value to simplify Query Changes elide to only inject models annotated with @Inject (#1299) * Changes elide to only inject models annotated with @Inject * Codacy fix Co-authored-by: Aaron Klish <[email protected]> Enhancemnents to Aggregation Store (#1300) Co-authored-by: moiz arafat <[email protected]> QueryEngine result cache (#1279) * Add QueryEngine cache API * Remove outdated javadoc text Elide Async Feature Unit and Integration Tests (#1311) * Added integration,unit test setup for Async Module with few tests Co-authored-by: moizarafat <[email protected]> * Adding Licence headers, Sample entity and working Async test for POST,GET Co-authored-by: moizarafat <[email protected]> * Adding additional integration tests Co-authored-by: moizarafat <[email protected]> * Adding remaining integration tests and updating javadocs Co-authored-by: moizarafat <[email protected]> * Adding all unit tests Co-authored-by: moizarafat <[email protected]> * Updating tests to work with Singleton changes Co-authored-by: moizarafat <[email protected]> * Modifying tests to work with singleton logic Co-authored-by: moizarafat <[email protected]> * Resolving some codacy errors Co-authored-by: moizarafat <[email protected]> * Adding DSL for GraphQL and addressing review comments Co-authored-by: moizarafat <[email protected]> * Fixing unit tests and checkstyle errors Co-authored-by: moizarafat <[email protected]> * Moving tests to new format Co-authored-by: moizarafat <[email protected]> * Fixing session not closed error and checkstyle errors * Fixing imports Co-authored-by: moizarafat <[email protected]> * Removing unused method and updating harness Co-authored-by: moizarafat <[email protected]> * Updating unit tests Co-authored-by: moizarafat <[email protected]> * Updating ResourceConfig for AsyncTest Co-authored-by: moizarafat <[email protected]> * Moving logic for resource config to new test binder Co-authored-by: moizarafat <[email protected]> * Adding bindFactory logic for async services Co-authored-by: moizarafat <[email protected]> * Fixing IT * Fixing IT Co-authored-by: Abhino <[email protected]> * Adding Remaining integration tests Co-authored-by: moizarafat <[email protected]> * Consolidating filter logic for integration tests Co-authored-by: moizarafat <[email protected]> * Fixing legacy Hibernate entity manager store (so it doesn't recycle the entity manager * Adding additional integration tests for Standalone and Spring boot Co-authored-by: moizarafat <[email protected]> * Fixing codacy errors Co-authored-by: moizarafat <[email protected]> * Review Comments Co-authored-by: abhino <[email protected]> Co-authored-by: avijay <[email protected]> Co-authored-by: moizarafat <[email protected]> Co-authored-by: moiz arafat <[email protected]> Co-authored-by: Aaron Klish <[email protected]> in memory compilation integrated with dynamic config helpers (#1255) * Rebase against 5.x Co-authored-by: Ayeswarya <[email protected]> Co-authored-by: AvaniMakwana <[email protected]> * pom fix Co-authored-by: Ayeswarya <[email protected]> Co-authored-by: AvaniMakwana <[email protected]> * pom fix Co-authored-by: Ayeswarya <[email protected]> Co-authored-by: AvaniMakwana <[email protected]> * Merging 1220 Co-authored-by: rishi-aga <[email protected]> Co-authored-by: AvaniMakwana <[email protected]> * Spring Boot Changes Co-authored-by: Ayeswarya <[email protected]> * Spring Boot Changes Co-authored-by: Ayeswarya <[email protected]> * Spring Boot Changes Co-authored-by: Ayeswarya <[email protected]> * Review Comments Co-authored-by: Ayeswarya <[email protected]> * Update DynamicConfigTest.java * Review Comments Co-authored-by: Ayeswarya <[email protected]> Co-authored-by: moiz arafat <[email protected]> Co-authored-by: AvaniMakwana <[email protected]> Co-authored-by: rishi-aga <[email protected]> * Fixed issues with rebase * Finished rebase * Async ID change from UUID to String and Dynamic Config FIx (#1325) * Async UUID to String, Dynamic Config NPE * Update DefaultAsyncQueryDAOTest.java * TANDS-19093-transactionRegistry-interface (#1332) * TANDS-19093-transactionRegistry-interface * TANDS-19093-transactionRegistry-interface * TANDS-19093-transactionRegistry-interface * TANDS-19093-fixing-bugs * TANDS-19093-fixing-bugs * TANDS-19093-fixing-bugs * TANDS-19093-fixing-bugs Co-authored-by: Ramsha Rao <[email protected]> * add requestId (#1331) Co-authored-by: aaggarwal <[email protected]> * [maven-release-plugin] prepare release 5.0.0-pr9 * [maven-release-plugin] prepare for next development iteration * Introduce QueryResult class for QueryEngine caching (#1333) * Make QueryEngine cache bypass flag part of Query * Forming the cache key is not free, so don't a use stub cache * Add QueryResult class * Add pageTotals to QueryResult * Pass page totals through QueryResult instead of Pagination * Codacy doesn't know @Value makes fields private * remove missing javadoc warnings (#1337) Co-authored-by: Chandrasekar Rajasekar <[email protected]> * Elide 5.x dynamic config standalone (#1259) * Fix method call, Swagger Doc Update Co-authored-by: AvaniMakwana <[email protected]> * Swagger doc update Co-authored-by: AvaniMakwana <[email protected]> * update pom.xml Co-authored-by: AvaniMakwana <[email protected]> * update pom.xml Co-authored-by: AvaniMakwana <[email protected]> * review comments Co-authored-by: AvaniMakwana <[email protected]> * Update ElideStandaloneSettings.java * Update pom.xml * Update pom.xml Co-authored-by: moiz arafat <[email protected]> * addTransaction-removeTransaction (#1338) * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169 * TANDS-19169-changing-CVSS-score * addTransaction-removeTransaction * addTransaction-removeTransaction * addTransaction-removeTransaction * addTransaction-removeTransaction * addTransaction-removeTransaction * addressing-comments * addressing-comments * addressing-comments * addressing-comments * addressing-comments * addressing-comments * addressing-comments * addressing-comments * addressing-comments * making DataStore an abstract class * making DataStore an abstract class * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * adding-transaction-id to transaction implementations * adding-transaction-id to transaction implementations * adding-transaction-id to transaction implementations * adding-transaction-id to transaction implementations * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * fixing bugs * addressing comments * addressing comments * addressing comments * fixing bugs Co-authored-by: Ramsha Rao <[email protected]> * Fixed rebase * Removed old elide example modules * [maven-release-plugin] prepare release 5.0.0-pr10 * [maven-release-plugin] prepare for next development iteration * Validation for Model Configs (#1306) * Dynamic Config Validator Co-authored-by: rishi-aga <[email protected]> * Review Comments * Review Comments * Use DynamicConfigValidator instead of ElideConfigParser Co-authored-by: moiz arafat <[email protected]> * add explicit join (#1364) Co-authored-by: Chandrasekar Rajasekar <[email protected]> * Fixes - Honor Doc Version in SwaggerController and Use ISO8601Dates Serde Format in Date Formatting (#1336) * misc_fixes * review comments * Review Comments * DyFixes * DyFixes * Elide dynamic config model verification (#1354) * sign & Verify model * review comments * typo fix * indent fix * review comment * additional testcases * revert system exit * fix codacy * add return * add system exit Co-aut…