Orquesta vars can no longer contain '.' in 3.2

[nmaludy]

SUMMARY

In StackStorm 3.2, if an Orquesta workflow contains a . in any of its vars (ie context) then the workflow will fail to run with an error:

$ st2 run default.orquesta_vars_with_periods.
id: 5eac3faeca07c849e9b4f280
action.ref: default.orquesta_vars_with_periods
parameters: None
status: failed
start_timestamp: Fri, 01 May 2020 15:26:38 UTC
end_timestamp: Fri, 01 May 2020 15:26:38 UTC
result: 
  errors:
  - message: key 'with.period' must not contain '.'
  output: null

STACKSTORM VERSION

$ st2 --version
st2 3.2.0, on Python 2.7.5

OS, environment, install method

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.8 (Maipo)

# install method = puppet-st2

Steps to reproduce the problem

/opt/stackstorm/packs/default/actions/orquesta_vars_with_periods.yaml

---
description: "Demos bug with orquesta vars with periods"
enabled: true
runner_type: orquesta
entry_point: workflows/orquesta_vars_with_periods.yaml
name: orquesta_vars_with_periods
pack: default
parameters:
  demo:
    type: string
    default: "demo"

/opt/stackstorm/packs/default/actions/workflows/orquesta_vars_with_periods.yaml

---
version: 1.0

description: "Demos bug with orquesta vars with periods"

input:
  - demo

vars:
  - myvar.with.period: 'this line is going to cause an error'

tasks:
  dispatch:
    action: core.noop

Register and run the action

$ st2 action create /opt/stackstorm/packs/default/actions/orquesta_vars_with_periods.yaml 
$ st2 run default.orquesta_vars_with_periods
.
id: 5eac47b9ca07c849e9b4f289
action.ref: default.orquesta_vars_with_periods
parameters: None
status: failed
start_timestamp: Fri, 01 May 2020 16:00:57 UTC
end_timestamp: Fri, 01 May 2020 16:00:58 UTC
result: 
  errors:
  - message: key 'myvar.with.period' must not contain '.'
  output: null

Expected Results

Workflow runs without error

Actual Results

Workflow errors trying to write state to the database

[nmaludy]

Investigation

Rolled back to v3.1 and this same workflow executes successfully.

Looks like this problem was introduced in this PR: #4854

It appears there were some other problems with '.' in different places for workflows and subsequent rollback PRs were introduced to fix those bugs:

• Rollback field changes on workflow and task execution collection #4875
• Rollback workflow DB change on errors field #4892

Diffing from 3.1 to 3.2 i can see the following fields were changed:

$ git diff v3.1.0 v3.2.0 -- st2common/st2common/models/db/workflow.py
diff --git a/st2common/st2common/models/db/workflow.py b/st2common/st2common/models/db/workflow.py
index b93b40686..b642df8eb 100644
--- a/st2common/st2common/models/db/workflow.py
+++ b/st2common/st2common/models/db/workflow.py
@@ -36,11 +36,11 @@ class WorkflowExecutionDB(stormbase.StormFoundationDB, stormbase.ChangeRevisionF
     RESOURCE_TYPE = types.ResourceType.EXECUTION     action_execution = me.StringField(required=True)
-    spec = stormbase.EscapedDictField()
-    graph = stormbase.EscapedDictField()
+    spec = me.DictField()
+    graph = me.DictField()
     input = stormbase.EscapedDictField()
-    notify = stormbase.EscapedDictField()
-    context = stormbase.EscapedDictField()
+    notify = me.DictField()
+    context = me.DictField()
     state = stormbase.EscapedDictField()
     status = me.StringField(required=True)
     output = stormbase.EscapedDictField()
@@ -62,7 +62,7 @@ class TaskExecutionDB(stormbase.StormFoundationDB, stormbase.ChangeRevisionField
     task_name = me.StringField(required=True)
     task_id = me.StringField(required=True)
     task_route = me.IntField(required=True, min_value=0)
-    task_spec = stormbase.EscapedDictField()
+    task_spec = me.DictField()
     delay = me.IntField(min_value=0)
     itemized = me.BooleanField(default=False)
     items_count = me.IntField(min_value=0)

Not yet sure which of these fields need to be reverted back to stormbase.EscapedDictField() in order to fully fix the issue. Working on this now.

[cognifloyd]

Do you use an IP address or domain name as a var name? Why do you have a period in a var name?

[nmaludy]

@cognifloyd in this case we have keys that represent datastore keys that we pass to bolt so it can do an st2kv lookup from within the Bolt process in order to securely transfer secrets from st2 to bolt.

example:

vars:
  - attempt_count: 0
  - st2kv_config_linux:
      ssh.password: system.linux.password
      ssh.user: system.linux.username
      ssh.tmpdir: system.linux.tmpdir
  - st2kv_config_windows:
      winrm.password: system.ad.password
      winrm.user: system.ad.username
      winrm.port: system.winrm.port

Although, yes, it would be the same problem for a key being a domain name or IP address.

[cognifloyd]

If we drop mongo 3.4 and use mongoengine >=0.19.1, then we don't have to use the EscapedDictField() to have periods in keys. That won't work for now (ie dropping mongo 3.4 in a bugfix release is a no-no), but that means we can have hope of re-adding these performance fixes in a future release.
See: MongoEngine/mongoengine#1594 and MongoEngine/mongoengine#2193 (merged in MongoEngine/mongoengine@b205314 )

[arm4b]

There are plans to drop MongoDB 3.4 in favor of 3.6 in next release:
See

• Move MongoDB 3.4 -> 4.0 for EL7/U16 due to EOL st2-packages#643
• https://github.com/StackStorm/st2docs/pull/968/files