MongoDB collection/index name used by ST2 is too long for AWS DocumentDB

[GuiTeK]

SUMMARY

While migrating from ST2 v2.9 to ST2 v3.0, we encountered the following problem: one MongoDB index name used by ST2, namely st2.action_execution_scheduling_queue_item_d_b.scheduled_start_timestamp (C.F. st2/st2common/st2common/models/db/execution_queue.py) is too long for AWS DocumentDB. It does work with an official MongoDB server.

AWS DocumentDB should be compatible/honor the official limits of MongoDB (here, 127 characters for fully qualified index names), but I contacted the AWS DocumentDB support which confirmed me that it was a known issue in DocumentDB and that they didn't have an estimated time of resolution for this problem. They suggested me to shorten the index name if I can, but I think it's currently not customizable (which is okay, this should not be needed).

I realize that this is not a StackStorm bug per se, however AWS DocumentDB is a big player and it might be interesting for you to shorten this index name so ST2 users/customers can use DocumentDB with ST2 V3+.

ISSUE TYPE

• Bug Report

STACKSTORM VERSION

st2 3.0.0, on Python 2.7.12

OS / ENVIRONMENT / INSTALL METHOD

N/A

STEPS TO REPRODUCE

• Install StackStorm V3.0+
• Configure an AWS DocumentDB as the MongoDB server
• Run a command that will try to create the DB, e.g. st2ctl reload --register-all

EXPECTED RESULTS

st2ctl reload --register-all should successfully create the DB and work.

ACTUAL RESULTS

st2ctl reload --register-all exits with an error because it can't create the index name st2.action_execution_scheduling_queue_item_d_b.scheduled_start_timestamp.

[Kami]

Thanks for reporting this.

As you have said - that's not a bug in StackStorm per say, because we officially only support MongoDB 3.6 and 4.0 on Ubuntu Bionic.

Having said that, if a change is relatively easy and ensures it also works with AWS DocumentDB, then I'm fine with it.

There is a bigger question here though - in the future, we plan to use and support MongoDB 4.0 (that's already the case for Ubuntu Bionic packages). As far as I know, AWS only claims API compatibility with 3.6 because of the licensing change on the MongoDB side.

Right now, we don't actually utilize any specific functionality from 4.0 yet, but we might in the future.

[Kami]

@GuiTeK On a related note - do you have information on what's the current limit for index name on AWS DocumentDB?

It looks like it may be 70 characters, but it would be good to confirm that.

[GuiTeK]

Thank you for your answer @Kami!

The limit length for a fully qualified index name (db_name.collection_name.index_name) isn't clear to me and seems to be variable (it should not) depending on the length of each part (DB name, collection name, index name) of the fully qualified index name.

With the following index: st2.action_execution_scheduling_queue_item_d_b.scheduled_start_timestamp, the limit is 65 characters. However, if you try other db names/collection names/index names, you'll see that you might be able to go up to 67, 68 characters... Limiting fully qualified index names to 60 characters seems safe.

The AWS support told me that "the name of the index size is limited to 2048 bytes apart from the
6 - 127 characters limit". I don't really know what that means though.

[Kami]

I was able to reproduce the issue on actual DocumentDB cluster. It looks like anything up to 70 characters should work and should be a safe limit.

Here is a list of all the indexes we create and their name lengths:

st2.rule_d_b.enabled   20
st2.rule_d_b.action.ref   23
st2.rule_d_b.trigger   20
st2.rule_d_b.context.user   25
st2.rule_d_b.metadata_file   26
st2.rule_d_b.tags.name   22
st2.rule_d_b.tags.value   23
st2.rule_d_b.uid   16
st2.rule_d_b.name.pack   22
st2.sensor_type_d_b.name   24
st2.sensor_type_d_b.enabled   27
st2.sensor_type_d_b.trigger_types   33
st2.sensor_type_d_b.metadata_file   33
st2.sensor_type_d_b.uid   23
st2.sensor_type_d_b.name.pack   29
st2.trigger_type_d_b.metadata_file   34
st2.trigger_type_d_b.tags.name   30
st2.trigger_type_d_b.tags.value   31
st2.trigger_type_d_b.uid   24
st2.trigger_type_d_b.name.pack   30
st2.trigger_d_b.name   20
st2.trigger_d_b.type   20
st2.trigger_d_b.parameters   26
st2.trigger_d_b.uid   19
st2.trigger_d_b.name.pack   25
st2.trigger_instance_d_b.occurrence_time   40
st2.trigger_instance_d_b.trigger   32
st2.trigger_instance_d_b.occurrence_time.trigger   48
st2.trigger_instance_d_b.status   31
st2.action_d_b.name   19
st2.action_d_b.pack   19
st2.action_d_b.ref   18
st2.action_d_b.metadata_file   28
st2.action_d_b.tags.name   24
st2.action_d_b.tags.value   25
st2.action_d_b.uid   18
st2.action_d_b.name.pack   24
st2.action_execution_d_b.rule.ref   33
st2.action_execution_d_b.action.ref   35
st2.action_execution_d_b.liveaction.id   38
st2.action_execution_d_b.start_timestamp   40
st2.action_execution_d_b.end_timestamp   38
st2.action_execution_d_b.status   31
st2.action_execution_d_b.parent   31
st2.action_execution_d_b.rule.name   34
st2.action_execution_d_b.runner.name   36
st2.action_execution_d_b.trigger.name   37
st2.action_execution_d_b.trigger_type.name   42
st2.action_execution_d_b.trigger_instance.id   44
st2.action_execution_d_b.context.user   37
st2.action_execution_d_b.status.action.ref.start_timestamp   58
st2.action_execution_d_b.workflow_execution   43
st2.action_execution_d_b.task_execution   39
st2.action_execution_state_d_b.query_module   43
st2.action_execution_state_d_b.execution_id   43
st2.action_alias_d_b.name   25
st2.action_alias_d_b.enabled   28
st2.action_alias_d_b.formats   28
st2.action_alias_d_b.metadata_file   34
st2.action_alias_d_b.uid   24
st2.action_alias_d_b.name.pack   30
st2.live_action_d_b.action.start_timestamp   42
st2.live_action_d_b.start_timestamp   35
st2.live_action_d_b.end_timestamp   33
st2.live_action_d_b.action   26
st2.live_action_d_b.status   26
st2.live_action_d_b.context.trigger_instance.id   47
st2.live_action_d_b.workflow_execution   38
st2.live_action_d_b.task_execution   34
st2.runner_type_d_b.uid   23
st2.runner_type_d_b.name   24
st2.key_value_pair_d_b.name   27
st2.key_value_pair_d_b.expire_timestamp   39
st2.key_value_pair_d_b.uid   26
st2.key_value_pair_d_b.scope.name   33
st2.runner_type_d_b.uid   23
st2.runner_type_d_b.name   24
st2.action_execution_d_b.rule.ref   33
st2.action_execution_d_b.action.ref   35
st2.action_execution_d_b.liveaction.id   38
st2.action_execution_d_b.start_timestamp   40
st2.action_execution_d_b.end_timestamp   38
st2.action_execution_d_b.status   31
st2.action_execution_d_b.parent   31
st2.action_execution_d_b.rule.name   34
st2.action_execution_d_b.runner.name   36
st2.action_execution_d_b.trigger.name   37
st2.action_execution_d_b.trigger_type.name   42
st2.action_execution_d_b.trigger_instance.id   44
st2.action_execution_d_b.context.user   37
st2.action_execution_d_b.status.action.ref.start_timestamp   58
st2.action_execution_d_b.workflow_execution   43
st2.action_execution_d_b.task_execution   39
st2.action_execution_output_d_b.execution_id   44
st2.action_execution_output_d_b.action_ref   42
st2.action_execution_output_d_b.runner_ref   42
st2.action_execution_output_d_b.timestamp   41
st2.action_execution_output_d_b.output_type   43
st2.action_execution_state_d_b.query_module   43
st2.action_execution_state_d_b.execution_id   43
st2.action_execution_scheduling_queue_item_d_b.action_execution_id   66
st2.action_execution_scheduling_queue_item_d_b.liveaction_id   60
st2.action_execution_scheduling_queue_item_d_b.original_start_timestamp   71
st2.action_execution_scheduling_queue_item_d_b.scheduled_start_timestamp   72
st2.live_action_d_b.action.start_timestamp   42
st2.live_action_d_b.start_timestamp   35
st2.live_action_d_b.end_timestamp   33
st2.live_action_d_b.action   26
st2.live_action_d_b.status   26
st2.live_action_d_b.context.trigger_instance.id   47
st2.live_action_d_b.workflow_execution   38
st2.live_action_d_b.task_execution   34
st2.action_alias_d_b.name   25
st2.action_alias_d_b.enabled   28
st2.action_alias_d_b.formats   28
st2.action_alias_d_b.metadata_file   34
st2.action_alias_d_b.uid   24
st2.action_alias_d_b.name.pack   30
st2.policy_type_d_b.name   24
st2.policy_type_d_b.name.resource_type   38
st2.policy_d_b.name   19
st2.policy_d_b.resource_ref   27
st2.policy_d_b.resource_ref.policy_type   39
st2.policy_d_b.name.pack   24
st2.rule_enforcement_d_b.trigger_instance_id   44
st2.rule_enforcement_d_b.execution_id   37
st2.rule_enforcement_d_b.rule.id   32
st2.rule_enforcement_d_b.rule.ref   33
st2.rule_enforcement_d_b.enforced_at   36
st2.rule_enforcement_d_b.enforced_at   36
st2.rule_enforcement_d_b.enforced_at.rule.ref   45
st2.rule_enforcement_d_b.status   31
st2.rule_enforcement_d_b.tags.name   34
st2.rule_enforcement_d_b.tags.value   35

It only seems to have a problem with two indexes which names are over 70 characters. Perhaps to be on the safe side, we should limit it to 65 characters, for now.

Also, as you have said, it doesn't seem to be directly related to full index name length, but to the individual components (and in this case, collection name is already long. renaming it would be quite a hassle).

[GuiTeK]

Thank you so much @Kami for being so quick to solve this problem! That's amazing! ⚡️

I don't know if you have the answer to this question, but when will 3.0.1 be released in APT repository so we can install it?

[Kami]

@GuiTeK v3.0.1 should be out sometime in the next 2 weeks.

I'm still working on the release (I've been blocked by various race issues we found in our CI/CD testing infrastructure when need to be fixed so I can proceed with the release process).