Add implementation of CHASM List/Count Runs #8662

awln-temporal · 2025-11-18T06:12:44Z

What changed?

Add implementation of CHASM List/Count Runs. Given a CHASM archetype and supported MemoType, a Visibility Query can be ran against both custom and CHASM search attributes. Both operations wrap around the Visibility Manager methods to List/Count Workflows.

In the initial handler of the query in visibility_store.go, the query converter is passed the chasm VisibilitySearchAttributesMapper and the archetypeID. When resolving query parameters, the order of resolving aliases to field names is Custom -> CHASM -> System/Predefined attribute.

Removes chasm transient dependency on persistence package

Fixes visibility task executor to emit Warning Log instead of returning error when custom search attribute is not found. (Edge case when search attributes get deregistered between task creation and execution)

Why?

Required to support CHASM Visibility and Component authors.

How did you test it?

chasm/visibility.go

common/persistence/visibility/store/elasticsearch/visibility_store.go

service/history/chasm_engine.go

common/searchattribute/mapper.go

common/persistence/visibility/types/types.go

chasm/test_visibility.go

chasm/visibility.go

common/persistence/visibility/manager/visibility_manager.go

common/persistence/visibility/visibility_manager_impl.go

common/persistence/visibility/visiblity_manager_metrics.go

service/history/visibility_queue_task_executor.go

service/history/chasm_engine.go

chasm/engine.go

service/history/chasm_engine.go

chasm/engine.go

chasm/ref.go

chasm/search_attribute.go

rodrigozhou · 2025-11-20T19:28:05Z

chasm/visibility.go

+	UserMemoPrefix  = "__user__"
+	ChasmMemoPrefix = "__chasm__"


Suggested change

UserMemoPrefix = "__user__"

ChasmMemoPrefix = "__chasm__"

UserMemoKey = "__user__"

ChasmMemoKey = "__chasm__"

Please change the var names.

common/searchattribute/name_type_map.go

service/history/shard/context_factory.go

rodrigozhou · 2025-11-20T23:15:01Z

service/history/visibility_queue_task_executor.go

 		fieldName, err := searchAttributesMapper.GetFieldName(alias, namespaceEntry.Name().String())
 		if err != nil {
-			return err
+			t.logger.Warn("Failed to get field name for alias, ignoring search attribute", tag.NewStringTag("alias", alias), tag.Error(err))


If you are ignore, you need to continue

Suggested change

t.logger.Warn("Failed to get field name for alias, ignoring search attribute", tag.NewStringTag("alias", alias), tag.Error(err))

t.logger.Warn("Failed to get field name for alias, ignoring search attribute", tag.NewStringTag("alias", alias), tag.Error(err))

continue

service/history/visibility_queue_task_executor.go

bergundy

I will let others review the implementation but the API LGTM.

Just please document all of the exported functions. CHASM will be used by a lot of developers and we want to help people out as much as possible.

yycptt · 2025-11-22T00:39:46Z

chasm/engine.go

+	NextPageToken []byte
+}
+
+type ListExecutionsOptions struct {


can be removed now.

yycptt · 2025-11-22T00:40:38Z

chasm/engine.go

+	NamespaceID   string
+	NamespaceName string


why we need to take in both?

NamespaceID is what we store in persistence, and NamespaceName is needed for dynamic config and custom search attributes mapper.

yycptt · 2025-11-22T00:45:06Z

service/matching/fx.go

interesting that matching needs visibility manager as well.

The scheduler has some constraints in their query, so it needs the visibility manager to build the query converter IIRC.

yycptt · 2025-11-22T00:48:26Z

service/history/visibility_queue_task_executor.go

 		}
 	}
+
+	combinedMemo := make(map[string]*commonpb.Payload)


Suggested change

combinedMemo := make(map[string]*commonpb.Payload)

combinedMemo := make(map[string]*commonpb.Payload, 2)

yycptt · 2025-11-22T00:52:28Z

chasm/registry.go

 	return fqn, ok
 }

+// ComponentByID returns the registrable component for a given archetype ID.


plz also comment that those methods should not be used by chasm library developer. and also for RegistrableComponent.SearchAttributesMapper()

we probably need to create a separate interface for developer facing registry later.

yycptt · 2025-11-22T00:57:38Z

chasm/tree_test.go

+	s.Len(root.currentSA, 3)
 	s.Contains(root.currentSA, "TemporalDatetime01")
 	s.True(root.currentSA["TemporalDatetime01"].(VisibilityValueTime).Equal(VisibilityValueTime(now)))
-	s.True(root.currentMemo[TestComponentStartTimeMemoKey].(VisibilityValueTime).Equal(VisibilityValueTime(now)))


we should be able to assert the memo value? here and below?

yycptt · 2025-11-22T01:03:15Z

chasm/tree.go

 	if ok {
 		newMemo := memoProvider.Memo(immutableContext)
-		if !maps.EqualFunc(n.currentMemo, newMemo, isVisibilityValueEqual) {
+		if !proto.Equal(n.currentMemo, newMemo) {


hmm I think now that we are using proto.Message we no longer have guarantee that there's no error upon encoding in task executor, but guess that should never happen.

Something for later, since the currentMemo/SA here are always consistent with CHASM tree state, in visibility task executor we actually don't have to invoke SA/Memo provider again, we can just serialize & buffer the value here (and now we can also return error upon closeTransaction if it can't be encoded), and use it in the vis task executor.

yycptt · 2025-11-22T01:09:37Z

common/persistence/visibility/manager/visibility_manager.go

+		Count int64
+	}
+
+	ChasmExecutionInfo struct {


Now that persistence can depend on chasm package, for those new struct definitions, is it possible to reuse the corresponding definitions in chasm package and thus avoiding the conversion between the different struct definition of essentially the same thing in chasm_engine.go?

yycptt · 2025-11-22T01:26:39Z

service/frontend/admin_handler.go

 			InitialFailoverVersion:   resp.GetInitialFailoverVersion(),
 			IsGlobalNamespaceEnabled: resp.GetIsGlobalNamespaceEnabled(),
 			IsConnectionEnabled:      request.GetEnableRemoteClusterConnection(),
-			IsReplicationEnabled:     request.GetIsReplicationEnabled(),


is this intentional?

yycptt · 2025-11-22T01:55:27Z

common/persistence/visibility/visibility_manager_impl.go

+		var userMemo *commonpb.Memo
+		var chasmMemoPayload *commonpb.Payload
+
+		// Check if archetype matches to decide how to split memo


hmm why not just check if __chasm__ is a key in the combinedMemo?

if we are worrying about potential conflict (though I feel that chance is very low...), I'd probably just check if TemporalNamespaceDivision string is a number or not (just the first char should be enough), since we are going to use this method to query across archetypes, e.g. in namespace migration case, I just want all visibility records.

Yes, I'm concerned about potential conflict (although very low).
I'm ok with checking if TemporalNamespaceDivision is a number.

rodrigozhou · 2025-11-22T07:46:15Z

chasm/visibility.go


+// GetType returns the type of a CHASM search attribute field.
+// Returns an error if the field is not found in the type map.
+func (v *VisibilitySearchAttributesMapper) GetType(fieldName string) (enumspb.IndexedValueType, error) {


Rename function: ValueType or ValueTypeOf

rodrigozhou · 2025-11-22T07:47:51Z

chasm/visibility.go

+	UserMemoPrefix  = "__user__"
+	ChasmMemoPrefix = "__chasm__"


Please change the var names.

rodrigozhou · 2025-11-22T07:54:13Z

common/persistence/visibility/store/query/converter_legacy.go

+	// Get field type, checking CHASM mapper first if applicable
+	var tp enumspb.IndexedValueType
+	if sadefs.IsChasmSearchAttribute(colName) {
+		var err error


err is already defined in the scope of this function, no need to create a new var.

rodrigozhou · 2025-11-22T07:55:18Z

common/persistence/visibility/store/query/converter_legacy.go

+		tp, err = c.chasmMapper.GetType(colName)
+		if err != nil {
+			return nil, NewConverterError("invalid search attribute: %s", colName)
+		}
+	} else {
+		var err error
+		tp, err = c.saNameType.GetType(colName)


I wonder if you can actually combine these maps and do the check together. These are field names, so there shouldn't be any conflicting keys.

rodrigozhou · 2025-11-22T07:58:07Z

common/persistence/visibility/store/sql/visibility_store.go

 		enableUnifiedQueryConverter dynamicconfig.BoolPropertyFn
 	}
+
+	listWorkflowExecutionsRequestInternal struct {


Suggested change

listWorkflowExecutionsRequestInternal struct {

listExecutionsInternalRequest struct {

awln-temporal requested review from a team as code owners November 18, 2025 06:12

awln-temporal requested review from bergundy and rodrigozhou November 18, 2025 06:18

rodrigozhou requested changes Nov 18, 2025

View reviewed changes

rodrigozhou requested a review from yycptt November 18, 2025 21:22

awln-temporal force-pushed the CHASMVisibilityQuery branch from 457d6b9 to e63973b Compare November 19, 2025 20:52

awln-temporal requested a review from rodrigozhou November 19, 2025 21:00

rodrigozhou requested changes Nov 19, 2025

View reviewed changes

awln-temporal force-pushed the CHASMVisibilityQuery branch 4 times, most recently from 1b7b1d1 to db42bca Compare November 20, 2025 18:52

awln-temporal requested a review from rodrigozhou November 20, 2025 19:01

awln-temporal force-pushed the CHASMVisibilityQuery branch 3 times, most recently from c71cd57 to 6d0b6fa Compare November 20, 2025 19:10

awln-temporal commented Nov 20, 2025

View reviewed changes

service/history/chasm_engine.go Outdated Show resolved Hide resolved

chasm/engine.go Outdated Show resolved Hide resolved

service/history/chasm_engine.go Outdated Show resolved Hide resolved

chasm/engine.go Outdated Show resolved Hide resolved

rodrigozhou requested changes Nov 20, 2025

View reviewed changes

awln-temporal force-pushed the CHASMVisibilityQuery branch 6 times, most recently from 256aa5d to 74fae89 Compare November 21, 2025 21:47

Implement CHASM List/Count Executions

23c1374

awln-temporal force-pushed the CHASMVisibilityQuery branch from 74fae89 to 23c1374 Compare November 21, 2025 21:58

bergundy reviewed Nov 21, 2025

View reviewed changes

yycptt reviewed Nov 22, 2025

View reviewed changes

rodrigozhou reviewed Nov 22, 2025

View reviewed changes

	t.logger.Warn("Failed to get field name for alias, ignoring search attribute", tag.NewStringTag("alias", alias), tag.Error(err))
	t.logger.Warn("Failed to get field name for alias, ignoring search attribute", tag.NewStringTag("alias", alias), tag.Error(err))
	continue

	combinedMemo := make(map[string]*commonpb.Payload)
	combinedMemo := make(map[string]*commonpb.Payload, 2)

	listWorkflowExecutionsRequestInternal struct {
	listExecutionsInternalRequest struct {

Add implementation of CHASM List/Count Runs #8662

Are you sure you want to change the base?

Add implementation of CHASM List/Count Runs #8662

Uh oh!

Conversation

awln-temporal commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed?

Removes chasm transient dependency on persistence package

Fixes visibility task executor to emit Warning Log instead of returning error when custom search attribute is not found. (Edge case when search attributes get deregistered between task creation and execution)

Why?

How did you test it?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bergundy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yycptt Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

awln-temporal commented Nov 18, 2025 •

edited

Loading

yycptt Nov 22, 2025 •

edited

Loading