Move2Kube : Detailed Insights into Configuration File Generation

[nitinrana-deloitte]

Move2Kube dynamically adjusts its questioning process based on different scenarios.
Our goal is to gain a deeper understanding of the configuration file generation process without having to execute the transformation.

Dynamic Questioning based on Cluster Type: When changing the cluster type, such as to OpenShift, Move2Kube modifies subsequent questions, like requesting route creation.
We want to explore how this effects the generation of the config file.

Variation in Questions for Multiple Services: In scenarios where multiple services are detected, Move2Kube poses specific questions tailored to each service. Now, these questions can vary based on the number of services it detects, for a particular service there can be more questions compared to the other detected services.

We are aware of the ability of the config file, but our main goal is to gain a deeper understanding of the configuration file generation process so as to build the config file by taking minimal input from user ( rather than asking users to answer the entire transformation questions) and feed it to pipeline hosting m2k to generate manifest file. ****

[ashokponkumar]

Due to the dynamic composing nature of transformers, the questions change depending on the transformers that get activated. Going through the relevant transformers code (https://github.com/konveyor/move2kube/tree/main/transformer) might give an idea. Or we might have to end up trying out with say a cluster type to see what gets into the config. Using --qa-skip might help quickly check the questions that get asked (at least on the default flow).

@HarikrishnanBalagopal do we have any documents explaining the wild card capabilities of a config file?
@Akash-Nayak can you also point to the question categorisation work?

[HarikrishnanBalagopal]

@nitinrana-deloitte Our yaml config file generation is heavily based on https://github.com/mikefarah/yq and json path. Especially the ability to create new yaml documents https://mikefarah.gitbook.io/yq/

The config file generation process is very simple.

1. It starts with an empty dictionary (key value pairs)
2. When a question is asked, the answer is stored in the dictionary. The key is the question ID and the value is the answer to the question that the user provided.
3. The key is often in the form aaa.bbb.ccc.ddd where the . is used as a separator. The key is split into sub keys using the seperator. The subkeys are used to decide where to store the answer.

Example:
Initially the dict is empty

{}

We ask our 1st question with ID aaa.bbb.ccc and get the answer myanswer1. The dictionary now looks like this

{
  "aaa": {
    "bbb": {
      "ccc": "myanswer1"
    }
  }
}

Now we ask our 2nd question with ID aaa.bbb.ddd and get the answer myanswer2. The dictionary now looks like this

{
  "aaa": {
    "bbb": {
      "ccc": "myanswer1",
      "ddd": "myanswer2",
    }
  }
}

The above works for input, multiline, select, and boolean question types.

For multi-select questions, the answer is an array so we use a special symbol [] to indicate the location along the tree where the array goes.

By default we don't store password type questions in the config (although there is a CLI flag to change that).

[nitinrana-deloitte]

@HarikrishnanBalagopal, thanks for this. Could we also get to know more about the decision tree? Are there only a fixed number of questions which m2k is going to ask, we want to answer these questions for the users beforehand, but with increasing code base we have to deal with an increased number of questions, is there any way where we can handle this?

[HarikrishnanBalagopal]

@HarikrishnanBalagopal, thanks for this. Could we also get to know more about the decision tree? Are there only a fixed number of questions which m2k is going to ask, we want to answer these questions for the users beforehand, but with increasing code base we have to deal with an increased number of questions, is there any way where we can handle this?

There is no limit to the number of questions that can be asked. By default we try to avoid asking the user too many questions to keep things simple.

Using custom transformers you can ask as many questions as you need. Example using a custom Starlark transformer
https://move2kube.konveyor.io/transformers/external/starlark#qa-engine
https://github.com/konveyor/move2kube-transformers/blob/a91025e0fe23f288ea1331bb04cf0a34c58ba0d8/cloud-foundry-to-ce-iks-roks/cedockerfile/cedockerfile.star#L52-L54

[seshapad]

@nitinrana-deloitte There is a feature in move2kube to switch on/off categories of questions in the flow. Here is the tutorial for this: https://move2kube.konveyor.io/tutorials/qa-categorization
This could potentially help in removing question sets from a flow if it is not relevant to a particular application.
Please give it a try.

[Akash-Nayak]

@nitinrana-deloitte There is a feature in move2kube to switch on/off categories of questions in the flow. Here is the tutorial for this: https://move2kube.konveyor.io/tutorials/qa-categorization This could potentially help in removing question sets from a flow if it is not relevant to a particular application. Please give it a try.

@nitinrana-deloitte Here is the mapping file, which contains the names of different categories and also the mapping between the different categories and the questions. If we disable a category, say cicd, then Move2Kube won't ask any questions that fall in this category to the user and take the default answer for the questions.

$ move2kube transform -s src --qa-disable cicd

Multiple categories can be disabled by passing the --qa-disable flag multiple times.

$ move2kube transform -s src --qa-disable cicd --qa-disable git

[nitinrana-deloitte]

@Akash-Nayak Thanks for this, I just wanted to clarify that we are implementing move2kube via the move2kube API.
Can we also pass these flags during the API implementation?

[seshapad]

@nitinrana-deloitte
Cluster selector: https://move2kube.konveyor.io/tutorials/customize-cluster-selector
Custom transformers:
Tutorials: https://move2kube.konveyor.io/tutorials/customizing-the-output
Custom transformers: https://github.com/konveyor/move2kube-transformers
please take a look at this: https://move2kube.konveyor.io/tutorials/customizing-the-output/custom-annotations

[nitinrana-deloitte]

@seshapad could we get a document which talks about the wildcard in the config file?

[HarikrishnanBalagopal]

@seshapad could we get a document which talks about the wildcard in the config file?

@nitinrana-deloitte I will try to summarize the semantics here for now.

We always try to get the answer from the config file first. We do this by looking at the ID of the question, splitting that ID into a bunch of sub keys separated by . and looking for the value pointed to by those sub keys in the config map/dict. See my previous answer for an example #1151 (comment)

If we are unable to find the answer in the config, then we ask the user for the answer.

There are 2 wildcards [] and *.

• The [] is meant to be used in question IDs of Multi-Select type questions.
• The * is meant to be used in the config file.

move2kube/common/constants.go

Lines 74 to 77 in 565394b

	// Special is the special case indicator of the multi-select problems
	Special = "[]"
	// MatchAll is used to set the default for a set of keys. Example: move2kube.services.*.ports=[8080]
	MatchAll = "*"

Wildcard *

The * symbol is simpler.

For a question with ID a.mysvc1.port and a config like

{
  "a": {
    "*": {
      "port": "8080"
    }
  }
}

It would not match anything at first.

So then we try to match against * symbols present in the config. We do this by replacing parts of the key with the symbol * and doing a literal match. In the above example the key a.mysvc1.port turns into a.*.port which matches and gets the value 8080.

move2kube/types/qaengine/config.go

Lines 103 to 104 in 565394b

	// starting from 2nd last subkey replace with match all selector *
	// Example: Given a.b.c.d.e this matches a.b.c.*.e, then a.b.*.d.e, then a.*.c.d.e

Wildcard []

The [] symbol is used for Multi-Select type questions. It can match against any key in a map/dict.
Multi-Select type questions take a list of strings as the answer to the question.

move2kube/types/qaengine/config.go

Lines 135 to 137 in 565394b

	// Given 'a.b.[].d' there should be at least one key 'c' such that 'a.b.c.d' exists.
	// Note that 'c' in 'a.b.c.d' does not have to be a valid option for the question.
	// We take the mere existence of 'a.b.c.d' to indicate that the user wants to skip the question.

There are 2 ways such a question can be stored in the config:

move2kube/types/qaengine/config.go

Lines 245 to 256 in 565394b

	// multi-select problem has 2 cases
	key := p.ID
	idx := strings.LastIndex(key, common.Special)
	if idx < 0 {
	// normal case key1 = [val1, val2, val3, ...]
	set(key, p.Answer, c.yamlMap)
	set(key, p.Answer, c.writeYamlMap)
	return nil
	}
	// special case
	baseKey, lastKeySegment := key[:idx-len(common.Delim)], key[idx+len(common.Special)+len(common.Delim):]

For a Multi-Select question with id a.b.c, options ["ans1", "ans2", "ans3"] and answers ["ans1", "ans3"] we get a config that looks like this

{
  "a": {
    "b": {
      "c": ["ans1", "ans3"]
    }
  }
}

For a Multi-Select question with id a.[].c, options ["ans1", "ans2", "ans3"] and answers ["ans1", "ans3"] we get a config that looks like this

{
  "a": {
    "ans1": {
      "c": true
    },
    "ans2": {
      "c": false
    },
    "ans3": {
      "c": true
    }
  }
}

[nitinrana-deloitte]

@HarikrishnanBalagopal, thanks.

So let's say we have this config file which is only having one service (src), now if the service count is to be increased for future runs, in case of a different code base, so can this wildcard functionality be used for the same?

move2kube:
minreplicas: "1"
services:
src:
"5001":
servicetype: ClusterIP
deployment: Deployment
enable: true
port: "5001"
target:
cicd:
tekton:
gitrepobasicauthsecret: ""
gitreposshsecret: ""
registrypushsecret: ""
default:
clustertype: Openshift
ingress:
host: python-sample.com

[ashokponkumar]

{
  "a": {
    "*": {
      "port": "8080"
    }
  }
}

It just needs to be "*" where you have "src" now.

[nitinrana-deloitte]

Hi @ashokponkumar, using the wildcard *, can we construct the config file in such a way so that we still have the ability to ask the user certain questions for instance port number, while implementing it like this it gets skipped during the transformation.

move2kube:
minreplicas: "1"
services:
"*":
"5001":
servicetype: ClusterIP
deployment: Deployment
enable: true
port: "5001"

  I was trying to use the [] selector, but I guess it works only with the multi selector questions

[ashokponkumar]

Hi @ashokponkumar, using the wildcard *, can we construct the config file in such a way so that we still have the ability to ask the user certain questions for instance port number, while implementing it like this it gets skipped during the transformation.

move2kube:
minreplicas: "1"
services:
"*":
"5001":
servicetype: ClusterIP
deployment: Deployment
enable: true
port: "5001"

  I was trying to use the [] selector, but I guess it works only with the multi selector questions

Remove the port number from this config

move2kube:
minreplicas: "1"
services:
"":
"":
servicetype: ClusterIP
deployment: Deployment
enable: true