Using several templates in a string may cause errors

[ildar170975]

Checklist:

• [x ] I updated to the latest version available
• [ x] I cleared the cache of my browser

Release with the issue:
1.3.7-beta.1

Last working release (if known):

Browser and Operating System:
Chrome 134.0.6998.36
Win10x64

Description of problem:

Consider this example:

type: vertical-stack
cards:
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: ${VAR}
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: '${VAR}'
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: "${VAR}"
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: >-
        ${VAR}

All cards only differ in a way of using a template:
-- no quotes
-- in single quotes
-- in double quotes
-- with >- multiline-yaml syntax

Results are same correct:

Now let's test with several templates in a string:

type: vertical-stack
cards:
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: ${VAR} ${VAR}
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: '${VAR} ${VAR}'
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: "${VAR} ${VAR}"
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: >-
        ${VAR} ${VAR}

In all cases it gives an error:

Using more quotes gives a result - but values are embraced by quotes:

type: vertical-stack
cards:
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: '''${VAR} ${VAR}'''
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: "'${VAR} ${VAR}'"
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: '"${VAR} ${VAR}"'
  - type: custom:config-template-card
    entities:
      - sun.sun
    variables:
      VAR: states['sun.sun'].state
    card:
      type: markdown
      content: >-
        '${VAR} ${VAR}'

Javascript errors shown in the web inspector (if applicable):

Additional information:

[PaulSD]

Yes, I previously discovered this was broken in 1.3.7-beta.1 ... It is fixed in #159.

More specifically, this commit fixes it: 551e10c

[ildar170975]

The problem is that I tested with a js file taken from "Build" yesterday.
OK, thanks for a feedback, will re-download js & retest!

Took js from here:

Now I see some long process is going on... Is it expected?

[ildar170975]

Retested with the latest build, still see this:

BUT placing any character after the last "}" helps:

I do not think that adding any character should be a way to use several templates...

[PaulSD]

I do not think that adding any character should be a way to use several templates...

Unfortunately, this is a case where there is no ideal solution, so we have to choose one of a few less-than-ideal solutions.

The currently implemented behavior is documented here: 97c213a#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R207

--

The primary problem is that advanced templates may require the use of a } character in the template, and parsing nested } characters is complicated. For example, consider the following template (which we do not currently support):

${ prefix } ${ String(entities.map((e) => { if(e.startsWith('a')) { return 'a'; } else { return 'b'; } }) } {}

If prefix is PREFIX, and entities is ['a'] then you would expect the above should evaluate to:

prefix a {}

However, parsing it correctly requires matching up ${ delimiters with the appropriate } delimiter, and could get very complicated in various corner cases like the following:

${ prefix } ${ String(entities.map((e) => { if(e.startsWith('a')) { return 'a'; } else { return '}'; } }) } {}
${ prefix } ${ String(entities.map((e) => { if(e.startsWith('a')) { return 'a'; } else { return '}'; } }) } }
{ ${ prefix } }{ ${ String(entities.map((e) => { if(e.startsWith('a')) { return 'a'; } else { return '}'; } }) } }
${ prefix+'}' } } ${ String(entities.map((e) => { if(e.startsWith('a')) { return 'a'; } else { return '}'; } }) }

--

In order to parse templates from values in a simple and efficient way, we currently use the following regex to handle multiple templates:

/\${[^}]+}/g

This is a simple regex that is easy to read/understand/troubleshoot. However, it cannot handle any nested } characters.

As a work-around for that limitation, we currently split this to two cases that we handle separately:

1. If the value begins with ${ and ends with } then we assume the whole value is a single template, which makes handling nested } characters trivial (all nested } characters are part of the template and are not template delimiters). Hence, multiple templates are not supported, but nested } are supported in this case.
2. If the value does not begin with ${ and end with } then we use the above regex to parse it, in which case multiple templates are supported, but nested } characters within a template are not supported.

While this is admittedly unintuitive for the user, the implementation is extremely simple, easy to understand, and fast to execute in all cases.

Also, since every possible value that uses multiple templates can be expressed in an alternative way using a single template, this design should support all possible use cases. (The solution for some use cases may be less intuitive than we would like, but there are no use cases that cannot be supported.)

--

While it is technically possible to implement an alternative parser that fully supports both multiple templates and nested } characters, that will significantly complicate the code, which may make it harder for future CTC contributors to understand and modify the code.

For example, a solution I have used in the past is to use a "recursive independent subexpression" regex to identify matching delimiters, like this:

/(\{((?:(?>[^{}]+)|(?-2))*)\})/g

However, this regex is extremely difficult to read/understand/troubleshoot, it may still not handle all possible corner cases, there may be some corner cases with very poor performance, and the Javascript regex engine does not support this (so a separate library or custom regex implementation would be needed).

Alternatively, we could explicitly write code to loop through the characters in the value, count or match { and } characters, and identify the matching pairs that are used as template delimiters, but this would also add a lot of code and complexity.

--

If you feel strongly that we need to fully support multiple templates, then that can be done. However, the current implementation was an intentional design choice in order to minimize implementation complexity.

[PaulSD]

After thinking about this a bit more, maybe a better solution would be to change the template end delimiter from } to }$. This would disambiguate } characters in many corner cases and make parsing easier.

We could continue to support the original ${...} syntax for single-template values, but require ${...}$ for multiple-template values.

[ildar170975]

Paul, please check DM in Community.

but require ${...}$ for multiple-template values.

Not the best way. It could be confusing since “${}” is used usually…

[PaulSD]

Perhaps we should use a different delimiter entirely.

"${ }" is usually used for Javascript template literals, but:

• Javascript template literals are parsed by the Javascript lexer, which has a lot more context for matching delimiters than we can get without re-implementing at least a partial Javascript lexer.
• CTC supports a lot more Javascript syntax within the template than Javascript template literals do, so it may not even be possible for us to properly disambiguate everything CTC supports, unless we apply the Javascript template literal restrictions to CTC (which I don't think we want to do, since that would prohibit a lot of things users are likely to want to do in templates). For example, what if someone wants to use a Javascript template literal within a CTC template?

The more I think about it, the more I think that "${ }" is not really viable for CTC's use case and should be changed (without breaking backward compatibility).

We should obviously avoid {{ }}, since that is what Jinja uses and we don't want any further confusion with Jinja used elsewhere in HA.

EJS and ERB templates use <% %>, so we should probably avoid those, since they have lots of variants and subtleties that we probably wouldn't support, and we want to avoid confusion for any users who are familiar with that syntax.

What about <$ $> or <@ @> ?

[PaulSD]

I have added a commit to #159 which implements <$ $> template delimiters.

I now strongly believe that ${ } was a poor choice for template delimiters, and before we finalize the addition of the new features we have added recently (especially support for multiple templates per value), we should definitely switch to using a new unique delimiter that has no conflicts with Javascript syntax. This will greatly simplify both the documentation and internal implementation relative to what would be required to stick with ${ }.

At this stage, it is easy to change this delimiter to something else (like <@ @>) if needed, but we should finalize the selection new delimiters before the next CTC release.

${ } is still supported in all of the cases that CTC 1.3.6 allowed, but the new delimiter is required for multi-template syntax.

I have updated the Configuration section of README.md to document these changes, but I have not yet updated the Examples.

[ildar170975]

Using {} brackets is a usual thing in JS.
Paul, what about using [[[ and ]]] ?
These things are currently used in button-card.
(unrelated) Also, it uses "return" in every template...