Cannot view rule with UTF-8 character in name

[blag]

SUMMARY

The ST2 API does not return results when viewing a rule with a UTF-8 character in the name.

I suspect that this will also happen with action names, alias names, and pack names.

STACKSTORM VERSION

st2 3.4.1, on Python 3.6.13

This was during the 3.4.1 release.

OS, environment, install method

Ubuntu 16.04, st2vagrant

Steps to reproduce the problem

Show how to reproduce the problem, using a minimal test-case. Make sure to include any content
(pack content - workflows, actions, etc.) which are needed to reproduce the problem.

1. Create a rule in the web UI with a utf-8 character in its name. I used test_rule_utf8_náme. Creating the rule should work just fine.
2. Once you create the rule, the web UI will switch back to the rule list and flash up a notification in the lower left that creating the rule was successful. It will also try to GET the rule that was just created. This request will fail with an HTTP 404 and flash up a notification that it was unable to get the rule.
3. Trying to get this rule on the command list does not work either.

Expected Results

What did you expect to happen when running the steps above?

Creating a rule with a utf-8 character in its name should work (it currently does).
Viewing a rule with a utf-8 character in its name should work (it currently does not).

Actual Results

What happened? What output did you get?

Getting the rule via CLI does not work:

vagrant@st2vagrant:~$ st2 rule get default.test_rule_utf8_náme
Rule "default.test_rule_utf8_náme" is not found.

Note that listing rules returns the rule with the correct name:

vagrant@st2vagrant:~$ st2 rule list
+-----------------------------------------+----------+-----------------------------------------------------------------------------+---------+
| ref                                     | pack     | description                                                                 | enabled |
+-----------------------------------------+----------+-----------------------------------------------------------------------------+---------+
| chatops.notify                          | chatops  | Notification rule to send results of action executions to stream for        | True    |
|                                         |          | chatops                                                                     |         |
| chatops.notify-errbot                   | chatops  | Notification rule to send results of action executions to stream for        | True    |
|                                         |          | errbot.                                                                     |         |
| default.test_rule_utf8_náme             | default  | asdf                                                                        | True    |
| examples.notify_inquiry                 | examples | Notify relevant users of an Inquiry action                                  | False   |
| examples.notify_slack                   | examples | Sample rule firing on action completion.                                    | True    |
| examples.sample_rule_file_watch         | examples | Sample rule custom trigger type - add a file to be watched by               | True    |
|                                         |          | file_watch_sensor in linux pack.                                            |         |
| examples.sample_rule_with_actiontrigger | examples | Sample rule firing on action completion.                                    | True    |
| examples.sample_rule_with_timer         | examples | Sample rule using an Interval Timer.                                        | False   |
| fixtures.tests.interval_timer           | fixtures | Test rule using an Interval Timer.                                          | False   |
+-----------------------------------------+----------+-----------------------------------------------------------------------------+---------+

The log formatting for logging exceptions is a little messed up, so I tweaked it to make the exception traceback a little bit more readable:

Reformatted log

2021-03-13 07:15:42,225 140260273373864 INFO logging [-] 0c6a8478-e6a5-42ca-b3cf-c7efc0c724ef - GET /v1/rules/views/default.test_rule_utf8_n%C3%A1me with query={} (method='GET',path='/v1/rules/views/default.test_rule_utf8_n%C3%A1me',remote_addr='127.0.0.1',query={},request_id='0c6a8478-e6a5-42ca-b3cf-c7efc0c724ef')
2021-03-13 07:15:42,228 140260277417984 ERROR handlers [-] --- Logging error --- Traceback (most recent call last):
   File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)
 UnicodeEncodeError: 'ascii' codec can't encode character '\xe1' in position 177: ordinal not in range(128)
Call stack:   File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 119, in handle
    super(EventletWorker, self).handle(listener, client, addr)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 121, in __call__
    return self.app(environ, start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/request_id.py", line 51, in __call__
    return self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 149, in __call__
    return self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/logging.py", line 79, in __call__
    values, extra=values)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)
Message: "69e44ae1-1276-4576-8a49-9de4d70e1564 - GET /v1/ruleenforcements/views with query={'rule_ref': ['default.test_rule_utf8_nme'], 'limit': ['10']}"
Arguments: ()--- Logging error --- Traceback (most recent call last):
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 523, in _get_one
    include_fields=include_fields)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 596, in _get_by_ref_or_id
    raise StackStormDBObjectNotFoundError(msg)
 st2common.exceptions.db.StackStormDBObjectNotFoundError: Resource with a reference or id "default.test_rule_utf8_nme" not found

During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
   File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)
 UnicodeEncodeError: 'ascii' codec can't encode characters in position 116-117: ordinal not in range(128)
Call stack:   File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 119, in handle
    super(EventletWorker, self).handle(listener, client, addr)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 69, in __call__
    return self.app(environ, start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/request_id.py", line 51, in __call__
    return self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 149, in __call__
    return self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/logging.py", line 91, in __call__
    retval = self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/cors.py", line 90, in __call__
    return self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/error_handling.py", line 49, in __call__
    return self.app(environ, start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/streaming.py", line 48, in __call__
    return self.app(environ, start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 599, in as_wsgi
    resp = self(req)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 516, in __call__
    resp = func(**kw)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/v1/rule_views.py", line 105, in get_one
    requester_user=requester_user, from_model_kwargs=from_model_kwargs)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 525, in _get_one
    LOG.exception(six.text_type(e))
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)
   File "/usr/lib/python3.6/logging/__init__.py", line 1343, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)
Message: 'Resource with a reference or id "default.test_rule_utf8_nme" not found'
Arguments: ()--- Logging error --- Traceback (most recent call last):
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 523, in _get_one
    include_fields=include_fields)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 596, in _get_by_ref_or_id
    raise StackStormDBObjectNotFoundError(msg)
 st2common.exceptions.db.StackStormDBObjectNotFoundError: Resource with a reference or id "default.test_rule_utf8_nme" not found

During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 516, in __call__
    resp = func(**kw)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/v1/rule_views.py", line 105, in get_one
    requester_user=requester_user, from_model_kwargs=from_model_kwargs)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 526, in _get_one
    abort(http_client.NOT_FOUND, six.text_type(e))
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 79, in abort
    raise exc.status_map[status_code](message)
 webob.exc.HTTPNotFound: Resource with a reference or id "default.test_rule_utf8_nme" not found

During handling of the above exception, another exception occurred:
 Traceback (most recent call last):
   File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)
 UnicodeEncodeError: 'ascii' codec can't encode characters in position 238-239: ordinal not in range(128)
Call stack:   File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 119, in handle
    super(EventletWorker, self).handle(listener, client, addr)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 69, in __call__
    return self.app(environ, start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/request_id.py", line 51, in __call__
    return self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 149, in __call__
    return self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/logging.py", line 91, in __call__
    retval = self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/cors.py", line 90, in __call__
    return self.app(environ, custom_start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/error_handling.py", line 49, in __call__
    return self.app(environ, start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/streaming.py", line 48, in __call__
    return self.app(environ, start_response)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 599, in as_wsgi
    resp = self(req)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 523, in __call__
    (func.__name__, endpoint['operationId'], six.text_type(e)))
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)
   File "/usr/lib/python3.6/logging/__init__.py", line 1343, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)
Message: 'Failed to call controller function "get_one" for operation "st2api.controllers.v1.rule_views:rule_view_controller.get_one": Resource with a reference or id "default.test_rule_utf8_nme" not found'
Arguments: ()6a8478-e6a5-42ca-b3cf-c7efc0c724ef - 404 109 43.653ms (method='GET',path='/v1/rules/views/default.test_rule_utf8_n%C3%A1me',remote_addr='127.0.0.1',status=404,runtime=43.653,content_length=109,request_id='0c6a8478-e6a5-42ca-b3cf-c7efc0c724ef')
e44ae1-1276-4576-8a49-9de4d70e1564 - 200 3820 53.23ms (method='GET',path='/v1/ruleenforcements/views',remote_addr='127.0.0.1',status=200,runtime=53.23,content_length=3820,request_id='69e44ae1-1276-4576-8a49-9de4d70e1564')

Full log

2021-03-13 07:15:42,225 140260273373864 INFO logging [-] 0c6a8478-e6a5-42ca-b3cf-c7efc0c724ef - GET /v1/rules/views/default.test_rule_utf8_n%C3%A1me with query={} (method='GET',path='/v1/rules/views/default.test_rule_utf8_n%C3%A1me',remote_addr='127.0.0.1',query={},request_id='0c6a8478-e6a5-42ca-b3cf-c7efc0c724ef')
2021-03-13 07:15:42,228 140260277417984 ERROR handlers [-] --- Logging error ---

2021-03-13 07:15:42,228 140260277417984 ERROR traceback [-] Traceback (most recent call last):

2021-03-13 07:15:42,228 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,229 140260277417984 ERROR traceback [-]   File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)

2021-03-13 07:15:42,229 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,229 140260277417984 ERROR traceback [-] UnicodeEncodeError: 'ascii' codec can't encode character '\xe1' in position 177: ordinal not in range(128)

2021-03-13 07:15:42,229 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,229 140260277417984 ERROR handlers [-] Call stack:

2021-03-13 07:15:42,229 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)

2021-03-13 07:15:42,229 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,230 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 119, in handle
    super(EventletWorker, self).handle(listener, client, addr)

2021-03-13 07:15:42,230 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,230 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)

2021-03-13 07:15:42,230 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,230 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)

2021-03-13 07:15:42,230 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,231 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 121, in __call__
    return self.app(environ, start_response)

2021-03-13 07:15:42,231 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,231 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/request_id.py", line 51, in __call__
    return self.app(environ, custom_start_response)

2021-03-13 07:15:42,231 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,232 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 149, in __call__
    return self.app(environ, custom_start_response)

2021-03-13 07:15:42,232 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,232 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/logging.py", line 79, in __call__
    values, extra=values)

2021-03-13 07:15:42,233 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,233 140260277417984 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)

2021-03-13 07:15:42,233 140260277417984 ERROR traceback [-]
2021-03-13 07:15:42,233 140260277417984 ERROR handlers [-] Message: "69e44ae1-1276-4576-8a49-9de4d70e1564 - GET /v1/ruleenforcements/views with query={'rule_ref': ['default.test_rule_utf8_nme'], 'limit': ['10']}"
Arguments: ()

2021-03-13 07:15:42,239 140260273373864 ERROR handlers [-] --- Logging error ---

2021-03-13 07:15:42,240 140260273373864 ERROR traceback [-] Traceback (most recent call last):

2021-03-13 07:15:42,240 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,241 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 523, in _get_one
    include_fields=include_fields)

2021-03-13 07:15:42,241 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,241 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 596, in _get_by_ref_or_id
    raise StackStormDBObjectNotFoundError(msg)

2021-03-13 07:15:42,241 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,241 140260273373864 ERROR traceback [-] st2common.exceptions.db.StackStormDBObjectNotFoundError: Resource with a reference or id "default.test_rule_utf8_nme" not found

2021-03-13 07:15:42,242 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,242 140260273373864 ERROR traceback [-]
During handling of the above exception, another exception occurred:


2021-03-13 07:15:42,242 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,243 140260273373864 ERROR traceback [-] Traceback (most recent call last):

2021-03-13 07:15:42,243 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,243 140260273373864 ERROR traceback [-]   File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)

2021-03-13 07:15:42,243 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,244 140260273373864 ERROR traceback [-] UnicodeEncodeError: 'ascii' codec can't encode characters in position 116-117: ordinal not in range(128)

2021-03-13 07:15:42,244 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,244 140260273373864 ERROR handlers [-] Call stack:

2021-03-13 07:15:42,245 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)

2021-03-13 07:15:42,245 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,245 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 119, in handle
    super(EventletWorker, self).handle(listener, client, addr)

2021-03-13 07:15:42,245 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,245 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)

2021-03-13 07:15:42,246 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,246 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)

2021-03-13 07:15:42,247 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,247 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 69, in __call__
    return self.app(environ, start_response)

2021-03-13 07:15:42,247 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,247 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/request_id.py", line 51, in __call__
    return self.app(environ, custom_start_response)

2021-03-13 07:15:42,248 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,248 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 149, in __call__
    return self.app(environ, custom_start_response)

2021-03-13 07:15:42,248 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,248 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/logging.py", line 91, in __call__
    retval = self.app(environ, custom_start_response)

2021-03-13 07:15:42,248 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,249 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/cors.py", line 90, in __call__
    return self.app(environ, custom_start_response)

2021-03-13 07:15:42,249 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,249 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/error_handling.py", line 49, in __call__
    return self.app(environ, start_response)

2021-03-13 07:15:42,249 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,249 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/streaming.py", line 48, in __call__
    return self.app(environ, start_response)

2021-03-13 07:15:42,250 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,250 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 599, in as_wsgi
    resp = self(req)

2021-03-13 07:15:42,250 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,250 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 516, in __call__
    resp = func(**kw)

2021-03-13 07:15:42,250 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,251 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/v1/rule_views.py", line 105, in get_one
    requester_user=requester_user, from_model_kwargs=from_model_kwargs)

2021-03-13 07:15:42,251 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,251 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 525, in _get_one
    LOG.exception(six.text_type(e))

2021-03-13 07:15:42,251 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,251 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)

2021-03-13 07:15:42,252 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,252 140260273373864 ERROR traceback [-]   File "/usr/lib/python3.6/logging/__init__.py", line 1343, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)

2021-03-13 07:15:42,252 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,252 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)

2021-03-13 07:15:42,252 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,252 140260273373864 ERROR handlers [-] Message: 'Resource with a reference or id "default.test_rule_utf8_nme" not found'
Arguments: ()

2021-03-13 07:15:42,253 140260273373864 ERROR handlers [-] --- Logging error ---

2021-03-13 07:15:42,253 140260273373864 ERROR traceback [-] Traceback (most recent call last):

2021-03-13 07:15:42,253 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,253 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 523, in _get_one
    include_fields=include_fields)

2021-03-13 07:15:42,254 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,254 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 596, in _get_by_ref_or_id
    raise StackStormDBObjectNotFoundError(msg)

2021-03-13 07:15:42,254 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,254 140260273373864 ERROR traceback [-] st2common.exceptions.db.StackStormDBObjectNotFoundError: Resource with a reference or id "default.test_rule_utf8_nme" not found

2021-03-13 07:15:42,254 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,255 140260273373864 ERROR traceback [-]
During handling of the above exception, another exception occurred:


2021-03-13 07:15:42,255 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,255 140260273373864 ERROR traceback [-] Traceback (most recent call last):

2021-03-13 07:15:42,255 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,255 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 516, in __call__
    resp = func(**kw)

2021-03-13 07:15:42,255 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,256 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/v1/rule_views.py", line 105, in get_one
    requester_user=requester_user, from_model_kwargs=from_model_kwargs)

2021-03-13 07:15:42,256 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,256 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/controllers/resource.py", line 526, in _get_one
    abort(http_client.NOT_FOUND, six.text_type(e))

2021-03-13 07:15:42,256 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,256 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 79, in abort
    raise exc.status_map[status_code](message)

2021-03-13 07:15:42,257 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,257 140260273373864 ERROR traceback [-] webob.exc.HTTPNotFound: Resource with a reference or id "default.test_rule_utf8_nme" not found

2021-03-13 07:15:42,257 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,258 140260273373864 ERROR traceback [-]
During handling of the above exception, another exception occurred:


2021-03-13 07:15:42,258 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,258 140260273373864 ERROR traceback [-] Traceback (most recent call last):

2021-03-13 07:15:42,258 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,259 140260273373864 ERROR traceback [-]   File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)

2021-03-13 07:15:42,259 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,259 140260273373864 ERROR traceback [-] UnicodeEncodeError: 'ascii' codec can't encode characters in position 238-239: ordinal not in range(128)

2021-03-13 07:15:42,259 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,259 140260273373864 ERROR handlers [-] Call stack:

2021-03-13 07:15:42,260 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)

2021-03-13 07:15:42,260 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,260 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 119, in handle
    super(EventletWorker, self).handle(listener, client, addr)

2021-03-13 07:15:42,260 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,260 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 56, in handle
    self.handle_request(listener_name, req, client, addr)

2021-03-13 07:15:42,261 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,261 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)

2021-03-13 07:15:42,261 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,261 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 69, in __call__
    return self.app(environ, start_response)

2021-03-13 07:15:42,261 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,261 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/request_id.py", line 51, in __call__
    return self.app(environ, custom_start_response)

2021-03-13 07:15:42,262 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,262 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/instrumentation.py", line 149, in __call__
    return self.app(environ, custom_start_response)

2021-03-13 07:15:42,262 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,262 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/logging.py", line 91, in __call__
    retval = self.app(environ, custom_start_response)

2021-03-13 07:15:42,263 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,263 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/cors.py", line 90, in __call__
    return self.app(environ, custom_start_response)

2021-03-13 07:15:42,263 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,263 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/error_handling.py", line 49, in __call__
    return self.app(environ, start_response)

2021-03-13 07:15:42,264 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,264 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/middleware/streaming.py", line 48, in __call__
    return self.app(environ, start_response)

2021-03-13 07:15:42,264 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,265 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 599, in as_wsgi
    resp = self(req)

2021-03-13 07:15:42,265 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,265 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 523, in __call__
    (func.__name__, endpoint['operationId'], six.text_type(e)))

2021-03-13 07:15:42,265 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,266 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)

2021-03-13 07:15:42,266 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,266 140260273373864 ERROR traceback [-]   File "/usr/lib/python3.6/logging/__init__.py", line 1343, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)

2021-03-13 07:15:42,267 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,267 140260273373864 ERROR traceback [-]   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/log.py", line 160, in func_wrapper
    return func(*args, **kwargs)

2021-03-13 07:15:42,267 140260273373864 ERROR traceback [-]
2021-03-13 07:15:42,268 140260273373864 ERROR handlers [-] Message: 'Failed to call controller function "get_one" for operation "st2api.controllers.v1.rule_views:rule_view_controller.get_one": Resource with a reference or id "default.test_rule_utf8_nme" not found'
Arguments: ()

2021-03-13 07:15:42,269 140260273373864 INFO logging [-] 0c6a8478-e6a5-42ca-b3cf-c7efc0c724ef - 404 109 43.653ms (method='GET',path='/v1/rules/views/default.test_rule_utf8_n%C3%A1me',remote_addr='127.0.0.1',status=404,runtime=43.653,content_length=109,request_id='0c6a8478-e6a5-42ca-b3cf-c7efc0c724ef')
2021-03-13 07:15:42,281 140260277417984 INFO logging [-] 69e44ae1-1276-4576-8a49-9de4d70e1564 - 200 3820 53.23ms (method='GET',path='/v1/ruleenforcements/views',remote_addr='127.0.0.1',status=200,runtime=53.23,content_length=3820,request_id='69e44ae1-1276-4576-8a49-9de4d70e1564')

[blag]

Confirmed: the same thing happens with actions/workflows created with utf-8 characters in their names. Listing actions includes the utf8-named actions, including the utf-8 characters, but attempting to get the details of those actions returns an "Action is not found." error.

[Kami]

IIRC, at some point in the (distant) past when storing names with unicode characters we actually stored unicode escape sequence instead of actual unicode values (I forget if that was only under Python 2 or if some other environment things such as a locale affected it).

It could be that now we try to (correctly) directly store the unicode value.

@blag Per the log output it actually looks like the locale is not set to utf-8 (otherwise the logging exception would not be there), is this indeed the case? Also, does it affect other distros?

[Kami]

So yeah, I had a quick look (latest master, utf-8 locale) and it seems like this will be indeed a pain to handle correctly in all the cases.

Here is an error on the server and the client side:

    respiter = self.wsgi(environ, resp.start_response)
  File "/home/vagrant/st2/st2common/st2common/middleware/instrumentation.py", line 47, in __call__
    endpoint, _ = self.router.match(request)
  File "/home/vagrant/st2/st2common/st2common/router.py", line 243, in match
    path = url_unquote(req.path)
  File "/home/vagrant/st2/virtualenv/lib/python3.6/site-packages/webob/request.py", line 476, in path
    bpath = bytes_(self.path_info, self.url_encoding)
  File "/home/vagrant/st2/virtualenv/lib/python3.6/site-packages/webob/descriptors.py", line 70, in fget
    return req.encget(key, encattr=encattr)
  File "/home/vagrant/st2/virtualenv/lib/python3.6/site-packages/webob/request.py", line 165, in encget
    return bytes_(val, 'latin-1').decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 35: invalid continuation byte
[2021-03-13 10:39:09 +0000] [2827] [DEBUG] worker: SIGWINCH ignored.
[2021-03-13 10:39:09 +0000] [2767] [INFO] Handling signal: winch



Traceback (most recent call last):
  File "/home/vagrant/st2/st2client/st2client/commands/resource.py", line 195, in get_resource_by_pk
    instance = self.manager.get_by_id(pk, **kwargs)
  File "/home/vagrant/st2/st2client/st2client/models/core.py", line 42, in decorate
    return func(*args, **kwargs)
  File "/home/vagrant/st2/st2client/st2client/models/core.py", line 214, in get_by_id
    self.handle_error(response)
  File "/home/vagrant/st2/st2client/st2client/models/core.py", line 173, in handle_error
    response.raise_for_status()
  File "/home/vagrant/st2/virtualenv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error
Unable to retrieve detailed message from the HTTP response. Expecting value: line 1 column 1 (char 0)
 for url: http://127.0.0.1:9101/v1/rules/examples.test_rule_utf8_n%ED%B3%83%ED%B2%A1me
ERROR: 'utf-8' codec can't encode characters in position 31-32: surrogates not allowed

Client sends URL path which looks like this examples.test_rule_utf8_n%ED%B3%83%ED%B2%A1me and server fails to handle it correctly.

I also tested the both scenarios and both result an error, albeit a different one since in the second scenario request is processed correctly, but there is an issue in DB layer in how we translate the key.

• http://127.0.0.1:9101/v1/rules/examples.test_rule_utf8_n%ED%B3%83%ED%B2%A1me
• http://127.0.0.1:9101/v1/rules/examples.test_rule_utf8_náme
  test_rule_utf8_náme

I checked the database and the values are indeed stored as actual unicode characters (which is something I would expect and it's the right thing). There is still a question though if that was the case in the past with Python 2.

{ "_id" : ObjectId("604c962c5779c52cbe31a481"), "tags" : [ ], "uid" : "rule:examples:test_rule_utf8_náme", "metadata_file" : "", "name" : "test_rule_utf8_náme", "ref" : "examples.test_rule_utf8_náme", "description" : "Sample rule firing on action completion.", "pack" : "examples", "type" : { "ref" : "standard", "parameters" : {  } }, "trigger" : "core.2df70229-bed3-40cf-8a29-d3553eebe260", "criteria" : { "trigger．channel" : { "pattern" : "slack", "type" : "equals" } }, "action" : { "ref" : "slack.post_message", "parameters" : { "message" : "{{trigger.message}}" } }, "context" : { "user" : "stanley" }, "enabled" : true }

So we will need to answer the following questions and decide how to proceed:

1. Should the client (and API consumers) include actual unicode characters in the URL path or escape sequences (should also check HTTP RFC)?
2. Should we support both approaches per 1)?
3. How to handle Python 2 -> Python 3 compatibility? It's likely there are edge cases there per my comment above).
4. How were database objects with unicode characters in the object names / refs / uids stored in the database in the past under Python 2?

Some related issues:

• GET and POST behavior w.r.t. utf-8 decoding errors Pylons/webob#161
• Handle UnicodeDecodeError on bad UTF-8 URLs GoogleCloudPlatform/webapp2#152

[Kami]

OK, so this change fixes it for curl - b367ef2.

curl -X GET -H  'User-Agent: python-requests/2.23.0' -H  'Accept-Encoding: gzip, deflate' -H  'Accept: */*' -H  'Connection: keep-alive' http://127.0.0.1:9101/v1/rules/examples.test_rule_utf8_náme

curl -X GET -H  'User-Agent: python-requests/2.23.0' -H  'Accept-Encoding: gzip, deflate' -H  'Accept: */*' -H  'Connection: keep-alive' "http://127.0.0.1:9101/v1/rules/examples.test_rule_utf8_n%C3%A1me"

When sending a request with unicode characters in the path, it will get correctly encoded using URI encoding sequences (examples.test_rule_utf8_náme -> examples.test_rule_utf8_n%C3%A1me).

This works correctly with curl, but not with our CLI. I'm still digging in why it doesn't work correctly with the client, but it seems that somewhere in our CLI layer, that unicode value gets incorrectly decoded to a byte string with unicode escape sequences (instead of being left alone and then parsing unicode value to the http client which should url encode / quote it correctly).

It appears the issue is that actual sys.argv will hold byte string with unicode surrogate escape sequences which will be incorrectly sent to our client code.

I believe we will need to do something like this in the client code to handle those scenarios correctly - in short we want actual unicode string with unicode characters and not byte string with surrogate escape sequences.

sys_argv_value = sys_argv_value.encode('ascii', 'surrogateescape').decode('utf-8')

Right now the main question is where to do that. I think that doing in the the HTTPClient layer before passing url to the requests is probably fine.

I confirmed the same issue also exists under Python 3 in the older releases.

[blag]

I confirmed that the same issue existed on ST2 v3.4.0, so this is not a regression in 3.4.1.

This is probably something that we should circle back to, and write tests for. But for 3.4.1, it's not a problem.

[Kami]

Yep, it's not a regression (so not a blocker for v3.4.12), likely been there for a long time (I tested with very old release under Python 3 and it still appears to be there).

Here is also WIP fix for the client side - 4e0bb04.

I confirmed it works end to end, but likely I will end up going with a different approach and already re-encoding sys.argv arg value earlier in the pipeline.