v3.8.1

3.8.1 - December 13, 2023

https://stackstorm.com/2023/12/v3-8-1-released/

Fixed

• Fix proxy auth mode in HA environments #5766 #6049
  Contributed by @floatingstatic

• Fix issue with linux pack actions failed to run remotely due to incorrect python shebang. #5983 #6042
  Contributed by Ronnie Hoffmann (@ZoeLeah Schwarz IT KG)

• Fix CI usses #6015
  Contributed by Amanda McGuinness (@amanda11 intive)

• Bumped paramiko to 2.10.5 to fix an issue with SSH Certs - paramiko/paramiko#2017 (security)
  Contributed by @jk464

• Avoid logging sensitive information in debug (fix #5977)

• Fix codecov failures for stackstorm/st2 tests. #6035, #6046, #6048

• Fix #4676, edge case where --inherit-env is skipped if the action has no parameters

• Fix ST2 Client for Windows Clients. PWD is a Unix only Libary. #6071
  Contributed by (@philipphomberger Schwarz IT KG)

• Fix Snyk Security Finding Cross-site Scripting (XSS) in contrib/examples/sensors/echo_flask_app.py #6070
  Contributed by (@philipphomberger Schwarz IT KG)

• Update cryptography 3.4.7 -> 39.0.1, pyOpenSSL 21.0.0 -> 23.1.0, paramiko 2.10.5 -> 2.11.0 (security). #6055

• Bumped eventlet to 0.33.3 and gunicorn to 21.2.0 to fix RecursionError bug in setting SSLContext minimum_version property. (security) #6061
  Contributed by @jk464

• Update orquesta to v1.6.0 to fix outdated dependencies (security). #6050

• Fix KV value lookup in actions when RBAC is enabled #5934

• Update version 3.1.15 of gitpython to 3.1.18 for py3.6 and to 3.1.37 for py3.8 (security). #6063

• Update importlib-metadata from 3.10.1 to 4.8.3 for py3.6 and to 4.10.1 for py3.8 (security). #6072
  Contributed by @jk464

• For "local-shell-script" runner, on readonly filesystems, don't attempt to run chmod +x on script_action. Fixes #5591
  Contributed by @jk464

Added

• Move git clone to user_home/.st2packs #5845

• Error on st2ctl status when running in Kubernetes. #5851
  Contributed by @mamercad

• Continue introducing pants <https://www.pantsbuild.org/docs>_ to improve DX (Developer Experience)
  working on StackStorm, improve our security posture, and improve CI reliability thanks in part
  to pants' use of PEX lockfiles. This is not a user-facing addition.
  #5778 #5789 #5817 #5795 #5830 #5833 #5834 #5841 #5840 #5838 #5842 #5837 #5849 #5850
  #5846 #5853 #5848 #5847 #5858 #5857 #5860 #5868 #5871 #5864 #5874 #5884 #5893 #5891
  #5890 #5898 #5901 #5906 #5899 #5907 #5909 #5922 #5926 #5927 #5925 #5928 #5929 #5930
  #5931 #5932 #5948 #5949 #5950
  Contributed by @cognifloyd

• Added a joint index to solve the problem of slow mongo queries for scheduled executions. #5805

• Added publisher to ActionAlias to enable streaming ActionAlias create/update/delete events. #5763
  Contributed by @ubaumann

• Expose environment variable ST2_ACTION_DEBUG to all StackStorm actions.
  Contributed by @maxfactor1

• Python 3.9 support. #5730
  Contributed by Amanda McGuinness (@amanda11 intive)

• Run the st2 self-check in Github Actions and support the environment variable TESTS_TO_SKIP to skip tests when running st2-self-check. #5609
  Contributed by @winem

Changed

• Remove distutils dependencies across the project. #5992
  Contributed by @AndroxxTraxxon

v3.8.0

3.8.0 - November 18, 2022

https://stackstorm.com/2022/12/v3-8-0-released/

Fixed

• Fix redis SSL problems with sentinel #5660

• Fix a bug in the pack config loader so that objects covered by an patternProperties schema
  or arrays using additionalItems schema(s) can use encrypted datastore keys and have their
  default values applied correctly. #5321

  Contributed by @cognifloyd

• Fixed st2client/st2client/base.py file to check for http_proxy and https_proxy environment variables for both lower and upper cases.

  Contributed by @S-T-A-R-L-O-R-D

• Fixed a bug where calling 'get_by_name' on client for getting key details was not returning any results despite key being stored. #5677

  Contributed by @bharath-orchestral

• Fixed st2client/st2client/base.py file to use https_proxy(not http_proxy) to check HTTPS_PROXY environment variables.

  Contributed by @wfgydbu

• Fixed schema utils to more reliably handle schemas that define nested arrays (object-array-object-array-string) as discovered in some
  of the ansible installer RBAC tests (see #5684). This includes a test that reproduced the error so we don't hit this again. #5685

• Fixed eventlet monkey patching so more of the unit tests work under pytest. #5689

• Fix and reenable prance-based openapi spec validation, but make our custom x-api-model validation optional as the spec is out-of-date. #5709
  Contributed by @cognifloyd

• Fixed generation of st2.conf.sample to show correct syntax for [sensorcontainer].partition_provider (space separated key:value pairs). #5710
  Contributed by @cognifloyd

• Fix access to key-value pairs in workflow and action execution where RBAC rules did not get applied #5764

  Contributed by @m4dcoder

• Add backward compatibility to secret masking introduced in #5319 to prevent security-relative issues.
  Migration to the new schema is required to take advantage of the full output schema validation. #5783

  Contributed by @m4dcoder

Added

• Added graceful shutdown for workflow engine. #5463
  Contributed by @khushboobhatia01

• Add ST2_USE_DEBUGGER env var as alternative to the --use-debugger cli flag. #5675
  Contributed by @cognifloyd

• Added purging of old tokens. #5679
  Contributed by Amanda McGuinness (@amanda11 intive)

• Begin introducing pants <https://www.pantsbuild.org/docs>_ to improve DX (Developer Experience)
  working on StackStorm, improve our security posture, and improve CI reliability thanks in part
  to pants' use of PEX lockfiles. This is not a user-facing addition. #5713 #5724 #5726 #5725 #5732 #5733 #5737 #5738 #5758 #5751 #5774 #5776 #5777 #5782
  Contributed by @cognifloyd

Changed

• BREAKING CHANGE for anyone that uses output_schema, which is disabled by default.
  If you have [system].validate_output_schema = True in st2.conf AND you have added
  output_schema to any of your packs, then you must update your action metadata.

  output_schema must be a full jsonschema now. If a schema is not well-formed, we ignore it.
  Now, output can be types other than object such as list, bool, int, etc.
  This also means that all of an action's output can be masked as a secret.

  To get the same behavior, you'll need to update your output schema.
  For example, this schema:

    output_schema:
      property1:
        type: bool
      property2:
        type: str

should be updated like this:

    output_schema:
      type: object
      properties:
        property1:
          type: bool
        property2:
          type: str
      additionalProperties: false

#5319

Contributed by @cognifloyd

• Changed the X-XSS-Protection HTTP header from 1; mode=block to 0 in the conf/nginx/st2.conf to align with the OWASP security standards. #5298

  Contributed by @LiamRiddell

• Use PEP 440 direct reference requirements instead of legacy PIP VCS requirements. Now, our *.requirements.txt files use
  package-name@ git+https://url@version ; markers instead of git+https://url@version#egg=package-name ; markers. #5673
  Contributed by @cognifloyd

• Move from udatetime to ciso8601 for date functionality ahead of supporting python3.9 #5692
  Contributed by Amanda McGuinness (@amanda11 intive)

• Refactor tests to use python imports to identify test fixtures. #5699 #5702 #5703 #5704 #5705 #5706
  Contributed by @cognifloyd

• Refactor st2-generate-schemas so that logic is in an importable module. #5708
  Contributed by @cognifloyd

Removed

• Removed st2exporter service. It is unmaintained and does not get installed. It was
  originally meant to help with analytics by exporting executions as json files that
  could be imported into something like elasticsearch. Our code is now instrumented
  to make a wider variety of stats available to metrics drivers. #5676
  Contributed by @cognifloyd

v3.7.0

https://stackstorm.com/2022/05/10/stackstorm-3-7-0-released/

3.7.0 - May 05, 2022

Added

• Added st2 API get action parameters by ref. #5509
  API endpoint /api/v1/actions/views/parameters/{action_id} accepts ref_or_id.
  Contributed by @DavidMeu

• Enable setting ttl for MockDatastoreService. #5468
  Contributed by @ytjohn

• Added st2 API and CLI command for actions clone operation.

  API endpoint /api/v1/actions/{ref_or_id}/clone takes ref_or_id of source action.Request method body takes destination pack and action name. Request method body also takes optional parameter overwrite. overwrite = true in case of destination action already exists and to be overwritten.

  CLI command st2 action clone <ref_or_id> <dest_pack> <dest_action> takes source ref_or_id, destination pack name and destination action name as mandatory arguments. In case destination already exists then command takes optional argument -f or --force to overwrite destination action. #5345

  Contributed by @mahesh-orch.

• Implemented RBAC functionality for existing KEY_VALUE_VIEW, KEY_VALUE_SET, KEY_VALUE_DELETE and new permission types KEY_VALUE_LIST, KEY_VALUE_ALL. RBAC is enabled in the st2.conf file. Access to a key value pair is checked in the KeyValuePair API controller. #5354
  Contributed by @m4dcoder and @ashwini-orchestral

• Added service deregistration on shutdown of a service. #5396
  Contributed by @khushboobhatia01

• Added pysocks python package for SOCKS proxy support. #5460
  Contributed by @kingsleyadam

• Added support for multiple LDAP hosts to st2-auth-ldap. #5535, StackStorm/st2-auth-ldap#100
  Contributed by @ktyogurt

• Implemented graceful shutdown for action runner. Enabled graceful_shutdown in st2.conf file. #5428
  Contributed by @khushboobhatia01

• Enhanced 'search' operator to allow complex criteria matching on payload items. #5482
  Contributed by @erceth

• Added cancel/pause/resume requester information to execution context. #5554
  Contributed by @khushboobhatia01

• Added trigger.headers_lower to webhook trigger payload. This allows rules to match webhook triggers without dealing with the case-sensitive nature of trigger.headers, as triggers.headers_lower providers the same headers, but with the header name lower cased. #5038
  Contributed by @Rand01ph

• Added support to override enabled parameter of resources. #5506
  Contributed by Amanda McGuinness (@amanda11 Intive)

• Add new api.auth_cookie_secure and api.auth_cookie_same_site config options which specify values which are set for secure and SameSite attribute for the auth cookie we set when authenticating via token / api key in query parameter value (e.g. via st2web).

  For security reasons, api.auth_cookie_secure defaults to True. This should only be changed to False if you have a valid reason to not run StackStorm behind HTTPs proxy.

  Default value for api.auth_cookie_same_site is lax. If you want to disable this functionality so it behaves the same as in the previous releases, you can set that option to None.

  #5248
  Contributed by @Kami.

• Add new st2 action-alias test <message string> CLI command which allows users to easily test action alias matching and result formatting.

  This command will first try to find a matching alias (same as st2 action-alias match command) and if a match is found, trigger an execution (same as st2 action-alias execute command) and format the execution result.

  This means it uses exactly the same flow as commands on chat, but the interaction avoids chat and hubot which should make testing and developing aliases easier and faster. #5143

  #5143
  Contributed by @Kami.

• Add new credentials.basic_auth = username:password CLI configuration option.

  This argument allows client to use additional set of basic auth credentials when talking to the StackStorm API endpoints (api, auth, stream) - that is, in addition to the token / api key native StackStorm auth.

  This allows for simple basic auth based multi factor authentication implementation for installations which don't utilize SSO.

  #5152
  Contributed by @Kami.

• Add new audit message when a user has decrypted a key whether manually in the container (st2 key get [] --decrypt)
  or through a workflow with a defined config. #5594
  Contributed by @dmork123

• Added garbage collection for rule_enforcement and trace models #5596/5602
  Contributed by Amanda McGuinness (@amanda11 intive)

• Added garbage collection for workflow execution and task execution objects #4924
  Contributed by @srimandaleeka01 and @amanda11

Changed

• Minor updates for RockyLinux. #5552

  Contributed by Amanda McGuinness (@amanda11 intive)

• Bump black to v22.3.0 - This is used internally to reformat our python code. #5606

• Updated paramiko version to 2.10.3 to add support for more key verification algorithms. #5600

Fixed

• Fix deserialization bug in st2 API for url encoded payloads. #5536
  Contributed by @sravs-dev

• Fix issue of WinRM parameter passing fails for larger scripts.#5538
  Contributed by @ashwini-orchestral

• Fix Type error for time_diff critera comparison. convert the timediff value as float to match timedelta.total_seconds() return. #5462
  Contributed by @blackstrip

• Fix issue with pack option not working when running policy list cli #5534
  Contributed by @momokuri-3

• Fix exception thrown if action parameter contains {{ or {% and no closing jinja characters. #5556
  contributed by @guzzijones12

• Link shutdown routine and sigterm handler to main thread #5555
  Contributed by @khushboobhatia01

• Change compound index for ActionExecutionDB to improve query performance #5568
  Contributed by @khushboobhatia01

• Fix build issue due to MarkUpSafe 2.1.0 removing soft_unicode
  Contributed by Amanda McGuinness (@amanda11 intive) #5581

• Fixed regression caused by #5358. Use string lock name instead of object ID. #5484
  Contributed by @khushboobhatia01

• Fix st2-self-check script reporting falsey success when the nested workflows runs failed. #5487

• Fix actions from the contrib/linux pack that fail on CentOS-8 but work on other operating systems and distributions. (bug fix) #4999 #5004
  Reported by @blag and @dove-young contributed by @winem.

• Use byte type lock name which is supported by all tooz drivers. #5529
  Contributed by @khushboobhatia01

• Fixed issue where pack index searches are ignoring no_proxy #5497
  Contributed by @minsis

• Fixed trigger references emitted by linux.file_watch.line. #5467

  Prior to this patch multiple files could be watched but the rule reference of last registered file
  would be used for all trigger emissions causing rule enforcement to fail. References are now tracked
  on a per file basis and used in trigger emissions.

  Contributed by @nzlosh

• Downgrade tenacity as tooz dependency on tenacity has always been < 7.0.0 #5607
  Contributed by @khushboobhatia01

• Pin typing-extensions<4.2 (used indirectly by st2client) to maintain python 3.6 support. #5638

v3.6.0

https://stackstorm.com/2021/12/16/stackstorm-v3-6-0-released/

3.6.0 - October 29, 2021

Added

• Added possibility to add new values to the KV store via CLI without leaking them to the shell history. #5164

• st2.conf is now the only place to configure ports for st2api, st2auth, and st2stream.

  We replaced the static .socket sytemd units in deb and rpm packages with a python-based generator for the
  st2api, st2auth, and st2stream services. The generators will get <ip>:<port> from st2.conf
  to create the .socket files dynamically. #5286 and st2-packages#706

  Contributed by @nzlosh

Changed

• Modified action delete API to delete action files from disk along with backward compatibility.

  From CLI st2 action delete <pack>.<action> will delete only action database entry.
  From CLI st2 action delete --remove-files <pack>.<action> or st2 action delete -r <pack>.<action>
  will delete action database entry along with files from disk.

  API action DELETE method with {"remove_files": true} argument in json body will remove database
  entry of action along with files from disk.
  API action DELETE method with {"remove_files": false} or no additional argument in json body will remove
  only action database entry. #5304, #5351, #5360

  Contributed by @mahesh-orch.

• Removed --python3 deprecated flag from st2client. #5305

  Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

  Contributed by @blag.

• Fixed __init__.py files to use double quotes to better align with black linting #5299

  Contributed by @blag.

• Reduced minimum TTL on garbage collection for action executions and trigger instances from 7 days to 1 day. #5287

  Contributed by @ericreeves.

• update db connect mongo connection test - isMaster MongoDB command depreciated, switch to ping #5302, #5341

  Contributed by @lukepatrick

• Actionrunner worker shutdown should stop Kombu consumer thread. #5338

  Contributed by @khushboobhatia01

• Move to using Jinja sandboxed environment #5359

  Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

• Pinned python module networkx to versions between 2.5.1(included) and 2.6(excluded) because Python v3.6 support was dropped in v2.6.
  Also pinned decorator==4.4.2 (dependency of networkx<2.6) to work around missing python 3.8 classifiers on decorator's wheel. #5376

  Contributed by @nzlosh

• Add new --enable-profiler flag to all the servies. This flag enables cProfiler based profiler
  for the service in question and dumps the profiling data to a file on process
  exit.

  This functionality should never be used in production, but only in development environments or
  similar when profiling code. #5199

  Contributed by @Kami.

• Add new --enable-eventlet-blocking-detection flag to all the servies. This flag enables
  eventlet long operation / blocked main loop logic which throws an exception if a particular
  code blocks longer than a specific duration in seconds.

  This functionality should never be used in production, but only in development environments or
  similar when debugging code. #5199

• Silence pylint about dev/debugging utility (tools/direct_queue_publisher.py) that uses pika because kombu
  doesn't support what it does. If anyone uses that utility, they have to install pika manually. #5380

• Fixed version of cffi as changes in 1.15.0 meant that it attempted to load libffi.so.8. #5390

  Contributed by @amanda11, Ammeon Solutions

• Updated Bash installer to install latest RabbitMQ version rather than out-dated version available
  in OS distributions.

  Contributed by @amanda11, Ammeon Solutions

Fixed

• Correct error reported when encrypted key value is reported, and another key value parameter that requires conversion is present. #5328
  Contributed by @amanda11, Ammeon Solutions

• Make update_executions() atomic by protecting the update with a coordination lock. Actions, like workflows, may have multiple
  concurrent updates to their execution state. This makes those updates safer, which should make the execution status more reliable. #5358

  Contributed by @khushboobhatia01

• Fix "not iterable" error for output_schema handling. If a schema is not well-formed, we ignore it.
  Also, if action output is anything other than a JSON object, we do not try to process it any more.
  output_schema will change in a future release to support non-object output. #5309

  Contributed by @guzzijones

• core.inject_trigger: resolve trigger payload shadowing by deprecating trigger param in favor of trigger_name.
  trigger param is still available for backwards compatibility, but will be removed in a future release. #5335 and #5383

  Contributed by @mjtice

v3.5.0

https://stackstorm.com/2021/06/29/stackstorm-v3-5-0-released/

Added

• Added web header settings for additional security hardening to nginx.conf: X-Frame-Options,
  Strict-Transport-Security, X-XSS-Protection and server-tokens. #5183

  Contributed by @Shital.

• Added support for limit and offset argument to the list_values data store
  service method (#5097 and #5171).

  Contributed by @anirudhbagri.

• Various additional metrics have been added to the action runner service to provide for better
  operational visibility. (improvement) #4846

  Contributed by @Kami.

• Added sensor model to list of JSON schemas auto-generated by make schemasgen that can be used
  by development tools to validate pack contents. (improvement)

• Added the command line utility st2-validate-pack that can be used by pack developers to
  validate pack contents. (improvement)

• Fix a bug in the API and CLI code which would prevent users from being able to retrieve resources
  which contain non-ascii (utf-8) characters in the names / references. (bug fix) #5189

  Contributed by @Kami.

• Fix a bug in the API router code and make sure we return correct and user-friendly error to the
  user in case we fail to parse the request URL / path because it contains invalid or incorrectly
  URL encoded data.

  Previously such errors weren't handled correctly which meant original exception with a stack
  trace got propagated to the user. (bug fix) #5189

  Contributed by @Kami.

• Make redis the default coordinator backend.

• Fix a bug in the pack config loader so that objects covered by an additionalProperties schema
  can use encrypted datastore keys and have their default values applied correctly. #5225

  Contributed by @cognifloyd.

• Add new database.compressors and database.zlib_compression_level config option which
  specifies compression algorithms client supports for network / transport level compression
  when talking to MongoDB.

  Actual compression algorithm used will be then decided by the server and depends on the
  algorithms which are supported by the server + client.

  Possible / valid values include: zstd, zlib. Keep in mind that zstandard (zstd) is only supported
  by MongoDB >= 4.2.

  Our official Debian and RPM packages bundle zstandard dependency by default which means
  setting this value to zstd should work out of the box as long as the server runs
  MongoDB >= 4.2. #5177

  Contributed by @Kami.

• Add support for compressing the payloads which are sent over the message bus. Compression is
  disabled by default and user can enable it by setting messaging.compression config option
  to one of the following values: zstd, lzma, bz2, gzip.

  In most cases we recommend using zstd (zstandard) since it offers best trade off between
  compression ratio and number of CPU cycles spent for compression and compression.

  How this will affect the deployment and throughput is very much user specific (workflow and
  resources available). It may make sense to enable it when generic action trigger is enabled
  and when working with executions with large textual results. #5241

  Contributed by @Kami.

• Mask secrets in output of an action execution in the API if the action has an output schema
  defined and one or more output parameters are marked as secret. #5250

  Contributed by @mahesh-orch.

Changed

• All the code has been refactored using black and black style is automatically enforced and
  required for all the new code. (#5156)

  Contributed by @Kami.

• Default nginx config (conf/nginx/st2.conf) which is used by the installer and Docker
  images has been updated to only support TLS v1.2 and TLS v1.3 (support for TLS v1.0 and v1.1
  has been removed).

  Keep in mind that TLS v1.3 will only be used when nginx is running on more recent distros
  where nginx is compiled against OpenSSL v1.1.1 which supports TLS 1.3. #5183 #5216

  Contributed by @Kami and @Shital.

• Add new -x argument to the st2 execution get command which allows
  result field to be excluded from the output. (improvement) #4846

• Update st2 execution get <id> command to also display execution log attribute which
  includes execution state transition information.

  By default end_timestamp attribute and duration attribute displayed in the command
  output only include the time it took action runner to finish running actual action, but it
  doesn't include the time it it takes action runner container to fully finish running the
  execution - this includes persisting execution result in the database.

  For actions which return large results, there could be a substantial discrepancy - e.g.
  action itself could finish in 0.5 seconds, but writing data to the database could take
  additional 5 seconds after the action code itself was executed.

  For all purposes until the execution result is persisted to the database, execution is
  not considered as finished.

  While writing result to the database action runner is also consuming CPU cycles since
  serialization of large results is a CPU intensive task.

  This means that "elapsed" attribute and start_timestamp + end_timestamp will make it look
  like actual action completed in 0.5 seconds, but in reality it took 5.5 seconds (0.5 + 5 seconds).

  Log attribute can be used to determine actual duration of the execution (from start to
  finish). (improvement) #4846

  Contributed by @Kami.

• Various internal improvements (reducing number of DB queries, speeding up YAML parsing, using
  DB object cache, etc.) which should speed up pack action registration between 15-30%. This is
  especially pronounced with packs which have a lot of actions (e.g. aws one).
  (improvement) #4846

  Contributed by @Kami.

• Underlying database field type and storage format for the Execution, LiveAction,
  WorkflowExecutionDB, TaskExecutionDB and TriggerInstanceDB database models has
  changed.

  This new format is much faster and efficient than the previous one. Users with larger executions
  (executions with larger results) should see the biggest improvements, but the change also scales
  down so there should also be improvements when reading and writing executions with small and
  medium sized results.

  Our micro and end to benchmarks have shown improvements up to 15-20x for write path (storing
  model in the database) and up to 10x for the read path.

  To put things into perspective - with previous version, running a Python runner action which
  returns 8 MB result would take around ~18 seconds total, but with this new storage format, it
  takes around 2 seconds (in this context, duration means the from the time the execution was
  scheduled to the time the execution model and result was written and available in the database).

  The difference is even larger when working with Orquesta workflows.

  Overall performance improvement doesn't just mean large decrease in those operation timings, but
  also large overall reduction of CPU usage - previously serializing large results was a CPU
  intensive time since it included tons of conversions and transformations back and forth.

  The new format is also around 10-20% more storage efficient which means that it should allows
  for larger model values (MongoDB document size limit is 16 MB).

  The actual change should be fully opaque and transparent to the end users - it's purely a
  field storage implementation detail and the code takes care of automatically handling both
  formats when working with those object.

  Same field data storage optimizations have also been applied to workflow related database models
  which should result in the same performance improvements for Orquesta workflows which pass larger
  data sets / execution results around.

  Trigger instance payload field has also been updated to use this new field type which should
  result in lower CPU utilization and better throughput of rules engine service when working with
  triggers with larger payloads.

  This should address a long standing issue where StackStorm was reported to be slow and CPU
  inefficient with handling large executions.

  If you want to migrate existing database objects to utilize the new type, you can use
  st2common/bin/migrations/v3.5/st2-migrate-db-dict-field-values migration
  script. (improvement) #4846

  Contributed by @Kami.

• Add new result_size field to the ActionExecutionDB model. This field will only be
  populated for executions which utilize new field storage format.

  It holds the size of serialzed execution result field in bytes. This field will allow us to
  implement more efficient execution result retrieval and provide better UX since we will be
  able to avoid loading execution results in the WebUI for executions with very large results
  (which cause browser to freeze). (improvement) #4846

  Contributed by @Kami.

• Add new /v1/executions/<id>/result[?download=1&compress=1&pretty_format=1] API endpoint
  which can be used used to retrieve or download raw execution result as (compressed) JSON file.

  This endpoint will primarily be used by st2web when executions produce very large results so
  we can avoid loading, parsing and formatting those very large results as JSON in the browser
  which freezes the browser window / tab. (improvement) #4846

  Contributed by @Kami.

• Update jinja2 dependency to the latest stable version (2.11.3). #5195

• Update pyyaml dependency to the latest stable version (5.4). #5207

• Update various dependencies to latest stable versions (bcrypt, appscheduler, pytz,
  python-dateutil, psutil, passlib, gunicorn, flex, cryptography.
  eventlet, ``gr...

v3.4.1

https://stackstorm.com/2021/03/10/stackstorm-v3-4-1-security-fix/

Added

• Update the service start up code to warn if the service locale encoding is not set to utf-8 #5184
  Contributed by @Kami

Changed

• Use sudo -E to fix GitHub Actions tests #5187
  Contributed by @cognifloyd
• Properly handle unicode strings in logs #5184
  Contributed by @Kami

v3.4.0

https://stackstorm.com/2021/03/04/v3-4-0-released/

Added

• Added support for GitLab SSH URLs on pack install and download actions. (improvement) #5050
  Contributed by @asthLucas

• Added st2-rbac-backend pip requirements for RBAC integration. (new feature) #5086
  Contributed by @hnanchahal

• Added notification support for err-stackstorm. (new feature) #5051

• Added st2-auth-ldap pip requirements for LDAP auth integartion. (new feature) #5082
  Contributed by @hnanchahal

• Added --register-recreate-virtualenvs flag to st2ctl reload to recreate virtualenvs from scratch. (part of upgrade instructions) [#5167]
  Contributed by @winem and @blag

Changed

• Updated deprecation warning for python 2 pack installs, following python 2 support removal. #5099
  Contributed by @amanda11

• Improve the st2-self-check script to echo to stderr and exit if it isn't run with a
  ST2_AUTH_TOKEN or ST2_API_KEY environment variable. (improvement) #5068

• Added timeout parameter for packs.install action to help with long running installs that exceed the
  default timeout of 600 sec which is defined by the python_script action runner (improvement) #5084

  Contributed by @hnanchahal

• Upgraded cryptography version to 3.2 to avoid CVE-2020-25659 (security) #5095

• Converted most CI jobs from Travis to GitHub Actions (all except Integration tests).

  Contributed by @nmaludy, @winem, and @blag

• Updated cryptography dependency to version 3.3.2 to avoid CVE-2020-36242 (security) #5151

Fixed

• Pin chardet version as newest version was incompatible with pinned requests version #5101
  Contributed by @amanda11

• Fixed issue were st2tests was not getting installed using pip because no version was specified.
  Contributed by @anirudhbagri

• Added monkey patch fix to st2stream to enable it to work with mongodb via SSL. (bug fix) #5078 #5091

• Fix nginx buffering long polling stream to client. Instead of waiting for closed connection
  wait for final event to be sent to client. (bug fix) #4842 #5042

  Contributed by @guzzijones

• StackStorm now explicitly decodes pack files as utf-8 instead of implicitly as ascii (bug fix)
  #5106, #5107

• Fix incorrect array parameter value casting when executing action via chatops or using
  POST /aliasexecution/match_and_execute API endpoint. The code would incorrectly assume the
  value is always a string, but that may not be the cast - they value could already be a list and
  in this case we don't want any casting to be performed. (bug fix) #5141

  Contributed by @Kami.

• Fix @parameter_name=/path/to/file/foo.json notation in the st2 run command which didn't
  work correctly because it didn't convert read bytes to string / unicode type. (bug fix) #5140

  Contributed by @Kami.

• Fix broken st2 action-alias execute command and make sure it works
  correctly. (bug fix) #5138

  Contributed by @Kami.

Removed

• Removed --python3 pack install option #5100
  Contributed by @amanda11

• Removed submit-debug-info tool and the st2debug component #5103

• Removed check-licence script (cleanup) #5092

  Contributed by @kroustou

• Updated Makefile and CI to use Python 3 only, removing Python 2 (cleanup) #5090

  Contributed by @blag

• Remove st2resultstracker from st2ctl, the development environment and the st2actions setup.py (cleanup) #5108

  Contributed by @winem

v3.3.0

https://stackstorm.com/2020/10/22/stackstorm-v3-3-0-released/

Added

• Add make command to autogen JSON schema from the models of action, rule, etc. Add check
  to ensure update to the models require schema to be regenerated. (new feature)

• Improved st2sensor service logging message when a sensor will not be loaded when assigned to a
  different partition (@punkrokk) #4991

• Add support for a configurable connect timeout for SSH connections as requested in #4715
  by adding the new configuration parameter ssh_connect_timeout to the ssh_runner
  group in st2.conf. (new feature) #4914

  This option was requested by Harry Lee (@tclh123) and contributed by Marcel Weinberg (@winem).

• Added a FAQ for the default user/pass for the tools/launch_dev.sh script and print out the
  default pass to screen when the script completes. (improvement) #5013

  Contributed by @punkrokk

• Added deprecation warning if attempt to install or download a pack that only supports
  Python 2. (new feature) #5037

  Contributed by @amanda11

• Added deprecation warning to each StackStorm service log, if service is running with
  Python 2. (new feature) #5043

  Contributed by @amanda11

• Added deprecation warning to st2ctl, if st2 python version is Python 2. (new feature) #5044

  Contributed by @amanda11

Changed

• Switch to MongoDB 4.0 as the default version starting with all supported OS's in st2
  v3.3.0 (improvement) #4972

  Contributed by @punkrokk

• Added an enhancement where ST2api.log no longer reports the entire traceback when trying to get a datastore value
  that does not exist. It now reports a simplified log for cleaner reading. Addresses and Fixes #4979. (improvement) #4981

  Contributed by Justin Sostre (@saucetray)

• The built-in st2.action.file_writen trigger has been renamed to st2.action.file_written
  to fix the typo (bug fix) #4992

• Renamed reference to the RBAC backend/plugin from enterprise to default. Updated st2api
  validation to use the new value when checking RBAC configuration. Removed other references to
  enterprise for RBAC related contents. (improvement)

• Remove authentication headers St2-Api-Key, X-Auth-Token and Cookie from webhook payloads to
  prevent them from being stored in the database. (security bug fix) #4983

  Contributed by @potato and @knagy

• Updated orquesta to version v1.2.0.

Fixed

• Fixed a bug where type attribute was missing for netstat action in linux pack. Fixes #4946

  Reported by @scguoi and contributed by Sheshagiri (@Sheshagiri)

• Fixed a bug where persisting Orquesta to the MongoDB database returned an error
  message: key 'myvar.with.period' must not contain '.'. This happened anytime an
  input, output, publish or context var contained a key with a . within
  the name (such as with hostnames and IP addresses). This was a regression introduced by
  trying to improve performance. Fixing this bug means we are sacrificing performance of
  serialization/deserialization in favor of correctness for persisting workflows and
  their state to the MongoDB database. (bug fix) #4932

  Contributed by Nick Maludy (@nmaludy Encore Technologies)

• Fix a bug where passing an empty list to a with items task in a subworkflow causes
  the parent workflow to be stuck in running status. (bug fix) #4954

• Fixed a bug in the example nginx HA template declared headers twice (bug fix) #4966
  Contributed by @punkrokk

• Fixed a bug in the paramiko_ssh runner where SSH sockets were not getting cleaned
  up correctly, specifically when specifying a bastion host / jump box. (bug fix) #4973

  Contributed by Nick Maludy (@nmaludy Encore Technologies)

• Fixed a bytes/string encoding bug in the linux.dig action so it should work on Python 3
  (bug fix) #4993

• Fixed a bug where a python3 sensor using ssl needs to be monkey patched earlier. See also #4832, #4975 and gevent/gevent#1016 (bug fix) #4976

  Contributed by @punkrokk

• Fixed bug where action information in RuleDB object was not being parsed properly
  because mongoengine EmbeddedDocument objects were added to JSON_UNFRIENDLY_TYPES and skipped.
  Removed this and added if to use to_json method so that mongoengine EmbeddedDocument
  are parsed properly.

  Contributed by Bradley Bishop (@bishopbm1 Encore Technologies)

• Fix a regression when updated dnspython pip dependency resulted in
  st2 services unable to connect to mongodb remote host (bug fix) #4997

• Fixed a regression in the linux.dig action on Python 3. (bug fix) #4993

  Contributed by @blag

• Fixed a bug in pack installation logging code where unicode strings were not being
  interpolated properly. (bug fix)

  Contributed by @misterpah

• Fixed a compatibility issue with the latest version of the logging library API
  where the find_caller() function introduced some new variables. (bug fix) #4923

  Contributed by @Dahfizz9897

• Fixed another logging compatibility issue with the logging API in Python 3.
  The return from the logging.findCaller() implementation now expects a 4-element
  tuple. Also, in Python 3 there are new arguments that are passed in and needs to be
  acted upon, specificall stack_info that determines the new 4th element in the returned
  tuple. (bug fix) #5057

  Contributed by Nick Maludy (@nmaludy Encore Technologies)

Removed

• Removed Mistral workflow engine (deprecation) #5011

  Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

• Removed CentOS 6/RHEL 6 support #4984

  Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

• Removed our fork of codecov-python for CI and have switched back to the upstream version (improvement) #5002

v3.2.0

https://stackstorm.com/2020/04/30/stackstorm-v3-2-0-released/

Added

• Add support for blacklisting / whitelisting hosts to the HTTP runner by adding new
  url_hosts_blacklist and url_hosts_whitelist runner attribute. (new feature)
  #4757
• Add user parameter to re_run method of st2client. #4785
• Install pack dependencies automatically. #4769
• Add support for immutable_parameters on Action Aliases. This feature allows default
  parameters to be supplied to the action on every execution of the alias. #4786
• Add get_entrypoint() method to ActionResourceManager attribute of st2client.
  #4791
• Add support for orquesta task retry. (new feature)
• Add config option scheduler.execution_scheduling_timeout_threshold_min to better control the cleanup of scheduled actions that were orphaned. #4886

Changed

• Install pack with the latest tag version if it exists when branch is not specialized.
  (improvement) #4743

• Implement "continue" engine command to orquesta workflow. (improvement) #4740

• Update various internal dependencies to latest stable versions (apscheduler, eventlet,
  kombu, amqp, pyyaml, mongoengine, python-gnupg, paramiko, tooz, webob, bcrypt).

  Latest version of mongoengine should show some performance improvements (5-20%) when
  writing very large executions (executions with large results) to the database. #4767

• Improved development instructions in requirements.txt and dist_utils.py comment headers
  (improvement) #4774

• Add new actionrunner.stream_output_buffer_size config option and default it to -1
  (previously default value was 0). This should result in a better performance and smaller
  CPU utilization for Python runner actions which produce a lot of output.
  (improvement)

  Reported and contributed by Joshua Meyer (@jdmeyer3) #4803

• Add new action_runner.pip_opts st2.conf config option which allows user to specify a list
  of command line option which are passed to pip install command when installing pack
  dependencies into a pack specific virtual environment. #4792

• Refactor how orquesta handles individual item result for with items task. Before the fix,
  when there are a lot of items and/or result size for each item is huge, there is a negative
  performance impact on write to the database when recording the conductor state. (improvement)

• Remove automatic rendering of workflow output when updating task state for orquesta workflows.
  This caused workflow output to render incorrectly in certain use case. The render_workflow_output
  function must be called separately. (improvement)

• Update various internal dependencies to latest stable versions (cryptography, jinja2, requests,
  apscheduler, eventlet, amqp, kombu, semver, six) #4819 (improvement)

• Improve MongoDB connection timeout related code. Connection and server selection timeout is now
  set to 3 seconds. Previously a default value of 30 seconds was used which means that for many
  connection related errors, our code would first wait for this timeout to be reached (30 seconds)
  before returning error to the end user. #4834

• Upgrade pymongo to the latest stable version (3.10.0.). #4835 (improvement)

• Updated Paramiko to v2.7.1 to support new PEM ECDSA key formats #4901 (improvement)

• Remove .scrutinizer.yml config file. No longer used.

• Convert escaped dict and dynamic fields in workflow db models to normal dict and dynamic fields.
  (performnce improvement)

• Add support for PEP 508 <https://www.python.org/dev/peps/pep-0508/stackstorm/st2#environment-markers>_
  environment markers in generated requirements.txt files. (improvement) #4895

• Use pip-compile from pip-tools instead of pip-conflict-checker (improvement) #4896

• Refactor how inbound criteria for join task in orquesta workflow is evaluated to count by
  task completion instead of task transition. (improvement)

• The workflow engine orquesta is updated to v1.1.0 for the st2 v3.2 release. The version upgrade
  contains various new features and bug fixes. Please review the release notes for the full list of
  changes at https://github.com/StackStorm/orquesta/releases/tag/v1.1.0 and the st2 upgrade notes
  for potential impact. (improvement)

Fixed

• Fix the action query when filtering tags. The old implementation returned actions which have the
  provided name as action name and not as tag name. (bug fix) #4828

  Reported by @angrydeveloper and contributed by Marcel Weinberg (@winem)

• Fix the passing of arrays to shell scripts where the arrays where not detected as such by the
  st2 action_db utility. This caused arrays to be passed as Python lists serialized into a string.

  Reported by @kingsleyadam #4804 and contributed by Marcel Weinberg (@winem) #4861

• Fix ssh zombies when using ProxyCommand from ssh config #4881 [Eric Edgar]

• Fix rbac with execution view where the rbac is unable to verify the pack or uid of the execution
  because it was not returned from the action execution db. This would result in an internal server
  error when trying to view the results of a single execution.
  Contributed by Joshua Meyer (@jdmeyer3) #4758

• Fixed logging middleware to output a content_length of 0 instead of Infinity
  when the type of data being returned is not supported. Previously, when the value was
  set to Infinity this would result in invalid JSON being output into structured
  logs. (bug fix) #4722

  Contributed by Nick Maludy (@nmaludy Encore Technologies)

• Fix the workflow execution cancelation to proceed even if the workflow execution is not found or
  completed. (bug fix) #4735

• Added better error handling to contrib/linux/actions/dig.py to inform if dig is not installed.
  Contributed by JP Bourget (@punkrokk Syncurity) #4732

• Update dist_utils module which is bundled with st2client and other Python packages so it
  doesn't depend on internal pip API and so it works with latest pip version. (bug fix) #4750

• Fix dependency conflicts in pack CI runs: downgrade requests dependency back to 0.21.0, update
  internal dependencies and test expectations (amqp, pyyaml, prance, six) (bugfix) #4774

• Fix secrets masking in action parameters section defined inside the rule when using
  GET /v1/rules and GET /v1/rules/<ref> API endpoint. (bug fix) #4788 #4807

  Contributed by @Nicodemos305 and @jeansfelix

• Fix a bug with authentication API endpoint (POST /auth/v1/tokens) returning internal
  server error when running under gunicorn and whenauth.api_url config option was not set.
  (bug fix) #4809

  Reported by @guzzijones

• Fixed st2 execution get and st2 run not printing the action.ref for non-workflow
  actions. (bug fix) #4739

  Contributed by Nick Maludy (@nmaludy Encore Technologies)

• Update st2 execution get command to always include context.user, start_timestamp and
  end_timestamp attributes. (improvement) #4739

• Fixed core.sendmail base64 encoding of longer subject lines (bug fix) #4795

  Contributed by @stevemuskiewicz and @guzzijones

• Update all the various rule criteria comparison operators which also work with strings (equals,
  icontains, nequals, etc.) to work correctly on Python 3 deployments if one of the operators is
  of a type bytes and the other is of a type unicode / string. (bug fix) #4831

• Fix SSL connection support for MongoDB and RabbitMQ which wouldn't work under Python 3 and would
  result in cryptic "maximum recursion depth exceeded while calling a Python object" error on
  connection failure.

  NOTE: This issue only affected installations using Python 3. (bug fix) #4832 #4834

  Reported by @alexku7.

• Fix the amqp connection setup for WorkflowExecutionHandler to pass SSL params. (bug fix) #4845

  Contributed by Tatsuma Matsuki (@mtatsuma)

• Fix dependency conflicts by updating requests (2.23.0) and gitpython (2.1.15). #4869

• Fix orquesta syntax error for with items task where action is misindented or missing. (bug fix)
  PR StackStorm/orquesta#195.

• Fix orquesta yaql/jinja vars extraction to ignore methods of base ctx() dict. (bug fix)
  PR StackStorm/orquesta#196. Fixes #4866.

• Fix parsing of array of dicts in YAQL functions. Fix regression in YAQL/Jinja conversion
  functions as a result of the change. (bug fix) PR StackStorm/orquesta#191.

  Contributed by Hiroyasu Ohyama (@userlocalhost)

Removed

• Removed Ubuntu 14.04 from test matrix #4897

v3.1.0

Changed

• Allow the orquesta st2kv function to return default for nonexistent key. (improvement) #4678
• Update requests library to latest version (2.22.0) in requirements. (improvement) #4680
• Disallow "decrypt_kv" filter to be specified in the config for values that are marked as
  "secret: True" in the schema. (improvement) #4709
• Upgrade tooz library to latest stable version (1.65.0) so it uses latest version of
  grpcio library. (improvement) #4713
• Update st2-pack-install and st2-pack-download CLI command so it supports installing
  packs from local directories which are not git repositories. (improvement) #4713

Fixed

• Fix orquesta st2kv to return empty string and null values. (bug fix) #4678
• Allow tasks defined in the same task transition with fail to run for orquesta. (bug fix)
• Fix workflow service to handle unexpected coordinator and database errors. (bug fix) #4704 #4705
• Fix filter to_yaml_string to handle mongoengine base types for dict and list. (bug fix) #4700
• Fix timeout handling in the Python runner. In some scenarios where action would time out before
  producing any output (stdout, stder), timeout was not correctly propagated to the user. (bug fix)
  #4713
• Update st2common/setup.py file so it correctly declares all the dependencies and script
  files it provides. This way st2-pack-* commands can be used in a standalone fashion just by
  installing st2common Python package and nothing else. (bug fix) #4713
• Fix st2-pack-download command so it works in the environments where sudo binary is not
  available (e.g. Docker). (bug fix) #4713