Khepri 0.18.0

At this point, Khepri should be considered Beta. The API will likely evolve and future releases will likely introduce breaking changes.

What's new in Khepri 0.18.0

Support multi-table projections

Khepri 0.18.0 supports extended projections that need to manage several ETS tables. This is useful to maintain multiple views of the same set of records, or to organise a complex structured view of that set of records accross several tables to provide more efficient updates and lookups.

To declare a multi-table projection, you can provide the list of tables along with their specific ETS options:

Options = #{tables => #{my_table_1 => #{type => set},
                        my_table_2 => #{type => bag}},
            read_concurrency => true},
khepri_projection:new(my_projection, fun extended_fun/4, Options).

Your projection function is then called with a map of table names/TIDs instead of a single TID:

extended_fun(Tables, Path, OldProps, NewProps) ->
    #{my_table_1 := Tid1,
      my_table_2 := Tid2} = Tables,
    ...

See #366.

Other changes

• Add an internal API to wait for a machine version or behaviour to be effective in a cluster (#374).
• Ra was bumped from Ra 2.16.12 to 3.1.2 (#340, #347, #364, #375, #378, #382).

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.18.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.18.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.18.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.17.7

At this point, Khepri should be considered Beta. The API will likely evolve and future releases will likely introduce breaking changes.

What's new in Khepri 0.17.7

The release uses Horus 0.4.0. This new version brings a lock-related bug fix as well as some preparation work for future changes (to make this version compatible with future versions of Horus).

It is a new minor version but it is fully compatible with the Horus 0.3.x series.

See #379 and #380.

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.17.7

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.17.7"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.17.7

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.17.6

At this point, Khepri should be considered Beta. The API will likely evolve and future releases will likely introduce breaking changes.

What's new in Khepri 0.17.6

The release comes with several non-critical bug fixes:

• Fix use of patterns when we want to handle several expired nodes in a single pass in remove_expired_nodes1/2 (#376).
• Handle exception from ra:trigger_election/2 when we expand a Khepri cluster; it was already the case when starting a Ra server (#369).
• Cache effective machine version after recovery (#373).
• Allow the use of ?LOG_*() macros in transactions (#377).

The remaining changes are related to docs, Dialyzer fixes or small no-op refactors:

• Refactor the code that needs to know the Ra leader when khepri_machine wants to remember if the latest command was local or remote (#361).
• Fix type specs (#363, #368).
• Use the correct guard to check if an argument is a store ID (#370).
• Fix typo if "maple" (#365).

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.17.6

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.17.6"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.17.6

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.17.5

At this point, Khepri should be considered Beta. The API will likely evolve and future releases will likely introduce breaking changes.

What's new in Khepri 0.17.5

A couple bug fixes:

• Send the #dedup_ack{} to the leader instead of the local Ra server if the leader is known (#355).
• Fix handling of deletions when triggering stored proceudres (#357).

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.17.5

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.17.5"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.17.5

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.17.4

At this point, Khepri should be considered Beta. The API will likely evolve and future releases will likely introduce breaking changes.

What's new in Khepri 0.17.4

Improved keep_while conditions

The focus of this release was put on keep_while conditions and in particular, bugs and performance issues. These were discovered while working on topic exchanges in RabbitMQ.

The first bug is about child nodes of a deleted parent nodes missing from the returned result. The second one was leaked entries in the keep_while reverse index in the machine state. Both were addressed by #352.

The performance issues were related to the way we squash version bumps after the handling of keep_while conditions, and the way we handled many tree nodes expiring as part of the application of one machine command. See #353.

Other changes

Horus was updated from 0.3.1 to 0.3.2 (#354).

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.17.4

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.17.4"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.17.4

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.17.3

At this point, Khepri should be considered Beta. The API will likely evolve and future releases will likely introduce breaking changes.

What's new in Khepri 0.17.3

Faster queries

When khepri_machine computes the list of default options, it verifies which machine version is currently effective, regardless of what it supports locally, because the overall cluster might use an older machine code.

This machine version check uses Ra counters, but they remain expensive to read. This is specially visible when querying a Khepri store. A significant part of the time it takes to process a query is spent in reading the Ra counters.

This version caches the effective machine version when the Khepri machine is started and when the machine version is upgraded using a persistent_term. This is way more efficient and improves the performance of queries. This improves the life of the machine itself, but the impact is less visible.

See #349.

Other changes

• Handle {error, normal} return value from Ra in khepri_machine:process_sync_command/3 (#350).
• Ignore keep_while condition at time a tree node is created when the keep_while targets a child node (#351).
• Address several warnings from Dialyzer (#345, #346).

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.17.3

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.17.3"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.17.3

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.17.2

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.17.2

Several small fixes:

• Handle {error, normal} return value from Ra in khepri_cluster:do_reset/4 (#335).
• Log an error if khepri_cluster:trigger_election/2 fails during start (#336).
• Protect concurrent updates to the ?PT_STORE_IDS persistent term with a lock (#337).
• Address several warnings from Dialyzer (#341).

Also, Ra was bumped from 2.16.7 to 2.16.12 (#333, #339).

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.17.2

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.17.2"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.17.2

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.17.1

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.17.1

Ra was bumped from 2.16.6 to 2.16.7. It fixes a bug with the handling of #info_reply{} that affects Khepri, as discovered by the khepri_mnesia_migration testsuites.

See #329.

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.17.1

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.17.1"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.17.1

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.17.0

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.17.0

Return tree nodes deteled by keep-while conditions in put and delete return values

All put and delete operations now return all nodes that were deleted by the operation. This includes nodes deleted by keep-while conditions as well as child tree nodes in the case of delete operations. See #305 and #309.

The deletion reason was added in the return tree node properties to allow the caller to distinguish nodes explicitly deleted from the nodes that were deleted as a consequence of a put or delete.

To permit that, the return values of advanced put and delete operations were made uniform with other operations, always returning a map of tree node properties (#303, #310). Here is an example of the change to make to your code:

• Up to Khepri 0.16.x:

  {ok, #{payload_version := _}} = khepri_adv:put(StoreId, Path, Value)

• Starting from Khepri 0.17.0:

  {ok, #{Path := #{payload_version := _}}} = khepri_adv:put(StoreId, Path, Value)

Backward-compatible transaction functions

The breaking change above impacted the khepri_tx_adv API as well and you may need to rework transactions to take into account the possible old and new return values. Indeed, transactions submitted from a Khepri member are executed on all cluster members, regardless of the version of Khepri running there. An old node might submit a transaction executed on a new node, and a new node might submit a transaction executed on an old node.

You can use the new khepri_tx:does_api_comply_with/1 to determine what to expect. Here is an example:

khepri:transaction(
  StoreId,
  fun () ->
          %% We query Khepri to determine how the API will behave.
          UsesUniformWriteRet = try
                                    khepri_tx:does_api_comply_with(uniform_write_ret)
                                catch
                                    error:undef ->
                                        false
                                end,

          %% We use the result to match the return value appropriately. This
          %% transaction function willl work equally on Khepri 0.17.0 and older
          %% versions.
          case khepri_tx_adv:delete(Path) of
              {ok, #{Path := #{data := _}}} when UsesUniformWriteRet ->
	          %% The API used the new return value format.
		  ...;
              {ok, #{data := _}} when not UsesUniformWriteRet ->
	          %% The API used the old return value format.
		  ...;
              {ok, _} ->
                  ok
          end
  end, rw).

See #326.

Optimize the lookup of matching keep_while conditions

Khepri uses a new khepri_prefix_tree internal structure instead of the previous map to manage the reverse index. This allows to quickly eliminate non-matching entries.

See #298.

Unify khepri_cluster:members/2 and khepri_cluster:nodes/2 with other query APIs

khepri_cluster:members/2 and khepri_cluster:nodes/2 now take a query options map (see the documentation of the khepri:query_options() type) instead of a timeout value only.

This allows to select between a leader query and a local query with the same favor option as any store query function. Therefore, favor => consistency will use a leader query while favor => low_latency will use a local query.

Here is an example of the change to make to your code:

• Up to Khepri 0.16.x:

  khepri_cluster:members(StoreId, Timeout),
  khepri_cluster:locally_known_nodes(StoreId)

• Starting from Khepri 0.17.0:

  khepri_cluster:members(StoreId, #{timeout => Timeout}),
  khepri_cluster:nodes(StoreId, #{favor => low_latency})

See #300.

Other changes

• Allow calls to khepri_path in transactions (#312).
• Make the internal khepri_tree:tree_v*() and khepri_tree:keep_while_conds_revidx_v*() types opaque types (#299).
• Handle expired dedups on tick to remove some load on commands (#297).
• Ignore unknown props to return to take machine breaking changes into account (#308).
• Fix bang function return value spec (#311).
• Fix documentation (#316, #321).
• Fix how we determine the need for a fence preliminary query (#317).
• Fix how we check for the dedup mechanism support (#320).
• Filter query/command options that are passed to transactions (#322, #327).
• Fix keep_while corner cases (#324).
• Horus was bumped from 0.3.0 to 0.3.1 (#307).
• Ra was bumped from 2.14.0 to 2.16.6 (#306, #314, #318, #323, #325).

Download

Khepri is available from Hex.pm: https://hex.pm/packages/khepri/0.17.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.17.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.17.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri is available on Hex.pm.

Khepri 0.16.0

At this point, Khepri should be considered Beta and not production ready. The API will likely evolve and future releases will likely introduce breaking changes. The goal of this release is to make it easy for anyone to try it and possibly give feedback.

What's new in Khepri 0.16.0

This release focuses on bug fixes and optimizations. It does not bring new features.

• Fix a bug in the khepri_cluster:reset() code that was caused by a race between the local node and the rest of the cluster (#294).
• Fix a bug when a projection is triggered in the same batch of aux effects as the projection deletion (#293).
• Stop spamming the cluster with join/leave commands if it is not ready (#295).
• Improve the performance of khepri_cluster:is_store_running() (#292, #296).

Download

Khepri 0.16.0 is available from Hex.pm: https://hex.pm/packages/khepri/0.16.0

Upgrade

Using Rebar:

1. Update your rebar.config:

   %% In rebar.config
   {deps, [{khepri, "0.16.0"}]}.

2. Run rebar3 upgrade khepri.

Using Erlang.mk:

1. Update your Makefile:

   %% In your Makefile
   dep_khepri = hex 0.16.0

2. Remove the deps/khepri directory. The new version will be fetched the next time you build your project.

Documentation

The documentation for Khepri 0.16.0 is available on Hex.pm.