Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions advanced-and-troubleshooting/advanced/add-operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,20 @@ description: >-
# Adding Operators

{% hint style="warning" %}
This is an experimental feature and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

You can add operators to your cluster using the `charon alpha edit add-operators` command. This operation keeps all validator public keys unchanged whilst adding new operators to the cluster.
You can add operators to your cluster using the `charon alpha edit add-operators` command. This operation keeps all distributed validator public keys unchanged whilst adding new operators to the cluster.

## Prerequisites

1. Review the `edit add-operators` command [CLI reference](../../learn/charon/charon-cli-reference.md#add-operators-to-a-cluster).
2. **For existing operators**: Keep the DV node running during the process and ensure you have a copy of the current cluster lock file and validator private key shares.
2. **For existing operators**: Keep the DV node running during the process and ensure you have a copy of the current [cluster lock file](../../learn/charon/cluster-configuration#cluster-lock-file) and validator private key shares.
3. **For new operators**: Obtain a copy of the existing cluster lock file from the existing operators and have your Charon ENR private key file ready.
4. Obtain the Charon ENR addresses of all new operators being added to the cluster.

{% hint style="info" %}
The ceremony uses a different p2p relay from your running cluster to avoid conflicts. The default relay address is already configured differently, so no special action is required.
The command uses a different set of p2p-relays to `charon run` to avoid conflicts with your running cluster.
{% endhint %}

## Adding Operators Process
Expand Down Expand Up @@ -62,31 +62,37 @@ The example below is designed for the [CDVN repository](https://github.com/ObolN
1. To start using the new configuration (with the added operators), stop the current Charon and validator client instances:

```bash
# Stop the containers
docker compose stop charon lodestar
```

2. Back up and remove the existing `.charon` directory, then move the `output` directory to `.charon`:

```bash
# Put the original artifacts in a backup location
mv .charon .charon-backup
# Copy the output from the add-operators command into the location of the original files
mv output .charon
```

3. Restart the Charon and validator client instances:

```bash
# Restart charon and the validator client with the new data
docker compose up -d charon lodestar
```

Lodestar's boot script (`lodestar/run.sh`) will automatically import all keys, removing any existing keys and cache. Charon will load the new `cluster-lock.json` and recognise all validators in the cluster with the updated operator set.

{% hint style="warning" %}
All existing operators must fully shut down their cluster nodes before starting with the new configuration. The old cluster must be completely stopped before the new cluster with the expanded operator set can begin operating. Unlike add-validators, this is not a gradual migration.

It is advisable to shut the cluster down for at least two epochs, to minimise any risk of unintended double signing during the controlled restart.
{% endhint %}

## Current Limitations
## Current Considerations

- The new cluster configuration will not be reflected on the Launchpad.
- The new cluster configuration will not yet be reflected on the Obol Launchpad.
- The new cluster configuration will have a new cluster hash, so the observability stack will display new cluster data under a different identifier.
- All operators (both existing and new) must participate in the add-operators ceremony for it to complete successfully.
- The cluster's threshold value remains unchanged after adding operators because the existing set of operators already possesses enough shares to create full signatures.
18 changes: 11 additions & 7 deletions advanced-and-troubleshooting/advanced/add-validators.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,18 @@ description: >-
# Adding Validators

{% hint style="warning" %}
This is an experimental feature available starting with Charon v1.6 and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

You can add validators to your cluster using the `charon alpha edit add-validators` command. The example below is designed for the default configuration provided by this repository and assumes that the stack uses the Lodestar validator client.
You can add validators to your cluster using the `charon alpha edit add-validators` command. The example below is designed for the [CDVN repository](https://github.com/ObolNetwork/charon-distributed-validator-node) and assumes a Lodestar validator, but the process is similar for other setups.

## Prerequisites

1. Review the `edit add-validators` command [CLI reference](../../learn/charon/charon-cli-reference.md#add-validators-to-a-cluster).
2. Keep the DV node running during the process and ensure you have a copy of the current cluster lock file and validator private key shares (or use `--unverified` if validator keys are not accessible).

{% hint style="info" %}
The ceremony uses a different p2p relay from your running cluster to avoid conflicts. The default relay address is already configured differently, so no special action is required.
The command uses a different set of p2p-relays to `charon run` to avoid conflicts with your running cluster.
{% endhint %}

## Adding Validators Process
Expand All @@ -42,19 +42,23 @@ The example below is designed for the [CDVN repository](https://github.com/ObolN
1. To start using the new configuration (with the added validators), stop the current Charon and validator client instances:

```bash
# Stop the containers
docker compose stop charon lodestar
```

2. Back up and remove the existing `.charon` directory, then move the `output` directory to `.charon`:

```bash
# Put the original artifacts in a backup location
mv .charon .charon-backup
# Copy the output from the add-validators command into the location of the original files
mv output .charon
```

3. Restart the Charon and validator client instances:

```bash
# Restart charon and the validator client with the new data
docker compose up -d charon lodestar
```

Expand All @@ -64,9 +68,9 @@ Lodestar's boot script (`lodestar/run.sh`) will automatically import all keys, r
Steps 1–3 must be performed independently by all node operators, likely at different times. During this process, some nodes will use the old configuration and others the new one. Once the number of upgraded nodes reaches the BFT threshold, the newly added validators will begin participating in the cluster.
{% endhint %}

## Current Limitations
## Current Considerations

- The new cluster configuration will not be reflected on the Launchpad.
- The new cluster configuration will have a new cluster hash, so the observability stack will display new cluster data under a different identifier.
- If Charon has no access to the existing validator keys (for example, if they're stored in a remote KeyManager), you must use the `--unverified` flag. This flag allows the addition to proceed but skips hashing and signing the new cluster lock data. When using this flag, you must start `charon run` with the `--no-verify` flag or set the `CHARON_NO_VERIFY=true` environment variable.
- The new cluster configuration will not be reflected on the Obol Launchpad.
- The new cluster configuration will have a new cluster hash, so the observability stack will display new cluster under a different identifier.
- If Charon has no access to the existing validator keys (for example, if they're stored in a remote KeyManager), you must use the `--unverified` flag. This flag allows the addition to proceed but skips hashing and signing the new cluster lock data. However when using cluster artifacts created with this flag, you must start `charon run` with the `--no-verify` flag or set the `CHARON_NO_VERIFY=true` environment variable.
- If you use different validator clients, review the keys import script. The old keys in `.charon/validator_keys` remain unchanged, so verify that importing the same keys will not disrupt the validator client's state.
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: >-
# Recreating Private Keys

{% hint style="warning" %}
This is an experimental feature and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

You can recreate the private key shares for your cluster using the `charon alpha edit recreate-private-keys` command. This operation creates new private key shares to replace the existing validator private keys whilst retaining the same operator identities and validator public keys.
Expand Down
2 changes: 1 addition & 1 deletion advanced-and-troubleshooting/advanced/remove-operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: >-
# Removing Operators

{% hint style="warning" %}
This is an experimental feature and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

You can remove operators from your cluster using the `charon alpha edit remove-operators` command. This operation leaves all validators intact whilst removing specified operators from the cluster.
Expand Down
2 changes: 1 addition & 1 deletion advanced-and-troubleshooting/advanced/replace-operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: >-
# Replacing an Operator

{% hint style="warning" %}
This is an experimental feature and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

You can replace an operator in your cluster using the `charon alpha edit replace-operator` command. This operation keeps all validators intact whilst swapping one operator for another in the cluster.
Expand Down
10 changes: 5 additions & 5 deletions learn/charon/charon-cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -605,7 +605,7 @@ Use "charon alpha edit [command] --help" for more information about a command.
The `charon alpha edit add-validators` command allows you to generate new validators and add them to an existing cluster. This process is very similar to the `charon dkg` ceremony, which requires all node operators to participate, because under the hood it runs the same DKG protocol with additional actions and verifications.

{% hint style="warning" %}
It is not yet recommended to use this command for Mainnet clusters.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

```markdown
Expand Down Expand Up @@ -648,7 +648,7 @@ Flags:
The `charon alpha edit add-operators` command adds new operators to an existing distributed validator cluster whilst keeping all validator public keys unchanged. All existing operators and new operators must participate in this ceremony.

{% hint style="warning" %}
This is an experimental feature and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

```markdown
Expand Down Expand Up @@ -686,7 +686,7 @@ Flags:
The `charon alpha edit remove-operators` command removes operators from an existing distributed validator cluster whilst leaving all validators intact. Remaining operators must participate in this ceremony.

{% hint style="warning" %}
This is an experimental feature and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

```markdown
Expand Down Expand Up @@ -726,7 +726,7 @@ Flags:
The `charon alpha edit recreate-private-keys` command creates new private key shares to replace the existing validator private keys whilst retaining the same operator identities and validator public keys. All operators must participate in this ceremony.

{% hint style="warning" %}
This is an experimental feature and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

```markdown
Expand Down Expand Up @@ -763,7 +763,7 @@ Flags:
The `charon alpha edit replace-operator` command replaces an existing operator in a distributed validator cluster with a new operator, whilst keeping all validators intact. All remaining and new operators must participate in this ceremony.

{% hint style="warning" %}
This is an experimental feature and should not be used in production (Mainnet) yet.
This is an alpha feature and is not yet recommended for production use.
{% endhint %}

```markdown
Expand Down