feat: ✨ Enhanced Helm deployment with ingress management and solochain support (#115)

This PR enhances the Helm-based deployment system with several key
improvements organized into the following areas:

## New Features

### Ingress Management
- **Ingress per replica**: Added `ingress-per-replica` chart template
that automatically generates an ingress for each node replica, exposing
individual pod instances
- **Traefik integration**: Added chart values to deploy Traefik as the
ingress controller for local K8s clusters, enabling proper ingress
testing (requires adding names to `/etc/hosts`)

### Solochain Support
- **New relay chart**: Added dedicated Helm chart for Solochain relay
deployment
- **CLI integration**: Updated DataHaven CLI to deploy both Execution
and Solochain relayers

## Configuration Improvements

### Environment Structure
- **Modular configs**: Refactored environmental configurations from
single `values.yaml` files into separate component-specific overrides
for better organization

### Node Configuration
- **Base config updates**: Improved base configurations for bootnode and
validator nodes
- **Network protocol**: Reverted from litep2p back to libp2p to resolve
node communication issues on Stagenet
- **Archive node routing**: In Stagenet, relayers now connect to the
bootnode (configured as archive node) instead of validator nodes

## Storage & Deployment

### Persistent Storage
- **Relayer database**: Added support for persistent volumes to store
relayer databases instead of using ephemeral storage

### Deployment Scripts
- **Documentation cleanup**: Removed obsolete test deploy.sh script and
updated the deployment README with clearer instructions

These changes provide a more robust, scalable, and maintainable
deployment system for DataHaven infrastructure.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for deploying new relayer types: Solochain and Execution
relayers, with dedicated configuration and secret management.
* Introduced per-replica ingress configuration for node deployments,
allowing each replica to have its own ingress resource and hostname.
* Added persistent storage options for relay data, configurable via
storage path, class, and size.
* Added new deployment configuration files for local and stagenet
environments, including Traefik ingress controller setup.
* Introduced a new relay category, "Solochain Relayers," for standalone
chain operations and cross-chain communication.

* **Improvements**
* Updated deployment configurations to use container-specific
environment YAML files for more granular control.
* Enhanced relay and node configurations with new flags and backend
options, including dynamic peer ID generation and automatic bootnode
discovery.
* Updated relayer endpoints to consistently use the bootnode for
connections.
* Refined relay configuration files for improved structure, clarity, and
endpoint management.

* **Bug Fixes**
* Corrected deployment logic to reference the correct
environment-specific configuration files during Helm deployments.

* **Documentation**
* Simplified and updated deployment documentation to focus on CLI-based
deployment, removing outdated manual instructions and adding a concise
overview of components and relayer types.

* **Chores**
* Removed deprecated deployment scripts and outdated configuration files
to streamline the deployment process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude <[email protected]>

Author: stiiifff

deploy/README.md

@@ -32,15 +32,15 @@ deploy/
 
 ## Deployment
 
-To deploy to a specific environment:
+The recommended way to deploy is using the DataHaven CLI with the deploy command:
 
 ```bash
-./scripts/deploy.sh <environment>
+cd test && bun cli deploy --e <environment>
 ```
 
 Example:
 ```bash
-./scripts/deploy.sh local
+cd test && bun cli deploy --e local
 ```
 
 Available environments:
@@ -92,99 +92,4 @@ The deployment process:
   - **Beacon Relay**: Relays Ethereum beacon chain data
   - **BEEFY Relay**: Relays BEEFY consensus data for finality
   - **Execution Relay**: Relays Ethereum execution layer data
-
-## Development
-
-For local development:
-1. Ensure you have a local Kubernetes cluster (e.g., Minikube, Kind)
-2. Deploy using the local environment:
-   ```bash
-   ./scripts/deploy.sh local
-   ```
-
-## Troubleshooting
-
-Common issues and solutions:
-
-1. Namespace doesn't exist:
-   ```bash
-   kubectl create namespace kt-datahaven-<env>
-   ```
-
-2. Helm dependencies need updating:
-   ```bash
-   helm dependency update charts/node
-   helm dependency update charts/relay
-   ```
-
-3. View deployment status:
-   ```bash
-   kubectl get all -n kt-datahaven-<env>
-   ```
-
-4. Preview deployment changes:
-   ```bash
-   ./scripts/deploy.sh <env> true
-   ```
-
-## Manual Deployment (Advanced)
-
-For advanced users who want to deploy components individually or need more control:
-
-### DataHaven Bootnode & Validators
-
-#### Deploy individual components
-```bash
-cd deploy
-helm upgrade --install dh-bootnode charts/node -f charts/node/datahaven/dh-bootnode.yaml -f ../environments/<env>/values.yaml -n kt-datahaven-<env>
-helm upgrade --install dh-validator charts/node -f charts/node/datahaven/dh-validator.yaml -f ../environments/<env>/values.yaml -n kt-datahaven-<env>
-```
-
-#### Access validator node with Polkadot.js apps
-```bash
-kubectl port-forward svc/dh-validator-0 -n kt-datahaven-<env> 9955:9955
-# Then visit: https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9955#/explorer
-```
-
-#### Remove DataHaven components
-```bash
-helm uninstall dh-bootnode -n kt-datahaven-<env>
-helm uninstall dh-validator -n kt-datahaven-<env>
-```
-
-#### Cleanup volumes
-```bash
-kubectl delete pvc -l app.kubernetes.io/instance=dh-bootnode -n kt-datahaven-<env>
-kubectl delete pvc -l app.kubernetes.io/instance=dh-validator -n kt-datahaven-<env>
-```
-
-### Snowbridge Relayers
-
-#### Create required secrets
-```bash
-kubectl create secret generic dh-beefy-relay-ethereum-key --from-literal=pvk="<ETHEREUM_PRIVATE_KEY>" -n kt-datahaven-<env>
-kubectl create secret generic dh-beacon-relay-substrate-key --from-literal=pvk="<SUBSTRATE_PRIVATE_KEY>" -n kt-datahaven-<env>
-kubectl create secret generic dh-execution-relay-substrate-key --from-literal=pvk="<SUBSTRATE_PRIVATE_KEY>" -n kt-datahaven-<env>
-```
-
-#### Deploy individual relay components
-```bash
-cd deploy
-helm upgrade --install dh-beacon-relay charts/relay -f charts/relay/snowbridge/dh-beacon-relay.yaml -f ../environments/<env>/values.yaml -n kt-datahaven-<env>
-helm upgrade --install dh-beefy-relay charts/relay -f charts/relay/snowbridge/dh-beefy-relay.yaml -f ../environments/<env>/values.yaml -n kt-datahaven-<env>
-helm upgrade --install dh-execution-relay charts/relay -f charts/relay/snowbridge/dh-execution-relay.yaml -f ../environments/<env>/values.yaml -n kt-datahaven-<env>
-```
-
-#### Remove relay components
-```bash
-helm uninstall dh-beacon-relay -n kt-datahaven-<env>
-helm uninstall dh-beefy-relay -n kt-datahaven-<env>
-helm uninstall dh-execution-relay -n kt-datahaven-<env>
-```
-
-#### Delete relay secrets
-```bash
-kubectl delete secret dh-beefy-relay-ethereum-key -n kt-datahaven-<env>
-kubectl delete secret dh-beacon-relay-substrate-key -n kt-datahaven-<env>
-kubectl delete secret dh-execution-relay-substrate-key -n kt-datahaven-<env>
-```
+- **Solochain Relayers**: Handle standalone chain operations and cross-chain communication

deploy/charts/node/datahaven/dh-bootnode.yaml

@@ -12,24 +12,27 @@ imagePullSecrets:
 
 node:
   command: datahaven-node
-  # customChainspec: true # see extraInitContainers, chainspec-generator
+  customChainspec: true # see extraInitContainers, chainspec-generator
   role: full
   replicas: 1
   chainData:
-    pruning: archive
+    pruning: 1000
     storageClass: "gp2"
   chainKeystore:
     mountInMemory:
       enabled: true
-  # perNodeServices:
-  #   relayP2pService:
-  #     enabled: true
-  # to generate new key run: docker run moonsonglabs/datahaven key generate-node-key
-  # 12D3KooWRpzRTivvJ5ySvgbFnPeEE6rDhitQKL1fFJvvBGhnenSk
-  customNodeKey: 80c30ac6ba927c6e5c0c9681aa9674f1d181d180853bcd3485cee9d18e931238
+  persistGeneratedNodeKey: true
   flags:
     - "--allow-private-ipv4"
     - "--discover-local"
+    - "--network-backend libp2p"
+
+ingress:
+  enabled: false
+  perReplica: false
+  wildcardDomain: datahaven.local
+  # If enabled, this would generate:
+  # - dh-bootnode-0.datahaven.local
 
 # Generate chainspec, and expose it as url
 extraInitContainers:
@@ -43,15 +46,32 @@ extraInitContainers:
       - |
         apt update || true
         apt install -y jq
+        
+        # Wait for node key to be generated by the persist-generated-node-key init container
+        echo "Waiting for node key generation..."
+        for i in {1..60}; do
+          [ -f /keystore/node-key ] && break
+          echo "Node key not found, waiting ($i/60)…"
+          sleep 2
+        done
+        [ -f /keystore/node-key ] || { echo "Node key generation timed out"; exit 1; }
+
+        # Extract the peer ID from the generated node key
+        NODE_PEER_ID="$({{ .Values.node.command }} key inspect-node-key --file /keystore/node-key)"
+        echo "Using generated peer ID: ${NODE_PEER_ID}"
+        
+        # Generate chainspec with dynamic peer ID
         {{ .Values.node.command }} build-spec --chain {{ .Values.node.chain }} > base.json
-        echo '{"bootNodes":["/dns/dh-bootnode-0/tcp/30333/p2p/12D3KooWRpzRTivvJ5ySvgbFnPeEE6rDhitQKL1fFJvvBGhnenSk"]}' > override1.json
+        echo "{\"bootNodes\":[\"/dns/dh-bootnode-0/tcp/30333/p2p/${NODE_PEER_ID}\"]}" > override1.json
         jq  -s '.[0] * .[1]' base.json override1.json | sed 's/1e+18/1000000000000000000/' > plain.json
         cut -c -256 plain.json
         {{ .Values.node.command }} build-spec --chain plain.json --raw  > chainspec.json
         cp chainspec.json {{ .Values.node.customChainspecPath }}
     volumeMounts:
       - mountPath: /chain-data
         name: chain-data
+      - mountPath: /keystore
+        name: chain-keystore
 extraContainers:
   - name: chainspec
     image: nginxinc/nginx-unprivileged:stable

deploy/charts/node/datahaven/dh-validator.yaml

@@ -12,8 +12,8 @@ imagePullSecrets:
 
 node:
   command: datahaven-node
-  # customChainspecUrl: http://dh-bootnode:8080/chainspec.json
-  # forceDownloadChainspec: true
+  customChainspecUrl: http://dh-bootnode:8080/chainspec.json
+  forceDownloadChainspec: true
   role: authority
   replicas: 2
   chainData:
@@ -40,19 +40,19 @@ node:
       type: beef
       scheme: ecdsa
       extraDerivation: '$([ "${HOSTNAME##*-}" = "0" ] && echo "//Alice" || echo "//Bob")'
-  customNodeKey:
-    # To generate new key run: docker run --rm -t moonsonglabs/datahaven:latest key generate-node-key
-    # 12D3KooWL5Av1ZZSKkaittmxXBmZpzP7zgiB1AAnWHEw7MxzqnFp
-    - bdf71a910354e231095366230621eaefb5f99465045f1501478fd3d9b5deef98
-    # 12D3KooWAxFonTS177T81CTDeH6mfvJQWYEJeVQ1gPrnULjNY8Cn
-    - 2a775a9db9fb0ff40afacb4aa7ccbf2a5d04c6d980bb1437c196c8e38a6cd948
+  persistGeneratedNodeKey: true
   flags:
-    - "--bootnodes /dns/dh-bootnode-0/tcp/30333/p2p/12D3KooWRpzRTivvJ5ySvgbFnPeEE6rDhitQKL1fFJvvBGhnenSk"
+    - "--network-backend libp2p"
+    # Note: Bootnode discovery will happen automatically via the chainspec downloaded from customChainspecUrl
   enableOffchainIndexing: true
-  perNodeServices:
-    apiService:
-      enabled: true
-      type: NodePort
+
+ingress:
+  enabled: false
+  perReplica: true
+  wildcardDomain: datahaven.local
+  # If enabled, this would generate:
+  # - dh-validator-0.datahaven.local
+  # - dh-validator-1.datahaven.local
 
 extraInitContainers:
   - name: dump-state-and-wasm

deploy/charts/node/templates/ingress-per-replica.yaml

@@ -0,0 +1,50 @@
+{{- if and .Values.ingress.enabled .Values.ingress.perReplica -}}
+{{- $fullName := include "node.fullname" . -}}
+{{- $replicas := .Values.node.replicas | int -}}
+{{- range $i := until $replicas }}
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: {{ $fullName }}-{{ $i }}
+  labels:
+  {{- $labels := include "node.labels" $ | fromYaml }}
+  {{- range $key, $value := $labels }}
+    {{- if eq $key "app.kubernetes.io/instance" }}
+    {{ $key }}: {{ $value | replace $fullName (printf "%s-%d" $fullName $i) | quote }}
+    {{- else }}
+    {{ $key }}: {{ $value | quote }}
+    {{- end }}
+  {{- end }}
+    replica: "{{ $i }}"
+  {{- with $.Values.ingress.annotations }}
+  annotations:
+  {{- range $key, $value := . }}
+    {{- if eq $key "meta.helm.sh/release-name" }}
+    {{ $key }}: {{ $value | quote }}
+    {{- else }}
+    {{ $key }}: {{ $value | replace $fullName (printf "%s-%d" $fullName $i) | quote }}
+    {{- end }}
+  {{- end }}
+  {{- end }}
+spec:
+  {{- if $.Values.ingress.ingressClassName }}
+  ingressClassName: {{ $.Values.ingress.ingressClassName | quote }}
+  {{- end }}
+  rules:
+    - host: {{ printf "%s-%d.%s" $fullName $i $.Values.ingress.wildcardDomain | quote }}
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: {{ $fullName }}-{{ $i }}
+                port:
+                  number: {{ $.Values.node.perNodeServices.apiService.rpcPort }}
+  {{- with $.Values.ingress.tls }}
+  tls:
+  {{- toYaml . | nindent 4 }}
+  {{- end }}
+{{- end }}
+{{- end }}

deploy/charts/node/templates/ingress.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.ingress.enabled -}}
+{{- if and .Values.ingress.enabled (not .Values.ingress.perReplica) -}}
   {{- $fullName := include "node.fullname" . -}}
   {{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
 apiVersion: networking.k8s.io/v1

deploy/charts/relay/configs/beacon-relay.json

@@ -13,14 +13,14 @@
                 }
             },
             "datastore": {
-                "location": "/data",
+                "location": "/relay-data",
                 "maxEntries": 100
             }
         }
     },
     "sink": {
         "parachain": {
-            "endpoint": "ws://dh-validator-0:9944",
+            "endpoint": "ws://dh-bootnode-0:9944",
             "maxWatchedExtrinsics": 8,
             "headerRedundancy": 20
         },

deploy/charts/relay/configs/beefy-relay.json

@@ -1,7 +1,7 @@
 {
     "source": {
         "polkadot": {
-            "endpoint": "ws://dh-validator-0:9944"
+            "endpoint": "ws://dh-bootnode-0:9944"
         }
     },
     "sink": {

deploy/charts/relay/configs/execution-relay.json

@@ -1,41 +1,40 @@
 {
-  "source": {
-    "ethereum": {
-      "endpoint": "ws://el-1-reth-lodestar:8546"
-    },
-    "contracts": {
-      "Gateway": "0x8f86403A4DE0BB5791fa46B8e795C547942fE4Cf"
+    "source": {
+        "ethereum": {
+            "endpoint": "ws://el-1-reth-lodestar:8546"
+        },
+        "contracts": {
+            "Gateway": "0x8f86403A4DE0BB5791fa46B8e795C547942fE4Cf"
+        },
+        "beacon": {
+            "endpoint": "http://cl-1-lodestar-reth:4000",
+            "stateEndpoint": "http://cl-1-lodestar-reth:4000",
+            "spec": {
+                "syncCommitteeSize": 512,
+                "slotsInEpoch": 32,
+                "epochsPerSyncCommitteePeriod": 256,
+                "forkVersions": {
+                    "deneb": 0,
+                    "electra": 0
+                }
+            },
+            "datastore": {
+                "location": "/relay-data",
+                "maxEntries": 100
+            }
+        }
     },
-    "channel-id": "",
-    "beacon": {
-      "endpoint": "http://cl-1-lodestar-reth:4000",
-      "stateEndpoint": "http://cl-1-lodestar-reth:4000",
-      "spec": {
-        "syncCommitteeSize": 512,
-        "slotsInEpoch": 32,
-        "epochsPerSyncCommitteePeriod": 256,
-        "forkVersions": {
-          "deneb": 0,
-          "electra": 0
+    "sink": {
+        "parachain": {
+            "endpoint": "ws://dh-bootnode-0:9944",
+            "maxWatchedExtrinsics": 8,
+            "headerRedundancy": 20
         }
-      },
-      "datastore": {
-        "location": "",
-        "maxEntries": 100
-      }
-    }
-  },
-  "sink": {
-    "parachain": {
-      "endpoint": "ws://dh-validator-0:9944",
-      "maxWatchedExtrinsics": 8,
-      "headerRedundancy": 20
+    },
+    "instantVerification": false,
+    "schedule": {
+        "id": null,
+        "totalRelayerCount": 1,
+        "sleepInterval": 1
     }
-  },
-  "instantVerification": false,
-  "schedule": {
-    "id": null,
-    "totalRelayerCount": 1,
-    "sleepInterval": 1
-  }
-}
+}