Feature: Support offline consistent backups

[tamalsaha]

User Story
As a user, I want to take consistent backup of my workloads even if that means I have to take it offline. Examples are: nightly back up of database of Confluence, sentry etc. Some of these applications are deployed via Helm. So, user has no control over how the DB is provisioned.

[tamalsaha]

Design Ideas
Here are some rough ideas on how this can be done:

• We add a new stash backup command that takes one off backup and exits. This should also use leader election to deal with replicas > 1
• Instead of adding stash sidecar, we add stash as init containers
• Then we create a CronJob. This will run on schedule and delete the pods of selected deployments. This will cause them to restart and run the init container (taking backup).
  • https://github.com/appscode/Dockerfiles/pull/11
  • https://github.com/johanhaleby/kubetail/blob/master/kubetail#L178:1
• Using CronJob will mean, we only support Kube 1.8+

[tamalsaha]

type BackupType string
const (
  BackupOnline BackupType = "online" // default, injects sidecar
  BackupOffline BackupType = "offline" // injects init cotainer
)

ResticSpec {
 Type BackupType `json:"type"`
}

[tamalsaha]

• Launch a post restic check job
• Delete sub Job created for CronJob, Check Job, Recovery Job ?
• Setup RBAC for check jobs
• Use Discovery to detect kubectl version