Skip to content

AIDE migration #537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Vincent056 wants to merge 1 commit into from
Closed

AIDE migration #537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions Dockerfile.ci
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,33 @@ WORKDIR /go/src/github.com/openshift/file-integrity-operator

ENV GOFLAGS="-mod=vendor"

RUN dnf -y install git make gcc automake autoconf libtool flex gettext-devel e2fsprogs-devel audit-libs-devel libattr-devel flex bison zlib-devel libgcrypt-devel audit-libs-devel libacl-devel libselinux-devel libtool && dnf clean all

COPY . .

RUN git clone https://github.com/autoconf-archive/autoconf-archive.git && \
cp autoconf-archive/m4/*.m4 /usr/share/aclocal/
Copy link
Contributor

@rhmdnd rhmdnd Jul 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a dependency for building AIDE 0.18?

Copy link
Contributor Author

@Vincent056 Vincent056 Jul 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I think rhel might has rpm for this, but this builder does not


RUN make build

RUN make aide-0.18

# Step two: containerize file-integrity-operator and AIDE together
FROM registry.fedoraproject.org/fedora-minimal:37
RUN microdnf -y install aide-0.16
RUN microdnf -y install aide golang && microdnf clean all

# Install AIDE 0.16 and dependencies
RUN microdnf -y install aide-0.16 golang && microdnf clean all

ENV OPERATOR=/usr/local/bin/file-integrity-operator \
USER_UID=1001 \
USER_NAME=file-integrity-operator

RUN mkdir -p /usr/local/libs

# install operator binary
COPY --from=builder /go/src/github.com/openshift/file-integrity-operator/build/bin/manager ${OPERATOR}
COPY build/bin /usr/local/bin
COPY --from=builder /go/src/github.com/openshift/file-integrity-operator/build/bin/aide-0.18 /usr/sbin/aide-0.18
RUN /usr/local/bin/user_setup

ENTRYPOINT ["/usr/local/bin/entrypoint"]
Expand Down
25 changes: 25 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,31 @@ else
$(eval OPENSHIFT_USER = $(shell oc whoami))
endif


.PHONY: aide-0.18
aide-0.18:
cd build && \
rm -rf aide && \
git clone https://github.com/aide/aide.git && \
cd aide && \
Copy link
Contributor

@rhmdnd rhmdnd Jul 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a build directory where we stash other related tools like controller-gen and kustomize. We could keep the entire repository in there so it gets cleaned up using make clean. Otherwise, we might need to go in and clean it up manually if we encounter a build failure.

git checkout v0.18.8 && \
sh ./autogen.sh && \
./configure \
--with-zlib \
--disable-static \
--with-posix-acl \
--with-gcrypt \
--with-selinux \
--with-xattr \
--with-e2fsattrs \
--with-audit && \
$(MAKE) && \
chmod +x ./aide && \
cd .. && \
cd .. && \
cp ./build/aide/aide ./build/bin/aide-0.18


.PHONY: check-operator-version
check-operator-version:
ifndef VERSION
Expand Down
17 changes: 14 additions & 3 deletions build/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,38 @@
# Step one: build file-integrity-operator
FROM golang:1.22 as builder
FROM registry.ci.openshift.org/openshift/release:rhel-9-release-golang-1.22-openshift-4.17 AS builder
USER root

WORKDIR /go/src/github.com/openshift/file-integrity-operator

ENV GOFLAGS="-mod=vendor"

RUN dnf -y install git make gcc automake autoconf libtool flex gettext-devel e2fsprogs-devel audit-libs-devel libattr-devel flex bison zlib-devel libgcrypt-devel audit-libs-devel libacl-devel libselinux-devel libtool && dnf clean all

COPY . .

RUN git clone https://github.com/autoconf-archive/autoconf-archive.git && \
cp autoconf-archive/m4/*.m4 /usr/share/aclocal/

RUN make build

RUN make aide-0.18
Copy link
Contributor

@rhmdnd rhmdnd Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another alternative we talked about earlier was to build this using a separate ubi8 base image container and then copying it over, too.

Or, at least in this case, we could just copy it out of the fedora 40 image and see how that works.


# Step two: containerize file-integrity-operator and AIDE together
FROM registry.fedoraproject.org/fedora-minimal:37
RUN microdnf -y install aide-0.16
RUN microdnf -y install aide golang && microdnf clean all

# Install AIDE 0.16 and dependencies
RUN microdnf -y install aide-0.16 golang && microdnf clean all

ENV OPERATOR=/usr/local/bin/file-integrity-operator \
USER_UID=1001 \
USER_NAME=file-integrity-operator

RUN mkdir -p /usr/local/libs

# install operator binary
COPY --from=builder /go/src/github.com/openshift/file-integrity-operator/build/bin/manager ${OPERATOR}
COPY build/bin /usr/local/bin
COPY --from=builder /go/src/github.com/openshift/file-integrity-operator/build/bin/aide-0.18 /usr/sbin/aide-0.18
Copy link
Contributor

@rhmdnd rhmdnd Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then we don't need to pollute the builder container image with utilities and code unrelated to golang, or building FIO.

RUN /usr/local/bin/user_setup

ENTRYPOINT ["/usr/local/bin/entrypoint"]
Expand Down
34 changes: 22 additions & 12 deletions cmd/manager/daemon.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,13 @@ const (
aideReadDBFileName = "aide.db.gz"
aideReadLogFileName = "aide.log"
// Files output from AIDE init and scan log. We copy the files AIDE writes from here.
aideWritingDBFileName = "aide.db.gz.new"
aideWritingLogFileName = "aide.log.new"
aideReinitFileName = "aide.reinit"
aideConfigFileName = "aide.conf"
backupTimeFormat = "20060102T15_04_05"
pprofAddr = "127.0.0.1:6060"
aideWritingDBFileName = "aide.db.gz.new"
aideWritingLogFileName = "aide.log.new"
aideReinitFileName = "aide.reinit"
aideConfigFileName = "aide.conf"
aideMigrateConfigFileName = common.PreMigrationConfDataKey
backupTimeFormat = "20060102T15_04_05"
pprofAddr = "127.0.0.1:6060"
)

var DaemonCmd = &cobra.Command{
Expand Down Expand Up @@ -78,6 +79,7 @@ type daemonConfig struct {
FileDir string
RunDir string
ConfigDir string
RunMigrationCheck bool
}

type daemonRuntime struct {
Expand Down Expand Up @@ -196,6 +198,7 @@ func defineFlags(cmd *cobra.Command) {
cmd.Flags().Bool("debug", false, "Print debug messages")
cmd.Flags().Bool("local", false, "Run the daemon locally, using KUBECONFIG. Should only be used when debugging.")
cmd.Flags().Bool("pprof", false, "Enable /debug/pprof endpoints. Should only be used when debugging.")
cmd.Flags().Bool("migration-check", false, "Run the migration check")
}

func parseDaemonConfig(cmd *cobra.Command) *daemonConfig {
Expand All @@ -207,7 +210,8 @@ func parseDaemonConfig(cmd *cobra.Command) *daemonConfig {
conf.RunDir = getValidStringArg(cmd, "aideruntimedir")
conf.ConfigDir = getValidStringArg(cmd, "aideconfigdir")
conf.LogCollectorNode = os.Getenv("NODE_NAME")
conf.LogCollectorConfigMapName = getConfigMapName(getValidStringArg(cmd, "lc-config-map-prefix"), conf.LogCollectorNode)
conf.RunMigrationCheck, _ = cmd.Flags().GetBool("migration-check")
conf.LogCollectorConfigMapName = getConfigMapName(getValidStringArg(cmd, "lc-config-map-prefix"), conf.LogCollectorNode, conf.RunMigrationCheck)
conf.LogCollectorTimeout, _ = cmd.Flags().GetInt64("lc-timeout")
conf.Interval, _ = cmd.Flags().GetInt64("interval")
conf.MaxBackups, _ = cmd.Flags().GetInt("maxbackups")
Expand Down Expand Up @@ -279,11 +283,17 @@ func daemonMainLoop(cmd *cobra.Command, args []string) {
// integrityInstanceLoop is not added to the waitgroup because the watcher would stall wg.Wait() indefinitely. It
// will just die with the process and does not have anything to clean up.
go integrityInstanceLoop(ctx, rt, conf, errChan)
wg.Add(4)
go reinitLoop(ctx, rt, conf, errChan, &wg)
go holdOffLoop(ctx, rt, conf, errChan, &wg)
go aideLoop(ctx, rt, conf, errChan, &wg)
go logCollectorMainLoop(ctx, rt, conf, errChan, &wg)
// check if we are running in a migration scenario and if so, only run the migration check
if conf.RunMigrationCheck {
wg.Add(1)
go migrationCheckLoop(ctx, rt, conf, errChan, &wg)
} else {
wg.Add(4)
go reinitLoop(ctx, rt, conf, errChan, &wg)
go holdOffLoop(ctx, rt, conf, errChan, &wg)
go aideLoop(ctx, rt, conf, errChan, &wg)
go logCollectorMainLoop(ctx, rt, conf, errChan, &wg)
}

var finalErr error
for {
Expand Down
32 changes: 30 additions & 2 deletions cmd/manager/daemon_util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ func aideConfigPath(c *daemonConfig) string {
return path.Join(c.ConfigDir, aideConfigFileName)
}

func aideMigrateConfigPath(c *daemonConfig) string {
return path.Join(c.ConfigDir, aideMigrateConfigFileName)
}

func logAndTryReportingDaemonError(ctx context.Context, rt *daemonRuntime, conf *daemonConfig, fmt string, err error) {
LOG(fmt, err)
if reportErr := reportDaemonError(ctx, rt, conf, fmt, err); reportErr != nil {
Expand All @@ -92,9 +96,16 @@ func createErrorEvent(ctx context.Context, rt *daemonRuntime, conf *daemonConfig
}

func reportDaemonError(ctx context.Context, rt *daemonRuntime, conf *daemonConfig, format string, err error) error {
if reportErr := reportError(ctx, fmt.Sprintf(format, err), conf, rt); reportErr != nil {
return reportErr
if err != nil {
if reportErr := reportError(ctx, fmt.Sprintf(format, err), conf, rt); reportErr != nil {
return reportErr
}
} else {
if reportErr := reportError(ctx, format, conf, rt); reportErr != nil {
return reportErr
}
}

return createErrorEvent(ctx, rt, conf, format, err)
}

Expand All @@ -118,6 +129,23 @@ func runAideInitDBCmd(ctx context.Context, c *daemonConfig) error {
}, backoff.WithMaxRetries(backoff.NewExponentialBackOff(), maxRetries))
}

// runAideMigrationCheckCmdSaveOutput runs the AIDE migration check command and saves the output
func runAideMigrationCheckCmdSaveOutput(ctx context.Context, c *daemonConfig) (string, error) {
configPath := aideMigrateConfigPath(c)

cmd := exec.CommandContext(ctx, "aide-0.18", "-c", configPath, "-D")
_, err := cmd.Output()
if err != nil {
// Check if the error is an exec.ExitError to get the stderr output
if exitErr, ok := err.(*exec.ExitError); ok {
errString := string(exitErr.Stderr)
return strings.TrimSpace(strings.TrimSuffix(errString, "\n")), err
}
return "", err
}
return "", nil
}

func runAideScanCmd(ctx context.Context, c *daemonConfig) error {
configPath := aideConfigPath(c)
// CWE-78 - configPath is only made of user input during standalone debugging
Expand Down
42 changes: 32 additions & 10 deletions cmd/manager/logcollector_util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,10 @@ const (
configMapMaxSize = 1048570 // 1MB for etcd limit. Over this, you get an error.
)

func getConfigMapName(prefix, nodeName string) string {
func getConfigMapName(prefix string, nodeName string, migrationMode bool) string {
if migrationMode {
return prefix + "-migration"
}
return prefix + "-" + nodeName
}

Expand Down Expand Up @@ -176,7 +179,7 @@ func newLogConfigMap(owner *unstructured.Unstructured, configMapName, contentkey
}
}

func newInformationalConfigMap(owner *unstructured.Unstructured, configMapName string, node string, annotations map[string]string) *corev1.ConfigMap {
func newInformationalConfigMap(owner *unstructured.Unstructured, configMapName string, node string, annotations map[string]string, labels map[string]string) *corev1.ConfigMap {
return &corev1.ConfigMap{
TypeMeta: metav1.TypeMeta{
APIVersion: "v1",
Expand All @@ -193,21 +196,38 @@ func newInformationalConfigMap(owner *unstructured.Unstructured, configMapName s
UID: owner.GetUID(),
},
},
Labels: map[string]string{
common.IntegrityOwnerLabelKey: owner.GetName(),
common.IntegrityLogLabelKey: "",
common.IntegrityConfigMapNodeLabelKey: node,
},
Labels: labels,
},
}
}

func getLogCollectorConfigMapLabels(owner *unstructured.Unstructured, node string, migrationMode bool) map[string]string {
if migrationMode {
return map[string]string{
common.IntegrityOwnerLabelKey: owner.GetName(),
common.IntegrityMigrationLabelKey: "",
}
} else {
return map[string]string{
common.IntegrityOwnerLabelKey: owner.GetName(),
common.IntegrityLogLabelKey: "",
common.IntegrityConfigMapNodeLabelKey: node,
}
}
}

// reportOK creates a blank configMap with no error annotation. This is treated by the controller as an OK signal.
func reportOK(ctx context.Context, conf *daemonConfig, rt *daemonRuntime) error {
DBG("creating temporary configMap '%s' to report a successful scan result", conf.LogCollectorConfigMapName)
return backoff.Retry(func() error {
fi := rt.GetFileIntegrityInstance()
confMap := newInformationalConfigMap(fi, conf.LogCollectorConfigMapName, conf.LogCollectorNode, nil)
// check if we are on migration mode if so we need to create a configmap with the migration annotation
annotations := map[string]string{}
if conf.RunMigrationCheck {
annotations[common.AideConfigAutoMigrationFailedAnnotationKey] = "false"
}
labels := getLogCollectorConfigMapLabels(fi, conf.LogCollectorNode, conf.RunMigrationCheck)
confMap := newInformationalConfigMap(fi, conf.LogCollectorConfigMapName, conf.LogCollectorNode, annotations, labels)
_, err := rt.clientset.CoreV1().ConfigMaps(conf.Namespace).Create(ctx, confMap, metav1.CreateOptions{})
return err
}, backoff.WithMaxRetries(backoff.NewExponentialBackOff(), maxRetries))
Expand All @@ -218,9 +238,11 @@ func reportError(ctx context.Context, msg string, conf *daemonConfig, rt *daemon
return backoff.Retry(func() error {
fi := rt.GetFileIntegrityInstance()
annotations := map[string]string{
common.IntegrityLogErrorAnnotationKey: msg,
common.IntegrityLogErrorAnnotationKey: msg,
common.AideConfigAutoMigrationFailedAnnotationKey: "true",
}
confMap := newInformationalConfigMap(fi, conf.LogCollectorConfigMapName, conf.LogCollectorNode, annotations)
labels := getLogCollectorConfigMapLabels(fi, conf.LogCollectorNode, conf.RunMigrationCheck)
confMap := newInformationalConfigMap(fi, conf.LogCollectorConfigMapName, conf.LogCollectorNode, annotations, labels)
_, err := rt.clientset.CoreV1().ConfigMaps(conf.Namespace).Create(ctx, confMap, metav1.CreateOptions{})
return err
}, backoff.WithMaxRetries(backoff.NewExponentialBackOff(), maxRetries))
Expand Down
37 changes: 37 additions & 0 deletions cmd/manager/loops.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,43 @@ func aideLoop(ctx context.Context, rt *daemonRuntime, conf *daemonConfig, errCha
}
}

// The migrationCheckLoop checks if the migrated AIDE config file exists and if the new AIDE config works.
func migrationCheckLoop(ctx context.Context, rt *daemonRuntime, conf *daemonConfig, errChan chan<- error, wg *sync.WaitGroup) {
defer wg.Done()
migrateCtx, migrateCancel := context.WithCancel(ctx)
defer migrateCancel()

for {
select {
case <-migrateCtx.Done():
DBG("Migration loop cancelled by the main routine!")
Copy link
Contributor

@rhmdnd rhmdnd Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case - has the operation been aborted due to an error? Can, or should, the user do anything to restart the migration?

return
default:
output, err := runAideMigrationCheckCmdSaveOutput(migrateCtx, conf)
aideResult := common.GetAideExitCode(err)
if aideResult == 0 {
LOG("AIDE migration check passed. The configuration is safe to run using AIDE 0.18")
if err := reportOK(migrateCtx, conf, rt); err != nil {
// Considering this a non-fatal error right now.
LOG("Failed reporting migration check result: %v", err)
}
return
} else if aideResult == 17 {
// This is an AIDE config line error.
newErr := fmt.Sprintf("Detected configuration error during the migration check for AIDE 0.18: %s, output: %s ", common.GetAideErrorMessage(aideResult), output)
Copy link
Contributor

@rhmdnd rhmdnd Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool - this error moves us in the right direction I think by giving users the information they need to update AIDE configs. Curious to see how this renders in the logs with an actual AIDE error.

logAndTryReportingDaemonError(migrateCtx, rt, conf, newErr, nil)
errChan <- err
return
} else {
logAndTryReportingDaemonError(ctx, rt, conf, fmt.Sprintf("Error running migration check: %s",
common.GetAideErrorMessage(aideResult)), err)
errChan <- err
}
time.Sleep(time.Second * time.Duration(conf.Interval))
}
}
}

// The holdoff file is the signal from the node controller to pause the aide scan.
// We do not make this pause the logCollector loop, and we might want to.
func holdOffLoop(ctx context.Context, rt *daemonRuntime, conf *daemonConfig, errChan chan<- error, wg *sync.WaitGroup) {
Expand Down
Loading