Skip to content

feat(hydrator): parallelize repo-server calls (#24451) #24436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 9, 2025
Merged

feat(hydrator): parallelize repo-server calls (#24451) #24436

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c282449
chore(refactor): use contexts in hydration operations
crenshaw-dev Sep 5, 2025
d7d34a6
codegen
crenshaw-dev Sep 6, 2025
68444f9
chore(refactor): simplify hydrator manifest generation
crenshaw-dev Sep 5, 2025
087486a
feat(hydrator): parrallelize repo-server calls
crenshaw-dev Sep 6, 2025
3204831
Merge remote-tracking branch 'origin/master' into parallel-hydration
crenshaw-dev Sep 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 29 additions & 11 deletions controller/hydrator/hydrator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@ import (
"context"
"encoding/json"
"fmt"
"sync"
"time"

"golang.org/x/sync/errgroup"

log "github.com/sirupsen/logrus"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
Expand Down Expand Up @@ -279,17 +282,32 @@ func (h *Hydrator) hydrate(logCtx *log.Entry, apps []*appv1.Application, project
syncBranch := apps[0].Spec.SourceHydrator.SyncSource.TargetBranch
targetBranch := apps[0].Spec.GetHydrateToSource().TargetRevision

var paths []*commitclient.PathDetails
var targetRevision string
var err error
// TODO: parallelize this loop
for _, app := range apps {
var pathDetails *commitclient.PathDetails
targetRevision, pathDetails, err = h.getManifests(context.Background(), app, targetRevision, projects[app.Spec.Project])
if err != nil {
return "", "", fmt.Errorf("failed to get manifests for app %q: %w", app.QualifiedName(), err)
}
paths = append(paths, pathDetails)
// Get a static SHA revision from the first app so that all apps are hydrated from the same revision.
targetRevision, pathDetails, err := h.getManifests(context.Background(), apps[0], "", projects[apps[0].Spec.Project])
if err != nil {
return "", "", fmt.Errorf("failed to get manifests for app %q: %w", apps[0].QualifiedName(), err)
}
paths := []*commitclient.PathDetails{pathDetails}

eg, ctx := errgroup.WithContext(context.Background())
Copy link
Collaborator

@leoluz leoluz Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting usage of errgroup. It seems that you are using it just to avoid having to do wg.Add(1)? In go 1.25 this baked directly in the sync package:
https://antonz.org/go-1-25/#go-wait-group-go

Copy link
Member Author

@crenshaw-dev crenshaw-dev Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using it for the error handling and fail-fast context. I'm hoping those will make their way into the standard library soon as well.

var mu sync.Mutex

for _, app := range apps[1:] {
app := app
eg.Go(func() error {
_, pathDetails, err = h.getManifests(ctx, app, targetRevision, projects[app.Spec.Project])
if err != nil {
return fmt.Errorf("failed to get manifests for app %q: %w", app.QualifiedName(), err)
}
mu.Lock()
paths = append(paths, pathDetails)
mu.Unlock()
return nil
})
}
err = eg.Wait()
if err != nil {
return "", "", fmt.Errorf("failed to get manifests for apps: %w", err)
}

// If all the apps are under the same project, use that project. Otherwise, use an empty string to indicate that we
Expand Down
Loading