When Your GitOps Monorepo Starts Fighting Argo CD

One small Git commit should not make thousands of Argo CD Applications line up for work. That was the shape of the problem we were dealing with. We were seeing high Argo CD reconcile latency.

The Shape of Our Setup

Our Argo CD control plane is itself managed through GitOps.

A top-level cluster Application creates a system-tools Application, and that system-tools Application owns the Argo CD control plane Application.

+-------------------------+
| top-level cluster app   |
+-----------+-------------+
            |
            v
+-------------------------+
| system tools app        |
+-----------+-------------+
            |
            v
+-------------------------+
| Argo CD control plane   |
+-------------------------+

The Argo CD control plane runs in the argocd namespace and is synced from Git like the rest of the platform.

Git
 |
 v
Argo CD control plane manifests
 |
 v
argocd namespace

At the time of review, the control plane had:

• 1 application controller StatefulSet
• 3 repo-server replicas
• 1 API server
• 1 ApplicationSet controller
• 1 Redis deployment
• 1 Dex deployment
• 1 notifications controller

Across the whole Argo CD instance, we were operating at 2,000+ Application scale, with most application workloads generated from the same application monorepo.

The application monorepo is structured around service and deployment scope. The general path format looks like this:

manifests/<service>/<country>/<environment>

So one service deployed to multiple countries and environments becomes many Argo CD Applications:

manifests/account-service/MY/dev
manifests/account-service/SG/dev
manifests/account-service/MY/production
manifests/payment-backend/TH/production

That structure is useful because each Application has a clear ownership boundary. It also means the same repository and revision are reused many times across different source paths.

Our app repo is a monorepo. Many Argo CD Applications point to the same Git repository and revision:

repoURL: https://github.com/example/application-gitops.git
targetRevision: HEAD

Each Application has its own source path:

path: manifests/account-service/MY/dev

But without additional path hints, when a commit lands in the monorepo, Argo CD mostly sees:

the repo changed
HEAD changed
many apps use this repo at HEAD

So even if a commit only changes one service, Argo CD may refresh many Applications that share the same repo and revision.

That creates the painful chain:

small commit to application GitOps monorepo
        |
        v
many Applications marked for refresh
        |
        v
large app_reconciliation_queue
        |
        v
many manifest generation requests to repo-server
        |
        v
repo-server git/cache/lock pressure
        |
        v
higher p99 reconcile latency

Why Argo CD Was Working Well Until Now

The default model worked well for us for a long time. At smaller scale, broad refresh behavior is usually acceptable:

repo changes
  -> Argo CD refreshes apps
  -> repo-server generates manifests
  -> controller compares desired state with live cluster state

That is the normal GitOps loop.

The problem only became visible as the system grew:

• more services
• more countries
• more environments
• more generated Applications
• more frequent image and tag commits
• more apps sharing the same monorepo at HEAD

At small app counts, a broad refresh wave is tolerable. At 2,000+ Application scale, it becomes expensive.

The system still works, but tail latency starts to hurt because the reconcile queue gets longer and repo-server work piles up.

The fix was to make that broad refresh behavior path-aware. We added this annotation to generated app Applications:

argocd.argoproj.io/manifest-generate-paths: "."

For an app like this:

spec:
  source:
    repoURL: https://github.com/example/application-gitops.git
    targetRevision: HEAD
    path: manifests/account-service/MY/dev

the annotation value . means:

watch this app's own source.path

So Argo CD resolves it as:

manifests/account-service/MY/dev

When GitHub sends a webhook to Argo CD, the webhook payload includes the list of changed files. Argo CD can compare those changed files against each Application’s manifest generation paths.

For example, if this file changes:

manifests/account-service/MY/dev/values.yaml

Argo CD should refresh:

dev.my.account-service

But it can skip unrelated apps:

dev.my.user-service
dev.sg.payment-backend
dev.th.order-backend

The annotation does not make manifest generation itself faster. It prevents unnecessary manifest generation from being requested.

Before the change, the behavior looked like this:

commit to one service
        |
        v
many apps using the monorepo at HEAD refresh
        |
        v
large reconciliation queue
        |
        v
repo-server pressure
        |
        v
high p99 latency

After adding manifest-generate-paths, the behavior became:

commit to one service
        |
        v
Argo CD checks changed files from webhook
        |
        v
only apps whose manifest paths changed refresh
        |
        v
smaller reconciliation queue
        |
        v
less repo-server pressure
        |
        v
better p99 latency

The reason . works well here is that our generated Applications already use service, country, and environment-specific paths:

manifests/{{ service.name }}/{{ country }}/{{ env }}

So . gives us the tightest useful scope.

Example:

source.path = manifests/account-service/MY/dev
annotation  = .
effective watched path = manifests/account-service/MY/dev

If we used .., the watched path would become broader:

manifests/account-service/MY

That could cause changes in sibling environments or nearby shared files to refresh more apps than necessary. For our current structure, . is the cleanest fit.

The Main Lesson

One Git commit could fan out into thousands of app refreshes because many Applications shared the same monorepo and HEAD revision.

Horizontal scaling can help absorb load, but the better first optimization is to avoid creating unnecessary work in the first place.

The annotation changed the behavior from:

repo changed, refresh broadly

to:

repo changed, refresh only apps whose manifest paths changed

That is the core lesson: at small scale, broad reconciliation is fine. At monorepo scale, path-aware reconciliation becomes important. Our scale made the default behavior expensive.