Building a GitOps Platform Developers Chose to Use
How I designed and rolled out a GitOps delivery platform for 35+ services and 10+ teams, what the trade-offs were, how teams came to trust it, and what broke along the way.
One of the most impactful systems I have built was a GitOps delivery and developer platform for a product used by millions of people.
It spans five AWS EKS clusters, two of them in production, and more than 35 services owned by over 10 engineering teams. I owned it from start to finish. That meant evaluating the existing delivery model, designing the architecture and the way teams would work with it, building it from scratch, and carrying it through production rollout and adoption.
GitOps keeps application configuration in Git. Argo CD compares that desired state with each cluster and reconciles differences according to the environment's sync policy. A release starts with a recorded Git change; production still needs a manual sync.
The ownership gap
The existing GitLab CI/CD setup worked well. The gap was developer ownership of delivery: teams still needed specialist help for routine releases.
Too many routine actions still meant learning the underlying tooling or asking DevOps for help. The organization shared the goal of letting developers build, test, deploy and release without depending on DevOps for every routine step. I took responsibility for turning that goal into a platform.
A model built around how developers work
Developers think in applications, environments and releases, not in Kubernetes objects, so that is the model the platform gives them.
On the common path a team provides a small set of configuration values. The platform takes care of the Kubernetes resources, the Argo CD configuration and anything that differs between environments. Nobody needs direct access to the clusters, or needs to know how they are organized underneath.
Developers choose the action they actually need.
- Build a commit
- Deploy a build that already exists
- Deploy a feature change from a pull request
- Promote a release
- Roll a release back
Nothing builds or deploys unless someone asked for it. One of the first things people noticed was that changing an environment variable no longer meant rebuilding the whole application.
Feature deployments start from a pull request or a comment on it. Developers own their feature and development environments and can iterate there freely. The same application model carries forward into staging and production, with stronger controls at each step.
Product teams decide when to release. The platform decides how releases happen and which guardrails they pass through.
What runs underneath
- A developer changes the service repository or requests a release.
- Shared workflows build an image and record its tag back in Git.
- ApplicationSets discover the service; Argo CD reconciles development and staging.
- The promotion gate checks and writes production values to Git. A person syncs production.
ApplicationSets generate Argo CD applications; AppProjects restrict their allowed repositories and destinations. A platform-owned chart scans GitHub for a repository topic and reads a small file per environment. A team registers once, then adds services without a platform pull request.
Shared GitHub Actions workflows build each image and record its tag in that environment's configuration. Three details keep that safe.
- A dedicated GitHub App writes the image tag as a signed commit to the protected main branch.
- The commit is marked so it does not trigger another build, which would otherwise loop forever.
- A guard skips any write that would move an environment to an older build, so builds that finish out of order cannot quietly roll anything back.
Development has one rule that matters a lot in practice. The main branch never deploys to dev automatically. When it did, feature deployments and main fought over the same resources and dev became unpredictable. That rule now lives in the template itself, so it holds by construction rather than by everyone remembering it.
Production is gated and fails closed. A release is cut for audit, and the promotion workflow only writes the production tag when every one of these holds.
- The commit is on the main branch
- Its build passed
- It moves production forward, never back
- An approver has signed off
A person then syncs each production application. Argo CD history provides an immediate rollback path, but it leaves the newer production image tag in Git. A durable rollback would need an approved exception to the forward-only promotion rule: update the image tag in Git to the intended version before the next sync, or the newer version can return. Argo Rollouts adds canary and blue/green deployments; Terraform with Terragrunt manages infrastructure across several AWS accounts.
I also built a CLI and an MCP server on top of the platform, so engineers and AI-assisted tools can run deployment and operational workflows without opening the Argo CD UI.
Balancing autonomy and control
This was the biggest trade-off. I could not build a separate platform for every team, and I did not want one that only worked for easy services. I settled on three tiers.
- A default path with sensible shared configuration, so most services move on their own
- Opt-in configuration for teams that need more
- A controlled exception path, with more platform involvement, for services that do not fit the standard model
Adoption kept testing those abstractions. Every team wanted features that matched its own workflow, and some requests showed me where my original design was too restrictive or too complicated. For each one I decided whether it belonged in the shared platform, should stay opt-in, or truly needed an exception. That kept the default simple for everyone while still serving the hard cases.
I also treated the GitOps model as a long-term platform contract, not as Argo CD being the final destination. Higher-level systems such as Crossplane or an internal developer portal can sit on top later, with the same governance and delivery model underneath.
Earning trust
Getting teams to adopt the platform was harder than building it. We were changing workflows people were comfortable with, so I had to show it was a real improvement and not one more platform to learn.
- I started with simpler services, to test the model with real developers and fix the experience before moving harder workloads.
- I worked directly with engineers to understand what frustrated them, and walked teams through the new workflows.
- I wrote documentation and recorded walkthrough videos from a developer's point of view.
- I showed practical wins, such as a configuration change going out without a rebuild, or a feature environment running without a DevOps ticket.
As confidence grew we moved complex and critical services onto the platform, including services with direct financial impact. The moment principal engineers and the teams who own those systems trusted the platform with critical production work was the milestone that mattered most to me.
We migrated more than 35 services across development, staging and production, and supported more than 10 product teams through the change, without customer-facing downtime during the migrations.
What broke, and what I changed
After production adoption, a short-lived repository-scanning problem removed two Argo CD Applications before they were recreated. Their Application deletion finalizers triggered deletion of the stateful workloads they managed. The highly available setup prevented customer impact, but the failure mode was real.
I set preserveResourcesOnDeletion on workload ApplicationSets and checked the behavior after rollout. That stops the controller adding the deletion finalizer to newly generated Applications; it does not remove finalizers already present on existing ones. Existing Applications need a separate finalizer check. Preserved workloads also need deliberate cleanup if an Application is removed.
Argo CD: Application deletion and resource preservation (opens in a new tab)I deliberately left the policy ApplicationSets out of that change. They grant permissions, and a permission is revoked by removing its entry. Protecting those from deletion would have kept revoked permissions alive.
A safety guard is not one-size-fits-all. Before adding one, ask whether deleting is the dangerous action or the protective one.
A few smaller problems taught me as much.
- The deploy App's commits kept being rejected even though it was allowed to bypass the branch ruleset. An older branch protection rule also applied, and that bypass does not cover it. Each protection layer needs its own exception.
- A shared workflow failed at startup after I trimmed the permissions its caller granted. The caller has to grant everything the shared workflow declares, not just what the steps it runs use.
- Manual deploys stopped working because a lookup found builds by branch alone and picked up a scheduled nightly run, which a later check rejected. Filtering by the kind of event fixed it.
- One service could never appear in Argo CD because a chart it depended on lived in a private storage bucket Argo CD could not reach. Its first production configuration also had to start from the image already running, or the first sync would have deployed the latest image instead.
Decisions I would revisit
- We deploy by commit tag rather than by image digest, because the shared chart only understood tags. Digest pinning and provenance checks should come before any fully automated production promotion.
- Shared workflows were referenced by a moving major version while handling the deploy App's key. The App's installation permissions limit the blast radius; pinning workflows to immutable commits and protecting release tags would reduce supply-chain risk.
How I knew it worked
Success was never just Argo CD running or services having moved. Three things had to be true.
- Developers shipped with much less DevOps dependency, and chose the platform because it gave them more control, not because they were told to.
- The platform was trusted with critical production workloads.
- The model held up as more teams joined, growing to meet new requirements instead of collapsing under exceptions.
The hard part was never Argo CD or Kubernetes on their own. It was balancing developer autonomy against production safety, finding common ground across teams with different needs, and earning enough trust that engineers actually wanted to use what we built.