The layer underneath the data

The surface reasons were the usual ones, and they were real: pull data together to answer questions globally, across regions and across functions, instead of region by region. End the duplication and endless reconciliation by building a single source of truth. Standardise how data is collected, processed, stored, accessed, and governed.

But there was a deeper risk, and it’s the one that shaped our part of the work. The organisation didn’t have in-house cloud-architecture and security competence, so it ran a real chance of either doing this wrong or becoming over-reliant on vendors. It had been to the cloud before, including earlier data-mart efforts, and each time the design and architectural knowledge left with the vendor at handover. Only the operational knowledge stayed. Enterprise information security required detailed architecture reviews before major decisions, and the cloud custodians were deliberately conservative about granting access to sensitive services like IAM, KMS, and networking. Momentum was expensive. That combination defined the brief.

The honest scope matters, because it’s where the value actually was. We were not the platform’s builder. The data modelling, the pipelines, the Snowflake build, and the reconciliation sat with the data engineering teams, core and vendor. We were the cloud infrastructure, integration, and security architect for the platform: the foundation beneath the data, and the connective tissue around it.

The first decision wasn’t technical. It was about who would still understand the system a year after the vendors left. There’s a recurring failure worth naming: a vendor builds something competent, hands it over, and walks off with the design rationale. That’s how an organisation ends up operating a platform nobody internal can evolve.

Accountability, and the judgement behind it, has to live with people who stay. We made that an org-chart decision, not a hope.

Most of the architecture was the resolution of a standing tension between three groups, each wanting something incompatible with the others.

Information security wanted multiple layers of data protection, in transit and at rest, strict access control, and standards met and then exceeded. The enterprise cloud team owned the AWS account and, taught by past experience, fenced each division in: separate accounts per environment, no freedom to add new services, and tight control over any change to IAM, KMS, or networking. The data engineering teams wanted the opposite: broad, fast access during exploration, and even later, the access they could articulate was far coarser than the fine-grained policies real least-privilege demands. Holding all three at once, without any of them being wrong, was the job.

The answer to “how do you keep five inherited security models from becoming six” is to stop configuring security by hand. We expressed it as code, and not merely infrastructure-as-code. The access model itself became code.

We designed custom IAM roles to the least-privilege principle, then had each Terraform module grant those roles only the minimum access they need to function against that module’s resources. Conditional checks used tag values as attributes, attribute-based access on top of role-based access, and enterprise identities were federated into the AWS roles. The result: access policy written as Terraform module declarations, mapping resources, roles, and the least-privilege grants between them in one place. Resource-level policies on the sensitive stores (S3, KMS, Secrets) added another layer on top.

A word on the tool, because it was a deliberate choice: CloudFormation was too constrained for the structuring and reuse this needed. The CDK went the other way; its full programming power made guardrails hard. Terraform’s HCL sat in the sweet spot: near-declarative, with just enough iteration and conditionals, and modules for structure. The judgement was in picking the constraint level, not the brand.

The platform sat on AWS for several converging reasons: the core team’s existing skills, the maturity of both the platform and the enterprise cloud team, and an ecosystem that integrates closely with open-source providers, itself a hedge against lock-in. But the decisive reason was security. AWS’s combination of resource-level and role-level policy, and in particular an explicit deny that overrides everything else, made a posture possible that most setups can’t reach.

The other half of the choice was storage. AWS’s S3-based lakehouse is mature precisely because it started early, and combined with KMS access policies, it offered the versatile, multi-layered access control a regulated data store needs.

The most dangerous part of any multi-cloud platform is data crossing a trust boundary: AWS to Azure, Salesforce, Snowflake, on-prem. We didn’t make that safe with one big grant. We made it safe by layering, so no single failure exposes anything.

Usability and security pull in opposite directions, and the temptation in every migration is to consolidate fast and tidy the security later. We engineered the trade-off instead of choosing a side. A fully open sandbox, with no external connectivity, let the development teams experiment and discover what they actually needed. Beyond it, environments graduated: permissive while a service was new, tightened to least-privilege via code as the solution matured, so the higher environments were least-privilege by construction. It even defused a predictable political moment, when a vendor PM blamed the cloud architect for a missed deadline. The graduated scheme gave teams room to move without leaving the high environments loose.

And it has a failure worth telling, because it’s the proof the discipline matters. Once CI/CD for the infrastructure code was in place, a manual change was made directly in the console under deadline pressure, bypassing the pipeline. The drift between the declared state and reality grew until it needed its own dedicated project to repair. Policy-as-code is only as real as the discipline that stops anyone editing around it.

Governance as code isn’t a file format. It’s a promise that nothing changes outside the code, and that promise breaks the first time someone clicks “save” in a console under pressure.

The single most consequential design decision wasn’t any one policy. It was making the whole thing cohere. We designed Terraform modules not just for resources but for the fine-grained access policy too, at both the resource and IAM level, using attribute-based access. We built CI/CD for the infrastructure code, and managed Terraform state per subsystem so the platform could grow without the state file becoming a single point of contention.

That coherence is the part a less experienced architect, or a team split into separate developer, cloud-engineer, and build-engineer roles, rarely produces. Each role optimises its own slice, and nobody owns the seam between them. Spanning those roles to design one consistent system is exactly the judgement the engagement was for.

Five-plus warehouses became one. The platform became the foundation for multiple production AI systems. The natural-language-to-SQL assistant that let business users query this very warehouse in plain English was one of them, and the count has only grown since. It’s still running, and it genuinely did become the bedrock the later builds stood on.

We won’t claim governance was “100% as code,” because that wouldn’t be true. The drift episode is proof some change happened outside the code. The honest, and stronger, statement is that governance was implemented as code by design, and that the moment it wasn’t is exactly what taught the lesson. Precision about what’s true is the whole point of the accountability layer.

One observation outlives this particular platform. A data team that builds a foundation faces a quiet fork: stay a pure data provider, or climb toward co-owning the solutions built on top. The provider role is safer and more clearly bounded, but it leaves the most valuable work, including the AI use cases the platform makes possible, to other teams. The platforms that compound are the ones whose owners move up the value chain, from serving data to helping build what the data is for. Foundations are worth more when their builders don’t stop at the foundation.

Strip away the services and the work reduces to a handful of durable ideas.

• →Least privilege starts from the role's need. Grant exactly what a role must do to function; deny the rest - and back it with a deny that even admins can't override.
• →Make the access model code, not just the infrastructure. Roles, resources, and the grants between them belong in the same declarative place.
• →Policy-as-code is only as real as its CI/CD. One console change under pressure, and the promise and the state drift apart.
• →Defend in depth. Network, security groups, roles, encryption - no single grant should be load-bearing.
• →Tighten as you mature. Open sandbox for discovery, least-privilege by code where it counts - engineer the trade-off instead of choosing a side.
• →Keep the architecture in-house. Vendors can build; the know-how must stay with people who don't leave.