Blog
Private endpoints, customer-managed keys and zero retention. The three things to insist on before proprietary data goes near a foundation model.
-By Otto Gori, Lead Cloud Engineer & Engagement Manager at Rewire.
To say that data sovereignty – i.e. owning and controlling your data and making sure that there’s no unauthorized third-party access – is top of mind would be an understatement. In the first of a series of articles, I discuss the constraints and technology options to achieve data sovereignty while still enjoying the benefits of Generative AI.
The data sovereignty challenge
Consuming Generative AI at enterprise scale creates inherent complexity and tension between the data governance of your industrial secrets and what this new tool brings as capability to your company.
While consumer AI interfaces rely on shared API endpoints with "unknown" data retention or use by third party companies like OpenAI, Anthropic and so forth, enterprise architectures require strict data isolation, zero-retention guarantees, and precise lineage tracking.
The central thesis of modern AI architecture is straightforward: Enterprises do not necessarily need to build foundation models from scratch to achieve full data sovereignty. By abstracting the intelligence layer from the underlying data hosting layer, organizations can maintain absolute administrative cryptographic and operational control over their proprietary assets while leveraging managed foundation models.
So problem solved? Not quite. That’s because there are a bunch of requirements that constrain data governance. Let’s review some of the most critical ones.
The “non negotiables” of production-grade data governance
Production readiness requires moving beyond raw API calls to robust control planes.
The first pillar of control is network perimeter isolation. Enterprise AI solutions must route inference prompts and fine-tuning artifacts through private link endpoints, preventing third-party telemetry or sensitive payload traffic from traversing the public internet.
The second pillar centers on zero data retention (ZDR) and cryptographic boundaries. Model providers must operate strictly as stateless processors. Customer prompts, context windows, and retrieval-augmented generation (RAG) embeddings must never be used to train foundational weights or stored in persistent provider logs. Furthermore, customer-managed encryption keys (CMEK) ensure that fine-tuned weights and vector embeddings remain unreadable to the platform provider at rest.
The third pillar is cloud agnosticism. Relying on proprietary toolchains creates strategic vulnerabilities. A resilient architecture decouples model invocation from business logic, treating large language models as interchangeable processing units behind unified API gateways.
But before we go agnostic, which solutions out there follow these principles, and you can easily interface with? Data sovereignty after all is not strictly defined by where your models run, but by who controls the compliance, encryption keys, network routes, retention logs and the physical location of the datacenters.
An overview of common technology solutions – and pointers on how to choose
While self-hosted open-source models (e.g., Llama, Mistral) on dedicated GPU clusters offer maximum theoretical control, they carry significant operational overhead. AWS Bedrock, Google Cloud Vertex AI, and Azure Foundry together control roughly 70% of the European hyperscale market, so your engineering team is almost certainly familiar with one or more of them. They provide the fastest path to production because they allow enterprises to apply existing governance policies, identity management (IAM), and network perimeters to foundation models without maintaining bare-metal infrastructure. They also greatly reduce the engineering challenge of implementing this solution in-house. Not to mention the very attractive price of experimentation when comparing with the acquisition of expensive hardware.
Rather than evaluating or recommending a single platform, this overview presents the primary enterprise options as reference. Each offers distinct governance models and isolation configurations; selecting the optimal solution depends entirely on your organization’s unique regulatory mandates and existing cloud footprint.
Amazon Bedrock
Implements a serverless API paradigm centered on strict data boundary enforcement. Prompts and completions are rendered statelessly and can be configured for zero content logging or model training by third-party model providers. Through AWS PrivateLink, inference calls remain entirely within the customer’s Amazon VPC. Fine-tuning jobs for models like Meta Llama or Amazon Titan run inside, isolated compute environments, encrypted with customer-managed AWS KMS keys.
Google Cloud Vertex AI
Provides integrated enterprise governance through Private Service Connect and VPC Service Controls. It allows organizations to deploy Gemini models alongside custom open-weight architectures within a controlled security perimeter. Vertex AI enforces strict zero data retention policies for enterprise contracts and integrates natively with Cloud KMS, guaranteeing that prompt histories and customer embeddings remain isolated within the enterprise boundary.
Azure Foundry
Enterprise-grade orchestration across Azure OpenAI Service and open-source model catalog offerings. By anchoring infrastructure in Azure Virtual Network private endpoints and Microsoft Entra ID role-based access control, organizations prevent data exposure. Azure Foundry ensures that customer data is never used to train global base models, while enabling fine-tuning, RAG indexing in Azure AI Search, and prompt evaluation under strict data loss prevention policies.
Figure 1. A comparative matrix of the three most common solutions
| Feature | Amazon Bedrock | Google Cloud Vertex AI | Azure AI Foundry |
|---|---|---|---|
| Primary network isolation | AWS PrivateLink | VPC Service Controls and Private Service Connect | Azure Private Link |
| Key management | AWS KMS (CMK) | Cloud KMS (CMEK) | Azure Key Vault (CMK) |
| Zero data retention | Configurable and contractual | Contractual guarantees | Opt-in and contractual |
| Multi-model flexibility | Broadest serverless API catalogue | Native Gemini plus custom open-weight support | Strong OpenAI integration plus open-source catalogue |
| Identity and access | AWS IAM | Google Cloud IAM | Microsoft Entra ID (RBAC) |
Scroll the table sideways to see all three platforms.
Choosing the right platform usually comes down to where your data already lives and which models your workloads demand, but in a nutshell, this is a quick reference if you are still in doubt:
Choose AWS Bedrock if: Your application architecture is already heavily invested in AWS. Bedrock's serverless approach makes model swapping painless via unified APIs, and its PrivateLink integration is rock-solid for strict VPC isolation. Bedrock shines when you want to experiment with different model providers (like Anthropic vs. Meta) without rewriting your orchestration logic or managing compute instances.
Choose Google Cloud Vertex AI if: You require deep integration with BigQuery to leverage advanced data analytics or want to fine-tune open-weight models alongside Google’s Gemini family. Vertex AI’s VPC Service Controls provide unmatched security perimeters if you are already handling complex data pipelines on GCP, though configuring the security boundaries requires rigorous IAM hygiene.
Choose Azure Foundry if: Your enterprise relies heavily on the Microsoft ecosystem (Azure VNets, Microsoft Entra ID) or requires direct access to OpenAI’s frontier models within a dedicated enterprise tenant. Azure makes identity management and Role-Based Access Control (RBAC) seamless through Entra ID, making it a favorite for security teams to audit end-user access to RAG pipelines.
Conclusion
Achieving true data sovereignty in enterprise Generative AI is ultimately an architectural discipline, not a product feature. While Amazon Bedrock, Google Vertex AI, and Azure Foundry offer the stateless invocation, private perimeters, and encryption capabilities required for compliance, no managed platform will solve governance for you out of the box. Sovereign AI isn't about where the intelligence is hosted—it’s about who holds the keys to the kingdom.
And as a personal hint: use KMS Keys for your kingdom =)
Sources - If you would like to extend your research, start here.
Google Cloud Vertex AI
- Google Cloud Generative AI Privacy & Data Governance Commitment. Google’s explicit contractual commitments regarding data ownership, non-retention of prompts/responses for base model training, and adherence to enterprise privacy standards.
- Vertex AI Data Governance & Security Controls. How Vertex AI protects the complete AI life cycle—from training to runtime inference—within enterprise perimeters.
- Vertex AI Dataset Encryption with CMEK. Technical implementation of Customer-Managed Encryption Keys (CMEK) via Google Cloud KMS for encrypting Vertex AI datasets and model artifacts.
- VPC Service Controls Documentation. Mitigating data exfiltration risks by enforcing private network boundaries and isolating multi-tenant Google Cloud resources.
Amazon Bedrock
- Amazon Bedrock Security, Privacy, and Responsible AI. Amazon Bedrock’s core data policy, establishing that customer inputs and outputs are never shared with model providers or used to train base foundation models.
- [Data Protection in Amazon Bedrock (User Guide). Encrypting custom models, agent sessions, and knowledge bases using AWS Key Management Service (KMS) customer-managed keys.
- Amazon Bedrock VPC & PrivateLink Setup. Establishing private VPC network connectivity through AWS PrivateLink to execute inference requests without exposing traffic to the public internet.
Azure Foundry / Azure OpenAI
- Data, Privacy, and Security for Models Sold in Microsoft Foundry. Proof that prompts, generated content, and fine-tuning data in Azure Foundry are strictly customer-isolated, double-encrypted at rest, and omitted from base model training.
- Data, Privacy, and Security for Azure Foundry Agent Service. Specific data isolation rules for stateful entity memory, RAG integrations, and custom agent tool execution.
- Security Features in Azure Content Understanding & Foundry Tools. Implementing Virtual Network (VNet) private endpoints, Microsoft Entra ID role-based access control (RBAC), and Customer-Managed Keys (CMK) within Azure AI infrastructure.
Agents reward organisations that ask harder questions first.
The organisations getting compounding value from agentic AI share one trait: they invested in getting the foundations right before scaling.
We've built agent systems across financial services, energy, telecom, and public sector, from early proofs of concept to production systems running at scale. We've seen what separates the deployments that deliver from the ones that don't.
