View all study notes here

---

Responsible & Secure AI

Data Privacy Boundaries in Vertex AI

When using Vertex AI (Gemini Enterprise Agent Platform) to call Google’s foundation models, your data is not used to train Google’s public foundation models. This is a hard contractual and technical boundary.

• Enterprise customers’ prompts, completions, and fine tuning data stay private to the customer’s project
• Customer data is not used to improve base model weights for other customers
• This is enforced at the infrastructure level, not just by policy

Model Armor

A new security service that provides prompt and response protection for AI applications. Sits in front of LLM calls to intercept and filter content.

Protects against

• Prompt injection
  • Malicious content in user inputs attempting to hijack model behaviour
• Jailbreak attempts
  • Inputs trying to bypass safety constraints
• Tool poisoning
  • Malicious tool calls in agentic workflows
• Sensitive data leakage
  • PII or confidential data appearing in model responses
• Harmful content
  • Harassment, dangerous content, violence

How it works

• Two enforcement modes
  • INSPECT_ONLY (log and allow)
  • INSPECT_AND_BLOCK (log and stop)
• Configured via “floor settings”
  • Minimum detection thresholds applied organization wide
• Integrates directly with the Gemini API in Vertex AI / Agent Platform
• Logs violations to Cloud Logging for audit trails

Sensitive Data Protection (formerly Cloud DLP) For AI

Scanning pipelines that feed data into AI models should include Sensitive Data Protection to:

• Detect and de-identify PII before it enters training datasets
• Scan BigQuery tables and Cloud Storage files used as AI training data
• Prevent regulated data (PHI, PCI) from appearing in model outputs
• Sensitive Data Protection sits in the data pipeline before Vertex AI sees the data.

IAM for AI Workloads

Standard IAM principles apply, with AI specific roles:

• roles/aiplatform.user: submit training jobs, make predictions
• roles/aiplatform.developer: create and manage models, pipelines, datasets
• roles/aiplatform.admin: full Vertex AI / Agent Platform administration
• Workload Identity Federation: external ML frameworks authenticating to Vertex AI without service account keys

VPC Service Controls + Vertex AI

• Create a service perimeter that includes aiplatform.googleapis.com to prevent data exfiltration.
• Ensures training data in Cloud Storage cannot be accessed from outside the perimeter, and model outputs cannot be exfiltrated.
• Critical for HIPPA and PCI-DSS compliance scenarios.

Agent Identity

For agentic workloads (autonomous agents making decisions across systems), Ahent Identity assigns every deployed agent a unique cryptographic ID. Provides:

• Complete audit trail of every action taken by each agent.
• Scoped permissions, each agent only has access it needs.
  • Apply least privilege to agent based workflows.
• Revocation capabilities, disable a specific agent’s access without affecting others

Agent Gateway

Security and governance layer for multi agent systems. Functions as “air traffic control” for agent interactions.

• Enforces consistent security policies across all agent-to-tool and agent-to-agent communication.
• Integrates Model Armor protection.
• Provides unified logging and observability for agent activity.

---

New & Updated Products

Gemini Cloud Assist

AI assistant in Google Cloud Console

• Provides architecture recommendations inline while you work.
• Can query you cloud resources in natural language.
• Does NOT retain conversation context between sessions in a day that violates data residency policies.
• Is subject to the same IAM restrictions as the user operating it.

Colab Enterprise

Google’s managed Jupyter notebook environment fro enterprise ML workflows.

• Integrated with Vertex AI and BigQuery
• Supports team collaboration with access controls
• Connects to enterprise data without data ever leaving GCP
• Replaces older AI Platform Notebooks / Workbench

Vertex AI Pipelines (MLOps Focus)

Automate, version, and reproduce ML training workflows.

• Based on Kubeflow Pipelines
• Each step in a pipeline is a containerized component
• Pipelines are reproducible, re-running reproduces the same results
• Integrates with Cloud Build for CI/CD of ML pipelines
• Artifacts and metadata tracked automatically in Vertex ML Metadata

NotebookLM

Google’s AI native research and note taking tool. Is a product built on Vertex AI.

---

Key Architectural Patterns

1. Enterprise RAG Application

Build a Q&A bot that answers questions from our internal knowledge base

Architecture

• Cloud Storage or BigQuery -> source data
• Vertex AI Embeddings API -> generate vectors
• Vertex AI Vector Search -> index & retrieve
• Agent Builder / RAG Engine -> orchestrate retrieval + generation
• Gemini model -> generate final response
• Model Armor -> inspect prompts and responses
• Cloud Logging -> audit all interactions

2. Custom Model Training & Serving

Train a custom classification model on our proprietary data and serve it at low latency

Architecture

• BigQuery or Cloud Storage -> training data storage
• Vertex AI Workbench / Colab Enterprise -> experimentation
• Vertex AI Pipelines -> automated training workflow
• Cloud Storage -> store model artifacts
• Vertex AI Model Registry -> version the model
• Vertex AI Endpoints (online prediction) -> serve predictions
• Cloud monitoring + Vertex AI Model Monitoring -> detect drift

3. Large Scale LLM Fine Tuning

Fine tune Gemini on our domain specific data for better accuracy

Architecture

• Cloud Storage -> data preparation (JSONL format)
• Vertex AI -> supervised fine tuning (SFT) or parameter efficient fine tuning (PEFT)
• Run on TPU v4 or GPU cluster via Vertex AI training job
• Model Registry -> register fine tuned model
• Vertex AI Endpoint -> deploy with traffic splitting for A/B testing NOTE: full fine tuning vs PEFT vs prompt engineering
• If scenario is “limited labelled data” or “low cost”, choose prompt engineering.
• If “domain specific accuracy requirements”, choose fine tuning
• If “customize without retraining full model”, choose PEFT.

4. Autonomous Agent Architecture

Build an agent that can query out databases, send emails, and book meetings.

Architecture

• Agent Development Kit (ADK) -> build agent logic
• Agent Platform -> deploy & manage agent runtime
• Agent Identity -> assign cryptographic ID for auditability
• Agent Gateway + Model Armor -> secure agent interactions
• MCP Servers (Cloud BigQuery, Google Maps, etc) -> connect tools to agent
• Memory Baank -> persist cross session context
• Cloud Logging + Unified Trace Viewer -> observe agent reasoning

---

Well-Architected Framework Applied to AI

Map every AI/ML architecture decision through the six pillars:

Pillar	AI/ML Considerations
Operational Excellence	Vertex AI Pipelines for reproducible MLOps; automated retraining triggers; model versioning in Model Registry; Colab Enterprise for collaborative development
Security	Model Armor for prompt/response protection; VPC Service Controls perimeter including aiplatform.googleapis.com; Sensitive Data Protection before training; Agent Identity for agentic systems; CMEK for training data at rest
Reliability	Traffic splitting on Vertex AI Endpoints for canary rollouts; Model Monitoring for drift detection; fallback logic if endpoint is unavailable; regional redundancy for critical endpoints
Performance	Right hardware selection (TPU vs GPU); batch vs online prediction trade-offs; Vector Search index configuration for low-latency RAG; model distillation for smaller, faster models
Cost Optimization	Batch prediction for non-latency-sensitive workloads; TPU V5e/Ironwood for cost-efficient inference; Gemini Flash over Pro for high-volume/lower-complexity tasks; idle endpoint shutdown
Sustainability	TPUs are more energy-efficient than GPUs for equivalent workloads; prefer managed services over self-managed GPU clusters; batch workloads during off-peak hours

---

Appendix

AI/ML workflows can be separated into two primary lanes: Predictive ML (using historical data to forecast trends/numbers. e.g. fraud detection), and Generative/Agentic AI (using foundation models to generate text, code, or execute autonomous workflows).

Google Cloud has unified these paradigms under the Gemini Enterprise Agent Platform (next gen Vertex AI).

Basic Terminologies

Foundation Models (FMs)

• A massive ML model trained on a vast generalized dataset (e.g. entire public internet, libraries of books, code repositories) that can be adapted to a wide range of downstream tasks.
• e.g. a new university graduate with a broad, well rounded education; not yet trained for any specializations, but have massive baseline of language,, logic and reasoning skills.
• Do not train foundation models yourself as it costs millions in supercomputing infrastructure.
• Use managed foundation model (e.g. Gemini 3 Pro, Flash) from the Model Garden and use architectures like RAG or turning to tailor it to business needs.

Temperature

• A configuration parameter that controls the randomness and creativity of a model’s output.
• Ranges from 0.0 to 2.0 (or sometimes 1.0 depending on framework).
• Lower temperatures force the model to highly prioritize the most mathematically probable next word.
• Higher temperatures flatten the probabilities, allowing the model to choose less obvious, more creative words.
• Set temperature to 0.0 for deterministic, factual tasks (e.g. code generation, analyzing financial spreadsheets, customer support bots drawing from knowledge base).
• Set temperature higher (e.g. 0.7 to 1.0) for creative workflows (e.g. marketing copy generation or brainstorming tools).

Tokens (and Context Window)

• A token is the basic unit of text that model processes.
• Not exactly a word; typically a syllable or fragment of a word (~4 characters or 0.75 words in English).
• The context window is the maximum number of tokens a models can read and write in a single session.
  • Like the model’s short term memory.
• Gemini has 2 million + token context window, meaning an architect can feed an entire codebase, hours of video, or thousands of pages of reports directly into a single prompt without running out of memory.
• Tokens are also the direct billing metric for GenAI (pay per 1000 input and output tokens).

Grounding (Retrieval Augmented Generation / RAG)

• The practice of connecting a foundation model to an external, verified source of truth (e.g. company’s private cloud databases or document repositories) to ensure its answers are factual and accurate.
• e.g. like taking an open book test instead of relying purely on what’s memorized (the pre-trained weights).
• Important for stopping hallucination (when a model confidently invents false facts).
• Enterprise Grounding if a company needs an AI agent to look up user account details, real time inventory, or specific company legal clauses.

Embeddings and Vector Databases

• An embedding is a way of converting data (words, sentences, or entire images) into a long string of numbers (a vector).
• A vector database is a database designed specifically to store and mathematically compare these strings of numbers.
• Words or concepts with similar meanings are assigned numbers that sit close to each other in a multi-dimensional mathematical space.
• When building a grounding/RAG pipeline, must first convert all data into embeddings and store them in a vector database (like Vertex AI Vector Search).
• When a user searches for something, the database performs a fast mathematical calculation to pull up the most relevant documents based on conceptual meaning, not just exact keyword matches.

Fine Tuning vs Prompt Engineering

• Prompt engineering
  • Writing, structuring, and optimizing the text prompt given to the model to guide it’s behaviour.
  • Changing the input instructions without changing the model itself.
• Fine tuning
  • Taking an existing foundation model and training it further on a smaller, highly specific dataset to permanently change its internal mathematical weights.
• Fine tuning is expansive, requires deep data science expertise, and locks the model into a specific point in time.
• Architects should always try prompt engineering and grounding first.
• Only use find tuning if the model needs to learn a completely unique dialect, tone, or highly specialized industry formatting style (e.g. formatting raw medical machine telemetry).

Common End to End AI/ML Workflow

1. Exploration & proto
2. Data prep & feature
3. Orchestration & training
4. Registry & versioning
5. Model server/API
6. Monitoring & tracing

Ideation, Exploration & Prototyping

Before writing scale out training code, data scientists and prompt engineers need a sandbox.

• GenAI
  • Environment to test prompts and map autonomous reasoning paths.
• Predictive ML
  • Interactive scratchpads to write Python, analyze data chunks, visualize charts.

GCP Components

• Agent Studio / Vertex AI Studio
  • Low-code/no-code visual playground to experiment with foundation models, fine tune prompts, test parameters (e.g. temperature), and design agent reasoning loops.
• Model Garden
  • A curated directory of pre-trained models.
  • Can choose from Google’s native foundation models (Gemini 3) or open weights options (Gemma, Llama).
• Colab Enterprise / Vertex AI Workbench
  • Fully managed, cloud hosted Jupyter Notebook environments where data scientists write code.

Underlying Technology

• GenAI Interfacing
  • REST/gRPC API wrappers over massive distributed neural networks hosted globally by the cloud provider.
• Notebooks
  • VMs containerizing web based IDEs attached to a persistent storage drive.

Industry Alternatives

• Open source: JupyterLaab, Hugging Face Spaces / Model Hub
• AWS: Amazon Bedrock Playgrounds (GenAI), SageMaker Studio Notebooks (Predictive)
• Azure: Azure AI Foundry / Azure ML Studio

Data Management & Feature Engineering

Models are only as good as the data fed into them.

• In predictive ML
  • Raw data is transformed into features (e.g. converting a raw timestamp into is_weekend = True)
  • Features must be calculated exactly the same way during training as they are during real time production serving to prevent data leakage.
• In generative AI
  • Connecting the model to private datasets to supply context, a technique known as Retrieval Augmented Generation (RAG).

GCP Components

• Vertex AI Feature Store
  • A centralized, time travel capable repository to store, share, and serve ML features consistently across offline training and online serving.
    • Time travel: point in time correctness, ability to fetch feature values exactly as they existed at a specific timestamp in the past.
• Enterprise Grounding (RAG Engine / Vertex AI Search)
  • Automatically chunks, embeds, and indexes internal corporate documents (from Cloud Storage or BigQuery) so Gemini can retrieve real time facts securely.

Underlying Technology

• Feature Store
  • A dual storage architecture.
  • A low latency key-value store (like Redis or Bigtable) handles real time online lookups, while a high throughput data lake (like BigQuery or Apache Iceberg) handles historical batch fetches.
• Grounding/RAG
  • Vector databases
  • Text data is converted into arrays of numbers (embeddings) using n embedding model and stored in a specialized index designed for mathematical similarity search (Approximate Nearest Neighbour/ANN algorithms).

Industry Alternatives

• Open source: Feast (Feature Store), Pinecone/Milvus/Qdrant (Vector databases), LangChain / LlamaIndex (RAG orchestration).
• AWS: SageMaker Feature Store, Amazon Bedrock Knowledge Bases (RAG)
• Azure: Azure ML Feature Store, Azure AI Search (Vector/RAG)

Pipeline Orchestration & Model Training

A production system cannot reply on a data scientist manually hitting run on a notebook cell. Must build a repeatable, automated factory pipeline that ingests new data, validates it, pre-processes it, trains the model, and evaluates the performance.

GCP Components

• Vertex AI Pipelines
  • A serverless orchestrator that executes multi-step ML workflows safely and tracks metadata automatically.

Underlying technology

• Kubeflow Pipelines (KFP) & TensorFlow Extended (TFX)
  • Vertex AI Pipeline is built entirely on top of open source Kubeflow.
    • Kubeflow: Used to manage, orchestrate, and scale ML workflows and pipelines.
  • Each step in pipeline is executed inside its own isolated Docker container running on an underlying managed Kubernetes cluster.

Industry Alternatives

• Open source: Apache Airflow, Kubeflow, Perfect, MLflow Pipelines
• AWS: SageMaker Pipelines
• Azure: Azure Machine Learning Pipelines

Governance: The Model Registry

Once a model is trained or a custom prompt agent is built, it must be saved, version controlled, and audited. A central ledger that tracks version histories, who trained it, what data was used (lineage), and whether it has been approved for production.

GCP Components

• Vertex AI Model Registry / Skill Registry
  • A centralized repository to manage the lifecycle of ML models and AI agent skills.

Underlying technology

• A secure object storage registry backed by a metadata database tracking artifact URIs (e.g. a pointer to a model.tar.gz file in Cloud Storage), framework versions, and deployment tags (e.g. Staging, Production).

Industry Alternatives

• Open source: MLflow Model Registry, Weights & Biases
• AWS: SageMaker Model Registry
• Azure: Azure ML Model Registry

Deployment & Model Serving

Expose models over the network so other applications can query it.

• Online / real time
  • Application sends request and expects an answer in milliseconds
• Batch
  • Running predictions over millions of database rows offline overnight.

GCP Components

• Vertex AI Endpoints
  • Dedicated, auto scaling web endpoints designed to serve predictions from custom predictive models at low latency.
• Agent Engine
  • Managed runtime used to host and serve production ready autonomous generative AI agents

Underlying Technology

• Inference Serving Engines
  • Under the hood, Vertex AI Endpoints spin up a load balancer in front of containerized web servers running optimized inference engines like Triton Inference Server, TS Serving or vLLM (for open large language models), utilizing GPUs or TPUs (Tensor processing unit, Google’s specialized microchip for AI) as compute backbones.

Industry Alternatives

• Open source: TorchServe, KServe, vLLM, BentoML
• AWSL SageMaker Real-time Endpoints
• Azure: Azure ML Online Endpoints

Explainability, Evaluation & Observability

Once model or agent is live in production, monitor it to ensure it remains accurate over time, debug its internal logic, and understand why it made a specific prediction (explainability), or tracing how an agent reached a conclusion (observability).

GCP Components

• Vertex AI Explainable AI
  • Provide breakdown of how much each data feature contributed to a predictive model’s final output
• Unified Trace Viewer / Agent Evaluation
  • Tools to step through the exact multi-turn reasoning paths, tool calls, and prompt injection affecting active AI agents.

Underlying Technology

• Explainability Math
  • Uses advanced cooperative game theory and calculus algorithms like Shapley Values (SHAP) or Integrated Gradients to mathematically attribute weight to input parameters.
• Observability Telemetry
  • Standard Application Performance Monitoring (APM) patterns adapted for AI, capturing token generation rates, input / output logs, and latency traces.

Industry Alternatives

• Open source: SHAP/LIME libraries (for explanation); Arize AI, TruLens, LangSmith (for LLM tracing and evaluation).
• AWS: SageMaker Clarify (exxplainability) & Amazon Bedrock Model Evaluattion
• Azure: Azure AI Evaluation SDK & Azure Machine Learning Responsible AI Dashboard