View all study notes here

---

Vertex AI Platform (Gemini Enterprise Agent Platform)

Unified, managed platform for building, deploying, and operating AI/ML workloads. It spans the full lifecycle: data preparation -> model training -> deployment -> monitoring -> agent orchestration. Consolidates previously three separate products:

• Vertex AI
  • MLOps, model serving, training pipelines
• Agentspace
  • Enterprise search and agent workspace for employees
• Gemini Code Assist Enterprise
  • AI coding assistant and agent

Core Components

Vertex AI Studio / Agent Studio

• Interactive UI for experimenting with prompts, testing models, and designing generative AI applications.
• Used for prototyping before being to production.
• Supports text, image, video, and multimodal inputs.

Model Garden

• Curated catalogue of 200+ pre-trained models available for direct deployment or customization.
• Deploy pre-built model without training from scratch.
• Includes:
  • Google’s models
    • Gemini 3 Pro
      • Primary reasoning backbone for complex multistep orchestration, coding, heavy analytical tasks.
    • Gemini 3 Flash
      • Optimized for low latency, high frequency agentic workflows and real time processing.
    • Gemma
      • Light weight, open weights model family.
      • Use when customer requires local deployment or complete fine tuning control inside their own compute boundaries.
  • Partner models
    • Anthropic Claude, Meta Llama, Mistral, etc
  • Open source models
    • Available for download and self hosting.
• For custom prediction logic on a Model Garden model, create a custom container, push it to Artifact Registry, and deploy to a Vertex AI endpoint (not Cloud Run, nor Compute Engine)

Vertex AI Workbench / Colab Enterprise

• Managed Jupyter notebook environment for data science and model development.
• Colab Enterprise is the newer, more integrated version of Vertex AI Workbench.
• Where data scientists experiment with ML code in a managed, collaborative environment.

Vertex AI Pipelines (based on Kubeflow)

• Orchestration for ML workflows.
• Chaining together data prep, training, evaluation, and deployment steps as repeatable, versioned pipelines.
• MLOps backbone.
• Used to automate and version end to end ML training workflow.

Vertex AI Feature Store

• Centralized repository for ML features (engineered input variables).
• Provides consistent feature serving for both training and online prediction, preventing training-serving skew.
• Monitor latency to see how quickly a Feature Store application responds to users.

Model Registry

• Tracks all version of our trained models with metadata, lineage, and deployment history.
• Integrates with Vertex AI Endpoints for controlled rollouts.

Vertex AI Endpoints

• Managed infrastructure for serving predictions.
• Transform trained models into production ready services through managed infrastructure that handles serving.
• Supports:
  • Online prediction
    • Low latency, synchronous requests (REST/gRPC)
  • Batch prediction
    • Asynchronous, large scale jobs
  • Streaming prediction
    • For continuous data

Vertex AI Explainable AI

• Makes model predictions interpretable and auditable.
• Critical for regulated industries (finance, healthcare).
• Lives inside Model Registry and Endpoints, not as a separate product.
• Three feature attribution methods
  • Sampled Shapley
    • Best for tree based/ensemble models on tabular data.
  • Integrated Gradients
    • Best for differentiable models (neural nets) on tabular or image (pixel level data)
  • XRAI
    • Best for image region level explanations
• Use to make Vertex AI model more transparent without deploying a new service

---

Agent Builder & Grounding (RAG Architecture)

Vertex AI Agent Builder (now part of Agent Platform)

Tools to build conversational agents and search powered applications without needing to train custom models.

Key Components

• Search (formerly Vertex AI Search)
  • Connects enterprise data (documents, databases, websites) to LLMs so the model can answer questions based on your private data.
  • A key component in RAG
  • Supports unstructured data (PDFs, HTML, documents) and structured data (BigQuery tables).
• RAG Engine
  • A managed pipeline for RAG
  • Handles full RAG workflow
    • Document ingestion
    • Chunking
    • Embedding
    • Vector storage
    • Retrieval
    • Generation
• Vector Search (formerly Matching Engine)
  • The vector database layer
  • Stores high dimensional embeddings and performs approximate nearest neighbour (ANN) search to find semantically similar documents or data points.
  • Used in RAG pipelines, recommendation systems, and semantic search.

Standard RAG Architecture

e.g. building a customer support bot from private company documents:

1. Store source documents in Cloud Storage
2. Embed using Vertex AI Embedding API (convert text to vector)
3. Index vectors in Vertex AI Vector Search
4. Retrieve relevant chunks via semantic similarity query
5. Generate a response by passing retrieved context + user query to Gemini model Note: Do NOT route through Cloud Run, keep it within the Vertex AI / Agent Platform ecosystem using managed endpoints.

Gemini Cloud Assist

AI powered assistant integrated directly into Google Cloud Console. It provides:

• Architecture recommendations and best practice guidance.
• Natural language querying of your cloud resources.
• Assistance with writing Terraform, reviewing configurations, debugging
• Integrates with existing IAM context and audit logging.
• Leverage to design workflows without creating compliance or data residency issues.

---

AI Infrastructure - GPUs vs TPUs vs AI Hypercomputer

The Decision Framework

Scenario	Best Choice
Training large transformer / LLM models (BERT, Gemini, T5)	TPU
High-throughput inference at scale for GenAI	TPU (V5e, Ironwood)
General deep learning (TensorFlow, PyTorch, JAX)	GPU or TPU
Computer vision, image classification, object detection	GPU (typically NVIDIA A100, H100)
Research / experimentation with flexible frameworks	GPU
Lift-and-shift ML workloads, custom CUDA code	GPU
Large-scale recommendation systems	TPU
Cost-optimized inference serving for GenAI	TPU V5e or Ironwood
VM-level control + specialized hardware	Compute Engine with GPU/TPU attached

GPUs (Graphics Processing Units)

• Use case
  • Flexible, supports CUDA ecosystem, great for image/video workloads, research, and workloads with custom processing.
• Available on GCP
  • NVIDIA T4, V100, A100, H100 series
• Attach to
  • Compute Engine VMs, GKE nodes, Vertex AI training jobs
• When to choose GPU over TPU
  • When your framework relies on CUDA libraries, when you need maximum framework flexibility, or when your model isn’t optimized for XLA compilation.

TPUs (Tensor Processing Units)

Google’s custom ASICs purpose built for matrix multiplication (the dominate operation in neural network training and inference).

• Key versions
  • TPU v4
    • Large scale training for foundation models
    • Used by Google internally and partners like Anthropic and Cohere
  • TPU v5e
    • Cost optimized for inference at scale
    • Excellent price performance for serving GenAI
  • Ironwood
    • Newest generation, focused on cost efficient inference for very large generative AI models
• Programming
  • Uses Google’s XLA (Accelerated Linear Algebra) compiler
  • JAX is the preferred framwork
  • TensorFlow also natively supports TPUs
• When to choose TPU s GPU
  • Training large transformer models, high throughput LLM inference serving, when you want best performance per dollar for Google’s recommended model architectures
• Common pattern: “serving Gemma 3 on GKE with TPUs and vLLM”
• GKE can run TPU backed node pools for AI inference workloads.

AI Hypercomputer

Google’s architecture pattern for large scale AI training and inference. Not aa single product, it’s an integrated system combining:

• Hardware
  • TPU or GPU cluster (TPU v4/v5 pods, NVIDIA H100 clusters)
• Networking
  • Ultra high bandwidth interconnects between accelerators
• Storage
  • High throughput storage (Cloud Storage + Firestore for checkpointing)
• Software
  • Cluster Toolkit (formerly HPC Toolkit) for provisioning Slurm based clusters, XLA, JAX, and Google’s distributed training libraries
• Managed layer
  • Vertex AI as the orchestration surface
• Use AI Hypercomputer in scenarios involving training very large models (billions+ parameters) and the constraint is maximizing training throughput or minimizing training time.