Deploying ML Models to Production: FastAPI + Docker + Kubernetes

A trained ML model sitting in a Jupyter notebook delivers zero business value. This guide covers the complete path from trained model to a production-grade, scalable API endpoint.

Serving the Model with FastAPI

from fastapi import FastAPI
from pydantic import BaseModel
import joblib

app = FastAPI(title="Article Classifier API")
model = joblib.load("models/article_classifier.pkl")

class PredictRequest(BaseModel):
    text: str

class PredictResponse(BaseModel):
    category: str
    confidence: float

@app.post("/predict", response_model=PredictResponse)
async def predict(request: PredictRequest):
    proba = model.predict_proba([request.text])[0]
    cat_idx = proba.argmax()
    return PredictResponse(
        category=model.classes_[cat_idx],
        confidence=float(proba[cat_idx])
    )

Optimizing for Inference

For deep learning models, convert PyTorch models to ONNX format and serve with ONNX Runtime—typically 2-4x faster than native PyTorch inference. For even higher throughput, use NVIDIA Triton Inference Server with model batching and GPU acceleration.

Kubernetes Deployment with GPU Nodes

resources:
  requests:
    nvidia.com/gpu: "1"
  limits:
    nvidia.com/gpu: "1"

Health Checks and Model Metadata

Always expose /health and /metrics endpoints. Include model version, training date, and feature schema in the /info endpoint. This makes debugging production issues dramatically easier when multiple model versions are deployed.

Production Multi-Stage Dockerfile Blueprint

Below is a secure, multi-stage production Dockerfile designed to minimize image size and eliminate security vulnerabilities by running as a non-privileged user:

# Stage 1: Build virtual env
FROM python:3.11-slim AS builder
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends gcc libpq-dev && rm -rf /var/lib/apt/lists/*
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Stage 2: Final lightweight image
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends libpq-dev && rm -rf /var/lib/apt/lists/*
COPY --from=builder /opt/venv /opt/venv
COPY . .
ENV PATH="/opt/venv/bin:$PATH"
ENV PYTHONUNBUFFERED=1
RUN useradd -u 10001 appuser && chown -R appuser:appuser /app
USER 10001
EXPOSE 8080
CMD ["gunicorn", "--bind", "0.0.0.0:8080", "run:app"]

Model Performance & Retrieval Profiles

Below is the performance comparison profile for our processing pipeline tested in staging against sanitized validation datasets:

US & UK Regulatory Standards for Artificial Intelligence

Deploying machine learning models in the US and UK markets requires strict alignment with local regulatory frameworks. In the United States, applications must respect the guidelines set by the FTC regarding algorithmic transparency, alongside the Executive Order on Safe, Secure, and Trustworthy AI. In the United Kingdom, AI systems must comply with the UK General Data Protection Regulation (UK GDPR), which enforces strict rules on automated profiling (under Article 22). Conducting bias auditing and maintaining explainable decision paths is critical to avoiding compliance sanctions in both jurisdictions.