LabHub

Blog

MLOps Pipeline Design

2026-03-01 · 26 min read · #mlops#ml-pipeline#production#mlflow

한국어English日本語

1. What Is MLOps

1.1 Definition and Background

MLOps (Machine Learning Operations) is a set of practices for systematically managing the development, deployment, and operation of ML models. Just as DevOps was born to close the gap between software development and operations, MLOps aims to close the gap between ML model development and production operations.

Google Cloud's official MLOps guide defines MLOps as follows:

MLOps is an ML engineering culture and practice that aims at unifying ML system development (Dev) and ML system operation (Ops).

In a traditional software system you only have to manage code, but in an ML system you have to manage three axes at once: code, data, and models. That fundamental difference is what makes MLOps a separate discipline.

1.2 Key Differences from DevOps

DevOps and MLOps share the same principles of automation, CI/CD, and monitoring, but a number of differences arise from the characteristics unique to ML systems.

AspectDevOpsMLOps
What is managedCodeCode + data + models
TestingUnit/integration testsData validation + model validation + code tests
DeploymentService/application deploymentModel deployment + prediction service deployment
MonitoringSystem performance, error rateModel performance decay, Data Drift, Concept Drift
Rollback basisWhen an error occursBased on model performance metrics
Version controlCode version controlCode + data + model + hyperparameter version control
ReproducibilityBuild reproductionExperiment reproduction (including data, environment, parameters)

1.3 Technical Debt in ML Systems: Analyzing the Google Paper

The paper "Hidden Technical Debt in Machine Learning Systems" (Sculley et al., NeurIPS 2015), published by a Google research team in 2015, is the key paper that systematically laid out the technical debt problem in ML systems. Its core message is as follows:

In a real production ML system, the ML code is only a tiny fraction of the whole system. Most of the rest consists of surrounding infrastructure: data collection, validation, feature extraction, configuration management, monitoring, serving infrastructure, and so on.

The types of technical debt unique to ML systems that the paper identifies are as follows:

It is to manage this technical debt systematically that an MLOps pipeline is needed.

2. Google MLOps Maturity Model

The MLOps maturity model presented by the Google Cloud Architecture Center consists of 3 stages, from Level 0 to Level 2. Each level is distinguished by how automated the ML pipeline is and how far CI/CD is integrated.

2.1 Level 0: Manual Process

Level 0 is where most ML teams start, with every step performed manually.

Characteristics:

The core problem: at Level 0, model performance decays over time once the model is in production, but there is no mechanism to detect it. The Google guide states that "models can decay in more ways than conventional software systems" and stresses that changes in the data profile are the main cause of model performance decay.

2.2 Level 1: ML Pipeline Automation

The goal of Level 1 is Continuous Training (CT) --- that is, automating the ML pipeline so the model can be trained continuously.

Core building blocks:

The key difference from Level 0: at Level 0 you deploy a trained "model", while at Level 1 you deploy a "pipeline". The same pipeline runs symmetrically in the development and production environments, and each stage of the pipeline is modularized and containerized.

2.3 Level 2: CI/CD Pipeline Automation

Level 2 is the stage where updates to the ML pipeline itself are automated as well. Where Level 1 automated continuous training (CT) of the model, Level 2 also covers continuous integration (CI) and continuous deployment (CD) of the pipeline code.

The 6 pipeline stages:

  1. Development and experimentation: new algorithms and modeling techniques are tried out in orchestrated steps
  2. Continuous Integration (CI): source code is built, tested, and packaged. This includes unit tests of Feature Engineering logic, verification that model training converges, and component integration tests
  3. Continuous Delivery (CD): pipeline artifacts are deployed to the target environment
  4. Automated triggers: the pipeline is executed automatically in the production environment
  5. Model CD: the trained model is deployed as a prediction service
  6. Monitoring: model performance and pipeline efficiency are tracked continuously

Deployment strategy:

Test items included in CI:

3. Data Management

3.1 Feature Store: Feast

A Feature Store is the system that manages and serves Features centrally in an ML pipeline. Feast (Feature Store) is a leading open-source Feature Store whose core goal is to solve the Feature consistency problem between training and serving (Training-Serving Skew).

Feast's core architecture:

                    +-------------------+
                    |   Feature Store   |
                    |     (Feast)       |
                    +-------------------+
                   /                     \
    +------------------+        +------------------+
    |  Offline Store   |        |  Online Store    |
    |  (Historical)    |        |  (Low-latency)   |
    +------------------+        +------------------+
    | - BigQuery       |        | - Redis          |
    | - Redshift       |        | - DynamoDB       |
    | - Snowflake      |        | - Datastore      |
    | - File (Parquet) |        | - SQLite         |
    +------------------+        +------------------+
           |                            |
    Model Training               Model Serving
    (Batch Feature Retrieval)    (Online Feature Retrieval)

Why use Feast:

  1. Preventing Training-Serving Skew: it guarantees that the Feature transformation logic used at training time is identical to the logic used at serving time
  2. Feature reusability: a Feature defined once can be reused across several models
  3. Point-in-time Correctness: Feature values as of a past point in time can be retrieved accurately, preventing Data Leakage
  4. Feature Discovery: available Features can be searched and explored from the central store

Example Feast Feature definition:

from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64
from datetime import timedelta

# Define the data source
driver_stats_source = FileSource(
    path="data/driver_stats.parquet",
    timestamp_field="event_timestamp",
)

# Define the Entity
driver = Entity(
    name="driver_id",
    description="Driver identifier",
)

# Define the Feature View
driver_stats_fv = FeatureView(
    name="driver_hourly_stats",
    entities=[driver],
    ttl=timedelta(hours=2),
    schema=[
        Field(name="conv_rate", dtype=Float32),
        Field(name="acc_rate", dtype=Float32),
        Field(name="avg_daily_trips", dtype=Int64),
    ],
    source=driver_stats_source,
)

3.2 Data Versioning: DVC

DVC (Data Version Control) is an open-source tool that versions data and ML pipelines on top of Git. Just as Git manages code, DVC efficiently versions large datasets, model files, and intermediate outputs.

DVC's core principle:

Only the metadata about data files (the .dvc files and dvc.yaml) is committed to the Git repository, while the actual data is stored in Remote Storage (S3, GCS, Azure Blob, and so on).

# Initialize DVC
dvc init

# Start tracking a data file
dvc add data/training_dataset.csv

# Only the metadata files are committed to Git
git add data/training_dataset.csv.dvc data/.gitignore
git commit -m "Add training dataset v1"

# Configure Remote Storage
dvc remote add -d myremote s3://my-bucket/dvc-storage
dvc push

DVC Pipeline definition (dvc.yaml):

stages:
  preprocess:
    cmd: python src/preprocess.py
    deps:
      - src/preprocess.py
      - data/raw/
    outs:
      - data/processed/

  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/processed/
    outs:
      - models/model.pkl
    metrics:
      - metrics/scores.json:
          cache: false

  evaluate:
    cmd: python src/evaluate.py
    deps:
      - src/evaluate.py
      - models/model.pkl
      - data/test/
    metrics:
      - metrics/eval_results.json:
          cache: false

A DVC pipeline plays the role of a "Makefile" for an ML project. Because each stage's inputs, outputs, and dependencies are declared explicitly, the single command dvc repro reproduces the whole pipeline.

4. Experiment Tracking

4.1 MLflow Tracking

MLflow is an open-source platform for managing the entire lifecycle of ML experiments, developed by Databricks. MLflow Tracking is its core experiment management component, systematically recording and comparing parameters, metrics, artifacts, and code versions.

Core concepts of MLflow Tracking:

MLflow Tracking usage example:

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split

# Set the Tracking Server URI
mlflow.set_tracking_uri("http://mlflow-server:5000")
mlflow.set_experiment("fraud-detection-experiment")

# Run the experiment
with mlflow.start_run(run_name="rf_baseline_v1"):
    # Log the hyperparameters
    params = {
        "n_estimators": 100,
        "max_depth": 10,
        "min_samples_split": 5,
        "random_state": 42
    }
    mlflow.log_params(params)

    # Train the model
    model = RandomForestClassifier(**params)
    model.fit(X_train, y_train)

    # Predict and compute metrics
    y_pred = model.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    f1 = f1_score(y_test, y_pred, average='weighted')

    # Log the metrics
    mlflow.log_metrics({
        "accuracy": accuracy,
        "f1_score": f1
    })

    # Log the model artifact
    mlflow.sklearn.log_model(model, "random_forest_model")

    # Log additional artifacts (the confusion matrix image and so on)
    mlflow.log_artifact("plots/confusion_matrix.png")

4.2 Weights & Biases (W&B)

Weights & Biases is an integrated MLOps platform for experiment tracking, hyperparameter optimization, and model management. It offers functionality similar to MLflow, but its strengths lie especially in real-time dashboards and team collaboration features.

W&B's key features:

import wandb

# Initialize W&B
wandb.init(
    project="fraud-detection",
    config={
        "learning_rate": 0.001,
        "epochs": 50,
        "batch_size": 32,
        "architecture": "ResNet50"
    }
)

# Log metrics inside the training loop
for epoch in range(wandb.config.epochs):
    train_loss = train_one_epoch(model, train_loader)
    val_loss, val_acc = evaluate(model, val_loader)

    wandb.log({
        "epoch": epoch,
        "train_loss": train_loss,
        "val_loss": val_loss,
        "val_accuracy": val_acc
    })

wandb.finish()

MLflow vs W&B comparison:

ItemMLflowW&B
LicenseOpen source (Apache 2.0)Freemium (free for individuals)
HostingSelf-hosted / ManagedCloud (SaaS)
Real-time dashboardBasicAdvanced and interactive
Team collaborationLimitedStrong collaboration features
Model RegistryBuilt inManaged through Artifacts
Hyperparameter TuningNot supported (needs a separate tool)Sweeps built in

5. Model Registry and Version Management

5.1 MLflow Model Registry

The MLflow Model Registry is the component that centrally manages the entire lifecycle of an ML model. It systematically supports model version management, stage transitions, and approval workflows.

Core concepts of the Model Registry:

Model Registry workflow:

Development (Experiment) --> Model registration --> Staging --> Production
                         |              |           |
                     Version 1      Test/validate    Serve
                     Version 2        A/B test
                     Version 3

MLflow Model Registry usage example:

import mlflow
from mlflow import MlflowClient

client = MlflowClient()

# Train and log the model (done in Tracking)
with mlflow.start_run() as run:
    mlflow.sklearn.log_model(model, "model")
    model_uri = f"runs:/{run.info.run_id}/model"

# Register in the Model Registry
model_version = mlflow.register_model(
    model_uri=model_uri,
    name="fraud-detection-model"
)

# Set the Alias (designate it as champion)
client.set_registered_model_alias(
    name="fraud-detection-model",
    alias="champion",
    version=model_version.version
)

# Load the champion model in production
champion_model = mlflow.pyfunc.load_model(
    model_uri="models:/fraud-detection-model@champion"
)
predictions = champion_model.predict(input_data)

Why use a Model Registry:

  1. Audit Trail: you can trace the full history of which model was deployed to production, when, by whom, and from which experiment
  2. Easy rollback: when a model performance problem arises you can roll back to the previous version immediately
  3. Approval workflow: you can require a model to go through review and approval before it is deployed to production
  4. Reproducibility: by tracking which Run each model version is linked to, you can reproduce the training data, parameters, and code exactly

6. CI/CD for ML

6.1 Differences Between Traditional CI/CD and ML CI/CD

CI/CD in an ML project has to bring not only code but also data and models into the pipeline. This is called CI/CD/CT (Continuous Integration / Continuous Delivery / Continuous Training).

6.2 CML (Continuous Machine Learning)

CML is an open-source tool from Iterative.ai (the makers of DVC) that integrates with GitHub Actions and GitLab CI to implement CI/CD for ML projects.

CML's key features:

GitHub Actions + CML workflow example:

# .github/workflows/ml-pipeline.yml
name: ML Pipeline CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  train-and-evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - uses: iterative/setup-cml@v2

      - name: Install dependencies
        run: |
          pip install -r requirements.txt

      - name: Pull data with DVC
        run: |
          dvc pull
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Train model
        run: |
          dvc repro

      - name: Create CML report
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # Compare metrics
          echo "## Model Performance Report" >> report.md
          echo "" >> report.md
          dvc metrics diff --md >> report.md
          echo "" >> report.md

          # Add the visualization
          echo "## Training Curves" >> report.md
          cml asset publish plots/training_curve.png --md >> report.md
          echo "" >> report.md
          echo "## Confusion Matrix" >> report.md
          cml asset publish plots/confusion_matrix.png --md >> report.md

          # Post it as a comment on the PR
          cml comment create report.md

This workflow trains the model automatically whenever a Pull Request is opened and posts the performance metrics and visualizations as a PR comment. A code reviewer can then review not just the code change but the change in model performance.

6.3 Testing Strategy in an ML Pipeline

On top of traditional software testing, an ML project needs the following kinds of ML-specific testing.

+----------------------------------------------------+
|              ML Testing Pyramid                     |
|                                                     |
|                  /\                                  |
|                 /  \     End-to-End Pipeline Test    |
|                /----\                                |
|               / Model \   Model Quality Tests       |
|              /  Quality \  (Performance Threshold)   |
|             /------------\                           |
|            /   Training    \  Training Tests         |
|           /   Convergence   \ (NaN, Inf, Overfit)    |
|          /-------------------\                       |
|         /   Data Validation    \ Schema, Statistics  |
|        /     & Feature Tests    \                    |
|       /-------------------------\                    |
|      /     Unit Tests (Code)      \  Standard Unit   |
|     /______________________________\  Tests          |
+----------------------------------------------------+

7. Model Serving Patterns

Once a model is trained and validated, it has to be served so it can deliver predictions to real users. The serving style and deployment strategy are decided by business requirements and system characteristics.

7.1 Batch Serving vs Real-time Serving

Batch Serving:

# Batch Serving example: Apache Spark + MLflow
from pyspark.sql import SparkSession
import mlflow

spark = SparkSession.builder.appName("batch-prediction").getOrCreate()

# Load the production model
model = mlflow.pyfunc.load_model("models:/fraud-detection@champion")

# Batch prediction over a large volume of data
input_df = spark.read.parquet("s3://data-lake/daily/transactions/")
predictions = model.predict(input_df.toPandas())

# Save the results
result_df = input_df.toPandas()
result_df["prediction"] = predictions
spark.createDataFrame(result_df).write.parquet("s3://data-lake/predictions/")

Real-time Serving:

7.2 Deployment Strategies

A/B Testing:

Two or more model versions are deployed to production at the same time, traffic is split between them, and their effect on business metrics is compared statistically.

User request ──────> Load Balancer
                        |
              +---------+---------+
              |                   |
         Model A (70%)       Model B (30%)
         (current model)     (new model)
              |                   |
         return response     return response
              |                   |
              +----> collect metrics <----+
                        |
                  statistical significance analysis

Shadow Deployment:

The new model receives a copy of production traffic and produces predictions, but those predictions are not used in the actual response. Only the existing model serves the real response.

User request ──────> existing model (Production)
       |                    |
       |               returns the real response
       |
       +----------> new model (Shadow)
                         |
                    stores the prediction (not used in the response)
                         |
                    performance comparison analysis

Canary Deployment:

The new model is rolled out first to a small set of users (for example, 5% of total traffic), and if nothing goes wrong the traffic share is raised gradually.

Phase 1:  existing model 95% ──|── new model 5%     (monitoring)
Phase 2:  existing model 75% ──|── new model 25%    (validation)
Phase 3:  existing model 50% ──|── new model 50%    (confirmation)
Phase 4:  existing model 0%  ──|── new model 100%   (complete)

8. Monitoring: Drift Detection

Monitoring after a model reaches production is the most important area in MLOps, and often the most overlooked. Model performance will inevitably degrade over time, and the key is detecting that early and responding to it.

8.1 Types of Drift

Data Drift:

This is when the statistical distribution of the input data diverges from the distribution of the data used in training. The model itself has not changed, but the characteristics of the input data have, and the model's prediction accuracy degrades.

Concept Drift:

This is when the relationship between the input (X) and the output (Y) itself changes. The correct answer for the same input has changed.

Model Drift:

This is when the model's prediction performance measures (accuracy, precision, recall, and so on) degrade over time. It often appears as a consequence of Data Drift or Concept Drift.

8.2 Drift Monitoring with Evidently AI

Evidently AI is an open-source Python library for monitoring ML model and data quality. It provides more than 20 prebuilt Drift detection methods and more than 100 built-in metrics.

Evidently's key features:

Evidently Data Drift detection example:

import pandas as pd
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, DataQualityPreset
from evidently.metrics import DataDriftTable

# Load the training data (reference) and the production data (current)
reference_data = pd.read_csv("data/training_data.csv")
current_data = pd.read_csv("data/production_data_latest.csv")

# Generate the Data Drift report
drift_report = Report(metrics=[
    DataDriftPreset(),
    DataQualityPreset(),
])

drift_report.run(
    reference_data=reference_data,
    current_data=current_data
)

# Save the HTML report
drift_report.save_html("reports/data_drift_report.html")

# Access the results programmatically
report_dict = drift_report.as_dict()
dataset_drift = report_dict["metrics"][0]["result"]["dataset_drift"]

if dataset_drift:
    print("Dataset Drift detected! Triggering model retraining.")
    # Retraining pipeline trigger logic

Setting up an automated monitoring pipeline:

from evidently.test_suite import TestSuite
from evidently.tests import (
    TestShareOfDriftedColumns,
    TestColumnDrift,
    TestShareOfMissingValues
)

# Define the test suite (for CI/CD integration)
data_stability_tests = TestSuite(tests=[
    TestShareOfDriftedColumns(lt=0.3),  # fewer than 30% of columns may have drifted
    TestColumnDrift(column_name="transaction_amount"),
    TestColumnDrift(column_name="user_age"),
    TestShareOfMissingValues(lt=0.05),  # missing value proportion under 5%
])

data_stability_tests.run(
    reference_data=reference_data,
    current_data=current_data
)

# Check the test results
test_results = data_stability_tests.as_dict()
all_passed = all(
    test["status"] == "SUCCESS"
    for test in test_results["tests"]
)

if not all_passed:
    # Send an alert and trigger retraining
    send_alert("Data stability test failed!")

8.3 Establishing a Monitoring Strategy

Effective ML monitoring calls for a multi-layered approach.

Monitoring layerTargetToolsCadence
InfrastructureCPU/GPU utilization, memory, networkPrometheus, GrafanaReal time
ServiceResponse latency, error rate, throughputPrometheus, DatadogReal time
Data qualitySchema, missing values, outliersEvidently, Great ExpectationsBatch/real time
Data DriftFeature distribution changesEvidently, NannyMLDaily/weekly
Model performanceAccuracy, F1, AUC, and so onEvidently, W&BDaily/weekly
Business metricsConversion rate, revenue, click-through rateIn-house analytics toolsDaily/weekly

9. Full Architecture Diagram

Below is a complete MLOps pipeline architecture corresponding to MLOps Level 2.

┌────────────────────────────────────────────────────────────────────┐
│                        MLOps Pipeline Architecture                 │
├────────────────────────────────────────────────────────────────────┤
│                                                                    │
│  ┌─────────────┐    ┌──────────────┐    ┌──────────────────────┐  │
│  │ Data Sources │───>│ Data Pipeline │───>│ Feature Store (Feast)│  │
│  │ (DB, API,   │    │ (Ingestion,  │    │ ┌────────┐┌────────┐│  │
│  │  Streaming) │    │  Validation, │    │ │Offline ││Online  ││  │
│  └─────────────┘    │  Transform)  │    │ │Store   ││Store   ││  │
│        │            └──────────────┘    │ └────────┘└────────┘│  │
│        │                   │            └──────────────────────┘  │
│        │                   │                   │          │       │
│        ▼                   ▼                   ▼          │       │
│  ┌──────────┐    ┌────────────────┐    ┌────────────┐    │       │
│  │   DVC    │    │  Data Version  │    │  Training   │    │       │
│  │ (Data    │    │  & Validation  │    │  Pipeline   │    │       │
│  │ Version) │    │  (Evidently)   │    │             │    │       │
│  └──────────┘    └────────────────┘    └──────┬─────┘    │       │
│                                               │          │       │
│                                               ▼          │       │
│  ┌──────────────────────────────────────────────────┐    │       │
│  │           Experiment Tracking                     │    │       │
│  │    ┌─────────────┐    ┌──────────────────┐       │    │       │
│  │    │   MLflow     │    │   Weights &      │       │    │       │
│  │    │  Tracking    │    │   Biases          │       │    │       │
│  │    └──────┬──────┘    └──────────────────┘       │    │       │
│  └───────────┼──────────────────────────────────────┘    │       │
│              ▼                                           │       │
│  ┌──────────────────┐   ┌───────────────────────────┐    │       │
│  │  Model Registry  │   │    CI/CD Pipeline          │    │       │
│  │  (MLflow)        │──>│  (GitHub Actions + CML)    │    │       │
│  │  ┌────────────┐  │   │  ┌─────┐ ┌────┐ ┌─────┐  │    │       │
│  │  │ @champion  │  │   │  │ CI  │ │ CD │ │ CT  │  │    │       │
│  │  │ @challenger│  │   │  └─────┘ └────┘ └─────┘  │    │       │
│  │  └────────────┘  │   └────────────┬──────────────┘    │       │
│  └──────────────────┘                │                   │       │
│                                      ▼                   │       │
│  ┌───────────────────────────────────────────────────────┐│       │
│  │              Model Serving Layer                      ││       │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────────────┐   │▼       │
│  │  │  Batch   │  │Real-time │  │ Deployment Strategy│   │       │
│  │  │ Serving  │  │ Serving  │  │ - A/B Testing     │   │       │
│  │  │ (Spark)  │  │ (REST/   │  │ - Canary          │   │       │
│  │  │          │  │  gRPC)   │  │ - Shadow          │   │       │
│  │  └──────────┘  └──────────┘  └──────────────────┘   │       │
│  └───────────────────────────────────────────────────────┘       │
│                            │                                     │
│                            ▼                                     │
│  ┌───────────────────────────────────────────────────────┐       │
│  │              Monitoring Layer                          │       │
│  │  ┌────────────┐ ┌──────────────┐ ┌─────────────────┐ │       │
│  │  │ Data Drift │ │ Model Perf   │ │ Infrastructure  │ │       │
│  │  │(Evidently) │ │ Monitoring   │ │ (Prometheus/    │ │       │
│  │  │            │ │              │ │  Grafana)       │ │       │
│  │  └──────┬─────┘ └──────┬───────┘ └─────────────────┘ │       │
│  └─────────┼──────────────┼─────────────────────────────┘       │
│            │              │                                      │
│            ▼              ▼                                      │
│  ┌──────────────────────────────┐                                │
│  │   Alert & Retrain Trigger   │──── trigger the retraining pipeline │
│  └──────────────────────────────┘                                │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

Data flow between the architecture components:

  1. Data collection and preprocessing: data is collected from a variety of sources (DB, API, streaming) and preprocessed through a data pipeline. DVC versions the data and Evidently validates data quality.

  2. Feature Engineering: the preprocessed data is stored in the Feature Store (Feast). The Offline Store serves Features for training and the Online Store serves Features for serving.

  3. Model training and experiment management: the model is trained in the Training Pipeline and the experiment is tracked with MLflow Tracking or W&B. Parameters, metrics, and artifacts are recorded systematically.

  4. Model registration and deployment: the validated model is registered in the MLflow Model Registry and automatically tested, validated, and deployed through the CI/CD pipeline (GitHub Actions + CML).

  5. Model Serving: predictions are served through Batch Serving (Spark) or Real-time Serving (REST/gRPC), applying deployment strategies such as A/B Testing, Canary, and Shadow.

  6. Monitoring and retraining: Evidently monitors Data Drift and model performance while Prometheus/Grafana monitor the infrastructure. When an anomaly is detected, the retraining pipeline is triggered automatically.

The key thing in this architecture is the feedback loop (Feedback Loop). Problems detected by monitoring trigger retraining automatically, and the retrained model is validated, registered, and deployed again - that cycle is the essence of MLOps.

10. Conclusion: A Guide to Adopting MLOps

For a team adopting MLOps for the first time, an incremental approach is recommended.

Phase 1 - Laying the foundation (starting at Level 0):

Phase 2 - Automating the pipeline (reaching Level 1):

Phase 3 - Integrating CI/CD (reaching Level 2):

At each stage it matters to choose tools and processes that fit the organization's ML maturity, team size, and business requirements. Rather than trying to build a perfect MLOps system from the outset, concentrating on the biggest current bottleneck and expanding gradually is the realistic and effective strategy.

References

  1. Google Cloud, "MLOps: Continuous delivery and automation pipelines in machine learning" - https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
  2. Google Cloud, "Practitioners Guide to MLOps" - https://cloud.google.com/resources/mlops-whitepaper
  3. Sculley et al., "Hidden Technical Debt in Machine Learning Systems", NeurIPS 2015 - https://papers.neurips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
  4. MLflow Official Documentation - https://mlflow.org/docs/latest/
  5. MLflow Tracking - https://mlflow.org/docs/latest/ml/tracking/
  6. MLflow Model Registry - https://mlflow.org/docs/latest/ml/model-registry/
  7. Feast Official Documentation - https://docs.feast.dev
  8. DVC Official Documentation - https://doc.dvc.org/
  9. DVC Data Pipelines - https://doc.dvc.org/start/data-pipelines/data-pipelines
  10. CML (Continuous Machine Learning) - https://cml.dev/
  11. Iterative CML GitHub - https://github.com/iterative/cml
  12. Evidently AI Official - https://www.evidentlyai.com
  13. Evidently AI GitHub - https://github.com/evidentlyai/evidently
  14. Evidently AI, "What is data drift in ML" - https://www.evidentlyai.com/ml-in-production/data-drift
  15. Evidently AI, "What is concept drift in ML" - https://www.evidentlyai.com/ml-in-production/concept-drift
  16. Weights & Biases Documentation - https://docs.wandb.ai
  17. Weights & Biases Experiment Tracking - https://wandb.ai/site/experiment-tracking/

Read the original: https://www.youngju.dev/blog/mlops/mlops_pipeline_design.en

Comments

No comments yet.

Sign in to leave a comment