MLOps Best Practices
What is MLOps?
MLOps is DevOps principles — automation, version control, continuous integration and delivery, testing, and monitoring — applied to the machine learning lifecycle. The goal is the same as DevOps: ship reliable systems quickly and repeatably. But ML breaks several assumptions that ordinary software engineering quietly relies on.
The one idea to take away
In traditional software, behavior is fully determined by code. In ML, behavior depends on code + data + model. All three must be versioned and reproducible, because a prediction is a function of which code ran, on which data, to produce which model artifact. Change any one and the system behaves differently.
That extra coupling is why you cannot just "git push" an ML system and call it done. The data changes on its own — the world drifts — so even with frozen code the model silently decays. MLOps exists to make this whole moving system reproducible, testable, and self-healing.
Why ML is harder than normal software
| Concern | Traditional software | Machine learning |
|---|---|---|
| What to version | Code | Code + data + model + config |
| Reproducibility | Same input → same output | Needs pinned data version, seed, and environment too |
| Testing | Unit / integration tests | Also data validation + model validation (behavioral) |
| Decay over time | Stable unless code changes | Degrades on its own as data drifts |
| CI/CD | Continuous Integration / Delivery | Plus Continuous Training (CT) |
The ML lifecycle is a loop, not a line
A common mistake is to picture ML as a one-way assembly line that ends at deployment. In reality it is a continuous loop. After deployment, monitoring watches the live model and feeds signals back to retraining, which re-enters the pipeline. The stages:
data ingestion → data validation → feature engineering → training → evaluation → deployment → monitoring → (drift / SLO breach) → retraining → …back to training
The two arrows that matter most are the gates and the loop-back. Validation and evaluation are gates: they can stop a run cold so bad data or a worse model never reaches users. Monitoring → retraining is the loop-back that makes the system self-healing — this automated retraining is what people mean by Continuous Training (CT), the extra "C" that ML adds to CI/CD.
Interactive: walk the MLOps lifecycle
Click any stage to see what happens there, the tools used, and the failure mode it prevents. Then run a scenario and watch a job flow through the pipeline. Try injecting a problem — bad data caught at validation, a candidate model blocked at the evaluation gate, or data drift in production that triggers an automatic retrain looping back to training.
Click any stage to inspect it. Run a scenario to watch a job flow through the loop — green = passed, purple = running, red = gated/blocked. The dashed purple arc is the automated retrain loop.
Data Validation
Check the incoming data against an expected schema and statistics: column types, ranges, null rates, category sets, and distribution vs. the previous batch. Bad data is gated out before it ever reaches training.
Garbage-in / garbage-out. Catches schema drift, broken upstream pipelines, and silent data corruption before they poison the model.
Why it loops
A model is never "done." The world drifts away from the training data, so monitoring continuously feeds signals back into retraining. The gates (validation, evaluation) are what keep that automation safe.
Notice the gates
"Inject bad data" never reaches training, and "fails the gate" never reaches users. Automation is only safe because validation and evaluation can halt it.
Notice the loop
"Trigger data drift" shows monitoring firing a retrain that flows back into training with no human in the loop. That feedback arc is the heart of MLOps.
The core practices
Experiment tracking
Log every run's hyperparameters, code commit, data version, metrics, and artifacts so experiments are comparable and the "best" model is reproducible. MLflow, Weights & Biases.
Data & model versioning + a registry
Version data like code (DVC, LakeFS) and register models with stage tags (staging / production / archived) in a model registry so promotion and rollback are deliberate and auditable.
Reproducible pipelines
Express the lifecycle as code (Airflow, Dagster, Kubeflow) with pinned environments, so a run can be re-executed identically rather than reconstructed from a notebook by memory.
Feature store
Compute each feature's logic once and serve it to both training and online inference. This is the canonical defense against train/serve skew — the bug where the model sees differently-computed features in production than it trained on. Feast, Tecton.
CI / CD / CT automation
CI tests code and pipelines, CD ships the model service, and CT automatically retrains on new data or drift triggers. The whole loop runs without a human babysitting it — gated by the tests below.
Testing — beyond unit tests
ML needs data validation (schema, ranges, distribution checks) and model validation (is the candidate better than production? does it pass on critical slices? is it fair?) in addition to ordinary unit and integration tests.
Monitoring & automated retraining triggers
Track data drift, prediction drift, latency, and (when labels arrive) live accuracy. When a metric crosses a threshold, fire a retraining trigger automatically. See the model monitoring and deployment lessons for the details.
Hidden technical debt in ML systems
The influential paper Hidden Technical Debt in Machine Learning Systems (Sculley et al., Google, 2015) makes a point every ML engineer eventually feels: the model training code is a tiny box in a much larger system. Surrounding it are configuration, data collection, feature extraction, serving infrastructure, monitoring, and process- management glue — and that glue is where most of the debt accumulates.
Adapted from Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015). The actual ML code (blue) is a tiny fraction of a real production ML system.
ML systems incur all the usual software debt plus ML-specific kinds:
- Glue code & pipeline jungles — sprawling one-off scripts that stitch data and models together, brittle and hard to test.
- Configuration debt — huge, fragile config files where a single wrong flag silently changes model behavior.
- Entanglement (CACE) — "Changing Anything Changes Everything." Because features and signals interact, you cannot tweak one input in isolation.
- Undeclared consumers & data dependencies — downstream systems quietly depend on a model's outputs or on input signals no one is tracking.
- Feedback loops — the model's own predictions influence the future data it trains on, which is hard to reason about.
The practical lesson: most of MLOps is about taming the system around the model — versioning, validation gates, monitoring, and reproducible pipelines — not about the modeling itself.
MLOps maturity levels
Teams adopt MLOps gradually. A widely used framing (Google Cloud) describes three levels by how much of the lifecycle is automated:
| Level | What is automated | Characteristics |
|---|---|---|
| Level 0 Manual | Nothing — notebook-driven | Manual, script-and-handoff process; rare retraining; training/serving disconnected; no CI/CD, little monitoring. |
| Level 1 Pipeline automation | The training pipeline (enables CT) | Automated, reproducible training pipeline; retraining on new data via triggers; feature store; data & model validation gates in the pipeline. |
| Level 2 CI/CD pipeline automation | The pipeline itself, via CI/CD | New pipeline code is automatically tested, built, and deployed; full CI/CD + CT; rapid, reliable, frequent updates to pipelines and models in production. |
The point of the ladder is not that everyone should be at Level 2 — it is to match your investment to your needs. A model retrained twice a year does not need full CI/CD; a fraud model retrained nightly absolutely does.
Common pitfalls
✗ Anti-patterns
- • Tracking code in git but not data or model artifacts
- • Computing features differently in training vs. serving (skew)
- • Deploying once and never monitoring for drift
- • Only unit-testing code; never validating data or the model
- • Retraining manually and ad-hoc, when someone remembers
- • Notebooks as production; no reproducible pipeline
✓ Best practices
- • Version code + data + model together; use a registry
- • Serve features from a shared feature store
- • Monitor drift, latency, and accuracy continuously
- • Add data-validation and model-validation gates
- • Automate retraining with triggers (CT)
- • Express the lifecycle as reproducible pipeline code
Key Takeaways
- MLOps is DevOps for ML — but ML behavior depends on code + data + model, so all three must be versioned and reproducible.
- The ML lifecycle is a continuous loop: data → validate → features → train → evaluate → deploy → monitor → retrain, not a one-way line.
- Validation and evaluation are gates that keep automated pipelines safe by blocking bad data and worse models.
- A feature store serves identical features to training and serving, preventing train/serve skew.
- ML adds Continuous Training (CT) to CI/CD: monitoring triggers automatic retraining when the world drifts.
- Most of an ML system is the glue around the model — "hidden technical debt" lives in config, data, and serving infrastructure, not the modeling code.
- MLOps maturity is a ladder (manual → pipeline automation → CI/CD automation); match your automation to how often the model must change.