Tag: #deepeval
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 5 posts
LLM Observability & Prompt Tools 2026 — Helicone / LangSmith / Langfuse / Braintrust / Athina / Comet Opik / Portkey Deep Dive
The 2026 LLM ops map. Helicone (YC) · LangSmith (LangChain) · Langfuse (open source, Series A) · W&B Weave · Arize Phoenix · Braintrust · Athina · Comet Opik (released March 2025) · Vellum · PromptHub · Portkey AI Gatewa
2026-05-16 · 22 min read #llm-observability#prompt-engineering#helicone#langsmith#langchainAI Safety, Evals and Red-Teaming in 2026 — Deep Dive into Inspect AI, Garak, PyRIT, Promptfoo, OpenAI Evals, lm-eval-harness
A single-page map of the 2026 AI safety, evaluation, and red-teaming ecosystem. Inspect AI (Anthropic, adopted by UK AISI), Garak (NVIDIA then independent), PyRIT (Microsoft), Promptfoo (YC), OpenAI Evals, lm-evaluation-
2026-05-16 · 22 min read #ai-safety#red-teaming#evaluation#inspect-ai#garakLLM Evaluation and Benchmarking Guide: Measuring What Actually Matters
A comprehensive guide to LLM evaluation and benchmarking. Covers standard academic benchmarks (MMLU, HELM, MT-Bench), production evaluation pipelines, LLM-as-judge, human evaluation, RAG evaluation, safety testing, and b
2026-03-17 · 18 min read #llm#evaluation#benchmark#mmlu#helmRAG Quality Evaluation and Failure Pattern Analysis: Diagnosing and Improving Retrieval-Augmented Generation
A systematic guide to evaluating RAG (Retrieval-Augmented Generation) system quality and analyzing common failure patterns. Covers evaluation metrics for Retriever, Reranker, and Generator components, framework compariso
2026-03-07 · 15 min read #rag#llm#evaluation#ragas#deepevalRAG Chatbot Evaluation in Practice: From Offline/Online Quality Measurement to Production Guardrails
Covers the evaluation system for reliably operating RAG chatbots in production. Addresses offline benchmarks, LLM-as-a-Judge, online experiments, alert thresholds, and regression prevention pipelines with code.
2026-03-04 · 16 min read #chatbot#rag#evaluation#llmops#production