Tag: #langsmith
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 5 posts
LLM Observability & Prompt Tools 2026 — Helicone / LangSmith / Langfuse / Braintrust / Athina / Comet Opik / Portkey Deep Dive
The 2026 LLM ops map. Helicone (YC) · LangSmith (LangChain) · Langfuse (open source, Series A) · W&B Weave · Arize Phoenix · Braintrust · Athina · Comet Opik (released March 2025) · Vellum · PromptHub · Portkey AI Gatewa
2026-05-16 · 22 min read #llm-observability#prompt-engineering#helicone#langsmith#langchainAgent Evaluation Systems in 2026 — Inspect AI vs Promptfoo vs Phoenix vs LangSmith vs OpenAI Evals (You're Measuring the Agent, Not the Model)
LLM evals measure the model. Agent evals measure whether the model plus the harness plus the tools actually carry a task to completion. They are different problems. This is a map of the 2026 landscape — Inspect AI from U
2026-05-14 · 20 min read #agent-evaluation#inspect-ai#promptfoo#phoenix#langsmithLLM Evaluation & Observability: Eval Harness, LLM-as-Judge, Tracing, Regression Prevention (2025)
Claiming a new model "seems better" is not science. The 1 failure mode of 2025 LLM products is "not measuring". Eval harness design, LLM-as-judge pitfalls and calibration, 3-layer observability (Trace/Span/Metric), produ
2026-04-15 · 11 min read #llm-eval#observability#llm-judge#distributed-tracing#langsmithComparing LLM Production Monitoring Platforms: A Practical Operations Guide for LangSmith, LangFuse, and Arize Phoenix
A comprehensive comparison guide of three LLM production monitoring platforms (LangSmith, LangFuse, Arize Phoenix). Covers trace collection, prompt version management, evaluation pipelines, cost monitoring, quality dashb
2026-03-09 · 23 min read #ai-platform#llm-monitoring#langsmith#langfuse#arizeChatbot Performance Monitoring and Conversation Quality Analysis: From Metric Design to A/B Test Automation
A practical guide to chatbot performance monitoring and conversation quality analysis. Covers core metric design, tracing with LangSmith/Langfuse, automated quality evaluation pipelines, and A/B testing framework with co
2026-03-08 · 30 min read #chatbot#monitoring#analytics#ab-testing#langsmith