BOOK LOG
ARTICLE LOG
Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman $ρ= 0.14$). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95\% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8\% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency---signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.
Maybe We Shouldn't Be Reviewing All This Code
Or, perhaps the problem isn't that AI has broken code review, maybe it’s that we've been using code review to solve the wrong problems
The MCP Abstraction Tax | Justin Poehnelt
Every layer from App to API to MCP loses fidelity. An exploration of what gets lost and why it matters for enterprise APIs.
You Need to Rewrite Your CLI for AI Agents | Justin Poehnelt
Human DX optimizes for discoverability. Agent DX optimizes for predictability. What I learned building a CLI for agents first.
Writing is Thinking: Learning to Write with Confidence
Learning to write is learning to think, because writing is thinking. They are two peas in a powerful pod. This article goes over my writing process, focused on minimizing the activation energy to get started.
How to write a good software design doc
As a software engineer, I spend a lot of time reading and writing design documents. After having gone through hundreds of these docs, I’ve…
How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server
This article has been revised since it was first published, to correct several significant technical inaccuracies in the original text. Kubernetes has long been the default platform for distributed workloads, and writing your own controller for it is now a matter of a few hours. The common path — Golang, using kubebuilder on top of controller-runtime — gives you a project scaffold, types, and a reconciler. For typical scenarios that is more than enough. But as soon as load grows or the controller starts behaving in ways you did not expect, a whole class of edge cases shows up. Most of them trace back to the same root cause: a fuzzy mental model of how controller-runtime works inside. If you write Kubernetes controllers in Go, this article should help you build a coherent picture and avoid expensive surprises in production.
How to Write an Effective Software Design Document
Best practices for writing a design doc based on my experience working as a developer at Google and Microsoft.
Extending Claude's capabilities with skills and MCP | Claude by Anthropic
Learn how skills and MCP work together to build agents that follow your workflows and use external tools effectively. Best practices and real-world examples.
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose $τ$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.
llm-wiki
llm-wiki. GitHub Gist: instantly share code, notes, and snippets.
How the Open Knowledge Format can improve data sharing | Google Cloud Blog
Learn how the Open Knowledge Format helps secure data sharing and improves collaboration across teams with standardized documentation.
Benchmarking Agentic Workflow Generation
Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is a crucial step in this process. Existing workflow evaluation frameworks either focus solely on holistic performance or suffer from limitations such as restricted scenario coverage, simplistic workflow structures, and lax evaluation standards. To this end, we introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. Additionally, we present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms to accurately quantify the LLM agent's workflow generation capabilities. Through comprehensive evaluations across different types of LLMs, we discover distinct gaps between the sequence planning capabilities and graph planning capabilities of LLM agents, with even GPT-4 exhibiting a gap of around 15%. We also train two open-source models and evaluate their generalization abilities on held-out tasks. Furthermore, we observe that the generated workflows can enhance downstream tasks, enabling them to achieve superior performance with less time during inference. Code and dataset are available at https://github.com/zjunlp/WorfBench.
Understanding is the new bottleneck
Agents can write code faster than we can absorb it. Here's why it still matters for humans to understand what they build — and some techniques for doing that efficiently: explainer docs, quizzes, micro-worlds, and shared spaces.
Demystifying evals for AI agents
Demystifying evals for AI agents
Kubernetes Gateway API - Blog by Roman Glushko
Why Ingress Is Being Replaced and Which Gateway Controller to Pick