BOOK LOG
ARTICLE LOG
How to write a good software design doc
As a software engineer, I spend a lot of time reading and writing design documents. After having gone through hundreds of these docs, I’ve…
How the controller-runtime Cache Actually Works, and Why Your Controller Does Not Crash the API Server
This article has been revised since it was first published, to correct several significant technical inaccuracies in the original text. Kubernetes has long been the default platform for distributed workloads, and writing your own controller for it is now a matter of a few hours. The common path — Golang, using kubebuilder on top of controller-runtime — gives you a project scaffold, types, and a reconciler. For typical scenarios that is more than enough. But as soon as load grows or the controller starts behaving in ways you did not expect, a whole class of edge cases shows up. Most of them trace back to the same root cause: a fuzzy mental model of how controller-runtime works inside. If you write Kubernetes controllers in Go, this article should help you build a coherent picture and avoid expensive surprises in production.
How to Write an Effective Software Design Document
Best practices for writing a design doc based on my experience working as a developer at Google and Microsoft.
Extending Claude's capabilities with skills and MCP | Claude by Anthropic
Learn how skills and MCP work together to build agents that follow your workflows and use external tools effectively. Best practices and real-world examples.
Trace an LLM application tutorial - Docs by LangChain
Add LangSmith observability to an LLM application across prototyping, beta testing, and production.
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose $τ$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.
llm-wiki
llm-wiki. GitHub Gist: instantly share code, notes, and snippets.
How the Open Knowledge Format can improve data sharing | Google Cloud Blog
Learn how the Open Knowledge Format helps secure data sharing and improves collaboration across teams with standardized documentation.
Benchmarking Agentic Workflow Generation
Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is a crucial step in this process. Existing workflow evaluation frameworks either focus solely on holistic performance or suffer from limitations such as restricted scenario coverage, simplistic workflow structures, and lax evaluation standards. To this end, we introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. Additionally, we present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms to accurately quantify the LLM agent's workflow generation capabilities. Through comprehensive evaluations across different types of LLMs, we discover distinct gaps between the sequence planning capabilities and graph planning capabilities of LLM agents, with even GPT-4 exhibiting a gap of around 15%. We also train two open-source models and evaluate their generalization abilities on held-out tasks. Furthermore, we observe that the generated workflows can enhance downstream tasks, enabling them to achieve superior performance with less time during inference. Code and dataset are available at https://github.com/zjunlp/WorfBench.
Understanding is the new bottleneck
Agents can write code faster than we can absorb it. Here's why it still matters for humans to understand what they build — and some techniques for doing that efficiently: explainer docs, quizzes, micro-worlds, and shared spaces.
Demystifying evals for AI agents
Demystifying evals for AI agents
Kubernetes Gateway API - Blog by Roman Glushko
Why Ingress Is Being Replaced and Which Gateway Controller to Pick