| 1 | optimize agent tools and promptsAnthropic 1Agent coding interview: using provided starter code (Anthropic API client, tool-definition templates, basic agentic loop), optimize an agent… | Writing effective tools for AI agents—using AI agentsCode execution with MCP: building more efficient AI agentsTool use with Claude (docs)Tool consolidation, response shaping, token-efficient tool surfaces |
| 1 | implement HellaSwag evalUnspecified frontier 1Implement a research paper live in 45 minutes, from the evaluation side: implement the HellaSwag benchmark evaluation | HellaSwag: Can a Machine Really Finish Your Sentence?lm-evaluation-harnessHellaSwag likelihood scoring; harness task structure |
| 1 | LLM agent loop with tool useAnthropic 1Phone screen prompt: "This interview will test writing code with LLMs as building blocks and prompting to create agents. You should be conce… | Tool use with Claude (docs)Building agents with the Claude Agent SDKReAct: Synergizing Reasoning and Acting in Language Modelstool_use/tool_result loop, gather-act-verify, ReAct pattern |
| 1 | PRs with Claude CodeAnthropic 1Agentic-coding round (2026): a GitHub-like site with four PRs, using Claude Code to make changes | Best practices for Claude CodeClaude Code workflow: explore, plan, code, test, commit |
| 1 | LLM app system design areasGoogle DeepMind 1Applied AI Engineer ML system design blending model choice at scale, RAG factuality/grounding, efficiency (quantization, distillation), agen… | AI Engineering: Building Applications with Foundation ModelsBuilding A Generative AI PlatformWho Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesModel choice, grounding, inference optimization, judge-human alignment |
| 1 | API rate limits and token optimizationOpenAI 1Be ready with stories about production incidents you've handled; expect quizzing on OpenAI API quirks such as rate limiting and token optimi… | Rate limits (OpenAI API docs)Conversation state (OpenAI API docs)RPM/TPM limits, backoff, batching, caching to cut tokens |
| 1 | debug and improve tool-using agentAnthropic 1"Stock Trading Agent Loop": debug an agent built with an LLM API and tools, get it working, then improve its workflow | AgentsTool use with Claude (docs)Writing effective tools for AI agents—using AI agentsAgent failure modes, tool-call mechanics, tool fixes |
| 1 | ChatGPT system designOpenAI 1 · 1 staff+Design ChatGPT. | Scaling ChatGPT: Five Real-World Engineering ChallengesBuilding A Generative AI PlatformEfficient Memory Management for Large Language Model Serving with PagedAttentionEnd-to-end chat architecture and serving bottlenecks |
| 1 | stateful RAG in productionUnspecified frontier 1Your RAG chatbot works locally; behind a load balancer with 3 replicas it forgets what users asked and degrades with each restart. Why? | Conversation state (OpenAI API docs)Building A Generative AI PlatformExternalize conversation state from replicas; shared stores and caches |
| 1 | LLM cost spikeUnspecified frontier 1Your API's LLM spend jumped from $200/day to $4,000 overnight with no new users, features or code changes. What do you check first? | Rate limits (OpenAI API docs)Conversation state (OpenAI API docs)How we built our multi-agent research systemCache-hit drops, retry storms, agent token multipliers and batch pricing |
| 1 | rerankers and cross-encodersCohere 1What is a reranker, when should you use one, and what does a cross-encoder cost you? | Rerankers and Two-Stage RetrievalColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTIntroducing Contextual RetrievalCross-encoder cost vs bi-encoder recall; measured rerank gains |
| 1 | RAG pipeline evaluationCohere 1How would you evaluate the quality of a RAG pipeline: retrieval and generation separately? | Ragas: Automated Evaluation of Retrieval Augmented GenerationEvaluating Long-Context Question & Answer SystemsAI Evals: Everything You Need to Know (Evals FAQ)Separate retrieval and faithfulness metrics; long-context QA eval design |
| 1 | embedding semantic driftCohere 1What causes semantic drift in embedding search and how do you detect it? | The Embedding Drift Problem: How Your Semantic Search Silently DegradesMMTEB: Massive Multilingual Text Embedding BenchmarkDrift causes, canary queries, embedding-model benchmarking |
| 1 | agent tool-call error handlingOpenAI 1How do you handle tool-call errors, timeouts and retries in an agentic loop? | Designing robust and predictable APIs with idempotencyWriting effective tools for AI agents—using AI agentsTool use with Claude (docs)Idempotent retries, backoff, informative tool errors, is_error results |
| 1 | structured output vs function callingMistral 1What is the difference between structured output and function calling? | Tool use with Claude (docs)Function calling vs constrained-decoding structured outputs |
| 1 | tool schema design for agentsAnthropic 1How many tools is too many? How do you design tool schemas an LLM can actually use correctly? | Writing effective tools for AI agents—using AI agentsCode execution with MCP: building more efficient AI agentsTool count, namespacing, schema clarity, code-execution alternative |
| 1 | long-running agent memory designAnthropic 1Design memory for a long-running agent: what do you store, where, and how do you retrieve it? | Effective context engineering for AI agentsMemGPT: Towards LLMs as Operating SystemsEffective harnesses for long-running agentsNote-taking, hierarchical memory, progress files across context windows |
| 1 | human-in-the-loop agent approvalOpenAI 1Design human-in-the-loop approval for an agent that takes consequential actions. | Our framework for developing safe and trustworthy agentsLangGraph overviewHuman control principles; interrupt-and-resume approval checkpoints |
| 1 | prompting vs RAG vs fine-tuningOpenAI 1, Mistral 1, Cohere 1Prompting, RAG or fine-tuning: give me your decision framework with cost and latency attached. | Optimizing LLM AccuracyAI Engineering: Building Applications with Foundation ModelsPrompt vs RAG vs fine-tune framework with cost and latency |
| 1 | eval set without ground truthCohere 1How do you build an eval set when there is no labelled ground truth and experts are expensive? | Product Evals in Three Simple StepsAI Evals: Everything You Need to Know (Evals FAQ)Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesSmall labelled seed set, aligned judges, criteria drift |
| 1 | hallucination detection in RAGAnthropic 1, OpenAI 1How do you detect and measure hallucinations in a production RAG system? | FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text GenerationRagas: Automated Evaluation of Retrieval Augmented GenerationAtomic-fact verification, sampling consistency, faithfulness metrics |
| 1 | eval regression gate for shippingAnthropic 1Design the regression gate that decides whether a prompt or model change ships. | Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsDemystifying evals for AI agentsEvaluation best practices (OpenAI API docs)Paired tests and power, regression suites, prompt-regression gate |
| 1 | coding agent harness loopAnthropic 1What matters more for an agentic coding tool like Claude Code: the model or the harness? Design the loop. | Raising the bar on SWE-bench Verified with Claude 3.5 SonnetEffective harnesses for long-running agentsThe Harness LecturesMinimal SWE-bench harness, long-run harness design, harness papers |
| 1 | coding agent tool surfaceAnthropic 1Design the tool surface for a coding agent: which tools exist, what their schemas look like, and how results come back. | Raising the bar on SWE-bench Verified with Claude 3.5 SonnetSWE-bench: Can Language Models Resolve Real-World GitHub Issues?Writing effective tools for AI agents—using AI agentsBash/edit tools, agent-computer interface design, tool response shaping |
| 1 | customer RAG hallucination triageAnthropic 1An enterprise customer says “Claude hallucinates too much” in their RAG-based knowledge assistant. You're the applied engineer on the accoun… | A Field Guide to Rapidly Improving AI ProductsSeven Failure Points When Engineering a Retrieval Augmented Generation SystemEvaluating Long-Context Question & Answer SystemsError analysis process, RAG failure points, faithfulness evals |
| 1 | reliable tool-calling agent loopOpenAI 1You are building a production agent that calls tools (function calling). What makes the loop reliable enough to ship? | Demystifying evals for AI agentsτ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsDesigning robust and predictable APIs with idempotencypass^k reliability, simulated-user tests, retry-safe tool calls |
| 1 | model upgrade regression complaintOpenAI 1A customer says “the model got worse” after you upgraded model versions in their deployment. How do you verify and respond? | A postmortem of three recent issuesAdding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsEvaluation best practices (OpenAI API docs)Real quality-regression postmortem, paired comparisons, regression evals |
| 1 | deployed system latency diagnosisOpenAI 1An enterprise customer reports that responses from your deployed system have gotten slow. Walk me through the diagnosis. | Rate limits (OpenAI API docs)Efficient Memory Management for Large Language Model Serving with PagedAttentionA postmortem of three recent issuesLatency levers, batching/KV pressure, infra-level root causes |
| 1 | claims processing automation scopingOpenAI 1An enterprise customer says: “We want AI to automate our claims processing.” You're the engineer in the room. What do the first two weeks lo… | A Field Guide to Rapidly Improving AI ProductsBuilding effective agentsWhat We've Learned From A Year of Building with LLMsError-analysis-led discovery, simplest workflow first, strategy lessons |
| 1 | frontier model eval harnessGoogle DeepMind 1Build the evaluation harness for a new frontier model release. What does it need to do? | Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationlm-evaluation-harnessInspectAdding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsHarness design, benchmark tasks, sandboxed agent evals, error bars |
| 1 | assistant release eval systemMeta MSL 1How would you build the evaluation system for a Meta AI assistant before and after each model release? | The Llama 3 Herd of ModelsJudging LLM-as-a-Judge with MT-Bench and Chatbot ArenaAdding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsMeta's Llama 3 eval suites, pairwise human/judge evals, significance |
| 1 | LLM-backed service pair programmingMistral 1Pair-programming: build a service that takes a user question, enriches it with data from a third-party API, and answers via a chat-model API… | Designing robust and predictable APIs with idempotencyTool use with Claude (docs)Claude CookbooksRetry-safe external calls, tool-calling structure, service notebooks |
| 1 | reliable function callingMistral 1How does function calling actually work with an LLM, and how do you make it reliable enough for production agents? | Tool use with Claude (docs)Toolformer: Language Models Can Teach Themselves to Use ToolsTool schema to tool_use mechanics; training-time tool learning |
| 1 | on-prem open-weight deploymentMistral 1Design an on-prem deployment of an open-weight model for a European bank that cannot send data to any external API. | llm-dgpt-oss-120b & gpt-oss-20b Model CardEfficient Memory Management for Large Language Model Serving with PagedAttentionvLLM/llm-d on-prem stack, open-weight sizing, KV memory planning |
| 1 | two-stage retrieval with rerankerCohere 1You have an embedding model and a reranker. Why sell both? Design the two-stage retrieval pipeline and tell me when the reranker earns its l… | Rerankers and Two-Stage RetrievalColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTBEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval ModelsTwo-stage pipeline, late interaction, out-of-domain evidence for reranking |
| 1 | embedding compression for vector indexCohere 1An enterprise wants semantic search over ~100M documents but is balking at vector-index cost. Walk me through embedding compression options … | Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper RetrievalThe Faiss libraryBinary/int8/Matryoshka compression maths, PQ and IVF trade-offs |
| 1 | multilingual retrieval evaluationCohere 1How would you evaluate multilingual retrieval quality when employees query in French and Korean over mostly-English documents? | MMTEB: Massive Multilingual Text Embedding BenchmarkMultilingual/cross-lingual benchmarks and metrics (MMTEB, MIRACL) |
| 1 | RAG quality drop after scaling indexCohere 1A customer 10x'd their indexed documents and reports answer quality “got noticeably worse.” Drive the investigation. | Lost in the Middle: How Language Models Use Long ContextsSeven Failure Points When Engineering a Retrieval Augmented Generation SystemThe Faiss libraryDistractor passages, missed top-k, ANN recall at larger scale |
| 1 | enterprise workflow agentCohere 1Design an agent that automates an enterprise workflow, say, drafting RFP responses from internal documents and a CRM. What does “enter-prise… | Building effective agentsRAG with Access ControlOur framework for developing safe and trustworthy agentsWorkflow-vs-agent choice, ACL-aware retrieval, human approval gates |
| 1 | air-gapped full-stack deploymentCohere 1A bank wants the whole stack (model, RAG, agents) deployed air-gapped on their own GPUs. What actually changes versus your SaaS? | llm-dgpt-oss-120b & gpt-oss-20b Model CardSelf-hosted serving stack and open-weight model sizing |
| 1 | RAG eval without labelsCohere 1An enterprise customer wants to deploy your RAG system but has no labelled data. How do you evaluate it before and after launch? | Ragas: Automated Evaluation of Retrieval Augmented GenerationProduct Evals in Three Simple StepsA Field Guide to Rapidly Improving AI ProductsReference-free metrics, aligned judges, production error analysis |
| 1 | self-adapting agentic systemAnthropic 1Design an agentic AI system that can autonomously adapt to new tasks. | Agentic Context Engineering: Evolving Contexts for Self-Improving Language ModelsStanford CS329A: Self-Improving AI AgentsContext-adaptation playbooks; self-improvement via verifiers and RL |
| 1 | Claude API in regulated industryAnthropic 1If a customer in a regulated industry wanted to use Claude APIs, how would you design the implementation? | API and data retention (Claude docs)Our framework for developing safe and trustworthy agentsData retention, compliance reports, agent control principles |
| 1 | long-running Claude workflow timeoutsAnthropic 1How would you build a reliable Claude workflow for a long-running task that risks timing out? | Effective harnesses for long-running agentsDesigning robust and predictable APIs with idempotencyRate limits (OpenAI API docs)Checkpointed harness, durable execution, batch API for long jobs |
| 1 | Claude deployment security complianceAnthropic 1How would you handle security and compliance when deploying Claude for a government contractor? | API and data retention (Claude docs)Model Context Protocol Specification (2025-11-25)Compliance controls, ZDR, MCP authorization model |
| 1 | handling hallucinations in deploymentAnthropic 1, OpenAI 1How would you handle hallucinations in a generative AI model deployed to users? | FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text GenerationPatterns for Building LLM-based Systems & ProductsIntroducing Contextual RetrievalHallucination measurement, guardrails and UX, better retrieval |
| 1 | prompt engineering approachAnthropic 1What's your approach to prompt engineering? | Prompt engineering overview (Claude docs)A Field Guide to Rapidly Improving AI ProductsPrompting techniques iterated through error analysis |
| 1 | semantic product search for ChatGPTOpenAI 1Build a semantic search system over Amazon products so ChatGPT could access it. | Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product SearchBinary and Scalar Embedding Quantization for Significantly Faster & Cheaper RetrievalModel Context Protocol Specification (2025-11-25)Product-search eval set, cheap embeddings, exposing search as MCP tool |
| 1 | RAG over customer private dataOpenAI 1Design a retrieval-augmented generation system over a customer's private data. | RAG with Access ControlAPI and data retention (Claude docs)Retrieval-Augmented Generation for Large Language Models: A SurveyPermission-aware retrieval, data controls, RAG design space |
| 1 | agent eval suite accuracy and costOpenAI 1How would you build the evaluation suite to confirm an AI agent meets its accuracy and cost targets? | AI Agents That MatterDemystifying evals for AI agentsHolistic Agent Leaderboard: The Missing Infrastructure for AI Agent EvaluationAccuracy-cost Pareto evals, graders, cost-tracking harness |
| 1 | customer discovery before AI designOpenAI 1A customer wants to use AI to solve a specific business problem. What questions would you ask before designing anything, and how would you a… | A Field Guide to Rapidly Improving AI ProductsWhat We've Learned From A Year of Building with LLMsDiscovery questions, success criteria, look at data first |
| 1 | 10x traffic spike latency costOpenAI 1A customer's traffic spikes 10x overnight. Walk through how you keep the system within latency and cost targets. | Rate limits (OpenAI API docs)Mooncake: A KVCache-centric Disaggregated Architecture for LLM ServingConversation state (OpenAI API docs)Rate limits, overload scheduling and load shedding, caching |
| 1 | MCP long-running task reliabilityAnthropic 1Respond to an error message and propose a solution that improves model reliability on long-running tasks, using Anthropic's AI tools (MCP to… | Code execution with MCP: building more efficient AI agentsModel Context Protocol Specification (2025-11-25)Effective harnesses for long-running agentsCode-execution MCP, protocol transports, long-run harness patterns |
| 1 | improve internal LLM tool codeAnthropic 1Analyze and improve code from an Anthropic internal tool to strengthen performance, reliability and context-window management. | Effective context engineering for AI agentsWriting effective tools for AI agents—using AI agentsConversation state (OpenAI API docs)Context management, tool improvements, caching and trimming |
| 1 | on-prem enterprise deploymentMistral 1Design an enterprise on-premise deployment with model updates, A/B testing, and data-sovereignty guarantees. | llm-dAdding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsAPI and data retention (Claude docs)Self-hosted rollout, A/B significance, data-sovereignty controls |
| 1 | search RAG ML designOpenAI 1Search & RAG ML Design (spoken) (60 min) | Retrieval Augmented Generation (RAG)Retrieval-Augmented Generation for Large Language Models: A SurveyBEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval ModelsRetriever design, RAG architecture space, retrieval benchmarks |
| 1 | LLM assistant eval systemGoogle DeepMind 1Design an Evaluation System for an LLM Assistant: An assistant product ships a new candidate model every week. | Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsJudging LLM-as-a-Judge with MT-Bench and Chatbot ArenaThe Llama 3 Herd of ModelsWeekly model comparison stats, pairwise evals, release eval suites |
| 1 | agent loop with toolsAnthropic 1Building an Agent Loop with Tool Use: A notebook defines client, an object with one method, client.create(model, system, messages, tools, ma… | Tool use with Claude (docs)Building agents with the Claude Agent SDKClaude CookbooksImplementing the tool loop against a create() client |
| 1 | structured output prompting evalAnthropic 1Prompting and Engineering with LLMs: Structured Output and Evaluation: A notebook defines complete, a function that sends one turn to a lang… | Tool use with Claude (docs)Prompt engineering overview (Claude docs)Product Evals in Three Simple StepsStructured outputs, prompt design, simple eval harness |
| 1 | metric up users say worseGoogle DeepMind 1Your evaluation metric has improved over the last several training runs, but early users report that outputs feel worse — how do you investi… | Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human PreferencesA Field Guide to Rapidly Improving AI ProductsEvaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)Criteria drift, judge bias, error analysis on user traces |
| 1 | agentic code-review design critiqueAnthropic 1Critique a design document for an agentic code-review auto-approval system (system design round as design review). | Establishing Best Practices for Building Rigorous Agentic BenchmarksOur framework for developing safe and trustworthy agentsAI Agents That MatterBenchmark validity checklist, human-control principles, cost-aware evaluation |
| 1 | Mistral RAG agentMistral 1Build a minimal, but production-minded, retrieval-augmented generation (RAG) agent in Python that uses the Mistral API. | RAG Quickstart (Mistral docs)Seven Failure Points When Engineering a Retrieval Augmented Generation SystemMinimal Mistral RAG, production failure points |
| 1 | reproducible LLM dev envMistral 1Set up a clean, fully reproducible local Python environment to develop a small Mistral-based agent and verify it with a real API call. | RAG Quickstart (Mistral docs)uv-based reproducible environment plus Mistral API smoke test |
| 1 | scalable RAG cost latencyMistral 1Design a scalable Retrieval-Augmented Generation (RAG) system from scratch, detailing how you would manage cost, throughput, and latency. | Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper RetrievalThe Faiss libraryIntroducing Contextual RetrievalBuilding A Generative AI PlatformIndex compression, ANN cost, retrieval quality, platform caching |
| 1 | agentic workflow state machinesMistral 1Explain how you would implement and manage complex agentic workflows using frameworks like LangGraph or custom state machines. | LangGraph overviewBuilding effective agentsGraph/state-machine agents vs simple workflows |
| 1 | real-time assistant social dataxAI 1How would you approach building a real-time AI assistant with access to social media data? | Building A Generative AI PlatformAgentic Retrieval-Augmented Generation: A Survey on Agentic RAGUsing server-sent events (MDN)Freshness-aware retrieval, agentic RAG, streaming responses |