Building with the Claude API, part 1: Requests, evals and prompting

A walkthrough of the Claude Academy course Building with the Claude API (Build with the API track, 67 lessons, 8 quizzes), split into four parts; this is part 1. Each lesson is summarised with my own tables and examples; take the course for the full lessons. All the courses are listed in the Claude Academy course index.

Note: This summary comes mainly from the public lesson text. A few details were added from docs.claude.com or the sibling Bedrock/Vertex courses, such as reranking, contextual retrieval and adaptive thinking. Verify those details against the lessons and current docs.

Prerequisites: Python (functions, dicts, async basics for MCP), JSON, a terminal and Jupyter. You also need a Claude Console account with an API key, and a VoyageAI key for the RAG module. Node.js is needed for Claude Code.

Course map

#	Module	Lessons
1	Accessing Claude with the API	1–8 + Quiz 1
2	Prompt evaluation	9–14 + Quiz 2
3	Prompt engineering techniques	15–19 + Quiz 3
4	Tool use with Claude	20–31 + Quiz 4
5	Retrieval Augmented Generation (RAG)	32–38
6	Features of Claude	39–46 + Quiz 5
7	Model Context Protocol (MCP)	47–56 + Quiz 6
8	Anthropic apps: Claude Code & computer use	57–60
9	Agents and workflows	61–67 + Quiz 7, Final assessment (Quiz 8)

---

1. Accessing Claude with the API

Request lifecycle

• The flow is: client → your server → Anthropic API → model → your server → client.
• Never call the API from client-side code. The API key would be exposed. Your server sits between the client and the API.
• Every request needs four things: an API key, a model, messages and max_tokens.
• Inside the model, text goes through four stages:
  1. Tokenization: text is split into tokens.
  2. Embedding: each token becomes a list of numbers, a “numeric definition”.
  3. Contextualization: each embedding is adjusted based on neighbouring tokens. For example, “quantum” could mean physics or just “very small”.
  4. Generation: the model produces probabilities for the next token and samples one. Sampling mixes probability with randomness, then the process repeats.
• Generation stops when one of these happens:
  • max_tokens is reached.
  • The model emits a natural end-of-sequence token.
  • A stop sequence is generated.
• The response contains three things:
  • message: the generated content.
  • usage: input and output token counts.
  • stop_reason: why generation stopped.

Setup and first request

• To get a key, go to the Claude Console, choose Get API Key, then Create Key. Store the key in .env as ANTHROPIC_API_KEY.

%pip install anthropic python-dotenv
from dotenv import load_dotenv; load_dotenv()
from anthropic import Anthropic

client = Anthropic()            # reads ANTHROPIC_API_KEY
model = "claude-haiku-4-5"      # course default (fast & cheap)

message = client.messages.create(
    model=model,
    max_tokens=1000,            # a safety ceiling, NOT a target length
    messages=[{"role": "user", "content": "What is quantum computing? Answer in one sentence"}],
)
print(message.content[0].text)

Multi-turn conversations

• The API is stateless and does not remember earlier turns. You must store the history and resend it on every request.
• Messages alternate between the user role (human input) and the assistant role (model output).

def add_user_message(messages, text):
    messages.append({"role": "user", "content": text})

def add_assistant_message(messages, text):
    messages.append({"role": "assistant", "content": text})

def chat(messages, system=None, temperature=1.0, stop_sequences=[]):
    params = {"model": model, "max_tokens": 1000, "messages": messages,
              "temperature": temperature, "stop_sequences": stop_sequences}
    if system:                      # API rejects system=None, so add conditionally
        params["system"] = system
    return client.messages.create(**params).content[0].text

System prompts

• The system= parameter sets Claude’s role, tone and behaviour. In the course’s math-tutor example, the system prompt tells Claude to give hints instead of handing over the answer.
• Build params conditionally, because passing system=None fails.

Temperature (0–1)

• Low temperature gives more deterministic output. High temperature gives more varied, creative output.

Range	Use cases
0.0–0.3	Factual answers, coding, data extraction, content moderation
0.4–0.7	Summarization, educational content, problem-solving
0.8–1.0	Brainstorming, creative writing, marketing copy, jokes

Streaming

• Streaming improves the user experience: text appears chunk by chunk instead of after a 10–30 s wait.
• Events arrive in this order:

Event	Meaning
MessageStart	A new message is being sent
ContentBlockStart	Start of a text, tool-use or other block
ContentBlockDelta	A chunk of generated text; this is the part you display
ContentBlockStop	The current block is complete
MessageDelta	Top-level message changes, such as stop_reason and usage
MessageStop	End of the message

# Raw events
for event in client.messages.create(model=model, max_tokens=1000, messages=messages, stream=True):
    print(event)

# Simplified text stream + final assembled message (for storage/history)
with client.messages.stream(model=model, max_tokens=1000, messages=messages) as stream:
    for text in stream.text_stream:
        print(text, end="")          # forward via WebSocket/SSE in production
    final_message = stream.get_final_message()

Controlling output: prefill and stop sequences

• Prefill: end the message list with an assistant message. Claude continues exactly where that text ends and does not repeat it.
  • Example: prefilling "Coffee is better because" makes Claude argue for coffee.
• Stop sequences: generation ends just before the stop string would be output.
  • Example: “Count 1–10” with stop_sequences=[", 5"] returns "1, 2, 3, 4".

Structured data (prefill + stop sequence)

• Use this when you want only the raw content, such as JSON, code, CSV or regex, with no commentary.

messages = []
add_user_message(messages, "Generate a very short event bridge rule as json")
add_assistant_message(messages, "```json")
text = chat(messages, stop_sequences=["```"])
import json; rule = json.loads(text.strip())

---

2. Prompt Evaluation

• After you draft a prompt, you can test it once, test it a few times by hand, or run an eval pipeline. The course recommends the eval pipeline.
• Prompt engineering means techniques for writing better prompts. Prompt evaluation means measuring objectively how well a prompt works.

The eval workflow

1. Draft a prompt.
2. Build an eval dataset of inputs.
3. Run each input through Claude using the prompt.
4. Grade each output, for example on a 1–10 scale.
5. Change the prompt, re-run the evals and compare scores. In the course example, v1 averaged 7.66 and v2 averaged 8.7.

Generating a dataset with Claude

• You can use a fast model to generate test cases. The course’s example is AWS tasks that need Python, JSON or regex output.
• Claude returns a JSON array using the prefill and stop-sequence trick. Save the result to dataset.json.
• Give each case a format field so a code grader can check it.

Core pipeline

def run_prompt(test_case):
    prompt = f"Please solve the following task:\n{test_case['task']}"
    messages = []; add_user_message(messages, prompt)
    return chat(messages)

def run_test_case(test_case):
    output = run_prompt(test_case)
    model_grade = grade_by_model(test_case, output)
    syntax_score = grade_syntax(output, test_case)
    score = (model_grade["score"] + syntax_score) / 2
    return {"output": output, "test_case": test_case, "score": score,
            "reasoning": model_grade["reasoning"]}

def run_eval(dataset):
    results = [run_test_case(tc) for tc in dataset]
    avg = statistics.mean(r["score"] for r in results)
    return results, avg

Grader types

Grader	How it works	Good for
Code	Programmatic checks such as length, keywords, syntax and readability	Format and validity; cheap and deterministic
Model	Another API call scores the output	Quality, instruction following, completeness and helpfulness; flexible
Human	A person reviews the output	Nuanced judgement; slow and costly

• Model grader tip: ask for JSON with strengths, weaknesses, reasoning and score. If you ask for a score alone, models tend to give middling scores of about 6.
• Code grader: the validate_json, validate_python and validate_regex functions use json.loads, ast.parse and re.compile respectively. Each returns 10 if parsing succeeds and 0 if it fails.
• Pass extra criteria, such as the solution criteria for each case, into the grader prompt to make grading consistent.

---

3. Prompt Engineering Techniques

• The course uses a reusable PromptEvaluator(max_concurrent_tasks=…) class with two methods:
  • generate_dataset(task_description, prompt_inputs_spec, output_file, num_cases)
  • run_evaluation(run_prompt_function, dataset_file, extra_criteria), which also produces an HTML report.
• The worked example builds a one-day meal plan for an athlete. Scores improved at each step:

Step	Score
Baseline prompt	2.32
+ Clear & direct	3.92
+ Being specific	7.86
+ XML tags / examples	Further gains

Technique	Key ideas
Be clear & direct	The first line matters most. Write an instruction with an action verb (“Generate a one-day meal plan…”) rather than a question.
Be specific	(a) Add output quality guidelines such as length, structure and required attributes; use these almost always. (b) Add process steps for complex problems such as troubleshooting or decision-making.
Structure with XML tags	Wrap interpolated content in descriptive tags such as <sales_records>, <my_code>, <docs> or <athlete_information>, so Claude can tell instructions apart from data.
Provide examples	One-shot or multi-shot examples wrapped in <sample_input> and <ideal_output>. Use them for corner cases such as sarcasm in sentiment analysis. Take the highest-scoring eval outputs as examples and explain why each one is ideal.

prompt = f"""
Generate a one-day meal plan for an athlete that meets their dietary restrictions.

<athlete_information>
- Height: {prompt_inputs["height"]}
- Weight: {prompt_inputs["weight"]}
- Goal: {prompt_inputs["goal"]}
- Restrictions: {prompt_inputs["restrictions"]}
</athlete_information>

Guidelines:
1. Include accurate daily calorie amount
2. Show protein, fat, and carb amounts
3. Specify when to eat each meal
4. Use only foods that fit restrictions
5. List all portion sizes in grams
"""

---