What you're building
In Part 11 you built a small agent loop in plain Python: no framework, just a loop, a system prompt, a couple of tools and some bookkeeping. This project reuses that same shape for something more useful: a command-line research assistant that takes a question, decides what to search for, reads the pages it finds, keeps short notes as it goes, and writes a final answer where every claim is tied to a specific source it actually read. Along the way it uses most of what this path has covered so far: the observe-think-act loop from Part 2, tool definitions the model can call from Part 3, the workflow-versus-agent judgement call from Part 4, notes as an external memory store from Part 6, decisions about what goes into the model's context on each turn from Part 7, a stopping condition that does not depend on the model behaving perfectly from Part 8, and an optional human checkpoint before the agent touches the open internet from Part 9. By the end you will have three small Python files, tools.py, agent.py and main.py, that together form a working, deliberately modest, research agent you can run from a terminal.
One thing worth saying up front: this is not the shortest way to summarize a web page. If you always want the same three sources fetched and summarized the same way, a fixed workflow, in the sense of Part 4, is simpler, cheaper and more predictable than an agent loop. The agent pattern earns its keep here because the right number of searches, the right pages to read and when there is enough information genuinely depend on the question, and we want the model to make those calls rather than hard-coding them.
Before you start
You will need the following.
- Python 3.9 or later
- An API key for a chat model that supports tool calling (the examples here use OpenAI's API; the tool-calling pattern is similar with other providers, but the exact client code will differ, so check your provider's own documentation)
- Internet access, since the agent makes real web requests
- About an hour, plus whatever it costs you in API usage to run a handful of test questions
Install the packages used in this project. The code below was written against the versions listed here. If a function call does not match what you have installed, run pip show <package> and check that package's own changelog, since both the search wrapper and the model provider's client library have changed their interfaces before and may change again.
- openai 1.x, using the chat.completions.create(..., tools=...) interface
- requests 2.31
- beautifulsoup4 4.12
- duckduckgo-search 5.3, a third-party wrapper around DuckDuckGo's search, unaffiliated with OpenAI or MLHub
pip install openai requests beautifulsoup4 duckduckgo-search
Set your API key as an environment variable so it never ends up in your source code.
export OPENAI_API_KEY="sk-..."
# on Windows PowerShell:
# $env:OPENAI_API_KEY="sk-..."
Decide what "done" looks like before you write code
It is tempting to start coding a tool loop straight away, but it helps to write down what a good run looks like first, the same way Part 8 argued for deciding stopping conditions before you need them. For this project, a successful answer meets all of the following.
- Every factual claim in the answer carries a citation number
- Every citation number points to a URL the agent actually fetched and read, not just a URL that turned up in a search result snippet
- The agent stops within a bounded number of steps, whether or not it found a perfect answer
- If it cannot find good sources, it says so instead of guessing
And here are the failure modes worth watching for, because they are easy to produce by accident and easy to miss if you only glance at the final text.
- Citing a URL that appeared in search results but was never opened and read
- Searching for the same thing twice in slightly different words, burning steps without new information
- Running until an external timeout or rate limit kills the process, rather than stopping on its own terms
- A confident-sounding answer with no sources at all
Designing the assistant
Before writing any code, it is worth mapping each design decision back to a part of this path, because every one of them is a direct answer to a question raised earlier.
The loop
The shape is the same as Part 2 and Part 11: build a list of messages, call the model with the available tools, and if the model asks to call a tool, run it and feed the result back as a new message, then call the model again. Repeat until the model calls a finish tool that passes validation, or until a hard step limit is reached.
The tools
Four tools are enough for this project. web_search looks things up and returns titles, URLs and short snippets. read_page fetches one URL and returns a plain-text excerpt of its content. take_note saves a short fact together with the URL it came from, which is how the agent builds up the evidence it will later cite. finish is called once, at the end, and it is where the final answer and its sources are handed back, in a structured form the code can check rather than a free-text paragraph the code has to parse and hope for the best.
Memory: what the agent remembers
Following Part 6, there are two kinds of memory at work here. The conversation itself, the list of messages sent back and forth with the model, is short-term memory: it exists for this run and is what the model directly sees. The notes list is a small, deliberate piece of external memory: facts the agent chose to write down, each tagged with its source. Notes are cheaper to keep around than full page text, and they are also the fallback answer if the agent runs out of steps before calling finish.
Context engineering: keeping the conversation lean
This is where Part 7 shows up directly. The full text of a page, up to a few thousand characters, is useful right after read_page returns it, because the model needs to actually read it before taking notes. But if the agent reads four or five pages over the course of a run, keeping every page's full text in the conversation forever wastes tokens, slows the model down, and can crowd out more recent and more relevant context. The project trims older read_page results down to a short placeholder once newer ones have arrived, trusting the notes to carry the important facts forward instead of the raw text.
Stopping conditions
Part 8's advice was to have more than one layer of defence, since any single stopping rule can fail. Here there are three. MAX_SEARCHES and MAX_PAGES cap how much the agent can do with each tool individually. MAX_STEPS is a hard ceiling on the whole loop, so a confused agent cannot run forever. And the finish tool call itself is validated before it is accepted: an answer with no sources, or with a citation pointing at a page that was never actually read, is rejected and sent back to the model as an error to fix, rather than treated as a legitimate stop.
Human in the loop
This agent fetches arbitrary URLs chosen by a language model, which is exactly the kind of action Part 9 flagged as worth a checkpoint. The project adds an optional require_approval flag with two gates: one before fetching a page, one before returning the final answer, so a person can see, and veto, what is about to happen or be said. It defaults to off so the examples in this article run unattended, but turning it on is a one-line change when you want a person watching.
A citation is a claim about the world: this specific fact came from this specific page. If the code cannot point to the fetch that produced it, the agent should not be allowed to make the claim.
Step 1: project layout
Create a folder with three empty files to start from.
mkdir research_assistant
cd research_assistant
touch tools.py agent.py main.py
tools.py holds the plain functions that do the actual work: searching and fetching pages. agent.py holds the tool schemas, the system prompt, and the loop itself. main.py is the small command-line wrapper you will actually run. Keeping the functions that touch the network separate from the loop that talks to the model makes both easier to test on their own, which matters later.
Step 2: the tools the agent can call
These are ordinary Python functions, nothing more. They do not know anything about the model or about tool calling; that layer is added in agent.py. Keeping them plain makes them easy to test by calling them directly from a Python shell before the agent ever touches them.
"""Tools the research agent can call.
Tested against:
requests==2.31.0
beautifulsoup4==4.12.3
duckduckgo-search==5.3.0
The duckduckgo-search package is a third-party wrapper and has changed
its interface before. If web_search() below raises an error about a
missing method or argument, check the README of the version you have
installed.
"""
import requests
from bs4 import BeautifulSoup
from duckduckgo_search import DDGS
MAX_CHARS_PER_PAGE = 4000
REQUEST_TIMEOUT = 10 # seconds
USER_AGENT = "mlhub-research-assistant/0.1 (educational project)"
def web_search(query, max_results=5):
"""Search the web and return a list of {title, url, snippet} dicts."""
results = []
with DDGS() as ddgs:
for item in ddgs.text(query, max_results=max_results):
results.append({
"title": item.get("title", ""),
"url": item.get("href", ""),
"snippet": item.get("body", ""),
})
return results
def read_page(url):
"""Fetch a URL and return a plain-text excerpt of its main content.
Returns {"url": url, "text": "..."} on success, or
{"url": url, "error": "..."} on failure. Never raises, so the agent
loop can treat a failed fetch as information rather than a crash.
"""
try:
response = requests.get(
url,
timeout=REQUEST_TIMEOUT,
headers={"User-Agent": USER_AGENT},
)
response.raise_for_status()
except requests.RequestException as exc:
return {"url": url, "error": f"could not fetch page ({exc})"}
soup = BeautifulSoup(response.text, "html.parser")
for tag in soup(["script", "style", "nav", "footer", "header", "aside", "form"]):
tag.decompose()
paragraphs = [p.get_text(" ", strip=True) for p in soup.find_all("p")]
text = "\n".join(p for p in paragraphs if p)
if not text:
return {"url": url, "error": "page fetched but no readable text was found"}
return {"url": url, "text": text[:MAX_CHARS_PER_PAGE]}
A few choices here are worth calling out explicitly. read_page never raises an exception; every outcome, success or failure, comes back as a plain dictionary. That matters because the agent loop in the next step treats tool results as data the model gets to reason about, in the spirit of Part 8: a failed fetch is something the agent can react to, by trying a different source, rather than something that crashes the whole run. The timeout keeps one slow or unresponsive server from stalling the entire agent. The custom User-Agent header identifies the request honestly rather than pretending to be a browser. And the truncation to MAX_CHARS_PER_PAGE is a context engineering decision from Part 7 made at the source: rather than pulling an entire long article into the conversation and dealing with the cost later, the tool itself caps how much text it ever hands back.
Note what read_page does not do: it does not try to detect or parse PDFs, JavaScript-rendered pages, or anything behind a login. For those, BeautifulSoup will typically find no <p> tags and the function will return the "no readable text was found" error, which the agent treats like any other failed read and moves on from. That graceful failure is good enough for this project; extending it to other content types is listed as an extension later on.
Step 3: the agent loop
This is the core file. It defines how the tools are described to the model, what the model is told to do with them, how a finish call is checked before it is accepted, and the loop that ties all of that together. Read through it once, then the sections after it walk through each part.
"""The research agent loop: observe, think, act.
Tested against openai==1.30.0. Any 1.x release that supports the
`tools` parameter on chat.completions.create should work. Swap MODEL
for any tool-calling-capable chat model your provider offers; check
your provider's current model names, since these change over time.
"""
import json
import re
from openai import OpenAI
from tools import web_search, read_page
MODEL = "gpt-4o-mini"
MAX_STEPS = 8
MAX_SEARCHES = 4
MAX_PAGES = 5
KEEP_FULL_TEXT_FOR_LAST_N_READS = 2
client = OpenAI() # reads OPENAI_API_KEY from the environment
SYSTEM_PROMPT = """You are a research assistant. You answer questions by
searching the web and reading pages before you write anything down.
Rules:
1. Use web_search to find candidate sources, then use read_page to
actually read at least one of them before answering. Do not rely on
memory alone for facts that could be out of date or that you are not
completely sure about.
2. After reading a page, call take_note with the specific fact you plan
to use and the url it came from. Keep each note short and factual.
3. When you have enough information, call finish. In the answer field,
put an inline citation marker like [1] right after each claim it
supports. In the sources field, list every number you used together
with the exact url you read it on. Only cite urls you fetched with
read_page.
4. If a page fails to load or has nothing useful, search again or try a
different result rather than giving up.
5. You have a limited number of searches and page reads. Be deliberate:
a few well-chosen sources beat reading everything you can find."""
TOOLS = [
{
"type": "function",
"function": {
"name": "web_search",
"description": "Search the web. Returns up to max_results items, each with a title, url and short snippet.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"max_results": {"type": "integer"},
},
"required": ["query"],
},
},
},
{
"type": "function",
"function": {
"name": "read_page",
"description": "Fetch a web page by url and return a plain-text excerpt of its content.",
"parameters": {
"type": "object",
"properties": {"url": {"type": "string"}},
"required": ["url"],
},
},
},
{
"type": "function",
"function": {
"name": "take_note",
"description": "Save a short factual note extracted from a page you have read, with its source url.",
"parameters": {
"type": "object",
"properties": {
"url": {"type": "string"},
"note": {"type": "string"},
},
"required": ["url", "note"],
},
},
},
{
"type": "function",
"function": {
"name": "finish",
"description": "Give the final answer. Only cite urls fetched with read_page.",
"parameters": {
"type": "object",
"properties": {
"answer": {"type": "string"},
"sources": {
"type": "array",
"items": {
"type": "object",
"properties": {
"number": {"type": "integer"},
"url": {"type": "string"},
"title": {"type": "string"},
},
"required": ["number", "url"],
},
},
},
"required": ["answer", "sources"],
},
},
},
]
def validate_finish(args, fetched_urls):
answer = args.get("answer", "")
sources = args.get("sources", [])
if not answer.strip():
return False, "the answer field is empty"
if not sources:
return False, "no sources were given; read_page at least one page before finishing"
cited = {int(n) for n in re.findall(r"\[(\d+)\]", answer)}
numbered = set()
for s in sources:
if "number" not in s or "url" not in s:
return False, "every source needs a number and a url"
numbered.add(s["number"])
if s["url"] not in fetched_urls:
return False, f"source [{s['number']}] points to {s['url']}, which was never read with read_page"
missing = cited - numbered
if missing:
return False, f"the answer cites {sorted(missing)} but there is no matching entry in sources"
return True, None
def confirm(prompt):
reply = input(f"{prompt} [y/n] ").strip().lower()
return reply in ("y", "yes")
def run_agent(question, require_approval=False, verbose=True):
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": question},
]
notes = []
fetched_urls = set()
read_page_message_indices = []
search_count = 0
for step in range(1, MAX_STEPS + 1):
if len(read_page_message_indices) > KEEP_FULL_TEXT_FOR_LAST_N_READS:
for idx in read_page_message_indices[:-KEEP_FULL_TEXT_FOR_LAST_N_READS]:
messages[idx]["content"] = json.dumps(
{"note": "page text omitted to save context; see earlier notes"}
)
response = client.chat.completions.create(
model=MODEL, messages=messages, tools=TOOLS, tool_choice="auto"
)
msg = response.choices[0].message
assistant_entry = {"role": "assistant", "content": msg.content or ""}
if msg.tool_calls:
assistant_entry["tool_calls"] = [
{
"id": tc.id,
"type": "function",
"function": {"name": tc.function.name, "arguments": tc.function.arguments},
}
for tc in msg.tool_calls
]
messages.append(assistant_entry)
if not msg.tool_calls:
if verbose:
print(f"[step {step}] model answered without calling finish; stopping")
return {"answer": msg.content, "sources": [], "notes": notes, "steps": step}
for tc in msg.tool_calls:
name = tc.function.name
try:
args = json.loads(tc.function.arguments or "{}")
except json.JSONDecodeError:
args = {}
if verbose:
print(f"[step {step}] tool call: {name}({args})")
if name == "web_search":
if search_count >= MAX_SEARCHES:
result = {"error": "search budget exhausted; work with what you have or read a page"}
else:
search_count += 1
result = web_search(
args.get("query", ""), max_results=args.get("max_results", 5)
)
elif name == "read_page":
url = args.get("url", "")
if len(fetched_urls) >= MAX_PAGES:
result = {"error": "page-read budget exhausted; use take_note and finish"}
elif require_approval and not confirm(f"Fetch {url}?"):
result = {"error": "fetch was not approved by the human operator"}
else:
result = read_page(url)
if "text" in result:
fetched_urls.add(url)
elif name == "take_note":
notes.append({"url": args.get("url", ""), "note": args.get("note", "")})
result = {"status": "saved", "note_count": len(notes)}
elif name == "finish":
ok, problem = validate_finish(args, fetched_urls)
if ok:
if require_approval and not confirm("Return this answer to the user?"):
result = {"error": "the human operator asked for changes before finishing"}
else:
return {
"answer": args["answer"],
"sources": args["sources"],
"notes": notes,
"steps": step,
}
else:
result = {"error": problem}
else:
result = {"error": f"unknown tool: {name}"}
tool_message = {
"role": "tool",
"tool_call_id": tc.id,
"content": json.dumps(result),
}
messages.append(tool_message)
if name == "read_page" and "text" in result:
read_page_message_indices.append(len(messages) - 1)
fallback = "I ran out of steps before producing a fully cited answer. Notes gathered so far:\n"
fallback += "\n".join(f"- {n['note']} ({n['url']})" for n in notes) or "(no notes were saved)"
return {"answer": fallback, "sources": [], "notes": notes, "steps": MAX_STEPS}
The system prompt
SYSTEM_PROMPT does three jobs at once: it tells the model the expected order of operations (search, then read, then note, then finish), it states the citation format explicitly rather than hoping the model guesses a sensible one, and it tells the model it has a limited budget so it does not treat reading ten pages as free. None of these rules are enforced by the prompt alone; they are backed up by the budgets and by validate_finish. A prompt is a request, not a guarantee, which is exactly why the code checks the finish call rather than trusting it.
The tool schemas
TOOLS follows the function-calling format from Part 3: each entry names a function, describes what it does in plain language, and gives a JSON schema for its arguments. The description fields matter more than they look like they should, since the model relies on them, not on your code comments, to decide when and how to call each tool. The one worth studying closely is finish. Instead of asking the model to just write a paragraph with citations in free text, sources is a structured array of {number, url, title} objects. That structure is what makes validate_finish possible: checking whether a URL was really fetched is trivial against a list of dictionaries, and would be fragile and error-prone against a block of prose you would otherwise have to parse with more regular expressions and hope you covered every phrasing the model might use.
Validating the finish call
validate_finish is the part of this project doing the most work for the least code. It checks that the answer is not empty, that at least one source is given, that every source's url is in fetched_urls (the set of URLs that read_page actually succeeded on, built up during the run), and that every [n] marker appearing in the answer text has a matching source number. If any check fails, the finish call is not accepted; instead, the loop sends the reason back to the model as a tool result, the same way any other tool error is reported, and the model gets another turn to fix it. This is the Part 8 idea of treating failure as something to recover from rather than something to crash on, applied specifically to the one failure mode that matters most for a tool that is supposed to produce citations: a citation that points nowhere real.
Trimming old page text
read_page_message_indices keeps track of which entries in messages hold full page text from a successful read. At the top of every loop iteration, all but the most recent KEEP_FULL_TEXT_FOR_LAST_N_READS of them are overwritten with a short placeholder. The model still has access to everything it chose to write down with take_note, and it still sees the full text of the one or two pages it read most recently, but it is not carrying the complete text of every page it has ever opened for the rest of the run. This is the context engineering idea from Part 7 in its most literal form: deciding, explicitly, what stays in the model's working context and what gets moved out to a cheaper, external store.
Budgets and the hard stop
MAX_SEARCHES and MAX_PAGES are checked inside the tool-handling code, before the real functions in tools.py are even called, so a model that tries to search a ninth time simply gets an error message telling it the budget is gone, rather than another real network request. MAX_STEPS bounds the whole loop regardless of what the model does. If the loop reaches the end of that range without a validated finish call, it does not crash or hang; it returns a fallback answer built directly from the notes collected so far, which is honest about having run out of budget rather than silent about it.
The optional human checkpoints
Setting require_approval=True adds two input() prompts: one before any read_page call actually hits the network, and one before a validated finish call is allowed to return. Both follow the simple pattern described in Part 9, pause and ask, rather than anything more elaborate like a review queue. It is off by default here so the worked example later can run without you sitting at the keyboard, but for a first real run against a question you care about, turning it on for a session or two is a reasonable way to see exactly what the agent is doing before you trust it unattended.
Step 4: main.py, the command-line entry point
"""Command-line entry point for the research assistant.
Usage:
python main.py "What are the main differences between HTTP/2 and HTTP/3?"
Requires OPENAI_API_KEY to be set in the environment.
"""
import sys
from agent import run_agent
def main():
if len(sys.argv) < 2:
print('Usage: python main.py "your question"')
sys.exit(1)
question = " ".join(sys.argv[1:])
result = run_agent(question, require_approval=False, verbose=True)
print("\n" + "=" * 60)
print("ANSWER")
print("=" * 60)
print(result["answer"])
if result["sources"]:
print("\nSOURCES")
for s in sorted(result["sources"], key=lambda s: s["number"]):
title = s.get("title", "")
print(f"[{s['number']}] {title} - {s['url']}")
print(f"\n(finished in {result['steps']} step(s))")
if __name__ == "__main__":
main()
This file does nothing clever on purpose. It reads the question from the command line, calls run_agent, and prints the answer followed by a numbered source list. With verbose=True, run_agent also prints each tool call as it happens, which is the cheapest debugging tool you have when something looks wrong: you can see exactly what was searched, what was read, and in what order, without adding any logging of your own.
Running it: a worked example
The transcript below shows the shape of a run, not a real captured session. To avoid presenting fabricated search results as if they were verified, the URLs use example.com and example.org, domains reserved for documentation rather than live content. When you run this yourself, the agent will search and cite whatever real pages are live on the web at the time, and both the wording and the exact sources will differ from one run to the next.
$ python main.py "What are the main differences between HTTP/2 and HTTP/3?"
[step 1] tool call: web_search({'query': 'HTTP/2 vs HTTP/3 differences'})
[step 2] tool call: read_page({'url': 'https://example.com/http2-vs-http3'})
[step 2] tool call: take_note({'url': 'https://example.com/http2-vs-http3', 'note': 'HTTP/3 runs over QUIC on UDP instead of TCP'})
[step 3] tool call: web_search({'query': 'QUIC connection migration'})
[step 4] tool call: read_page({'url': 'https://example.org/quic-explainer'})
[step 4] tool call: take_note({'url': 'https://example.org/quic-explainer', 'note': 'QUIC supports connection migration across network changes'})
[step 5] tool call: finish({'answer': 'HTTP/3 replaces the TCP transport used by HTTP/2 with QUIC, a protocol built on UDP [1]. QUIC also supports connection migration, letting a connection survive a network change without a fresh handshake [2].', 'sources': [{'number': 1, 'url': 'https://example.com/http2-vs-http3', 'title': 'HTTP/2 vs HTTP/3'}, {'number': 2, 'url': 'https://example.org/quic-explainer', 'title': 'QUIC Explained'}]})
============================================================
ANSWER
============================================================
HTTP/3 replaces the TCP transport used by HTTP/2 with QUIC, a protocol
built on UDP [1]. QUIC also supports connection migration, letting a
connection survive a network change without a fresh handshake [2].
SOURCES
[1] HTTP/2 vs HTTP/3 - https://example.com/http2-vs-http3
[2] QUIC Explained - https://example.org/quic-explainer
(finished in 5 step(s))
Walking through it: the agent searched once, read one page, and took one note, then noticed it had only covered the transport change and searched a second, narrower query to cover connection migration specifically. It read a second page, took a second note, and only then called finish. If it had instead tried to cite example.org/quic-explainer without having called read_page on it, validate_finish would have rejected the call with an error naming that exact URL, and the loop would have given the model another turn to either read the page for real or drop the claim. That rejection path is the part of this project doing the real work; everything else is plumbing to get the model to a point where it can be checked.
Hardening: things that go wrong on real runs
Repeated searches and stalls
The version above caps the number of searches and page reads, but it does not notice if the model searches for near-duplicate queries, for example "HTTP/3 benefits" immediately after "HTTP/3 advantages", without making progress. A simple check you can add is to compare how much new information a step produced against how much it produced before, and nudge the model to stop or change tack if nothing changed.
def has_stalled(prev_note_count, prev_url_count, notes, fetched_urls):
"""Return True if the latest step added neither a new note nor a new url."""
return len(notes) == prev_note_count and len(fetched_urls) == prev_url_count
if __name__ == "__main__":
notes = [{"url": "https://example.com", "note": "a fact"}]
fetched_urls = {"https://example.com"}
stalled = has_stalled(prev_note_count=1, prev_url_count=1, notes=notes, fetched_urls=fetched_urls)
print("stalled:", stalled) # stalled: True, nothing changed since the last check
To wire this into run_agent, record len(notes) and len(fetched_urls) at the top of each iteration, compare them at the bottom, and if has_stalled returns True, append a plain user message to the conversation telling the model that the last step added nothing new and it should either try a different page or call finish with what it already has. This is a small addition, but it is the kind of thing that only becomes obvious once you have watched a real run spin on near-identical searches, which is a good reason to run the verbose version a few times before trusting the agent unattended.
Slow or hostile pages
REQUEST_TIMEOUT already stops one unresponsive server from hanging the whole run, and the truncation to MAX_CHARS_PER_PAGE already stops one enormous page from blowing the context budget. The remaining gap is content read_page was never built for, such as PDFs or pages that render their text with JavaScript after the initial HTML loads. In both cases, BeautifulSoup will typically find no <p> tags, and read_page will return its "no readable text was found" error, which the agent treats exactly like any other failed read: it tries a different source. That is a reasonable default for a learning project. If you want to support PDFs specifically, you would add a separate code path that detects the content type and runs a PDF text extractor instead of BeautifulSoup, which is listed as an extension later on rather than built here, to keep this project's surface area manageable.
Be a polite guest
read_page already identifies itself with a custom User-Agent rather than pretending to be a browser, which is a small but real courtesy. Two things this project does not do, and that you should add before pointing it at any site beyond occasional personal use, are checking robots.txt before fetching a page, which Python's standard library can help with through urllib.robotparser, and adding a short delay between requests so you are not hitting the same server repeatedly in quick succession. Neither is implemented here, both are worth doing, and both are also a reminder that this is a personal research tool built for learning, not a general-purpose web crawler, and the terms of service of any site you query still apply.
Testing the citation check
validate_finish is pure logic with no network calls, which makes it the easiest part of this project to test on its own, and arguably the most important part to test, since it is the thing standing between the agent and a fabricated citation. One wrinkle: importing agent.py creates an OpenAI client at module load time, and that client raises an error if no API key is available. Set OPENAI_API_KEY to any non-empty placeholder string before running this test file if you only want to check validate_finish and do not plan to call the real API.
"""Quick sanity checks for validate_finish.
Importing agent.py creates an OpenAI client, which raises an error if no
API key is set. Set OPENAI_API_KEY to any non-empty placeholder value if
you only want to run this test and do not plan to call the real API.
"""
from agent import validate_finish
def test_rejects_uncited_source():
fetched = {"https://real-source.example/a"}
args = {
"answer": "A grounded claim [1].",
"sources": [{"number": 1, "url": "https://never-fetched.example/b"}],
}
ok, problem = validate_finish(args, fetched)
assert not ok
print("rejected as expected:", problem)
def test_accepts_grounded_answer():
fetched = {"https://real-source.example/a"}
args = {
"answer": "A grounded claim [1].",
"sources": [{"number": 1, "url": "https://real-source.example/a"}],
}
ok, problem = validate_finish(args, fetched)
assert ok
print("accepted as expected")
if __name__ == "__main__":
test_rejects_uncited_source()
test_accepts_grounded_answer()
Running this file should print both success messages and exit without an assertion error. The value of a test like this is not in catching exotic bugs; it is in catching the regressions you will introduce yourself later, for example if you rewrite the system prompt, swap the model, or change the citation marker format and accidentally break the regular expression in validate_finish without noticing until an ungrounded citation slips through in a real run.
Common mistakes
- Letting the agent cite URLs it never actually fetched: a search result snippet is not evidence that the page was read; only validate the claim against URLs that read_page actually succeeded on
- Skipping the hard step, search and page-read caps and trusting the model to stop on its own, which is the exact gap Part 8 warned about
- Keeping every page's full text in the conversation forever, which quietly inflates both the token cost and the latency of every later call
- Treating a finish tool call as proof the agent is actually done, rather than checking its contents the same way you would check any other output you did not write yourself
- Fetching pages with no timeout and no identifying User-Agent, so one slow or blocking server can stall or get you blocked from an entire run
- Running near-duplicate searches without noticing, which burns through the search budget without adding new information
- Giving a tool that reaches the open internet no human checkpoint at all, when even a simple log of what was fetched, Part 9's lightest option, costs almost nothing to add
- Reaching for the full agent loop when a fixed three-step workflow, search then read the top result then summarize, would answer the same question more cheaply and more predictably, the judgement call from Part 4
Limitations and good next steps
- read_page only handles pages with readable HTML paragraphs; PDFs, JavaScript-rendered pages and paywalled content will come back as a clean error rather than useful text, which the agent handles gracefully but cannot work around
- There is no memory across runs; every question starts from nothing. Part 6's ideas about longer-term memory, a notes file or a small database keyed by topic, would let the agent reuse earlier research instead of rediscovering it
- There is no explicit planning step; the model decides what to search for one step at a time. Part 5's task decomposition could be added by asking the model to list two or three sub-questions before it starts searching, which tends to help on compound questions with several parts
- It is a single model doing both the searching and the writing; there is no split between a researcher role and a writer role, which some agent designs use to keep each prompt more focused
- There is no robots.txt check and no rate limiting between requests; both are straightforward to add with the standard library and are worth adding before you rely on this beyond occasional personal use
- Both the search wrapper and the model provider's tool-calling API are moving targets; if something that used to work stops working, check their current documentation before assuming the code here is wrong
Summary and what's next
This project put almost the whole path to work on one small, concrete task. The loop from Part 2 and Part 11 drove the back-and-forth between model and tools. The tool schemas from Part 3 told the model what it could do. The decision to build an agent rather than a workflow came from the judgement call in Part 4, because the right number and choice of sources genuinely depends on the question. Notes served as the external memory from Part 6, the trimming of old page text was context engineering from Part 7 made literal, the search and page-read budgets plus the validated finish call were the layered stopping conditions from Part 8, and the optional approval gates were the human checkpoints from Part 9. None of these pieces is complicated on its own; the result is useful because they were combined deliberately rather than bolted on after something went wrong.
From here, the most useful thing you can do is point this same pattern at a question you actually want answered and watch it work, with require_approval turned on for the first few runs so you can see exactly what it fetches before you let it run unattended. If you want to go further, pick one item from the limitations list, memory across runs, an explicit planning step, or a PDF reader, and add it yourself; each one is a small, self-contained change to a project you now understand end to end. And if you are weighing whether a future task needs this much machinery at all, Part 1 and Part 4 are worth rereading: an agent earns its complexity by handling real variability, and a fixed workflow is still the better choice whenever the steps do not actually need to change from one run to the next.
Comments (0)
No comments yet. Be the first to share your thoughts.