Where we are, and what this part covers

In part 1 you decided what the doorstep greeter should actually do, and worked through the rules for filming people on your own property: a visible sign, consent before storing anyone's face, and a firm line that the camera must never film the public pavement or neighbours more than necessary. In part 2 you chose the hardware it will run on: a Raspberry Pi 5 with an AI HAT+ (or a Jetson Orin Nano as the alternative), a Camera Module 3 in its NoIR variant with an infrared illuminator for night use, a small speaker, a PIR motion sensor, and an outdoor enclosure.

This part has no project code to run; that starts in part 4 when you set up the Pi itself. Instead, treat this article as the owner's manual for the software engine you are about to install. Every practical decision you will make later in this series, which similarity threshold to pick, how many photos to take when enrolling a person, why the system struggles at dusk, follows directly from four ideas: detection, alignment, embeddings and matching. Understanding them now will make the code in parts 5 and 6 feel like an implementation of something you already understand, rather than a recipe you are copying.

The four-stage pipeline: detect, align, embed, match

Imagine describing a visitor to a friend over the phone so they can tell whether it is someone you both know. You would not read out a list of pixel brightness values. You would first notice that there is a face there at all, as opposed to a coat or a parcel. You would then mentally straighten it out, correcting for the fact that they are looking slightly to one side or tilting their head. You would then reduce what you saw to a handful of distinguishing features, the shape of the jaw, the spacing of the eyes, rather than the whole image. Finally, your friend would compare that description against the people they already have in mind and decide whether it matches well enough to be confident.

A face recognition system does the same four things, just with numbers instead of words:

  • Detection: find where in the camera frame a face is, and draw a box around it.
  • Alignment: straighten and resize that boxed region so the face is in a standard pose and a standard size.
  • Embedding: feed the aligned face through a trained neural network that turns it into a list of numbers, a vector, that represents its identity.
  • Matching: compare that vector against the vectors of people you already know, and decide whether any of them is close enough to count as a match.

Each stage is a separate, swappable piece of software. In part 5 you will pick one specific face detector to run on the Pi, and in part 6 one specific embedding model, most likely from InsightFace or a similarly maintained open-source library, with exact version numbers pinned at that point rather than here. But whichever library you end up using, the shape of the pipeline above stays the same, and that shape is what this article is about.

Stage 1: Finding faces in the frame (detection)

Detection answers one narrow question: is there a face in this image, and if so, roughly where? The output is usually a bounding box, four numbers describing a rectangle, x, y, width and height, together with a confidence score, and often a handful of landmark points marking the centres of the eyes, the tip of the nose and the corners of the mouth. What detection does not tell you is whose face it is. That is an important distinction to hold onto, because it is easy to assume that a detector being very confident that a face is present means the system is equally confident about identity. It does not. A detector can be entirely certain there is a face at the door and have no idea at all whether it belongs to a household member, a delivery driver, or a stranger. Recognition is a separate job, built on top of detection, not included in it.

Face detectors have gone through a few generations. Early systems, notably the Viola-Jones method from the early 2000s, slid a small window across the image at many scales and checked each position against simple patterns of light and dark rectangles, called Haar features, that tend to appear around eyes and noses. It was fast enough to run on the modest hardware of its day, but it struggled with faces that were tilted, partly covered, or poorly lit. Later approaches combined hand-designed features called Histogram of Oriented Gradients with a linear classifier, an approach used for years inside the popular dlib library, which improved robustness somewhat but still relied on features a person had designed by hand rather than features learned from data.

Modern detectors, the kind you will actually use in part 5, are convolutional neural networks trained end to end on large collections of photographs where every face has been manually marked with a box. Names you will come across include MTCNN, RetinaFace, and SCRFD, the last of which is the detector bundled with InsightFace and designed specifically to be fast on modest hardware, which is exactly why it is a sensible choice for a Raspberry Pi. These networks learn, from thousands of examples, what a face looks like across a wide range of poses, lighting conditions, ages and partial occlusions, rather than relying on a programmer's guess about which rectangles of brightness matter.

Detection is not infallible in either direction. A false positive is the detector reporting a face where there is none, a shadow, a door knocker, a patterned coat can occasionally be mistaken for one, especially by less accurate detectors. A false negative is a real face that the detector misses entirely, which happens more often than you might expect when someone is looking almost side-on to the camera, has their face partly covered by a hood or a hand, or is standing in strong backlight with the sun behind them. This is exactly why the project plan in part 2 pairs the camera with a PIR motion sensor rather than relying on detection alone to notice that someone has arrived, and why part 10 later in this series is devoted entirely to making detection and everything downstream of it work reliably at night, in rain, and in glare.

Stage 2: Lining the face up (alignment)

A neural network that turns a face into an identity vector sees pixels, not geometry. If the same person's face is tilted ten degrees in one photo and perfectly level in another, those two images contain quite different arrangements of pixels even though the underlying face is identical. Without correcting for this, the network would have to learn to be invariant to every possible rotation, scale and position a face might appear at within a camera frame, on top of everything else it needs to learn about identity. Alignment removes that burden by doing the geometric correction explicitly, with simple mathematics, before the image ever reaches the identity network.

The landmarks produced alongside the bounding box in stage 1, typically the centre of each eye plus a few other points, are used to work out how far the face needs to be rotated, scaled and cropped so that it ends up in a standard pose: eyes level, face centred, a fixed distance between the eyes, and a fixed output size in pixels, commonly a small square such as 112 by 112, though the exact figure depends on whichever embedding model you pair it with in part 6, so treat that number as typical rather than universal.

Here is the geometry behind the rotation step, worked through with small numbers so you can see exactly what is happening. Suppose the detector reports the left eye at pixel coordinates (30, 42) and the right eye at (70, 40), using image coordinates where y increases downward.

🐍Python
import numpy as np

# Example eye coordinates reported by the face detector, in pixels (x, y)
left_eye = np.array([30, 42])
right_eye = np.array([70, 40])

# Vector from the left eye to the right eye
dx = right_eye[0] - left_eye[0]
dy = right_eye[1] - left_eye[1]

# Angle of the eye line relative to horizontal, in degrees
angle_degrees = np.degrees(np.arctan2(dy, dx))
print(f"dx={dx}, dy={dy}")
print(f"Eye-line angle: {angle_degrees:.2f} degrees")

Running this gives dx = 40 and dy = -2, so the eye line is tilted by about -2.86 degrees from horizontal, meaning the right eye sits very slightly higher than the left in the image. Alignment rotates the cropped face by the opposite of that angle, about +2.86 degrees here, so the eyes end up level. In a real photo the tilt is often larger than this toy example, perhaps ten or twenty degrees if someone is leaning towards the camera or standing at an angle, and the same arctan2 calculation handles that just as easily. After rotating, the face region is rescaled so the distance between the eyes matches a fixed target, and cropped to the fixed output size the embedding network expects. This whole alignment step is pure geometry, implemented with simple image-warping functions already available in OpenCV, and costs very little computation compared with the two neural-network stages either side of it, which matters on a Raspberry Pi where compute budget is limited.

Stage 3: Turning a face into numbers (the embedding)

This is the stage that does the actual work of representing identity, and it is worth building a clear mental picture of what it produces before you see any code for it in part 6. Picture a space with several hundred dimensions, far more than the three you can draw, in which every aligned face photograph becomes a single point. A well-trained embedding network places points so that different photographs of the same person, taken on different days, in different clothes, with different expressions, cluster close together in that space, while photographs of different people land further apart. The alignment step in stage 2 has already removed differences in pose and scale; the embedding network's job is to go further and also become insensitive to lighting, expression and minor variations in camera angle, while staying sensitive to the underlying facial structure that actually distinguishes one person from another.

The network that produces this vector is a convolutional neural network, trained in advance on a very large collection of face photographs labelled by identity, meaning many photos each for many thousands of different people. You will not train this network yourself; part 6 uses a ready-made, pre-trained model, most likely from InsightFace, and the exact package and version will be given there rather than guessed at here. It still helps to know roughly how such a network is trained, because it explains why the resulting vectors behave the way they do. Two training approaches are common in published face recognition research. One, triplet loss, takes three images at a time, an anchor photo of a person, a positive photo of the same person, and a negative photo of someone else, and nudges the network's weights so the anchor and positive end up closer together in the embedding space than the anchor and the negative are, by at least some margin. The other, used by ArcFace-style models including those in InsightFace, adds a margin directly in terms of the angle between vectors during training, which has the effect of pushing different identities further apart in the embedding space than a plain classification loss would. The details matter mainly to researchers training these models from scratch; what matters to you is the result, a function that reliably turns an aligned face photo into a vector where closeness means likely the same person.

The output is a fixed-length vector regardless of anything about the input photo beyond its content, commonly a few hundred numbers, 512 is a figure you will often see quoted for ArcFace-style models, though this varies between models and you should check whichever one you actually install in part 6 rather than assume a specific length. Two properties of this vector are worth remembering because they explain the matching stage that follows. First, it is the direction of the vector in that high-dimensional space that carries identity information, not simply its raw values, which is why similarity between two embeddings is usually measured with an angle-based comparison rather than comparing the numbers one by one. Second, an embedding is not an image and cannot be viewed as one; reconstructing a recognisable photograph from it is a non-trivial research problem rather than something that can be done trivially, but it has not been shown to be impossible either, and more importantly it is still derived from someone's face. Treat it, and store it, as biometric personal data, which under UK and EU GDPR and under US state laws such as Illinois's BIPA, discussed in part 1, is exactly what it legally is. That is why this project stores embeddings rather than raw photographs, encrypts them where practical, deletes them on request, and applies an automatic retention limit, all of which will be built in the later hands-on parts.

Stage 4: Deciding who's who (matching)

Once you have an embedding for the person currently standing at the door, the final step is to compare it against a small gallery, the embeddings of people you have already enrolled and stored in the local SQLite database described in the project plan. Ideally each known person has more than one stored embedding, taken from a couple of different photos, to capture a little natural variation in lighting or expression. Matching, sometimes called one-to-many or 1:N search in the research literature, means comparing the new embedding against every embedding in the gallery and finding the closest one.

The most common way to measure closeness between two face embeddings is cosine similarity, which looks at the angle between two vectors rather than the straight-line distance between them. Two vectors pointing in almost the same direction, regardless of their length, get a cosine similarity close to 1. Two vectors pointing in completely unrelated directions get a value close to 0, and in theory opposite directions give -1, though in practice well-trained face embeddings for different people rarely end up pointing in genuinely opposite directions. Here is the idea worked through with small, made-up vectors standing in for the real 512-dimensional ones, so the arithmetic stays visible.

🐍Python
import numpy as np

def cosine_similarity(a, b):
    a = np.array(a, dtype=float)
    b = np.array(b, dtype=float)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

# Toy 4-dimensional "embeddings" standing in for the real ~512-dimensional ones
alice_enrolled = [0.8, 0.1, 0.3, 0.5]   # stored when Alice was enrolled
alice_today    = [0.75, 0.15, 0.35, 0.48]  # computed from today's doorstep photo
stranger       = [0.1, 0.9, 0.2, -0.3]      # computed from an unrelated visitor

print("Alice enrolled vs Alice today:", cosine_similarity(alice_enrolled, alice_today))
print("Alice enrolled vs a stranger: ", cosine_similarity(alice_enrolled, stranger))

Running this prints a similarity of roughly 0.99 between Alice's two embeddings, taken on different days, and roughly 0.08 between Alice's embedding and the stranger's. In this toy example the gap between a genuine match and a non-match is huge and obvious. In real life, with real cameras, real lighting and real pose variation, the gap is usually still clear but narrower than this, and occasionally two different people's embeddings land closer together than you would like, or the same person's embedding on a bad lighting day lands further away than usual. That is why you set a threshold, a similarity value above which you accept the match and below which you treat the visitor as unknown, and why you tune that threshold using your own test photos in part 6 rather than borrowing a number from a tutorial or forum post written for a different camera, a different library version, and different lighting. Some libraries report Euclidean distance between normalised vectors instead of cosine similarity; for vectors that have been scaled to length 1, the two measures are directly related to each other, so it matters far less which one your chosen library defaults to than it matters that you understand which direction counts as a better match, smaller distance or larger similarity, for whichever one you are using.

Moving the threshold changes two kinds of mistake in opposite directions. A false accept is the system deciding a stranger is someone it knows, which happens more often when the threshold is set too low. A false reject is the system failing to recognise someone it does genuinely know, treating a household member as an unknown visitor, which happens more often when the threshold is set too high. For a doorstep greeter, a false reject is mildly annoying, Alice gets asked her name again, or the system simply does not greet her by name that time. A false accept matters more, particularly for the flagged-visitor feature covered later in this series, where a match drives an alert to the owner's phone. This is precisely why the project's rule, set out back in part 1 and worth restating here because it follows directly from everything in this section, is that a match never triggers an automatic action on its own, not unlocking a door, not any other action beyond a greeting or a notification. A match is a statistical suggestion produced by comparing two vectors against a threshold you chose; it is not proof of identity, and the system is built throughout this series so that a human, the owner, always makes the actual decision about what a flagged match means.

Why these details matter for the greeter on your doorstep

Everything above has concrete consequences for the device you are building. Outdoor lighting changes far more than indoor lighting ever does, from bright midday sun to deep shade to streetlight orange at night, and while the NoIR camera and infrared illuminator chosen in part 2 help keep brightness more consistent after dark, extreme shadows and strong backlight can still shift an embedding enough to affect matching, which is one of several reasons part 6 recommends enrolling each person with more than one photo, and part 10 deals with outdoor conditions specifically. Camera angle matters too: a doorbell-height camera usually looks slightly up at a visitor's face rather than straight on, and alignment corrects the in-plane tilt of the eyes but cannot fully undo a head that is genuinely turned away from the lens, which is part of why mounting height and angle get their own attention in part 2 and again when the enclosure goes up in part 12.

Detection and embedding are the two stages that run a full neural network over the image, and they are the two stages that benefit from the AI HAT+ or Jetson Orin Nano you chose in part 2, which exist specifically to run these networks fast enough that a visitor is recognised in roughly the time it takes to say hello, rather than after a multi-second delay on the Pi's general-purpose CPU alone. Alignment and matching, by contrast, are comparatively simple arithmetic, image warping and vector comparisons, and run comfortably on the CPU without needing acceleration. Finally, storing embeddings rather than photographs, as this project does, is good practice for keeping the amount of sensitive data on the device small, but it does not change the legal status of what you are storing: an embedding derived from someone's face remains biometric personal data under GDPR in the UK and EU and under biometric privacy laws in several US states, so consent before storage, encryption at rest where practical, a sensible retention limit, and an easy way to ask for deletion all remain necessary regardless of whether you are storing a photo or a vector of numbers, exactly as planned in part 1.

Common mistakes and things that go wrong

A handful of misunderstandings come up repeatedly with systems like this, and it is worth naming them plainly now, before you write any code, so you recognise them when you meet them in practice.

  • Confusing a detection confidence score with an identity confidence score. A detector being very sure there is a face at a given position says nothing at all about whose face it is; that judgement comes entirely from the later embedding and matching stages.
  • Enrolling a person from a single photograph. One photo captures one pose, one lighting condition and one expression; a little variety across two or three photos makes matching noticeably more reliable, especially outdoors where lighting is inconsistent.
  • Copying a similarity threshold from a tutorial, forum post or someone else's project without testing it against your own camera, your own lighting and the actual faces of the people who live in or visit your house.
  • Assuming an embedding is safe to store loosely because it is not a photograph. Treat it as biometric personal data in every practical sense, because that is both the cautious choice and the one consistent with GDPR and similar laws.
  • Overlooking that accuracy is not the same for everyone. Face recognition systems, like other computer vision systems, have repeatedly been shown to work less reliably for some skin tones, ages and lighting conditions than others, depending on the data a given model was trained on. Test the system with everyone who lives in or regularly visits the house, under the lighting conditions you will actually use, and treat any inconsistency as a reason to add more enrollment photos or adjust the thresho
  • Treating any match, however high the similarity score, as proof of who someone is. A match is a strong hint produced by a model that occasionally gets things wrong, not a verdict, which is exactly why this project never lets a match trigger an automatic action by itself.
  • Forgetting that a photo or video of someone's face, held up to the camera, can fool a plain detection and matching pipeline with no further checks. This is a genuine limitation of everything described in this article, and it is exactly what part 11, on liveness checks, deals with later in this series.

The fairness point deserves a little more space, because it is easy to treat as an abstract concern rather than a practical one. The underlying cause is usually that a given pre-trained model was trained on a dataset that did not represent every skin tone, age group or lighting condition equally well, so its embeddings separate some groups of people more cleanly than others. You cannot retrain the underlying model yourself as part of this project, but you can and should test it properly: enrol everyone who actually lives in or visits the house, take enrollment photos in the lighting conditions the camera will really see, and if you notice the system struggling more with some household members than others, respond by adding more enrollment photos for them or adjusting the threshold for the system as a whole, rather than assuming the system is simply right. This is also a good reason, independent of fairness, never to let a flagged-visitor match trigger anything beyond an alert that a human reviews, which the project already commits to.

A quick mental model to carry forward

It helps to be able to hold the whole pipeline in your head in one go before you start building it. A camera frame arrives. A detector scans it and reports a bounding box, a confidence score and a handful of landmarks for each face it finds, answering only "is there a face, and where." Alignment uses those landmarks to rotate, scale and crop the face into a standard pose and a standard size, removing pose and scale differences before identity is ever assessed. A trained embedding network turns that standardised crop into a vector of a few hundred numbers, placing it in a space where closeness is meant to reflect shared identity. Matching compares that vector against a small gallery of vectors belonging to people you have already enrolled, using cosine similarity or an equivalent distance measure, and checks the result against a threshold you have chosen and tested yourself. Above the threshold, the system has a suggested name. Below it, the visitor is unknown. Either way, that output feeds into what the system says or who it alerts, never into anything that happens automatically without a person deciding.

Summary and what's next

Face recognition, as used in this project, is four distinct steps rather than one mysterious one: detection finds a face, alignment straightens it, an embedding network turns it into a vector that represents identity, and matching compares that vector against people you already know using a similarity score and a threshold. Each step has its own failure modes, missed or false detections, pose that alignment cannot fully correct, lighting that shifts an embedding, and a threshold that is too loose or too tight, and none of those failure modes turn a match into proof, which is exactly why the project keeps a human owner in charge of every decision beyond a greeting.

Part 4 picks the practical thread back up, setting up the Raspberry Pi itself, the camera, and the Python environment you will build everything on. Part 5 turns the detection stage described here into running code, drawing boxes around faces in a live camera feed. Part 6 adds alignment, embeddings and matching, building the gallery of known faces in SQLite and recognising the people you enrol, with real version numbers and real thresholds tested on your own camera rather than assumed in advance. Everything after that, voice, new-visitor enrollment with consent, flagged-visitor alerts, outdoor robustness and liveness checks, builds on the same four ideas covered here, so if a line of code later in the series seems opaque, it is worth returning to this article to see which of the four stages it belongs to.