Named Entity Recognition
Introduction
Named Entity Recognition (NER) is the task of finding named entities in text and labeling them with a category. It does two things at once: it locates the span of words that make up an entity, and it classifies that span into a type such as PERSON, ORGANIZATION, LOCATION, or DATE.
For the sentence "Barack Obama was born in Hawaii", an NER system should output that Barack Obama is a PERSON and Hawaii is a LOCATION. NER is a foundational step for information extraction, question answering, knowledge-graph construction, search, and document anonymization.
The hard part is that the same word can be different things in different contexts: "Washington" may be a person, a city, a state, or an organization. NER must use surrounding context, not just the word itself.
Framing NER as Token Classification
Entities are spans of one or more words, but most machine learning models naturally produce one label per token. NER bridges this gap by reframing span detection as a token classification problem: assign every single token a tag, and recover the spans from the pattern of tags. The standard tagging scheme for this is BIO (also called IOB).
B- (Begin)
Marks the first token of an entity, e.g. B-ORG for the start of an organization.
I- (Inside)
Marks a token inside / continuing the same entity, e.g. I-ORG for the second word of a two-word organization.
O (Outside)
Marks a token that is not part of any entity. Most tokens in ordinary text are O.
Each entity type gets its own B- and I- tag, so the tag set for a system that recognizes persons and organizations is { B-PER, I-PER, B-ORG, I-ORG, O }. An entity of length n is encoded as one B- followed by n−1 I- tags:
Why we need B- as well as I-
The B- tag is what lets the scheme split two adjacent entities of the same type. Consider "... met France Germany ..." (two separate countries with no word between them). Tagging both as I-LOC I-LOC would merge them into one entity; the correct encoding is B-LOC B-LOC, where the second B- signals that a new entity begins. An I- tag only ever continues the entity started by the most recent B- of the same type.
(Variants exist: the original IOB1 used B- only to separate adjacent same-type entities; the more common IOB2 / BIO used here always begins every entity with B-. Schemes like BIOES add explicit E- (end) and S- (single-token) tags.)
Interactive BIO Tagger
Below is a small rule + gazetteer tagger running live in your browser. It looks each token up in a dictionary of known names, organizations, and locations, applies a few rules (month names become dates, unknown capitalized words are guessed as people), and emits BIO tags. The first token of each entity gets B-, continuing tokens get I-. Toggle the example sentences, and try adding a word to the gazetteer to see the tags change.
Highlighted entities
Per-token BIO tags
3 entities found. B- begins an entity, I- continues it, and O is outside any entity.
Edit the gazetteer
Add a word or phrase to the tagger's dictionary and watch the tags update. Try adding treaty as an ORG, or a name the tagger currently misses.
Notice the weaknesses of pure rules: the capitalization heuristic can mislabel ordinary capitalized words, and anything missing from the gazetteer is silently dropped. Real systems learn these patterns from data instead — which is what the next section is about.
Features and Context Cues
Before deep learning, NER models relied on hand-engineered features computed per token. Even modern systems benefit from understanding which signals are informative:
- The word itself and its lowercased / lemmatized form.
- Orthographic shape: is it capitalized, all-caps, a number, mixed case (e.g. iPhone), does it contain digits or punctuation?
- Affixes: prefixes and suffixes (-ton, -ville, Inc., Corp.).
- Context window: the previous and next words and their tags. Trigger words like "Mr.", "said", "in", or "Inc." strongly hint at the neighboring entity type.
- Gazetteers: membership in curated lists of known people, companies, and places.
- Part-of-speech tags (entities are usually proper nouns).
Context is decisive. In "I flew to Jordan" the model should lean LOC; in "Michael Jordan scored" it should lean PER. The word alone cannot decide.
Modeling Approaches
NER models have to label tokens jointly because a token's correct tag depends on its neighbors' tags — for instance, I-PER can only legally follow B-PER or I-PER. Approaches differ mainly in how they encode context and how they enforce these label dependencies.
1. Conditional Random Fields (CRF)
A linear-chain CRF is a discriminative sequence model that scores an entire tag sequence y given the tokens x. It combines two kinds of features: emission scores (how well a token fits a tag) and transition scores (how likely one tag follows another). Because it models label dependencies explicitly, it learns that I-PER cannot follow O or B-ORG. The most likely tag sequence is decoded globally with the Viterbi algorithm, rather than choosing each tag independently and greedily.
2. BiLSTM-CRF
The classic neural NER architecture. A bidirectional LSTM reads the sentence left-to-right and right-to-left, producing a context-aware vector for each token (so the representation of "Jordan" reflects the words on both sides). Those vectors feed a CRF layer on top, which adds the same global transition constraints as a standalone CRF. The LSTM provides rich learned features; the CRF guarantees a coherent BIO sequence. Often the LSTM is fed word embeddings concatenated with character-level representations to handle rare and unseen words.
3. Transformer Token Classifiers (BERT fine-tuning)
The modern standard. A pretrained Transformer like BERT encodes the sentence with self-attention, producing deeply contextual token vectors. For NER you add a single linear classification head over each token's final hidden state that predicts its BIO tag, then fine-tune the whole model on labeled data. A CRF layer can still be added on top, but strong pretrained encoders often do well with just a softmax per token. One subtlety: BERT uses subword tokenization, so a word may split into several pieces — the convention is to tag the first subword and ignore the rest during training and decoding.
| Approach | Context | Label dependencies | Notes |
|---|---|---|---|
| Rules / Gazetteer | Local, hand-coded | None | Brittle, but interpretable; the demo above |
| CRF | Hand-engineered features | Explicit (transitions + Viterbi) | Strong classic baseline |
| BiLSTM-CRF | Learned, bidirectional | Explicit (CRF layer) | Pre-Transformer neural standard |
| BERT fine-tuning | Deep self-attention | Implicit (optional CRF head) | State of the art; subword tokenization |
Evaluation: Entity-Level F1
NER is not scored on per-token accuracy. Because most tokens are O, a model that labeled everything O could score 90%+ token accuracy while finding zero entities. Instead NER uses entity-level precision, recall, and F1, where a predicted entity counts as correct only on an exact span match: the predicted span must have the same boundaries and the same type as a gold entity.
where TP = predicted entities that exactly match a gold entity, FP = predicted entities with no exact gold match, and FN = gold entities the model missed.
Partial matches get no credit
If the gold entity is [Barack Obama]PER but the model predicts only [Obama]PER, that is both a false positive (a wrong span was predicted) and a false negative (the true span was missed). The same applies if the boundaries match but the type is wrong. This strict scoring is what the standard conlleval script and the seqeval library implement.
Because there are many entity types, scores are typically aggregated with a micro-average (pool all TP/FP/FN across types, then compute one F1 — weights types by frequency) or a macro-average (compute F1 per type, then average — weights every type equally).
Common Pitfalls
Watch out for
- • Invalid tag sequences (I- without a preceding matching B-)
- • Ambiguous words that need context (Jordan, Apple, Washington)
- • Nested or overlapping entities (BIO cannot represent these)
- • Inconsistent annotation of boundaries (include titles? articles?)
- • Subword tokenization misalignment with word-level labels
- • Domain shift: a news-trained model fails on biomedical or legal text
Good practice
- • Use a CRF or constrained decoding to forbid illegal transitions
- • Always report entity-level F1, never token accuracy
- • Add character / subword features for rare and unseen names
- • Combine learned models with gazetteers for high-precision recall
- • Fine-tune or adapt to your target domain
- • Keep annotation guidelines explicit and consistent
Key Takeaways
- NER both locates and classifies named entities (PERSON, ORG, LOC, DATE, and more).
- Span detection is reframed as token classification using the BIO scheme: B- begins an entity, I- continues it, O is outside.
- Adjacent same-type entities are separated because each entity starts with B- — an I- only continues the current span.
- Context cues — capitalization, affixes, neighboring trigger words, gazetteers — drive the prediction far more than the word in isolation.
- CRFs model label dependencies explicitly via transitions and Viterbi decoding; BiLSTM-CRF adds learned bidirectional features; BERT fine-tuning uses a contextual Transformer with a per-token classification head.
- Evaluate with entity-level precision / recall / F1 on exact span match — partial overlaps and wrong types earn no credit.