TF-IDF

Also: term frequency–inverse document frequency, tf-idf, TFIDF

A classic term-weighting scheme: term frequency (TF) rewards words that are frequent in a document, and inverse document frequency (IDF) penalizes words common across the corpus. High scores mark words that are distinctive for that document.

TF-IDF(t, d) = TF(t, d) × log(N / df(t))

Example

In the lesson's 6-document corpus, "learning" appears in 4 documents, so IDF = ln(6/4) ≈ 0.41, while "neural" appears in only 1, so IDF = ln(6) ≈ 1.79. In "Deep learning uses neural networks with multiple layers", "neural" outweighs "learning" by over 4×.

See it in the lesson: TF-IDF →

Related terms

More in Language & LLMs