Natural Language Processing (NLP)

From PRIMO.ai
Jump to navigation Jump to search

YouTube ... Quora ...Google search ...Google News ...Bing News

Natural Language Processing (NLP) is the set of techniques computers use to understand and generate human language, including text classification and Sentiment Analysis, using machine learning algorithms, statistical models, and linguistic rules. Today the most visible example is the Large Language Model (LLM), but NLP also covers search, translation, speech interfaces, information extraction, spelling and grammar checking, document classification, and the many small text-processing steps that sit underneath those applications.

The field sits where computer science meets linguistics. Its history runs through a handful of landmark moments: the 1954 Georgetown-IBM experiment, which machine-translated more than 60 Russian sentences into English and set off a wave of optimism; Noam Chomsky's Syntactic Structures (1957), which gave the field formal grammars; ELIZA (1966) and SHRDLU (1970), early dialogue and language-understanding programs; the 1966 ALPAC report, which cut machine translation funding after the early optimism faded; the statistical turn of the late 1980s and 1990s, driven by annotated corpora such as the Penn Treebank (1993); word embeddings (word2vec, 2013); neural sequence-to-sequence models with attention (2014 to 2016); the Transformer (2017); BERT (2018); GPT-3 (2020); ChatGPT (2022); and reasoning models trained with reinforcement learning (2024 to 2025).

The core problem has never changed. Human language is ambiguous at every level: a word can mean several things (lexical ambiguity), a sentence can have several structures (syntactic ambiguity), a structure can carry several meanings (semantic ambiguity), and a pronoun can point to several earlier mentions (anaphoric ambiguity). Meaning also depends on context, speaker intent, and world knowledge that never appears in the text. Every NLP technique on this page is a strategy for resolving that ambiguity well enough to do something useful.

The central mechanism today is statistical learning from very large text collections. A modern model learns to predict missing or upcoming pieces of text, and in the process it builds internal vector representations of words, phrases, and ideas. This rests on the distributional hypothesis from mid-century linguistics (Zellig Harris, J. R. Firth): words that occur in similar contexts tend to have similar meanings. Scale that idea up to trillions of words and billions of parameters, and you get systems that can translate, summarize, answer questions, and write code without being explicitly programmed for any of those tasks.

That is the paradigm shift. For decades, an NLP system was a pipeline of separate, hand-built components (a tokenizer, a part-of-speech tagger, a parser, a named entity recognizer) feeding a task-specific model trained on one labeled dataset. Writing in 2019, Victor Sanh described the first stage of the change The Best and Most Current of Modern Natural Language Processing | Victor Sanh - Medium:

  • For a long time, we used pre-trained word embeddings such as Word2Vec or Global Vectors for Word Representation (GloVe) to initialize the first layer of a neural network, followed by a task-specific architecture that is trained in a supervised way using a single dataset.
  • Recently, several works demonstrated that we can learn hierarchical contextualized representations on web-scale datasets leveraging unsupervised (or self-supervised) signals such as language modeling and transfer this pre-training to downstream tasks (Transfer Learning). Excitingly, this shift led to significant advances on a wide range of downstream applications ranging from Question Answering, to Natural Language Inference through Syntactic Parsing…

Since then the shift has gone much further. Pretrain-then-fine-tune gave way to prompting a single general model, then to instruction tuning and human feedback, and most recently to models trained with reinforcement learning to reason step by step and to use tools. The classic techniques have not disappeared. They remain essential for search engines, data preparation, privacy protection, low-resource languages, interpretability research, and evaluation, and they explain what large models are doing internally. That is why this page covers both.

Era Approximate years Core idea Representative systems
Rules and symbols 1950s to 1980s Hand-written grammars, dictionaries, and pattern rules Georgetown-IBM, ELIZA, SHRDLU
Statistical NLP Late 1980s to 2000s Probabilities estimated from annotated corpora HMM taggers, IBM alignment models, maximum entropy models, CRFs
Neural representations 2010 to 2017 Dense word vectors, recurrent networks, attention word2vec, GloVe, LSTMs, neural machine translation
Pretrain and fine-tune 2018 to 2021 Contextual representations transferred to many tasks ELMo, ULMFiT, BERT, GPT-2, T5
Prompted, instruction-tuned LLMs 2020 to 2024 In-context learning, instruction tuning, human feedback GPT-3, ChatGPT, Claude, Llama, Gemini
Reasoning and agentic models 2024 to present Reinforcement learning on verifiable tasks, test-time compute, tool use o1, DeepSeek-R1, and their successors


Language is the interface to most human knowledge, and NLP is the project of teaching machines to use it. The field moved from hand-written rules to statistics to learned representations to general-purpose language models, but the classic levels of language (sound, word, sentence structure, meaning, discourse, and context) never disappeared. They moved inside the models, where researchers are still learning to find them.


How NLP Systems Work

The classic pipeline

Before large language models, most NLP systems processed text in stages, each one adding a layer of structure. The sections later on this page cover each stage in depth:

  1. Acquire and clean text. Strip markup, fix encodings, remove boilerplate, normalize Unicode. See Text Preprocessing.
  2. Split and tokenize. Break text into sentences and tokens. See Tokenization / Sentence Splitting.
  3. Normalize. Case folding, stemming or lemmatization, and stop word removal, mainly for search and counting-based models. See Normalization.
  4. Tag. Assign a part of speech to each token. See Part-of-Speech (POS) Tagging.
  5. Parse. Recover sentence structure as constituency trees, dependency graphs, or shallow chunks. See Syntax (Parsing).
  6. Extract meaning. Find named entities, relations, coreference chains, and semantic slots. See Semantics.
  7. Model discourse and context. Track topics, dialogue state, and speaker intent. See Discourse (Dialog) and Pragmatics.
  8. Perform the task. Classify, retrieve, translate, summarize, or answer.

The weakness of this design is error cascading: a mistake in tagging corrupts the parse, which corrupts entity extraction, and so on. Each stage also needed its own annotated training data. Its strength is transparency. You can inspect every intermediate result, which is why pipeline tools such as NLTK, spaCy, Stanford CoreNLP, GATE, and Apache UIMA remain in heavy use.

The neural language model pipeline

A modern language model collapses most of those stages into one learned function. The sequence looks like this:

  1. Tokenization. Text is split into subword units drawn from a fixed vocabulary, typically 30,000 to 250,000 entries, using algorithms such as byte pair encoding (BPE), WordPiece, or the SentencePiece unigram model. Each token becomes an integer ID.
  2. Embedding. Each ID is looked up in a learned table and turned into a vector of a few hundred to several thousand numbers. Position information is added, today usually through rotary position embeddings (RoPE).
  3. Transformer layers. Dozens of layers alternate two operations. Attention lets every token gather information from other tokens; a feed-forward network (MLP) transforms each token's vector. A residual connection carries a running representation, often called the residual stream, through the whole stack.
  4. Prediction. The final vectors are turned into a probability distribution over the vocabulary. Decoder models such as GPT predict the next token; encoder models such as BERT predict masked tokens; diffusion language models iteratively fill in many masked positions at once.
  5. Pretraining. The model learns by making these predictions over trillions of tokens of web pages, books, code, and other text, adjusting its parameters to reduce prediction error.
  6. Post-training. Supervised fine-tuning on instruction-and-response examples turns a text predictor into an assistant. Preference optimization (reinforcement learning from human feedback, or direct preference optimization) shapes style and safety. Reinforcement learning with verifiable rewards (for example, checking math answers or running code tests) teaches extended step-by-step reasoning.
  7. Inference. At run time the model generates one token at a time (or, for diffusion models, refines many at once), using decoding strategies such as greedy selection, temperature sampling, or nucleus sampling. Systems often add retrieval (RAG), tool calls, and structured output formats.

Encoders, decoders, and encoder-decoders

The three transformer families suit different jobs:

  • Encoder-only models (BERT, RoBERTa, DeBERTa, ModernBERT) read the whole input at once in both directions. They are fast and compact, and they remain the workhorses for classification, named entity recognition, and retrieval embeddings.
  • Decoder-only models (the GPT family, Llama, Claude, Gemini, Qwen, DeepSeek) read left to right and generate text. Almost all general-purpose assistants are decoders.
  • Encoder-decoder models (T5, BART, many translation systems) encode an input and then generate an output conditioned on it, a natural fit for translation and summarization.

Adapting a model to a task

Practitioners now choose among several adaptation strategies, roughly in order of cost:

  • Zero-shot prompting: describe the task in natural language.
  • Few-shot (in-context) learning: include a handful of worked examples in the prompt.
  • Embeddings plus a small classifier: encode text with a pretrained embedding model and train a lightweight model on top.
  • Retrieval augmentation: fetch relevant documents and place them in the prompt. See RAG and Semantic Search.
  • Fine-tuning: update some or all model weights on task data, often with parameter-efficient methods such as LoRA. See Fine-tuning.

Where uncertainty remains

We can describe these mechanisms precisely and still not know exactly what a trained model represents. Researchers continue to debate how much linguistic structure models learn versus how much they approximate with surface statistics, why they sometimes produce confident falsehoods (hallucination), why performance can collapse on rephrased inputs, and how well benchmark scores predict real-world reliability. Interpretability research (covered under Recent Developments) is beginning to answer some of these questions from the inside.

The Information Science Underneath

NLP has no forces or fields, but it does have physics-like regularities: information theory, statistical laws of word frequency, the geometry of vector spaces, and scaling relationships that predict how models improve. These explain why the mechanisms above work and where they break.

Language as a probability distribution

A language model assigns a probability to any sequence of tokens by multiplying conditional probabilities (the chain rule):

P(w_1, ..., w_n) = P(w_1) * P(w_2 | w_1) * ... * P(w_n | w_1, ..., w_(n-1))

Training minimizes cross-entropy, the average number of bits (or nats) the model needs to encode the true next token. The standard intrinsic metric, perplexity, is the exponential of that cross-entropy:

Perplexity = exp( -(1/N) * sum_i ln P(w_i | context) )

Intuitively, a perplexity of 20 means the model is as uncertain as if it were choosing uniformly among 20 options at each step. Claude Shannon estimated in 1951 that printed English carries roughly 0.6 to 1.3 bits of information per character once long-range context is exploited, far below the 4.7 bits needed to pick uniformly among 26 letters. That redundancy is exactly what language models learn to exploit, and it is why prediction is such a powerful training signal: to predict text well, a model must capture grammar, facts, and reasoning patterns.

Zipf's law and Heaps' law

Word frequencies follow a heavy-tailed distribution. Under Zipf's law, the frequency of the word with rank r is roughly proportional to 1 / r. In English, "the" alone accounts for around 7% of running words, the top hundred words cover about half of a typical text, and a vast tail of words appears only once or twice. Heaps' law describes vocabulary growth: the number of distinct words grows roughly as V(N) = K * N^beta, with beta typically between 0.4 and 0.6, so new words keep appearing no matter how much text you read.

These laws drive several design choices on this page. Stop word removal exists because the head of the distribution carries little topical information. Stemming and lemmatization exist to pool counts across rare variants. Subword tokenization exists because any fixed word vocabulary will always miss part of the long tail.

The distributional hypothesis and embedding geometry

Word embeddings turn the distributional hypothesis into geometry. Words become vectors, and similarity becomes the cosine of the angle between them:

cos(a, b) = (a . b) / (|a| * |b|)

The skip-gram model of word2vec trains vectors so that a word predicts its neighbors; Levy and Goldberg (2014) showed that this is implicitly equivalent to factorizing a matrix of pointwise mutual information (PMI) between words and contexts. The famous analogy result, vec(king) - vec(man) + vec(woman) landing near vec(queen), shows that some relationships become roughly linear directions. Modern contextual models produce a different vector for each occurrence of a word, so "bank" in "river bank" and "bank loan" end up in different regions of the space.

Attention and its cost

The transformer's core operation is scaled dot-product attention:

Attention(Q, K, V) = softmax( Q K^T / sqrt(d_k) ) V

Each token produces a query (Q), a key (K), and a value (V). The dot product of a query with every key decides how much of each value to mix in. Dividing by the square root of the key dimension keeps the softmax from saturating. Because every token attends to every other token, compute grows with the square of sequence length n: doubling the context quadruples the attention cost. During generation, the stored keys and values (the KV cache) grow linearly with context and often dominate memory. This quadratic cost motivates sliding-window and sparse attention, linear-attention variants, and state space models such as Mamba, as well as hybrid architectures that mix them. A 2026 ACL best paper studied exactly what local (windowed) attention can and cannot express, a reminder that efficiency choices change what a model can represent.

Tokenization as compression

Byte pair encoding starts from individual bytes or characters and repeatedly merges the most frequent adjacent pair into a new symbol until the vocabulary reaches a target size. The result is a compression scheme: common words become single tokens, rare words split into pieces. For English, GPT-style tokenizers average roughly four characters per token. Compression is uneven across languages, though. Petrov and colleagues (NeurIPS 2023) found that the same text translated into different languages can require up to 15 times as many tokens, a "tokenization premium" that raises cost and latency and shrinks the usable context window for speakers of under-served languages.

Tokenizer-free models replace the fixed vocabulary with learned segmentation. Meta's Byte Latent Transformer groups bytes into variable-length patches using the entropy of a small byte-level model's next-byte prediction:

H(x_i) = - sum_v p(v | x_1, ..., x_(i-1)) * log p(v | x_1, ..., x_(i-1))

A new patch starts where entropy is high (the next byte is hard to predict), so the expensive main model spends more compute on surprising regions of text and less on predictable ones.

Scaling laws

Model loss falls smoothly and predictably as parameters, data, and compute grow. The Chinchilla analysis (Hoffmann and colleagues, 2022) fit:

L(N, D) = E + A / N^alpha + B / D^beta

where N is the number of parameters, D the number of training tokens, and E an irreducible loss. Training compute is approximately C = 6 * N * D floating-point operations, and the compute-optimal recipe trains on roughly 20 tokens per parameter. Many current models deliberately "overtrain" smaller models on far more tokens because smaller models are cheaper to run. Scaling also has a supply side: Epoch AI researchers estimated the stock of high-quality public human text at roughly 300 trillion tokens and projected that frontier training could exhaust it sometime between 2026 and 2032, which helps explain the growing interest in synthetic data, licensed data, and more data-efficient training.

Human versus machine data efficiency

Children acquire their native language from fewer than 100 million words of input by early adolescence. Large language models typically need three to four orders of magnitude more data. That gap is both an engineering opportunity and a scientific puzzle, and it motivates the BabyLM Challenge described below. It also matters for the roughly 7,000 languages that will never have trillions of tokens of text.

Why This Matters

Language is the interface to knowledge. Most of what people know is recorded in text and speech: law, medicine, science, software documentation, history, and everyday communication. Systems that can read and write language at scale change how people search, learn, write, program, and access services. NLP powers search engines, translation, accessibility tools such as captions and screen readers, customer service, contract review, clinical documentation, education, and the conversational assistants listed at the top of this page.

Engineering consequences. The shift to general models has changed how NLP is built. Many tasks that once required a labeled dataset and a custom model can now be prototyped with a prompt in minutes. At the same time, cost, latency, privacy, and reliability push many production systems back toward smaller fine-tuned encoders, retrieval systems, and classic preprocessing, which is why the "old" techniques on this page still matter.

Scientific consequences. Language models give linguists and cognitive scientists a new kind of experimental subject: a system that learns language from data and can be probed, modified, and retrained. A lively debate, summarized under Recent Developments, asks what these models can teach us about human language learning and whether they challenge or support long-standing linguistic theories.

Equity and coverage. Performance remains strongly skewed toward English and other high-resource languages. Tokenization premiums, scarce training data, and thin evaluation coverage mean that speakers of many languages pay more for worse results. Multilingual benchmarks, openly licensed corpora, and tokenizer-free architectures are partly responses to this gap.

Risks and limitations. Language systems can hallucinate facts, reproduce social biases from training data, leak private information, and generate persuasive misinformation at scale. Their apparent fluency can mask shallow understanding, and benchmark gains do not always transfer to messy real-world inputs. Reasoning models reduce some errors on verifiable tasks but can produce long, convincing explanations for wrong answers.

What would change, and what remains uncertain. If tokenizer-free models, diffusion decoding, or more data-efficient training mature, NLP systems could become cheaper, faster, and fairer across languages. If interpretability research succeeds, we may be able to verify what a model knows and why it answers as it does. It remains unclear how far scaling and reinforcement learning will carry reasoning beyond domains with automatically checkable answers, and whether models' internal representations resemble human linguistic knowledge in any deep sense.

Recent Developments

The period from late 2024 to October 2026 brought changes at every level of the NLP stack, from how text is split into units to how models reason, and renewed scientific debate about what language models reveal about language itself.

Reasoning models and reinforcement learning

OpenAI's o1 (September 2024) showed that models trained to produce long internal chains of reasoning before answering perform far better on math, coding, and logic problems. DeepSeek's R1 (released January 2025) made the recipe public and, in September 2025, became the first major large language model described in a peer-reviewed Nature paper. The team showed that reasoning behavior can be incentivized through reinforcement learning that rewards only correct final answers, without human-written reasoning examples. Their first model, R1-Zero, developed long, self-checking reasoning traces on its own but mixed languages and produced hard-to-read output, so the final R1 pipeline added a small amount of supervised "cold-start" data and a reward for language consistency. The paper reported that the reinforcement learning stage cost about $294,000 on 512 H800 GPUs, on top of the cost of the underlying base model.

What this establishes: reinforcement learning with verifiable rewards can substantially improve reasoning on tasks with checkable answers, and independent groups have reproduced the basic effect. What it does not establish: that the same approach works as well where answers cannot be checked automatically, or that the visible reasoning trace faithfully reflects the computation the model actually performs. Reviewers of the Nature paper noted that how the model arrives at its answers remains only partly understood.

The end of the tokenizer?

Subword tokenization has been one of the few hand-engineered components left in the NLP stack, and it causes well-known problems: difficulty counting letters, brittleness to typos, uneven costs across languages, and odd behavior on numbers and code. In a fitting bit of timing, the 2016 paper that introduced BPE for neural machine translation (Sennrich, Haddow, and Birch) received ACL's Test-of-Time award in 2026, just as serious alternatives emerged.

  • Byte Latent Transformer (BLT). Meta and University of Washington researchers (December 2024; ACL 2025) built a byte-level model that groups bytes into dynamic, entropy-based patches. In the first FLOP-controlled scaling study of byte-level models, up to 8 billion parameters and 4 trillion training bytes, BLT matched tokenizer-based Llama-style models while improving inference efficiency and robustness to character-level noise.
  • H-Net. Sukjun Hwang, Brandon Wang, and Albert Gu (Carnegie Mellon and Cartesia; July 2025; ICLR 2026) introduced dynamic chunking, a mechanism that learns content- and context-dependent segmentation end to end. At matched compute and data, a byte-level H-Net with one stage of hierarchy outperformed a strong BPE-tokenized transformer, and a two-stage H-Net matched a token-based transformer twice its size. The authors report the current implementation trains about twice as slowly as a standard model, and follow-up work has applied the idea to morphologically rich languages.

Neither approach has yet displaced tokenizers in frontier production models. They are best read as strong evidence that the tokenizer is removable, not yet as proof that removing it is cheaper at the largest scales.

The encoder revival

While generative models dominate attention, small encoders still do most of the world's classification and retrieval work. In December 2024, Answer.AI, LightOn, and collaborators released ModernBERT, the first major update to the BERT recipe in years: trained on 2 trillion tokens including code, with a native 8,192-token context (versus 512 for most older encoders), rotary position embeddings, alternating local and global attention, and hardware-aware optimizations that made it the fastest encoder in its class. Its release post noted that the original BERT was still among the most-downloaded models on Hugging Face more than six years after release.

Specialized encoders also compete with large models on extraction tasks. GLiNER (NAACL 2024) matches entity-type descriptions to text spans with a compact bidirectional encoder and, in the authors' zero-shot evaluations, outperformed both ChatGPT and fine-tuned large models on several NER benchmarks while extracting all entities in parallel. The practical lesson for 2026: use a large model to prototype, then consider whether a small encoder can do the job at a fraction of the cost and latency.

Embeddings and retrieval

Embeddings power semantic search, clustering, recommendation, and retrieval-augmented generation. The Massive Multilingual Text Embedding Benchmark (MMTEB) (ICLR 2025), a community expansion of MTEB, covers more than 500 quality-controlled tasks across 10 task categories and over 250 languages, including instruction following, long-document retrieval, and code retrieval. A notable finding: although embedding models built on large language models led on some languages, the best publicly available model across languages was the much smaller multilingual-e5-large-instruct. MMTEB also introduced downsampled task sets that preserve model rankings at a fraction of the compute, lowering the barrier for low-resource communities.

Diffusion language models

Diffusion models, which dominate image generation, are now being applied to text. Instead of writing left to right, they start from a fully masked sequence and refine many positions in parallel over several steps.

  • LLaDA (February 2025) scaled a masked diffusion model to 8 billion parameters and reported performance competitive with Llama 3 8B on in-context learning, along with better handling of the "reversal curse" (failing to infer "B is A" after learning "A is B").
  • Gemini Diffusion, demonstrated by Google DeepMind at I/O in May 2025, reported a sampling speed of about 1,479 tokens per second. As of mid-2026 it remained an experimental demo rather than a generally available model.
  • Mercury, from Inception Labs, is marketed as the first commercial-scale diffusion language model, with vendor-reported speeds above 1,000 tokens per second on H100 GPUs.
  • LLaDA 2.0 (Ant Group, December 2025) scaled diffusion to a 100-billion-parameter mixture-of-experts model with open weights.

Demonstrated: large speed advantages and competitive quality at small and medium scale. Still open: whether diffusion models can match the best autoregressive models on difficult reasoning and long-form writing, and how to adapt post-training and caching techniques built for left-to-right generation. Most published speed figures are vendor-reported.

Long context and its limits

Context windows grew from a few thousand tokens to a million or more, raising the question of whether retrieval still matters. A July 2025 study from Chroma, Context Rot, held task difficulty constant while varying only input length across 18 models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3. Performance degraded as inputs grew, often non-uniformly and well before the advertised limit, even on tasks as simple as repeating a string. Counterintuitively, models did better on randomly shuffled distractor text than on logically coherent documents. This extends earlier findings such as Lost in the Middle (2024), which showed that information placed mid-context is used less reliably. The practical consequence: careful retrieval and "context engineering" remain important even with very long windows. Chroma builds retrieval infrastructure, so independent replication on other task types is valuable.

Inside the model: interpretability meets linguistics

In March 2025, Anthropic published On the Biology of a Large Language Model, using attribution graphs built on a cross-layer transcoder with about 30 million interpretable features to trace how Claude 3.5 Haiku turns specific prompts into answers. Two findings matter directly for NLP:

  • Shared multilingual representations. The model uses a mix of language-specific and language-independent circuits. When the same simple sentence was processed in English, French, and Chinese, much of the computation ran through shared, abstract features, and the shared fraction grew with model scale: Haiku shared more than twice the proportion of its features across languages compared with a smaller model.
  • Planning ahead. When writing rhyming poetry, the model selected candidate rhyme words before generating the line that leads to them, evidence that next-token prediction does not imply purely one-step-at-a-time computation.

These are case studies on one model using a method that captures only part of the computation, so they are hypotheses refined by intervention experiments rather than complete explanations. They connect to the old Sapir-Whorf question in a new form: models appear to build a partly language-neutral conceptual space.

Linguistics and language models: a live debate

Can language models tell us anything about human language? Noam Chomsky and others argued that models learn "impossible" languages as easily as real ones, so they cannot illuminate human language. Julie Kallini, Christopher Potts, and colleagues tested this in Mission: Impossible Language Models, which won a best paper award at ACL 2024. They trained small GPT-2 models on English and on synthetic languages with unnatural rules (random word shuffles, rules based on counting word positions) and found the models learned the impossible languages less efficiently than English.

The result did not settle the question. Tim Hunter (UCLA) argued in Computational Linguistics (2025) that the key comparison had a confound and did not test languages built on constituency structure. Richard Futrell and Kyle Mahowald then made the broader case in a Behavioral and Brain Sciences target article, How Linguistics Learned to Stop Worrying and Love the Language Models (published online 2025; journal issue 2026), arguing that models are useful "model systems" and proofs of concept for gradient, usage-based approaches to language, without replacing linguistic theory. More than two dozen commentaries followed, from critiques titled "Studies with impossible languages falsify LMs as models of human language" and "LLMs are not children" to calls for more attention to speech and to languages other than English. The authors' April 2026 reply identified two recurring mistakes: assuming that statistical learners cannot be linguistically interesting, and assuming that today's models are the limit of what the approach can show.

Data efficiency is the other front. The BabyLM Challenge asks teams to train models on at most 100 million words, roughly a child's exposure. In the first round, winning submissions using the LTG-BERT architecture outperformed models trained on trillions of words on the challenge's evaluations, while curriculum learning (ordering data from easy to hard) was largely unsuccessful. The third challenge (2025) expanded into a workshop on cognitively plausible modeling. At ACL 2026, a best paper award went to The Imperfective Paradox in Large Language Models, a study of how models handle grammatical aspect (the puzzle that "she was crossing the street" does not entail "she crossed the street"), and an outstanding paper introduced a minimal-pair benchmark for constructional understanding. Fine-grained linguistic probing remains central to the field, not peripheral.

Data: openly licensed corpora and the data wall

Most large models are trained on web text gathered without explicit permission, which has drawn legal and ethical scrutiny. In June 2025, researchers from the University of Toronto, EleutherAI, Hugging Face, the Allen Institute for AI, and others released the Common Pile v0.1 (NeurIPS 2025 Datasets and Benchmarks), an 8-terabyte collection of public domain and openly licensed text from 30 sources, including research papers, code, books, encyclopedias, educational materials, and audio transcripts. Two 7-billion-parameter models trained on it (Comma v0.1, on 1 and 2 trillion tokens) performed competitively with Llama 1 and Llama 2 7B at similar compute. That does not match today's frontier, but it shows that openly licensed data can support capable models. At the same time, curated web corpora such as Hugging Face's FineWeb (about 15 trillion tokens) have become standard open baselines, and data supply limits are pushing labs toward synthetic and licensed data.

Machine translation and multilingual evaluation

The 2025 Conference on Machine Translation (WMT25) general task covered 30 language pairs and evaluated 60 systems: 36 submitted by participants and 24 commercial LLMs and online translation services. Its findings paper carried a pointed title: Time to Stop Evaluating on Easy Test Sets. Organizers built harder test sets through difficulty sampling and more complex source data, moved further toward document-level translation, and used professional annotators with the Error Span Annotation protocol (and MQM for two pairs). The direction mirrors the rest of NLP evaluation: as general models saturate older test sets, benchmarks must become harder, longer, and more realistic to remain informative. See Language Translation and Benchmarks.

The research community at scale

The field's growth brings its own problems. ACL 2026 (San Diego, July 2026) received a record of roughly 12,000 submissions, up about 45% from the previous year. One of its outstanding papers, HalluCitation Matters, analyzed papers from ACL, NAACL, and EMNLP in 2024 and 2025 and documented hundreds of papers containing hallucinated references, a sign that language model misuse has reached the literature about language models.

Capabilities

Natural Language Understanding (NLU)

Youtube search... | Quora search... ...Google search


Natural-language understanding (NLU) or natural-language interpretation (NLI) is a subtopic of natural-language processing in artificial intelligence that deals with machine reading comprehension. There is considerable commercial interest in the field because of its application to automated reasoning, machine translation, question answering, news-gathering, text categorization, voice-activation, archiving, and large-scale content analysis. (Note that in current research "NLI" more often stands for natural language inference, the task of deciding whether one sentence entails, contradicts, or is neutral toward another.)

In classic voice-assistant architectures, NLU is the post-processing of text, after the use of NLP algorithms (identifying parts-of-speech, etc.), that utilizes context from recognition devices; automatic speech recognition (ASR), vision recognition, last conversation, misrecognized words from ASR, personalized profiles, microphone proximity etc.), in all of its forms, to discern meaning of fragmented and run-on sentences to execute an intent from typically voice commands. NLU has an ontology around the particular product vertical that is used to figure out the probability of some intent. An NLU has a defined list of known intents that derives the message payload from designated contextual information recognition sources. The NLU will provide back multiple message outputs to separate services (software) or resources (hardware) from a single derived intent (response to voice command initiator with visual sentence (shown or spoken) and transformed voice command message to different output messages to be consumed for M2M communications and actions) Natural-language understanding | Wikipedia

NLU uses algorithms to reduce human speech into a structured ontology. Then AI algorithms detect such things as intent, timing, locations, and sentiments. ... Natural language understanding is the first step in many processes, such as categorizing text, gathering news, archiving individual pieces of text, and, on a larger scale, analyzing content. Real-world examples of NLU range from small tasks like issuing short commands based on comprehending text to some small degree, like rerouting an email to the right person based on basic syntax and a decently-sized lexicon. Much more complex endeavors might be fully comprehending news articles or shades of meaning within poetry or novels. NLP vs. NLU: from Understanding a Language to Its Processing | Sciforce

From intents to instructions. The intent-and-slot design above (used by platforms such as Alexa, Dialogflow, and Rasa) required designers to enumerate every intent in advance and label training examples for each. Large language models changed this in two ways. First, zero-shot classification lets a model map an utterance onto a list of intents described in plain language, so new intents can be added without retraining. Second, function calling or tool use lets the model fill structured arguments for an API directly from a request, replacing hand-built slot fillers with a schema the model reads at run time. Many production systems now combine both: a small, fast classifier routes common requests, and a large model handles the long tail.

Measuring understanding. The Stanford Natural Language Inference (SNLI) corpus (2015) introduced about 570,000 human-labeled sentence pairs for entailment; MultiNLI extended the idea across genres. Models soon scored highly, but follow-up studies showed many relied on shortcuts, for example treating the word "not" as a strong signal of contradiction. That experience, repeated across many datasets, is why NLU evaluation has moved toward adversarial, multi-step, and real-world tasks. See Benchmarks.

1*Uf_qQ0zF8G8y9zUhndA08w.png

Image Source: Understanding Natural Language Understanding | Bill MacCartney

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

NLU Design Landscape | Cobus Greyling - Medium

Phonology (Phonetics)

Youtube search... ...Google search

Phonology is a branch of linguistics concerned with the systematic organization of sounds in spoken languages and signs in sign languages. It used to be only the study of the systems of phonemes in spoken languages (and therefore used to be also called phonemics, or phonematics), but it may also cover any linguistic analysis either at a level beneath the word (including syllable, onset and rime, articulatory gestures, articulatory features, mora, etc.) or at all levels of language where sound or signs are structured to convey linguistic meaning.

Phonetics and phonology are related but distinct. Phonetics studies the physical production and acoustics of speech sounds; phonology studies how a particular language organizes those sounds into a system of contrasts.

A Phoneme is the most basic sound unit of sound; any of the perceptually distinct units of sound in a specified language that distinguish one word from another, for example p, b, d, and t in the English words pad, pat, bad, and bat. Phoneme | Wikipedia

A Grapheme is the smallest unit of a writing system of any given language. An individual grapheme may or may not carry meaning by itself, and may or may not correspond to a single phoneme of the spoken language

Why phonology matters for NLP. Speech systems must bridge sounds and spellings. Grapheme-to-phoneme (G2P) conversion turns written words into pronunciations for text-to-speech; speech recognition does the reverse. English makes this hard because spelling and sound diverge ("though", "through", "tough"). Older systems chained an acoustic model, a pronunciation dictionary, and a language model. Modern end-to-end systems, such as OpenAI's Whisper (trained on about 680,000 hours of weakly supervised audio), learn the mapping directly, and the newest assistants process speech natively without converting it to text first. See End-to-End Speech and Synthesize Speech. Phonetic matching algorithms such as Soundex apply the same ideas to spelling variation in names.

Load video
YouTube
Load video
YouTube


Lexical (Morphology)

Lexical Ambiguity: Words have multiple meanings

The study of words, how they are formed, and their relationship to other words in the same language. It analyzes the structure of words and parts of words, such as stems, root words, prefixes, and suffixes. Morphology also looks at parts of speech, intonation and stress, and the ways context can change a word's pronunciation and meaning. About the words that make up the sentence, how they are formed, and how do they change depending on their context. Some examples of these include:

  • Prefixes/suffixes
  • Singularization/pluralization
  • Gender detection
  • Word inflection (modification of word to express different grammatical categories such tenses, case, voice etc..). Other forms of inflection includes conjugation (inflection of verbs) and declension (inflection of nouns, adjectives, adverbs etc…).
  • Lemmatization (the base form of the word, or the reverse of inflection)
  • Spell checking

Morphological typology. Languages differ enormously in how much work happens inside words. Isolating languages such as Mandarin use mostly invariant words; fusional languages such as Spanish pack several grammatical meanings into one ending; agglutinative languages such as Turkish and Finnish string many separate morphemes together, so a single Turkish word can translate into a whole English sentence. This matters for modern NLP because subword tokenizers trained mostly on English often split agglutinative words at points that ignore morpheme boundaries, which inflates token counts and can hurt quality. Tokenizer-free models with learned segmentation (see The end of the tokenizer?) are partly motivated by this problem.

Text Preprocessing

Cleaning and preparation the information for use, such as punctuation removal, spelling correction, lowercasing, stripping markup tags (HTML,XML)

What counts as good preprocessing depends on the model. For counting-based methods (Bag-of-Words (BoW), Term Frequency–Inverse Document Frequency (TF-IDF), Latent Dirichlet Allocation (LDA)), aggressive normalization (lowercasing, punctuation removal, stop word removal, stemming) reduces noise and vocabulary size. For transformer models, the opposite is usually true: they were pretrained on raw text, so removing case, punctuation, or function words throws away signal they know how to use.

Large-scale pretraining corpora use a different kind of preprocessing:

  • Unicode normalization (for example NFC or NFKC) so visually identical characters share one encoding.
  • Boilerplate removal to strip navigation menus, ads, and footers from web pages (see jusText under Corpora).
  • Language identification to route text to the right pipeline.
  • Deduplication with techniques such as MinHash, since repeated text wastes compute and encourages memorization.
  • Quality filtering with heuristics or trained classifiers that score how educational or well-formed a document is.
  • Privacy filtering to remove personal information (see Identity Scrubbing).

Load video
YouTube

Regular Expressions (Regex)

Youtube search... ...Google search

Search for text patterns, validate emails and URLs, capture information, and use patterns to save development time.

Regular expressions describe patterns over characters. They are still the fastest way to handle well-structured text such as dates, phone numbers, product codes, and log lines, and they remain central to tokenizers themselves: GPT-style tokenizers use a regex to pre-split text into chunks (words, numbers, punctuation) before applying byte pair encoding. Formally, regular expressions recognize exactly the regular languages, which is why they cannot reliably match nested structures such as balanced parentheses or full HTML. A good rule of thumb: use regex for shallow, rigid patterns; use a parser or a learned model when structure nests or meaning depends on context.

regex-example.png

Load video
YouTube

Soundex

Youtube search... ...Google search

a phonetic algorithm for indexing names by sound, as pronounced in English. The goal is for homophones to be encoded to the same representation so that they can be matched despite minor differences in spelling. The Soundex code for a name consists of a Letter alphabet followed by three numerical digits: the letter is the first letter of the name, and the digits encode the remaining consonants. Consonants at a similar place of articulation share the same digit so, for example, the labial consonants B, F, P, and V are each encoded as the number 1. Wikipedia

The correct value can be found as follows:

  1. Retain the first letter of the name and drop all other occurrences of a, e, i, o, u, y, h, w.
  2. Replace consonants with digits as follows (after the first letter):
    • b, f, p, v → 1
    • c, g, j, k, q, s, x, z → 2
    • d, t → 3
    • l → 4
    • m, n → 5
    • r → 6
  3. If two or more letters with the same number are adjacent in the original name (before step 1), only retain the first letter; also two letters with the same number separated by 'h' or 'w' are coded as a single number, whereas such letters separated by a vowel are coded twice. This rule also applies to the first letter.
  4. If you have too few letters in your word that you can't assign three numbers, append with zeros until there are three numbers. If you have more than 3 letters, just retain the first 3 numbers.

For example, "Robert" and "Rupert" both encode to R163, so a search for one finds the other. Soundex was patented in 1918 and became famous through its use in indexing United States census records. Its weaknesses (English-specific rules, sensitivity to the first letter, many false matches) led to successors such as NYSIIS, Metaphone (1990), and Double Metaphone (2000), which model English spelling rules more closely and handle names from other languages better. Modern record-linkage systems often combine phonetic codes with edit distance and learned similarity models.

Load video
YouTube


Tokenization / Sentence Splitting

Youtube search... ...Google search

Tokenization is the process of demarcating (breaking text into individual words or word pieces) and possibly classifying sections of a string of input characters. The resulting tokens are then passed on to some other form of processing. The process can be considered a sub-task of parsing input. A related but distinct idea is the n-gram: a contiguous sequence of n items from a given sample of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application. Tokens are the units; n-grams are runs of consecutive units (a bigram is two tokens in a row, a trigram three).

Sentence splitting looks trivial but is not. A period can end a sentence or belong to an abbreviation ("Dr.", "U.S."), a decimal number, or an ellipsis. Rule-based splitters use abbreviation lists; statistical splitters such as NLTK's Punkt learn abbreviations from unlabeled text; and many modern pipelines use a trained model.

Word-level tokenization splits on whitespace and punctuation, with language-specific rules for contractions ("don't" becomes "do" + "n't" in the Penn Treebank convention). Languages written without spaces, such as Chinese, Japanese, and Thai, need dedicated word segmenters.

Subword tokenization is what modern language models use:

  • Byte pair encoding (BPE): start with characters or bytes and repeatedly merge the most frequent adjacent pair. Introduced to neural machine translation by Sennrich, Haddow, and Birch (2016), a paper that won ACL's Test-of-Time award in 2026. GPT-2 and later GPT models apply BPE to raw bytes, so any input can be represented.
  • WordPiece: used by BERT; similar to BPE but chooses merges that most increase the likelihood of the training data.
  • Unigram language model (SentencePiece): starts with a large vocabulary and prunes it, and treats text as a raw stream (spaces included), which suits languages without whitespace.

Tokenization explains many famous model quirks. A model that sees "strawberry" as two or three opaque tokens has no direct view of its letters, which is why counting letters was historically hard. Numbers may be split inconsistently ("1234" as "12" + "34" or "123" + "4"), which complicates arithmetic. Unusual tokens that appeared in tokenizer training but rarely in model training (the "SolidGoldMagikarp" phenomenon) once triggered erratic outputs. And because tokenizers compress some languages far better than others, the same text can cost many times more tokens in under-served languages (see Tokenization as compression).

In-this-time-series-Google-Ngram-Viewer-is-used-to-compare-some-literature-for-children.png

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Normalization

Youtube search... ...Google search

Process that converts a list of words to a more uniform sequence. Normalization creates equivalence classes of terms so that different surface forms match: "U.S.A." and "USA", "color" and "colour", "Automobile" and "automobile". The techniques below (stemming, lemmatization, case folding) are the main tools. Normalization improves recall in search and reduces sparsity for counting-based models, but it can also merge words that should stay distinct, so the right amount depends on the application.

Load video
YouTube
Load video
YouTube

Stemming (Morphological Similarity)

Youtube search... ...Google search

Stemmers remove morphological affixes from words, leaving only the word stem. Refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes.

The most widely used algorithm is the Porter stemmer (Martin Porter, 1980), which applies ordered suffix-stripping rules: "connection", "connected", and "connecting" all reduce to "connect". Its successor, the Snowball framework, provides stemmers for many languages. Stemmers are fast and need no dictionary, but they make two kinds of error: over-stemming merges unrelated words ("university" and "universe" both become "univers"), and under-stemming misses related ones. Stemming remains standard in keyword search engines (for example in BM25-based retrieval) but is rarely useful as input to neural models.

figure3.png

Load video
YouTube
Load video
YouTube

Lemmatization

Youtube search... ...Google search

Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma . If confronted with the token saw, stemming might return just s, whereas lemmatization would attempt to return either see or saw depending on whether the use of the token was as a verb or a noun. The two may also differ in that stemming most commonly collapses derivationally related words, whereas lemmatization commonly only collapses the different inflectional forms of a lemma. Stemming and lemmatization | Stanford.edu NLTK's lemmatizer knows "am" and "are" are related to "be."

Because the right lemma depends on part of speech, good lemmatizers run a POS tagger first. Tools such as spaCy and Stanza combine lookup tables with trained models, and for morphologically rich languages lemmatization is a substantial machine learning task in its own right.

Load video
YouTube
Load video
YouTube

Capitalization / Case Folding

Youtube search... ...Google search

A common strategy is to do case-folding by reducing all letters to lower case. Often this is a good idea: it will allow instances of Automobile at the beginning of a sentence to match with a query of automobile. It will also help on a web search engine when most of your users type in ferrari when they are interested in a Ferrari car. On the other hand, such case folding can equate words that might better be kept apart. Many proper nouns are derived from common nouns and so are distinguished only by case, including companies (General Motors, The Associated Press), government organizations (the Fed vs. fed) and person names (Bush, Black). We already mentioned an example of unintended query expansion with acronyms, which involved not only acronym normalization (C.A.T. → CAT) but also case-folding (CAT → cat). Capitalization/case-folding | Stanford

Early BERT releases came in both "uncased" and "cased" versions; cased models perform better on tasks where capitalization carries meaning, such as named entity recognition. Nearly all current large language models are cased. Case folding is mostly applied at the search and indexing layer, not at model input.

Stop Words

Youtube search... ...Google search

One of the major forms of pre-processing is to filter out useless data. In natural language processing, useless words (data), are referred to as stop words. A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore, both when indexing entries for searching and when retrieving them as the result of a search query.

Stop word removal follows directly from Zipf's law: a few very frequent words make up a large share of text but say little about topic. It helps Bag-of-Words (BoW) models, TF-IDF, and topic models. It is usually harmful for anything that depends on meaning or syntax. "Flights to Boston" and "flights from Boston" differ only in stop words, and removing "not" flips sentiment. Modern search engines often keep stop words and let ranking functions such as BM25 down-weight them automatically. (This section previously sat under Syntax; it moved here because stop word filtering is a lexical preprocessing step.)

Stop-word-removal-using-NLTK.png

Load video
YouTube
Load video
YouTube

Similarity

Youtube search... ...Google search

Similarity can be measured at several levels: characters (edit distance, useful for spelling correction and fuzzy matching), words (dictionary relations or embeddings), and whole texts (see Sentence/Document Similarity).

Word Similarity

Youtube search... ...Google search

There are two main families of word similarity. Knowledge-based measures use a hand-built lexical database such as WordNet, developed at Princeton, which groups words into sets of synonyms (synsets) linked by relations such as "is a kind of" (hypernymy). Measures such as path similarity and Wu-Palmer similarity score two words by how close they sit in that hierarchy. Distributional measures use word embeddings and compute the cosine similarity between vectors. Distributional similarity tends to capture relatedness ("coffee" and "cup") as well as true similarity ("coffee" and "tea"), which is useful for search but can mislead when you specifically need synonyms.

Text Clustering

Youtube search... ...Google search

Text clustering groups documents by similarity without predefined labels. The classic recipe represents documents as TF-IDF vectors and applies k-means or hierarchical clustering. The current recipe encodes each document with a sentence-embedding model, optionally reduces dimensionality (for example with UMAP), and clusters with a density-based method such as HDBSCAN, which does not require choosing the number of clusters in advance. A language model can then name each cluster by reading a few representative documents. See Clustering and Topic Modeling.

header_short.jpg

Load video
YouTube
Load video
YouTube


Syntax (Parsing)


Syntactic Ambiguity: Sentence is having multiple parse trees.

The set of rules, principles, and processes that govern the structure of sentences (sentence structure) in a given language, usually including word order. The term syntax is also used to refer to the study of such principles and processes. Focus on the relationship of the words within a sentence: how a sentence is constructed. In a way, syntax is what we usually refer to as grammar. To derive this understanding, syntactical analysis is usually done at a sentence-level, where as for morphology the analysis is done at word level. When we’re building dependency trees or processing parts-of-speech, we’re basically analyzing the syntax of the sentence.

There are two main ways to represent sentence structure:

  • Constituency (phrase-structure) parsing groups words into nested phrases (noun phrases, verb phrases) according to a grammar. Classic algorithms include the CKY dynamic-programming parser over probabilistic context-free grammars, trained on treebanks such as the Penn Treebank. See Constituency Tree.
  • Dependency parsing links each word to the word it modifies with a labeled arc ("subject of", "object of"). It suits languages with free word order and is the basis of Universal Dependencies (UD), a cross-linguistically consistent annotation scheme with treebanks for well over 100 languages. Modern dependency parsers are either transition-based (building the tree with shift and reduce actions) or graph-based (scoring every possible arc, as in the biaffine parser of Dozat and Manning, 2017).

The classic example of syntactic ambiguity is "I saw the man with the telescope": the telescope can belong to the seeing or to the man. Do large language models represent syntax even though nobody gives them trees? Probing studies suggest yes, to a significant degree. Hewitt and Manning (2019) showed that parse-tree distances can be recovered from BERT and ELMo representations with a simple linear transformation, and later work found syntactic structure spread across layers of larger models. Explicit parsers remain valuable where you need a guaranteed, inspectable structure, such as information extraction, grammar checking, and linguistic research.

1*Bi_s86b68I5kDEC2kmU39A.png

Relating Text

Understanding how the words relate to each other and the underlying grammar by segmenting the sentences syntax

Part-of-Speech (POS) Tagging

Youtube search... ...Google search

Part-of-speech tagging (POST), also called grammatical tagging or word-category disambiguation, is the process of marking up a word in a text (corpus) as corresponding to a particular part of speech, based on both its definition and its context, i.e., its relationship with adjacent and related words in a phrase, sentence, or paragraph. A simplified form of this is commonly taught to school-age children, in the identification of words as nouns, verbs, adjectives, adverbs, etc.

Tag sets vary. The Penn Treebank uses about 45 fine-grained English tags (distinguishing, for example, singular nouns NN from plural nouns NNS and past-tense verbs VBD from past participles VBN), while Universal Dependencies defines 17 coarse universal tags shared across languages. English POS tagging accuracy reached about 97% on standard newswire benchmarks years ago, and the remaining errors concentrate on genuinely ambiguous words ("that", "around") and unusual text such as social media. The history of taggers mirrors the history of the field: rule-based systems, then hidden Markov models decoded with the Viterbi algorithm, then maximum entropy and conditional random field models, then bidirectional LSTMs, and now transformer encoders.

Taggers:

lsmcqqk.jpg

Load video
YouTube
Load video
YouTube

Constituency Tree

Youtube search... ...Google search

a one-to-one-or-more relation; every word in the sentence corresponds to one or more nodes in the tree diagram; employ the convention where the category acronyms (e.g. N, NP, V, VP) are used as the labels on the nodes in the tree. The one-to-one-or-more constituency relation is capable of increasing the amount of sentence structure to the upper limits of what is possible.

In a constituency tree, the sentence "The cat sat on the mat" becomes an S node with two children: a noun phrase (NP: "The cat") and a verb phrase (VP: "sat on the mat"), which in turn contains a prepositional phrase (PP: "on the mat"). Probabilistic context-free grammars attach a probability to each rewrite rule and choose the most likely tree with dynamic programming. Neural constituency parsers now score spans directly with transformer encoders. The influence of this idea extends beyond parsing: John Hale's 2001 paper using a probabilistic parser as a model of human sentence processing received ACL's 2026 Test-of-Time award, a reminder that parsing has long connected NLP to cognitive science.

E-ICA-01.jpg

Load video
YouTube
Load video
YouTube

Chunking

Youtube search... ...Google search

In natural language processing, chunking (also called shallow parsing) groups the tokens of a sentence into non-overlapping, non-nested phrases such as noun phrases and verb phrases, without building a full parse tree. In "[The quick brown fox] [jumped] [over] [the lazy dog]", the bracketed spans are chunks. Chunking is cheaper and more robust than full parsing, which made it a standard step for information extraction.

Chunkers are typically built in one of two ways:

  • Rule-based chunking with patterns over part-of-speech tags. NLTK's RegexpParser, for example, can define a noun phrase as an optional determiner, any number of adjectives, and a noun: NP: {<DT>?<JJ>*<NN>}.
  • Sequence labeling with IOB tags, where each token is marked as Beginning a chunk, Inside a chunk, or Outside any chunk. The CoNLL-2000 shared task established this framing, and the same IOB scheme is used for named entity recognition.

Note on terminology: an earlier version of this section described the "Hierarchy of Ideas" from Neuro-Linguistic Programming, a self-help discipline that shares the acronym NLP but is unrelated to natural language processing. That material is kept here for reference only: "The Hierarchy of Ideas (also known as chunking) is a linguistic tool used in NLP that allows the speaker to traverse the realms of abstract to specific," and it concerns how people group ideas, not how software groups tokens. Hierarchy of Ideas or Chunking in NLP | Excellence Assured. "Chunking" also has a third meaning in modern RAG systems: splitting long documents into passages before embedding them.

image-result-for-chunking-examples-in-marketing.gif

Load video
YouTube
Load video
YouTube

Chinking

Youtube search... ...Google search

The process of removing a sequence of tokens from a chunk. If the matching sequence of tokens spans an entire chunk, then the whole chunk is removed; if the sequence of tokens appears in the middle of the chunk, these tokens are removed, leaving two chunks where there was only one before. If the sequence is at the periphery of the chunk, these tokens are removed, and a smaller chunk remains.

Chinking is the complement of chunking: it is often easier to say what should not be in a chunk than what should. In NLTK's grammar notation, chink rules are written with reversed braces, so a grammar can first chunk everything and then carve out verbs and prepositions.

chinking.png

Load video
YouTube


Semantics


  • Semantic Ambiguity: Sentence having multiple meanings

The linguistic and philosophical study of meaning in language, programming languages, formal logics, and semiotics. It is concerned with the relationship between signifiers (like words, phrases, signs, and symbols) and what they stand for in reality, their denotation. The “meaning” of the sentence. It’s difficult to (i) link raw text to a knowledge base of mathematical facts in our system and (ii) combine pieces of knowledge together to infer an answer. Meaning in Natural Language is a multi-facetted concept with:

  • semantic,
  • pragmatic,
  • cognitive and
  • social aspects.

Computational semantics works at several levels:

  • Lexical semantics: what individual words mean, including word sense disambiguation (choosing between the financial and river senses of "bank").
  • Compositional semantics: how word meanings combine into sentence meanings, classically represented as logical forms.
  • Semantic role labeling: who did what to whom, when, and where, using resources such as PropBank and FrameNet.
  • Meaning representations: graph formats such as Abstract Meaning Representation (AMR) that abstract away from surface wording.

Large language models mostly encode meaning implicitly in vectors rather than explicit structures. Whether that counts as "understanding" is a long-running debate, discussed under Linguistics and language models: a live debate.

Word Embeddings

Youtube search... ...Google search

The collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers.

Embeddings have gone through three generations:

  1. Static word embeddings (Word2Vec in 2013, GloVe in 2014, fastText in 2016) give each word a single vector regardless of context. fastText added character n-grams so it can build vectors for unseen words.
  2. Contextual embeddings (ELMo, BERT) give each occurrence of a word a different vector depending on its sentence, which resolves much lexical ambiguity.
  3. Sentence and document embeddings (Sentence-BERT in 2019 and many successors) are trained with contrastive objectives so that texts with similar meaning land close together. These power Semantic Search and RAG. Current models often support Matryoshka representations, whose leading dimensions can be truncated to save storage with modest accuracy loss.

Embedding models are compared on the MTEB and MMTEB benchmarks (see Embeddings and retrieval). Rankings change often, and a model's leaderboard average can hide large differences across languages and domains, so it pays to test candidates on your own data.

Example-of-embedding-of-time-series-xi-from-the-temporal-space-left-into-the-pairwise.png

Load video
YouTube
Load video
YouTube

Named Entity Recognition (NER)

Youtube search... ...Google search

Named Entity Recognition (also known as entity identification, entity chunking, and entity extraction) is a subtask of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. Most research on NER systems has been structured as taking an unannotated block of text, and producing an annotated block of text that highlights the names of entities. NER is often framed as a sequence-labeling task like Part-of-Speech (POS) Tagging, using the same IOB scheme as chunking, but it is a distinct task: POS tagging labels every word's grammatical category, while NER finds specific spans that name things.

Benchmarks shaped the field. CoNLL-2003 defined four entity types (person, organization, location, miscellaneous) for English and German news, while OntoNotes 5.0 expanded to 18 types including dates, quantities, and works of art. Systems progressed from gazetteers and hand-written rules, to feature-rich CRFs, to BiLSTM-CRF networks, to fine-tuned transformers. The newest shift is open or zero-shot NER, where users name the entity types at run time. Large language models can do this from a prompt, and compact encoders such as GLiNER match entity-type descriptions to spans and, in their authors' evaluations, outperform ChatGPT in zero-shot settings while extracting all entities in parallel. NER is usually followed by entity linking, which connects each mention to a unique entry in a knowledge base (see Wikifier).

maxthonsnap20170219114627.png

Load video
YouTube

Identity Scrubbing

Youtube search... ...Google search

Patient ID: P89474 

Mary Phillips is a 45-year-old woman with a history of diabetes.
She arrived at New Hope Medical Center on August 5 complaining
of abdominal pain. Dr. Gertrude Philippoussis diagnosed her
with appendicitis and admitted her at 10 PM.

into this:

Patient ID: [ID] 

[NAME] is a [AGE]-year-old woman with a history of diabetes.
She arrived at [HOSPITAL] on [DATE] complaining
of abdominal pain. Dr. [PHYSICIAN] diagnosed her
with appendicitis and admitted her at 10 PM.

or this:

Patient ID: ID586 

Sandy Parkinson is a 34-year-old woman with a history of diabetes.
She arrived at Mercy Hospital on July 10 complaining
of abdominal pain. Dr. Myron Prendergast diagnosed her
with appendicitis and admitted her at 10 PM.

The second output is redaction (replacing identifiers with type labels); the third is surrogate substitution (replacing them with realistic fake values), which keeps text readable and makes any missed real identifier harder to spot among the fakes. De-identification is applied NER: it depends on finding names, dates, locations, and ID numbers reliably. In the United States, the HIPAA Safe Harbor method lists 18 categories of identifiers that must be removed from health records. Current tools include rule-and-model hybrids such as Microsoft Presidio, fine-tuned transformer taggers, and LLM-based de-identification. Two cautions apply: recall matters more than precision (one missed name can expose a patient), and removing direct identifiers does not guarantee anonymity, since combinations of remaining details can sometimes re-identify a person. This also matters for training data: personal information in pretraining corpora can be memorized and later reproduced by models. (This section previously sat under Syntax; it moved here because de-identification is an application of named entity recognition.)

Semantic Slot Filling

Youtube search... ...Google search

One way of making sense of a piece of text is to tag the words or tokens which carry meaning to the sentences. There are three main approaches to solve this problem:

  1. Rule Based; regular expressions, context-free grammars
  2. Probabilistic modeling & Machine Learning; likelihood maximization, linear classifiers
  3. Deep Learning; recurrent neural networks, convolutional neural networks

A fourth approach now dominates new systems:

  1. Instruction-following language models; a schema (for example, a JSON description of an API's parameters) is given to a large language model, which fills the slots directly through function calling or constrained structured output.

Slot filling is usually paired with intent detection. In "Book a flight from Boston to Denver next Friday", the intent is book_flight and the slots are origin = Boston, destination = Denver, date = next Friday. Classic benchmarks include ATIS (airline travel queries) and SNIPS (voice-assistant commands), and joint models such as JointBERT predict the intent and the slots together. With LLM-based slot filling, validation becomes the hard part: checking that extracted values exist, are well-formed, and are actually supported by the user's words.

1*bpkI_VpU1j0wXv4VqW5e5Q.png

Relation Extraction

Youtube search... ...Google search

task of extracting semantic relationships from a text. Extracted relationships usually occur between two or more entities of a certain type (e.g. Person, Organisation, Location) and fall into a number of semantic categories (e.g. married to, employed by, lives in). Relationship Extraction

Approaches have evolved through hand-written patterns ("X, the CEO of Y"), supervised classifiers trained on datasets such as TACRED, distant supervision (automatically labeling sentences that mention two entities already related in a knowledge base, a cheap but noisy source of training data), and transformer models that extract entities and relations jointly. Large language models can now extract relations from a schema described in the prompt, which makes it practical to build domain-specific knowledge graphs quickly. The main risks are hallucinated relations and inconsistent labels across documents, so production pipelines usually require each extracted relation to be grounded in a quoted source span.

TypicalRelationExtractionPipeline.png

Relation Extraction | Jun Tian

Load video
YouTube


Discourse (Dialog)

Discourse is the creation and organization of the segments of a language above as well as below the sentence. It is segments of language which may be bigger or smaller than a single sentence but the adduced meaning is always beyond the sentence.

  • identifying the discourse structure of a connected text, i.e. the nature of the discourse relationships between sentences (e.g. elaboration, explanation, contrast).
  • recognizing and classifying the speech acts in a chunk of text (e.g. yes-no question, content question, statement, assertion, etc.).

The objects of discourse analysis (discourse, writing, conversation, communicative event) are variously defined in terms of coherent sequences of sentences, propositions, speech, or turns-at-talk.

The problems addressed in discourse research aim to answer two general kinds of questions:

  1. what information is contained in extended sequences of utterances that goes beyond the meaning of the individual utterances themselves?
  2. how does the context in which an utterance is used affect the meaning of the individual utterances, or parts of them?

Chapter 6 Discourse and Dialogue

Computational frameworks. Rhetorical Structure Theory (RST, Mann and Thompson, 1988) represents a text as a tree of relations such as elaboration, contrast, and cause between spans. The Penn Discourse Treebank (PDTB) annotates relations signaled, or merely implied, by connectives such as "because" and "however". Discourse parsing supports summarization, argument mining, and essay scoring (see Automated Scoring).

Dialogue systems. Task-oriented dialogue systems traditionally tracked a dialogue state (the slots filled so far) across turns, evaluated on datasets such as MultiWOZ. Open-domain chatbots aimed for engaging conversation without a fixed goal. Large language models have largely merged the two: a single model maintains conversational context, answers questions, and calls tools. Discourse problems have not gone away, though. Models still lose track of commitments over long conversations, contradict earlier statements, and struggle with long-context coherence (see Long context and its limits). Spoken dialogue adds turn-taking, interruptions, and backchannels ("mm-hm"), which newer speech-native assistants are beginning to handle.

Compared to traditional linguistics Contrary to much of traditional linguistics, discourse analysts not only study language use 'beyond the sentence boundary' but also prefer to analyze 'naturally occurring' language use, not invented examples. Text linguistics is a closely related field. The essential difference between discourse analysis and text linguistics is that discourse analysis aims at revealing socio-psychological characteristics of a person/persons rather than text structure.


Pragmatics

Anaphoric Ambiguity: Phrase or word which is previously mentioned but has a different meaning.

Studies the ways in which context contributes to meaning. Pragmatics encompasses speech act theory, conversational implicature, talk in interaction and other approaches to language behavior in philosophy, sociology, linguistics and anthropology: understanding the text as a whole. Popular problems that we’re trying to solve at this stage are:

  • Topic modelling
  • Coreference/Anaphora
  • Summarization
  • Question & Answering

Core pragmatic phenomena include:

  • Implicature: meaning implied rather than stated. Philosopher Paul Grice proposed that listeners assume speakers are cooperative (informative, truthful, relevant, clear), so "Some of the students passed" usually implies that not all did.
  • Presupposition: background assumptions a sentence takes for granted ("She stopped smoking" presupposes she used to smoke).
  • Speech acts: utterances that do things, such as promising, requesting, or apologizing. "Can you pass the salt?" is a request, not a question about ability.
  • Deixis: words whose meaning depends on the situation ("here", "now", "you").

How well large language models handle pragmatics is an open research question with mixed evidence. Models often interpret indirect requests and common implicatures correctly, but evaluations find weaker performance on less conventional implicatures, sarcasm, and reasoning about what a particular speaker knows. Related work on "theory of mind" in language models is contested: some studies report strong performance on false-belief tasks, while others show that small changes to the task wording can cause failures, suggesting reliance on familiar patterns. ACL 2026 recognized work on steering models' theory-of-mind behavior among its best theme papers, a sign of how active the area is.

Grouping

Sentence/Document Similarity

Youtube search... ...Google search

Word embeddings have become widespread in Natural Language Processing. They allow us to easily compute the semantic similarity between two words, or to find the words most similar to a target word. However, often we're more interested in the similarity between two sentences or short texts. Comparing Sentence Similarity Methods | Yves Peirsman - NLPtown

Modern systems use two complementary designs:

  • Bi-encoders embed each text separately, so millions of documents can be pre-encoded and compared by cosine similarity in a vector index. This is fast and scales to large collections.
  • Cross-encoders read both texts together and output a relevance score. They are more accurate but too slow to run against every document, so they are typically used to re-rank the top results from a bi-encoder.

Lexical methods are not obsolete. BM25, a refinement of TF-IDF ranking, remains a strong baseline, especially for rare terms, product codes, and names that embeddings handle poorly. Many production search systems use hybrid retrieval that combines BM25 and embedding scores.

image-1.png

Load video
YouTube

Text Classification

Youtube search... ...Google search

Tasks:

  • predict tags or categories
  • predict sentiment
  • filter spam eMails

Text Classification approaches:

  • Flat - there is no inherent hierarchy between the possible categories the data can belong to (or we chose to ignore it). Train either a single classifier to predict all of the available classes or one classifier per category (1 vs All)
  • Hierarchically - organizing the classes, creating a tree or DAG (Directed Acyclic Graph) of categories, exploiting the information on relationships among them. Although there are different types of hierarchical classification approaches, the difference between both modes of reasoning and analysing are particularly easy to understand in these illustrations, taken from a great review on the subject by Silla and Freitas (2011). Taking a top-down approach, training a classifier per level (or node) of the tree (again, although this is not the only hierarchical approach, it is definitely the most widely used and the one we’ve selected for our problem at hands), where a given decision will lead us down a different classification path.

Choosing a method in 2026. The options form a cost and accuracy ladder:

  • Linear models on TF-IDF features (logistic regression, linear SVMs, naive Bayes) train in seconds, are easy to explain, and remain hard to beat when you have thousands of labeled examples and short, topical texts.
  • Fine-tuned encoders (BERT, DeBERTa, ModernBERT) usually give the best accuracy per unit of cost when you have hundreds to thousands of labels.
  • Few-shot methods such as SetFit fine-tune a sentence-embedding model with contrastive learning on as few as 8 examples per class.
  • Zero-shot or few-shot LLM prompting needs no training at all and handles nuanced or fast-changing label sets, at much higher cost per document. A common pattern is to use an LLM to label data, then train a small classifier on those labels for production.

Whatever the method, evaluate with metrics suited to imbalanced classes (macro-averaged F1, per-class precision and recall) rather than raw accuracy, and check for label noise, which often limits performance more than the model does.

1.png 2.png

Load video
YouTube

Topic Modeling

Youtube search... ...Google search

A type of statistical modeling for discovering the abstract “topics” that occur in a collection of documents. Latent Dirichlet Allocation (LDA) is an example of topic model and is used to classify text in a document to a particular topic

LDA (Blei, Ng, and Jordan, 2003) treats each document as a mixture of topics and each topic as a probability distribution over words, then infers both from word co-occurrence. It is interpretable and still widely used in the social sciences, but it ignores word order, needs careful preprocessing, and struggles with short texts such as tweets. Embedding-based topic models such as BERTopic (Grootendorst, 2022) encode documents with sentence embeddings, cluster them, and describe each cluster with a class-based variant of TF-IDF. LLM-based approaches such as TopicGPT (2024) prompt a language model to propose and assign readable topic labels. Topic model output should always be validated by people who know the documents, because coherent-looking word lists can still group documents in misleading ways. See Social Science.

40064_2016_3252_Fig5_HTML.gif

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube


Neural Coreference

Youtube search... ...Google search

Coreference is the fact that two or more expressions in a text (like pronouns or nouns) link to the same person or thing. It is a classical Natural language processing task that saw a strong revival of interest around 2016 to 2018, as several research groups applied deep-learning and reinforcement-learning techniques to it. It is also one of the key building blocks to building conversational Artificial intelligence.

In "Maria told her sister that she had won", resolving "she" requires knowing which reading the writer intended, and often world knowledge. The Winograd Schema Challenge built entire test items around such cases ("The trophy didn't fit in the suitcase because it was too big": what was too big?). The landmark neural approach, end-to-end span-based coreference (Lee and colleagues, 2017), scored every possible text span as a mention and linked mentions directly, without a separate parser; transformer encoders such as SpanBERT later raised accuracy substantially on the OntoNotes (CoNLL-2012) benchmark. Large language models resolve most everyday pronouns implicitly, which is part of why they read so fluently, but explicit coreference systems remain useful for information extraction, summarization, and analysis of long documents, where consistent entity tracking must be verified.

Load video
YouTube


Whole Word Masking

Youtube search... ...Google search

Background: BERT's masking recipe. Training the language model in Bidirectional Encoder Representations from Transformers (BERT) is done by predicting 15% of the tokens in the input, that were randomly picked. These tokens are pre-processed as follows: 80% are replaced with a “[MASK]” token, 10% with a random word, and 10% use the original word. The intuition that led the authors to pick this approach is as follows (Thanks to Jacob Devlin from Google for the insight):

  • If we used [MASK] 100% of the time the model wouldn’t necessarily produce good token representations for non-masked words. The non-masked tokens were still used for context, but the model was optimized for predicting masked words.
  • If we used [MASK] 90% of the time and random words 10% of the time, this would teach the model that the observed word is never correct.
  • If we used [MASK] 90% of the time and kept the same word 10% of the time, then the model could just trivially copy the non-contextual embedding.

What whole word masking changes. The recipe above selects individual WordPiece tokens. Because BERT splits rare words into pieces (for example, "philammon" becomes "phil", "##am", "##mon"), the original method often masked only part of a word, letting the model guess the hidden piece from the visible pieces of the same word, which is too easy. Whole word masking, released by Google for BERT in May 2019, masks all pieces of a word together whenever any piece is chosen, while keeping the overall masking rate at about 15%. It produced modest but consistent gains on benchmarks such as SQuAD and MultiNLI and was especially helpful for Chinese, where Chinese BERT-wwm masked whole words rather than single characters. Later work went further: SpanBERT masks contiguous spans, and some models mask whole named entities or phrases, all aimed at forcing the model to use wider context.

Load video
YouTube

Managed Vocabularies

Managed vocabularies are curated, controlled sets of terms (corpora, ontologies, taxonomies, thesauri) that give NLP systems a stable reference for meaning. They matter most in regulated and specialized domains such as medicine (for example, coding clinical text to standard terminologies), law, and finance, where an answer must map onto an agreed-upon concept rather than any plausible wording.

Corpora

Youtube search... ...Google search

A corpus (plural corpora) or text corpus is a large and structured set of texts (nowadays usually electronically stored and processed). In corpus linguistics, they are used to do statistical analysis and hypothesis testing, checking occurrences or validating linguistic rules within a specific language territory.

Corpora have grown by about seven orders of magnitude:

  • The Brown Corpus (1961): about 1 million words of American English in 15 genres, the first major computer-readable corpus.
  • The British National Corpus (1990s): about 100 million words of written and spoken British English.
  • The Penn Treebank (1993): newswire text annotated with part-of-speech tags and parse trees, the training ground for statistical parsing.
  • Common Crawl (from 2008): a public archive of web crawls, the raw material for most modern pretraining sets.
  • The Pile (EleutherAI, 2020): an 825 GiB mix of 22 diverse sources, an early open pretraining corpus.
  • Dolma (Allen Institute for AI, 2024) and FineWeb (Hugging Face, 2024): documented open corpora of roughly 3 trillion and 15 trillion tokens.
  • The Common Pile v0.1 (2025): 8 terabytes of public domain and openly licensed text from 30 sources (see Recent Developments).

Two concerns now dominate corpus work: provenance (who created the text, under what license, and with what consent) and documentation (datasheets that describe what a corpus contains and omits). Both matter because a corpus shapes everything a model learns, including its biases.

Load video
YouTube
NLTK Corpora - Natural Language Processing With Python and NLTK p.9
Part of the NLTK Corpora is WordNet. I wouldn't totally classify WordNet as a Corpora, if anything it is really a giant Lexicon, but, either way, it is super useful. With WordNet we can do things like look up words and their meaning according to their parts of speech, we can find synonyms, antonyms, and even examples of the word in use.

Load video
YouTube
WordNet - Natural Language Processing With Python and NLTK p.10
Part of the NLTK Corpora is WordNet. I wouldn't totally classify WordNet as a Corpora, if anything it is really a giant Lexicon, but, either way, it is super useful. With WordNet we can do things like look up words and their meaning according to their parts of speech, we can find synonyms, antonyms, and even examples of the word in use.


Building your Corpora | Yassine Iabdounane

Load video
YouTube
#1 Introduction to Corpus Linguistics - What is Corpus Linguistics? (For Absolute Beginners)
Hello there! My name is Yassine Iabdounane and I'm a Moroccan MA student of Applied Linguistics and ELT. My areas of interest are Syntax, Phonology, Semantics, Cognitive Linguistic, and Corpus Linguistics. This video is made specifically for those who don't know much about Corpus Linguistics and would like to get started in the field.

Load video
YouTube
#2 Introduction to Corpus Linguistics - Types of Corpora
Hello there! In this video I present to you some of the most common types of corpora. This is by no means an exhaustive list but it's enough to get you started :)

Load video
YouTube
# 3 Introduction to Corpus Linguistics - Compiling and Analysing our First Corpus
Hello there! In this video, we compile our first corpus and analyse it using AntConc. We generate frequency lists, concordances, look for collocates, and discover a cool hack by the end of the video!

Load video
YouTube
#4 Introduction to Corpus Linguistics - Part-of-Speech Tagging and Working with Tagged Data
Hello there! In this video, we see how to tag a corpus for part of speech and how to work with tagged data using two methods/tools. We'll learn about how to look for specific parts of speech, how to restrict the search to only specific word classes, and how to use two regular expressions.

Load video
YouTube
#5 Introduction to Corpus Linguistics - Rank Words that Belong to a Specific Word Class
Hello there! In this video we see a really easy way to rank words that belong to specific word classes. If you want to rank nouns only, verbs only, adverbs only and so on, this video is for you! :D

Load video
YouTube
#6 Introduction to Corpus Linguistics - Look for Constructions in a Corpus
Hello there! In this video we see how to look for constructions in a tagged corpus. I've tagged the corpus using TagAnt. And, I've got my AC/DC shirt on :)

Load video
YouTube
#7 Introduction to Corpus Linguistics - Searching with Regular Expressions (Regex)
Hey there! In this video we see how to look for specific parts of speech or constructions using regular expressions.

Load video
YouTube
#8 Now what? What can I do in Corpus Linguistics?
Hey there! In this video we have a look at some fields and areas of research that use corpora to study language phenomena.

Ontology

An ontology is a formal specification of the concepts in a domain and the relations among them: classes, properties, and constraints that a computer can reason over. Embeddings and ontologies are complementary. Embeddings capture fuzzy similarity learned from usage; ontologies capture precise, agreed-upon relationships that support logical inference and auditing. Current practice increasingly uses language models to draft ontology entries, map free text onto ontology concepts, and suggest missing relations, with domain experts reviewing the results before they become authoritative.

Taxonomy

Youtube search... ...Google search

The practice and science of classification of things or concepts, including the principles that underlie such classification.

A taxonomy is a simpler cousin of an ontology: usually a single hierarchy of "is a kind of" relations (a golden retriever is a dog, which is a mammal). Automatic taxonomy construction classically used lexical patterns such as "X such as Y" (Hearst patterns, 1992) to harvest candidate relations from text, then organized them into a hierarchy. Language models can now propose taxonomies directly, but they tend to produce plausible-looking structures that drift in granularity and consistency, so taxonomy learning benchmarks and human review remain important. Taxonomies also underpin hierarchical text classification.

Taxonomy-of-linguistic-analyzers.png

Load video
YouTube

Related

Workbench / Pipeline

Youtube search... ...Google search

A practical NLP toolkit in 2026 typically draws on several layers:

  • Classic pipelines: NLTK (teaching and research), spaCy (fast production pipelines, with extensions that call large language models), Stanford CoreNLP and Stanza (linguistically detailed analysis across dozens of languages), GATE and Apache UIMA (document-processing frameworks for large annotation projects).
  • Model libraries: Hugging Face Transformers and sentence-transformers for loading, fine-tuning, and serving pretrained models.
  • Application frameworks: LangChain and similar libraries for chaining models, retrieval, and tools into applications.
  • Operations: versioned datasets, evaluation suites, and monitoring, often orchestrated with workflow tools such as Kubeflow Pipelines (see AIOps/MLOps).

The engineering question has shifted from "which components do I chain together?" to "where do I need an explicit, inspectable step instead of asking one large model to do everything?" Explicit steps (tokenization, de-identification, entity extraction, validation) remain the right choice wherever auditability, cost, or latency matter.

technology-stack1.png 4-Figure3-1.png A-global-model-of-the-Power-Workbench.png

Load video
YouTube
Load video
YouTube

Sapir Whorf Hypothesis - How language shapes the way we think

Youtube search... ...Google search

The Sapir-Whorf hypothesis, named for linguists Edward Sapir and Benjamin Lee Whorf, holds that the language you speak shapes how you think. The strong version (linguistic determinism: language fixes the boundaries of thought) has little support. The weak version (linguistic relativity: language nudges attention, memory, and categorization) has experimental support in areas such as color discrimination, spatial reference frames (some languages describe locations with cardinal directions rather than "left" and "right"), and grammatical gender.

Language models give the question a new twist. They learn from many languages at once, and interpretability research suggests that large models build partly language-independent conceptual representations, with shared multilingual features that become more prominent as models grow (see Inside the model). At the same time, models trained mostly on English can carry English-centric assumptions into other languages. Whether a shared conceptual space in a model tells us anything about human thought is an open question, but it gives researchers a controllable system for testing ideas that are difficult to test in people.

Load video
YouTube
Load video
YouTube

General Natural Language Processing (NLP) Videos

Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube
Load video
YouTube

Natural Language Processing (NLP) Techniques

Youtube search... | Quora search... ...Google search

Speech recognition, (speech) translation, understanding (semantic parsing) complete sentences, understanding synonyms of matching words, Sentiment Analysis, and writing/generating complete grammatically correct sentences and paragraphs.

Sources and Viewing

Key Sources

Reasoning and training

Tokenization and architectures

Embeddings, extraction, and long context

Interpretability and the science of language

Data and evaluation

Additional Viewing

Load video
YouTube

Stanford CS336 Language Modeling from Scratch, Spring 2025, Lecture 1: Overview and Tokenization (Stanford Online, about 79 minutes). Percy Liang and Tatsunori Hashimoto open their course by laying out how modern language models are built end to end, then work through tokenization in detail, from character and word schemes to byte pair encoding. It is an excellent bridge between the classic Tokenization section and the neural pipeline described on this page. Watch on YouTube

Load video
YouTube

Let's build the GPT Tokenizer (Andrej Karpathy, February 2024, about 2 hours 13 minutes). Karpathy builds the byte-level BPE tokenizer used by GPT models from scratch in code and then explains how tokenization causes many familiar model failures, from trouble with spelling and arithmetic to odd behavior on rare tokens. It is the most thorough practical companion to the tokenization and tokenization-as-compression discussions above. Watch on YouTube

Load video
YouTube

Let's build GPT: from scratch, in code, spelled out (Andrej Karpathy, about 2 hours). Karpathy implements a small transformer language model step by step following Attention Is All You Need, showing exactly how embeddings, self-attention, and next-token prediction fit together. Watch it alongside The Physics section to see the attention and cross-entropy equations come to life. Watch on YouTube

Load video
YouTube

Deep Dive into LLMs like ChatGPT (Andrej Karpathy, February 2025, about 3 hours 31 minutes). A general-audience but technically careful tour of the full training stack: pretraining data such as FineWeb, tokenization, the transformer, supervised fine-tuning, reinforcement learning, and reasoning models including DeepSeek-R1, plus the "psychology" of model failures such as hallucination. It covers most of The Mechanism and the reasoning-model developments in a single sitting. Watch on YouTube

Load video
YouTube

Stanford CS336 Language Modeling from Scratch, Spring 2026, Lecture 12: Evaluation (Stanford Online, May 2026, about 79 minutes). The lecture traces how NLP evaluation evolved from perplexity through exam-style benchmarks, human and LLM-judged chat comparisons, agentic tasks, and safety tests, and closes on contamination, realism, and dataset quality. It explains why the field keeps building harder benchmarks, the same trend visible in WMT25 and MMTEB. See also Benchmarks. Watch on YouTube

Retrieved from "https://primo.ai/w/index.php?title=Natural_Language_Processing_(NLP)&oldid=53486"