What Are Embeddings? How AI Turns Text Into Vectors

By Pushpjeet Cholkar, 17 years of corporate experience, starting in databases and now working in cloud, AI and ML.

Last updated: October 20, 2026

Short answer: An embedding is a list of numbers (a vector) that a model produces for a piece of text, an image or another item, arranged so that items with similar meaning get vectors that point in similar directions. That lets software compare meaning with simple arithmetic, which powers semantic search, RAG, recommendations and clustering.

Why do we need embeddings?

Computers compare numbers well and words badly. A keyword search sees “bill” and “invoice” as two unrelated strings. A customer typing “where do I get my bill” into a help center built on keyword matching gets nothing back, even though an article called “Download your monthly invoice as a PDF” exists.

An embedding fixes this by giving every piece of text a position in a space of numbers, where position reflects meaning. Then “find related text” becomes “find nearby points”, and that is fast arithmetic.

I spent the first part of my career in databases, where a query matches exactly what you ask for. Embeddings are the opposite contract: you get the nearest matches by meaning, never a guaranteed exact one. Keeping that difference in mind prevents most of the design mistakes later in this article.

What is an embedding? Five levels

Level 1: the intuition

Think of a map. Every city has two numbers, latitude and longitude, and cities that are close on the map have close numbers. An embedding does the same for meaning, except the map has hundreds of directions instead of two. Sentences about passwords cluster in one region, sentences about billing in another.

Level 2: the technical definition

An embedding model is a function that takes an input (a word, sentence, paragraph, image) and returns a fixed-length vector of floating-point numbers. The model is trained so that inputs with similar meaning produce vectors with a high cosine similarity.

• The vector length (the dimension) is fixed by the model. The sentence-transformers model all-MiniLM-L6-v2 maps sentences and paragraphs to a 384-dimensional vector. Amazon Titan Text Embeddings V2 returns 1,024 numbers by default, or 512 or 256 if you request them.
• Individual numbers usually have no human-readable meaning. “Dimension 17 = sweetness” is a teaching story, not how real models work. Meaning lives in the direction of the whole vector.
• The vectors are dense: nearly every number is non-zero. That contrasts with older sparse representations, such as one column per vocabulary word, where almost every entry is zero.

Level 3: the mathematics

Write the model as a function

$$f: \text{text} \rightarrow \mathbb{R}^{d}$$

Here $f$ is the embedding model, $\mathbb{R}^{d}$ means “a list of $d$ real numbers”, and $d$ is the dimension (384 for MiniLM).

To compare two texts, compare their vectors $\mathbf{a} = f(\text{text}_1)$ and $\mathbf{b} = f(\text{text}_2)$ with cosine similarity:

$$\cos(\theta) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert} = \frac{\sum_{i=1}^{d} a_i b_i}{\sqrt{\sum_{i=1}^{d} a_i^2}\,\sqrt{\sum_{i=1}^{d} b_i^2}}$$

• $\theta$ is the angle between the two vectors.
• $\mathbf{a} \cdot \mathbf{b}$ is the dot product: multiply matching positions $a_i b_i$ and add them up.
• $\lVert \mathbf{a} \rVert$ is the length (L2 norm) of $\mathbf{a}$. Dividing by both lengths removes the effect of size and keeps only direction.
• The result lies between $-1$ and $1$. Higher means more similar in meaning.

If every vector is first scaled to length 1 (normalized), the denominator becomes $1 \times 1$ and cosine similarity is just the dot product. That is why most systems normalize at indexing time: search then needs only multiplications and additions. The maths section of the RAG tutorial works through a full example by hand, including why the raw dot product can pick the wrong document. If vectors and dot products are new to you, start with the vectors section of Mathematics for Machine Learning.

How does the model learn good vectors? Modern sentence-embedding models are usually trained with a contrastive objective: show the model pairs of texts that belong together (a question and its answer, two paraphrases) and push their vectors together while pushing apart the other texts in the same batch. One common form of that loss, for a query $q$ and its matching text $p^{+}$ in a batch of $B$ candidate texts $p_1, \dots, p_B$, is

$$\mathcal{L} = -\log \frac{\exp\left(s \cdot \cos(f(q), f(p^{+}))\right)}{\sum_{j=1}^{B} \exp\left(s \cdot \cos(f(q), f(p_j))\right)}$$

• $\mathcal{L}$ is the loss the training minimizes for one query.
• The numerator grows when the query and its true partner point the same way.
• The denominator adds up the same quantity for every candidate in the batch, so the loss is small only when the true partner beats the others.
• $s$ is a scale factor (a fixed constant or a learned temperature) that sharpens the comparison.

The all-MiniLM-L6-v2 model card describes this approach: cosine similarity between every possible pair in a batch, then cross-entropy loss against the true pairs, trained on more than a billion sentence pairs.

Level 4: build one from scratch

You don’t need a neural network to see the core idea. The oldest route to word vectors is the distributional hypothesis: words used in similar contexts have similar meanings. Count which words appear near each other, treat each word’s row of counts as its vector, and similar words get similar rows.

The script below uses only NumPy. It builds count vectors from 16 short sentences about pets and AWS, then compresses them to 2 numbers per word with singular value decomposition (SVD), the same linear algebra behind older methods such as latent semantic analysis.

"""Build tiny word embeddings from scratch with NumPy (no ML library)."""
import numpy as np

corpus = [
    "the cat drinks milk", "the dog drinks water",
    "the cat chases the mouse", "the dog chases the ball",
    "the kitten drinks milk", "the puppy drinks water",
    "the kitten chases the ball", "the puppy chases the mouse",
    "aws stores files in s3", "aws runs servers on ec2",
    "s3 stores objects in buckets", "ec2 runs virtual servers",
    "lambda runs code on aws", "s3 stores backups",
    "ec2 runs code", "lambda runs code",
]
STOP_WORDS = {"the", "a", "in", "on"}

# 1. Tokenize and build the vocabulary (every distinct word gets an id).
sentences = [[w for w in line.split() if w not in STOP_WORDS] for line in corpus]
vocab = sorted({w for words in sentences for w in words})
word_to_id = {w: i for i, w in enumerate(vocab)}
V = len(vocab)

# 2. Co-occurrence matrix: counts[i, j] = how often word j appears
#    within WINDOW positions of word i. Row i is word i's first "embedding".
WINDOW = 2
counts = np.zeros((V, V))
for words in sentences:
    for i, center in enumerate(words):
        for j in range(max(0, i - WINDOW), min(len(words), i + WINDOW + 1)):
            if i != j:
                counts[word_to_id[center], word_to_id[words[j]]] += 1


def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))


pairs = [("cat", "kitten"), ("dog", "puppy"), ("cat", "dog"), ("ec2", "lambda"),
         ("milk", "water"), ("cat", "s3"), ("milk", "servers")]

print(f"Vocabulary: {V} words, so each count vector has {V} numbers")
print("Row for 'cat':", {vocab[j]: int(c) for j, c in enumerate(counts[word_to_id['cat']]) if c})
print("\nCosine similarity of count vectors")
for a, b in pairs:
    print(f"  {a:>6} vs {b:<8} {cosine(counts[word_to_id[a]], counts[word_to_id[b]]):.2f}")

# 3. Compress: SVD keeps the directions that explain most of the counts.
#    Keeping 2 of them turns each 23-number row into a dense 2-number embedding.
U, S, _ = np.linalg.svd(counts)
DIM = 2
dense = U[:, :DIM] * S[:DIM]

print(f"\nDense {DIM}-D embeddings (from SVD)")
for w in ["cat", "kitten", "milk", "s3", "ec2", "lambda"]:
    x, y = dense[word_to_id[w]]
    print(f"  {w:>6} -> [{abs(x) if abs(x) < 5e-4 else x:6.2f}, {abs(y) if abs(y) < 5e-4 else y:6.2f}]")

print(f"\nCosine similarity of {DIM}-D embeddings")
for a, b in pairs:
    score = cosine(dense[word_to_id[a]], dense[word_to_id[b]])
    print(f"  {a:>6} vs {b:<8} {abs(score) if abs(score) < 5e-3 else score:.2f}")

Real output (Python 3.13.5, NumPy 2.5.4, run on October 19, 2026):

Vocabulary: 23 words, so each count vector has 23 numbers
Row for 'cat': {'chases': 1, 'drinks': 1, 'milk': 1, 'mouse': 1}

Cosine similarity of count vectors
     cat vs kitten   0.75
     dog vs puppy    0.75
     cat vs dog      0.50
     ec2 vs lambda   0.82
    milk vs water    0.67
     cat vs s3       0.00
    milk vs servers  0.00

Dense 2-D embeddings (from SVD)
     cat -> [  0.00,   1.43]
  kitten -> [  0.00,   1.43]
    milk -> [  0.00,   1.43]
      s3 -> [ -0.31,   0.00]
     ec2 -> [ -2.73,   0.00]
  lambda -> [ -2.08,   0.00]

Cosine similarity of 2-D embeddings
     cat vs kitten   1.00
     dog vs puppy    1.00
     cat vs dog      1.00
     ec2 vs lambda   1.00
    milk vs water    1.00
     cat vs s3       0.00
    milk vs servers  0.00

What the output teaches:

1. Context creates similarity. Nobody told the code that a kitten is a young cat. “cat” and “kitten” score 0.75 because both appear next to “drinks”, “milk”, “chases” and “mouse”. “cat” and “s3” never share a neighbor, so they score 0.00.
2. Compression keeps the big structure. In 2 dimensions every pet word sits on one axis and every AWS word on the other. The two topics are perfectly separated.
3. Too few dimensions lose detail. In 2-D, “cat” and “milk” get the same vector, so the model can no longer tell an animal from a drink. This is the reason real models use hundreds or thousands of dimensions: they need room for many distinctions at once.

On another machine the SVD signs may flip (for example 1.43 instead of -1.43), because an SVD direction and its opposite are equally valid. The similarities don’t change.

Level 5: a real embedding model

Count-based vectors break down at scale: they don’t handle word order, whole sentences or phrases that mean the same thing with different words. Pretrained neural models do. The next script compares a plain keyword match with all-MiniLM-L6-v2, a small open-source model that runs on a laptop CPU.

Install the dependencies first (the model, about 90 MB, downloads on first run):

pip install sentence-transformers numpy

"""Semantic search with a pretrained embedding model vs. plain keyword overlap."""
import numpy as np
from sentence_transformers import SentenceTransformer

help_articles = [
    "How to reset a forgotten password",
    "Update the credit card on your account",
    "Download your monthly invoice as a PDF",
    "Turn on two-factor authentication",
    "Cancel your subscription",
    "Change the email address you sign in with",
]
queries = [
    "I can't log in to my account",
    "where do I get my bill",
    "stop charging me every month",
]

STOP_WORDS = {"i", "to", "my", "me", "a", "the", "do", "in", "on", "your", "how", "as", "you", "with"}


def keyword_score(query, doc):
    """Fraction of query words (minus stop words) found in the document. No meaning involved."""
    q = set(query.lower().split()) - STOP_WORDS
    d = set(doc.lower().split()) - STOP_WORDS
    return len(q & d) / len(q)


model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
doc_vectors = model.encode(help_articles, normalize_embeddings=True)   # shape (6, 384)
query_vectors = model.encode(queries, normalize_embeddings=True)       # shape (3, 384)
print("Document matrix shape:", doc_vectors.shape)
print("First 5 numbers of the first article's embedding:", np.round(doc_vectors[0][:5], 3))
print("Vector length after normalization:", round(float(np.linalg.norm(doc_vectors[0])), 3))

# Vectors are unit length, so the dot product IS the cosine similarity.
scores = query_vectors @ doc_vectors.T                                  # shape (3, 6)

for qi, query in enumerate(queries):
    kw = [keyword_score(query, doc) for doc in help_articles]
    kw_best = int(np.argmax(kw))
    kw_text = repr(help_articles[kw_best]) if kw[kw_best] > 0 else "no article shares a word"
    order = np.argsort(-scores[qi])
    print(f"\nQuery: {query!r}")
    print(f"  Keyword best: {kw_text} (overlap {kw[kw_best]:.2f})")
    print("  Embedding top 2:")
    for rank in order[:2]:
        print(f"    {scores[qi, rank]:.3f}  {help_articles[rank]}")

Real output (sentence-transformers 6.1.0, PyTorch 2.14.1 CPU, run on October 19, 2026; download progress messages removed):

Document matrix shape: (6, 384)
First 5 numbers of the first article's embedding: [-0.021 -0.021 -0.07   0.018 -0.05 ]
Vector length after normalization: 1.0

Query: "I can't log in to my account"
  Keyword best: 'Update the credit card on your account' (overlap 0.33)
  Embedding top 2:
    0.474  How to reset a forgotten password
    0.384  Change the email address you sign in with

Query: 'where do I get my bill'
  Keyword best: no article shares a word (overlap 0.00)
  Embedding top 2:
    0.428  Download your monthly invoice as a PDF
    0.351  Update the credit card on your account

Query: 'stop charging me every month'
  Keyword best: no article shares a word (overlap 0.00)
  Embedding top 2:
    0.488  Cancel your subscription
    0.382  Download your monthly invoice as a PDF

Line by line, the important parts:

• SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") loads the model. Every text you embed later, documents and queries alike, must go through this same model.
• model.encode(..., normalize_embeddings=True) returns one 384-number vector per text, scaled to length 1. The printed length of 1.0 confirms it.
• query_vectors @ doc_vectors.T computes every query-to-article cosine similarity in one matrix multiplication. A vector database does the same thing at scale, with an index so it doesn’t have to check every row.
• np.argsort(-scores[qi]) sorts articles from most to least similar.

The results are the point of the whole article. The keyword match picked the credit card article for a login problem, because both contain “account”. For the billing and cancellation questions, keyword matching found nothing at all. The embedding model ranked the right article first for all three queries, with no shared words.

Notice the scores too: the best matches sit around 0.43 to 0.49, not near 1.0. Absolute scores depend on the model, so judge an embedding search by its ranking, and pick any score cutoff by testing it on your own data.

Types of embeddings

For GenAI work, the second row is the one you’ll use most. When people say “embeddings” in a RAG or vector database discussion, they almost always mean dense sentence or passage embeddings.

Where are embeddings used?

• Semantic search. Help centers, internal wikis and code search, as in the demo above.
• Retrieval-augmented generation (RAG). Document chunks are embedded once and stored. Each question is embedded with the same model, the nearest chunks are retrieved, and those chunks go into the LLM prompt. The RAG tutorial in Python builds this end to end, and RAG vs fine-tuning covers when retrieval is the right fix in the first place.
• Clustering and topic discovery. Group thousands of support tickets or survey answers by meaning before anyone reads them.
• Deduplication. Find near-duplicate documents, questions or product listings whose wording differs.
• Classification with little labeled data. Train a simple classifier, such as logistic regression, on top of embeddings instead of raw text.
• Recommendations. “Customers who viewed this also viewed” based on vector closeness between items or between a user and items.

Embedding models on Amazon Bedrock

If you build on AWS, Amazon Bedrock hosts embedding models behind an API, and Amazon Bedrock Knowledge Bases can use them to build a managed RAG index. These details come from the Bedrock user guide, checked on October 19, 2026. Model lists change, so confirm them in the current docs before you design around one.

Pricing isn’t listed here because it changes; check the Amazon Bedrock pricing page for current figures.

Here is the Titan V2 call in Python. It is not executed for this article, because it calls a paid AWS service and needs your own credentials, Region and model access. The request fields come from the Bedrock user guide.

import json

import boto3

client = boto3.client("bedrock-runtime", region_name="us-east-1")

request = {
    "inputText": "How to reset a forgotten password",
    "dimensions": 512,      # accepted: 1024 (default), 512, 256
    "normalize": True,      # default is true; unit-length vectors
}
response = client.invoke_model(
    modelId="amazon.titan-embed-text-v2:0",
    body=json.dumps(request),
)
result = json.loads(response["body"].read())
vector = result["embedding"]
print(len(vector), result["inputTextTokenCount"])

Expected behavior, per the docs: vector holds 512 floats and inputTextTokenCount reports how many tokens the input used. Swap the local model for this one only if you re-embed every document too.

How do you choose an embedding model?

Work through these in order:

1. Language and domain. English-only text can use an English model. Mixed-language data needs a multilingual one. Heavy domain jargon (legal, medical, code) is the case where you should test several models on your own data rather than trust a general leaderboard.
2. Input length. Check the model’s limit. all-MiniLM-L6-v2 truncates input longer than 256 word pieces by default; Titan Text Embeddings V2 accepts up to 8,192 tokens. Either way, you will usually chunk documents for retrieval.
3. Dimensions vs. cost. Every vector is stored and searched. Fewer dimensions mean less storage and faster search, at some cost in accuracy. Titan V2’s 256 and 512 options and binary output exist for that trade-off.
4. Where it runs. A managed API (Bedrock) means no servers to run but per-request cost and data leaving your process. A local open-source model means you run the compute yourself, but data stays inside your environment.
5. Your own evaluation. Write 20 to 50 real questions with the document each should find, then measure how often the right one appears in the top results (hit rate@k). The RAG tutorial’s evaluation section shows how. A model that tops a public benchmark can still lose on your data.

Common mistakes with embeddings

1. Mixing models. Indexing documents with one model and embedding queries with another (or a newer version of the same model) gives meaningless similarities, often with no error. Store the model name and version next to the vectors.
2. Changing dimensions without re-indexing. A vector index is created for one dimension. Moving Titan V2 from 1,024 to 512 means re-embedding every document and rebuilding the index.
3. Silent truncation. Feed a whole PDF page to a model with a 256-word-piece limit and everything after the limit is ignored. Chunk first, then embed.
4. Treating a score as a probability. A cosine of 0.47 is not “47% relevant”. Score ranges vary by model, so tune any cutoff on your own data.
5. Using a raw dot product on vectors that aren’t normalized. Longer vectors win for the wrong reason. Normalize, or use cosine similarity explicitly.
6. Expecting embeddings to match exact identifiers. Order numbers, error codes and SKUs are where keyword search is stronger. Production systems often run hybrid search: keyword and vector results merged.
7. Assuming embeddings remove the need for access control. Retrieval returns whatever is nearest. If a user shouldn’t see a document, filter it out with metadata before or during the search.

Frequently asked questions

What is an embedding in simple terms?

An embedding is a list of numbers that represents a piece of data, such as a sentence, so that items with similar meaning get similar lists. Software can then compare meaning by measuring how close two lists of numbers are.

What is the difference between embeddings and keyword search?

Keyword search matches the words you typed. Embedding search matches meaning, so “where do I get my bill” can find an article titled “Download your monthly invoice” with no shared words. Many production systems combine both in hybrid search.

What does embedding dimension mean?

It is how many numbers are in each vector. all-MiniLM-L6-v2 outputs 384; Amazon Titan Text Embeddings V2 outputs 1,024 by default, or 512 or 256 if you ask. More dimensions can hold finer distinctions but cost more storage and search time.

Can I compare embeddings from two different models?

No. Each model defines its own vector space, so a vector from one model means nothing to another, even when the lengths match. Index documents and embed queries with the same model and version, and re-embed everything when you switch.

How are embeddings used in RAG?

Document chunks are embedded once and stored in a vector index. At question time the question is embedded with the same model, the closest chunks are retrieved by cosine similarity, and those chunks are added to the prompt sent to the LLM.

Which embedding models can I use on AWS?

Amazon Bedrock offers Amazon Titan Text Embeddings V2 and Cohere Embed English and Multilingual for text, plus multimodal embedding models, and Bedrock Knowledge Bases can use them for RAG. You can also host an open-source model such as all-MiniLM-L6-v2 yourself.

Where to go next

For the glossary-level view of embeddings alongside other AI concepts, see AI/ML fundamentals. To use embeddings in a working retrieval pipeline, follow the RAG tutorial in Python. More GenAI explainers are collected in the Generative AI tutorials hub.

Sources

All checked on October 19, 2026.

• Amazon Bedrock User Guide: Amazon Titan Text Embeddings models (model ID, 8,192-token and 50,000-character input limits, 1,024/512/256 output sizes, chunking recommendation)
• Amazon Bedrock User Guide: Amazon Titan Embeddings Text request and response (inputText, dimensions, normalize default true, embeddingTypes)
• Amazon Bedrock User Guide: Supported models and Regions for Amazon Bedrock knowledge bases (embedding models, vector types and dimensions)
• Amazon Nova User Guide: Using Nova Embeddings (modalities, 8K-token and 30-second limits, 3,072/1,024/384/256 dimensions)
• Hugging Face: sentence-transformers/all-MiniLM-L6-v2 model card (384 dimensions, 256-word-piece truncation, contrastive training on more than 1 billion sentence pairs)
• Reimers and Gurevych (2019): Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Mikolov et al. (2013): Efficient Estimation of Word Representations in Vector Space (word2vec)
• Amazon Bedrock pricing (check live figures; this article states none)

Learn embeddings and RAG with guided practice. For live training on embeddings, retrieval and LLM applications, see the RAG and LLM training page. If you’re in the United States, see online AWS and AI/ML training in the USA.