EmbeddingGemma: Micro embeddings for mobile AI

On-device AI is moving fast, and compact text embeddings are at the heart of that shift. EmbeddingGemma is a small, efficient family of text-only embedding models designed for phones, Raspberry Pi, and other edge targets. It supports matryoshka embeddings—letting developers choose compact vector sizes without retraining—making it a strong fit for semantic search, RAG, classification, and clustering entirely offline.

What EmbeddingGemma is and why it matters

EmbeddingGemma is purpose-built for low-latency, low-memory environments while remaining competitive on standard evaluations. According to the official announcement, the models are text-only, support sequence lengths around 2K tokens, and expose matryoshka outputs so you can select the dimensionality that best fits your device and use case. That means you can generate a 768-dimensional vector for maximum accuracy or slice down to 128 for ultra-compact storage and faster retrieval—all from the same model output. See the announcement for details: Introducing EmbeddingGemma.

These models extend the broader Gemma family, which includes on-device chat-centric variants and encoder–decoder research models. For background on Gemma and supported runtimes, start with the official hub: Gemma documentation and model hub.

Where EmbeddingGemma fits in the Gemma family

Recent Gemma releases have trended in two complementary directions: (1) highly optimized on-device models suitable for phones and small edge boxes, and (2) research-friendly variants (including encoder–decoder architectures inspired by T5) for experimentation. EmbeddingGemma slots into the on-device track, giving developers a compact, production-ready embedding model they can pair with a lightweight local LLM for fully offline RAG or semantic features. The matryoshka design lets teams trade a small drop in accuracy for significant gains in speed and memory.

Matryoshka embeddings in plain terms

Matryoshka embeddings are trained so that lower-dimensional prefixes of the vector preserve most of the task-relevant information. Instead of maintaining multiple separate models for 128, 256, 512, and 768 dimensions, you export one embedding and use the first D dimensions to match your constraints. The original idea is described in ‘Matryoshka Representation Learning’: paper on arXiv. In practice, this means you can start at 768 dims during development, then roll out 256 or 128 dims in production to fit stricter latency or storage targets without retraining.

Performance notes and evaluation

Embedding models are commonly compared on MTEB (Massive Text Embedding Benchmark), which covers retrieval, classification, clustering, reranking, and more. While larger models still top the charts, compact options like EmbeddingGemma rank competitively against similarly sized baselines across multilingual and English tasks. You can explore current results and filters on the live leaderboard: MTEB leaderboard.

Practical capabilities on-device

• Semantic search and local question answering (RAG) without network access
• Topic classification, intent detection, and routing
• Clustering for deduplication, content grouping, and recommendations
• Lightweight personalization on mobile devices
• Creative utilities such as mapping text prompts to curated palettes or tags

With matryoshka outputs, you can pick 768 dims during development and later reduce to 256 or 128 dims to fit tight memory or speed budgets. This is especially useful for mobile devices with limited storage or for vector databases running on a Raspberry Pi where RAM is precious.

Getting started with Sentence-Transformers

EmbeddingGemma is compatible with the popular Sentence-Transformers ecosystem, making it straightforward to prototype and benchmark. The project’s documentation covers installation, pooling options, and similarity utilities: Sentence-Transformers docs.

Install the essentials

pip install sentence-transformers numpy

Encode a query and documents, then rank by similarity

from sentence_transformers import SentenceTransformer, util; import numpy as np; model_id = 'your-embeddinggemma-model-id'; model = SentenceTransformer(model_id, device='cpu'); query = 'Which planet is known as the red planet?'; docs = ['Venus is the second planet from the Sun.', 'Mars, known for its reddish hue, is often called the Red Planet.', 'Jupiter is the largest planet in the Solar System.', 'Saturn is notable for its prominent ring system.']; q_emb = model.encode(query, normalize_embeddings=True); d_emb = model.encode(docs, normalize_embeddings=True); sims = util.cos_sim(q_emb, d_emb).tolist()[0]; best = int(np.argmax(sims)); print('Best match:', docs[best], ' Score:', sims[best])

To use compact vectors with matryoshka-style slicing, keep the same embeddings and slice the prefix: for 256 dims use emb[:256], for 128 dims use emb[:128]. You’ll trade a bit of accuracy for speed and memory—often a favorable exchange on-device.

Build a tiny, fully local RAG

A minimal offline RAG system pairs EmbeddingGemma with a small local LLM and a lightweight vector store. The example below uses LangChain and Chroma. Explore the frameworks here: LangChain quickstart and Chroma documentation.

Prototype pipeline

from langchain.text_splitter import RecursiveCharacterTextSplitter; from langchain_community.vectorstores import Chroma; from langchain.embeddings import HuggingFaceEmbeddings; from transformers import pipeline; corpus = 'Paste your knowledge base text here (docs, FAQs, notes).'; splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120); chunks = [c.page_content for c in splitter.create_documents([corpus])]; embed = HuggingFaceEmbeddings(model_name='your-embeddinggemma-model-id'); vdb = Chroma.from_texts(chunks, embed); retriever = vdb.as_retriever(search_kwargs={'k': 4}); llm = pipeline('text-generation', model='your-local-llm-id', device_map='auto'); def ask(q): ctx = retriever.get_relevant_documents(q); prompt = 'Answer the question using only the context.\n\nContext:\n' + '\n'.join([c.page_content for c in ctx]) + '\n\nQuestion: ' + q + '\nAnswer:'; return llm(prompt, max_new_tokens=256, temperature=0.2)[0]['generated_text']; print(ask('Summarize the key points.'))

On capable phones or a Raspberry Pi-class device, this setup can run entirely offline: embeddings on CPU, a small generative model accelerated by GPU/NPU if available, and a tiny vector store on disk.

Deployment notes for mobile and edge

• Quantization and ONNX: Exporting to ONNX and applying post-training quantization significantly reduces runtime memory and improves latency. See ONNX Runtime quantization.
• Batching: If you need to embed many texts at once, small micro-batches (8–32) often hit a sweet spot on mobile devices. Measure throughput vs. latency.
• Matryoshka tuning: Profile your accuracy at 768, 512, 256, and 128 dims. In many RAG workflows, the quality delta from 768→256 is modest compared to the gains in speed and storage.
• Storage planning: At 128 dims with float16, you can store millions of vectors on-device within a few gigabytes. Consider PQ/IVF indexes if you migrate to ANN libraries later.
• Prompt construction: For small local LLMs, provide concise, retrieval-grounded prompts. Keep generation temperatures low and cap output tokens to preserve latency.
• Observability: Log retrieval scores, top-k documents, prompt length, and latency. These signals help you tune chunking, dimensionality, and index parameters.

When to choose alternatives

If your application is strictly server-side or latency is less constrained, larger embedding models will still offer higher accuracy on challenging retrieval tasks. For compact baselines worth evaluating alongside EmbeddingGemma, check the current standings on the MTEB leaderboard. Many teams also compare against widely used small models from established libraries documented in the Sentence-Transformers docs.

FAQ

Does EmbeddingGemma support multimodal inputs?

No—these are text-only models. For multimodal tasks, pair text embeddings with a separate vision encoder or choose a dedicated multimodal model.

How should I pick the embedding dimension?

Prototype at 768 dims, collect evaluation metrics, then slice to 512/256/128 to test quality-vs-latency trade-offs. The matryoshka property lets you adapt without retraining.

What vector store should I use on-device?

Chroma is a simple starting point for local development. For production-grade ANN at scale, consider migrating to Faiss, Milvus, or Qdrant on the server side once data sizes grow.

Can I run everything on CPU?

Yes, embeddings typically run fast enough on CPU, and micro-LLMs can run on CPU for small prompts. Hardware acceleration (mobile NPU, GPU) still helps, especially for longer outputs.

Key takeaways

• EmbeddingGemma delivers compact text embeddings designed for phones, Raspberry Pi, and edge boxes.
• Matryoshka outputs allow flexible dimensionality (e.g., 768→128) without retraining.
• Performance is competitive among small models on common evaluations like MTEB.
• Paired with a small local LLM and a light vector store, you can ship fully offline RAG.
• Quantization and careful batching are essential for production-grade latency on-device.

For further reading and up-to-date resources, consult the official announcement (EmbeddingGemma overview), the Gemma documentation, the Sentence-Transformers docs, and the live MTEB leaderboard to compare results across tasks and model sizes.