AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

train-sentence-transformers

Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder, dense or static embedding model for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker, pair scoring for two-stage retrieval / pair classification), `SparseEncoder` (SPLADE, sparse embedding model for learned-sparse retrieval), and `MultiVectorEncoder` (ColBERT / late-interaction, per-token embeddings scored with MaxSim). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.

Original source
huggingface
License
Apache-2.0
Pinned revision
25910b3d0ba31a2ba102f901cdc9e008c278a69c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read huggingface--train-sentence-transformers at revision 25910b3d0ba31a2ba102f901cdc9e008c278a69c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 17

references/evaluators_multi_vector_encoder.md

Multi-Vector-Encoder Evaluators

All evaluators live in sentence_transformers.multi_vector_encoder.evaluation. They mirror the bi-encoder evaluators but use MaxSim scoring end-to-end.

Top-line pick

| Task | Evaluator | |---|---| | Retrieval on a suite of common English IR benchmarks | MultiVectorNanoBEIREvaluator | | Custom retrieval corpus (your own docs / queries / qrels) | MultiVectorInformationRetrievalEvaluator | | Distillation from a cross-encoder teacher | MultiVectorDistillationEvaluator | | Reranking a fixed candidate list per query | MultiVectorRerankingEvaluator | | Triplet accuracy (does anchor score positive > negative?) | MultiVectorTripletEvaluator |

Default recommendation for training: MultiVectorNanoBEIREvaluator on a subset of NanoBEIR datasets (e.g. ["msmarco", "nq", "fiqa2018"]) during training, and the full suite at end-of-run. Cheap, well-calibrated, and the metric key format is stable.

MultiVectorNanoBEIREvaluator

Runs the 13 NanoBEIR sub-datasets and returns nDCG@10 averaged across them.

from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorNanoBEIREvaluator

evaluator = MultiVectorNanoBEIREvaluator(
    dataset_names=["msmarco", "nq", "fiqa2018"],  # subset for training-time evals
    batch_size=16,
)

primary_metric: "NanoBEIR_mean_maxsim_ndcg@10" (unless you pass aggregate_key="...", in which case it becomes "NanoBEIR_{aggregate_key}_maxsim_ndcg@10"). Note that NanoBEIREvaluator composes its display name from aggregate_key (and optionally truncate_dim), it does not accept a name kwarg.

metric_for_best_model key: "eval_NanoBEIR_mean_maxsim_ndcg@10" (add the eval_ prefix, since Trainer adds it to evaluator metrics).

  • dataset_names: list of NanoBEIR sub-datasets. Full list: msmarco, nq, fiqa2018, hotpotqa, nfcorpus, arguana, scidocs, climatefever, dbpedia, fever, quoraretrieval, scifact, touche2020. Note it is quoraretrieval, not quora: an invalid name raises at construction.
  • batch_size: also drives corpus encoding, so scale to fit memory.
  • English-only. For non-English retrieval, use MultiVectorInformationRetrievalEvaluator with your own corpus.

MultiVectorInformationRetrievalEvaluator

Full IR evaluator over a corpus + queries + qrels you supply.

from sentence_transformers.multi_vector_encoder.evaluation import MultiVectorInformationRetrievalEvaluator

evaluator = MultiVectorInformationRetrievalEvaluator(
    queries={qid: text, ...},                       # dict of query_id to query_text
    corpus={did: text, ...},                        # dict of doc_id to doc_text
    relevant_docs={qid: {did1, did2, ...}, ...},    # qid to set of relevant doc_ids
    batch_size=16,
    name="my-eval",                                 # optional, prefixes the metric key
    write_csv=True,                                 # persists per-eval metrics
    ndcg_at_k=[10],                                 # customize k-values if you need @20/@100 etc.
)

primary_metric: "maxsim_ndcg@10" by default (built from score_function="maxsim" and max(ndcg_at_k)).

metric_for_best_model key with name="my-eval": "eval_my-eval_maxsim_ndcg@10".

  • main_score_function overrides the score used in primary_metric if you set a non-default. Default is None, which resolves to maxsim from the model's similarity_fn_name at call time. For MVE just leave it as None.
  • Reports MRR@k, nDCG@k, Recall@k, Precision@k, MAP@k for the k-values you pass (mrr_at_k, ndcg_at_k, etc.).
  • chunk_elements (default None) is the element budget for the padded (chunk, d_tokens, dim) documents plus the (num_queries, chunk, q_tokens, d_tokens) MaxSim scoring intermediate. Leave it as None: maxsim packs documents into chunks under a 100M-element budget (at most ~400 MB, half that in bf16 / fp16), adapting to the query count and document lengths. Lower it to cut evaluation memory further. It is a separate axis from corpus_chunk_size, which caps how many encoded documents are held at a time.

MultiVectorDistillationEvaluator

Regression of student MaxSim scores against teacher scores over a held-out set of (query, doc, teacher_score) rows. Use it when you're distilling.

primary_metric: "spearman" (correlation of student scores with teacher scores).

metric_for_best_model key with name="my-eval": "eval_my-eval_spearman".

  • Pair with MultiVectorMarginMSELoss or MultiVectorDistillKLDivLoss during training.
  • Higher is better (student ranking matches teacher).

MultiVectorRerankingEvaluator

Reranks a fixed list of candidates per query. Same shape as the bi-encoder RerankingEvaluator, MaxSim-scored.

  • Data: samples = [{"query": ..., "positive": [...], "negative": [...]} , ...].
  • Reports MAP, MRR@10, nDCG@10 by default.

MultiVectorTripletEvaluator

Fraction of (anchor, positive, negative) triplets where sim(anchor, positive) > sim(anchor, negative). Cheap sanity signal, not what you want to gate a real IR release on.

Named-evaluator metric key format (universal)

If you pass name="..." to any of these, the metric key format is:

eval_{name}_{primary_metric}

If you don't pass name, the key is just eval_{primary_metric}. The trainer's metric_for_best_model=... must match exactly, or load_best_model_at_end selects the wrong checkpoint. Run the evaluator once before training to observe the exact key format: primary_metric is None until the first evaluator(model) call, which composes it from the resolved score function and the ndcg_at_k you passed, then prefixes name. The production template does exactly this before building TrainingArguments.

Gotchas

  • Metric key mismatch on metric_for_best_model: silent failure. Training runs to completion and load_best_model_at_end picks a stale checkpoint. Always print evaluator(model) output once before starting the trainer and confirm the key format.
  • Eval-time OOM: MaxSim scoring builds (num_queries, chunk, q_tokens, d_tokens) intermediates, which maxsim bounds by default under its 100M-element budget. If you still OOM, lower chunk_elements, then drop batch_size. NanoBEIR corpora can be small enough that oversized batch_size bites before you'd expect.
  • MultiVectorNanoBEIREvaluator with a Non-English base: it's English-only. You'll get zero or garbage scores. Use MultiVectorInformationRetrievalEvaluator with an in-domain corpus.
  • Distillation eval on the same data as training: measures memorization, not generalization. Hold out a separate (query, doc, teacher_score) split.