AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

train-sentence-transformers

Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder, dense or static embedding model for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker, pair scoring for two-stage retrieval / pair classification), `SparseEncoder` (SPLADE, sparse embedding model for learned-sparse retrieval), and `MultiVectorEncoder` (ColBERT / late-interaction, per-token embeddings scored with MaxSim). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.

Original source
huggingface
License
Apache-2.0
Pinned revision
25910b3d0ba31a2ba102f901cdc9e008c278a69c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read huggingface--train-sentence-transformers at revision 25910b3d0ba31a2ba102f901cdc9e008c278a69c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 17

references/evaluators_sparse_encoder.md

Evaluators (Sparse Encoder)

All sparse-encoder evaluators live in sentence_transformers.sparse_encoder.evaluation. They mirror the bi-encoder versions with a Sparse prefix and default to dot product similarity (cosine on sparse vectors is less meaningful).

Choosing the right evaluator

| Task | Evaluator | |---|---| | Retrieval (nDCG, MRR, Recall), fast default | SparseNanoBEIREvaluator | | Retrieval on your own corpus / qrels | SparseInformationRetrievalEvaluator | | STS / continuous similarity | SparseEmbeddingSimilarityEvaluator | | Binary classification | SparseBinaryClassificationEvaluator | | Triplet accuracy | SparseTripletEvaluator | | Reranking (from retrieval candidates) | SparseRerankingEvaluator | | MSE vs. teacher (distillation) | SparseMSEEvaluator | | Translation (cross-lingual alignment) | SparseTranslationEvaluator | | Hybrid BM25 + sparse retrieval | ReciprocalRankFusionEvaluator |

Wrap multiple in SequentialEvaluator (from sentence_transformers.base.evaluation):

from sentence_transformers.base.evaluation import SequentialEvaluator
evaluator = SequentialEvaluator([sparse_nano_beir, my_custom_ir])

The default: SparseNanoBEIREvaluator

Small, fast subset of BEIR adapted for sparse retrieval. Typical runtime <1 minute on a mid-range GPU.

from sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator

evaluator = SparseNanoBEIREvaluator(
    dataset_names=["msmarco", "nfcorpus", "nq"],   # default: all 13 NanoBEIR datasets
    batch_size=32,
    show_progress_bar=False,
)

Output key for metric_for_best_model: eval_NanoBEIR_mean_dot_ndcg@10 (sparse defaults to dot product).

Sparsity tracking

Unlike the dense variant, the sparse evaluator also reports active dimension counts so you can monitor sparsity during training:

  • query_active_dims: non-zero entries per query vector
  • document_active_dims: non-zero entries per document vector

A healthy SPLADE checkpoint typically shows ~30-50 active dims for queries and ~150-250 for documents. If these drift toward the vocab size (~30k), the FLOPS regularization isn't doing its job. Raise query_regularizer_weight / document_regularizer_weight in SpladeLoss.

Retrieval on your own corpus

SparseInformationRetrievalEvaluator

Same shape as the dense version but operates on sparse vectors internally:

from sentence_transformers.sparse_encoder.evaluation import SparseInformationRetrievalEvaluator

evaluator = SparseInformationRetrievalEvaluator(
    queries={qid: text for qid, text in ...},
    corpus={doc_id: text for doc_id, text in ...},
    relevant_docs={qid: {doc_id, ...} for qid in ...},
    name="my-sparse-ir",
    ndcg_at_k=[10],
    mrr_at_k=[10],
    accuracy_at_k=[1, 5, 10],
    map_at_k=[100],
    batch_size=32,
)

Output keys: eval_{name}_dot_ndcg@10, eval_{name}_dot_mrr@10, etc. Also reports active-dims.

Heavy for large corpora. Use SparseNanoBEIREvaluator during training. Reserve full IR for post-training.

Hybrid retrieval

ReciprocalRankFusionEvaluator

Measures the performance of combining your sparse encoder with BM25 (or any other retriever) via reciprocal-rank fusion. Useful when shipping a hybrid system is the actual deployment target.

Other sparse evaluators

SparseEmbeddingSimilarityEvaluator

STS-style. Computes Pearson/Spearman between sparse vector similarities and gold labels. Uses dot product by default.

SparseBinaryClassificationEvaluator

For labeled pair classification with sparse embeddings.

SparseTripletEvaluator

For (anchor, positive, negative) triplets. Reports fraction where the positive is closer than the negative (by dot product).

SparseRerankingEvaluator

For custom re-ranking with sparse embeddings. Same semantics as the dense RerankingEvaluator.

SparseMSEEvaluator

For distillation setups. Compares sparse student embeddings against teacher outputs.

SparseTranslationEvaluator

For cross-lingual / make_multilingual-style alignment checking with sparse embeddings.

Writing metric_for_best_model

Pattern: f"eval_{evaluator.primary_metric}". Inspect after construction: print(evaluator.primary_metric). Common values:

  • eval_NanoBEIR_mean_dot_ndcg@10: SparseNanoBEIREvaluator default
  • eval_{name}_dot_ndcg@10: SparseInformationRetrievalEvaluator
  • eval_{name}_spearman_dot: SparseEmbeddingSimilarityEvaluator

Gotchas

  • Always run evaluator(model) once before training. This confirms the pipeline works (a fill-mask base scores ~0 on retrieval until trained).
  • Sparse evaluators default to dot product. Cosine on sparse vectors isn't meaningful.
  • Don't compare dense and sparse metrics directly: different scales (cosine ∈ [-1, 1] vs. dot ∈ [0, ∞)).
  • Always check query_active_dims / document_active_dims: thousands of active dims per doc means the FLOPS regularizer is mistuned, even if nDCG looks OK.