AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

train-sentence-transformers

Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder, dense or static embedding model for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker, pair scoring for two-stage retrieval / pair classification), `SparseEncoder` (SPLADE, sparse embedding model for learned-sparse retrieval), and `MultiVectorEncoder` (ColBERT / late-interaction, per-token embeddings scored with MaxSim). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.

Original source
huggingface
License
Apache-2.0
Pinned revision
25910b3d0ba31a2ba102f901cdc9e008c278a69c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read huggingface--train-sentence-transformers at revision 25910b3d0ba31a2ba102f901cdc9e008c278a69c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 17

references/losses_sparse_encoder.md

Sparse-Encoder Losses (SPLADE)

All losses live in sentence_transformers.sparse_encoder.losses.

This reference targets the SPLADE architecture (Transformer + SpladePooling). The sparse-encoder package also exports CSRLoss and CSRReconstructionLoss for the CSR architecture (Transformer + Pooling + SparseAutoEncoder). Those are out of scope here. See the sbert.net docs if you're training a CSR model.

Choosing a loss means (a) pick a base loss (contrastive, regression, distillation) and (b) wrap it in SpladeLoss to add FLOPS regularization.

Top-line decision table

| You have | Use | |---|---| | (anchor, positive) or triplet, SPLADE architecture | SpladeLoss(loss=SparseMultipleNegativesRankingLoss(model), ...) | | Same, want effective batch size of 256+ | CachedSpladeLoss(...) | | (text1, text2, score) labeled pairs | SparseCoSENTLoss or SparseCosineSimilarityLoss | | Distillation from cross-encoder teacher | SparseMarginMSELoss | | Listwise distillation | SparseDistillKLDivLoss | | Explicit triplet | SparseTripletLoss |

The core wrapper: SpladeLoss

SpladeLoss adds FLOPS regularization on top of another sparse loss. FLOPS regularization penalizes non-zero activations, keeping embeddings genuinely sparse.

loss = SpladeLoss(
    model=model,
    loss=SparseMultipleNegativesRankingLoss(model=model),
    query_regularizer_weight=5e-5,
    document_regularizer_weight=3e-5,
)
  • query_regularizer_weight: how much to penalize non-zero terms in query embeddings.
  • document_regularizer_weight: same for documents.
  • Typical range: 1e-5 to 1e-4. Higher = sparser embeddings, lower recall. Lower = denser, possibly better recall.
  • SparseEncoderTrainer automatically registers a SpladeRegularizerWeightSchedulerCallback whenever the loss is a SpladeLoss. The callback ramps the weights from 0 up to the target over the first ~33% of training. The default shape is SchedulerType.QUADRATIC (not linear). The ramp length and shape are configured on the callback (SpladeRegularizerWeightSchedulerCallback(loss=..., warmup_ratio=..., scheduler_type=...)), not on SpladeLoss. To override, instantiate the callback yourself and pass it via callbacks=[...]. This ramp is important. Starting with full regularization from step 0 kills learning.

Use CachedSpladeLoss for the GradCache variant.

Contrastive losses (no labels)

SparseMultipleNegativesRankingLoss

Sparse analog of bi-encoder MNRL. In-batch contrastive.

inner = SparseMultipleNegativesRankingLoss(model=model)
loss = SpladeLoss(model=model, loss=inner, query_regularizer_weight=5e-5, document_regularizer_weight=3e-5)
  • Always wrap in SpladeLoss for SPLADE architectures.
  • Set batch_sampler=BatchSamplers.NO_DUPLICATES on training args.

SparseTripletLoss

Classic triplet margin loss on explicit (anchor, positive, negative).

Labeled regression losses

SparseCoSENTLoss

Pairwise ranking loss for (text1, text2, score). Mirrors bi-encoder CoSENTLoss.

SparseCosineSimilarityLoss

MSE on cosine similarity. Simpler, usually worse than CoSENT.

SparseAnglELoss

Angle-based loss in complex space. Alternative to CoSENT.

Distillation losses

SparseMSELoss

Embedding MSE. Student sparse embedding should match teacher embedding.

  • Data: (text, teacher_embedding).
  • Teacher can be a dense bi-encoder or another sparse model.

SparseMarginMSELoss

Margin MSE from a cross-encoder teacher.

  • Data: (query, positive, negative, score_diff) where score_diff = teacher_score(query, positive) - teacher_score(query, negative).
  • Typical recipe for training SPLADE from cross-encoder labels (ms-marco distillation).
  • Wrap in SpladeLoss(model, loss=SparseMarginMSELoss(model), ...) for SPLADE.

SparseDistillKLDivLoss

Listwise KL-div distillation. Student's softmax distribution over candidates should match teacher's.

Independent regularizer

FlopsLoss

Standalone FLOPS regularizer. Usually you use this via SpladeLoss, not directly.

For regularizer-weight tuning and dense-output recovery, see troubleshooting.md ("SPLADE embeddings are dense"). MLM-head requirement: base_model_selection.md (SPARSE section). Active-dim sparsity targets and how to monitor them: evaluators_sparse_encoder.md (Sparsity tracking).

Gotchas

  • SparseMultipleNegativesRankingLoss without SpladeLoss wrapping on a SPLADE model: no FLOPS regularization -> dense outputs defeating the purpose of SPLADE. Always wrap.
  • CachedSpladeLoss + gradient_checkpointing=True: crash. Pick one.
  • Starting training with full FLOPS regularization at step 0: the model outputs zero everywhere and gets stuck. The built-in scheduler avoids this. Don't override it unless you know why.
  • query_regularizer_weight == document_regularizer_weight: usually wrong. Queries should be sparser than documents (fewer terms per query). Since higher regularization drives more zeros, give the query weight the larger value. query_regularizer_weight=5e-5, document_regularizer_weight=3e-5 is a good starting ratio.