AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

hypogenic

Plans and audits use of ChicagoHAI HypoGeniC/HypoRefine for LLM-assisted hypothesis generation from labeled text datasets. Use for the `hypogenic` package, its task configs, hypothesis banks, or HypoBench datasets—not for manual hypothesis formulation or scientific validation.

Original source
K-Dense-AI
License
MIT
Pinned revision
5f2bc0341354fd5590aef76a533ada1e1e5b4e62

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read k-dense-ai--hypogenic at revision 5f2bc0341354fd5590aef76a533ada1e1e5b4e62, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 7

references/evaluation.md

Local output inspection and evaluation

Reviewed 2026-10-01 against the unchanged 0.3.5 output serializer. All bundled operations in this reference are deterministic and model-free. They do not establish scientific truth.

Hypothesis-bank inspection

The pinned upstream serializer writes a JSON object whose keys are hypothesis text and whose values contain:

  • hypothesis: same text as the object key;
  • acc: finite number in [0, 1];
  • reward: finite ranking value;
  • num_visits: non-negative integer;
  • correct_examples: [row_index, label] pairs;
  • optionally num_select in some literature/union outputs.

Inspect without echoing text:

python3 scripts/inspect_outputs.py hypotheses \
  --input outputs/hypotheses.json \
  --root .

The report includes file SHA-256, bank size, normalized duplicate counts, length/statistic ranges, and a bounded sample of hypothesis SHA-256 values. Candidate strings are never printed or interpreted, including duplicate-key error messages.

Stored acc and reward are algorithm state from the generation/update workflow. They are not an independently reproduced evaluation and should not be described as p-values, confidence intervals, causal effects, or scientific validation.

Strict saved-result schema

The upstream inference CLI logs metrics but does not define a durable prediction artifact. This skill therefore uses a small local interchange schema:

{
  "schema_version": "1.0",
  "dataset_manifest_sha256": "<64 lowercase hex>",
  "hypothesis_bank_sha256": "<64 lowercase hex>",
  "split": "test",
  "records": [
    {
      "id": "stable-nonsecret-id",
      "label": "class-a",
      "prediction": "class-a"
    }
  ]
}

prediction may be null for an abstention or extraction failure. IDs must be unique; duplicate-ID errors identify the record index without printing the ID. Do not store prompts, chain-of-thought, provider responses, credentials, or raw sensitive features in this file. Replace the placeholder hashes in assets/result.example.json with hashes of the exact reviewed artifacts.

Inspect structure:

python3 scripts/inspect_outputs.py results \
  --input results/test_predictions.json \
  --root .

The report redacts IDs, labels, and predictions; category values are represented by short SHA-256 fingerprints.

Evaluation plan

Freeze a plan before looking at test metrics:

python3 scripts/evaluate_local.py plan \
  --config reviewed_run_config.json \
  --manifest dataset_manifest.json \
  --root .

The plan records:

  • immutable data source revision and manifest hash;
  • provider/model/destination;
  • train, validation, and test roles;
  • row and hypothesis caps;
  • planned metrics;
  • required provenance and interpretation limits.

It does not read dataset rows or invoke a model.

Metrics report

python3 scripts/evaluate_local.py report \
  --results results/test_predictions.json \
  --root . \
  --expected-split test

Implemented metrics:

  • coverage: non-null predictions divided by all records;
  • accuracy_all_records: exact matches divided by all records; null predictions count as incorrect;
  • accuracy_covered_records: exact matches among non-null predictions;
  • macro_f1_all_records: unweighted mean of per-label F1 over the union of observed true and non-null predicted labels;
  • redacted confusion matrix, with a separate <missing> prediction column.

All arithmetic uses the supplied saved strings exactly. There is no label normalization beyond schema validation. A custom label extractor must be frozen before test evaluation and its behavior documented.

Reporting checklist

Report at minimum:

  1. hypogenic version, source commit, and artifact hash;
  2. dataset repository, immutable revision, manifest SHA-256, and file hashes;
  3. task config hash, provider wrapper, exact model, and data destination;
  4. train/validation/test/OOD roles and sample counts;
  5. generation/inference settings, seeds, hypothesis count, and selection rule;
  6. token/request/cost caps and actual provider usage when available;
  7. result and hypothesis-bank hashes;
  8. coverage, accuracy, macro-F1, class support, and uncertainty across seeds;
  9. duplicate/leakage audit results;
  10. failures, exclusions, abstentions, retries, and deviations;
  11. provider retention/privacy terms reviewed for the run;
  12. a statement that generated hypotheses are candidates, not evidence.

The local tool does not calculate confidence intervals or significance tests. Choose those methods from a prespecified design that respects dependence, repeated seeds, multiple comparisons, class imbalance, and the data-generating process.

Scientific interpretation limits

Predictive benchmark performance can show that a textual heuristic was useful for a declared classification setup. It does not by itself establish:

  • that the proposed mechanism is true or causal;
  • that the pattern generalizes beyond the sampled population/time/domain;
  • novelty relative to all scientific literature;
  • robustness to paraphrases, near duplicates, annotation artifacts, or distribution shift;
  • clinical, policy, or safety validity;
  • absence of data leakage or provider/model memorization.

Validate promising candidates using independent data, domain-expert review, appropriate controls, alternative explanations, sensitivity analyses, and, where warranted, prospective or experimental tests.