references/resource_semantics.md
Resource Semantics
Research and behavior cut-off: 2026-10-01. See
sources.md for the official documentation used.
Core rule: inventory is not entitlement
Never treat a host-wide count as a promise that the current process can use it. Interpret usable resources as the intersection of independently observed constraints:
- host inventory;
- process affinity or processor-group scope;
- cgroup/container constraints;
- scheduler allocation;
- accelerator visibility and device permissions; and
- application runtime compatibility.
Missing evidence means unknown, not unlimited. Scheduler variables can describe an allocation without proving that affinity or cgroup enforcement is enabled. Conversely, a cgroup or affinity mask can be stricter than the scheduler request.
CPU
Physical and logical CPUs
- A logical CPU is an operating-system scheduling unit. Simultaneous multithreading can expose multiple logical CPUs on one physical core.
- A physical-core count describes topology, not the number of independent workers the process may start.
psutil.cpu_count(logical=True), macOShw.logicalcpu, and POSIXSC_NPROCESSORS_ONLNprovide host/system inventory. They can exceed the CPUs usable by the process.os.cpu_count()is a fallback: Python 3.13+-X cpu_countorPYTHON_CPU_COUNToverrides both Python CPU-count APIs. The detector prefers independent inventory and keeps Python process count as a separate planning input.os.process_cpu_count()(Python 3.13+) is process-aware. On supported platforms, affinity APIs provide a more explicit process constraint.- On Windows systems with multiple processor groups, a system-wide logical count and one process/thread group's usable count can differ. Windows 11 and Server 2022 default affinity spans groups; do not assume every modern Windows process is capped at 64 logical CPUs.
The detector reports host logical and physical counts separately. It never derives physical cores from a logical count.
Affinity, cpusets, and quotas
- Process affinity limits the logical CPUs on which a process may execute.
- Linux
cpuset.cpus.effectivereports CPUs actually granted after parent constraints. The requestedcpuset.cpuscan differ. /proc/self/statusexposesCpus_allowed_list, but the detector prefers affinity APIs and cgroup effective cpusets.- cgroup v2
cpu.maxis$MAX $PERIOD.maxmeans no local bandwidth limit. A finite ratio is CPU-time capacity, possibly fractional; it is not a core topology count. - Parent cgroups also constrain children, so the detector takes the most
restrictive finite visible ancestor quota. Membership is resolved through
/proc/self/mountinfocgroup2 root/mountpoint fields. A namespace or delegated mount may hide stronger ancestors. Missing files are not evidence of unlimited capacity; cgroup v1 is outside the collector's implemented scope.
For a cpu.max ratio of 1.5, the snapshot reports
capacity_cores: 1.5 and a conservative CPU-bound worker ceiling of 1. A
workload may choose more threads for latency hiding, but it should expect
throttling and must not describe those threads as 1.5 physical cores.
Python worker pools
- Current Python
multiprocessing.PoolandProcessPoolExecutordefaults useos.process_cpu_count()when available. - Python 3.14 no longer uses
forkas the default start method on any platform. Code that depends on a particular start method must request it deliberately. - Process workers do not make each worker's native-library threads disappear. BLAS/OpenMP threads multiplied by process workers commonly oversubscribe an allocation.
- Windows
ProcessPoolExecutorhas a documented maximum of 61 workers.
Use the workload planner's worker and threads-per-worker values as conservative ceilings, then benchmark a real representative workload. Do not run a synthetic stress test merely to discover capacity.
Memory
Host and effective memory
psutil.virtual_memory().totaland.availabledescribe system-visible memory, not necessarily the process limit.- Linux
/proc/meminfoMemAvailableis the standard-library fallback for a host-availability estimate. - cgroup v2
memory.currentis current cgroup usage. memory.highis a throttle/reclaim pressure boundary. Exceeding it does not itself invoke the cgroup OOM killer, and the value can be breached.memory.maxis the hard cgroup limit. If usage cannot be reduced at that boundary, the cgroup OOM killer can run.- Parent memory limits are hierarchical. Shared ancestor usage can reduce what
remains for a child, so effective remaining memory is the minimum finite
limit - currentobserved through the ancestor chain. - Slurm memory variables describe requested/allocated memory, but strict enforcement depends on site configuration.
The detector keeps host total/available values, cgroup values, scheduler values, and the conservative effective values distinct. A point-in-time "available" value can change immediately and is not a reservation.
macOS unified memory
On Apple silicon, CPU and integrated GPU share unified memory. Do not add a
fictional GPU VRAM amount to system RAM. The snapshot marks the memory model as
unified_cpu_gpu and leaves dedicated GPU memory null. Metal framework support
and model/operator support still require application-specific checks.
Accelerators
Accelerator usability has separate layers:
- Hardware or management visibility — a management query returns a device.
- Allocation visibility — scheduler and named visibility variables permit a device or a subset.
- Device permission — the process/container can open the required device interfaces.
- Driver/runtime compatibility — driver, CUDA/ROCm/Metal runtime, and framework versions are compatible.
- Workload compatibility — the requested operation and data type are implemented on that backend.
nvidia-smi success confirms NVIDIA management visibility only. It does not
prove that CUDA libraries exist or are compatible. NVIDIA documents driver and
runtime compatibility as a separate requirement.
AMD SMI or ROCm SMI success likewise does not prove HIP/ROCm runtime usability.
New deployments should prefer amd-smi; rocm-smi is retained as a read-only
fallback. The fixed AMD query selects --asic --vram, avoiding unrelated
static fields. Current JSON nests VRAM size as {value, unit}; legacy ROCm SMI
uses a flat byte field. AMD SMI labels binary MiB values MB, confirmed in its
source conversion. The detector exposes this as reported_total_bytes and
leaves dedicated memory null because APU/shared-memory topology is unresolved.
On Linux, AMD recommends ROCR_VISIBLE_DEVICES; on Windows it recommends
HIP_VISIBLE_DEVICES.
An Apple integrated GPU is a Metal candidate, not a CUDA GPU. AMD SMI records are ROCm candidates; macOS display records are Metal candidates regardless of hardware vendor. Neural engines, TPUs, FPGAs, and other accelerators must also remain distinct from CUDA devices if another inventory source adds them.
The detector reads only these accelerator variable names and redacts their values:
NVIDIA_VISIBLE_DEVICESCUDA_VISIBLE_DEVICESROCR_VISIBLE_DEVICESHIP_VISIBLE_DEVICES
Counts derived from those values are only upper bounds. Environment variables are not a security boundary and can be reset by an application; device namespace/cgroup controls are stronger isolation.
Management counts describe records in a particular query. A physical NVIDIA GPU can expose several MIG devices, and AMD partition modes can also change runtime enumeration. Use the workload framework in its actual allocation to count usable devices; do not infer it from these candidate counts.
Slurm and other schedulers
The detector allowlists named Slurm variables and never dumps the environment. It records field names, parsed bounded counts, and memory quantities, but not job IDs, GPU UUIDs, node names, submit hosts, or paths.
Important scopes:
SLURM_CPUS_PER_TASK: requested CPUs per task; suitable as a per-process upper bound for one task process.SLURM_CPUS_ON_NODE: CPUs allocated to the current batch step on the node; it can be shared among tasks.SLURM_JOB_CPUS_PER_NODE: a per-node allocation list, not a process count.SLURM_TASKS_PER_NODE: compressed counts such as1(x2),4; resolve the current entry withSLURM_NODEID. Without that index, only a uniform list is interpretable. Never substitute the first node's task count. The node ID itself is not emitted. RequestedSLURM_NTASKS_PER_NODEis a fallback only when the allocation list is absent. Step overrides and heterogeneous job components still require application/site-specific interpretation.SLURM_MEM_PER_CPU: memory per allocated CPU. It becomes a per-task bound only when CPUs per task is known.SLURM_MEM_PER_NODE: shared per-node memory upper bound.SLURM_GPUS_PER_TASK: requested GPUs per task.SLURM_GPUS_ON_NODE: GPUs allocated to the batch step on the node.
Slurm CPU confinement requires site configuration such as task affinity or
task/cgroup with core constraints. Memory requests are not strictly enforced
unless the site enables an enforcement mechanism. Use affinity and cgroup
observations as enforcement evidence; do not trust visible node inventory.
For other schedulers, use their documented allocation API or variables and the same rule: allocation metadata and kernel enforcement are different facts. Do not guess from generic environment names.
Containers and OCI
Docker containers have no CPU or memory limit by default. When configured, Docker maps CPU and memory flags to cgroup controls. OCI runtime configuration also defines CPU, memory, and device constraints.
Inside a container:
- host inventory may remain visible;
- a CPU quota can be smaller than the visible CPU set;
- a cpuset can be smaller than the quota's apparent capacity;
- cgroup memory can be smaller than host RAM;
- GPU management tools can see a different set from the application runtime; and
- mounted-volume capacity can differ from writable quota.
Always report observed cgroup controls and uncertainty. A container marker without a finite cgroup value does not imply a finite limit.
Disk
Capacity, free blocks, user-writable blocks, path permission, filesystem quota, and actual ability to complete a write are different:
- capacity is the filesystem's total size;
- POSIX
f_bfreeincludes blocks reserved from an unprivileged user; shutil.disk_usage().freeusesf_bavailon POSIX, so the detector readsstatvfsto keep filesystem free and user-available values separate;- POSIX
f_bavailestimates blocks available to the current user; os.access(..., os.W_OK)is a non-writing permission check, not proof that a future write will succeed;- project/user quotas and remote storage policies can be stricter than block counts.
The detector does not create a probe file. It redacts the absolute working path and labels the scope as the working filesystem.