AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

airunway-aks-setup

Set up AI Runway on AKS — from bare cluster to running model. Covers cluster verification, controller install, GPU assessment, provider setup, and first deployment. WHEN: "setup AI Runway", "onboard AKS cluster", "install AI Runway", "airunway setup", "deploy model to AKS", "GPU inference on AKS", "KAITO setup on AKS", "run LLM on AKS", "vLLM on AKS", "set up model serving on AKS", "AI Runway controller".

Original source
microsoft
License
MIT
Pinned revision
3f5b461b2314e01c400564051e92469114c5b68f

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read microsoft--airunway-aks-setup at revision 3f5b461b2314e01c400564051e92469114c5b68f, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 12

references/model-sizing.md

Model Sizing & Starter Recommendations

VRAM Sizing Guide

Estimate how much VRAM a model requires. Values are weights only — add 20–30% overhead for KV cache and activations during inference.

| Parameters | float16 / bfloat16 | int8 | int4 / GGUF Q4 | |-----------|-------------------|------|----------------| | 1B | ~2 GB | ~1 GB | ~0.5 GB | | 3B | ~6 GB | ~3 GB | ~1.5 GB | | 7–8B | ~14–16 GB | ~7–8 GB | ~3.5–4 GB | | 13B | ~26 GB | ~13 GB | ~6.5 GB | | 34B | ~68 GB | ~34 GB | ~17 GB | | 70B | ~140 GB | ~70 GB | ~35 GB |

Example: 4× A100 80 GB = 320 GB total. Llama-3.1-70B at bfloat16 ≈ 168 GB with overhead — fits across 4 GPUs with tensor parallelism.

Starter Model Recommendations

| Cluster Capacity | Model | Provider | Notes | |-----------------|-------|----------|-------| | CPU-only | google/gemma-3-1b-it-qat-q8_0-gguf | KAITO (llama.cpp) | GGUF Q8; runs on CPU | | 1× T4 (16 GB) | microsoft/Phi-3-mini-4k-instruct | KAITO (vLLM) | ~8 GB float16; fits with headroom | | 1× A10G/L4 (24 GB) | meta-llama/Llama-3.1-8B-Instruct | KAITO (vLLM) | ~16 GB bfloat16; gated — needs HF token | | 1× A100 40 GB | microsoft/Phi-3-medium-128k-instruct | KAITO (vLLM) | ~28 GB float16; non-gated; MIT license | | 1× A100 80 GB / H100 | meta-llama/Llama-3.1-8B-Instruct | KAITO (vLLM) | Oversized; upgrade to 70B if more GPUs available | | 4× A100 80 GB | meta-llama/Llama-3.1-70B-Instruct | KAITO (vLLM, TP) | ~168 GB; tensor parallelism; gated |

Gated Models

These models are gated on HuggingFace and require an access token:

  • meta-llama/Llama-3.1-8B-Instruct
  • meta-llama/Llama-3.1-70B-Instruct

Non-gated alternatives (no token required):

  • microsoft/Phi-3-mini-4k-instruct (MIT license)
  • microsoft/Phi-3-medium-128k-instruct (MIT license)
  • google/gemma-3-1b-it-qat-q8_0-gguf (Gemma license)