AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

agent-designer

Use when designing a multi-agent system (supervisor, swarm, pipeline), writing agent tool schemas, or reading agent run logs for cost, latency or failure bottlenecks. Not for Claude Code workflow files (workflow-builder).

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--agent-designer at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 5

references/evaluation_methodology.md

Multi-Agent System Evaluation Methodology

Overview

This document provides a comprehensive methodology for evaluating multi-agent systems across multiple dimensions including performance, reliability, cost-effectiveness, and user satisfaction. The methodology is designed to provide actionable insights for system optimization.

Evaluation Framework

Evaluation Dimensions

1. Task Performance

  • Success Rate: Percentage of tasks completed successfully
  • Completion Time: Time from task initiation to completion
  • Quality Metrics: Accuracy, relevance, completeness of results
  • Partial Success: Progress made on incomplete tasks

2. System Reliability

  • Availability: System uptime and accessibility
  • Error Rates: Frequency and types of errors
  • Recovery Time: Time to recover from failures
  • Fault Tolerance: System behavior under component failures

3. Cost Efficiency

  • Resource Utilization: CPU, memory, network, storage usage
  • Token Consumption: LLM API usage and costs
  • Operational Costs: Infrastructure and maintenance costs
  • Cost per Task: Economic efficiency per completed task

4. User Experience

  • Response Time: User-perceived latency
  • User Satisfaction: Qualitative feedback scores
  • Usability: Ease of system interaction
  • Predictability: Consistency of system behavior

5. Scalability

  • Load Handling: Performance under increasing load
  • Resource Scaling: Ability to scale resources dynamically
  • Concurrency: Handling multiple simultaneous requests
  • Degradation Patterns: Behavior at capacity limits

6. Security

  • Access Control: Authentication and authorization effectiveness
  • Data Protection: Privacy and confidentiality measures
  • Audit Trail: Logging and monitoring completeness
  • Vulnerability Assessment: Security weakness identification

Metrics Collection

Core Metrics

Performance Metrics

{
  "task_metrics": {
    "task_id": "string",
    "agent_id": "string", 
    "task_type": "string",
    "start_time": "ISO 8601 timestamp",
    "end_time": "ISO 8601 timestamp",
    "duration_ms": "integer",
    "status": "success|failure|partial|timeout",
    "quality_score": "float 0-1",
    "steps_completed": "integer",
    "total_steps": "integer"
  }
}

Resource Metrics

{
  "resource_metrics": {
    "timestamp": "ISO 8601 timestamp",
    "agent_id": "string",
    "cpu_usage_percent": "float",
    "memory_usage_mb": "integer",
    "network_bytes_sent": "integer",
    "network_bytes_received": "integer",
    "tokens_consumed": "integer",
    "api_calls_made": "integer"
  }
}

Error Metrics

{
  "error_metrics": {
    "timestamp": "ISO 8601 timestamp",
    "error_type": "string",
    "error_code": "string",
    "error_message": "string",
    "agent_id": "string",
    "task_id": "string",
    "severity": "critical|high|medium|low",
    "recovery_action": "string",
    "resolved": "boolean"
  }
}

Advanced Metrics

Agent Collaboration Metrics

{
  "collaboration_metrics": {
    "timestamp": "ISO 8601 timestamp",
    "initiating_agent": "string",
    "target_agent": "string",
    "interaction_type": "request|response|broadcast|delegate",
    "latency_ms": "integer",
    "success": "boolean",
    "payload_size_bytes": "integer",
    "context_shared": "boolean"
  }
}

Tool Usage Metrics

{
  "tool_metrics": {
    "timestamp": "ISO 8601 timestamp",
    "agent_id": "string",
    "tool_name": "string",
    "invocation_duration_ms": "integer",
    "success": "boolean",
    "error_type": "string|null",
    "input_size_bytes": "integer",
    "output_size_bytes": "integer",
    "cached_result": "boolean"
  }
}

Evaluation Methods

1. Synthetic Benchmarks

Task Complexity Levels

  • Level 1 (Simple): Single-agent, single-tool tasks
  • Level 2 (Moderate): Multi-tool tasks requiring coordination
  • Level 3 (Complex): Multi-agent collaborative tasks
  • Level 4 (Advanced): Long-running, multi-stage workflows
  • Level 5 (Expert): Adaptive tasks requiring learning

Benchmark Task Categories

benchmark_categories:
  information_retrieval:
    - simple_web_search
    - multi_source_research
    - fact_verification
    - comparative_analysis
  
  content_generation:
    - text_summarization
    - creative_writing
    - technical_documentation
    - multilingual_translation
  
  data_processing:
    - data_cleaning
    - statistical_analysis
    - visualization_creation
    - report_generation
  
  problem_solving:
    - algorithm_development
    - optimization_tasks
    - troubleshooting
    - decision_support
  
  workflow_automation:
    - multi_step_processes
    - conditional_workflows
    - exception_handling
    - resource_coordination

Benchmark Execution

def run_benchmark_suite(agents, benchmark_tasks):
    results = {}
    
    for category, tasks in benchmark_tasks.items():
        category_results = []
        
        for task in tasks:
            task_result = execute_benchmark_task(
                agents=agents,
                task=task,
                timeout=task.max_duration,
                repetitions=task.repetitions
            )
            category_results.append(task_result)
        
        results[category] = analyze_category_results(category_results)
    
    return generate_benchmark_report(results)

2. A/B Testing

Test Design

ab_test_design:
  hypothesis: "New agent architecture improves task success rate"
  success_metrics:
    primary: "task_success_rate"
    secondary: ["response_time", "cost_per_task", "user_satisfaction"]
  
  test_configuration:
    control_group: "current_architecture"
    treatment_group: "new_architecture" 
    traffic_split: 50/50
    duration_days: 14
    minimum_sample_size: 1000
  
  statistical_parameters:
    confidence_level: 0.95
    minimum_detectable_effect: 0.05
    statistical_power: 0.8

Analysis Framework

def analyze_ab_test(control_data, treatment_data, metrics):
    results = {}
    
    for metric in metrics:
        control_values = extract_metric_values(control_data, metric)
        treatment_values = extract_metric_values(treatment_data, metric)
        
        # Statistical significance test
        stat_result = perform_statistical_test(
            control_values, 
            treatment_values,
            test_type=determine_test_type(metric)
        )
        
        # Effect size calculation
        effect_size = calculate_effect_size(
            control_values, 
            treatment_values
        )
        
        results[metric] = {
            "control_mean": np.mean(control_values),
            "treatment_mean": np.mean(treatment_values),
            "p_value": stat_result.p_value,
            "confidence_interval": stat_result.confidence_interval,
            "effect_size": effect_size,
            "practical_significance": assess_practical_significance(
                effect_size, metric
            )
        }
    
    return results

3. Load Testing

Load Test Scenarios

load_test_scenarios:
  baseline_load:
    concurrent_users: 10
    ramp_up_time: "5 minutes"
    duration: "30 minutes"
    
  normal_load:
    concurrent_users: 100
    ramp_up_time: "10 minutes"
    duration: "1 hour"
    
  peak_load:
    concurrent_users: 500
    ramp_up_time: "15 minutes"
    duration: "2 hours"
    
  stress_test:
    concurrent_users: 1000
    ramp_up_time: "20 minutes"
    duration: "1 hour"
    
  spike_test:
    phases:
      - users: 100, duration: "10 minutes"
      - users: 1000, duration: "5 minutes"  # Spike
      - users: 100, duration: "15 minutes"

Performance Thresholds

performance_thresholds:
  response_time:
    p50: 2000ms    # 50th percentile
    p90: 5000ms    # 90th percentile  
    p95: 8000ms    # 95th percentile
    p99: 15000ms   # 99th percentile
  
  throughput:
    minimum: 10    # requests per second
    target: 50     # requests per second
    
  error_rate:
    maximum: 5%    # percentage of failed requests
    
  resource_utilization:
    cpu_max: 80%
    memory_max: 85%
    network_max: 70%

4. Real-World Evaluation

Production Monitoring

production_metrics:
  business_metrics:
    - task_completion_rate
    - user_retention_rate
    - feature_adoption_rate
    - time_to_value
  
  technical_metrics:
    - system_availability
    - mean_time_to_recovery
    - resource_efficiency
    - cost_per_transaction
  
  user_experience_metrics:
    - net_promoter_score
    - user_satisfaction_rating
    - task_abandonment_rate
    - help_desk_ticket_volume

Continuous Evaluation Pipeline

class ContinuousEvaluationPipeline:
    def __init__(self, metrics_collector, analyzer, alerting):
        self.metrics_collector = metrics_collector
        self.analyzer = analyzer
        self.alerting = alerting
    
    def run_evaluation_cycle(self):
        # Collect recent metrics
        metrics = self.metrics_collector.collect_recent_metrics(
            time_window="1 hour"
        )
        
        # Analyze performance
        analysis = self.analyzer.analyze_metrics(metrics)
        
        # Check for anomalies
        anomalies = self.analyzer.detect_anomalies(
            metrics, 
            baseline_window="24 hours"
        )
        
        # Generate alerts if needed
        if anomalies:
            self.alerting.send_alerts(anomalies)
        
        # Update performance baselines
        self.analyzer.update_baselines(metrics)
        
        return analysis

Analysis Techniques

1. Statistical Analysis

Descriptive Statistics

def calculate_descriptive_stats(data):
    return {
        "count": len(data),
        "mean": np.mean(data),
        "median": np.median(data),
        "std_dev": np.std(data),
        "min": np.min(data),
        "max": np.max(data),
        "percentiles": {
            "p25": np.percentile(data, 25),
            "p50": np.percentile(data, 50),
            "p75": np.percentile(data, 75),
            "p90": np.percentile(data, 90),
            "p95": np.percentile(data, 95),
            "p99": np.percentile(data, 99)
        }
    }

Correlation Analysis

def analyze_metric_correlations(metrics_df):
    correlation_matrix = metrics_df.corr()
    
    # Identify strong correlations
    strong_correlations = []
    for i in range(len(correlation_matrix.columns)):
        for j in range(i + 1, len(correlation_matrix.columns)):
            corr_value = correlation_matrix.iloc[i, j]
            if abs(corr_value) > 0.7:  # Strong correlation threshold
                strong_correlations.append({
                    "metric1": correlation_matrix.columns[i],
                    "metric2": correlation_matrix.columns[j],
                    "correlation": corr_value,
                    "strength": "strong" if abs(corr_value) > 0.8 else "moderate"
                })
    
    return strong_correlations

2. Trend Analysis

Time Series Analysis

def analyze_performance_trends(time_series_data, metric):
    # Decompose time series
    decomposition = seasonal_decompose(
        time_series_data[metric], 
        model='additive', 
        period=24  # Daily seasonality
    )
    
    # Trend detection
    trend_slope = calculate_trend_slope(decomposition.trend)
    
    # Seasonality detection
    seasonal_patterns = identify_seasonal_patterns(decomposition.seasonal)
    
    # Anomaly detection
    anomalies = detect_anomalies_isolation_forest(time_series_data[metric])
    
    return {
        "trend_direction": "increasing" if trend_slope > 0 else "decreasing" if trend_slope < 0 else "stable",
        "trend_strength": abs(trend_slope),
        "seasonal_patterns": seasonal_patterns,
        "anomalies": anomalies,
        "forecast": generate_forecast(time_series_data[metric], periods=24)
    }

3. Comparative Analysis

Multi-System Comparison

def compare_systems(system_metrics_dict):
    comparison_results = {}
    
    metrics_to_compare = [
        "success_rate", "average_response_time", 
        "cost_per_task", "error_rate"
    ]
    
    for metric in metrics_to_compare:
        metric_values = {
            system: metrics[metric] 
            for system, metrics in system_metrics_dict.items()
        }
        
        # Rank systems by metric
        ranked_systems = sorted(
            metric_values.items(), 
            key=lambda x: x[1],
            reverse=(metric in ["success_rate"])  # Higher is better for some metrics
        )
        
        # Calculate relative performance
        best_value = ranked_systems[0][1]
        relative_performance = {
            system: value / best_value if best_value > 0 else 0
            for system, value in metric_values.items()
        }
        
        comparison_results[metric] = {
            "rankings": ranked_systems,
            "relative_performance": relative_performance,
            "best_system": ranked_systems[0][0]
        }
    
    return comparison_results

Quality Assurance

1. Data Quality Validation

Data Completeness Checks

def validate_data_completeness(metrics_data):
    completeness_report = {}
    
    required_fields = [
        "timestamp", "task_id", "agent_id", 
        "duration_ms", "status", "success"
    ]
    
    for field in required_fields:
        missing_count = metrics_data[field].isnull().sum()
        total_count = len(metrics_data)
        completeness_percentage = (total_count - missing_count) / total_count * 100
        
        completeness_report[field] = {
            "completeness_percentage": completeness_percentage,
            "missing_count": missing_count,
            "status": "pass" if completeness_percentage >= 95 else "fail"
        }
    
    return completeness_report

Data Consistency Checks

def validate_data_consistency(metrics_data):
    consistency_issues = []
    
    # Check timestamp ordering
    if not metrics_data['timestamp'].is_monotonic_increasing:
        consistency_issues.append("Timestamps are not in chronological order")
    
    # Check duration consistency
    duration_negative = (metrics_data['duration_ms'] < 0).sum()
    if duration_negative > 0:
        consistency_issues.append(f"Found {duration_negative} negative durations")
    
    # Check status-success consistency
    success_status_mismatch = (
        (metrics_data['status'] == 'success') != metrics_data['success']
    ).sum()
    if success_status_mismatch > 0:
        consistency_issues.append(f"Found {success_status_mismatch} status-success mismatches")
    
    return consistency_issues

2. Evaluation Reliability

Reproducibility Framework

class ReproducibleEvaluation:
    def __init__(self, config):
        self.config = config
        self.random_seed = config.get('random_seed', 42)
        
    def setup_environment(self):
        # Set random seeds
        random.seed(self.random_seed)
        np.random.seed(self.random_seed)
        
        # Configure logging
        self.setup_evaluation_logging()
        
        # Snapshot system state
        self.snapshot_system_state()
    
    def run_evaluation(self, test_suite):
        self.setup_environment()
        
        # Execute evaluation with full logging
        results = self.execute_test_suite(test_suite)
        
        # Verify reproducibility
        self.verify_reproducibility(results)
        
        return results

Reporting Framework

1. Executive Summary Report

Key Performance Indicators

kpi_dashboard:
  overall_health_score: 85/100
  
  performance:
    task_success_rate: 94.2%
    average_response_time: 2.3s
    p95_response_time: 8.1s
  
  reliability:
    system_uptime: 99.8%
    error_rate: 2.1%
    mean_recovery_time: 45s
  
  cost_efficiency:
    cost_per_task: $0.05
    token_utilization: 78%
    resource_efficiency: 82%
  
  user_satisfaction:
    net_promoter_score: 42
    task_completion_rate: 89%
    user_retention_rate: 76%

Trend Indicators

trend_analysis:
  performance_trends:
    success_rate: "↗ +2.3% vs last month"
    response_time: "↘ -15% vs last month"
    error_rate: "→ stable vs last month"
  
  cost_trends:
    total_cost: "↗ +8% vs last month"
    cost_per_task: "↘ -5% vs last month"
    efficiency: "↗ +12% vs last month"

2. Technical Deep-Dive Report

Performance Analysis

## Performance Analysis

### Task Success Patterns
- **Overall Success Rate**: 94.2% (target: 95%)
- **By Task Type**:
  - Simple tasks: 98.1% success
  - Complex tasks: 87.4% success
  - Multi-agent tasks: 91.2% success

### Response Time Distribution
- **Median**: 1.8 seconds
- **95th Percentile**: 8.1 seconds
- **Peak Hours Impact**: +35% slower during 9-11 AM

### Error Analysis
- **Top Error Types**:
  1. Timeout errors (34% of failures)
  2. Rate limit exceeded (28% of failures)
  3. Invalid input (19% of failures)

Resource Utilization

## Resource Utilization

### Compute Resources
- **CPU Utilization**: 45% average, 78% peak
- **Memory Usage**: 6.2GB average, 12.1GB peak
- **Network I/O**: 125 MB/s average

### API Usage
- **Token Consumption**: 2.4M tokens/day
- **Cost Breakdown**:
  - Large-tier model: 68% of token costs
  - Small-tier model: 28% of token costs
  - Other models: 4% of token costs

3. Actionable Recommendations

Performance Optimization

recommendations:
  high_priority:
    - title: "Reduce timeout error rate"
      impact: "Could improve success rate by 2.1%"
      effort: "Medium"
      timeline: "2 weeks"
      
    - title: "Optimize complex task handling"
      impact: "Could improve complex task success by 5%"
      effort: "High"
      timeline: "4 weeks"
  
  medium_priority:
    - title: "Implement intelligent caching"
      impact: "Could reduce costs by 15%"
      effort: "Medium"
      timeline: "3 weeks"

Continuous Improvement Process

1. Evaluation Cadence

Regular Evaluation Schedule

evaluation_schedule:
  real_time:
    frequency: "continuous"
    metrics: ["error_rate", "response_time", "system_health"]
    
  hourly:
    frequency: "every hour"
    metrics: ["throughput", "resource_utilization", "user_activity"]
    
  daily:
    frequency: "daily at 2 AM UTC"
    metrics: ["success_rates", "cost_analysis", "user_satisfaction"]
    
  weekly:
    frequency: "every Sunday"
    metrics: ["trend_analysis", "comparative_analysis", "capacity_planning"]
    
  monthly:
    frequency: "first Monday of month"
    metrics: ["comprehensive_evaluation", "benchmark_testing", "strategic_review"]

2. Performance Baseline Management

Baseline Update Process

def update_performance_baselines(current_metrics, historical_baselines):
    updated_baselines = {}
    
    for metric, current_value in current_metrics.items():
        historical_values = historical_baselines.get(metric, [])
        historical_values.append(current_value)
        
        # Keep rolling window of last 30 days
        historical_values = historical_values[-30:]
        
        # Calculate new baseline
        baseline = {
            "mean": np.mean(historical_values),
            "std": np.std(historical_values),
            "p95": np.percentile(historical_values, 95),
            "trend": calculate_trend(historical_values)
        }
        
        updated_baselines[metric] = baseline
    
    return updated_baselines

Conclusion

Effective evaluation of multi-agent systems requires a comprehensive, multi-dimensional approach that combines quantitative metrics with qualitative assessments. The methodology should be:

  1. Comprehensive: Cover all aspects of system performance
  2. Continuous: Provide ongoing monitoring and evaluation
  3. Actionable: Generate specific, implementable recommendations
  4. Adaptable: Evolve with system changes and requirements
  5. Reliable: Produce consistent, reproducible results

Regular evaluation using this methodology will ensure multi-agent systems continue to meet user needs while optimizing for cost, performance, and reliability.