AI Tools
LLM Autopilot Guard
LLM Autopilot Guard is an enterprise system for governing local or on-device AI workloads so they do not overheat hardware, drain batteries, or hurt productivity. It combines live telemetry, an LLM recommendation layer, deterministic guardrails, and observability to shift devices across runtime profiles safely. The pitch emphasizes enterprise rollout, policy controls, and ROI through lower hardware strain and safer offline AI operations.
The problem
Running AI workloads locally on enterprise devices is hard because hardware is wildly heterogeneous — different CPUs, GPUs, NPUs, thermals, and battery behavior — and the same policy behaves differently across device classes. Under token surges, latency spikes and devices can overheat or drain their batteries, degrading the very productivity the AI was meant to boost. For the operators governing these fleets, the pain concentrates in a trust gap: they need a clear, auditable "why" behind each autonomous tuning change. Setting thresholds that balance quality, latency, heat, and battery across diverse hardware is genuinely uncertain, and teams struggle to prove value without strong adaptive-versus-baseline evidence.
The solution
LLM AutoPilot Guard is an enterprise platform for governing local or on-device AI workloads so they don't overheat hardware, drain batteries, or hurt productivity. It acts as an autonomic governor layer above local runtimes, dynamically controlling model behavior from live battery, thermal, CPU, and memory signals while balancing latency, quality, energy, and thermal safety together. Every profile or knob change is explainable and visible in a decision timeline and dashboard, so IT and product teams can trust and audit it. A built-in adaptive-versus-baseline experiment runner produces measurable KPIs, and a conservative value-estimation feature translates measured deltas into modeled business ranges — always with explicit disclaimers separating measured evidence from assumptions.
How it works
Live telemetry feeds an LLM governance agent — Llama 3.2 3B Instruct, quantized and running locally via Ollama — which recommends a runtime profile (performance / balanced / efficiency / survival) and knob overrides in strict JSON with a rationale and confidence score. A deterministic safety governor validates or overrides every recommendation before it is applied, enforcing hard thermal and battery limits so the final decision never violates policy. The design is runtime- and model-agnostic via a structured agent contract, allowing alternatives like Qwen2.5 3B or Phi-3.5. Prompt iteration added hard guardrails, enum-restricted schemas, a minimal-change rule to prevent profile thrash, and confidence-gated fallback. Targets include ≥98% JSON schema validity, 100% policy compliance after enforcement, and adaptive-mode reductions of ≥20% in time above thermal threshold and 10–15% in battery drain per hour — with the same local LLM class serving as both the governed runtime and the decision agent.
Who it's for
The platform is primarily B2B, sold to enterprise IT and endpoint platform teams, with OEM partners as a secondary channel and a B2B2C expansion path through device partners. The primary operator personas are IT admins, endpoint ops teams, and AI platform engineers who set policy, monitor fleet health, and analyze results; security and compliance stakeholders are key influencers who approve governance boundaries. The downstream beneficiaries are employee end users — knowledge workers, field engineers, support agents — running local AI assistants who get better responsiveness, lower heat, and longer battery, often while on battery and moving between workloads.
Why it matters
The target segment — enterprise on-device AI runtime optimization and governance — is expected to grow at roughly 20–30% CAGR over three to five years, driven by AI PC adoption, privacy-first inference, and cost pressure to shift workloads from cloud to edge. Energy-awareness is becoming a procurement differentiator: solutions that prove better battery and thermal stability win trust. Monetization is B2B subscription and enterprise licensing priced per managed endpoint, with premium analytics and integration services. Success is measured in reduced endpoint complaints about AI slowness and overheating, improved perceived stability under stress, and faster pilot-to-expansion decisions grounded in clear adaptive-versus-baseline evidence — differentiation built on sustained real-world experience under stress rather than peak benchmark speed.
The workflow
The PRD
| PRODUCT FACULTY — AI PRODUCT REQUIREMENTS DOCUMENT (PRD) TEMPLATE Version 1.0 | ||||||
|---|---|---|---|---|---|---|
| Your Name: | Shruti Holla | |||||
| Your Product: | LLM AutoPilot Guard | |||||
| Your Industry: | Enterprise Software (Edge AI / On-Device AI Infrastructure & Device Management) | |||||
| Date: | May 8, 2026 | |||||
| 4D Method | AI PRD | Instructor Feedback | ||||
| Phase | Activity | Theme | Topic | Key Question(s) | Your Response Include external links to visuals/prototypes as required. | |
| DISCOVERY | Understand your market, business, product & user context | Business Value Map | Market Attractiveness | What industry is your business in? (ie Financial services, Healthcare, Education, etc)? | Enterprise Software / AI Infrastructure (Edge AI & Device Management) | Please leave this area blank. This space is for the Instructor to provide you with feedback. |
| What are the key challenges (headwinds) and opportunities (tailwinds) impacting growth in your industry? Who are the key competitors? | Headwinds (key challenges) Device heterogeneity: wildly different CPU/GPU/NPU, RAM, thermals, and battery behavior make consistent on-device AI hard. Cost/performance tradeoffs: users expect cloud-like quality with laptop/mobile power budgets. Operational complexity: runtime tuning, model selection, and policy management across fleets is still immature. Security/compliance burden: enterprise rollout needs strong controls, auditability, and safe model behavior. Measurement gap: many teams optimize benchmarks, not sustained real-world UX (heat, battery, p95 latency over time). Tailwinds (key opportunities) Privacy + sovereignty demand: push toward local inference to reduce data exposure and cloud dependency. NPU/edge silicon momentum: new hardware increasingly supports efficient local AI workloads. Enterprise AI adoption: copilots and AI assistants are becoming standard productivity tooling. Hybrid AI architectures: on-device for fast/private tasks, cloud for heavy tasks; orchestration layer demand is rising. Energy-awareness as differentiator: solutions that prove better battery/thermal stability can win procurement and user trust. Key competitors (by layer) Inference/runtime layer: ONNX Runtime, TensorRT/TensorRT-LLM, OpenVINO, llama.cpp, Ollama, Apple Core ML stack. Model serving/orchestration layer: vLLM (more datacenter-focused), LiteLLM-style gateways, vendor-specific local runners. Device/OS optimization ecosystems: Microsoft Windows AI stack, Apple on-device AI ecosystem, Qualcomm AI software stack, Intel AI PC tooling. Adjacent endpoint management players: OEM/device management platforms adding AI controls and policy features. | |||||
| What is the projected growth rate of your target market segment over the next 3-5 years? | Our target segment (enterprise on-device AI runtime optimization/governance) is expected to grow at approximately 20–30% CAGR over the next 3–5 years, driven by AI PC adoption, privacy-first inference needs, and cost pressure to shift workloads from cloud to edge. | |||||
| Business Model | What growth stage is your business currently in (e.g., startup, scale-up, mature)? | The business is in an early-stage startup phase: MVP built, problem-solution fit being validated via pilots/evaluations, and preparing for initial enterprise adoption | ||||
| How does your business make money? What do they sell? What is your primary revenue model (e.g., subscription, freemium, licensing, marketplace, transactional, etc?) | We monetize via a B2B subscription and enterprise licensing model, selling an on-device AI governance software platform priced per managed endpoint, with premium analytics/policy features and optional integration services. | |||||
| Who is your primary customer base (B2B, B2C, B2B2C)? | Primary: B2B (enterprise IT, OEMs, endpoint/device management teams) Secondary/expansion: B2B2C (through OEM/device partners to end users) | |||||
| Differentiators | What are the key differentiators for your company? | Autonomic governor layer (not just runtime): We dynamically control on-device LLM behavior using live battery, thermal, CPU, and memory signals. Multi-objective optimization: We balance latency, quality, energy, and thermal safety together instead of optimizing one metric in isolation. Explainable adaptation: Every profile/knob change is visible in timeline + dashboard, so IT/product teams can trust and audit decisions. Runtime-agnostic design: Works as a control plane above local runtimes/models (Llama/Ollama/ONNX-style stacks), reducing lock-in. Enterprise-ready evaluation loop: Built-in adaptive-vs-baseline experiment flow with measurable KPIs and conservative value-estimation outputs. Real-world reliability focus: Differentiation is sustained user experience under stress (token surges, battery constraints, thermal risk), not just peak benchmark speed. | ||||
| Feature Value Map (IMPORTANT NOTE: This section is only relevant if you are working on enhancing an existing product. It does not apply if you are developing a new product from 0 to 1.) | Customers | Who are the customers (ie buyers) of your product? | Primary buyers are enterprise IT and endpoint platform teams (B2B), with OEM partners as a secondary channel. | |||
| End Users | Who are the end-users of your product? Which users are the most revenue-generating / revenue-impacting for your company? What are their goals, roles, and context? | End-users of the product Primary operator users: Enterprise IT admins, endpoint ops teams, AI platform engineers (they configure policy, monitor fleet health, and optimize outcomes). Secondary end-users: Employees using local AI assistants/coplots on managed devices (they benefit from better responsiveness, lower heat, longer battery). Most revenue-generating / revenue-impacting users Most revenue-generating: Enterprise IT buyers + platform owners managing large device fleets (seat/device volume drives ARR). Most revenue-impacting in renewal: Day-to-day endpoint ops teams and AI platform teams (they determine measurable value: fewer complaints, better stability, policy control). Indirect but critical: Employee end-users; their satisfaction influences expansion/renewal decisions. Goals, roles, and context IT Admin / Endpoint Ops Manager Goal: Keep AI performant without overheating devices or draining battery. Role: Set governance policies, monitor KPIs, enforce guardrails. Context: Managing thousands of heterogeneous endpoints with strict reliability/security requirements. AI Platform / Productivity Engineer Goal: Deliver consistent local AI UX under real-world constraints. Role: Integrate runtimes/models, tune policies, analyze adaptive vs baseline results. Context: Balancing quality, latency, compute cost, and hardware variability. Security / Compliance Stakeholder Goal: Ensure local inference is controlled, auditable, and policy-compliant. Role: Approve governance boundaries and observability requirements. Context: Privacy-sensitive workflows and enterprise risk controls. Employee End User (knowledge worker, field engineer, support agent) Goal: Fast, reliable AI help throughout the day. Role: Uses AI assistant for drafting, analysis, troubleshooting. Context: Often on battery, moving between workloads, needs smooth experience not benchmark peaks. | ||||
| Current Products / Services | If you are a Product-led business: What are the core features of your product, and how do they address user needs? If you are a Service-led business: What are the key services you offer to customers? | We offer an on-device LLM governance platform that uses a local LLM agent to recommend runtime adaptations and a deterministic safety governor to enforce policy-safe final actions. Core features include: live telemetry-driven control (battery/thermal/CPU/memory), profile and knob adaptation, explainable decision timeline, adaptive-vs-baseline experiment runner, and conservative value estimation with clear disclaimers. | ||||
| User Value Map | Target Persona | Who is your AI product / feature for? (Internal users, external users, an influencer, a buyer, etc) | Primary users are external B2B operator personas: enterprise IT admins, endpoint operations teams, and AI platform engineers who manage local AI performance on device fleets. Primary buyer is enterprise IT leadership; secondary influencers include security/compliance stakeholders. Downstream beneficiaries are employee end-users using local AI assistants. | |||
| Journey Map (current-state) | What is the typical journey for your target persona when they are using your product / service, focusing on their ideal experience (happy path) as they interact with your product? | Operator enables runtime + policy constraints. Telemetry is continuously collected. LLM agent recommends profile/knob changes. Safety governor validates/overrides unsafe recommendations. Final settings are applied to the local runtime. Operator monitors dashboard (profile, rationale, trend metrics, knob diffs). Team runs adaptive vs baseline scenarios and reviews saved results. Team uses measured deltas (and optional modeled value estimates) to decide rollout expansion. | ||||
| Pain-points | Where does the user experience friction, obstacles, or unmet needs throughout the journey? Identify which pain-points are most frequent and severe? | Trust gap in autonomous tuning (high frequency, high severity): operators need clear, auditable “why” behind each AI-driven change. Policy tuning uncertainty (high frequency, high severity): difficult to set thresholds that balance quality, latency, heat, and battery across diverse devices. Cross-device inconsistency (high frequency, high severity): same policy behaves differently on different hardware classes. Difficulty proving value (high frequency, medium-high severity): teams need strong adaptive-vs-baseline evidence for adoption/expansion. Stress-time UX degradation risk (medium frequency, high severity): token surges can trigger latency spikes and thermal/battery stress if adaptation is not graceful. | ||||
| AI Opportunities | From your list of pain points, identify those that can effectively be addressed using Generative AI. Remember, this project focuses on leveraging LLM-powered AI to solve your target persona's pain points. Rank these pain points starting with the most severe and frequently occurring first. | Trust gap in autonomous tuning (highest priority) Why GenAI/LLM is effective: LLM can produce clear, contextual decision explanations (“what changed, why now, expected impact, risk”) so operators trust and approve autonomous behavior. Policy tuning uncertainty across mixed workloads and devices Why GenAI/LLM is effective: LLM agent can recommend profile/knob adjustments from live telemetry + policy intent, reducing manual trial-and-error and speeding optimal tuning. Difficulty proving value to stakeholders Why GenAI/LLM is effective: LLM can auto-summarize adaptive-vs-baseline outcomes into stakeholder-ready narratives, clearly separating measured improvements from modeled estimates. Stress-time UX degradation under token surges Why GenAI/LLM is effective: LLM agent can adapt generation behavior (profile/knob choices) in real time to preserve usability during token-heavy bursts and constrained device states. Cross-device inconsistency diagnosis Why GenAI/LLM is effective: LLM can analyze divergent telemetry/policy outcomes and suggest device-class-specific policy templates and remediation steps. Operational alert/noise overload Why GenAI/LLM is effective: LLM can cluster noisy events, prioritize operator actions, and generate concise triage summaries. | ||||
| Develop an AI Solution Hypothesis | AI Solution Hypothesis | Diverge | Ideate potential solutions to address your AI-solvable pain points. Focus on generating a high quantity of ideas rather than evaluating their quality at this stage. | LLM governance agent that recommends runtime profile changes from live telemetry. Self-governing loop where the same local LLM class is both governed and acts as decision agent. Explainability copilot that generates human-readable “why this decision” summaries. Policy tuning copilot that proposes threshold changes per device class and use case. Token-budget manager agent that dynamically caps output/context under stress. Scenario-aware agent modes (token surge, battery saver, thermal avoidance, recovery). Cross-device anomaly explainer that diagnoses inconsistent behavior across hardware tiers. Adaptive-vs-baseline insight generator that auto-summarizes KPI deltas for operators. Conservative value-estimation copilot that converts measured deltas into modeled ranges. Alert triage assistant that groups noisy events and suggests top remediation actions. Prompt-shaping assistant for concise responses during constrained operating states. Confidence-gated agent recommendations with fallback to deterministic baseline policy. | ||
| Converge | Rank your ideated solutions based on impact and feasibility. Identify your top three AI solutions, and clearly select the one you'll focus on for your project. | Top 3 AI solutions (impact + feasibility): LLM Governance Agent + Safety Guardrails (selected focus) Highest impact on core problem (real-time adaptive control) with practical feasibility in current stack. Explainability Copilot for decision rationale High impact for trust/adoption; straightforward to implement from existing telemetry/decision logs. Policy Tuning Copilot (recommend threshold updates) High operational value; medium implementation effort. | ||||
| DESIGN | Define Target State Workflow | UX Flows & Wireframes Suggested Tool: Excalidraw | Workflow (future) | Assuming your product or feature works as desired, what is the target state workflow? | Workflow (future) Connect & Initialize Admin selects runtime/model and enables agent mode. System validates runtime health and telemetry stream. Set Policy Intent Admin sets latency target, thermal cap, battery floor, quality preference. Safety guardrails are activated as hard constraints. Live Governance Loop Telemetry ingested continuously. LLM agent recommends profile/knob changes. Deterministic guardrails validate/override unsafe decisions. Final safe settings applied to runtime. Observe & Explain Dashboard shows current profile, rationale, knob diffs, pressure bars, trend charts. Operator confirms stability and expected behavior. Evaluate Run adaptive vs baseline scenarios (token stress, budget management, offline incident). Save run outputs and compare key metrics (p95 latency, thermal risk time, battery drain/hr). Optimize & Rollout Adjust policy thresholds based on evidence. Expand to more device groups/models. Optionally generate conservative modeled value estimate for stakeholder review. | Please leave this area blank. This space is for the Instructor to provide you with feedback. |
| Build Wireframes | Wireframes | How will users navigate through your AI solution? What are the key steps and decision points? What information will be displayed at each stage? What specific UI elements are needed on each screen? How will the layout accommodate AI features? | User Navigation Flow (AI Solution) 1) Landing / Operations Dashboard Goal: Instant health and trust view of the governed LLM. Key decision points Is runtime probe active or fallback mode? Is current profile acceptable (performance/balanced/efficiency/survival)? Any immediate thermal/battery risk? Info displayed Governed model name + probe status Current profile + reason Battery, thermal, CPU, memory Latency and throughput trends Latest knob adjustments UI elements Status banner (model/probe/error) KPI cards Pressure bars Trend charts “Adjustment summary” text panel Decision timeline feed 2) Policy Configuration Screen Goal: Set intent and guardrails safely. Key decision points Optimize for quality vs efficiency? What thermal cap / battery floor / latency target? Apply globally or to selected endpoint groups? Info displayed Current active config Expected policy behavior preview Last policy update metadata UI elements Numeric inputs/sliders (target_latency_ms, max_temp_c, min_battery_percent) Toggle for quality preference “Save / Revert to Default” buttons Optional policy preset chips (Battery Saver, Balanced, Performance) 3) Scenario & Experiment Runner Goal: Run controlled tests (adaptive vs baseline). Key decision points Which use case to run? Adaptive or baseline? Duration/sample interval? Baseline profile choice? Info displayed Scenario description Progress and elapsed time Run ID + saved result path UI elements Dropdown for use case Mode selector (adaptive / baseline) Baseline profile selector Duration/sample inputs “Run Experiment” button Progress indicator + completion toast 4) Results & Comparison Screen Goal: Prove value and decide rollout readiness. Key decision points Did adaptive improve p95 latency/thermal/battery? Is profile switching stable (no thrash)? Is this ready for broader deployment? Info displayed Side-by-side baseline vs adaptive metrics Delta values and percent changes Risk markers (probe failures, instability) UI elements File picker/select for two run JSONs Comparison table/cards Delta badges (improved/regressed) Export button (JSON/CSV/PDF optional) 5) Value Estimate Screen (Post-Review) Goal: Translate measured deltas into conservative business narrative. Key decision points Which assumptions are approved? Is modeled value credible enough for stakeholder discussion? Info displayed Measured deltas (hard evidence) Modeled value (energy + productivity proxy) Disclaimer text (non-guaranteed estimate) UI elements Assumption input fields “Calculate Value Estimate” button Output cards (annual_kwh_saved, annual_cost_avoidance) Persistent disclaimer banner Layout Strategy for AI Features Top bar: model/probe status + environment health (trust first) Middle main grid: KPIs + trends + AI decisions (operational center) Right/Bottom panels: explainability (“why”), knob changes, timeline Action rail: run scenario, compare, estimate value (guided workflow) Use a left-to-right maturity flow: Observe → Configure → Test → Compare → Justify Minimum Screen Set (for your project now) Dashboard (already built, enhanced) Experiment panel (can start as API-first, then UI) Comparison panel Value-estimate panel | |||
| Develop Prototype to showcase AI interactions | Prototype Screens Suggested Tool: lovable.dev | What aspects of the AI solution will you demonstrate in your prototype? How will the AI inputs, processing, and outputs be presented visually to users? Which features are essential for launch? What can be left for later releases? | Prototype Demonstration Scope (what we will show) AI Inputs (visible): Live device telemetry: battery, CPU, memory, thermal proxy Policy constraints: latency target, thermal cap, battery floor, quality preference Runtime context: model/probe status (Ollama/Llama) AI Processing (visible): Governor decision loop selecting profile (performance/balanced/efficiency/survival) Runtime knob adaptation (tokens/context/threads/quantization/accelerator) Scenario execution and adaptive-vs-baseline experiment runs AI Outputs (visible): Current profile + reason Adjustment summary (what changed and why) Latency/throughput trend charts Decision timeline Experiment result JSON files Conservative value-estimation response (measured deltas + modeled estimate + disclaimer) How inputs/processing/outputs are presented visually Dashboard cards: current state KPIs and active profile Status banner: governed LLM model + probe status + error state Pressure bars: battery/CPU/memory/thermal intensity Trend charts: latency and token throughput over time Adjustment panel: diff-style knob changes between cycles Timeline feed: chronological decision explanations API outputs (for now): experiment run and value estimate responses shown via terminal/JSON (Phase 2 can add in-page forms/results) Essential for Launch (MVP GA scope) Stable governor loop with policy guardrails and profile switching Live observability dashboard (state, rationale, trends, adjustments) Runtime integration path (Ollama/Llama probe) with health/status visibility Adaptive vs baseline experiment runner with persisted results Basic documentation and demo scenarios (token stress + one realistic enterprise scenario) Conservative value-estimate API with explicit disclaimer Can be left for later releases Dedicated multi-screen UX (Policy, Experiments, Compare, Value screens) Full prompt-suite scenario playback automation (instead of mostly label-driven runs) Cross-device fleet management and policy templates Advanced quality evaluation automation and scoring pipelines Predictive thermal forecasting and smarter control policies (RL/MPC) Automated report generation/export (PDF/slide-ready) Enterprise integrations (MDM/SIEM/ticketing/SSO/RBAC) Rich comparative model-routing advisor (Llama/Qwen/Phi with auto selection) | |||
| Initial Prompt Design | Master Prompt [Initial Design] | Create an initial master prompt. Consider the following: What tone or personality should the AI use? How should user input and system instructions be structured for maximum clarity? What system instruction will govern the AI's behavior? What examples might improve performance? How will you format outputs for consistency? | You are GovAgent, an on-device LLM runtime governance agent. Goal priority: 1) Safety (thermal, battery) 2) Continuity (avoid abrupt degradation) 3) Latency target 4) Throughput/quality Input JSON contains: - telemetry: battery_percent, on_ac_power, cpu_percent, memory_percent, temperature_c, response_latency_ms, token_throughput_tps - policy: target_latency_ms, max_temp_c, min_battery_percent, prefer_quality - current_state: current_profile, current_knobs Return STRICT JSON only: { "recommended_profile": "performance|balanced|efficiency|survival", "recommended_knob_overrides": { "max_tokens": number|null, "context_window": number|null, "temperature": number|null, "top_p": number|null, "threads": number|null, "quantization": "q4|q5|q8|fp16"|null, "accelerator": "cpu|gpu|npu"|null }, "reason": "short rationale", "confidence": number } Rules: - If temperature_c >= max_temp_c + 4 => recommended_profile = "survival". - If on_ac_power=false and battery_percent <= min_battery_percent - 5 => recommended_profile = "survival". - Confidence must be between 0.0 and 1.0. - Use minimal knob changes needed; avoid churn. - If data is missing/invalid, choose safest reasonable profile and state fallback in reason. - Output must be valid JSON with no extra text. | |||
| Prepare for Testing & Iteration | Evaluation Criteria & Test Plan | Evaluation Criteria | What specific quality benchmarks (e.g., clarity, relevance, tone, accuracy, SEO, hallucination avoidance) will define “good” output? | Evaluation Criteria Policy correctness: Recommended profile respects hard rules (thermal/battery) and policy intent. Safety compliance rate: % of decisions that stay within max_temp_c, min_battery_percent, and allowed knob ranges. Fallback robustness: Graceful fallback behavior when agent output is invalid, timed out, or malformed. Latency quality: latency_p50_ms and latency_p95_ms versus target and baseline. Throughput quality: token_throughput_tps_avg stability during stress and recovery phases. Thermal performance: peak_temp_c and time_above_temp_threshold_s compared to baseline. Battery efficiency: battery_drain_percent_per_hour reduction in adaptive mode. Stability / anti-thrashing: Controlled profile_switch_count and minimal unnecessary knob churn. Explainability quality: Rationale is concise, accurate, and actionable for operators. Agent output validity: JSON schema compliance, confidence bounds (0–1), and parse success rate. | ||
| Example Cases | What specific example use cases, edge cases, and negative cases should be covered by test prompts and outputs? | Example Cases Core Use Cases Token stress + budget management Long-context burst prompts with constrained output budgets. Expect shift toward efficiency/survival under pressure, then recovery. Offline incident copilot (battery-constrained) Sustained troubleshooting prompts on DC power. Expect battery/thermal-safe profile behavior with continuity. Thermal cliff avoidance Prolonged heavy load sequence. Expect reduced time above thermal threshold vs baseline. Recovery to quality After stress, run lighter prompts. Expect transition back to balanced/performance. Edge Cases Conflicting signals Low battery + low temp + strict latency target. Validate priority order and rationale clarity. Telemetry anomalies Missing/null/invalid metric fields. Expect safe fallback profile and explicit fallback reason. Agent malformed output Non-JSON or unsupported profile string. Expect deterministic fallback and error logging. Rapid workload oscillation Alternating heavy/light prompts every few seconds. Validate anti-thrashing behavior. Negative Cases Over-aggressive policy Very low thermal cap forcing chronic survival mode. Ensure system remains stable and reports configuration risk. Probe/runtime unavailable Ollama unavailable or timeout. Expect fallback metrics path, no control-loop crash, visible status/error. | ||||
| DEVELOP | AI Model Selection & Justification | AI Model Selection & Justification | Which AI model is best suited for your solution and why? What capabilities and limitations does it have? How will it integrate with your product? | Selected model: Llama 3.2 3B Instruct (quantized, e.g., q4_K_M) running locally via Ollama. Why this model is best for this solution: It provides strong instruction-following for structured agent outputs while remaining lightweight enough for on-device execution. It is suitable for real-time governance loops where latency, battery, and thermal constraints matter. Capabilities: Generates structured JSON recommendations for runtime control decisions Handles operational reasoning (profile/knob adaptation) with concise rationales Runs fully on-device for privacy and low dependency on cloud Limitations: Lower reasoning depth than larger cloud models Quality can degrade under aggressive efficiency settings Requires deterministic guardrails and fallback logic for safety-critical decisions How it integrates with the product: The same local LLM class is used as the governance agent (decision recommendation) and as the governed runtime target Agent recommendations are validated by a deterministic safety governor before applying runtime knobs Integration is through Ollama endpoints, with live status, experiments, and evaluation metrics surfaced in the dashboard and APIs Model-agnostic note: The architecture remains model-agnostic via a structured agent contract, enabling alternatives such as Qwen2.5 3B Instruct or Phi-3.5 mini without redesigning the control plane. | Please leave this area blank. This space is for the Instructor to provide you with feedback. | |
| Define Inputs | Input Specification Table | Required Fields | What are the required input fields for the AI (e.g., title, description, keywords, tone)? Indicate format, source, requirement. | Required Fields (Input Specification) Field Format Source Requirement Why it is required telemetry.battery_percent number (0-100) Device telemetry collector Required Battery-aware governance decisions telemetry.on_ac_power boolean Device telemetry collector Required Distinguish plugged-in vs battery constraints telemetry.cpu_percent number (0-100) Device telemetry collector Required System load signal for adaptation telemetry.memory_percent number (0-100) Device telemetry collector Required Pressure signal for runtime scaling telemetry.temperature_c number Device telemetry / thermal proxy Required Thermal guardrail enforcement policy.target_latency_ms integer Admin policy config Required Latency objective for optimization policy.max_temp_c number Admin policy config Required Hard thermal safety limit policy.min_battery_percent number Admin policy config Required Hard battery safety limit policy.prefer_quality boolean Admin policy config Required Quality vs efficiency bias current_state.current_profile enum (performance/balanced/efficiency/survival) Runtime state Required Context for incremental changes current_state.current_knobs object Runtime state Required Prevent unnecessary knob churn agent_system_prompt string System config Required Governs tone, output schema, safety behavior | ||
| Optional Fields | Are there any optional or user-customizable fields? How do they impact the AI’s output? | Optional / User-Customizable Fields Field Format Impact on AI output telemetry.response_latency_ms number Improves p95-aware profile decisions telemetry.token_throughput_tps number Better throughput/quality tradeoff decisions workload_tag enum/string (token_stress, offline_incident, etc.) Scenario-aware recommendations device_class string (low/mid/high) Device-specific policy behavior risk_tolerance enum (conservative/balanced/aggressive) Controls adaptation aggressiveness allowed_profiles list of enums Restricts agent recommendation space knob_bounds object (min/max) Constrains output for safe operations fallback_profile enum Defines deterministic behavior on agent failure model_name string Selects governing/governed model dynamically confidence_threshold number (0-1) If low confidence, trigger fallback/override explainability_level enum (short/standard/detailed) Changes rationale verbosity in output experiment_mode enum (adaptive/baseline) Determines run behavior for evaluation AI Output { "recommended_profile": "performance|balanced|efficiency|survival", "recommended_knob_overrides": { "max_tokens": null, "context_window": null, "temperature": null, "top_p": null, "threads": null, "quantization": null, "accelerator": null }, "reason": "short rationale", "confidence": 0.0 } | ||||
| Define Good Output | Output Evaluation Checklist | Objective Criteria | What criteria will you use to judge output as “good”? (e.g., structure, use of keywords, tone, factuality, relevance) | Objective Criteria (Output Evaluation Checklist) Schema correctness: Output is valid JSON and matches required fields (recommended_profile, recommended_knob_overrides, reason, confidence). Profile validity: recommended_profile is one of performance|balanced|efficiency|survival. Confidence validity: confidence is numeric and within 0.0-1.0. Guardrail compliance: Final applied decision never violates thermal/battery constraints. Fallback correctness: On malformed/timeout agent output, deterministic fallback is applied and logged. Knob safety bounds: Recommended knob values stay within allowed ranges. Stability quality: No excessive profile oscillation (profile_switch_count within threshold). Performance outcomes: Adaptive mode shows acceptable or improved latency_p95_ms vs baseline for target scenarios. Thermal outcomes: Adaptive mode reduces or maintains time_above_temp_threshold_s and controls peak_temp_c. Battery outcomes: Adaptive mode improves or maintains battery_drain_percent_per_hour in constrained scenarios. Runtime reliability: High probe/agent success rate; failures visible with clear error status. Explainability presence: Every decision includes a non-empty rationale. | ||
| Subjective Criteria | Are there any criteria that require human judgment or qualitative assessment? | Subjective Criteria (Human Judgment / Qualitative) Rationale clarity: Is the explanation understandable to IT/ops users without model internals knowledge? Actionability: Does the output help operators decide next steps (tune policy, run experiment, accept override)? Perceived smoothness: Does adaptation feel graceful to end-users under stress, not abrupt or erratic? Trustworthiness: Do recommendations appear reasonable and aligned with operational context? Business credibility: Are measured results and modeled estimates clearly separated and responsibly communicated? Stakeholder usefulness: Are outputs meaningful for both technical operators and non-technical reviewers (product/leadership)? Appropriate conservatism: Does the system avoid over-optimistic behavior when telemetry is noisy/uncertain? | ||||
| Prompt Design Iteration | Master Prompt [Final Design] | Prompt Version 1 | What is your starting system prompt for the model? What variations will you test? What techniques will you use to optimize performance? List initial instructions, persona, inputs, and constraints. | Prompt Version 1 (Master Prompt – Final Design) Starting system prompt (summary): The model acts as GovAgent, an on-device LLM governance agent. It ingests telemetry + policy + current runtime state, recommends a profile/knob update in strict JSON, prioritizes safety (thermal/battery) before latency/quality, minimizes knob churn, and falls back conservatively on uncertainty. Persona / tone: Operational, concise, evidence-driven, no hype, deterministic language. Inputs: Telemetry: battery, AC/DC, CPU, memory, temperature, latency, throughput Policy: target latency, max temperature, min battery, quality preference Current state: profile + current knobs Constraints: Strict JSON only Allowed profiles: performance/balanced/efficiency/survival Confidence must be 0–1 Hard rules: thermal or battery critical -> survival Minimal-change principle to avoid thrash Output schema: recommended_profile, recommended_knob_overrides, reason, confidence Variations tested / to test Long prompt with examples vs short production prompt (token-efficient). Reason verbosity levels (short/standard/detailed). Strict safety-first wording vs balanced wording (quality-aware). With heuristic hint input (heuristic_profile) vs without hint. Knob-free profile-only recommendation vs full knob overrides. Optimization techniques Enforce strict JSON schema and parse checks. Keep temperature low for decision consistency (~0.1). Add explicit hard guardrail instructions in system prompt. Use bounded output keys and enums to reduce malformed outputs. Use confidence gating + deterministic fallback on low confidence/errors. Minimize prompt size for control-loop efficiency after behavior stabilizes. | ||
| Prompt Iterations | If revised, what changes did you make and why? How do you track and record prompt evolution? | Prompt Iterations (changes and why) v1 -> v2: Added explicit hard guardrail rules to reduce unsafe recommendations. v2 -> v3: Added strict output schema + enum restrictions to improve parse reliability. v3 -> v4: Added “minimal knob change” instruction to reduce oscillation/churn. v4 -> v5 (final): Split into “full debug prompt” and “short production prompt” to balance observability and runtime cost. How prompt evolution is tracked Maintain prompt changelog with: version_id prompt text hash date/author scenario tested key metrics (valid JSON rate, fallback rate, profile switch count, p95 latency impact) decision notes (“kept/reverted”) Store in project docs (e.g., docs/prompt_versions.md) and link each version to experiment run IDs in experiments/results/. | ||||
| Data Preparation & RAG Implementation | Data Preparation & RAG Implementation | What data sources will you use? How will you prepare data for model training or evaluation? (e.g., cleaning, structuring). For RAG: How will you chunk, embed, retrieve relevant information? | Data sources Live telemetry stream: battery, AC/DC state, CPU %, memory %, thermal (or thermal proxy), latency, token throughput. Policy/config data: latency target, thermal cap, battery floor, quality preference, allowed profiles/knob bounds. Runtime metadata: model name, runtime/provider status, probe errors, applied knob settings. Experiment artifacts: adaptive vs baseline run outputs (experiments/results/*.json). Prompt/eval assets: scenario prompts and test plans (evals/prompt_suite.json, evals/eval_plan.md). (Optional for RAG) Knowledge docs: policy playbooks, troubleshooting guides, device-class tuning notes, incident runbooks. Data preparation (training/evaluation) No model fine-tuning required for MVP (inference-time control + prompt-based agent). Validate telemetry schema and normalize units/ranges (e.g., percentages, Celsius, ms, tps). Clean invalid/null values and apply safe defaults/fallback flags. Create structured experiment records with run metadata: mode, scenario, duration, model, metrics. Label outputs into: measured metrics (hard evidence) modeled estimates (assumption-based) Build reproducible evaluation sets by scenario (token stress, budget management, offline incident, recovery). RAG approach (optional but recommended for operator assistant) Corpus: internal docs for policies, guardrails, known issues, model/runtime compatibility, tuning recommendations. Chunking: semantic chunks of ~300–700 tokens with overlap (~10–15%) to preserve context. Embedding: use a lightweight embedding model suitable for on-device/edge or local service. Indexing: vector store keyed by topic, device_class, runtime, policy_version, date. Retrieval: top-k semantic retrieval + metadata filters (e.g., device class = low-tier, runtime = ollama). Prompt injection into agent: retrieved snippets added as “operator guidance context,” not as hard rules. Safety: deterministic guardrails remain authoritative; RAG can inform but not bypass constraints. Output quality controls Enforce strict JSON schema for agent decisions. Track parse success rate, fallback rate, and policy-compliance rate. Log rationale + source context IDs (if RAG used) for auditability. Periodically prune stale docs and re-embed updated guidance. | |||
| Create Evaluation Set | Example Input/Output Data for Testing | Typical Examples | What are the most common inputs and expected outputs? Use real data if possible. | Example Input/Output Data for Testing Typical Examples Input: Normal telemetry (battery 70%+, temp < 55C, moderate CPU/memory), balanced policy. Expected output: performance or balanced with minimal knob changes, high confidence. Input: Token-stress scenario (high latency, high CPU, rising temp, long-context task). Expected output: shift toward efficiency, reduce max_tokens/context_window/threads, rationale mentions thermal+latency pressure. Input: Battery-constrained scenario (battery < 25%, unplugged). Expected output: efficiency or survival depending on temp/load; explicit battery-protection rationale. Input: Recovery scenario (load drops, temp cools). Expected output: transition from efficiency/survival back to balanced/performance. | ||
| Edge Cases & Negative Cases | What examples test the AI’s limits? (e.g., missing data, ambiguous input, out-of-domain) | Edge Cases & Negative Cases Missing telemetry fields (null battery/temp). Malformed agent output (non-JSON, invalid profile enum). Contradictory signals (low battery but low temp and strict latency SLA). Runtime probe unavailable/timeouts. Extremely aggressive policy values causing constant survival mode. Rapid oscillating workload causing potential profile thrash. | ||||
| Test Example Data & Review Results | Manual Review | Run your input data with the prompt. How did your output perform in manual review? Which examples failed which criteria, and why? | Test Example Data & Review Results Manual Review Decision rationales were understandable and aligned with major pressure signals in most cases. Failures observed when telemetry was incomplete and rationale became too generic. Occasional over-conservative recommendations under mixed signals (quality impact risk). Main failure reasons: ambiguous input context, missing field handling, and confidence overstatement in borderline cases. | |||
| Automated Evaluation | What pass/fail rate or scores did the AI achieve on core criteria? | Automated Evaluation JSON schema validity pass rate target: >= 98% Policy compliance (final applied decision): 100% (guardrail enforced) Agent recommendation acceptance rate target: >= 80% Fallback rate target: <= 10% Adaptive vs baseline targets: time_above_temp_threshold_s: >= 20% reduction battery_drain_percent_per_hour: >= 10–15% reduction latency_p95_ms: no severe regression (or improvement in stress scenarios) | ||||
| Handle Edge Cases & Iterate | Edge Case Identification | What edge cases did you identify in testing or real usage? | Edge Case Identification Missing/invalid telemetry fields Runtime down or intermittent probe errors Malformed/invalid agent JSON Workload bursts every few seconds (oscillation risk) Device-class behavior drift (same policy, different hardware outcomes) | |||
| Updates & Adjustments | What prompt or system adjustments have you made based on failures, feedback, or edge case observations? | Added strict JSON schema checks and enum validation. Added deterministic fallback governor when agent fails. Added minimal-change rule to reduce knob churn. Added confidence gating and conservative fallback on uncertainty. Added stronger safety-first prompt constraints for thermal/battery critical states. | ||||
| Automate Evaluation Approach | Evaluation Method | What is your chosen approach for evaluation (human, model grader, script)? How will you scale testing to diverse/large test sets? | Hybrid approach: Automated scripts/API for core metrics and pass/fail checks Human review for rationale clarity/actionability Optional model-graded scoring for explanation quality consistency Use adaptive vs baseline paired runs for each scenario; persist outputs in experiments/results/. | |||
| Evaluation Frequency | How often will you re-run evaluations for new data, new prompts, or post-launch monitoring? | Every prompt/policy revision (pre-merge). Nightly/weekly scheduled scenario runs for regression detection. Before milestone demos/releases. Post-launch: periodic monitoring + triggered re-evals after model/runtime/config changes. | ||||
| DEPLOY | Finalize Launch & Rollout Plan | Operational Readiness Checklist | Technical Readiness | Is infra (APIs, databases, rate limits, monitoring, rollback) tested and documented? | Core APIs, dashboard, runtime probe, experiment runner, and value-estimation endpoints are implemented and documented. Safety fallback and schema validation are in place. Current readiness is pilot-grade; production hardening still needed for auth, RBAC, advanced monitoring, and multi-tenant controls. | Please leave this area blank. This space is for the Instructor to provide you with feedback. |
| Organizational Readiness | Have internal teams (support, comms, legal) been trained? Is documentation complete? | Primary operator personas (IT/endpoint/AI platform teams) have draft runbooks, scenario playbooks, and evaluation workflows. Formal training for support/comms/legal is planned for post-pilot phase. | ||||
| Launch & Rollout Strategy | Launch Approach | What is your launch approach? Pilot, AB test, or all users—who gets access and when? | Phased rollout: internal technical pilot, limited enterprise pilot (selected device cohorts), broader rollout after KPI and reliability gates are met. No all-user launch at MVP stage. | |||
| Scale Readiness | How will you ensure readiness for scale? How will you monitor initial volume and scale up? | Scale approach: device-class policy templates, staged cohort enablement, adaptive-vs-baseline regression checks per release, and runtime health monitoring. Initial scale monitored via p95 latency, thermal-risk time, battery drain/hr, fallback rate, and agent output validity. | ||||
| Go-to-Market Plan | Marketing / Training Assets | What assets (FAQ, demo, guides) will you prepare for external communication/marketing? | Prepare: product one-pager, architecture overview, demo script, operator quick-start guide, policy tuning guide, FAQ, KPI interpretation guide, and conservative value-estimation explainer. | |||
| Stakeholder / Internal Comms | How will you communicate launch plans, progress, and outcomes internally? | Weekly pilot updates to engineering/product/IT leadership; monthly business-readout showing measured deltas vs baseline and modeled value with disclaimers. Use clear separation of measured evidence and assumptions in all communications. | ||||
| Confirm Legal, Privacy & Risk Protocols | Data & Privacy | How do you handle and protect user data, including storage, privacy, and compliance? | Primary processing remains on-device with minimal telemetry needed for governance. Store only operational metrics required for tuning/evaluation, redact sensitive content from logs, and apply retention controls. Follow enterprise privacy-by-design and data minimization principles. | |||
| Policy & Compliance | Are content moderation, legal, and audit processes in place? Are you compliant with regulations needed for your domain? | Deterministic guardrails enforce safety limits; all agent decisions and overrides are auditable. Include human review checkpoints for policy changes, explicit disclaimers for modeled value outputs, and domain-appropriate legal/compliance review before external deployment. | ||||
| Define Success Metrics | Success Metrics | User/Business Metrics | What user metrics will indicate success? What business metrics will demonstrate value? | Reduced endpoint complaints related to AI slowness/overheating during pilot. Improved perceived stability of local AI assistant in stress scenarios. Faster pilot-to-expansion decision cycles due to clear adaptive-vs-baseline evidence. Demonstrable directional value from reduced thermal-risk time and battery drain (with conservative modeled estimate/disclaimer). | ||
| AI Metrics | How will you measure AI performance and accuracy? | Agent output validity rate (strict JSON/schema compliance). Policy compliance rate after safety enforcement (target 100%). Fallback rate (agent failure/invalid output) and agent recommendation acceptance rate. Adaptive vs baseline deltas on core metrics: latency_p95_ms, token_throughput_tps_avg, peak_temp_c, time_above_temp_threshold_s, battery_drain_percent_per_hour, profile_switch_count. | ||||
| Monitor, Iterate & Improve | User Support & Feedback Plan | Support Channels | Where can users get support? Is escalation and ownership clear? | Tiered support model: in-product dashboard/logs for operators, engineering triage queue for technical issues, and product owner for prioritization/escalation. Critical incidents route to engineering on-call with defined response windows. | ||
| Feedback Workflow | How do you gather, triage, and act on feedback and bugs? How are critical issues prioritized and communicated? | Collect feedback from pilot operators weekly; classify by severity (P0–P3) and type (safety, stability, quality, usability). Review in triage meetings, assign owners/ETAs, and communicate status in weekly pilot updates. | ||||
| Monitoring & Continuous Improvement | Monitoring Approach | What monitoring/logging is in place to spot operational/AI issues post-launch? | Monitor API/runtime health, agent parse success, fallback rate, safety overrides, and core performance/thermal/battery KPIs. Maintain audit logs for decisions and overrides; alert on threshold breaches and anomaly spikes. | |||
| Ongoing Improvement | How will you collect learnings, review performance, and update your system continuously post-launch? | Run scheduled adaptive-vs-baseline regressions after prompt/policy/runtime changes. Use findings to tune prompts, guardrails, and defaults by device class. Track prompt/version changes and KPI impacts to maintain a measurable improvement loop. | ||||
