AI-Powered Automations.

Before building AI-powered automations.

Prerequisites and framing.

The individual AI capabilities are working. AI-powered automations compose voice pipelines, LLMs, and computer vision. The individual pieces need to be operational first. See [Voice Assistants](/home-assistant/ai/voice), [LLM Integrations](/home-assistant/ai/llm), and [Frigate and Computer Vision](/home-assistant/ai/frigate) for setup details.

Automation fundamentals comfortable. [Automation Fundamentals](/home-assistant/automations/fundamentals), [Advanced Automation Patterns](/home-assistant/automations/advanced-patterns), [Scripts and Scenes](/home-assistant/automations/scripts-scenes), and [Template Sensors](/home-assistant/automations/templates) cover the automation building blocks. AI-powered automations use all of them, with AI calls as additional actions or triggers.

A clear scope for AI in the automations. AI works well for information and interaction. AI is a poor fit for safety-critical decisions and autonomous control of systems where consequences matter. The patterns on this page stay within the information-and-interaction zone; the "What AI should not decide" section below covers why.

Cost awareness for cloud AI. Automations that trigger frequently and call cloud LLMs in their actions run up costs. Rate-limiting, caching, and preferring local LLMs for high-volume cases matter. For cloud AI used in automations, monitoring usage from the first deployment prevents surprises.

Composition patterns.

The common ways AI capabilities combine in automations.

Trigger, enrich, notify. An automation triggers on a sensor condition (temperature threshold, moisture low, motion detected). Before sending the notification, the automation calls an LLM to generate context — not just "temperature high" but "temperature has been above target for 15 minutes, trending up since 13:00, cooling automations have not fired." The notification arrives with useful context.

Trigger, query, respond. Voice input triggers the automation (through the Conversation integration). The automation queries Home Assistant's state, passes results to an LLM for natural-language response, and speaks the response through Piper. The grower asks "how is Zone 2 doing?" and hears a plain-language answer built from current data.

Detect, classify, decide, record. Frigate detects motion or a specific object. The automation determines whether the detection matters (time of day, zone, frequency). If it does, the automation generates a record (log entry, notification, dashboard update). AI as the detector, deterministic logic as the decider.

Scheduled, summarize, deliver. Time trigger fires. Automation queries Home Assistant for relevant data (overnight conditions, recent events, upcoming requirements). LLM summarizes the data into a briefing. Briefing goes to the grower through notifications, email, or voice.

Pattern match, annotate, alert. An anomaly detection (deterministic logic finds a condition that is unusual or outside expected bounds). LLM generates a plausible explanation and recommended follow-up. Alert goes to the grower with both the anomaly and the AI-generated interpretation.

Ask, tool-use, confirm. A voice or text command asks the system to do something. LLM with tool access interprets the command and identifies the action. The system confirms the interpretation before executing (especially for destructive actions). Execution happens after confirmation.

Alert enrichment.

One of the highest-leverage AI automation patterns.

The problem. A basic alert says "Zone 2 temperature exceeded 85°F." The grower receives it, maybe at an inconvenient time, and has to open Home Assistant to understand what is happening. How long has it been high? Is it trending up or stabilized? Has the cooling system responded? Are other zones affected?

The enriched alert. The automation, between detecting the condition and sending the notification, asks an LLM to describe the situation given relevant data. The LLM receives current temperature, recent trend, cooling automation state, outdoor temperature, other zone temperatures, and a prompt to describe the situation concisely. The LLM produces a paragraph: "Zone 2 is at 89°F, above the 85°F threshold for 12 minutes. Temperature has been trending up since 13:00. Cooling automation has not fired because the ventilation setpoint is still 82°F. Outdoor temperature is 97°F. Zone 1 is at 82°F; Zone 3 is at 83°F."

Prompt construction. The automation builds the prompt with specific data:

- The triggering condition (what fired the automation). - Current values of relevant entities (the zone's temperature and humidity, other zones' temperatures, outdoor conditions). - Recent history or trends (temperature direction over the last hour). - Automation state (what cooling or other response has happened recently). - A brief instruction to the LLM about format and tone ("summarize concisely, in two or three sentences, focusing on what the grower needs to know").

The fallback discipline. Every AI-enriched alert must have a non-AI fallback path. If the LLM call fails (timeout, provider outage, integration bug), the alert still goes out as a basic notification. The grower should never miss an alert because AI enrichment failed. The fallback is not a detail; it is a requirement for production use.

Rate limiting. An alert that fires repeatedly during an extended out-of-range period should not call the LLM every time. Rate-limit the enrichment — the first alert is enriched; subsequent alerts within a time window use a shorter, non-enriched form or skip the notification entirely. Input helpers tracking the last-enrichment-time make this straightforward.

Privacy consideration. The prompt sent to a cloud LLM includes operational context — current values, zone names, nearby zone data. For operations with sensitive data, review what gets sent and consider local LLMs for enrichment.

Morning and evening briefings.

Scheduled summaries that give the grower a digest view.

The pattern. An automation fires at a scheduled time (06:00 for morning, 20:00 for evening). The automation queries Home Assistant for the relevant data (overnight events, current conditions, yesterday's summary). An LLM summarizes the data. The summary goes to the grower through notification, email, or voice.

Morning briefing content. What the grower needs at the start of the day:

- Any alerts that fired overnight (and whether they are still active). - Current conditions across zones (temperature, humidity, VPD, last irrigation). - Yesterday's DLI for lighting-dependent zones; whether targets were met. - Any equipment or automation states that changed overnight. - Weather forecast for the day (if a weather integration is configured). - Anything scheduled for the day (calendar events, planned irrigation, fertigation, etc.).

Evening briefing content. What happened today:

- Key events from the day (alerts that fired, automations that ran, anomalies). - DLI accumulated per zone, total irrigation time, any notable excursions from setpoints. - Comparison to recent days (was today unusual in any dimension). - Anything requiring follow-up tomorrow.

Prompt structure. The automation builds a substantial prompt — more context than an alert-enrichment prompt. A detailed system prompt (who the LLM is, what the grower expects) plus the data for the period plus a request for the summary format. For morning briefings, the tone can be oriented toward planning the day; for evenings, reviewing what happened.

Delivery options. Morning briefing as a phone notification, as an email, as a dashboard update, or as a voice announcement through a satellite. Multiple deliveries are possible; the operation picks what fits. Voice briefings are particularly useful for growers who start the day in the greenhouse rather than at a desk.

Accuracy expectations. Briefings are AI-generated summaries. They can be slightly off — missing an event, mischaracterizing a trend, including a phrasing that does not quite fit the situation. The grower treats them as drafts that complement dashboard review, not as authoritative records. The dashboards remain the source of truth.

Vision-triggered automations.

Using Frigate events as automation triggers.

The basic pattern. Frigate detects an object (person, vehicle, custom-trained pest, etc.). The detection produces a Home Assistant event or state change. An automation triggers on the event and takes action.

Beyond simple triggers. AI-powered vision automations typically do more than just "detect, notify." They combine the detection with context — time of day, recent history, other events, operational state — to decide whether the detection matters.

Pest detection with assessment. A trap camera with custom pest detection fires. The automation checks how many detections have fired in the last 24 hours. If the rate is within normal background levels, the detection is logged but no alert is sent. If the rate is increasing meaningfully, the automation alerts the IPM manager with a snapshot and an AI-generated assessment of the trend.

Entry monitoring with time context. A camera detects a person at an entry point. During normal working hours, this is just a logged event. Outside working hours, the automation sends an alert with a snapshot and an LLM-generated note about whether this fits recent patterns or is unusual.

Equipment state change detection. A camera aimed at a piece of equipment detects state changes (person approaching, equipment running, safety guard moving). The automation coordinates with other sensors to maintain an accurate picture of equipment status.

Operational area coverage. Cameras at workstations, load-in areas, fertigation stations detect person-present and person-leaves events. The automation uses these to maintain a "who is where" picture useful for coordination or safety.

The logging layer. Vision detections should generally be logged regardless of whether they trigger alerts. The log provides the historical record that is useful for trend analysis, compliance, and investigation. Alerts are a subset of logged events — the ones that meet additional criteria for operator attention.

Voice-driven operational queries.

Natural-language queries that use LLM interpretation.

The architecture. Voice input captured by a satellite. Whisper transcribes it. The intent engine tries to match; if no match, the query falls back to an LLM. The LLM interprets the query, queries Home Assistant for relevant data through tool use, and formulates a response. Piper synthesizes the response audio. The satellite plays it.

Query categories.

- Simple state queries. "What's the temperature in Zone 1?" The built-in intent engine handles these; LLM fallback is not needed. - Multi-value summaries. "How are my zones doing?" The LLM queries each zone's conditions and builds a response. The built-in engine does not handle this kind of composition. - Historical queries. "What was the highest temperature yesterday?" The LLM queries history and formulates a response. Simple historical queries may have built-in handlers; complex ones use the LLM. - Operational questions. "Should I expect Zone 2 to hit target DLI today?" The LLM considers current accumulated DLI, current PPFD, time remaining in photoperiod, and estimates. The answer is informed but not authoritative — it is a calculation the grower could verify. - Summary requests. "Give me the daily update." The LLM produces a summary similar to scheduled briefings, but on-demand through voice.

Tool scope. The LLM handling voice queries has read access to exposed entities. It typically does not have write access through voice — voice-triggered actions route through the built-in intent engine, which is deterministic. The LLM is a query and response layer; actions happen through explicit mapped commands.

Response style for voice. Spoken responses should be shorter than written responses. A paragraph that reads well on a screen is exhausting as speech. The system prompt for voice responses should emphasize brevity: "Respond in one or two sentences suitable for being spoken aloud. Do not read lists or detailed numbers; summarize."

Offline behavior. Voice queries that depend on cloud LLMs stop working when the internet is down. The voice pipeline itself (Whisper, the intent engine, Piper) works locally; only the LLM fallback requires the cloud for cloud-configured setups. Operations where voice needs to work during outages should use local LLMs for the fallback, or accept that complex voice queries need internet.

Anomaly narration.

Turning statistical anomalies into human-readable explanations.

The problem. A deterministic anomaly-detection template sensor fires when a value is unusual (three standard deviations from recent average, or persisting outside expected range, or whatever criterion). The alert says "anomaly detected in Zone 1 humidity." What does that mean? How should the grower interpret it?

The solution. When the anomaly fires, an automation asks an LLM to describe what "anomaly" means in context. The LLM receives the current value, the historical baseline, the trend, and related conditions. It produces a description: "Humidity in Zone 1 is at 85%, compared to a 7-day average of 65%. The rise began around 14:30 and correlates with a drop in ventilation fan activity. Possible causes include a stuck damper, a misconfigured setpoint, or a condition-specific automation override."

Hedging language matters. The LLM should not assert causes it does not know. "Possible causes include..." is appropriate; "The cause is..." is not. The system prompt should instruct the LLM to hedge appropriately and distinguish between observations (data) and inferences (interpretation).

Pairing with the data. The narration should accompany the raw data, not replace it. The alert includes the values, the thresholds, the trend, and the narration. The grower sees both what the data says and what the AI interprets; they can verify the narration against the data.

When narration helps. Complex anomalies involving multiple systems — a situation where temperature, humidity, and equipment state all changed together. The narration connects the dots. For single-variable anomalies that are obvious from the data, narration adds little.

AI-assisted decision support.

AI as a thinking partner for decisions the grower is already making.

The pattern. The grower asks a question about the operation — "should I advance Zone 3 to the flowering stage this week?" or "is the Zone 1 soil moisture trend concerning?" — and the LLM responds with an analysis that incorporates current data, historical patterns, and general domain knowledge. The grower weighs the response and makes the decision.

Why this is decision support, not decision-making. The LLM does not execute the decision. It contributes a perspective. The grower retains authority. The LLM's output is one input among many — others include the grower's direct observation, market considerations, operational context, and judgment that the LLM does not have.

Where this fits well.

- Ambiguous situations. Conditions where the "right" answer is not obvious from the data alone. An LLM bringing general knowledge to bear can help surface considerations the grower might have overlooked. - Routine decisions. Where the decision is clear and the LLM's role is to summarize the relevant data rather than reach a novel conclusion. - Teaching moments. When the grower is training staff or explaining a decision, an LLM's articulation of the reasoning can be a useful starting point.

Where this does not fit.

- Time-critical decisions. Fast-moving situations where the delay of asking and waiting for a response matters. - Decisions requiring the grower's specific expertise. A 45-year-veteran grower's judgment on their specific crop in their specific operation is better than an LLM trained on general agricultural data. - Decisions with consequences the LLM cannot assess. Market timing, labor coordination, regulatory nuance — contextual factors the LLM does not see.

Integration with existing automations. A decision-support pattern does not usually need to be a Home Assistant automation at all. The grower uses voice or a text interface to ask; the response informs their decision; actions follow through existing manual or automated paths. The automation is the infrastructure that makes the asking and answering possible, not a step in the decision itself.

What AI should not decide.

The boundary that matters most.

Safety-critical decisions. Fire prevention (thermal cutoffs on heaters, fused circuits). Overflow prevention (mechanical float switches). Critical alerts to workers in danger. These are hardware-and-deterministic-automation territory. AI might suggest that conditions are concerning; it should not autonomously decide safety actions.

Irreversible actions. Destroying crops, terminating plants, scheduling major interventions. Actions with consequences that cannot be undone need deliberate human authorization. An AI that decides to discard a batch of propagation because it thinks conditions were bad is wrong in a way that matters.

Significant financial decisions. Buying inputs, selling outputs, adjusting pricing. These are business decisions that the operation owner makes based on factors the AI does not see.

Critical environmental controls. Target setpoints, thresholds, automation configuration. The grower decides these based on their crop expertise. AI can summarize data relevant to the decisions; AI should not autonomously change the configuration.

Compliance actions. Batch tracking updates, compliance record entries, regulatory submissions. These need accurate data, not AI-generated plausibilities. Deterministic tooling with explicit grower input handles compliance.

Who has access. Decisions about which users can access what. The grower configures access; AI does not reason about who gets access to what.

The test. Before letting AI autonomously drive any action, ask: if the AI does the wrong thing, what is the consequence, and is it recoverable? If the consequence is significant and hard to recover from, AI should not be the decider. AI can surface information that informs the decision; the grower acts.

Testing AI-powered automations.

AI introduces variability that needs explicit testing.

Deterministic components first. The automation's trigger, conditions, and non-AI actions should be tested like any automation. The trace viewer shows whether the flow is correct. AI calls come on top of a flow that is already known to work.

Test the AI call separately. Invoke the LLM or vision integration with the specific prompts and data the automation will use. Review the outputs. Do they make sense? Are they concise enough? Do they avoid hallucination?

Test the fallback path. Deliberately cause the AI call to fail (disable the integration, provide an invalid API key, throttle the LLM) and verify the automation still produces useful output. If the fallback produces nothing or an incoherent notification, fix it before production deployment.

Test with edge-case data. Conditions the LLM has not seen. Missing data. Unusual values. The AI should handle these gracefully — either by skipping gracefully, by acknowledging it does not know, or by flagging the data as unusual.

Monitor in production. After deployment, review AI outputs periodically. Summaries that do not summarize well, enrichments that mischaracterize situations, detections that fire for the wrong reasons — these reveal themselves in production use. The review does not have to be every output, but it should be regular.

Iterate on prompts. The system prompts that drive AI behavior are configuration. Expect to adjust them as issues emerge. Version-control the prompts so changes are tracked. Treat prompt tuning as ongoing, not one-time.

Common failure modes.

Specific AI-powered-automation problems from real deployments.

The enrichment that turned into gibberish. The LLM's prompt context exceeded its limits; the response came back truncated or malformed. Fix: keep prompts focused; limit the amount of context sent; test with full-context scenarios before deployment.

The briefing that was confidently wrong. The morning briefing said irrigation had run when it had not; the grower trusted the briefing and did not check. Fix: never let the LLM assert facts it cannot verify from provided data; the prompt should include only facts the LLM can reference; distinguish between "based on provided data" and general claims.

The automation that failed silently when AI was down. The cloud LLM provider had an outage; the automation's LLM calls returned errors; the automation did not send any notification at all. Fix: fallback paths that produce a basic alert when AI is unavailable; monitoring for AI integration health; critical alerts must not depend on AI.

The runaway cost from a high-frequency trigger. An automation triggered by motion sensor called an LLM each time; the motion fired thousands of times during a busy day; the monthly bill was surprising. Fix: rate-limit AI calls; aggregate events before processing; use local LLMs for high-volume cases; monitor usage.

The vision detection that kept causing false-positive enrichments. Frigate's default model fired spuriously; each false fire triggered an AI-enriched alert; the grower's phone buzzed repeatedly. Fix: tune Frigate's detection (confidence thresholds, zones) before adding AI on top; AI on top of unreliable detection produces unreliable alerts.

The voice summary that was too long. The LLM produced paragraphs when the grower wanted a sentence. Listening to long spoken responses in a greenhouse is uncomfortable. Fix: explicit brevity instructions in the voice system prompt; example responses showing the desired length; post-processing to truncate if length exceeds a threshold.

The LLM that leaked data across zones. A prompt about Zone 1 accidentally included Zone 2's data in the response. Fix: scope the prompt context narrowly; do not include more data than the query needs; review what the automation sends in each call.

The morning briefing that changed behavior after a model update. The LLM provider released a new model version; responses subtly changed (different tone, different structure, occasionally different content). Fix: pin to specific model versions where possible; test after model updates; have the system prompt stable enough that model changes have minimal effect.

The automation that called itself. A voice response was played through a satellite; the response included a wake word; the satellite woke up, heard its own response, and tried to interpret it as a command. Fix: mute the satellite during TTS output; avoid including wake words or near-wake-words in responses; monitor for feedback patterns.

The chain of AI calls that compounded errors. Frigate detected something; LLM summarized the detection for an alert; voice read the summary aloud; each step introduced possible distortion; the final spoken message was far from what the raw detection meant. Fix: minimize chains; keep raw data alongside AI interpretations so the grower can verify; review the chain end-to-end in testing.

What not to do.

Patterns to avoid.

Don't put AI on the critical path of safety automations. A cooling system that depends on an LLM-generated decision to fire is a broken cooling system when the LLM is unavailable. Safety automations stay deterministic.

Don't skip fallback paths. Every AI-powered automation needs a non-AI fallback that produces reasonable output when AI is not working. Alerts that disappear because AI is down are worse than alerts without enrichment.

Don't trust AI outputs without verification for actions that matter. The LLM's output is a draft. For anything the grower will act on, the underlying data should be reviewable. Summaries pair with raw data; enrichments include the values they reference.

Don't chain too many AI calls. Each call adds failure risk and cost. A single well-designed prompt that does the work of three chained simple prompts is usually better than the chain.

Don't use cloud LLMs for high-frequency automations. The cost scales with frequency. High-frequency automations should use local LLMs or avoid LLM calls altogether (deterministic logic may be enough).

Don't deploy AI automations without testing the unhappy paths. AI failure, partial failure, strange-data cases. These will happen in production; test them deliberately.

Don't let AI-powered automations creep into decision-making authority. The line between "provides information for decisions" and "makes decisions" is one to defend explicitly. AI automations that start deciding what to do rather than informing are usually a design error waiting to cause trouble.

Don't forget monitoring. AI behavior can drift with model updates, prompt changes, or usage-pattern shifts. Periodic review of AI outputs catches drift before it causes problems.

Don't ignore privacy when building AI automations. Each automation's AI call sends data somewhere. Review what is sent, where it goes, and whether the privacy implications are acceptable. Scope prompts to minimize exposure.