Google Gemini powered Coop Presence/Door Control

I've written previously about our two coops and the automatic doors we use on them - on our large chicken coop, we use an Omlet door and on the goose coop we use a larger Ador turkey door. I have both of these integrated with home assistant, the Omlet door via the omlet API and the Ador via a relay board attached to an esp8266 running ESPHome. For the past 6 months or so, I've had these on simple time based automations, closing the coop doors after dark and opening around sunrise. This works very well for the chickens, who reliably return to the coop with the sun, but the geese tend to be more finicky. They will come and go throughout the evening, often getting locked out of the coop when the door finally closes.

Inspired by this reddit post, I figured a better approach would be to pipe an image to Google Gemini asking it if the geese and turkey were in the coop, and then shut the door if that was the case. I already had a wz_mini_hacks flashed wyze camera inside capturing the entire coop - I took a quick snapshot from it and passed it to gemini flash 2.0 in the web GUI. The couple of examples I passed (both with birds inside and birds not inside) all worked. It seemed like this would be feasible!

First, I installed the Google Generative AI integration and created an API key to use it per the docs.

I broke the project into a couple of steps - first, I built a script to capture an image from the camera inside the coop. The "goose coop" target is from my frigate integration.

Next, I built the main automation. Every 5 minutes between 7PM and 11PM, and if the door is still open, I capture an image from the inside coop camera, ask Gemini if the geese and turkey are inside, and act on its response accordingly.

Breaking it down, the first portion just calls the capture image script described above and delays for 5s to make sure the image is written to disk and ready to be sent to Gemini.

Next, I use the Google GenAI "Generate Content" action, passing the path to the image as an attachment and using the prompt in the image below. I store gemini's response in the "goose_response" variable.

Next, I created an "input text" helper called goose_response to store Gemini's response and set the helper to the response from the previous step.

The final step is a conditional check to see if the response was "birds inside". If so, I call a switch - turn off action to shut the relay that controls my door (an ESPHome device) and then notify our phones that the door has shut.

As I wrote this up, I realized I could expand on this and create true coop presence sensors by asking Gemini to count the number of birds in each image. I created a new automation to do just that, taking a picture every 5 minutes from both coop's interior cameras, counting the birds, and setting an input_number helper for each coop.

This was effectively copy paste from the door automation, with the exception of changing the prompt (to request a count and a response with only a number), and having to cast the text response from Gemini to an integer to match the input_number helper data type.

I added both occupancy numbers to my coop cards in the lovelace UI:

As I wrote this, Gemini was within 1 bird of being 100% accurate in both coops - in the goose coop, the missing bird is off camera and in the chicken coop, heavily obscured. I was pleasantly surprised at it's accuracy.

The free tier gemini rate limit is very generous (15 requests per minute as of now) and should easily support even a significant expansion on this use case.

Occupancy sensors open up a whole new world of automation types - off the top of my head, I will likely automate the coop interior lights in the same manner I use Aqara mmwave radar sensors in the house.

Full yaml for everything discussed below:

Coop count automation:

alias: Gemini - Coop Occupancy Count
description: ""
triggers:
  - trigger: time_pattern
    minutes: /5
conditions: []
actions:
  - action: script.turn_on
    metadata: {}
    data: {}
    target:
      entity_id: script.take_snapshot_from_goose_coop_camera
  - action: script.turn_on
    metadata: {}
    data: {}
    target:
      entity_id: script.take_snapshot_from_chicken_coop_camera
  - delay:
      hours: 0
      minutes: 0
      seconds: 5
      milliseconds: 0
  - action: google_generative_ai_conversation.generate_content
    metadata: {}
    data:
      prompt: >+
        This is the inside of my goose coop. How many birds (chickens, geese,
        and ducks) are inside the coop?


        Respond only with a number.

      filenames:
        - /config/tmp/goose_coop_snap.jpg
    response_variable: goose_response
  - action: input_number.set_value
    metadata: {}
    data:
      value: "{{goose_response.text | int}}"
    target:
      entity_id: input_number.goose_coop_occupancy_count
  - action: google_generative_ai_conversation.generate_content
    metadata: {}
    data:
      prompt: >+
        This is the inside of my chicken coop. How many birds (chickens, geese,
        and ducks) are inside the coop?


        Respond only with a number.

      filenames:
        - /config/tmp/chicken_coop_snap.jpg
    response_variable: chicken_response
  - action: input_number.set_value
    metadata: {}
    data:
      value: "{{chicken_response.text | int}}"
    target:
      entity_id: input_number.chicken_coop_occupancy_count
mode: single

Door closing automation:

alias: Goose Coop - Gemini check
description: ""
triggers:
  - trigger: time_pattern
    minutes: /5
conditions:
  - condition: time
    after: "19:00:00"
    before: "23:00:00"
  - condition: state
    entity_id: switch.goose_coop_door_ador_channel_b
    state: "on"
actions:
  - action: script.turn_on
    metadata: {}
    data: {}
    target:
      entity_id: script.take_snapshot_from_goose_coop_camera
  - delay:
      hours: 0
      minutes: 0
      seconds: 5
      milliseconds: 0
  - action: google_generative_ai_conversation.generate_content
    metadata: {}
    data:
      prompt: >+
        I am trying to decide whether or not to close my goose coop. 3 geese and
        1 turkey live in this coop. The geese all look similar and are african
        geese with gray tones. The turkey is larger and all white.


        If there are at least 2 geese and one turkey in this image,  say: "Birds
        inside"


        Otherwise, or if you are unsure, say"Birds not inside". 



        Use only one of these two exact sentences.

      filenames:
        - /config/tmp/goose_coop_snap.jpg
    response_variable: goose_response
  - action: input_text.set_value
    metadata: {}
    data:
      value: "{{goose_response.text}}"
    target:
      entity_id: input_text.goose_coop_response
  - if:
      - condition: state
        entity_id: input_text.goose_coop_response
        state: Birds inside
    then:
      - action: switch.turn_off
        metadata: {}
        data: {}
        target:
          entity_id: switch.goose_coop_door_ador_channel_b
      - action: notify.mobile_app_jay_s24
        metadata: {}
        data:
          message: Goose coop door closed
      - action: notify.mobile_app_sm_s928u
        metadata: {}
        data:
          message: Goose coop door closed
mode: single

Capture image from camera script:

sequence:
  - action: camera.snapshot
    target:
      entity_id: camera.goose_coop
    data:
      filename: /config/tmp/goose_coop_snap.jpg
alias: Take snapshot from goose coop camera
description: ""