Managing Raspberry Pi Fleet using Ansible and Semaphore

Introduction

In one of our deployments, we manage hundreds of edge devices specifically Raspberry Pis scattered across the country, with that footprint expanding as we scale. Each Pi handles heavy, real-time workloads: ingesting streams from multiple IP cameras, uploading video segments to cloud storage, broadcasting live feeds, and running local machine learning inference. Because these are true edge environments, they frequently contend with limited or unstable connectivity. On-device storage is another constant constraint; without aggressive cleanup routines, SD cards quickly fill up with cached footage. Consequently, routine maintenance, service restarts, and troubleshooting are daily necessities.

We already had Prometheus and Grafana configured to track fleet health, ensuring failures like dropped camera feeds or crashed services triggered alerts immediately. The bottleneck was the remediation workflow. Whenever an alert fired, an engineer had to open a terminal, SSH into the affected device, diagnose the root cause, verify network paths, restart the service, and confirm recovery. Managing disk space was just as manual: someone had to log in, identify stale recordings, and delete them by hand.

Across hundreds of nodes, this manual triage broke down. The team spent hours every week executing repetitive operational tasks that should have run automatically in seconds. We needed a centralized platform to orchestrate these workflows, allowing us to trigger diagnostic checks, restart services, and prune storage without opening individual SSH sessions.

To solve this, we evaluated various runbook automation tools. In this post, we share our findings and break down why we chose Semaphore over alternatives like Rundeck.

Why we picked Semaphore for Ansible and Playbook Automation

After comparing different runbook automation tools, we chose Semaphore because it matched the way we wanted to manage and automate our Raspberry Pi fleet:

Self-hosted: Our devices communicate through a Tailscale mesh network and are not directly reachable from the internet. We needed a solution that we could host on our own infrastructure and use to reach the devices through the existing network.

MIT-licensed: Semaphore is open source and MIT-licensed, so there are no per-node licensing costs. We could run it on a single VM and use it to manage the entire fleet.

Web UI for the ops team: Not everyone on the team is comfortable working directly from the command line. Semaphore provides a simple web interface where team members can trigger a playbook, select the target host, and view the execution results without having to SSH into the device or remember Ansible commands.

API-first: Semaphore provides a REST API for its core functionality. This gives us the flexibility to automate things like inventory management, update target hosts dynamically, and integrate Semaphore with our existing tools and workflows.

Ansible-based: Semaphore uses Ansible for running automation tasks. This gave us access to Ansible's existing modules and ecosystem while Semaphore handled the UI, scheduling, and access control around the playbooks.

Our Solution

Smart inventory: only target the devices that need it

The first thing we built was a way to automatically identify which devices need attention instead of running a playbook against all Raspberry Pis.

Prometheus continuously collects metrics from every device, including a camera availability metric that shows the percentage of cameras currently recording at each facility. We use this metric to identify devices where camera availability has dropped below 100%.

We wrote a custom Ansible module called sync_inventory that queries Prometheus and gets the list of hosts where camera availability has remained below the expected level for a defined period. The module then reads our source inventory file, which is a static INI file containing the Tailscale IPs and SSH credentials for all devices. It extracts the connection details for only the affected hosts and pushes this filtered inventory to Semaphore through its REST API.

# sync_inventory.yml

- hosts: localhost
  connection: local
  tasks:
    - name: Query Prometheus and update Semaphore inventory
      sync_inventory:
        prometheus_url: '{{ prometheus_url }}'
        query: 'last_over_time(recorder_cameras_availability_ratio[5m]) < 1'
        semaphore_url: '{{ semaphore_url }}'
        semaphore_token: '{{ semaphore_token }}'
        semaphore_inventory_id: '{{ semaphore_inventory_id }}'
        inventory_file: '{{ source_inventory_file }}'
yaml

The result: when we run the self-heal playbook, it doesn't hit all devices. It targets the 8 or 12 that actually have problems. That's the difference between a 30-second playbook run and a 30-minute one.

We run this inventory sync as its own Semaphore task, typically right before the healing playbook. The two tasks are chained in our workflow: sync first, then heal.

Self-healing cameras: the core automation

This is the big one. The recorder_heal playbook is what runs when cameras go down. It's a multi-play orchestration that diagnoses the problem, attempts automated recovery, and reports back with a detailed summary.

Here's what happens when it runs:

Play 1: Per-Host Diagnosis and Recovery

For each host with failed cameras, the playbook walks through a diagnostic tree:

1. Read camera status - Each Raspberry Pi maintains the current state of its cameras. The automation reads this status and identifies which cameras are recording and which have failed.
2. Map cameras to IP addresses - The camera status only identifies cameras by their IDs, so the workflow also reads the device configuration to map each camera ID to its IP address. This gives us the information needed for network-level checks.
3. Check camera connectivity - The workflow pings each failed camera and records whether it is reachable over the network. This helps separate cameras that are still reachable from those that have a network or connectivity issue.
4. Collect logs for unreachable cameras - If a camera cannot be reached, the workflow collects the relevant logs from the device. This gives us additional information for troubleshooting even when the camera cannot be recovered remotely.
5. Check disk usage - The workflow checks the device's data partition and identifies whether disk usage has crossed the configured threshold. A full disk can prevent the recorder from writing new video and can result in recording failures.
6. Find old recordings - If disk usage is high, the workflow scans the recordings stored on the device and identifies older data that can be removed. It discovers the cameras and recording directories dynamically rather than relying on a fixed list of camera IDs.
7. Decide the recovery action - Based on the results of these checks, the workflow determines the appropriate action. For example, it can restart the recording service when the affected cameras are reachable, clean up old recordings when disk space is low, or flag the device for manual investigation when the cameras are unreachable.
8. Execute the recovery - When a service restart is required, the workflow restarts the recording service and waits for it to settle. It then checks the camera status again to verify whether the affected cameras have recovered.
9. Store the results - The diagnosis and recovery results from each host are stored and passed to the next stage of the workflow, where they are aggregated into a single report.

Play 2: Aggregated Summary and Reporting

The second play runs on localhost and pulls together all the data from every host:

• Generates a styled HTML report with per-facility sections, camera details, embedded logs, and disk usage bars
• Uploads the report to S3 with a timestamped key (so we have a history) and a "latest" key that always points to the most recent run
• Sends a Slack notification with a concise summary: which hosts had issues, what was recovered, what still needs manual attention

# recorder_heal.yml (simplified)

- hosts: "{{ target_host | default('all') }}"
  become: yes
  vars:
    status_file: /var/lib/recorder/recording-status.json
    config_file: /etc/recorder/config/recorder.yml
    data_dir: /var/lib/recorder/data
    ping_timeout: 3
    settle_time: 10
    disk_threshold: 90
    retention_days: 5
  tasks:
    - name: Read recording status
      recorder_status:
        status_file: '{{ status_file }}'
      register: rec_status
    - name: Exit if all cameras are recording
      ansible.builtin.meta: end_host
      when: rec_status.cameras_failed == 0
    - name: Read recorder config for camera IPs
      recorder_config:
        config_file: '{{ config_file }}'
      register: rec_config
    - name: Ping failed cameras
      camera_ping:
        camera_ids: '{{ rec_status.failed_cameras }}'
        camera_ip_map: '{{ rec_config.camera_ip_map }}'
        timeout: '{{ ping_timeout }}'
      register: ping_result
yaml

Disk cleanup: automated storage management

Disk cleanup deserves its own section because it solves a slightly different problem. Even when cameras are recording fine, recording data accumulates. A Pi with 6 cameras recording 24/7 generates a lot of data. If you don't clean it up, the disk fills up and then cameras start failing.

The disk_cleanup playbook runs on a schedule and proactively manages storage:

# disk_cleanup.yml

- hosts: "{{ target_host | default('all') }}"
  become: yes
  vars:
    data_dir: /var/lib/recorder/data
    retention_days: 5
    dry_run: false
  tasks:
    - name: Scan for old recording directories
      disk_cleanup:
        data_dir: '{{ data_dir }}'
        retention_days: '{{ retention_days }}'
        dry_run: '{{ dry_run }}'
      register: cleanup_result
yaml

The disk cleanup process automatically finds all the camera directories under /var/lib/recorder/data/, so we don't need to maintain a hardcoded list of camera IDs. It then checks the main/ directory for each camera, identifies the recording folders based on their dates, handles the different date formats we have used over time, calculates how much space they are using, and removes recordings that are older than the configured retention period.

After each run, it sends a Slack message with a per-camera breakdown: how many folders were deleted, how much space was freed, and which folders were kept. The ops team can see at a glance what happened without needing to check Semaphore logs.

This has eliminated disk-related outages entirely. Before automation, we would get 2-3 alerts per month about full disks. Now it's just handled in the background.

Uploader health: catching stale records

Each Pi runs an uploader service that uploads recorded video segments to cloud storage. The uploader keeps track of each recording in a local SQLite database using different status values, such as pending, uploading, failed, and completed.

Sometimes the uploader can get stuck or fall behind, causing old records to remain in the pending or failed state. As these records build up, the uploader may keep retrying older uploads that are no longer relevant. This can use unnecessary bandwidth and make it harder to identify the actual state of the uploader.

To handle this, we built two separate playbooks: one for monitoring the uploader and another for clearing stale records when needed.

Monitoring: The monitoring playbook checks the SQLite database on each host, counts the pending uploads by status, identifies the oldest pending recording, and checks whether the uploader service is running.

# test_uploader_status.yml

- hosts: "{{ target_host | default('all') }}"
  become: yes
  tasks:
    - name: Check uploader status
      uploader_status:
        db_path: /etc/recorder/config/data.db
        uploader_service: uploader
      register: uploader_result
yaml

The results are sent to Slack so we can keep an eye on the overall state of the fleet. This also helps us spot patterns, such as one device consistently having a much larger upload backlog than the others.

Remediation: When a backlog needs to be cleared, we use a separate playbook that first stops the uploader service, updates older records in the SQLite database, and then starts the service again.

# bring-uploader-to-current.yml

- hosts: "{{ target_host | default('all') }}"
  become: yes
  tasks:
    - name: Stop uploader service
      ansible.builtin.systemd:
        name: uploader
        state: stopped
    - name: Update old recordings to status 3
      uploader_db_update:
        db_path: /etc/recorder/config/data.db
        cutoff_date: yesterday
    - name: Start uploader service
      ansible.builtin.systemd:
        name: uploader
        state: started
yaml

The update marks records older than yesterday as completed so the uploader can move on to current recordings instead of continuing to process an old backlog.

We run the monitoring playbook daily and the remediation playbook weekly, or manually when we see a backlog building up. We use yesterday as the cutoff because recordings created today may still be actively uploaded, and we don't want to mark those records as completed while they are still in progress.

Design Principles

A few Semaphore features and design choices have made this setup easier to operate as the number of devices has grown.

Slack notifications with detailed output. Semaphore can send notifications when a playbook execution succeeds or fails, which gives us a quick view of the overall result. We wanted more than just a pass or fail notification, so we added Ansible modules to generate detailed results and send them to Slack. This includes things like which cameras failed, whether they were reachable, disk usage, recovery actions, and any relevant logs. This gives the team the summary from Semaphore along with the details needed to understand what happened without having to open the Semaphore UI.

HTML reports for deeper investigation. For cases where the Slack output isn't enough, our playbooks generate a more detailed HTML report and store it in S3. The report includes per-facility navigation, camera details, FFmpeg logs, and disk usage information. We also keep timestamped reports so we can look back at previous runs when investigating recurring issues.

API-driven workflows. Semaphore provides a REST API that we use to automate parts of the workflow outside the UI. For example, we use it to update inventories, pass variables to tasks, and trigger playbooks programmatically. This allows Semaphore to fit into our existing monitoring and automation workflows instead of requiring every action to be started manually.

The Impact

What's Next

The playbooks we've built so far mainly handle reactive issues, where something goes wrong and the automation helps diagnose and recover it. There is still a lot of manual work around managing the devices, and that's where we want to take this next.

Pi Provisioning: Setting up a new Raspberry Pi is still a multi-step process. It involves preparing the SD card, running the installation, configuring the network, pushing the initial configuration, and verifying that the device is ready. We want to automate this process so that a new Pi can go from a fresh installation to a fully configured and monitored device with minimal manual work.

Proactive Disk Management: Today, disk cleanup happens when usage reaches a certain threshold. The next step is to use the historical disk usage and recording patterns to identify when a device is likely to run out of space and take action before it becomes an issue.

More Self-Healing Workflows: The camera recovery workflow is just the starting point. We want to apply the same approach to other components running on the Pi, such as the uploader, ML services, and network services. Each workflow can diagnose the issue, take the appropriate recovery action, and verify that the service has recovered.

The main goal is to move more operational work into repeatable automation. Once the playbook logic is in place, Semaphore gives us a central place to trigger it, target the right devices, schedule runs, and see the results.

Instead of manually connecting to individual Raspberry Pis for every issue or routine task, we can keep adding automation as new operational requirements come up. That's where we see the biggest value in using Semaphore for managing our edge devices.

Why We Picked Semaphore for Ansible and Playbook Automation

Managing a growing fleet of edge devices can quickly become difficult without the right automation and operational practices. CloudRaft helps teams build reliable infrastructure automation with Ansible, Kubernetes, DevOps, and cloud-native technologies—so your systems stay consistent, secure, and easier to operate at scale. Talk to CloudRaft about your infrastructure automation needs.

Meet the CEO

Book a meeting with Anjul