Terrabot

Terrabot runs a three-layer agent architecture: a behavioral layer of FSM-driven control behaviors, an executive layer with real-time monitors, and a planning layer that generates and adapts daily schedules using constraint satisfaction. Each layer operates on a 1-second tick cycle, with the planner regenerating schedules daily based on plant growth stage and environmental conditions.

Foliage classification, segmenting plant pixels from soil, hardware, and background, is handled by a fine-tuned UNet with a ResNet34 encoder, pretrained on ImageNet and exported to ONNX.

An initial Vision Transformer approach was abandoned after it caused out-of-memory failures on the Raspberry Pi during inference. The UNet traded a little accuracy (98% to 97%) for a far smaller memory footprint, proving more practical for continuous on-device operation.

The system captures images on a scheduled cadence: an FSM raises LED levels to 400-600 lux, waits for stabilization, then triggers the camera with up to 3 retry attempts on failure.

Before segmentation, a color calibration pipeline normalizes lighting variation across captures. A second UNet detects the physical calibration target, then an affine color transform computed in LAB space maps each image's color distribution to a reference, compensating for LED drift, time-of-day variation, and white-balance shifts.

Plant pixel extraction achieved ~97% accuracy on test data. During deployment, moss growth on the rockwool substrate introduced an out-of-distribution challenge absent from training; the model still excluded the moss from segmentation masks, though it may have affected foliage boundaries.

A height estimation module uses a reference ruler mask and the segmented foliage boundary to measure growth in centimeters, interpolating between tick marks to handle partial occlusion. Both metrics feed a health evaluation function that tracks greenery percentage and height changes over time, generating daily status messages.

Two redundant sensor boards each carry temperature/humidity, light, soil-moisture, and weight sensors, plus a shared reservoir-level sensor, microphone, and camera, 13 channels total.

The microphone verifies actuator operation by detecting pump and fan activity acoustically. Redundancy enables cross-validation: if Board0 and Board1 disagree beyond a threshold, the system flags a potential hardware fault.

A SAT-based hardware diagnosis system models the full signal and power path, outlet through Raspberry Pi, Arduino, sensor boards, and actuators, as a Boolean constraint network.

When readings deviate from expected values, it generates all minimal diagnoses: the smallest sets of component failures or disconnections that explain the observations, using OR-Tools to enumerate solutions and prune supersets.

The planner models the 24-hour day as 48 half-hour blocks and solves for a valid schedule satisfying minimum run durations (e.g. 8 hours of light), mutual exclusion between conflicting actuator demands (raising and lowering temperature can't co-occur), night-time caps, and min/max spacing between behavior instances.

The schedule regenerates daily, adapting to the current growth stage, for example, increasing watering limits and shifting temperature setpoints after germination.

Every 24 hours the agent generates and sends an HTML email with the latest plant photo, foliage-segmentation and height overlays, time-series plots (daily and cumulative) for all sensor channels, and a watering summary with cumulative and delta volumes.

It also estimates plant weight, decomposing total weight change into water content vs. growth using soil-moisture correlation. A health-evaluation string built from greenery and height deltas gives an at-a-glance status per report.

The pipeline runs as its own FSM behavior on a 24-hour cadence, pulling data from across the system, vision outputs from the classifier and height measurer, sensor logs from the logging monitor, and watering totals from the soil-moisture behavior, making it a lightweight observability layer over the full agent.

Images captured automatically by the agent over the grow period, assembled into a timelapse from first sprouts through full canopy coverage.