TL;DR

We harvested cabbage, celery, carrots, and basil. A surprise spider and tomato plant were encountered in the field while weeding. It was another hot day, but hopefully the last of a three week heat wave, at least for a little bit before August.

Bugs are everywhere and can’t be simply ‘cancelled’ without affecting quality. On a farm you can’t use pesticides, herbicides, or fungicides without affecting the quality and safety of the produce. In my homelab code, the more time I spend on fixing bugs, the less time I have to spend on higher-order work. So my weekend work is to fix a specific bug before getting back to automating the metadata tagging of all the photos and semantic delineation of post contents.

To my corporate readers: what is your employer’s approach to addressing bugs, either in operations or in its products? If it’s not managed properly, chances are that broad-spectrum methods like lay-offs won’t keep the company afloat for very long.

Bugs Around the Farm

There are many, many types of insects on the farm. Not all are ‘pests’, but many are. There are cabbage butterflies (a.k.a. Pieris rapae) whose larva leave little poop and bite marks on the outer leaves (which we throw away anyway). There are aphids, leafhoppers, and other soft-bodied insects that suck the sap out of the plants. There are beetles, caterpillars, and other insects that eat the leaves. There are ants, bees, and other insects that pollinate the plants. There are spiders, centipedes, and other insects that eat the pests. There are greenhead flies and mosquitoes that eat on us and the animals.

One of the photos I took today was of a big brown spider in the basil eating what looks like a young leafhopper. As a gardener and farmer, I’m not a big fan of aphids or caterpillars, and any natural predators that help to minimize these pest populations are most welcome. Like I mentioned in my previous post, applying agro-chemicals like chlorothalonil (by ground or air) has recently been linked to the decline in reproductive capability of bees, and I can only imagine it has similar effects on other insect-preadators as well.

This farm doesn’t spray pesticides, which is really good for many reasons, but it also means that we have to deal with the bugs. Even with other farms that use pesticides, there are non-affected bugs. Organic gardening practices would tell you to do things like knock down aphid and mite populations with a water jet spray, apply a vinegar and/or neem oil spray, or even introduce nematodes to keep specific pest species. On a farm at scale, all of these methods get cost and/or time prohibitive.

There are always bugs in nature…and especially in software. Some are your fault in your own code, some are not (in dependencies you bring in), but you always eventually have to deal with them.

Bugs Around My Homelab

Not all bugs manifest themselves in the same way. Some are more obvious, annoying, destructive, or dangerous than others…just like bugs on the farm. In my opinion, the priority of addressing bugs goes:

1. Mission-critical functionality
2. Security-related vulnerabilities
3. Data destruction/loss issues
4. Performance and reliability issues
5. Everything else, including new functionality

The reason for ‘new functionality’ being at the bottom is not that it’s not important, but everything above it undermines new work if it’s not addressed. If I were a 10-person team then we could split up these responsibilities and work on them in parallel…but of course the more people, the more communication and coordination become a factor. Right now, my team is just me.

Cancellable Components in Autoscribe

Every day I’m on the farm, I record audio notes on my drive back home, then I upload them to a specific Google Drive folder because it’s easy to do so from my phone. About a minute after that, my own component ‘Autoscribe’ picks them up and summarizes them. In March, I finalized the first early version and put it in production, which as always exposes a plume of outlier and exception cases in real operation. Dealing with these bugs are expected after significant releases.

About two weeks ago, I noticed that sometimes Autoscribe would stop summarizing audio files. I was able to track this down to a bug in the way the Langchain and ollama libraries don’t provide signals to cancel ongoing requests. When diagnosing this in my development environment, I also realized that they don’t handle SIGTERM or SIGINT signals well either (the other way around vs. the original problem).

At that time, I looked at how others tend to deal with this and found that wrapping these types of calls in async subprocesses/threads using the ‘asyncio’ library was the conventional approach when it’s a code dependency used by the main application vs. a shell command with it’s own subprocess ID to kill. While functionally bug-free, this introduced an even more insidious and inconsistent bug that only happens in production after the container is running for a while and doesn’t happen when automatically retried again…the dreaded ‘Event loop is closed’ error I now see in Slack notifications as output of monitoring the process.

Canceling Workforces to Maintain Profitability

This weekend, I know I’ll fix this and move on, but it’s an example of unanticipated time-spend due to using a 3rd party library that I don’t control. This happens all the time in software development, not because of human error or oversight, but because of the nature of the work itself. You can’t cancel bugs…or the need for skilled tech workers…just because balance sheets and speculative management thinking say it’s the easier way to deal with maintaining profitability margins.

Who’s going to solve for all the work AI will never get right…the unanticipated, the insidious, the undocumented, and only operationally evident bugs? I just hope that companies stupid enough to be laying off skilled and needed workers will be the first ones to drive themselves out of existence when their lack of organizational and cognitive bandwidth hits critical mass.

Both expansion and contraction in workforces, in their pay, in requirements are all part of every industry. It’s the nature of the beast. Specifically, management culture is the primary driver of this. Mass lay-offs are not easy decisions, but they’re a damned sight easier on those making the decisions than on those being laid off. You can’t cancel your board or shareholders expectations for constant growth, or the real root problem which is that many tech companies were never built to survive contraction cycles. Companies might cancel workforces, but market forces cancel weak companies.

AI Only Gets Right What We Feed It

‘Autoscribe’, my AI-assisted transcription and summarization process component is relatively self-contained for now. It runs as a deployment in my Kubernetes homelab cluster and the longest span without restarting it was 117 days (only due to applying scheduled OS updates which moved the pod to another node).

It currently uses a version of Whisper.cpp that I compile from source with OpenVINO which utilizes either/both CPU and integrated GPUs of older hardware for the transcription from audio/voice to text. For the most part, this is a bug-free.

What isn’t bug-free is the summarization phase despite adding try-catch handling around cancel and finalization calls. The code problems I will fix, but I’d really like to get back to the real work of improving the quality of the summarizations by adjusting prompts, context, and models I use. Harvesting, in this case would be making progress on higher-order ‘business’ use cases such as easy print-outs of my blogs to send to my extended family members who aren’t on social media or email much.

For today, Autoscribe finally produced the below AI-generated summary…after the customary first-attempt failure of course:

Summary

Consolidated Summary

Main Themes

1. Weather: Overcast and humid morning with warming afternoon conditions.
2. Crop Harvesting: Multiple crops (cabbage, celery, carrots, basil, tomatoes) harvested with varying methods and maturity stages.
3. Farm Maintenance: Weeding, irrigation line adjustments, electric fence checks, onion washing, and pig watering.
4. Observations: Unusual natural occurrences (a large brown spider, a lone tomato plant).
5. Farm Operations: Packaging electric van, battery maintenance, and crop tracking.

Activities Performed

• Harvesting: Cabbage, celery, carrots, basil, and tomatoes collected.
• Weeding: Removed weeds from celery, summer squash, kale, and pepper rows.
• Irrigation: Moved sprinkler lines and hoses.
• Maintenance: Washed onions, packaged electric van, watered pigs, replaced electric fence batteries.
• Observations: Noted a large brown spider and a lone tomato plant.

New Things Encountered

1. A lone tomato plant growing without support.
2. A large brown spider (species unknown).
3. A specific tomato variety (likely wild or hybrid).

Questions for Further Research

1. What species is the large brown spider?
2. How did the lone tomato plant grow without support? Is it wild or hybrid?
3. How long do electric fence batteries last? Are any from last year or earlier?
4. Will the lone tomato plant continue to grow or produce fruit?

Suggested Actions

1. Document the Spider: Photo and consult an expert for identification.
2. Monitor the Tomato Plant: Track growth and fruit production over time.
3. Inspect Electric Fence Batteries: Test lifespan and replace outdated ones.
4. Record Weather Impact: Note how weather affects crop growth and harvesting.
5. Expand Weeding Efforts: Target persistent weed areas (e.g., around summer squash or peppers).

Part 1

Main Themes

1. Weather Conditions: Overcast and humid morning with a warming afternoon.
2. Crop Harvesting: Multiple crops (cabbage, celery, carrots, basil, tomatoes) were harvested, each with distinct methods and stages of maturity.
3. Farm Maintenance: Weeding, irrigation, and equipment maintenance (e.g., electric fence, sprinkler lines).
4. Observations: Unusual natural occurrences (a lone tomato plant, a large spider).
5. Farm Operations: Onion washing, pig watering, and battery maintenance for the electric fence.

Activities Performed by the Intern

1. Harvesting:
   • Cabbage: Picked mature, undamaged heads.
   • Celery: Harvested two blue buckets worth, noting growth progression.
   • Carrots: Used pitchforks to extract carrots 2 inches below the soil surface.
   • Basil: Cut 2 inches above soil, harvested in bunches for $5 value.
   • Tomatoes: Harvested ripe cherry tomatoes and hoop house tomatoes.
2. Weeding: Removed weeds from celery, summer squash, kale, and pepper/shishito pepper rows.
3. Irrigation: Moved sprinkler lines and hoses.
4. Farm Maintenance:
   • Washed onions.
   • Packaged electric van.
   • Watered pigs.
   • Checked and replaced electric fence battery.
5. Observations:
   • Noted a large brown spider (species unknown).
   • Discovered a lone tomato plant producing green tomatoes.

New Things Not Yet Encountered

1. A lone tomato plant growing without a trellis or support.
2. A large brown spider (species not identified).
3. A specific type of tomato plant (likely a wild or unmarked variety).

Questions and Future Research Areas

1. Spider Species: What is the species of the large brown spider observed?
2. Tomato Plant Origin: How did the lone tomato plant grow without support? Is it a wild or hybrid variety?
3. Battery Lifespan: How long do the electric fence batteries last? Are some from last year or earlier needing replacement?
4. Tomato Plant Growth: Will the lone tomato plant continue to grow or produce fruit? How does it compare to cultivated plants?

Suggested Actions

1. Document the Spider: Take a clear photo and consult an entomologist or online database to identify the species.
2. Monitor the Tomato Plant: Track its growth and fruit production over time to determine if it is a wild, hybrid, or cultivated variety.
3. Inspect Electric Fence Batteries: Test the battery lifespan and replace any that are over a year old.
4. Record Weather Impact: Note how the weather (overcast, humid, warming) affects crop growth and harvesting efficiency.
5. Expand Weeding Efforts: Focus on areas with persistent weeds (e.g., around summer squash or pepper plants).