Skip to content
LifeOS / Video Editing Agent
Production-grade · v1

Video Editing Agent

A production system that automates complex video editing workflows, built on a single hard rule: separate thinking from doing. The model plans, a deterministic worker executes, and every decision is written down.

Deterministic executionLLM planningTrace-first
01

Section 01

Summary & core philosophy

Many agentic systems lean on fuzzy state management. This one does the opposite: the orchestrator does all the thinking, the worker does all the doing, and the boundary between them is a structured plan rather than a guess.

LLM planning

A high-level orchestrator (GPT-4o or similar) reads the raw footage and the user's brief, then writes a structured EditPlan: a sequence of discrete, deterministic operations.

Deterministic worker

A Python worker consumes the EditPlan and executes it with FFmpeg and MoviePy. When a command fails, it reports the exact error straight back to the orchestrator.

No VectorDB for state

State lives in PostgreSQL with JSONB, not a vector store. A project's whole state is a serialized tree of decisions: reproducible, debuggable, and auditable.

02

Section 02

Database schema

Drizzle ORM manages the PostgreSQL schema, designed for high observability and strict data integrity. Four tables hold the entire lifecycle of a job.

TableRole
jobsTop-level edit request and its lifecycle status.
assetsSource media: raw footage, b-roll, and audio.
edit_plansThe structured LLM output, versioned per job.
tracesPer-step observability for every FFmpeg call.
schema.ts
import { pgTable, uuid, varchar, text, timestamp, jsonb, integer, pgEnum } from 'drizzle-orm/pg-core'; export const jobStatusEnum = pgEnum('job_status', ['pending', 'processing', 'completed', 'failed', 'review_required']); // Core Jobs Tableexport const jobs = pgTable('jobs', {  id: uuid('id').primaryKey().defaultRandom(),  userId: uuid('user_id').notNull(),  title: varchar('title', { length: 255 }).notNull(),  status: jobStatusEnum('status').default('pending').notNull(),  editPlanId: uuid('edit_plan_id').references(() => editPlans.id),  createdAt: timestamp('created_at').defaultNow().notNull(),  updatedAt: timestamp('updated_at').defaultNow().notNull(),}); // Assets Tableexport const assets = pgTable('assets', {  id: uuid('id').primaryKey().defaultRandom(),  jobId: uuid('job_id').references(() => jobs.id).notNull(),  storageUrl: text('storage_url').notNull(),  assetType: varchar('asset_type', { length: 50 }).notNull(), // 'raw_footage', 'b-roll', 'audio'  metadata: jsonb('metadata').notNull(), // duration, resolution, fps, bitrate  createdAt: timestamp('created_at').defaultNow().notNull(),}); // Edit Plans Table (The LLM Output)export const editPlans = pgTable('edit_plans', {  id: uuid('id').primaryKey().defaultRandom(),  jobId: uuid('job_id').references(() => jobs.id).notNull(),  planData: jsonb('plan_data').notNull(), // Array of instructions: cut, transition, overlay  version: integer('version').default(1).notNull(),  llmModel: varchar('llm_model', { length: 50 }),  promptTokens: integer('prompt_tokens'),  completionTokens: integer('completion_tokens'),  createdAt: timestamp('created_at').defaultNow().notNull(),}); // Observability & Trace Logsexport const traces = pgTable('traces', {  id: uuid('id').primaryKey().defaultRandom(),  jobId: uuid('job_id').references(() => jobs.id).notNull(),  traceId: varchar('trace_id', { length: 100 }).notNull(),  stepName: varchar('step_name', { length: 255 }).notNull(),  ffmpegArgs: text('ffmpeg_args'),  renderTimeMs: integer('render_time_ms'),  logs: text('logs'),  payload: jsonb('payload'), // Input/Output snapshots  isFailure: varchar('is_failure', { length: 5 }).default('false'),  createdAt: timestamp('created_at').defaultNow().notNull(),});
03

Section 03

Video evaluation pipeline

An automated scoring engine guards quality before a render is finalized. Six metrics run on every candidate edit.

  1. 01

    Clarity & resolution

    Verifies export settings against the source assets.

  2. 02

    Punchiness

    Reads cut density and audio-visual sync, like cuts landing on the beat.

  3. 03

    Retention scoring

    LLM analysis of the first 3 seconds (the hook) and overall narrative pacing.

  4. 04

    Caption readability

    OCR checks and contrast-ratio validation for burned-in captions.

  5. 05

    Brand fit

    Color-histogram analysis and watermark / logo placement checks.

  6. 06

    Watchability

    Audio normalization (LUFS) and flicker detection.

Golden rendersscore > 0.85

A Golden Render is an edit that passed every evaluation gate. These are tagged in storage and used as ground truth for future style training, or shipped as the final deliverable.

04

Section 04

Observability & tracing

A trace-first architecture makes complex FFmpeg pipelines debuggable. Every job carries a root trace_id; every sub-task inherits it and adds its own span.

Trace ID lifecycle

  1. 1

    Job created

    Every job is assigned a root trace_id at creation.

  2. 2

    Sub-tasks inherit

    Whisper transcription, FFmpeg cuts, and color grades inherit that trace_id and add a span_id.

  3. 3

    Real-time shipping

    Logs stream to the traces table as each step runs.

Logged properties

ffmpeg_args
The exact command-line string sent to the worker.
render_time
Precision timing for each filtergraph execution.
cpu_usage / gpu_usage
Resource monitoring captured per task.
failures
Full stack traces and FFmpeg stderr, stored in the logs column.
05

Section 05

Local-friendly stack

The whole system runs efficiently on high-end local hardware or private cloud instances. PostgreSQL doubles as the persistent queue and state store, so a worker restart never loses work.

TS orchestrator

Node.js · TypeScript

API, planning, and state management on Drizzle + PostgreSQL.

Python / FFmpeg worker

subprocess · moviepy

FFmpeg via subprocess; MoviePy for complex timeline composition.

Whisper local

faster-whisper

Local NVIDIA GPUs, or CPU with OpenVINO, for high-speed transcription.

Redis queue

BullMQ · RQ

Reliable task distribution across TypeScript and Python workers.

JSONB state

PostgreSQL

Persistent queue and state store, so no work is lost on worker restart.