Oliver
308 lines
16 KiB
HTML
308 lines
16 KiB
HTML
<!doctype html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="UTF-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||
<meta name="robots" content="index, follow" />
|
||
<link rel="canonical" href="https://derez.ai/blog/posts/why-backing-up-an-agent-is-so-hard.html" />
|
||
<title>Why Backing Up an AI Agent Is So Hard — Derez.ai Blog</title>
|
||
<meta name="description" content="69% of an agent's state lives outside its home directory. Real filesystem data on why home-dir backups fail — and why full-disk snapshots are the only viable solution." />
|
||
<meta property="og:type" content="article" />
|
||
<meta property="og:url" content="https://derez.ai/blog/posts/why-backing-up-an-agent-is-so-hard.html" />
|
||
<meta property="og:title" content="Why Backing Up an AI Agent Is So Hard — And How Full-Disk Snapshots Fix It" />
|
||
<meta property="og:description" content="69% of an agent's state lives outside its home directory. Real filesystem data on why home-dir backups fail — and why full-disk snapshots are the only viable solution." />
|
||
<meta name="twitter:card" content="summary_large_image" />
|
||
<meta name="twitter:title" content="Why Backing Up an AI Agent Is So Hard — And How Full-Disk Snapshots Fix It" />
|
||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||
<link href="https://fonts.googleapis.com/css2?family=Inter:opsz,wght@14..32,400;14..32,500;14..32,600;14..32,700&display=swap" rel="stylesheet" />
|
||
<script defer data-domain="derez.ai" src="https://plausible.odoo4projects.com/js/script.js"></script>
|
||
<style>
|
||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||
body {
|
||
background: #08080c; color: #e8e8f0;
|
||
font-family: 'Inter', sans-serif; line-height: 1.7;
|
||
padding: 40px 20px;
|
||
}
|
||
.container { max-width: 720px; margin: 0 auto; }
|
||
a { color: #00f5ff; text-decoration: none; }
|
||
a:hover { text-decoration: underline; }
|
||
.back { margin-bottom: 32px; display: inline-block; font-size: 0.9rem; color: #888; }
|
||
.back:hover { color: #00f5ff; }
|
||
h1 { font-size: 1.9rem; font-weight: 700; line-height: 1.3; margin-bottom: 12px; }
|
||
.meta { font-size: 0.85rem; color: #888; margin-bottom: 40px; display: flex; gap: 12px; align-items: center; flex-wrap: wrap; }
|
||
.area-tag {
|
||
display: inline-block; background: #a78bfa; color: #000;
|
||
font-size: 0.75rem; font-weight: 700; padding: 2px 10px;
|
||
border-radius: 20px; text-transform: uppercase; letter-spacing: 0.5px;
|
||
}
|
||
.chip-tag {
|
||
display: inline-block; background: rgba(0,245,255,0.1); color: #00f5ff;
|
||
border: 1px solid rgba(0,245,255,0.3); font-size: 0.75rem; font-weight: 600;
|
||
padding: 2px 10px; border-radius: 20px; text-transform: uppercase; letter-spacing: 0.5px;
|
||
}
|
||
h2 { font-size: 1.3rem; font-weight: 600; margin: 36px 0 12px; color: #f0f0ff; }
|
||
h3 { font-size: 1.1rem; font-weight: 600; margin: 24px 0 8px; color: #f0f0ff; }
|
||
p { margin-bottom: 16px; color: #c8c8d8; }
|
||
ul, ol { margin: 0 0 16px 24px; color: #c8c8d8; }
|
||
li { margin-bottom: 6px; }
|
||
code { background: #0e0e14; padding: 2px 6px; border-radius: 4px; font-size: 0.9em; color: #00f5ff; }
|
||
pre { background: #0e0e14; border: 1px solid #1a1a24; border-radius: 8px; padding: 16px; overflow-x: auto; margin: 16px 0; font-size: 0.85rem; color: #c8c8d8; }
|
||
strong { color: #f0f0ff; }
|
||
.pro-tip {
|
||
background: rgba(0,245,255,0.06); border-left: 3px solid #00f5ff;
|
||
border-radius: 8px; padding: 16px 20px; margin: 20px 0;
|
||
font-size: 0.92rem; color: #d0d0e0;
|
||
}
|
||
.pro-tip strong { color: #00f5ff; }
|
||
.cta-box {
|
||
background: #12121c; border: 1px solid rgba(0,245,255,0.2);
|
||
border-radius: 12px; padding: 24px; margin: 32px 0; text-align: center;
|
||
}
|
||
.cta-box h3 { margin-top: 0; }
|
||
.cta-box .btn {
|
||
display: inline-block; background: #00f5ff; color: #000;
|
||
font-weight: 700; padding: 12px 28px; border-radius: 8px;
|
||
margin-top: 12px; text-decoration: none;
|
||
}
|
||
.cta-box .btn:hover { background: #00ddff; }
|
||
.data-table {
|
||
width: 100%; border-collapse: collapse; margin: 16px 0;
|
||
font-size: 0.88rem;
|
||
}
|
||
.data-table th, .data-table td {
|
||
text-align: left; padding: 8px 12px;
|
||
border-bottom: 1px solid #1a1a24;
|
||
}
|
||
.data-table th { color: #00f5ff; font-weight: 600; }
|
||
.data-table td { color: #c8c8d8; }
|
||
.data-table tr:last-child td { border-bottom: none; }
|
||
.number { color: #f0f0ff; font-weight: 600; }
|
||
.highlight-red { color: #ff6b6b; font-weight: 600; }
|
||
@media (max-width: 600px) {
|
||
body { padding: 24px 16px; }
|
||
h1 { font-size: 1.5rem; }
|
||
.data-table { font-size: 0.78rem; }
|
||
.data-table th, .data-table td { padding: 6px 8px; }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="container">
|
||
<a class="back" href="https://derez.ai/#blog">← Back to Blog</a>
|
||
<h1>Why Backing Up an AI Agent Is So Hard — And How Full-Disk Snapshots Fix It</h1>
|
||
<div class="meta">
|
||
<span>June 12, 2026</span>
|
||
<span class="area-tag">Architecture</span>
|
||
<span class="chip-tag">hermes</span>
|
||
</div>
|
||
|
||
<p>
|
||
Ask any DevOps engineer how to back up a server, and they'll say "home directory, config files, databases." Standard playbook. Works for web apps, works for databases, works for most cloud workloads.
|
||
</p>
|
||
|
||
<p>
|
||
Ask them how to back up an <strong>AI agent</strong>, and suddenly nobody has a good answer. Because an agent isn't a config file with a database attached — it's a <strong>full Linux system</strong> whose state scatters across <code>/usr</code>, <code>/var</code>, <code>/etc</code>, and <code>/root</code> in ways that standard backup tools never expect.
|
||
</p>
|
||
|
||
<p>
|
||
This post walks through real filesystem data from a live Hermes agent that ran speed-runs over two weeks. The numbers may surprise you.
|
||
</p>
|
||
|
||
<h2>The Sprawl Problem</h2>
|
||
|
||
<p>
|
||
After one week of basic agent work, we measured the filesystem changes from a clean Debian 12 install:
|
||
</p>
|
||
|
||
<pre>
|
||
504.6M ./usr
|
||
43.3M ./var
|
||
448.9M ./root
|
||
54.0K ./etc
|
||
996.8M total</pre>
|
||
|
||
<p>
|
||
A home-directory-only backup (<code>/root</code>) captures <strong>448.9 MB — 45%</strong> of what changed. Already missing over half the agent's state.
|
||
</p>
|
||
|
||
<p>
|
||
Then we ran the agent through several speed-runs: Hermes Agent setup, competitor profiling, cold email configuration, Odoo integration, mobile.de research scraping. After all setups:
|
||
</p>
|
||
|
||
<pre>
|
||
4.3G ./usr ← 8.5× growth
|
||
487.0M ./var ← 11.2× growth
|
||
2.1G ./root ← 4.7× growth
|
||
1.9M ./etc ← 35× growth
|
||
6.8G total ← 6.8× growth</pre>
|
||
|
||
<p>
|
||
Now a home-dir-only backup captures <strong class="number">2.1 GB (31%)</strong> — meaning <strong class="highlight-red">69% of the agent's state lives outside its home directory</strong>.
|
||
</p>
|
||
|
||
<div class="pro-tip">
|
||
<p><strong>The 69% problem: </strong>If you back up only <code>/root</code> and restore, your agent comes back with its config files and skills — but with no Hermes runtime, no Python packages, no Node modules, no HuggingFace models, no system dependencies. It won't even start.</p>
|
||
</div>
|
||
|
||
<h2>Where Does All That State Live?</h2>
|
||
|
||
<p>Here's the breakdown of what drives an agent's storage footprint and where it scatters:</p>
|
||
|
||
<table class="data-table">
|
||
<tr>
|
||
<th>Location</th>
|
||
<th>Size</th>
|
||
<th>What It Is</th>
|
||
<th>Trigger</th>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/usr/local/lib</code></td>
|
||
<td class="number">2.3 GB</td>
|
||
<td>Hermes runtime, node_modules, Python 3.11 libs</td>
|
||
<td>Hermes installation</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/root/.npm/_cacache</code></td>
|
||
<td class="number">390 MB</td>
|
||
<td>npm package cache</td>
|
||
<td>Node.js tooling</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/root/.hermes</code></td>
|
||
<td class="number">412 MB</td>
|
||
<td>19 skills, 1 plugin, cron jobs, memories, config</td>
|
||
<td>All speed-runs</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/root/.cache/huggingface</code></td>
|
||
<td class="number">142 MB</td>
|
||
<td>HuggingFace model weights</td>
|
||
<td>STT / model downloads</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/root/.cache/uv</code></td>
|
||
<td class="number">186 MB</td>
|
||
<td>UV Python package cache</td>
|
||
<td>Hermes venv management</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/var/cache/apt</code></td>
|
||
<td class="number">379 MB</td>
|
||
<td>Debian package cache</td>
|
||
<td>System dependencies</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/var/lib/apt</code></td>
|
||
<td class="number">80 MB</td>
|
||
<td>APT package state</td>
|
||
<td>System dependencies</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/usr/local/bin</code></td>
|
||
<td class="number">28 MB</td>
|
||
<td>hermes CLI, other executables</td>
|
||
<td>Hermes installation</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/root/.config</code></td>
|
||
<td class="number">475 KB</td>
|
||
<td>App configurations</td>
|
||
<td>Misc tools</td>
|
||
</tr>
|
||
<tr>
|
||
<td><code>/etc</code></td>
|
||
<td class="number">1.9 MB</td>
|
||
<td>System config (hostname, apt sources, etc.)</td>
|
||
<td>OS configuration</td>
|
||
</tr>
|
||
</table>
|
||
|
||
<p>
|
||
<strong>Total blog content created during this period:</strong> 5 posts (Odoo, Hermes setup guide, competitor profiling, cold email, mobile.de research) — and those 5 posts alone drove the system past 1 GB.
|
||
</p>
|
||
|
||
<h2>Why It Gets Worse Over Time</h2>
|
||
|
||
<p>
|
||
The sprawl isn't static — it <strong>compounds</strong>. Each new skill adds Python or Node dependencies in <code>/usr/local</code>. Each new cron job writes logs and state. Each new model download caches weights in <code>/root/.cache</code>. Each system package lands in <code>/var</code>.
|
||
</p>
|
||
|
||
<p>
|
||
Look at the trend:
|
||
</p>
|
||
|
||
<ul>
|
||
<li><strong>Week 1:</strong> 55% of state outside <code>/root</code></li>
|
||
<li><strong>After 3 speed-runs:</strong> 69% outside <code>/root</code></li>
|
||
<li><strong>Projection:</strong> As skills and integrations grow, the percentage <em>increases</em> — more <code>/usr</code> packages, more <code>/var</code> data, more scattered caches</li>
|
||
</ul>
|
||
|
||
<p>
|
||
A "backup your home directory" strategy doesn't just miss some data — it misses a <strong>growing majority</strong> of it. Restoring from a home-dir backup would give you your <code>.hermes</code> skills and config back, but Hermes itself, all its runtime dependencies, system packages, and model weights would be gone. Your agent would boot to a broken environment.
|
||
</p>
|
||
|
||
<h2>How Full-Disk Snapshots Solve It</h2>
|
||
|
||
<p>
|
||
The only way to guarantee an agent restores to a <strong>working state</strong> is to back up its <em>entire filesystem</em> — not just the home directory, and not just config files. Here's how deduplicated snapshot tools like Borg solve the practical concerns:
|
||
</p>
|
||
|
||
<h3>1. Size isn't a problem — deduplication makes it efficient</h3>
|
||
<p>
|
||
Our 6.8 GB agent compresses and deduplicates down to roughly 1–2 GB for daily snapshots. Borg breaks the filesystem into chunks and only stores what changed — so a daily backup after the initial snapshot is often just <strong>50–200 MB</strong> of deltas.
|
||
</p>
|
||
|
||
<h3>2. Full-disk means full recovery</h3>
|
||
<p>
|
||
When you restore, you get back <em>everything</em>: the Hermes runtime, all Python packages, model weights, cron schedules, system config, apt state, and your agent's custom skills. The agent boots and <strong>works immediately</strong> — no reinstallation, no rebuilding.
|
||
</p>
|
||
|
||
<h3>3. Compression reduces storage cost</h3>
|
||
<p>
|
||
Borg's lz4 compression reduces the 6.8 GB to about 4.2 GB on disk. With 7 daily snapshots retaining the full machine state, you're looking at ~6–10 GB total — negligible for any modern VPS provider at ~$0.02/GB/month.
|
||
</p>
|
||
|
||
<h2>What Derez.ai Does Differently</h2>
|
||
|
||
<p>
|
||
Every Derez.ai agent instance comes with <strong>automatic full-disk snapshots</strong> — not home-directory backups, not config exports, but full filesystem snapshots.
|
||
</p>
|
||
|
||
<ul>
|
||
<li><strong>Daily automatic backups</strong> — every 24 hours, we snapshot the entire agent filesystem</li>
|
||
<li><strong>7-day retention</strong> — you can restore to any of the last 7 snapshots</li>
|
||
<li><strong>One-click restore</strong> — from the dashboard, pick a backup and restore. Your agent is back to its working state in minutes</li>
|
||
<li><strong>Encrypted at rest and in transit</strong> — your data never leaves encrypted</li>
|
||
</ul>
|
||
|
||
<div class="pro-tip">
|
||
<p><strong>Real talk: </strong>Most agent hosting services back up your config and call it a day. They don't tell you that restoring won't work because the runtime is missing. We built full-disk snapshots into Derez.ai from day one because "it works after restore" is the only standard that matters.</p>
|
||
</div>
|
||
|
||
<h2>Better Safe Than Sorry</h2>
|
||
|
||
<p>
|
||
The 69% problem isn't a theoretical edge case — it's what happens to every agent that runs real workloads. Every skill, every integration, every model download pushes more state outside the home directory.
|
||
</p>
|
||
|
||
<p>
|
||
If your agent hosting service backs up only <code>/home</code> or <code>/root</code>, test this: delete a major runtime dependency (say, <code>/usr/local/lib/python3.11</code>), then restore from their backup. If the agent doesn't work, you're paying for a false sense of security, not actual backup.
|
||
</p>
|
||
|
||
<p>
|
||
At Derez.ai, your agent's full disk gets backed up every day. No gaps, no missing runtimes, no surprises on restore.
|
||
</p>
|
||
|
||
<div class="cta-box">
|
||
<h3>Try it yourself — first month free</h3>
|
||
<p>Deploy a Hermes agent with automatic full-disk backups. Use code <strong>blog950</strong> for your first month free.</p>
|
||
<a class="btn" href="https://derez.ai/#pricing">Get Your Agent</a>
|
||
</div>
|
||
|
||
<p style="margin-top:48px;padding-top:24px;border-top:1px solid #1a1a24;font-size:0.85rem;color:#666;">
|
||
<a href="https://derez.ai" style="color:#00f5ff">Derez.ai</a> — Work with your AI agent in 10 minutes. Full-disk backups, SSH access, managed dashboard included.
|
||
</p>
|
||
</div>
|
||
</body>
|
||
</html> |