I gave my family an AI butler — self-hosted on Hermes Agent

Every chat assistant I’ve used is a goldfish. ChatGPT, Claude, Copilot — brilliant in the moment, amnesiac the next. You can’t say “remember that appointment I mentioned last week” and have it actually know. And the ones that are supposed to remember — Siri, Alexa — live in walled gardens and can’t touch my files, my email, or a cron job.

So I built my own. A persistent agent that lives on a server, reaches me on Telegram, remembers my family’s life, and quietly gets better the more I use it. It runs on Hermes Agent, it’s self-hosted, and it costs less than a Netflix subscription.

Here’s how it’s wired and what I learned.

Why Hermes (and not a coding agent)

I already use coding agents daily, but they’re the wrong shape for this. They live inside an editor and have no memory between sessions, no scheduled tasks, no way to message me on my phone. DIY frameworks (LangChain, AutoGPT) can do it, but you build the memory, the scheduler, and the platform adapters yourself.

Hermes sat in the gap: persistent memory, a multi-platform gateway, built-in cron, provider-agnostic models, and — the part that sold me — a self-improving skill loop. When it solves a hard problem, it can save a skill. When it learns something about me, it persists it. Over weeks, it genuinely gets better at the jobs I give it. And it’s MIT-licensed, so I can read every line.

The shape of it

You ── Telegram (phone) ──► Hermes Gateway (systemd service)
                                   │
                                   ▼
                       AIAgent (conversation loop)
              model: DeepSeek V4 Flash · memory: mem0 + Qdrant
                                   │
        ┌────────────┬────────────┼────────────┬──────────────┐
        ▼            ▼            ▼            ▼              ▼
     Terminal   File system   Web tools   Browser      Email cron
                                                       (Gmail intake)
                                   │
                                   ▼
            Storage: everything in ~/.hermes/ on the server
       SQLite sessions  ·  Qdrant vector memory  ·  skills library

A few decisions that mattered:

• Telegram is the front door. Always-on, push notifications, voice messages (transcribed locally with Whisper), file sharing. The CLI is only for debugging.
• DeepSeek V4 Flash as the default model. Fast time-to-first-token, cheap, and reliable at tool-calling. Good enough for assistant work without paying frontier prices on every query. And because Hermes is provider-agnostic, switching models is one command if it’s ever down.
• All data stays in ~/.hermes/ on the server. No cloud sync, no telemetry.

The part that makes it real: unlimited memory

Hermes ships with a small built-in memory — two curated files, a couple thousand characters each. For a family assistant tracking appointments, school terms, travel bookings and the occasional lab report, that fills up instantly. One blood-test PDF eats a quarter of the budget.

So the biggest customization was swapping that for a proper vector memory: mem0 for the fact-extraction pipeline, backed by Qdrant running locally in Docker.

vector_store:
  provider: qdrant
  config:
    collection_name: hermes_mem0
    host: localhost
    port: 6333
    on_disk: true

llm:                 # mem0's internal fact-extraction model
  provider: deepseek
  config:
    model: deepseek-chat
    temperature: 0.1

embedder:
  config:
    model: text-embedding-v3   # 1024-dim, via DashScope

Every fact gets stored as an embedding plus text, sorted into categories (family, health, travel, schedules, preferences…). Facts are captured mid-conversation — I don’t have to say “save this.” And recall is semantic, by meaning rather than keyword:

• “When’s the Bali trip again?” → returns the full itinerary.
• “What did the doctor say about my results?” → returns the health record.
• “Any scheduling clashes next month?” → cross-references trips, appointments, and school terms.

Without persistent memory, an AI assistant is just a chatbot with good tools. With it, it starts building an actual model of your life. That’s the whole game.

Teaching it a custom skill

Hermes has ~60 built-in skills, but I needed one for family logistics. I didn’t write it from scratch — it evolved. It started as a one-line “remember family info” instruction, and every time the agent hit a new case (extract dates from a school email, read a lab PDF, compare two people’s schedules), it patched its own skill file. The SKILL.md is now a living record of accumulated experience: intake, appointment tracking, travel itineraries, medical data, and date-conflict checking.

That self-patching loop is the thing I keep coming back to. I correct it once; it updates the skill so it doesn’t need correcting again.

Automated intake from email

The friction in any “personal knowledge base” is the manual entry. So I wired up a Gmail integration with a cron job that checks a dedicated inbox every 30 minutes, reads unseen mail, extracts the useful bits (dates, booking refs, action items), saves them to memory, and archives the email.

Now flight confirmations become travel itineraries, school circulars become term dates, and appointment reminders land in the health schedule — all without me forwarding anything by hand.

How it composes

No single feature is magic; the composition is. A real example:

A lab report arrives as a PDF by email. The cron job picks it up → the agent reads the PDF → categorizes results by body system → flags anything outside the reference range → saves a structured summary to memory. Weeks later, from Telegram: “what did the blood test show?” → semantic search returns the breakdown.

And my favorite, because it shows the agent thinking across its memory:

Me: “Book a doctor follow-up for the 11th of July.” Agent: “You’ve got a return flight landing at 18:30 on the 11th — different date, or a post-trip appointment?” Me: “Ah, I meant the 11th of June.” Agent: “Got it, saved ✅”

Email intake → PDF extraction → structured memory → semantic recall → cross-referencing → conflict detection. None of that exists in any single tool. It emerges from the pieces fitting together.

Privacy, because it knows a lot

A thing that holds your family’s health and travel data has to be handled carefully. Everything lives on the server in ~/.hermes/ — no cloud storage, no telemetry. Hermes redacts secrets from tool output before they hit logs, and destructive commands need manual approval. Queries do go to the DeepSeek API, so identifying details are kept out of prompts where practical. If even that’s too much, Hermes can run fully local models via Ollama or vLLM — you trade some tool-calling quality for absolute privacy.

What it costs

What I’d do differently

• Fact extraction is hit-or-miss. deepseek-chat is occasionally inconsistent about what’s worth saving. A small model tuned for extraction would be better.
• Local models still aren’t there for tool-calling. Full privacy means accepting noticeably lower reliability — for now.
• Health data is free-text. A structured schema would let me ask “how has my cholesterol moved over the year?” properly, instead of leaning on vector search.
• Calendar is one-way. Reading is wired up; two-way sync (the agent writing events) is the next unlock.

Why I bothered

A few years ago this needed a team and a cloud budget. Now it’s a curl | bash install and an afternoon of config. What you get isn’t a chatbot or a copilot — it’s a small digital entity that lives on your machine, knows who you are, remembers what you told it, and reaches you wherever you are.

Built with

• Tool: Hermes Agent (Nous Research, MIT) — gateway on Telegram, custom family-info-management skill
• Model: DeepSeek V4 Flash (default) · deepseek-chat for mem0 fact extraction · Qwen text-embedding-v3 for embeddings
• Memory: mem0 + Qdrant (local, on-disk)
• Hosting: self-hosted on an Azure VM (Ubuntu 24.04), running on the $150/mo Azure credit I get as a Microsoft employee — all data in ~/.hermes/
• Users: me, plus my wife from her own Telegram app on her phone