Adding Langfuse Observability to Hermes Agent (Self-Hosted, the Hard Parts Included)

Langfuse observability for Hermes Agent is the way to see, per query, what your self-hosted agent actually does: which model it called, how many tokens it burned and what it cost. Hermes Agent is not built on LangChain, so it reports traces through Langfuse’s native OpenTelemetry support instead of a framework plugin. In this post I wire self-hosted Langfuse into a self-hosted Hermes Agent on a single Hetzner VPS, and share the gotchas that only show up when you actually run it.

My Hermes agent runs 24/7 on my own VPS. It answers me on Telegram, fires cron jobs at 3 AM and swaps between half a dozen models depending on the task. Here is the embarrassing part: I had no idea if it was costing me ₹50 a month or ₹5,000. For someone who works in AI observability, being blind to my own agent’s spend felt a bit like a dentist walking around with a toothache.

I use the OpenCode provider for Hermes. My default profile runs on deepseek-v4-flash, which is cheap per token. But I also keep separate profiles like legal and coding that use pricier models like GLM-5.2, Minimax and DeepSeek-V4-Pro. OpenCode gives me usage per model, but never per query. To close that gap, I plugged Hermes into Langfuse.

1. Why observe a personal agent at all

I have 15+ cron jobs running round the clock. Some fire only in the dead of night between 2 and 5 AM, some every 30 minutes, some only on Sundays. I wanted to know which of these were quietly eating the most tokens, and which tools the agent kept reaching for and how often. This is exactly the visibility Langfuse gives me.

2. The stack

My whole setup sits on one Hetzner CX43 VPS (8 vCPU, 16GB RAM, 80GB NVMe) running Coolify to manage containers. Hermes runs in a Docker container, and Langfuse is self-hosted right next to it on the same box. That means I run the observability stack myself instead of shipping my traces to a third-party SaaS dashboard, and I don’t pay a rupee for it. To be precise: my LLM calls still go out to the model providers and Langfuse offloads trace blobs to Cloudflare R2, so this is about keeping control of the Langfuse deployment, not pretending that no byte ever leaves the box. The diagram below shows how the pieces fit together.

    flowchart LR
    User["Me
(Telegram, CLI, cron jobs)"] --> Hermes

    subgraph VPS["Hetzner CX43 VPS (Coolify)"]
        direction TB
        Hermes["Hermes Agent
(Docker container)"]
        Langfuse["Langfuse
(self-hosted)"]
        Hermes -- "OTLP traces
(OpenTelemetry)" --> Langfuse
    end

    Hermes -- "LLM calls" --> Providers["Model providers via OpenCode
(DeepSeek, GLM, ...)"]
    Langfuse -- "blob storage" --> R2["Cloudflare R2"]

3. Setting up Langfuse

Though, Coolify has a one-click install for Langfuse, It did not work for me due to S3 dependency. So, I had to tweak the docker-compose.yml file to remove the S3 dependency, still it did not work. Finally, I used Cloudflare R2 as blob storage by setting the following env vars in the Langfuse container:

• LANGFUSE_S3_EVENT_UPLOAD_BUCKET, your bucket name in R2
• LANGFUSE_S3_EVENT_UPLOAD_REGION, your R2 region. I have set it as auto.
• LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID, your R2 access key
• LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY, your R2 secret key
• LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT, your R2 endpoint. I have set it as https://<account_id>.r2.cloudflarestorage.com. Replace <account_id> with your Cloudflare account ID.

Rest all env vars can be set as per the Langfuse documentation. After setting these env vars, I was able to successfully run the Langfuse container.

You can create a new organization in Langfuse and get the public key, secret key and base URL for your self-hosted Langfuse instance. These will be used in the next step to enable the Langfuse plugin in Hermes Agent.

4. Enabling the plugin

Let’s move our attention to the Hermes Agent. We need to enable the Langfuse plugin in the Hermes Agent. You can do this by running the following command in the terminal inside the Coolify container or from your Chat interface of Hermes Agent (Telegram for me).

hermes plugins enable observability/langfuse

This command enables the Langfuse observability plugin for the Hermes Agent. In Telegram you can directly ask it in natural language like “Enable Langfuse plugin” and it will enable the plugin for you.

You can verify that the plugin is enabled by running:

hermes plugins list

It should show like above screen shot. Note that the plugin is nested under observability/, so make sure to use the full path observability/langfuse when enabling it.

One thing the enable command does not do is install the Langfuse Python SDK. On a clean Hermes install the plugin imports langfuse at startup, and if the SDK is missing it quietly turns itself into a no-op: no error, no traces, nothing to grep for.

The obvious move is a quick pip install langfuse inside the container. It works, until it does not. The official Hermes image only persists /opt/data, so anything you install into the container layer gets wiped the next time Coolify recreates or upgrades the service, and tracing silently dies again. I learned this the annoying way. Make the install durable instead. Two options that survive a redeploy:

1. Derived image (my pick): build a tiny image on top of the one Coolify already deploys and bake the SDK in.

# Base this on the exact Hermes image tag Coolify deploys
FROM <your-hermes-image>:<tag>
RUN pip install --no-cache-dir langfuse

Point the Coolify service at this image and the SDK is there for good.

2. Install into the persistent volume: install the SDK somewhere under /opt/data and set PYTHONPATH so the gateway’s Python finds it on every boot, redeploy or not.

Either way, prove it survives a redeploy, not just the first start.

5. Env vars, and getting them to actually reach the process

This is the most important section of this post. If you set this up correctly 90% of the work will be done.
The Langfuse plugin reads its configuration from environment variables. Hermes loads these from its own env file at ~/.hermes/.env (see the Hermes environment-variable reference). My mistake early on was dropping the variables into an arbitrary project .env that Hermes never reads. On a Coolify deployment the cleanest option is to set them directly as container environment variables, so they reach the Hermes process no matter where it starts from.

These are the three variables the bundled Hermes plugin itself uses:

• HERMES_LANGFUSE_PUBLIC_KEY
• HERMES_LANGFUSE_SECRET_KEY
• HERMES_LANGFUSE_BASE_URL You will get these from the Langfuse dashboard after creating a new organization. The public key and secret key are used for authentication, and the base URL is the URL of your self-hosted Langfuse instance.

Those three are the whole configuration the bundled plugin needs. It builds the Langfuse client straight from these values and creates its own exporter from the base URL and credentials. There are no extra transport variables to set for the plugin itself.

So when traces did not show up for me at first, reaching for more variables was the wrong instinct. An empty dashboard is almost always upstream of configuration: the langfuse SDK is not installed (Section 4), the gateway was not restarted after the change, or the base URL is wrong. That is where to look first.

A note on the OTLP variables

While I was flailing around trying to get traces, I also had these OpenTelemetry variables set on my container, and I cannot cleanly say which change finally flipped tracing on:

• OTEL_EXPORTER_OTLP_ENDPOINT, like https://your-langfuse-base-url/api/public/otel
• OTEL_EXPORTER_OTLP_HEADERS, like Authorization=Basic ${AUTH_STRING},x-langfuse-ingestion-version=4
  • Get the base64 AUTH_STRING with echo -n "pk-lf-...:sk-lf-..." | base64 (add -w 0 on GNU systems, base64 auto-wraps long keys).

Be honest with yourself here, because I was not at first: the bundled Hermes plugin does not read these variables. It configures Langfuse from the three HERMES_LANGFUSE_* values above. The OTEL_EXPORTER_OTLP_* variables only matter if you are wiring up separate, generic OpenTelemetry instrumentation that ships to Langfuse’s OTLP endpoint. If your three plugin variables are right and the SDK is installed, you do not need them.

If you do use that generic OTLP path, mind the endpoint: behind a normal Coolify HTTPS proxy, use your public base URL with no port (https://your-langfuse-base-url/api/public/otel). Port 3000 is the internal container port, so http://localhost:3000/api/public/otel is only for a direct local deployment.

Other than these, there are optional env vars that can be set:

• HERMES_LANGFUSE_ENV
• HERMES_LANGFUSE_RELEASE
• HERMES_LANGFUSE_SAMPLE_RATE
• HERMES_LANGFUSE_DEBUG
• HERMES_LANGFUSE_MAX_CHARS

I did not set these optional env vars.

Here is the explanation of the env vars.

6. Verifying it works

After setting the environment variables, you can verify that they reached the process by checking the /proc/$GWPID/environ file. First capture the gateway PID into GWPID, then grep for the variable name only, so you confirm the variable is present without printing its value:

GWPID=$(pgrep -f "hermes gateway run" | head -n1)
grep -c HERMES_LANGFUSE_SECRET_KEY /proc/$GWPID/environ

You can also confirm the variables are set in the Coolify container settings for the Hermes Agent. Remember to restart the gateway after changing env vars with hermes gateway restart in the terminal inside the Coolify container.

7. What you actually see

7.1. Dashboards

Ask a few questions to the Hermes Agent and you will see the traces in the Langfuse dashboard. You can see the multi-turn session, tool-call spans, and token/cost breakdown across providers. Here are a few screenshots of the Langfuse dashboard showing the traces of the Hermes Agent:

Langfuse homepage showing Hermes Agent traces

Langfuse out of the box provides three dashboards: Cost, Usage and Latency. You can see the cost dashboard below:

Langfuse Cost Dashboard

Langfuse Usage Dashboard

Langfuse Latency Dashboard

7.2. Traces

Langfuse Traces Homepage As you can see from the screenshot all are my cron jobs running at different time intervals. You can click on any of the traces to see the details of the trace. Here is a screenshot of a trace showing the multi-turn session, tool-call spans, and token/cost breakdown across providers:

Detailed Trace showing multi-turn session, tool-call spans, and token/cost breakdown across providers

7.3. What the traces actually told me

This is the part I actually cared about. Setting up the dashboards is plumbing. The payoff is the answer to the question I started with: where is my money going? After letting the traces pile up for a week, here is what I found.

• The cron flood: I already keep a regular eye on my cron jobs, but the traces told a different story. They are absolutely flooded with cron runs. Clearly I need to dial down how often some of them fire. Cron Jobs Graph

• Too many granular tool calls: The agent was processing each comment on a GitHub issue as a separate step. More comments meant more LLM calls, which meant more cost. A single issue turned 16 terminal tool calls into 10 LLM calls. I need to batch them so the model runs once, not once per comment. 16 Terminal Tool Calls which led to 10 LLM calls

• The blind spot I still cannot explain: The OpenCode usage graph shows the minimax-m3 model ran on 13th July and burned through my entire monthly Go plan limit in one go. I still cannot pin down where exactly that model got called from. Getting granular per-model attribution is on my list. Opencode Usage Graph

The moment my monthly limit resets, I will remove the wasteful cron jobs and cut the frequency of the ones burning the most tokens and tool calls. That one change alone should save me a good chunk. This is one of the reason to set Observability for your agents.

8. Ops footnotes

As an AI Observability Architect by profession, I know setting up the observability tools are just the beginning. The real value add comes from analyzing the traces and making improvements to the agent. However, I also know that observability tools can generate a lot of data and can become an ops problem if not managed properly.

For example, I will set the HERMES_LANGFUSE_SAMPLE_RATE to 0.5 to reduce the number of traces sent to Langfuse and reduce the cost. I will update this post if I do that and see any difference in the traces. This is one of the reasons my Hetzner disk was filling up fast at ~1.5GB per day.

Future Work

• You might have seen the blank score widgets on the dashboards. I will be working on integrating the Evals and prompt management for Hermes with Langfuse. This will allow me to see the scores of the prompts and the evaluations of the responses in the Langfuse dashboard. I will start with a specific set of prompts and evaluations. Cost is the main blocker.
• I will be integrating with Slack to get the notifications of the traces in the Slack channel. This will allow me to see the traces in real-time and take action if needed.
• Custom dashboards for my specific needs. I want further nuances dashboard for token cost and usage.
• Data retention and expiration policies for the traces. I want to keep the traces for a specific period of time and then delete them to save disk space.

Lessons Learned

1. Put env vars in Coolify, not a .env file: The .env file inside the container did not reach the Hermes process. Setting the variables directly in the Coolify environment section was the only thing that worked reliably. When in doubt, put config where the process actually reads it, not where it looks tidy.
2. An empty dashboard is not a variables problem: When the three keys gave me nothing, my instinct was to add more variables. Wrong instinct. The bundled plugin reads only the three HERMES_LANGFUSE_* values. An empty dashboard is almost always the missing SDK, a gateway that was not restarted, or a wrong base URL. Look there before you touch anything else.
3. Restart the gateway after every change: New env vars do not apply to a running gateway. Run hermes gateway restart (or ask Hermes to do it) or you will keep debugging a process that never picked up your changes.
4. The one-click install lied to me: Coolify’s one-click Langfuse needs S3-style blob storage, and it did not just work. Cloudflare R2 with the LANGFUSE_S3_EVENT_UPLOAD_* variables was the fix. Budget an hour for storage config, not five minutes.
5. Container overlays are temporary, volumes are not: On Coolify, anything outside the mounted volume gets wiped on redeploy. Install and persist accordingly.
6. The observer needs observing too: Self-hosted Langfuse ate disk at about 1.5GB per day and filled up my VPS. Observability is not free. Plan sampling and retention from day one, not after the disk alert.

Frequently Asked Questions

Can I add Langfuse to a custom agent that does not use LangChain?

Yes. Hermes ships a bundled Langfuse plugin that you enable and configure with the three HERMES_LANGFUSE_* variables, no LangChain required. For a different custom agent that has no such plugin, Langfuse also accepts traces over generic OpenTelemetry (OTLP), so any OTel-instrumented app can report to it without framework lock-in.

Why are no traces showing up in Langfuse even though the plugin is enabled?

This was my exact problem, and the fix was not more variables. The bundled plugin reads only the three HERMES_LANGFUSE_* values. If the dashboard stays empty, work through this order: confirm the langfuse SDK is installed and survives redeploys, restart the gateway with hermes gateway restart, then check the env vars actually reached the process and the base URL is correct. Adding OTLP exporter variables does not help, because the plugin does not read them.

Does Langfuse track cost for non-OpenAI models like DeepSeek or GLM?

Yes. My Hermes profiles run on deepseek-v4-flash, GLM-5.2 and DeepSeek-V4-Pro through OpenCode, and Langfuse breaks down cost and token usage per model. You may need to check that pricing is configured for less common models, but the provider-agnostic cost view is the main reason I set this up.

Do I have to self-host Langfuse or can I use the cloud version?

You can use Langfuse Cloud, but I self-host it on the same Hetzner VPS as Hermes. That keeps all my agent data private and costs nothing beyond the server I already run. The tradeoff is you own the ops: storage, disk growth and retention are now your problem.

How much disk does self-hosted Langfuse use?

More than I expected. On my setup it grew at roughly 1.5GB per day and was one of the main culprits filling up my VPS. Plan for sampling with HERMES_LANGFUSE_SAMPLE_RATE and a retention policy before you run out of disk, not after.

Why did the Coolify one-click Langfuse install fail for me?

It expects S3-style blob storage and would not start out of the box. I fixed it by pointing it at Cloudflare R2 using the LANGFUSE_S3_EVENT_UPLOAD_* environment variables with the region set to auto and the R2 endpoint URL. After that the container came up cleanly.

How much does this setup cost me?

• Hetzner VPS costs around $20/month but you can settle in for a lower spec VPS if you want to only run Hermes and Langfuse.
• OpenCode Go subscription costs $10/month.
• Cloudflare R2 is free for the first 10GB of storage and 1 million requests per month, which is more than enough for my Langfuse traces.
• Therefore, in total this setup costs me around $30/month, which is reasonable for a self-hosted AI agent with observability. As I do self host other services this cost can be managed at $20/month if I use a lower spec VPS and manage the storage better. If you run locally with local LLMs, you can avoid the VPS and LLM cost entirely.

Related Articles

• The Trends of Engineering in AI: Prompt, Context, Harness, Loop, and Evaluation: the five disciplines behind production agents, including the observability layer this post plugs into.
• Weekly Notes: Week 27, 2026: where I traced the runaway disk growth back to Hermes, OpenClaw and Langfuse and fixed it.
• Weekly Notes: Week 26, 2026: the full self-hosted stack running on my Hetzner VPS via Coolify.

References

• Hermes Agent docs for Langfuse Integration
• Langfuse docs for Hermes Integration
• Langfuse docs for OpenTelemetry Integration
• Coolify docs for Hermes Agent
• Coolify docs for Langfuse