Here’s an org chart that exists in almost every large company, told as a small tragedy in three acts.
There’s a centralized file-transfer team. They run the platform – the schedulers, the SFTP servers, the folders. They’re good at it. Dozens of teams across the company “order” transfers from them: every night at 02:00, move the price file from us to the retailer; every hour, pull the orders file from the logistics partner. The central team sets it up, and from that moment the transfer belongs to everyone and no one.
Act one. A transfer fails. In the best case the platform logs an error – somewhere, in a log the central team owns and the ordering team has never seen and couldn’t access if they wanted to. In the worst case there’s nothing to log at all: the job that should have run simply didn’t, and no error exists anywhere because there was no attempt. You can’t grep for the absence of an event.
Act two. Nothing happens. The consumer team that ordered the transfer has no window into it. Their first monitoring signal is a customer: a retailer calling about yesterday’s prices, a warehouse asking where the orders went. The customer is, functionally, the alerting channel, with excellent reach and terrible latency.
Act three. The consumer team opens a ticket with the central team. The central team, who manage hundreds of transfers and heard about this one just now, digs through logs, finds the failure from eleven hours ago, reruns the job. Everyone involved is competent. Everyone involved is frustrated. The whole loop cost half a day, and it will run again next month with a different file.
The root cause isn’t the platform and it isn’t the people. It’s that visibility follows ownership of the infrastructure, when it should follow ownership of the outcome. The central team owns the folders; the consumer team owns the consequence.
The fix: telemetry out of the folders, visibility per team
The good news is that the folders can speak for themselves: with off-the-shelf OpenTelemetry, no changes to the transfer platform.
The OpenTelemetry Collector’s file stats receiver watches a directory and emits metrics about the files in it: how many, how big, how old. Point it at each transfer’s drop folder, and every scheduled transfer becomes an observable fact: the price file arrived at 02:04 and it’s 3.1 MB. Or, the more valuable version: it’s 06:30 and the price file still isn’t there.
One Collector on the file server covers the lot. We’ve written up the full configuration in Monitor a file-drop integration (hot folder) with OpenTelemetry – the short version is one receiver per folder pattern and an OTLP exporter, and the central team only sets it up once. Nothing is installed on the sending or receiving side, nothing about the transfers themselves changes.
That gets the facts flowing. The second half is making the facts land with the right people.
Each transfer becomes an integration; each team gets its own view
In Sluicio, each file-transfer type is modeled as an integration – “nightly price file to retailer X” is a thing with a name, a state, and a history, not a row in the central team’s scheduler. And each consumer team becomes a group in Sluicio, seeing exactly the integrations they’ve ordered. The price-file team sees price files. The orders team sees orders. Nobody wades through four hundred transfers that aren’t theirs.
For every transfer type, the team gets two things they never had:
State they can see. Did last night’s run happen? When did the file land, and how big was it? A glance answers the question that used to be a ticket.
Alerts that beat the customer. One rule does most of the work: the file hasn’t landed by its deadline (the 02:00 file, still absent at 02:30). It’s the answer to the worst case from act one: you can’t log a job that never ran, but you can notice a file that never arrived. We keep a second, gentler rule beside it for a run that produced no file at all. Alert rules in Sluicio route to the team’s own notification channels – so the price-file team’s own channel pings at 02:30, not the central team’s inbox at 09:00 via a customer at 08:45.
What changes for each side
For the consumer team: they find out first. When they contact the central team, the conversation starts at “the 02:00 transfer to retailer X didn’t land” – with a timestamp – instead of “a customer says something’s wrong with the prices, can you check everything?”
For the central team: fewer archaeology tickets. The requests that do arrive are precise, and the transfer platform’s reputation improves without the platform changing at all – because failures stop marinating for eleven hours before anyone looks.
For the customer: they go back to being customers. Unpaid monitoring duty ends.
None of this required agents on partner systems, changes to the scheduler, or the central team giving anyone access to their logs. The telemetry was always sitting there in the folders; it just needed a standard way out (OpenTelemetry) and a place that models it the way the organization actually works (integrations owned by teams).
Try it on one transfer
Pick the transfer that generates the angriest phone calls. Point a Collector’s file stats receiver at its drop folder, make it an integration in Sluicio, hand it to the group that owns the outcome, and set the alert rules. The next failure will be a ping in the team’s own channel at 02:30 instead of a customer call at 08:45 – and that difference is the entire argument.
Links:
- Full Collector setup: Monitor a file-drop integration (hot folder) with OpenTelemetry
- Docs: sluicio.com/docs