Act IV: The Joinery
The Cheapest Middleware Is The One You Never Wrote
There is a kind of software that does not show up in the demo, does not appear on the storefront, and does not have a single screen a customer will ever see, and yet it is the software most likely to wake you at three in the morning. It is the connective tissue—the code that sits between your Shopify store and the other systems your business cannot live without, translating one’s idea of an order into another’s, carrying inventory from the warehouse system into the catalog, pushing the customer who just checked out into the ERP that will eventually bill them. We call it middleware, and at enterprise scale it is sometimes the only honest answer to a real problem. It is also, almost always, more expensive than the version of you reading the proposal believes it will be. The whole of this chapter can be compressed into a single line that we would like you to carry out of here and repeat at planning meetings until your colleagues find it irritating: the cheapest middleware is the one you never wrote.
We mean that as encouragement, not scolding. There are two situations where middleware is genuinely justified, and they are worth naming plainly so you can check your own case against them. The first is when a system you depend on has no native way to talk to Shopify—a third-party vendor, a logistics provider, an ERP from a vintage when integration meant a nightly file on an FTP server—and someone has to write the thing that makes the two systems speak. The second is subtler and more common than people admit: you already have an integration, off-the-shelf or inherited, and it technically works, except that the data coming out of it is wrong in a way that quietly poisons everything downstream. Both of these are real. Both of these justify building. Neither of them justifies building more than the case demands, and the reason that distinction matters so much is the thing the epigraph was warning you about.
Middleware does not get complicated gradually. That is the trap. You build the first integration—Shopify to the warehouse, say—and it is clean and small and a single engineer holds the whole of it in their head, and it works, and everyone is pleased, and you reasonably conclude that the next integration will cost about the same. It will not. Somewhere around the third system, the thing crosses a threshold that nobody marked on the map, and the cost stops being additive and starts being something closer to multiplicative. The data model that the warehouse uses and the data model the CRM uses and the data model Shopify uses do not agree, and now your middleware is not connecting systems, it is adjudicating between them, holding three slightly incompatible versions of the truth and deciding, on every record, which one wins. Caching that was fine becomes a source of subtle staleness. Visibility that you never needed becomes the thing you would trade a great deal to have. This is complexity creep, and the word creep undersells it, because creeping implies you can watch it happen. You mostly can’t. It feels controllable right up until the discontinuity, and then everything is suddenly demanding at once.
The useful picture, if you want one to hold in your head, is a puzzle piece. Your middleware is a custom-cut piece with knobs and holes—the knobs and holes are the channels through which it talks to each system—and it has a color, which is its data model. For the picture to work, the knobs have to fit the neighboring holes and the colors have to harmonize. Add one connection and you re-cut one edge. Add another and you re-cut another, and adjust the color slightly so it still fits the first, which now means re-checking the first edge, and so on. Every new system is not a new piece sitting tidily beside the others; it is a re-cutting of the one piece you already have, and the piece only gets more intricate, never simpler. That is why the second integration is not like the first, and why the fifth is nothing like either.
So before anyone writes a line, the work is to make the thing smaller—or, ideally, to make it not exist. Three disciplines do most of that work, and they are worth more than any architectural cleverness that comes after.
The first is to challenge the necessity, honestly and somewhat ruthlessly. Before you commit to building, exhaust what your existing vendors can already do—read their documentation, call their support, find out whether the capability you are about to build from scratch is sitting unused behind a settings page you never opened. Then build the business case, and build it with the maintenance tax included, because middleware is not a thing you write once; it is a thing you own forever. If the integration is a genuine convenience rather than a pressing need, the correct move is frequently to wait. (You are not a tech startup. You are a brand that sells things, and every dollar you don’t spend writing connective software is a dollar you get to spend on the part of the business that customers actually pay you for.) And when you do the math, do it the way an earlier chapter asked you to—at twice your current size, not your current size. The integration that is merely annoying to maintain at today’s volume is the one that becomes a second full-time job at double the orders, and the time to discover that is now, on paper, not later, in production.
The second discipline is to map the workflows before you build them—all of them, not just the happy path through the middle. The honest version of this is a workshop, and the honest version of the workshop is long; eight hours is a reasonable floor, not a ceiling, because the whole point is to surface the upstream dependency nobody mentioned and the downstream consumer nobody remembered and the adjacent process that turns out to read the same data for an entirely different reason. Most middleware disasters are not failures of code. They are failures of having understood, on the morning you started, what the data was actually for.
The third discipline is to build the way you’d cross a river you can’t see the bottom of: one foot before the other, rather than a single giant leap. Resist the monolithic delivery—the temptation to specify the whole integration up front and build it all before anything ships. Take one use case, build it, put it in front of the real world, and let what the real world tells you shape the next one. Evolutionary development is slower to look impressive and far faster to be right, because the feedback arrives while the cost of changing your mind is still low.
Suppose you have done all of that, honestly, and the answer is still: yes, build. Fine. From this moment the thing is no longer a project; it is infrastructure. It handles orders and customers and shipments, which means when it falls over, something a customer cares about falls over with it, and you will be judged accordingly. So the question becomes one of shape, and the first fork in the road is between two architectures that look similar on a whiteboard and behave nothing alike in production.
Reactive middleware responds to events as they happen—an order is placed, a webhook fires, your code reacts. It is the right choice when the operation is genuinely time-sensitive, and it is also the choice that bites. Shopify will deactivate a webhook subscription after repeated failures or timeouts, which means a reactive system that cannot keep up doesn’t just fall behind, it gets unsubscribed—the firehose is turned off precisely because you were struggling to drink from it. So a reactive design needs a queue between Shopify and your logic, to absorb the spikes and to keep the events safe while you catch up. And the code on the far side of that queue has to be built for a hostile world: idempotent, so that the same event arriving twice doesn’t bill the customer twice; resilient to race conditions, where two events about the same object arrive close enough together to step on each other; and unbothered by events that arrive out of order, because Shopify does not promise you that causes precede effects. None of this is exotic. All of it is the kind of thing you have to design in from the first day, because retrofitting idempotency into a system that assumed it would never need it is its own small migration.
Scheduled middleware is the quieter sibling. It runs on an interval—every fifteen minutes, every night—and does its work in bulk, and it is very often the better answer when you are integrating with a third-party system that has its own ideas about rate limits and payload sizes. You optimize it by batching, by respecting the limits on both ends, by being a polite citizen of someone else’s API rather than a denial-of-service attack with good intentions. It lacks the drama of real-time, and the lack of drama is the feature. A great many things that feel like they need to react actually only need to be correct by morning, and the cheapest correct system is the one that does its thinking once, on a schedule, while everyone is asleep.
Then there is the question of how many things you deploy, and here we are going to be brief, because the previous chapter already made this argument at length and we have no interest in making you read it twice.
The pitch you will hear—from a conference stage, from the cloud console’s gentle upsells, from the colleague who just finished a very good blog post about service-oriented architecture—is to break the middleware into pieces. A service per integration, a function per concern, each small and independent and scalable on its own. It is an appealing picture and it is, for most enterprise middleware, premature optimization wearing the costume of good engineering. The pieces have to be deployed separately, which multiplies your pipelines. Sharing code between them stops being a function call and becomes a published package with a version number. The network that now sits between your own components can fail, and does, in ways a function call never could. And the data models drift apart across the services until keeping them consistent is its own standing project. You have bought yourself the full operational tax of a distributed system, and you have bought it before you had the scale that would justify a single line of it.
The answer is the one the last chapter taped to your monitor, and we will simply point back to it: the Majestic Monolith. One well-structured, long-running application. Orthogonal changes become atomic because they live in one codebase; the API endpoint and the scheduled job share code by simply being in the same place; one deploy pipeline; no distributed-system complexity to reason about because there is no distributed system. It is, as we said, boring to operate, and that remains exactly the point. The discipline that earned you a clean monolith for your custom app is the same discipline that earns you a clean one for your middleware—same temperament, pointed at a different surface.
This is not zealotry, and we want to be careful not to sell it as such. Microservices have legitimate homes. A piece of middleware so trivial that it is bound to exactly one webhook and does exactly one thing can reasonably live alone—there is nothing to share and nothing to keep consistent, so the monolith buys you little. And occasionally a slice of your business logic genuinely needs a particular technology—a library, a runtime, a model—that doesn’t belong in the main application, and isolating it is the honest move. The rule is not never split. The rule is that the benefit of splitting has to clearly and demonstrably outweigh the complexity it adds, and that the burden of proof sits with the split, not with the monolith. Start whole. Carve off a piece only when you can point at the graph that demands it.
Everything to this point assumes you are building. There is a respectable alternative we have so far only gestured at, and for a particular kind of brand it is the right call: you can buy the plumbing instead of laying it yourself. The dominant name here is Celigo, an iPaaS—an integration platform as a service—which is a long way of saying centralized plumbing for your tech stack. Instead of writing code, you configure flows: a flow listens for an event (a new Shopify order), transforms the data into the shape the destination expects, and creates the corresponding record in the system on the other end (a sales order in your ERP). No deploy, no codebase, no on-call engineer at three in the morning. For the right brand, that trade is excellent.
The right brand has a fairly specific silhouette. It runs NetSuite, because Celigo’s deepest, most maintained connectors point at NetSuite—the two companies’ offices are, by reputation, close enough that you can draw your own conclusions about why the NetSuite story is so unusually strong—and it is large enough, call it fifty million in revenue and up, that integration failures cost real money and the platform’s price is noise against that cost. It has many systems to connect rather than few, because Celigo’s value compounds with connection count: consolidating Shopify, NetSuite, your ESP, your marketplaces, and your warehouse in one place is genuine operational leverage. And, crucially, it has someone whose job includes owning the thing.
Because the cost structure of iPaaS is mostly hidden, and it is hidden in places that flatter you on the way in and bill you on the way out. The “no-code” promise is the first one to read carefully: the visual builder makes simple things genuinely simpler, but it does not eliminate the need for technical competence—a complex integration is still a complex integration, and you are now building it in a drag-and-drop interface instead of a text editor, which is a different activity, not an easier one. Testing is the second: there is no unit test, no CI pipeline, no green checkmark before you ship; you are, in the plain sense, testing in production, and the discipline you took for granted in your codebase has to be reconstructed by hand. The third is the one that does the quiet damage. Because flows are so easy to create, they multiply, and because nobody owns the multiplying, they are never pruned. We have walked into brands running more than forty active flows where twelve were actually necessary—the rest a sediment of duplicates, abandoned experiments, and one-off workarounds that someone built on a deadline and nobody ever turned off. Each one is technical debt and a line on the bill. And the failures, when they come, are silent: data simply stops syncing, and no alarm rings, and you find out when inventory is wrong or an order has gone missing, which is to say you find out from a customer. The longer all of this runs, the more the flows accumulate, and the more expensive it becomes to ever leave—which is the lock-in, arriving not as a decision but as a slow accretion.
So custom wins over Celigo in the mirror-image situation: when you have only two or three systems to connect and their needs are simple; when you have a strong in-house development team that would rather have version control and code review and a real test suite than a visual builder; when your workflows are highly customized and represent something close to a competitive advantage; and when code quality and test coverage are not negotiable in your culture. And if you do choose Celigo, the price of keeping it healthy is operating discipline that mirrors how you’d treat any critical system: assign it a clear owner rather than letting the whole company edit it; treat every flow change like a code commit, with a note on what changed and why and what you expected it to do; audit the whole estate quarterly and disable what nobody needs before the sediment builds; and configure real alerts on the mission-critical flows, so that the silent failure has something to break the silence. Celigo earns its keep when your competitive advantage lives somewhere else entirely—when data synchronization is necessary infrastructure but not the thing that distinguishes you. If that’s true, paying someone else to run the plumbing is exactly right, and it frees you to spend your engineers on what actually sets the brand apart.
Which lands us back where the epigraph started, only now you have the whole map behind the warning. Middleware seems easy because the first piece is easy, and the first piece is honestly telling you the truth about the first piece and lying to you about all the rest. Whether you build it or buy it, the thing you are signing up for is not a feature; it is a standing obligation, a small piece of infrastructure that will demand attention for as long as the business runs on it. So challenge the necessity before you commit. Map the whole workflow before you build. Build one foot at a time. Keep it whole until the graph forces you to split it. And run the math at twice your size, because the integration that is cheap today is the one that quietly becomes a salary tomorrow.
Here lies the integration nobody owned: forty flows, twelve of them load-bearing, all of them syncing faithfully right up until the morning they didn’t.
The cheapest middleware is the one you never wrote. The second cheapest is the one you wrote on purpose, kept small, and could explain to a new hire in an afternoon. Everything past that, you are paying for whether the line item says so or not.