With any technology, companies perpetually face a decision: build or buy?
Integrating external data is no exception. External data is hard. A special kind of hard. Yet many companies choose to do it themselves. The process is long and costly, involving having to manage a variety of vendor relationships, navigate documentation and licensing agreements, build bespoke connectors for a variety of filetypes, and figure out what went wrong and what to do when you face the inevitable schema break.
The result is that an organization can expect 70% of its data teams’ time to be spent on anything other than drawing data insights.
Data-engineering teams are constantly striving toward optimization in order to expand capacity and find alpha. 70% of a team’s time just to prep and maintain data—without drawing any actual value from it—is suboptimal.
Moreover, integrating external data is unlikely to be any particular organization’s core competency. Trying to manage external data on one’s own, therefore, fundamentally means that the organization must run up against a capacity issue; at some point, every available data engineer will be burnt out—doing everything they can do with every available minute and every available resource just to keep the proverbial lights on.
(This is to say nothing of the fact that data engineers—of which there is a talent shortage—generally didn’t get into data engineering just so they could spend most of their time prepping data for integration.)
To continue to keep external-data integration in house—to continue to build—means additional investment. But external data is limitless—growing exponentially and frequently changing. At some point in the near future, the organization will face a capacity issue again—and, more or less, remain at capacity.
The problem is that, unlike data, capital is not infinite.
The Crux of the matter
If building doesn’t work, what about buying?
There are other questions to answer first:
How much have you spent on integrating external data? How much do you plan to spend? How much would be too much for you to spend?
Are any of those answers anywhere close to $160+ million?
That’s not a figure we plucked out of the air. It represents how much Crux has already raised and committed to solving the external-data problem.
We have invested over $160 million in catching and resolving schema differences and schema changes.
We have invested over $160 million in building a variety of connectors to onboard over 300 data sources.
We have invested over $160 million in making more than 60,000 datasets integration-ready.
We have invested over $160 million in developing close partnerships with our network of data suppliers, so that we and they are ready to act immediately the moment a customer’s pipeline breaks.
And we’re investing even more to improve our suite of technology and services, bridging the gap between data suppliers and enterprises to accelerate the time-to-analysis and the time-to-value of external data.
Our customers would never spend that much on external-data integration. That is a good and natural thing. Integrating external data isn’t why they got into business; their business is why they got into business.
This is the message we have been focused on spreading since we started Crux in 2017. When you augment your data operations with Crux, you get unmatched scalability and capacity. This leaves your DataOps teams free to do the exciting and fulfilling work you hired them for—analyzing data and drawing critical insights that drive business value.
So, build, or buy?
You could build…but why would you want to?
(Not convinced? Let us know here.)
Crux: Jan 2, 2024
