Maintaining the status quo for data consumption supports stagnancy, not scale.
No one can really predict the future. For example, no matter how hard I try, I cannot predict the weather for an upcoming vacation. Meticulously studying the forecast and noticing every single degree shift can’t really tell me if it will rain on our first day, but that doesn’t stop me from trying.
But we can use data to make predictions, inform hypotheses, and strategize for what’s to come, and in the world of external data we use–you guessed it–more external data to do just that. In our industry, one of those key external datapoints is customer insights. As we watch the marketplace consume data, we can take trends into account to help us create our next product and educate the market.
This exact scenario has led to a prediction (and for some, a current reality) that needs to change. If things stay the same, the future of the external data market is backlogs. The definition of a backlog, according to Google, is “an accumulation of something, especially uncompleted work or matters that need to be dealt with.” For most businesses, that isn’t a very bright future. It’s not too late to shift that prediction into something more favorable, but first it’s important to understand how we got here. This isn’t just something I’ve personally experienced–it’s a common thing I’m hearing in conversations with organizations with increasing ferver lately. Businesses will come to us because they’ve hit capacity for how much data they can onboard at once, and are stuck with these backlogs just collecting virtual dust on their to-do lists.
The Demand for Data Increases with Suppliers
The external data market is relatively new compared to older industries. It can be used across professions and fields to support critical decision making, train your machine learning (ML) and artificial intelligence (AI) models, and predict customer behavior or drive targeted advertising. Its growth runs parallel to the boom of the internet, and as more people prefer to exist on social media and shop online from the comfort of their homes, new external data sources are created daily. Any organization who wanted a competitive edge quickly began consuming this new data source, even without the technology to support it. Now, the technology available in the external data market is growing, and using external data to supplement your business is no longer an edge, but an expectation.
So organizations adapted–they hired data engineers and asked them to maintain pipelines, manage supplier relationships, and otherwise maintain these external data sources. And that worked at first, as organizations used 2-5 sources and scaled their team with the demand for data. But that method hits a limit when the available data sources become exponential, and talent is scarce.
As everyone wants more data, it’s become increasingly clear that businesses will not let capacity limit their ability to consume external data sources, which means data engineers have a continuous, never-ending backlog of new datasets to onboard, clean, transform, and manipulate before it can even be used by other teams to meet their needs.
This level of data preparation work quickly adds up, and creates a roadblock for data engineers to maintain their actual, revenue-driving responsibilities.
Every day, more and more niche data providers join the market and create more competition, create diversity, and ease the burden of access. But the same can’t be said for the data engineers on the other side, as requests for new sources continue to flood their kanban boards. If they’re lucky, their company will onboard new data engineers and expand their data operations budget to attempt to keep up with the demand, but that alone doesn’t scale after a certain point.
A shift in focus from data operations to data insights is the best way to avoid backlogs of the future. But what does that mean?
Shifting Focus from Operations to Insights
A lack of supporting technology for external data integration was a key factor in how we got to backlogs of the future, but it’s no longer an excuse for how businesses maintain their data operations. The budget required to keep up with their current onboarding backlogs and account for future scaling must shift from manpower to technology. By shifting governance, cloud platforms, and tech stacks away from a labor-heavy and value-light, money can be re-invested into technology that simplifies data onboarding, reduces operational expenses, speed, risk, and latency.
Choosing to invest in external data platforms and managed services provided by Crux and similar companies can transform your data operations machine into a data insights one that drives revenue by delivering insightful data faster, and at a better quality.
This shift can also save your data engineers. No one likes a bait-and-switch, and data engineers hired with the promise of analytical, science-based work will only tolerate data maintenance work for so long before becoming burnt out. Shifting your organization’s focus from operations to insights can:
- Reduce the cost of labor
- Create happier employees
- Allow engineers to offboard tedious, monotonous work
- Transform external data from an expense to an ROI driver
- Allow you to re-invest the previously hefty operations budget to smarter, revenue-generating projects
The demand for external data isn’t slowing down, and neither will the growth of your backlog without a change. There are two choices for moving forward:
- Continue to throw money and manpower at your backlog, and hope it doesn’t expand much more.
- Or, you can invest in external data technologies like Crux to reallocate your resources and budget into something more.
What’ll it be? Reach out to me at vince.tomaselli@cruxinformatics.com and let me know what you decide.
Crux: Jan 2, 2024
Vince Tomaselli