Forrester Research Reveals Data Teams Spend 70% of Their Time on Prep & Plumbing of External Data

Just 30% of the time is spent on data analysis as the outcry for a solution grows louder.

Forrester has completed new research in partnership with Crux on automating and optimizing data engineering to accelerate time to value, specifically when it comes to external data. Key findings highlight gaps across the board and a strong desire for better solutions in external data integration, observability, and transformation. 

Key Findings 

• Nearly 80% of surveyed data leaders want a faster, more efficient way to onboard new external data sources 
• 37% of those surveyed said a top process challenge is the lack of proper data governance to ensure data quality, appropriate access, and consistency
• On average, surveyed leaders said their teams spend 70% of their time with a new data set, prepping it for analysis vs. just 30% of their time on actual data analysis

The need for external data isn’t going away anytime soon, and it will continue to fuel the corporate world. But Forrester has provided quantitative proof that things must change for the better to improve time to value and enable data teams to work smarter, not harder. 

59% of business leaders agree they experience slow time to value when onboarding new external/third-party data sets. 79% agree we need a faster, more efficient way to onboard new external data sources to keep up with the demands of the business.

In its current state, the future of external data is backlogs that quickly spiral out of control with the speed of growth regarding external data demands. Forrester estimates that companies ingest 16 to 20 new datasets per month while simultaneously noting challenges with onboarding speed or lack thereof. 

A majority of data engineers spend up to 70% on data prep work, leaving as little as 30% of their time for actual analytics. Time is split between tasks such as data sourcing and procurement, ingestion, preparation, and management, leaving room for little else.  

The bandwidth of current DevOps teams proves to be an additional challenge facing speed to ROI is a deficiency in data preparation. Of the organizations surveyed, 23% identify a lack of employees dedicated to handling external data needs as a contributing factor for external data management limitations. 

Even in the face of these consistent challenges, leaders are keen to invest in tools and processes that can make this complex implementation even a little bit more efficient. 40% of leaders surveyed are ready and willing to increase collaboration with external partners to build and train models to augment workloads. 

If these pains are all too familiar to you and your team, you can download the full Forrester report here. 

If you’re ready to overcome these challenges, schedule your free External Data Assessment with a Crux expert.