Looking at the year ahead, it is certainly shaping up to be an exciting and meaningful year for data and the role it continues to play in the industry’s march to find insights that drive business performance to new levels. Here’s a synopsis of some of the key trends we’ve observed and continue to watch:
Data Management:
1. Preparing data for AI – 58% of firms plan to integrate AI systems in 2019, making data preparation a top priority. To prepare data for AI, 49% of firms will prioritize making sure their data meets regulatory requirements, and 47% of firms will prioritize identifying and aggregating data from across the respective firm, according to research by PwC.
2. Integrating alternative data – Adoption of alternative data continues to accelerate. Last year, 70% of hedge funds said they were already using or expecting to use alternative data, up from 52% in 2016, according to a survey by EY. In 2019, we see signals emerging from alternative data and new data sources and platforms that will help firms manage their growing sources emerging as well. Interest in this sector is rising among strategics and private equity firms, as demonstrated by Nasdaq’s acquisition of data aggregator Quandl and Vista Equity Partners’ acquisition of data and analytics provider 7Park.
3. Scalable data management – To increase efficiency, funds are seeking out partners who specialize in ingesting, processing, and managing data. More than half (60%) of institutional investors plan to outsource data management by the end of 2019, according to research by State Street. Today, leading data engineering firms can accelerate funds’ data workflows by automating repetitive tasks, providing skilled teams to manage data operations, and delivering data on demand.
4. Eliminating data bias – Firms deploying algorithms are taking steps to prevent biased decisions. Leading funds ask whether a dataset shows “the forest,” or “a few trees” before modeling. Selecting appropriate data, understanding the implicit biases of the person behind the model, complying with data security and privacy regulations, and diversifying algorithmic trading strategies can help funds mitigate risks. Cleaning data to remove bias is a key priority for firms preparing for AI this year, according to PwC.
Data Science:
5. Quantamental trading – The combined power of human and machine is bringing asset managers new opportunities to find patterns in data. Firms are using quantamental approaches to analyze sentiment data around stocks and to anticipate and avoid catastrophic risks such as market crashes. The fusion of traditional financial principles and deep learning, a form of AI, empowers funds to understand vast amounts of historical data, forecast, and allocate portfolios.
6. Finding signal in text – New applications of Natural Language Processing (NLP) are enhancing investment strategies by helping firms find insights within vast amounts of text data. NLP can analyze transcripts of corporate earning calls to determine tone, complexity, and analyst engagement. With NLP, firms are also finding signal in proprietary text records and in public sources like Twitter.
7. Deciphering the black box – Today’s self-learning AI systems can decide which data is most useful and even create their own data to improve their training. Yet humans still need to understand how and why algorithms make decisions and prepare for variations in algorithms’ performance under different market conditions. The best AI systems show the thinking behind their decisions so that humans can interpret them. “Interpretability” of data inputs, model factors, and decisions is essential to plan for extreme events and achieve regulatory compliance.
Team:
8. Data-driven cultural changes – Making the transition to use data is leading firms to structure teams for collaboration and design organizational culture to attract, retain, and develop top data talent. The best firms start by defining their data and talent requirements, taking steps to reduce competition among different roles in data teams, increasing data learning opportunities, training academic and business thinkers to communicate about data strategies and models, learning what motivates the best minds to stay, and scaling data talent over time based on lessons learned.
Technology:
9. Deploying cloud solutions – Cloud-based solutions are receiving strong support among asset managers. 68% of institutional investors are comfortable with cloud data storage, according to a survey by State Street. Now the challenge for firms is implementation which includes deploying a unified cloud data system that integrates all data sources that users in the front and back offices need.
Note: trends were compiled from evaluating a broad list of industry conferences, large research documents, and conversations on data with financial services professionals.
At our recent Hackathon, the Crux team held a competition to see how many data pipelines each member could build. Senior Developer David Tong built the greatest number of pipelines that week, but we were also impressed with the performance of our CEO Philip Brittan, who joined the Hackathon to build data pipelines along with the team.
This Hackathon not only accelerated our data engineering process, but also helped our team learn valuable lessons about how to communicate and collaborate on the future state of data engineering. In keeping with our culture of being a learning organization, here are 4 learnings to help you innovate with data engineering:
1. Systematize and document data engineering processes. Write down clear, step-by-step instructions. Imagine starting with a clean hard-drive – nothing installed. Document your firm’s processes so that anyone could follow them, regardless of technical experience. Systematizing processes is essential to scaling data ingestion.
2. Teach your whole team about data engineering, even if they’re not data engineers. Data engineering is essential before data analysis can take place. Let your team practice data engineering and learn from data engineers. Walking a bit in data engineers’ shoes can help your whole team better understand the complete process of working with data.
3. Ask data engineers to explore each dataset, uncover its unique qualities, and learn about its use cases. Every dataset is different, and there is no substitute for hands-on work with each one. It is vital that your team learn the properties and quirks of each dataset before you onboard it so that everyone can communicate clearly and identify potential challenges with the data ahead of time.
4. Create a team challenge to spark learning and innovation. To accomplish something big, get everyone in the same place, and define one common task. Communicating, collaborating, and committing to deliver on a shared goal helps people make faster and smarter progress in working with data. While technology today enables people to work remotely, your team will appreciate the opportunity to come together and work as one.
We were featured in the Hedge Fund Journal! This article explores various topics including how our economies of scale can reduce your data management costs, why neutrality is the core principle of the ideal data engineering solution, and how you can ingest data more efficiently. As our CEO Philip Brittan says, Crux’s neutral platform “operates and maintains firms’ data supply chains, so firms get past this first mile challenge of ingesting data and speed ahead to generating returns.”
Our Head of Go-to-Market, Elizabeth Pritchard, also published a blog post on our efficient approach to data which was widely-shared across various networks. You can also find her predictions for the future of data featured in this article by Devin Banerjee, Senior Financial Services Editor at LinkedIn, on the FinTech conference Money 20/20.
Are you intellectually curious and excited to build the data solution of the future? Do you love solving complex problems and helping clients overcome data challenges? If so, let’s talk. The Crux team is growing, and we’re seeking talented individuals who know data can be delightful. Check out our job openings here.
We’re quickly expanding the Crux Network with some amazing data partnerships. If you’re a data supplier looking to reach more clients and accelerate distribution with zero roadblocks, we invite you to join Crux. Connect with us.
Crux: Jan 2, 2024
Not long ago, we observed here in our blog that the critical insights that drive business value come from data that is both (1) fast and (2) reliable.
Crux: Oct 31, 2023
This past year has been exciting, representing the dawning of a new age for artificial intelligence (AI) and machine learning (ML)—with large...
Crux: Oct 24, 2023
How do you get white-glove customer service from a major data supplier?
