• by Ray Schultz , Columnist, August 20, 2024

If there’s one thing data engineers dread, it’s a middle-of-the night call that they have a problem. 

These calls cause a major or significant impact for 56%, including burnout and reduced productivity,  and more routine trouble for 28%, according to The State of Data & AI Integration 2024-2025, a study from Nexla, conducted by Ascend2. 

Yet the demand for efficient data integration has increased for 93% of firms, 43% significantly.  

Of the companies polled, 87% are using three or more data integration tools, including:

• Data—API (API Proxy) — 67% 
• ETL — 45%
• Streaming — 41%
• iPass — 25%
• ELT — 25%
• R-ETL — 23% 

Moreover, 75% of companies are using data integration for Gen AI projects, 27% extensively.  

But they face challenges, among them:

• Security concerns — 56% 
• Security and government — 44% 
• Time it takes to set up/use data — 38% 
• Incompatible data formats — 35%
• Importing from external vendors — 34% 
• Data output into correct system and format — 34%
• Limited resources — 32% 
• Lack of standardized processes — 29%
• Exporting to external partners — 29%

Moreover, 83% of data professionals fear data backlogs in their organizations. These are caused by: 

• Complexity of data sources and formats — 49%
• Insufficient manpower or skill gaps in data engineering teams — 43% 
• Lack of automation in data integration tasks — 34% 
• Inadequate infrastructure for handling large volumes of data — 33%
• Unmanageable number of project needs — 27% 

Another complication is the number of partners or customers they are sharing data with, or plan to: 

• 1 partner/customer — 4%
• 2-5 partners/customers — 21% 
• 6-10 partners/customers — 26% 
• 11-50 partners/customers — 21% 
• 51-99 partners/customers — 10%
• 100+ partners/customers — 14% 
• Not sure — 4%

How do they alert the right teams to fix data pipelines when they break? A majority of firms, at 61%, have an automated system in place, while 32% use manual means. Another 5% have no system in place and 2% are unsure. 

Ascend2 surveyed 314 data integration professionals. Of these, 42% work in firms with $500 million in revenue. And 25% report revenue of $250 million to 499.99 million, 22% $100 million to $249.99 million and 11% $50 million to $99.99 million.