With the increasing demand for advanced data engineering skills, professionals are encountering a growing challenge in adapting to rapid technological changes, especially in the era of AI.
Prakash Rajagopalan, head of technology consulting at Tiger Analytics, shared insights on the current state and future trajectory of LLMs and generative AI and how they will reshape the industry, especially for data engineering professionals.
Speaking at DES 2025, AIM’s flagship event, Rajagoplan explained that in the data engineering lifecycle, Gen AI accelerates tasks such as data modelling, source-to-target mapping, and code assistance for migration or new function development.
He explained that their approach involves three key areas: rethinking the data engineering lifecycle, adapting to new demand patterns, and restructuring their processes to leverage Gen AI effectively.
Citing AI tools like Cursor, he said, “One can generate compliant code without having to write it.” Rajagopalan added that tools such as static test data generators and data wrangling utilities help in understanding and integrating new data sources more efficiently.
Moreover, he said governance is crucial as LLMs require well-governed and catalogued data to produce accurate output. “Data governance and AI governance become very important in this scenario,” Rajagopalan said, stressing the need for guardrails and metadata cataloguing.
He further added that Gen AI can help employees discover relevant data assets via intelligent metadata search, improving access workflows.
On the data consumption layer, Rajagopalan said that knowledge base search facilitated by GenAI can improve understanding of data. Instead of navigating numerous dashboards, a “prompt-based chat interface to give you insights” can analyse data and provide observations and even recommendations, moving beyond simple data consumption.
Support functions also benefit. Automation can interpret support tickets, conduct log analytics, and optimise costs, providing “auto-healing” and predictive monitoring capabilities.
Rajagopalan also reflected on past technology shifts, comparing Gen AI’s rise to earlier waves such as offshoring, cloud adoption, and big data. He emphasised that “there is a new wave of changes, a rapid evolution where you go into a new way of working, a new operating model.”
Evolving Skillsets and Organisational Adaptation
Rajagopalan said while some tasks, particularly repetitive coding based on specifications, may be increasingly handled by agents, the need for programming, which involves imagination and creation, remains.
“Programming will still stay relevant, but the coding aspect of it will be taken over by agents,” he said, adding that for engineers, learning one programming language is often enough, since they are no longer responsible for writing the syntactical structure of auto-generated code. This broadens their playground but also increases their responsibilities.
Rajagopalan said that new jobs are emerging in the field. According to him, one clear area of growth lies in building robust semantic layers and metadata frameworks. These components are vital for LLMs and intelligent agents to understand and navigate data effectively, improving the accuracy and relevance of their outputs.
As a result, data engineers will increasingly develop and maintain these descriptive data definitions, ensuring models have reliable access to structured knowledge.
Rajagopalan further added that there is also a growing demand for integrating and analysing real-time and unstructured data sources.
Managing data quality, testing, and compliance is getting more challenging as automation increases and governments introduce new AI regulations.
He also addressed concerns about job displacement, saying that while current tasks may require fewer people, continuous upskilling and reskilling are necessary.
Rajagopalan stressed that organisations and individuals must adapt to these new ways of working.
He said that enterprises should set clear rules for using Generative AI. This includes guiding how prompts are written and how much information is fed into the system at once, since both can affect performance and cost.
He further added that awareness of the models’ knowledge cutoffs and their influence on recommended architectures is also essential. Organisations must stay updated on evolving vendor capabilities and integrate Gen AI strategically.
Rajagopalan concluded that professionals must embrace an AI-first approach, considering how AI tools can augment every stage of their work. He said the younger generation is naturally adept at adopting AI tools.
“Some of us need to get reverse mentoring to see how we can enable it. College kids out there are gaming their essays and everything using ChatGPT—they will game how to do better programming also through that.”
