In the fast-changing world of data engineering and the AI era, the role of professionals has evolved, moving from mere managers of data pipelines to key strategists making rapid decisions in real time.
At DES 25, Varun Saraogi, principal data architect at MathCo, vividly captured this transformation, drawing an insightful analogy from the high-stakes world of Formula 1 racing.
“Modern data engineers are like race engineers,” he said. Saraogi, an F1 enthusiast, noted that the comparison was casual but more structured.
In a Formula 1 race, each team gathers massive volumes of data every second. But, as he noted, there is no differentiation in how much data you can capture across teams. What differentiates them is how their teams use the data.
Like a race engineer relaying insight to the driver, today’s data teams must act with precision, speed, and deep understanding. He stated that the key issue is how the data engineering team, the strategist, and the race engineer comprehend and leverage that data to deliver contextual information.
Two Pit Stops: Context and Product Mindset
Saraogi’s central argument was that context is now fundamental to decision-making. From metadata and domain knowledge to access controls and interaction history, Saraogi outlined eight distinct layers that must be embedded, not bolted on, into pipelines. “Metadata cannot be an afterthought anymore,” he insisted. It has to be embedded into your pipeline.”
Beyond context, he called for a shift from pipeline-building to product thinking. Drawing from experience, Saraogi admitted that back in the day, he only focused on providing data to the team and did not understand the use cases. That mindset no longer holds.
Whether building a Customer 360 or preparing for real-time AI use cases, data engineers today must anticipate business intent. “It is not just about building platforms for AI, it is about building platforms with AI as well,” he said.
Saraogi also pointed to the need for real-time signals, such as holidays or sales trends, to be embedded directly into data flows, keeping context in mind. Traditional systems can’t keep up with open-ended queries or expectations for on-the-fly personalisation. “If you have to rewrite the playbook…the future is here,” he said, urging engineers to rethink how data platforms work and are built.
The Finish Line Isn’t Just About Speed
Saraogi reminded the audience that the fastest car doesn’t always win. “McLaren in the second half had the fastest car, but they did not win. It’s also about strategy, data, and how one uses it,” he said.
Likewise, a platform with high throughput and low latency won’t cut it unless the data has meaning. He stressed that the data has to be analysed to understand the users, the region, and more, not just in terms of sales numbers.
Context must be present from the beginning to deliver relevant and trustworthy AI outputs. Saraogi laid out a structured approach: identify context sources (structured, unstructured, metadata, and external), assemble context models (from relational to vector databases), and monitor their use to ensure feedback and trust.
So, the modern data engineer doesn’t just move data. They monitor, interpret, and strategise, like a race engineer perched on the pit wall. The race hasn’t slowed down, but the winners are not only building faster pipelines but also asking more thoughtful questions.
