Data has been a fundamental resource for enterprises, influencing decisions from the highest levels. Analysts and leaders in the industry consistently emphasize the necessity of the CEO’s active participation in data initiatives. However, what often remains unspoken is that a significant number of data projects fail to progress to production. In 2016, Gartner reported that only 15% of such projects succeed.
The implementation of data projects has emerged as a crucial element for organizations aiming to transform a flood of data into a functional digital transformation strategy. Building on the foundation laid by DevOps, DataOps is the next step. Nonetheless, Gartner has issued a caution: organizations without a sustainable data and analytics operational framework by 2024 will risk delaying their initiatives by up to two years.
Successful operationalization demands effective orchestration. Basil Faruqui, director of solutions marketing at BMC, explains, “When constructing a data pipeline—whether it’s a simple BI project or a complex AI or machine learning endeavor—several stages are involved, including data ingestion, storage, processing, and insight. Each stage utilizes various technologies.” Faruqui adds, “There’s a consensus that production processes should be automated.”
This is where Control-M from BMC, and specifically BMC Helix Control-M, plays a vital role. Control-M has supported organizations for over thirty years, facilitating the execution of hundreds of thousands of batch jobs daily while optimizing intricate operations like supply chain management. With the evolving technological landscape from on-premises to cloud and an increased reliance on SaaS orchestration, launching BMC Helix Control-M in 2020 became an obvious choice.
“While CRMs and ERPs have embraced the SaaS model for some time, there’s been a growing demand in operations for SaaS consumption,” states Faruqui.
The advantage of being a well-established company—BMC was founded in 1980—is that many clients have adapted Control-M for modern applications. A prime example is Hershey’s, a long-time BMC customer managing a highly intricate supply chain.
Given the time-sensitive nature of their business with perishable goods, Hershey’s has adopted Azure, transitioning some existing ETL applications to the cloud, all while operating within a complex SAP environment. Control-M, as noted by Hershey’s analyst Todd Lightner, ‘literally runs our business.’
Faruqui elaborates on how Hershey’s manages significant holiday campaigns or shipping decisions: “It’s entirely data-driven. They collect data from numerous record systems, including external sources, amalgamating it into large data lakes for running AI and ML algorithms to derive outcomes, which are then channeled into an analytics layer allowing executives to review dashboards and reports for decision-making.
“They exemplify how orchestration and automation using Control-M can serve as a strategic advantage,” adds Faruqui.
However, this leads to another crucial insight: while DataOps is integral to BMC’s strategy, it is not the sole focus. “Data pipelines depend on applications both above and below them,” explains Faruqui. “In the case of Hershey’s, determining the appropriate promotion involves data sourced from SAP, which is not a static system; it undergoes constant updates and workflows.
“So, how does the data pipeline ascertain that SAP has completed its updates and that the data is ready? Once strategy is determined, all this information must revert back to SAP because the ordering of raw materials depends on ERPs, not within the data pipeline itself,” continues Faruqui.
“Control-M excels at connecting across this layer, distinguishing it from many existing tools in the DataOps arena.”
Faruqui will be addressing these topics at the AI & Big Data Expo Europe in Amsterdam in September, discussing how orchestration and operationalization represent the next phase in organizations’ DataOps journeys. Attendees can expect stories and best practices illustrating successful transformations, strategies for creating data pipeline orchestration across hybrid environments, and insights into future trends—characterized by increasing complexity, according to Faruqui.
“One ongoing challenge is the proliferation of various tools and capabilities in the AI and ML arenas,” he states. “When you visit the websites of AWS, Azure, or Google and explore their AI/ML offerings, the range is vast, with new capabilities launched at every event. This is a challenge for vendors.”
“On the customer side, there’s a strong desire to quickly test and identify the tools that will be beneficial for them,” Faruqui adds. “This represents both a challenge and an opportunity for orchestration vendors. The challenge lies in keeping pace with advancements since orchestration requires integration with new technologies, yet the opportunity exists as our customers actively seek solutions.”
“They want to avoid having to recreate their orchestration frameworks every time they adopt new technologies.”
Photo by Larisa Birta on Unsplash
Interested in gaining further insights into AI and big data from industry experts? Explore the AI & Big Data Expo happening in Amsterdam, California, and London. This comprehensive event is co-located with Digital Transformation Week.
Essential aspects of managing SaaS environments will be examined in a free webinar titled Navigating SaaS Management: Enhancing Security and Operational Efficiency. Attend to hear insights from Calero, Marsh McLennan, and Merck Group. Register today.
Discover additional upcoming enterprise technology events and webinars powered by TechForge here.
