Process workflows in highly dynamic IT landscapes change practically every day, which means that the demands on workload automation have grown rapidly, and not just in data warehouse and big data environments. Today, almost every company operates complex, heterogeneous IT environments comprising servers located at multiple sites, multi-cloud environments and containers. When the processes are automated, errors during the processing of large volumes of data due to resource bottlenecks are much more common.
There is frequently no strict separation between development and production. This means that each developer, alongside performing their own work, is also involved in monitoring and operating their data warehouse processes. The complex dependency relationships in data warehouse workflows place exacting demands on the functionality of the deployed automation tool. Many conventional job schedulers do not (or only in part) provide these characteristics, and are therefore only suitable to a very limited extent for use in complex server landscapes.
Characteristics of a Workload Automation System
A Workload Automation System in a sophisticated IT operation scenario must also possess the following characteristics:
- In order to be able to react quickly to new requirements in the data warehouse, it must be possible to change processes whenever required without impacting already active processes, and the operator must always be able to influence active processes to be capable of reacting to exceptional situations in an appropriate manner.
- The installation, administration and operation of the system must not require any special system privileges (root privileges, etc). Access to workflows with regard to their definition, monitoring and operation, as well as the capability for executing jobs in certain environments (job server), must be safeguarded by user privileges.
- Definition, execution, monitoring and problem analysis have to be possible without any additional software installation and at any workstation (web application server).
- The system must provide suitable mechanisms for controlling the system resources to avoid or reduce errors caused by resource bottlenecks.
- The usage (job/batch definition, execution, monitoring, operation) must be easily and quickly learnable without the need for any programming skills. To achieve this, the system’s underlying concepts have to be simple, clear and understandable. The scheduling system must possess all the features required to map complex task definitions for workflow controlling to avoid having to resort to scripting solutions.
Workload Automation is a mature advanced development of job scheduling
These requirements were taken into consideration during the conception phase for BICsuite. That’s why BICsuite was a Workload Automation (WLA) solution long before EMA (Enterprise Management Associates) 2008 described the problems with traditional Enterprise Job Scheduling software as follows:
“Most organizations have one or more job scheduling solutions that automatically submit and monitor simple batch job or script workflows. However, in many cases typical job schedulers do not handle large, modern, complex, enterprise workloads. Workload Automation (WLA) is a mature evolution of job scheduling that supports much more complex workloads, building on traditional job scheduling tools by adding support for event-driven workload, Web services, composite applications, Service Oriented Architectures (SOAs), virtual infrastructures, enhanced interoperability, and business service alignment.” (https://www.enterprisemanagement.com/research/asset.php/777/SMART-Steps-Toward-Consolidated-Workload-Automation)
Exit status, batch and job hierarchies, dependencies, resources, triggers, and hierarchically structured workflow environments and their combinability provide for a simple, understandable and robust implementation of all tasks that occur in practice. These concepts go far beyond the requirements with which EMA describes workload automation tools. Long before the term “workload automation platform” was coined, BICsuite was already able to centrally orchestrate time and event-driven processes.
The BICsuite Job Scheduling Server
The core of the system is the BICsuite Job Scheduling Server, which takes over all the system’s logical functions. It keeps all the configuration, modelling, process state and logging data, etc., up to date in a relational database management system via JDBC. Features that BICsuite has been providing since 2001 and which Gartner describes as a future model for workload automation in 2014:
“Workload automation technology will underpin digital businesses by managing the movement of workloads to the most appropriate location to optimize resources in a heterogeneous environment.”
(https://www.gartner.com/en/documents/2855417-market-guide-for-workload-automation)
Since 2020, Gartner has called the software class which represents an evolution of Enterprise Job Scheduling and workload automation tools a “Service Orchestration and Automation Platform” (SOAP). The difference between SOAP systems and their predecessors, the workload automation systems, is that the services and workflows of the entire company with all the dependencies can be orchestrated and automated on an event-driven basis within a heterogeneous system landscape. With the revolutionary concepts developed in 2001, with BICsuite independIT anticipated the evolution of enterprise job scheduling and IT workload automation systems in the direction of a Service Orchestration and Automation Platform (SOAP). The BICsuite platform has been continuously developed ever since, and has been able to meet the challenges of a modern IT operation in every area.
The most important features of professional workload automation
May we show you how BICsuite can also optimize IT process automation in your company? Get in touch!