The global lockdown has made cloud technology imperative for business continuity. Of all the disruptive technologies, cloud has the biggest impact because of its adaptability, accessibility, scalability, resilience and cost.

Recognising this, the government plans to lead the growth of the cloud industry by adopting a cloud-first strategy in the public sector and appointing cloud service providers (CSPs) and managed service providers (MSPs) to build Malaysia’s capabilities in the segment.

While plans for cloud implementation have been in the works for several years, greater adoption of the technology is the focus of the Malaysia Digital Economy blueprint. The objective is to bring about a pivotal transition to achieve the country’s goal of becoming a regional leader in the digital economy.

This national objective offers a chance for a significant portion of local businesses, still relying on legacy technologies, to start investing in disruptive technologies, say experts.

There was a marked increase in the adoption of cloud services during the pandemic, spotlighting the critical need to strategically prepare for cloud adoption.

“With continued lockdowns and restricted movements around the world, many businesses have integrated cloud-based solutions to foster remote working environments, continue reaching their customers online and protect their data,” says Alvin Gan, head of technology consulting at KPMG Malaysia.

Under the blueprint in the MyDIGITAL initiative, four CSPs — Microsoft, Google, Amazon and Telekom Malaysia — have been given conditional approval to build and manage hyperscale data centres and cloud services in Malaysia.

Additionally, the government has proposed the appointment of three local information and communications technology (ICT) companies as MSPs. The MSPs will work with the CSPs to manage cloud services for the public sector. The three MSPs are Enfrasys Solution Sdn Bhd, Prestariang Systems Sdn Bhd and Cloud Connect Sdn Bhd.

With the digital infrastructure in place, the government hopes to move 80% of its services to the cloud by 2022 and in turn incentivise the private sector, especially small and medium enterprises (SMEs), to transition to the cloud.

As SMEs are the backbone of the local economy — comprising 98.5% of all business establishments in Malaysia — they should start investing in disruptive technologies such as big data analytics, Internet of Things (IoT) and cloud computing to avoid being left behind, says Geoff Soon, managing director, South Asia at Snowflake.

Although Malaysian SMEs are receptive to using digital technologies to enhance their business in general, they are still not availing themselves of digitalisation enablers such as cloud computing, IoT and data analytics, according to a white paper by SME Corp Malaysia and Huawei Technologies (M) Sdn Bhd.

The white paper, “Accelerating Malaysian Digital SMEs: Escaping the Computerisation Trap”, notes that while 44% of SMEs are using cloud computing, most of them have not adopted cloud software as a service (SaaS) to drive software process improvements. Instead, they are only using cloud storage services such as Dropbox to store personal documents, pictures and videos.

With the MyDIGITAL road map, SMEs will inevitably benefit from the many changes taking place with regard to cloud adoption, says Gan.

“The Covid-19 pandemic, in its own unique way, accelerated this change. From our observation, those that tend to be left behind are the ones that do not see the urgency to change and adapt.

“Many SMEs were toying with the idea of transforming digitally before the onset of the pandemic, but they decided not to pull the trigger. However, those that did (transform) were in an arguably better position to mitigate the challenges wrought by Covid-19,” he says.

Gan reasons that SMEs generally benefit from not having to deal with legacy issues, unlike established organisations, when embarking on digital transformation.

“In my work, it is often legacy systems that may not be fully equipped to be cloud-enabled. Even if it could be done, the cost is likely to balloon,” he adds.

But this transition need not require large investments, says K Raman, managing director of Microsoft Malaysia.

“Foundationally, there are cloud-based solutions available, such as Microsoft Azure with a pay-per-use model, as well as SaaS solutions like Microsoft 365 with a subscription model. These solutions require lower investment costs but deliver high impact.”

Cloud computing is a big shift from the traditional way businesses think about IT resources, he points out.

“The first immediate benefit in having a holistic journey to the cloud is cost effectiveness. Cloud computing eliminates the capital expense of buying hardware and software. In addition to the cost savings, implementing a holistic cloud journey helps to eliminate data silos.

“It brings together enterprise-wide data to create a comprehensive single source of truth across the business. For organisations today, data is the new currency. It is the key building block towards becoming a data-driven organisation.”

Cloud adoption has also increased as concerns about data security have significantly subsided, as both the underlying technology and market understanding have matured over the years, says Soon.

In fact, many organisations are realising that public clouds can provide a higher level of security than on-premise infrastructure when organisations adopt best practices, he notes.

“In addition, major data storage challenges are taken care of by cloud providers. Cloud technology helps with business productivity, enables remote working and helps organisations save costs with a flexible consumption model without having to rely solely on physical data centres, physical infrastructure or storage models. It eliminates the need for businesses to purchase, develop and operate infrastructure and applications, which typically require significant upfront investments.

“By adopting and leveraging cloud services and capabilities, organisations acquire added flexibility and convenience. Cloud services are available on-demand, allowing organisations to connect and process data from multiple discrete sources seamlessly.

“Cloud services also allow organisations to scale up or down rapidly, with an elasticity that means that they can immediately provide or release data according to what is required. All these benefits contribute to Malaysia’s goal of building a digital and competitive economy,” says Soon.

If all goes as planned, the local data centre industry is expected to contribute RM3.6 billion to the country’s revenue by 2026, states the Malaysia Digital Economy blueprint.

“We believe that going to the cloud is key to unlocking further digital transformation, and it is a gradual process. I also want to highlight that SMEs do not necessarily have to make large investments to facilitate cloud-based digital transformation,” says Raman.

According to the Harvey Nash/KPMG CIO Survey 2020, 47% of the respondents noted that the pandemic had permanently accelerated digital transformation and the adoption of emergent technologies, with cloud technology ranked third.

“In short, many businesses are gravitating towards cloud technology because it allows for quick deployment. Many models available today, for example, SaaS, platform as a service (PaaS) and infrastructure as a service (IaaS), provide a plethora of options for organisations to get things running quickly and reduce cost in operations,” says Gan.

Nevertheless, although the blueprint is an indication that the government is serious about having cloud technology as one of the fundamentals driving the country’s transformation agenda, it is important to ensure that there is demand, he points out.

“The last thing we need is more investments in infrastructure and cloud technology without knowing if they will be fully utilised. One of the ways to manage this is to ensure the synchronisation of all public sector agencies across all levels as part of the cloud agenda,” adds Gan.

Soon concurs that to achieve the ambitious goal, local organisations will have to join forces to adapt to the next normal. “This means more widespread adoption and implementation of cloud services across the country, from end-to-end governmental services to an increasing adoption of e-commerce by the private sector.

“It is imperative that the government considers tactical strategies to ensure the success of the MyDIGITAL initiative. An important first step is to understand the benefits of modern data analytics and cloud deployment. From that point, the government can move away from legacy technology, revisit security policies and review best practices for cloud deployment,” he says.

What Is Event Stream Processing?

Event stream processing (ESP) is a data-processing paradigm in which continuous, ordered sequences of events — generated by applications, sensors, devices, or user interactions — are ingested, analyzed, and acted upon in or near real time. Unlike traditional batch processing, which accumulates data before operating on it, ESP systems evaluate each event (or a sliding window of events) the moment it arrives, enabling sub-second decision-making at scale.

Typical event sources include IoT sensors, financial transaction feeds, clickstream data, network telemetry, and RFID readers. The output of an ESP pipeline might be a fraud alert, a dashboard update, an automated trade order, or a triggered microservice call.

Why Enterprises Are Adopting ESP

Several converging trends have made real-time stream processing a mainstream engineering requirement rather than a niche capability:

• IoT and edge device proliferation: Billions of connected devices generate continuous telemetry that is only useful when acted upon immediately. Batch analysis of sensor data hours after collection defeats the purpose of real-time monitoring and control.
• Demand for real-time business intelligence: Organizations increasingly expect operational dashboards, fraud systems, and recommendation engines to reflect what is happening right now, not what happened yesterday.
• Data volume growth: The sheer scale of modern event streams — millions of events per second across distributed systems — requires purpose-built streaming infrastructure rather than overloaded OLTP databases.
• Regulatory and compliance requirements: Industries such as financial services must detect and report suspicious transactions within tight time windows, making real-time processing a compliance necessity.

Deployment Models

ESP platforms are typically offered in three deployment configurations:

• On-premises: The organization runs the ESP cluster entirely within its own data center. This model provides maximum control over data residency and network latency but requires significant infrastructure investment and operational expertise.
• Managed (cloud): A cloud provider or SaaS vendor operates the streaming infrastructure on the customer’s behalf. Examples include Amazon Kinesis, Google Dataflow, Azure Stream Analytics, and Confluent Cloud. The organization pays for consumption and avoids cluster management overhead.
• Hybrid: Event processing is distributed across on-premises and cloud environments, often with edge pre-processing feeding a cloud aggregation layer. This model is common in industrial IoT and telecommunications scenarios where some data must stay on-site for latency or regulatory reasons.

Core Application Areas

Algorithmic Trading and Financial Services

High-frequency trading and real-time risk management were among the earliest production use cases for ESP. Trading platforms consume market-data feeds, evaluate complex event patterns (price movements, order book changes, cross-asset correlations), and emit buy or sell signals in microseconds. The same infrastructure is used for real-time position risk calculations and regulatory reporting.

Fraud Detection

Payment processors and banks apply streaming ML models and rule engines to every transaction as it occurs. By correlating a new transaction against a user’s recent behavioral pattern, device fingerprint, and geo-location data within milliseconds, fraud detection systems can decline suspicious transactions before they complete — something that is impossible in a batch model.

RFID Event Processing

Retail, logistics, and manufacturing companies use RFID readers to track assets and inventory in real time. ESP platforms filter the high-frequency, often noisy RFID event stream, deduplicate reads, and trigger business workflows (re-order alerts, shipment confirmations, shrinkage detection) as items move through a facility.

Process Monitoring and Observability

DevOps and SRE teams ingest log, metric, and trace event streams from distributed systems into ESP pipelines to detect anomalies, trigger alerts, and feed auto-remediation workflows. Tools like Apache Kafka Streams or Apache Flink are frequently used as the processing backbone behind observability platforms.

Location-Based Services in Telecommunications

Telcos process billions of signaling events per day — call detail records, handover events, data session starts — to perform real-time network optimization, personalized offer triggering, and lawful-intercept compliance. Geospatial stream queries are used to detect when a subscriber enters or leaves a defined area, enabling targeted promotions or emergency alerts.

Key Vendors and Open-Source Platforms

The ESP ecosystem spans both commercial products and widely adopted open-source frameworks. Notable participants in this space include:

Competition in this space is intense, with vendors differentiating on operational simplicity, latency guarantees, exactly-once semantics, native ML integration, and breadth of connectors to upstream and downstream systems.

Red Hat and Enterprise Automation: A Related Development

In September 2019, Red Hat announced the Red Hat Ansible Automation Platform, consolidating Ansible Tower, Ansible Engine, and Red Hat Network Automation into a unified enterprise product. While not an ESP product directly, automation platforms like Ansible are frequently used alongside streaming pipelines to act on ESP-generated alerts — for example, automatically remediating an infrastructure anomaly detected by a streaming observability pipeline. Red Hat’s broader portfolio (AMQ Streams, OpenShift) is directly relevant to organizations building Kubernetes-native event-driven architectures.

Regional Market Activity

Adoption of event stream processing solutions spans all major geographies, with the following general patterns:

• North America: Dominant market driven by large financial services firms, hyperscalers, and early IoT adopters. Home to many of the leading open-source committer communities.
• Europe: Strong growth in financial services and manufacturing (Industry 4.0), with GDPR driving interest in on-premises and hybrid deployments that keep personal data within the region.
• Asia Pacific: Rapid expansion fueled by telecommunications modernization, large-scale e-commerce event processing, and government smart-city initiatives.
• Middle East, Africa, and South America: Emerging adoption, particularly in telecommunications and government digital-transformation programs.

Architectural Considerations for DevOps Teams

When evaluating or operating an ESP platform, engineering teams should assess the following dimensions:

• Delivery guarantees: Does the platform support at-least-once, at-most-once, or exactly-once event delivery semantics? The answer has significant implications for financial and audit-sensitive workloads.
• State management: Stateful stream operations (windowed aggregations, joins across streams) require reliable, fault-tolerant state backends. Apache Flink’s RocksDB-backed state store and Kafka Streams’ changelog topics are common solutions.
• Backpressure handling: A well-designed pipeline must gracefully handle bursts without data loss or cascading failures. Evaluate how each platform propagates backpressure from sinks to sources.
• Operational tooling: Observability into consumer lag, partition skew, and processing latency is essential for operating ESP pipelines in production. Kafka’s consumer group metrics, Flink’s Web UI, and third-party tools like Burrow or Datadog provide this visibility.
• Schema management: Event schemas evolve. A schema registry (e.g., Confluent Schema Registry with Avro or Protobuf) prevents incompatible producers and consumers from silently corrupting downstream analytics.

Summary

Event stream processing has moved from a specialized capability used only by financial institutions and internet giants to a foundational component of modern data architecture across industries. The combination of IoT growth, cloud-native infrastructure, and the universal demand for real-time insight continues to drive both adoption of open-source frameworks like Apache Kafka and Flink and investment in managed cloud streaming services. For DevOps and platform engineering teams, understanding ESP concepts, deployment models, and operational requirements is increasingly a core competency rather than an optional specialization.