A lot of noise and misunderstanding exists in the user marketplace about what digital transformation is, and the roles that artificial intelligence (AI), machine learning (ML) and the Internet of Things (IoT) play in implementing it.

Digital transformation is of major importance to businesses and organizations. Alphabet, a driving force in the digital marketplace, reported record profits in the second fiscal quarter of 2021 based on very strong digital ad sales, internet searches, and the growth of cloud services. Many companies have recognized the importance of digital transformation as a key strategy to gain a competitive edge in the marketplace.

Digital Transformation

Digital transformation is a process that has been ongoing for decades whether businesses have focused on it as a paradigm shift in the way business is conducted or not. Email, ERP platforms, cloud computing, digital advertising, online retailing, telehealth, working remote and video streaming are a few examples of businesses and processes that have undergone digital transformation. All these changes were made possible by transformation to digital processes.

Businesses and organizations have begun to realize that digital transformation can be planned and implemented proactively to be more competitive, enhance customer experiences, and improve processes that are customer touch points. AI and ML tools and the IoT channel are readily available for the planning and implementation of a proactive digital transformation strategy.

Role of AI and ML

AI and ML are tools that can be used to plan and implement a digital transformation that focuses on what is needed to gain a competitive edge with customers now and in the future. They help companies develop comprehensive strategies for data transformation based on real-time market intelligence rather than piece-meal tactics of disjointed updating of systems and processes.

ML uses big data to analyze and correlate patterns which can offer insights into customer behavior and other events. It provides insights on solutions to improve business processes and customer experiences. Instead of finding out about a change in customer behavior after sales have already declined, ML can help to identify shifts in behavior and events as they happen so solutions can be devised to head off a decline in sales.

AI uses algorithms to create or adjust programs to take advantage of the insights gained with ML. AI mimics human intelligence to identify and react to behavior and events. Programs based on insights to behavior and events from ML can improve customer experiences and business processes.

Using AI and ML in planning and implementing a digital transformation puts businesses and organizations in position to stay ahead of the curve. With AI and ML information is analyzed in real-time to increase responsiveness to changes in customer behavior and events, and make business processes including IT operations more efficient.

Digital transformation directed and driven by AI and ML enables the gathering and analysis of data to better understand customers to help accelerate introduction of new products and services.

With AI and ML data transformation has become a constantly evolving process, providing insights and responses to the ever changing dynamics of customers and the marketplace.

Role of IoT

The IoT is the network of objects that have software, sensors, smart devices and other instruments incorporated in them to link and exchange data with other systems and devices over the internet. It is the channel for gathering data used in AI and ML, and for interacting with customers by way of the “things” in IoT to improve customer experiences and business processes. The internet and all the things it connects provides the ability to implement digital transformation. IoT is the conduit that makes data-driven, intelligent data transformation possible.

Businesses and organizations face two choices when they contemplate digital transformation. They can make decisions on digital transformation that are not coordinated or supported by market data, or base digital transformation strategies on the constantly evolving real-time data gathered via the IoT and analyzed and responded to with AI and ML. The organizations that choose the latter approach will gain competitive advantage in the marketplace.

When it comes to new AI automation, the Software 2.0 movement can be seen as akin to the auto industry’s recent evolution. Beginning in the 1990s and rapidly progressing into the 2000s, the auto industry saw a huge rise in AI automation such as digital vehicle diagnostics.

Throughout this, people worried that digital transformation might take jobs away, when instead it led to “greater profits, productivity, and competitiveness,” according to a 2008 study by the journal of Technological Forecasting and Social Change. In short, these innovations didn’t replace jobs, per se; they simply changed the processes of jobs from the manufacturer’s assembly line down to the mechanic’s garage.

An auto diagnostic specialist used to perform tedious tasks such as counting flashes and converting them to error codes on printed-out tables. Computerized diagnostic tools made this process faster, more reliable and, you might imagine, far less tedious. According to diagnostic specialists Helmut Frank and Uwe Schmidts, digital transformation went from being “a necessary evil to being a key to new, interesting and innovative functions.”

You might liken the above example to the innovations that AI automation and machine learning (ML) have offered to software developers, who until recently, were tasked with a much more hands-on approach to deploying and maintaining applications. In many ways, DevOps practices have heroically come in to break them free of this minutiae, automating many of the repetitive tasks involved in application management. By design, it’s increasing efficiency and security while decreasing tedium and the chance for human error.

Speaking to tedium, Mike Loukides and Ben Lorica, in their article “The road to Software 2.0” noted, “Up until now, we’ve built systems by carefully and painstakingly telling systems exactly what to do, instruction by instruction. The process is slow, tedious, and error-prone; most of us have spent days staring at a program that should work, but doesn’t.”

In short: Much like digital transformation brought efficiency to the life cycle of automobiles, AI automation is offering efficiency to every stage of the software development cycle.

Don’t Change the Players, Change the Game
Indeed, a new generation of AI automation and ML tools is now emerging, shaping what some call a Software 2.0 movement. But before you begin to fear that these tools will lead to a “they took our jobs!” revolt, consider that they may be celebrated instead as a means of not only augmenting the work of software developers but also helping offset the ongoing developer drought. According to a 2019 Stack Overflow survey, over a third of developers named “not enough people for the workload” as their primary challenge to productivity. Rather than replacing jobs, automation should be viewed as offering efficiency in an industry that desperately needs it.

As an example, the process of writing code, though more streamlined today than it was 25 years ago, has remained a mostly manual process. This has presented an excellent opportunity for the automated processes we are beginning to see, which bring in AI to automate aspects of code development such as program synthesis, smart code completion and static code analysis.

Assaf Araki spoke to this in the article, “Software 2.0 takes shape.” Araki described how automation is filling gaps in many other areas, such as regression testing, code analysis, AI for vulnerability detection and observability tools to monitor software health.

From Monolith to Microservices
Just like the recent, mysterious disappearing monolith in the Utah desert, the relevancy of monolithic development may be on its way out.

In the past, software cycles were longer. Teams updated monolithic-style deployments only monthly or yearly. Nowadays, companies are under pressure to push much more rapidly, often turning to microservices and progressive release strategies. This new paradigm requires much automation to increase release efficiencies and enable a seamless CI/CD.

As Araki pointed out, “Automation in software is best viewed through the lens of Continuous Integration (CI) and Continuous Deployment (CD).”

How to Know When AI Automation is Worth Its Salt
It’s smart to consider the investment of both money and time when deciding whether an AI solution is worth adopting. “AI systems need to add value that exceed any new costs and requirements (such as the need for labeled data, hiring new staff, etc.),” Araki said.

You should also be asking if the solutions are wide-ranging enough to meet the needs of your developers and anyone else on the team who may be employing them. Key considerations for AI automation, Araki noted, are focusing on tasks and systems that are frequently used, adjusted or tuned.

4 Examples of Versatile AI Automation and ML:
Data management systems from simple systems that employ ML to more complex systems that enable self-driving data management.
Dashboards that are automatically created, ideally with intuitive and visually appealing UX, that use AI automation and ML to bring up the right elements—more progress is expected in these areas in the coming years.
Websites and mobile apps that use AI automation to become highly personalized interfaces, such as those that employ real-time data for online and reinforcement learning—these are also expected to progress in the coming years.
UX and design tools that use simple automation to design and revise digital media, with ML being a promising way to kick the automation up a notch, such as deep learning for game developers.
AI Automation and ML to Grant Opportunities
In short, Software 2.0 doesn’t currently appear poised to threaten jobs or decrease developer need. Instead, it seems to be changing the processes engineers are using and creating efficiencies in their work, at a time when their expertise is in increasing demand. Demand for novel roles, like the Cloud Economist, demonstrates the emergence of new opportunities for humans with cloud knowledge who work alongside AI.

The new evolution of AI and ML also could create a unique subset of developers with skills keeping par with these new innovations—much like digital transformation in the auto industry led to evolving specialties in computerized vehicle systems.

Not to mention that just as low-code has paved the way for citizen developers and AutoML led to citizen data scientists to fill large gaps in demand, AI and ML solutions have the potential to further broaden and democratize software and code development.

The use of artificial intelligence (A.I.) and machine learning (ML), technologies that help people and organizations handle customer personalization and communication, data analytics and processing, and a host of other applications continues to grow.

An IDC report found three-quarters of commercial enterprise applications could lean on A.I. by next year alone, while an Analytics Insight report projects more than 20 million available jobs in artificial intelligence by 2023.

Due to A.I. and ML’s transformational reach, specialists with the right skills could find themselves with job opportunities across a wide range of industries. A global skills gap in the technologies means qualified applicants can expect good salaries and a strong bargaining position.

Gus Walker, director of product at Veritone, an A.I. tech company based in Costa Mesa, California, told Dice that, in 2021, there will be growing opportunities for A.I. and ML specialists in public safety, banking and fintech, and healthcare.

“These three industries have the money to invest in A.I. and ML right now and have the greatest opportunity to see the investment pay off, fast,” he said. “That being said, the pandemic has caused industries hit the hardest to take a step back and look at how they can leverage AI and ML to rebuild or adjust in the new normal.”

Data collection and preparation, as well as data analytics expertise, could end up the most in-demand skills when hiring for artificial intelligence. “Organizations need to hire individuals who can identify the correct training data and annotate the data accurately,” Walker added. “They need talent that can maintain growing training sets and analyze the data to create targeted datasets for customized model generation.”

This means companies will require professionals familiar with algorithm tuning and training; prime candidates will also have experience in A.I. operationalization and DevOps, in order to successfully set up datasets and continuous integration and continuous deployment pipelines so that algorithms stay up-to-date.

“Digitalization will make even more structured and unstructured data available to be leveraged, which will increase the need for more A.I. professionals,” Walker pointed out. “Businesses need someone who can maintain production deployments throughout.”

Here’s a breakdown from Burning Glass, which collects and analyzes millions of job postings from across the country, of how many technology jobs request machine learning skills. Although the current numbers are relatively small (at least when it comes to job postings), Burning Glass predicts significant growth over the next decade:

As the global telecoms industry begins to deploy 5G networks, technologies such as cloud-native RAN and automation platforms will play major roles in their evolution. In this blog post, we’ll cover impact on management and orchestration, the challenges that emerge, our view on the future of management and orchestration and what Ericsson brings to the table.

5G evolution realizing new values
5G is more than radio and core. It encompasses virtualization, management and orchestration, transport, software-defined networking (SDN), and adaptive policies powered by artificial intelligence (AI) and machine learning (ML). As more and more technological innovations enter the market, 5G will continue to leverage the capabilities of these technologies to evolve further.

The 5G networks are designed to serve consumers and enterprises alike and as we add capabilities like network slicing, edge computing, service exposure and new RAN applications, we are in effect creating a new innovation platform. An innovation platform that will enable exciting new business models and use cases that will deliver innovative services to countless industry verticals.

The impact on orchestration
The architecture of the 5G network is substantially different from previous network generations. One key change is the introduction of micro services-based architecture and network functions distributed over different cloud infrastructures. In addition, many services will go across different network functions and domains in the network and they need to work in unison to ensure end-user experiences. This impacts the way we manage and orchestrate the networks.

The openness and disaggregation started in transport and core with the introduction of Software Defined Networking (programmable networks) and virtualization. A similar shift is expected to happen in the access domain and certain aspects need to be considered in the orchestration systems as well.

Open source and standards become even more important than previously in this context. Ericsson is a leading supporter of several open-source initiatives. These include the Open Network Automation Platform (ONAP) and Open Radio Access Network (O-RAN). We also work hard to encourage industry alignment to ensure speed of adaptation and realization of values from the 5G platform.

The challenges from an orchestration perspective
As service providers’ networks evolve, and new technologies are added, the network is becoming ever more sophisticated and complex.

As network functions are virtualized, networks will end up consisting of Physical Network Functions (PNF), Virtual Network Functions (VNF) and Cloud native Network Functions (CNF).
Network-related services and end-user services will flow across different network functions and multiple domains.
Services will become more advanced with the introduction of network slicing, exposure and RAN applications.
Multiple domains (radio, transport, core and cloud infrastructure) end up having their own OSS systems for different reasons.
The challenge that arises is that service providers need to deliver on stringent -service level agreements towards enterprises, with many different business models and multiple partners. Orchestration and management need to deliver on this challenging task.

What’s needed is an intent-based AI/ML-enabled automation platform that is open and highly flexible to scale. This Automation platform is key to manage these complex networks and services and to achieve the necessary network and service flexibility and speed up time to market.

As cloud-native RAN is introduced in the network, Ericsson brings the knowledge and capability from our experience of managing and orchestrating end-to-end 5G networks that includes cloud-native 5G core.

Key considerations for services providers
As Cloud RAN is introduced, certain aspects need to be considered in the management and orchestration systems.

Artificial intelligence and machine learning will play important roles in traffic steering and optimization decisions that will improve network efficiency and customer experiences and is an absolute prerequisite in next generation management and orchestration.

Since 5G will provide services to many vertical industries, the use cases demand Quality of Experience (QoE) such as high bandwidth, low latency, high security and ultra-reliability. Managing QoE proactively will be central to maintaining SLAs and continuous optimization is needed. The introduction of cloud-native RAN and Service Management and Orchestration will add new capabilities to assure best possible Quality of Experience.

The virtualized radio and core will be deployed on many different cloud infrastructures and distributed multiple data centers and far edges of the networks very close to where the services are being consumed. The orchestration system will need to work across all these different cloud environments and manage the end-to-end lifecycle management of network functions and services.

These are some key considerations that service providers need to think of as they evolve their management and orchestration. In the next section we will explore our view on how to realize this and the values we bring.

Ericsson’s view on next-generation management and orchestration
The next-generation management and orchestration (as defined by O-RAN Alliance) consists of a number of interworking components – the Service Management and Orchestration (SMO), Non-Real Time Radio Intelligent Controller, rAPPs and the interfaces O1, O2 and A1 allowing for communication between these components.

Service Management Orchestration
Ericsson brings their unique experience in building, deploying and managing cloud native networks and orchestration systems to enable seamless and modular introduction of Service Management Orchestration (SMO) for both greenfield and brownfield networks. We work closely with our customers to create unique use cases, while at the same time driving rapid innovation by enabling an open app ecosystem for both customers and partners. SMO the management and orchestration for the future and have a few very important characteristics:

AI/ML capabilities will be crucial in SMO for innovation, automation and operational efficiency. A key capability to achieve a high degree of automation is to use artificial intelligence and machine learning together with advanced policy management and we call that intent- based AI/ML.
To achieve a high degree of automation, Ericsson brings unique know-how to package AI/ML powered use cases into hardened and scalable solutions using our telco expertise.
SMO should have infrastructure visibility and be able to scale compute resources up and down. This also helps optimize or reduce the overall capacity costs.
SMO can handle end-to-end network slicing where RAN slicing is an important component. For example, a SMO can continuously monitor and ensure that a specific enterprise SLA is maintained in an end-to-end network slice that is providing a mission critical, low latency, ultra-reliable service.
SMO needs to be aware of applications and their performance, network behavior, traffic and multiple domain information, and have the ability to dynamically place application workloads across multiple cloud infrastructures.
The SMO need to simultaneously handle life cycle management of PNFs, VNFs and CNFs. In the cloud native world software management changes drastically with the introduction of high degree of automation, continuous integration/continuous development of micro-services. To support very fast introduction of new SW and features into the network, canary deployment and A/B testing will grow in relevance.
Non–real time RAN Intelligent Controller
The non-Real time RAN Intelligent Controller is key component of the future management and orchestration as defined by O-RAN. Placed inside the SMO layer it can access external information to dynamically use policies to optimize the RAN powered by AI and ML.

The non-Real time RIC plays a central role in SMO hosting the ecosystem of applications acting across RAN, Core and Transport, to generate and apply new policies and network optimizations. Additionally, from a pure innovation platform point of view, this component will expose the SMO capabilities and the network data, to a vast variety of applications, that will coexist and collaborate, to achieve new business outcomes.

Finally, it needs to be flexible, aware and secure. It needs to be flexible enough to foster innovation, aware enough to understand the network behavior, and secure enough, since it provides access to operators’ networks and data pools.

The power of RAN applications (rAPPs)
As seen above, rAPPs are powerful and important tools in the future management and orchestration. rAPPs can be written for a various management tasks such as assurance, optimization, automation, fault management, performance management, configuration management, software management, domain-specific applications.

The introduction of rApps, made possible by an end-to-end automation platform, brings innovative aspects in the RAN domain. It means that tasks as those above can be realized outside the traditional node context. Working on their own or together with other rAPPs, they can create improvements across all network domains.

In summary, SMO and the next generation of management and orchestration is essential to cope with the rising complexity of data and network systems, achieving not only a more manageable end-to-end network but also achieving the business propositions sought by operators: increased QoE and time to market for services creation and rapid innovation to achieve the full 5G potential.

Ericsson is exploring the possibilities
Ericsson views the rapidly progressing automation capabilities as essential to maintaining operational efficiency while also maximizing the business potential of the increasingly sophisticated 5G networks. Building on our leadership in RAN, core, management and orchestration systems, we bring a unique experience in delivering end-to-end 5G networks and OSS solutions (management and orchestration) to manage the smooth introduction of open and interoperable cloud-native RAN. In addition, Ericsson brings the following key capabilities in its automation platform that is driven by management and orchestration systems:

A programmable network with intent based AI/ML capabilities in management and orchestration solution to drive efficiency and automation.
A management and orchestration system that manages the PNF, VNF and CNF that comes with the introduction of cloud-native RAN.
Ability to develop unique use cases, onboard new applications in the management systems and rapid innovation by enabling an open app ecosystem for customers and partners.
An intelligent management and orchestration system that can provide network and service flexibility to support various new business models and use cases.
Ericsson is exploring the possibilities offered by SMO architecture to create an attractive solution for next-generation management and orchestration and accelerate the automation journey together with our customers.

Artificial intelligence is used in loads of business scenarios. In some instances, organizations use data and machine learning to solve some of their most challenging problems, and have seen incredible returns. According to McKinsey Global Institute, the companies that absorb AI in their workflows by 2026 are set to dominate the 2030 world economy.

McKinsey even forecasts colossal growth, predicting 120 percent above cash-flow increases, a fairly significant number that Google Cloud hopes to capitalize on. Recently the cloud services provider released several AI/machine learning-oriented capabilities to assist organizations in doing just that. There are some challenges, however, in doing that, and according to Craig Wiley, Director, Product Management, Google CloudAI Platform, “Machine learning (ML) systems have a special capacity for creating technical debt if not managed well. He added in a blog post:

Google Cloud has a few goals with its AL/ML line of products designed to make it easier to perform certain business tasks. The first – unifying ML system development and operations. As such, the company announced a hosted offering for building and managing ML pipelines on the AI Platform earlier this year. Google Cloud said it would soon extend a fully-managed service for ML pipelines in preview this month.

The platform aims to simplify the management of models at scale and to help data scientists focus on models at risk of not meeting business objectives. “Continuous monitoring should be available by the end of 2020,” Google Cloud added in a statement. The foundation of the organization’s new tool serves as a machine learning metadata management service within the AI platform. It should have several implications, too, including boosting productivity across the board along with reducing development cycles, increasing deployment speed, etc.

Google Cloud also introduced Google Workspace, which boasts a host of useful features. The all-new user experience brings together tools like chat, email, voice, and video calling into a single, unified interface. This is the 14th rebrand of Google’s business productivity suite, most recently known as ‘G Suite.’ The suite includes Gmail, Calendar, Drive, Docs, Sheets, Slides, Meet, Chat, and more.

Google Meet even got a recent makeover, just in time for the school year. There’s the use of new digital whiteboards with Jamboard. The feature helps students and teachers collaborate whilst educators retain control over who can edit the whiteboard. There are also breakout rooms, attendance reports, Q&A capabilities, and polling.

Google Cloud is on a clear mission to beef up its functionalities in both the education and enterprise spaces – all given a massive nudge by the novel Coronavirus. The company told its employees they could work from home until July 2021, while its competitor Microsoft, is letting its employees work from home for good.

The Data Engineering Bottleneck Slowing AI Adoption

Data integration and big data management solutions provider Attunity announced an analytics platform designed to automate streaming data pipelines for AI, machine learning, and data science initiatives. The core problem it addresses is a familiar one: data engineering teams cannot keep pace with the demand for real-time datasets that machine learning applications require.

Data integration and migration are often manual, complex, and time-consuming processes. By the time a dataset is assembled and delivered to data scientists, it is frequently outdated — undermining the value of the analysis before it even begins. Attunity’s approach targets this lag directly by combining efficient change data capture (CDC) at scale with automated data transformations running on Apache Spark.

Attunity for Databricks Unified Analytics Platform

The integration — Attunity for Databricks Unified Analytics Platform — provides continuous change data capture, delivery, and automated data refinement to produce analytics-ready datasets inside Databricks. Databricks’ Unified Analytics Platform unifies data processing and machine learning in a single environment, with Apache Spark as the high-performance processing engine underneath.

By pairing Attunity’s CDC pipeline automation with Databricks, organizations gain a path from raw transactional source data to clean, ML-ready datasets without writing custom integration code. This matters in practice because manual pipeline development is a significant time sink for data engineers — time that could otherwise be spent on higher-value work.

“Our ability to reach into virtually all enterprise systems and deliver real-time, analytics-ready and transactionally consistent data without coding is a powerful enabler for data scientists, allowing them to generate the business insights that accelerate corporate decision making and growth.”

Itamar Ankorion, Chief Marketing Officer, Attunity

Key Capabilities

Accelerated Time to Value

Using Apache Spark as the processing engine, Attunity automates data delivery and creates analytics-ready datasets in Databricks. Data engineers can build reusable, automated pipelines that reduce reliance on manual coding and expensive custom development resources, shortening the delivery cycle from source systems to data scientists.

Reduced Data Preparation Overhead

Attunity’s pipeline automation reduces the data preparation burden on data scientists. Rather than spending significant time wrangling raw data, data scientists can focus on analysis and model development. More agile pipeline generation means that changes to source data schemas or business requirements can be accommodated without rebuilding pipelines from scratch.

Real-Time Data with CDC at Scale

Attunity’s change data capture (CDC) technology provides the low-latency, high-throughput characteristics needed for large-scale machine learning datasets in the cloud. CDC captures only the changes made to source data — inserts, updates, and deletes — and streams those changes continuously, rather than running expensive bulk extractions on a schedule.

Multi-Cloud Support

Attunity supports Databricks deployments on both Microsoft Azure and Amazon Web Services (AWS), as well as a broad range of data lake, data warehouse, and streaming services available on those platforms. This gives organizations flexibility to adopt the platform within their existing cloud strategy without vendor lock-in to a single cloud provider.

Transactionally Consistent, ACID-Compliant Data

Attunity provides full support for the Databricks Unified Analytics Platform’s ACID capabilities. Transactions are delivered and applied in the order they were committed on the source system, ensuring that data scientists and ML models work from a transactionally consistent view of the data — a critical requirement for models where data ordering and integrity affect accuracy.

Why This Matters for DataOps

The Attunity-Databricks integration reflects a broader shift toward DataOps practices — applying DevOps-style automation, collaboration, and continuous delivery principles to data pipelines. Automating the movement and transformation of data reduces the manual coordination between data engineering and data science teams, shortens feedback loops, and makes it easier to keep ML models trained on current data.

For enterprises with data spread across siloed storage systems — on-premises databases, cloud data warehouses, streaming platforms — the ability to unify those sources into a single analytics environment without hand-coded ETL is a meaningful operational improvement. The combination of CDC for real-time ingestion and Spark for transformation automation addresses the two most time-consuming stages of the modern data pipeline.

Note: Attunity was acquired by Qlik in May 2019, shortly after this announcement. Organizations evaluating this technology should review current product branding and availability under Qlik.