Before defining elastic search. I give you a brief explanation that you are in the field of data science and big data and you may have heard about elastic search and you might be wondering that how does search engine technology helps you in extracting meaning from your data at scale. How do you get answers in milliseconds? Let’s talk about that it started off as a scalable lucence version.it adds a horizontally scalable search engine. There are shards and each “shard” is an inverted index of documents but not just for full-text search.it only can handle structure data, & can aggregate data quickly and often doing a faster solution than Hadoop/ spark /flink/etc.

Often, you listen to engineers. Their sorts of priorities and various forms include in it. Now, among all engineering streams. Elastic is an integral part of the elastic search engineer. Let’s talk about an elastic certified engineer. An elastic engineer possesses the ability and skills on performing a necessary task to build complete solutions of elastic search and monitoring the functions of install, configure, and manage the cluster, indexing data into the clusters, query, and analyzing the index data.

What are the benefits of elastic engineer?

 Getting ideas about the elastic engineer. Now, it’s time to take action on benefits. What’re benefits do you getting from elastic engineering and you just can’t say words for it. After the completion of the exam and certifications then you behave as you saying. There are some of the benefits according to the ideology of elastic engineer as categorized:

• Scalability
• Really fast
• Multilingual
• Document oriented
• Auto-completion & instant search
• Scheme free

Why we should learn elastic engineer?

If you are talking about the elastic engineer then first you should assemble the conceptual knowledge about why you are learning or necessity along with the cause? Through, the learning path of elastic certification. It’s proven to be fruitful in the sectors of IT entities and organizations. Due to its Performa-based courses/training along with the professionals. That will give you the using concept with skills, task building techniques on the elastic search on the basis of two courses which are packed with elastic search engineer I and elastic search engineer II. Its certification is based on the elastic search engineer exam and after completing the procedure of all. It will create awesome technological advances, feed awareness about the industry and its impacts on the environment. One of the prominent roles of elastic certification is the job opportunity, creativity, and innovation in favor of drastic enhances.

How our elastic engineer course/training would help?

After taking the course structure of elastic search engineer. It will create ideology about the elastic search and perform functionalities in the areas of sharding techniques & strategies, tuning for ingest speed, monitoring the stack, backing up your data, and query theory.

Getting command all over the ability and data experiences will make you job achiever, exam certification and build techniques to qualify in performance considerations, high availability, always use bulk, increase refresh, disable replicas, templates/mapping, monitoring, delete snapshots, general things to avoid, general things to do, and tools for console & search profiler for better technology development. These training courses will help you to understand the advances in elastic search, develop skills, better performances and produce high-paid salary job opportunities towards the IT infrastructure. In this matter, I would like to prefer this course in the industry of software development for your future goals.

Related video:

What Are Google Cloud Machine Images?

Google Cloud introduced a feature called Machine Images designed to make it significantly easier for system administrators and data scientists to set up, replicate, and maintain specialized virtual machine (VM) environments in the cloud.

At its core, a machine image stores all the information required to fully restore a Compute Engine VM instance. This goes beyond what Google Cloud’s older custom images feature offered. While custom images are limited to a single boot disk, machine images capture:

• All attached disks (boot and data disks)
• Instance configuration properties
• Instance metadata
• Associated IAM permissions

This comprehensive snapshot means a machine image represents a complete, reproducible environment — not just an OS image.

Why This Matters for Data Science and DevOps Teams

Data science workloads typically require carefully configured environments: specific library versions, GPU drivers, mounted datasets, and fine-tuned system settings. Manually reproducing these environments across multiple cloud instances is time-consuming and error-prone.

Machine images eliminate this friction. Once a VM has been configured to the desired state, teams can capture it as a golden machine image and use it as the authoritative baseline for spinning up additional instances — all configured identically to the source.

According to Ari Liberman, a product manager for Google Compute Engine, machine images are designed to sit at the heart of scalability, backup, and disaster recovery strategies. They use the same differential disk backup technology that powers incremental snapshots, making instance backups both reliable and cost-effective.

The “Golden Image” Workflow

Liberman illustrated the intended workflow with a practical example:

“Imagine that you’ve created and configured an instance as part of your web application, and you want to use it as the basis of other instances. With machine images, you can capture your web application exactly as you want it and save it as a golden machine image. You can then use that machine image to launch as many instances as you need, configured in the exact same way as your source instance. You can also share the machine image with other projects.”

Ari Liberman, Product Manager, Google Compute Engine

While the example references a web application, the same pattern applies directly to data science and ML engineering contexts — for example, capturing a configured Jupyter environment with CUDA drivers and Python dependencies installed, then sharing it across a team or replicating it for parallel experiment runs.

Override Functionality: Flexibility Without Sacrifice

Machine images also support override functionality, allowing individual properties of new VM instances to be selectively modified at launch time. This means teams can use a shared golden image as a base while still customizing things like machine type, network tags, or startup scripts on a per-instance basis — without altering the source image itself.

Key Use Cases

• Scalability: Rapidly launch multiple identical VM instances from a single source image to handle increased workloads.
• Backup and recovery: Capture point-in-time snapshots of full VM environments, including all attached disks, for reliable disaster recovery.
• Environment standardization: Share a validated, reproducible VM configuration across teams and projects.
• Onboarding: Provide new team members with a pre-configured environment instantly, reducing setup time.

Machine Images vs. Custom Images vs. Snapshots

Note: This comparison reflects the state of these features as of the March 2020 announcement. Google Cloud feature capabilities may have evolved since then.

How to Access Machine Images

At the time of announcement, Machine Images were available in beta and could be accessed through:

• The Google Cloud Console
• The gcloud CLI
• The Compute Engine REST API

For the current status of this feature and up-to-date documentation, refer to the official Google Cloud Machine Images documentation.

It is no longer a surprise that the volume of enterprise data is growing rapidly. What is less widely discussed is that the origin of that data is shifting just as fast. An increasing share of data is now being generated in the cloud and at the network edge, not in traditional data centres. This shift forces engineering and operations teams to rethink where and how they analyse, understand, and manage data — ideally without routing everything through a centralised data centre as an intermediary.

Three areas are experiencing the most significant change when it comes to enterprise data strategy:

1. Edge Data Requires Edge Data Management

Edge computing buildout is already well underway, and its pace is accelerating. Key drivers include the Internet of Things (IoT), autonomous vehicles, and biometric systems — all of which generate large volumes of data far from centralised infrastructure. According to IDC forecasts cited in the original source, 50% of all enterprise IT infrastructure could be deployed at the edge by 2023, compared to roughly 10% at the time of writing.

This creates a straightforward operational challenge: if data is being generated at the edge, why move all of it to a data centre before you can do anything useful with it? The answer increasingly is that you should not. Edge data management addresses this by allowing teams to:

• Analyse data locally at the point of generation.
• Classify which data is high-value and needs to be sent back to the data centre or cloud for deeper processing.
• Process or discard lower-value data directly at the edge, reducing bandwidth costs and latency.

As more applications are purpose-built for edge deployment, the tooling around edge data management — including lightweight databases, stream processing, and local orchestration — will grow in maturity and adoption.

2. Multi-Cloud Adoption Makes Cross-Cloud Data Management a Requirement

Most large enterprises using public cloud services have already adopted some form of hybrid or multi-cloud strategy. Rather than committing to a single cloud provider, organisations are increasingly distributing workloads across two or more clouds — whether for cost optimisation, resilience, regulatory compliance, or to avoid vendor lock-in.

The operational consequence is data sprawl: copies of data, pipelines, and storage systems spread across AWS, Azure, Google Cloud, on-premises infrastructure, and potentially the edge. Managing this sprawl is genuinely difficult when each cloud provider offers its own native data tools with proprietary interfaces.

Cross-cloud data management platforms aim to solve this by providing a unified layer to discover, catalogue, govern, and move data regardless of where it physically resides. Key capabilities engineering teams should evaluate in such platforms include:

• Vendor-neutral connectivity: Integration with multiple storage and compute providers without requiring data to be normalised into a proprietary format.
• Unified metadata and cataloguing: A single view of what data exists, where it lives, and who owns it.
• Policy-based governance: Consistent access controls, retention policies, and compliance rules applied across all environments.
• Data portability: The ability to move or replicate data across clouds without manual re-engineering.

As multi-cloud becomes the default rather than the exception, solutions that are tightly coupled to a single cloud vendor’s ecosystem become a liability rather than an asset.

3. Analytics-Driven AI Moves from Buzzword to Operational Tool

Artificial Intelligence (AI) and Machine Learning (ML) have been prominent themes in enterprise technology conversations for several years. For much of that time, they occupied a space somewhere between genuine potential and marketing overstatement. That balance is shifting.

The application of AI to data management itself — rather than just to business analytics — is where some of the most practical near-term value lies. The core idea is straightforward: you cannot manage data you do not understand. By applying ML models to analyse data characteristics, usage patterns, access frequency, and content, data management platforms can make better automated decisions about storage tiering, archival, deletion, access control, and anomaly detection.

Concretely, analytics-driven AI in data management can enable:

• Intelligent data tiering: Automatically moving cold data to cheaper storage without manual intervention.
• Anomaly and threat detection: Identifying unusual data access patterns that may indicate a security incident.
• Data quality improvement: Flagging duplicate, incomplete, or inconsistent records across distributed repositories.
• Capacity forecasting: Predicting storage growth trends to inform infrastructure planning.

Data lakes spanning multiple storage repositories provide the breadth of signal needed to train and refine these models effectively. The practical output is a management layer that becomes smarter over time — reducing toil for operations teams while improving data hygiene across the organisation.

Bringing It Together: Manage Data Where It Lives

The common thread across all three of these trends is a shift away from the assumption that all data must eventually pass through a central data centre before it can be properly managed. That assumption made sense when almost all enterprise data was generated on-premises. It is increasingly untenable when data is being produced continuously at the edge, in multiple clouds, and across distributed application stacks.

Engineering and operations teams that get ahead of this shift will invest in:

• Edge-native data management tooling that reduces unnecessary data movement.
• Cloud-agnostic governance platforms that work across all environments without vendor lock-in.
• AI-assisted analytics that help teams understand and act on data at scale without proportionally growing headcount.

The organisations that treat data management as a first-class engineering concern — rather than an afterthought bolted on after architecture decisions are made — will be best positioned to derive real value from their data wherever it happens to live.

The IoT Data Problem

The rapid proliferation of Internet of Things (IoT) devices is creating a data challenge that traditional cloud infrastructure was not designed to absorb. As device counts grow into the tens of billions, the sheer volume of data generated strains every layer of the stack—from the device itself to the wide-area network and on to centralised cloud data centres.

The core tension is straightforward: cloud servers deliver powerful compute and storage at scale, but they are physically remote from IoT endpoints. Sending time-sensitive telemetry across wide geographic distances introduces latency, consumes bandwidth, and increases cost. For use cases such as autonomous vehicles, industrial automation, or real-time medical monitoring, that latency is unacceptable. This is the driving force behind edge and fog computing.

Edge Computing: Processing at the Source

Edge computing moves data processing physically closer to where data originates—on or near the IoT device itself, or at a local gateway. By shortening the path data must travel, edge computing delivers two immediate benefits:

• Reduced latency: Decisions can be made in milliseconds rather than waiting for a round-trip to a remote data centre.
• Lower bandwidth consumption: Only relevant, pre-processed data needs to be sent upstream, reducing WAN traffic and associated costs.

However, edge nodes are resource-constrained by design. Running complex analytics pipelines, training machine learning models, or retaining large datasets directly on an edge device is often impractical. This is where fog computing enters the picture.

Fog Computing: The Intermediate Layer

The term fog computing was coined by Cisco to describe a decentralised computing architecture that sits between cloud data centres and the data-generating edge. Rather than a binary choice between cloud and edge, fog computing introduces an intermediate tier—local or regional nodes that offer more compute, storage, and networking capability than a constrained edge device, while remaining far closer to end-users than a centralised cloud region.

Think of the architecture in three tiers:

1. Edge tier: IoT sensors, actuators, and gateways. Data is collected and lightly pre-processed here.
2. Fog tier: Local or regional compute nodes (on-premises servers, micro data centres, or colocation facilities). Heavier analytics, short-term storage, and filtering happen here before data is forwarded upstream.
3. Cloud tier: Centralised infrastructure for long-term storage, large-scale machine learning, business intelligence, and archival.

Fog computing is best understood as a complementary strategy rather than a replacement for either edge or cloud. It extends cloud capabilities downward while giving edge deployments access to resources they could not otherwise reach.

Key Differences: Edge vs. Fog

Edge and fog computing share the same underlying principle—bring computation closer to the data source—but they differ in where processing occurs and the resources available:

Why Centralised Cloud Alone Is Not Enough

Cloud computing remains indispensable for workloads that demand elastic scale, centralised data governance, or long-running training jobs. But it has structural limitations for IoT at scale:

• Geographic distance: Cloud regions are concentrated in a small number of locations. Routing all device traffic through them adds consistent latency.
• Bandwidth cost: Transmitting raw telemetry from billions of devices to the cloud is expensive and often technically infeasible on available WAN capacity.
• Centralised failure risk: A single cloud region outage can affect all connected workloads simultaneously if no intermediate layer exists.
• Regulatory constraints: Data sovereignty requirements in some jurisdictions prohibit certain data from leaving a geographic region, making centralised cloud problematic.

A Multi-Tier Distributed Architecture in Practice

Organisations addressing digital transformation at scale are increasingly moving toward multi-tier distributed architectures rather than treating infrastructure as a binary cloud-or-on-premises decision. A practical layered model looks like this:

1. Enterprise core: On-premises or private cloud hosting intellectual property, high-density compute, and model training workloads.
2. Cloud services layer: Public cloud for elastic burst capacity, managed services, and long-term data retention.
3. Distributed colocation: Multi-tenant, multi-site colocation facilities act as regional fog nodes—closer to end-users, connected to multiple network providers.
4. Edge delivery: On-device or near-device processing, potentially integrated with 5G infrastructure for ultra-low-latency delivery.

This architecture gives engineering and operations teams granular control over where specific workloads run, how capacity scales, and how data flows between tiers. It also shortens the physical and logical distance between compute resources and end-users—a key factor in delivering consistent application performance.

Operational Considerations for DevOps Teams

Adopting edge and fog computing introduces new operational complexity. DevOps and platform engineering teams should plan for:

• Fleet management: Edge and fog nodes must be provisioned, updated, and monitored at scale. Containerisation (e.g., Kubernetes at the edge via distributions like K3s or MicroK8s) simplifies workload portability across tiers.
• Observability: Distributed telemetry pipelines must aggregate metrics, logs, and traces from edge nodes back to a central observability platform without creating a bandwidth bottleneck.
• Security: A larger attack surface across distributed nodes demands zero-trust networking, mutual TLS between tiers, and automated certificate rotation.
• Data consistency: Applications that span edge, fog, and cloud tiers must handle eventual consistency, offline operation, and conflict resolution gracefully.
• CI/CD at the edge: Deployment pipelines need to accommodate constrained environments, staged rollouts to remote nodes, and rollback mechanisms that do not require cloud connectivity.

Summary

The data explosion driven by IoT growth cannot be solved by centralised cloud infrastructure alone. Edge computing reduces latency by moving processing to the device; fog computing provides an intermediate tier with greater compute capability while remaining closer to end-users than the cloud. Together, they form a layered architecture that allows organisations to place workloads where they run most efficiently—balancing latency, bandwidth, cost, and resilience.

For DevOps engineers, this means designing systems and pipelines that span multiple tiers, embracing fleet management tooling, and treating distributed consistency and security as first-class concerns rather than afterthoughts.

2019 saw more data organizations running advanced analytics, AI, and machine learning workloads than ever before. Heading into 2020, the expectation was a spike in both the number of technologies and the data teams required to support those workloads internally. AI and analytics teams were predicted to merge into unified data organizations, tackling challenges such as hybrid cloud strategy, Kubernetes-based analytics, and the ongoing evolution of the Hadoop ecosystem.

1. The Rise of the Hybrid Cloud

After years of discussion, hybrid cloud was expected to move from concept to practical reality in 2020. Large enterprises were already refusing to expand on-premises Hadoop deployments, choosing instead to invest in public cloud capacity. However, most were unwilling to migrate core enterprise data off-premises entirely.

The predicted model: data stays on-prem, compute bursts to the cloud. This approach targets peak demand and unpredictable workloads, where elastic cloud compute is more cost-effective than maintaining excess on-premises capacity. Technologies enabling seamless data-local-compute-remote architectures were expected to lead the hybrid cloud wave.

2. One Machine Learning Framework to Rule Them All

By late 2019, operationalizing ML model training had become a priority across organizations of all sizes. Multiple frameworks competed for dominance, but no clear winner had emerged. The analogy drawn was to earlier data tooling consolidations: Apache Spark became the standard for data transformation, and Presto was emerging as the leading choice for interactive querying.

The prediction for 2020 was that the model training space would see similar consolidation, with PyTorch and TensorFlow identified as the leading contenders for the dominant framework position.

3. Kubernetifying the Analytics Stack

Containers and Kubernetes had proven highly effective for stateless workloads — web servers, microservices, self-contained databases — but adoption for advanced analytics and AI workloads lagged behind. The prediction was that 2020 would mark the shift toward AI and analytics workloads becoming mainstream on Kubernetes.

The core technical challenge: solving data sharing and elasticity. Moving data from remote silos into Kubernetes clusters improves data locality, which is critical for performance-sensitive analytics jobs. This required new patterns for stateful workload management that the community was actively developing.

4. HDFS Is Dead — But Hadoop Compute Lives On

The “Hadoop is dead” narrative was common heading into 2020, but the reality was more nuanced. The prediction separated two distinct components of the Hadoop ecosystem:

• HDFS (Hadoop Distributed File System): Predicted to decline due to operational complexity, high cost, and a fundamental inability to support elastic compute scaling. Data stored in HDFS would migrate to cloud object storage (such as S3 or GCS) or on-premises object storage alternatives.
• Hadoop compute frameworks (Spark, Presto): Predicted to remain strong. These tools had already been adopted well beyond the Hadoop ecosystem and were seen as foundational for large-scale data processing regardless of the underlying storage layer.

The key insight: compute and storage decoupling was accelerating, making storage-layer-agnostic compute frameworks more valuable, not less.

5. AI and Analytics Teams Merge Into a Unified Data Organization

The organizational structure of data teams was also expected to evolve. Hadoop platform teams had already transitioned into analytics and data engineering teams. The next step was absorbing AI and ML functions into the same organizational unit, reflecting how deeply intertwined structured data analytics and machine learning had become.

The convergence of statistical modeling and computer science into modern ML meant that data, analytics, and AI teams were often working on the same datasets with overlapping goals. The predicted outcome: organizations would build dedicated, unified data stack teams responsible for storage, compute, and modeling across on-prem and cloud environments.

6. The Talent Gap Will Constrain Data Technology Adoption

A persistent challenge identified for 2020 was the shortage of engineers who could bridge data engineering, cloud architecture, and AI/ML workloads. Building production data stacks that incorporate modern cloud, Kubernetes, and ML tooling requires a rare combination of skills.

The concern: as executive-level interest in data-driven strategies grew, the pipeline of engineers capable of architecting those systems was not keeping pace. Companies making ambitious “data-everywhere” commitments were expected to fall short without a parallel investment in developing or recruiting the necessary talent.

7. China’s Cloud Adoption Leapfrog

While U.S. enterprises had been migrating to public cloud infrastructure for several years, Chinese enterprises had largely invested in on-premises infrastructure for data-intensive workloads. The prediction for 2020 was that this would change dramatically — with Chinese enterprises leapfrogging incremental cloud adoption to deploy advanced workloads directly in the public cloud at significant scale.

Public cloud providers operating in China were expected to see growth rates potentially outpacing those of established Western cloud giants, driven by new use cases, burst compute for peak loads, and migration of existing on-prem workloads.

Key Takeaways for Data Engineering Teams

• Plan for hybrid architectures: Separating data residency from compute elasticity is a practical near-term strategy, not a distant goal.
• Watch the ML framework landscape: Framework consolidation simplifies toolchain decisions; betting on widely-adopted frameworks reduces long-term migration risk.
• Invest in Kubernetes expertise for analytics: Stateful analytics workloads on Kubernetes require different patterns than stateless services — build that knowledge proactively.
• Decouple storage from compute: Moving away from HDFS toward object storage unlocks the elasticity needed for modern cloud-native analytics.
• Build unified data teams: Siloed analytics and AI functions create duplication; unified teams sharing infrastructure and data assets are more efficient.
• Address the talent gap directly: Upskilling existing engineers in cloud-native and ML tooling is often faster than competing for a limited pool of specialists.

The trends outlined here represent the data engineering inflection points anticipated heading into 2020 — a year when advanced analytics and AI were expected to move from experimental to operational across a much wider range of organizations.

Overview

Michael Berthold, CEO and co-founder of the open-source data analytics platform KNIME, delivered the keynote at the KNIME Fall Summit 2019. His talk focused on the complete end-to-end data science cycle, which he divided into two top-level categories—Create and Productionize—each containing two distinct phases.

Phase 1 – Gather & Wrangle

The first phase covers data acquisition and preparation. KNIME addresses this through a rich library of connector and transformation nodes, as well as big data extensions that integrate third-party technologies directly into a visual workflow. Supported integrations mentioned by Berthold include:

• Amazon Redshift
• H2 (in-memory database)
• Apache Hive and Impala
• Apache Spark

At the time of the keynote, KNIME had also extended its cloud file system connectivity, making it easier to read from and write to cloud storage buckets alongside on-premises data sources.

Phase 2 – Model & Visualize

This phase covers model building, library integration, and result visualisation. KNIME provides domain-specific nodes for several verticals, including:

• Life science – specialised cheminformatics and bioinformatics nodes
• Text processing – NLP preprocessing and classification
• Image analysis – pixel-level processing workflows
• Time-series analysis – lag features, smoothing, and forecasting nodes

For practitioners who need more control, KNIME integrates popular ML frameworks such as H2O and XGBoost, and allows custom code to be embedded via R and Python script nodes. This hybrid approach—low-code visual pipelines plus escape hatches for custom code—means teams with mixed skill levels can collaborate on the same workflow.

Phase 3 – Deploy & Manage

Once a model is ready for production, KNIME provides mechanisms to expose it to end users and downstream systems:

• Analytics applications – interactive web-based frontends for non-technical consumers
• Web services – REST endpoints that expose workflows as APIs
• KNIME Model Process Factory – enables on-demand model retraining, automated initialisation, and monitoring and alerting

KNIME also addresses production governance concerns—versioning, backwards compatibility, compliance controls, and documented best practices—so data science teams can maintain audit trails as models evolve.

Phase 4 – Consume & Optimize

The final phase closes the loop by making ML workflows accessible to business consumers. KNIME supports two primary consumption modes:

• Workflow as a Service – direct deployment so downstream systems can invoke a model programmatically
• Analytics Application – a guided UI that lets non-data-scientists run and interact with a model without writing code

Three Generations of Data Science Automation

Berthold framed the maturity of ML automation in three generations:

First Generation – Pre-canned Pipelines

Clean data enters a fixed pipeline, a standard set of transformations is applied, and a model is produced. There is little to no customisation or feedback loop. This covers basic batch scoring scenarios.

Second Generation – Interactive and Continuous

Customised interactions are introduced in the Model & Visualize and Deploy & Manage phases. Continuous deployment practices are applied, and users can provide feedback that influences subsequent model versions.

Third Generation – Mix-and-Match Automation

The Gather & Wrangle and Consume & Optimize phases become fully automated. Model building still involves customised human interaction, while deployment introduces guided interaction—where the platform makes suggestions and the practitioner confirms them rather than configuring everything manually.

Data Science Abstraction Through Components

Berthold concluded by discussing how KNIME uses Components—encapsulated, reusable workflow segments—to raise the level of abstraction across the pipeline. Components can encapsulate complex logic for steps such as:

• Feature selection and engineering
• Model interpretation and explainability
• Model management (versioning, comparison)
• Model deployment to production

By packaging these steps as components, organisations can standardise how ML problems are solved internally, reducing duplication and lowering the barrier for less experienced team members.

Four Adoption Profiles for Data Science

Berthold suggested that organisations should align their data science investment with their actual business context. He described four adoption profiles:

1. Standard problems with limited business impact – Simple, commodity data science tools are sufficient; the cost of a custom solution outweighs the benefit.
2. Applications requiring no specialist in-house expertise – Pre-built ML models or no-code tools can solve the problem without a dedicated data science team.
3. Productionising data science for value creation – Continuous delivery of models, integration of user feedback, and adoption of new technologies become critical differentiators.
4. Deep innovation embedded in the business – The organisation explores cutting-edge ML/AI research, integrates proprietary R&D, and treats data science as a core strategic capability.

KNIME Product Ecosystem

For teams looking to implement the patterns Berthold described, KNIME offers several products that map to different points in the end-to-end cycle:

• KNIME Analytics Platform – The open-source desktop environment for building and testing data science workflows.
• KNIME Server – A commercial server product that adds scheduling, team collaboration, and centralised deployment. Includes the KNIME Web Portal for browser-based workflow execution.
• Data Science as a Service – A managed offering for organisations that do not want to host their own KNIME Server infrastructure.
• KNIME Hub – A community and enterprise repository for sharing and discovering workflow components.
• KNIME Cloud Cognitive Services – Integrations with cloud-based AI APIs (announced at the time of the summit).

For the most up-to-date product information and documentation, visit the official KNIME website.

In this special guest feature, Ryohei Fujimaki, Ph.D., Founder and CEO of dotData, discusses how AI and ML are having a profound impact on enterprise digital transformation becoming crucial as a competitive advantage and even for survival. As the field grows, four trends emerge, shaping data science in the next five years. dotData is a spin-off of NEC Corporation and the first company focused on delivering full-cycle data science automation for the enterprise. Dr. Fujimaki is a world-renowned data scientist and was the youngest research fellow appointed in the 119-year history of NEC.

According to the Gartner Group, digital business reached a tipping point last year, with 49% of CIOs reporting that their enterprises have already changed their business models or are in the process of doing so. When Gartner asked CIOs and IT leaders which technologies they expect to be most disruptive, artificial intelligence (AI) was the top-mentioned technology.

AI and ML are having a profound impact on enterprise digital transformation becoming crucial as a competitive advantage and even for survival. As the field grows, four trends emerge, shaping data science in the next five years:

Accelerate The Full Data Science Life-Cycle

The pressure to grow ROI from AI and ML initiatives has pushed demand for new innovative solutions that accelerate AI and data science. Although data science processes are iterative and highly manual, more than 40% of data science tasks are expected to be automated by 2020, according to Gartner, resulting in increased productivity and broader usage of data across the enterprise.

Recently, automated machine learning (AutoML) has become one of the fastest-growing technologies for data science. Machine learning, however,  typically accounts for only 10-20% of the entire data science process. Real pains exist before the machine learning stage with data and feature engineering.  The new concept of data science automation goes beyond machine learning automation, including data preparation, feature engineering, machine learning, and the production of full data science pipelines. With data science automation, enterprises can genuinely accelerate AI and ML initiatives.

Leverage Existing Resources for Democratization

Despite substantial investments in data science across many industries, the scarcity of data science skills and resources often limits the advancement of AI and ML projects in organizations.  The shortage of data scientists has created a challenge for anyone implementing AI and ML initiatives, forcing a closer look at how to build and leverage data science resources.

Other than the need for highly specialized technical skills and mathematical aptitude, data scientists must also couple these skills with domain/industry knowledge that is relevant to a specific business area. Domain knowledge is required for problem definition and result validation and is a crucial enabler to deliver business value from data science. Relying on “data science unicorns” that have all these skill sets is neither realistic nor scalable.

Enterprises are focusing on repurposing existing resources as “citizen” data scientists. The rise of AutoML and data science automation can unlock data science to a broader user base and allow the practice to scale. By empowering citizen data scientists allowing them to execute standard use cases, skilled data scientists can focus on high-impact, technically-challenging projects to produce higher values.

Augment Insights for Greater Transparency

As more organizations are adopting data science in their business process, relying on AI-derived recommendations that lack transparency is becoming problematic. Increased regulatory oversight like the GDPR has exacerbated the problem. Transparent insights make AI models more “oversight” friendly and have the added benefit of being far more actionable.

White-box AI models help organizations maintain accountability in data-driven decisions and allow them to live within the boundaries of regulations. The challenge is the need for high-quality and transparent inputs (aka “features”), often requiring multiple manual iterations to achieve the needed transparency. Data science automation allows data scientists to explore millions of hypotheses and augments their ability to discover transparent and predictive features as business insights.

Operationalize Data Science in Business

Although ML models are often tiny pieces of code, when models are finally deemed ready for production, deploying them can be complicated and problematic. For example, since data scientists are not software engineers, the quality of their code may not be production-ready. Data scientists often validate the models with down-sampled datasets in labs environments and models may not be scalable enough for production-scale datasets. Also, the performance of deployed models decreases as data invariably changes, making model maintenance pivotal to extract business value from AI and ML models continuously. Data and feature pipelines are much bigger and more complex than ML models themselves, and operationalizing data and feature pipelines is even more complicated.  One of the promising approaches is to leverage concepts from continuous deployment through APIs. Data science automation can generate APIs to execute the full data science pipeline, accelerating deployments while also providing an ongoing connection to development systems to accelerate the optimization and maintenance of models.

Data science is at the heart of AI and ML. While the promise of AI is real, the problems associated with data science are also real. Through better planning, closer cooperation with line of business and by automating the more tedious and repetitive parts of the process, data scientists can finally begin to focus on what to solve, rather than how to solve.