Before defining elastic search. I give you a brief explanation that you are in the field of data science and big data and you may have heard about elastic search and you might be wondering that how does search engine technology helps you in extracting meaning from your data at scale. How do you get answers in milliseconds? Let’s talk about that it started off as a scalable lucence version.it adds a horizontally scalable search engine. There are shards and each “shard” is an inverted index of documents but not just for full-text search.it only can handle structure data, & can aggregate data quickly and often doing a faster solution than Hadoop/ spark /flink/etc.

Often, you listen to engineers. Their sorts of priorities and various forms include in it. Now, among all engineering streams. Elastic is an integral part of the elastic search engineer. Let’s talk about an elastic certified engineer. An elastic engineer possesses the ability and skills on performing a necessary task to build complete solutions of elastic search and monitoring the functions of install, configure, and manage the cluster, indexing data into the clusters, query, and analyzing the index data.

What are the benefits of elastic engineer?

 Getting ideas about the elastic engineer. Now, it’s time to take action on benefits. What’re benefits do you getting from elastic engineering and you just can’t say words for it. After the completion of the exam and certifications then you behave as you saying. There are some of the benefits according to the ideology of elastic engineer as categorized:

• Scalability
• Really fast
• Multilingual
• Document oriented
• Auto-completion & instant search
• Scheme free

Why we should learn elastic engineer?

If you are talking about the elastic engineer then first you should assemble the conceptual knowledge about why you are learning or necessity along with the cause? Through, the learning path of elastic certification. It’s proven to be fruitful in the sectors of IT entities and organizations. Due to its Performa-based courses/training along with the professionals. That will give you the using concept with skills, task building techniques on the elastic search on the basis of two courses which are packed with elastic search engineer I and elastic search engineer II. Its certification is based on the elastic search engineer exam and after completing the procedure of all. It will create awesome technological advances, feed awareness about the industry and its impacts on the environment. One of the prominent roles of elastic certification is the job opportunity, creativity, and innovation in favor of drastic enhances.

How our elastic engineer course/training would help?

After taking the course structure of elastic search engineer. It will create ideology about the elastic search and perform functionalities in the areas of sharding techniques & strategies, tuning for ingest speed, monitoring the stack, backing up your data, and query theory.

Getting command all over the ability and data experiences will make you job achiever, exam certification and build techniques to qualify in performance considerations, high availability, always use bulk, increase refresh, disable replicas, templates/mapping, monitoring, delete snapshots, general things to avoid, general things to do, and tools for console & search profiler for better technology development. These training courses will help you to understand the advances in elastic search, develop skills, better performances and produce high-paid salary job opportunities towards the IT infrastructure. In this matter, I would like to prefer this course in the industry of software development for your future goals.

Related video:

The DevSecOps Promise vs. Reality

For several years, DevSecOps has been positioned as the answer to a persistent problem: how do you ship cloud applications quickly without compromising security? The traditional approach — retrofitting complex security policies onto a hierarchical development pipeline — simply doesn’t scale in a world of rapid, collaborative cloud development.

The DevSecOps model proposes a cleaner solution: bake security into the software development lifecycle from the earliest stages. Instead of a separate security team reviewing finished work, every developer becomes a security stakeholder. This “shift left” philosophy is designed to be preventative rather than reactive — catch problems at the source, and you spend far less time firefighting later.

In theory, this makes particular sense on cloud platforms that assume rapid deployment, distributed teams, and continuous delivery. There is no longer a hard boundary between the development team, the security team, and the operations team. In practice, however, “shift left” is more talked about than actually practised. Many organizations have adopted the terminology without making the cultural or tooling changes required to back it up.

The Governance Gap in Cloud Development

One of the core challenges is visibility and control. As Guy Eisenkot, co-founder and VP of Product at Bridgecrew, puts it:

“There is a lack of control in this model. With the shift into cloud development and the fact that coders can always get a better answer on Stack Overflow and GitHub, it’s become practically impossible to track the supply chain. It’s a governance problem.”

Security has become a focal point of continuous development, but it remains a work in progress alongside competing priorities like performance and new features. And crucially, the available tools — while useful — are not sufficient on their own.

Consider a common scenario: an Elasticsearch cluster is inadvertently left in a publicly accessible state. Under a traditional hierarchical model, a security scan flags this issue and sends it to a ticketing system, where it may sit for hours before someone with the right access resolves it. The more effective approach is to surface this misconfiguration as part of the CI/CD pipeline, pushing an immediate alert to the team that introduced the error. Remediation can then happen at compile time rather than at runtime — preventing a potentially serious exposure rather than scrambling to contain one.

“It’s not simply that the tools are not enough — it’s that too many of them are reporting alerts and sending them to Slack channels, Jira tickets, and other queues instead of going to a developer’s desktop and CI tools.” — Guy Eisenkot, Bridgecrew

Infrastructure as Code: A Double-Edged Sword

Infrastructure as Code (IaC) is often cited as the mechanism that makes security “everyone’s business.” By turning the provisioning of cloud and data centre services into machine-readable code — bundled into automated workflows or templates — IaC removes the need for manual configuration steps and fragmented console tools.

Tools like Terraform, Chef, and Puppet have become central to this model. IaC configuration can be pushed directly to servers or pulled via a configuration server to a cloud provider’s infrastructure. Terraform, for example, typically treats resources as immutable: rather than modifying an existing setup, it tears down and rebuilds the resource from scratch.

But IaC introduces its own risks. The same mechanisms that give developers extraordinary speed and freedom also make it easy to embed security misconfigurations at scale. According to analysis by Bridgecrew, at least four in ten IaC templates contain a misconfiguration that can slip through into runtime vulnerabilities. The tools to provision cloud infrastructure have been successfully decentralised — but the expertise required to manage that provisioning securely has often remained centralised and is therefore becoming less effective as teams grow.

As Eisenkot explains, deployment velocity compounds the problem:

“It suddenly doesn’t take six months to deploy a change to an application — it takes a few minutes. This speed is a blessing in disguise because it creates challenges when we deploy software very fast. Where do we make the compromises?”

The Blind Spots in Modern Cloud Architectures

A particularly difficult challenge is that compliance and risk managers often cannot see what is happening across environments where developers have been given broad, often necessary freedoms.

“There are blind spots. How do you ensure you know what’s going on in your environment? The idea that we know everything that’s going on in our architecture is an assumption we can’t make any more.” — Guy Eisenkot

This plays out in mundane but consequential ways. Older IaC templates — some dating back years — may encode practices that are now considered insecure. Encryption at rest, for instance, was not enabled by default in some database packages until relatively recently. Surfacing these historical misconfigurations through automated scanning can yield quick wins on day one of a security improvement initiative.

What a Practical DevSecOps Platform Looks Like

Bridgecrew’s approach illustrates one way to bridge the gap between intent and practice. Rather than adding another alert queue, the platform is designed to integrate directly into the developer’s existing workflow — scanning IaC templates for misconfigurations and policy violations before errors reach deployed infrastructure, and issuing pull requests to surface issues where developers are already working.

Key capabilities in this kind of platform include:

• Build-time scanning: Identifying IaC misconfigurations and policy violations during the build phase — before they reach production infrastructure. This also enables the generation of compliance reports against standards such as PCI DSS.
• Cloud integration: Connecting to AWS, Microsoft Azure, and Google Cloud Platform, as well as scanning configurations expressed in CloudFormation, Kubernetes, and Terraform.
• Read-only deployment model: The platform can be deployed by running a CloudFormation or Terraform template on the target cloud account, which creates a read-only role. This enables scheduled scans of configuration states and continuous evaluation without requiring write access to production environments.
• Playbooks and remediation scripts: Automation that addresses common problems across cloud platforms, reducing the number of manual steps required to resolve a known class of misconfiguration.

Playbooks have real-world limitations, however. In highly automated cloud deployments, a change applied by a playbook can be inadvertently reversed by a subsequent automated change. Remediation therefore needs to be integrated at the pipeline level, not applied as a one-off fix.

Open Source Tools Supporting IaC Security

Bridgecrew has contributed several open source tools to this space:

• Checkov: A static analysis tool for IaC that scans Terraform, CloudFormation, and Kubernetes configurations for misconfigurations.
• AirIAM: An AWS Least Privilege Terraformer that generates a Terraform template designed to reduce IAM permissions to the minimum required — limiting the blast radius of a compromise.
• Terragoat: A deliberately vulnerable learning environment for experimenting with secure cloud configuration pipelines — useful for training developers to recognise common mistakes.

The Technical Challenges of Scanning IaC at Scale

Scanning IaC templates accurately is harder than it sounds. Eisenkot is candid about the complexity involved:

“It becomes extremely complicated when trying to build a data model that depicts how configuration will eventually manifest based on the prescribed plan. It is imperative to extract its variables and dependencies against the entire code base. Without this step, scans could result in both false positives — configurations that don’t extract all their dependencies — or false negatives that are not identified due to misinterpretation of the IaC plan.”

This highlights an important point for teams evaluating IaC scanning tools: a tool that produces too many false positives will be ignored by developers, while one that misses real issues due to incomplete dependency resolution can create a false sense of security. The quality of the underlying data model matters as much as the scanning rules themselves.

Beyond Tooling: Education and Culture

Ultimately, Bridgecrew’s experience points to a lesson that applies more broadly across DevSecOps: giving developers a new tool to detect bad decisions is only part of the solution. The more durable fix is education — helping developers understand why certain configurations are dangerous and how to make better decisions before writing the code in the first place.

“We learned that it’s not only about giving developers the panic button, but we also had to educate them on making the right decisions when building new infrastructure in the public cloud.”

By surfacing misconfigurations at the pull request or build report stage — with actionable guidance on how to fix them — teams can address problems before they reach production rather than after. This changes the developer’s relationship with security from a compliance burden to an integral part of the build process.

Key Takeaways for DevOps and Platform Teams

• “Shift left” only works if security feedback reaches developers in the tools they already use — not in a separate queue or alert channel.
• IaC accelerates both delivery and the spread of misconfigurations. Automated scanning at build time is essential, not optional.
• Compliance and risk visibility requires read access across the full cloud environment, including historical templates that may encode outdated practices.
• Static analysis tools like Checkov help, but their accuracy depends on correctly resolving variable dependencies across the entire codebase.
• Playbook-based remediation is a useful starting point, but must be integrated into the pipeline to survive subsequent automated changes.
• Developer education — not just tooling — is the most durable investment in reducing cloud misconfiguration risk.

In a relatively short time, the cloud has evolved from a powerful enabler to a source of significant operational risk when poorly governed. The answer is not more security tools bolted onto an existing workflow — it is a deeper integration of security as both a practice and a mindset throughout the development lifecycle.

Instaclustr Adds Managed Elasticsearch to Its Open Source Data Platform

Instaclustr, a provider of fully managed open source data infrastructure, announced the general availability of Instaclustr Managed Elasticsearch on March 30, 2020. The service runs on AWS and is built on the Open Distro for Elasticsearch — making Instaclustr the first managed service provider to offer Open Distro for Elasticsearch outside of AWS’s own native service.

Elasticsearch joins Apache Cassandra, Apache Kafka, and Apache Spark on the Instaclustr Managed Platform, giving enterprises a single destination for deploying, managing, and monitoring the full open source data layer.

Why Elasticsearch Matters for Enterprise Search and Analytics

Elasticsearch is built on Apache Lucene and was first released under the Apache 2.0 open source license roughly a decade before this announcement. It has since become the de-facto distributed search and analytics engine for enterprises, valued for:

• Linear horizontal scalability — clusters can grow incrementally as data and query volumes increase.
• Real-time search and analytics — near-instantaneous indexing and querying across large datasets.
• High availability and fault tolerance — distributed architecture with built-in replication keeps clusters resilient.
• Flexible data ingestion — supports structured, semi-structured, and unstructured data from virtually any source.

Despite these advantages, operating Elasticsearch at production scale is technically demanding. Tuning shard allocation, managing cluster state, handling rolling upgrades, and ensuring security compliance all require significant in-house expertise — resources that many engineering teams would rather direct toward product development.

Open Distro for Elasticsearch: A Fully Open Source Foundation

Instaclustr’s service is based on the Open Distro for Elasticsearch, an AWS-sponsored distribution that adds enterprise security, alerting, SQL query support, and performance analysis tooling on top of the 100% Apache 2.0-licensed Elasticsearch core. By choosing Open Distro rather than Elastic’s own commercial distribution, Instaclustr avoids the “open core” licensing model that can introduce vendor lock-in and unpredictable license costs for customers.

Instaclustr also claimed to be the first provider in the industry to offer enterprise support contracts for the Open Distro for Elasticsearch, including support for customers running self-managed or on-premises Elasticsearch installations.

Key Features of Instaclustr Managed Elasticsearch

The managed service includes the following capabilities at general availability:

• Cluster provisioning and configuration via the Instaclustr Management Console or REST APIs.
• Initial cloud support on AWS, with Microsoft Azure and Google Cloud Platform planned for future availability.
• Easy horizontal scaling to handle growing data volumes and search traffic.
• Flexible billing models — run clusters under Instaclustr’s cloud account at a fixed, infrastructure-inclusive price, or bring your own preferred cloud provider account.
• Private Network Cluster deployments, including the ability to deploy within an existing AWS VPC to eliminate cross-VPC network transfer costs.
• Security — encryption of data in transit and at rest, access control via IP allowlists or security groups, and SOC 2 compliance.
• Automated daily backups and point-in-time restore.
• Assisted zero-downtime migration onto the Instaclustr platform from existing deployments.
• Continual monitoring and alerting for cluster health and performance.
• SLAs up to 99.99% availability for production deployments.
• 24×7 expert support with defined support SLAs.
• Ongoing plugin additions — the feature set was committed to expand post-launch.

What This Means for DevOps and Platform Teams

For DevOps and platform engineering teams evaluating Elasticsearch-as-a-Service in 2020, Instaclustr’s announcement introduced a meaningful alternative to AWS’s own Elasticsearch Service (now Amazon OpenSearch Service). The key differentiators at launch were:

• 100% open source stack with no proprietary add-ons that could create future licensing exposure.
• Multi-cloud roadmap, reducing long-term dependency on a single cloud provider.
• Support for self-managed and on-premises Open Distro installations — useful for hybrid or regulated environments.
• Bring-your-own-account option, which can simplify cost allocation and cloud commitment management for enterprises with existing cloud spend agreements.

“Adding a new technology to the Instaclustr Managed Platform is never a decision we take lightly or make hastily. We have seen first-hand how enterprises are benefiting from Elasticsearch in production environments during an extensive early access program, and are now very excited to launch the general availability of Instaclustr Managed Elasticsearch.”

— Ben Slater, Chief Product Officer, Instaclustr

About Instaclustr

At the time of this announcement, Instaclustr reported managing more than 50 million node hours and 5 PB of data across its open source technology suite, which includes Apache Cassandra®, Apache Kafka®, Apache Spark™, and Elasticsearch. The company is headquartered in Redwood City, California.

Further Reading

For background on Open Distro for Elasticsearch, see the Open Distro for Elasticsearch project page (note: the project has since been superseded by OpenSearch, which AWS launched in 2021 as a fully open source fork following Elastic’s license change).

What Is AWS CloudFront?

AWS CloudFront is Amazon Web Services’ managed Content Delivery Network. It distributes content from edge locations — physically distributed data centres placed close to end users — rather than serving every request from a single origin server. When a user requests a file, CloudFront routes the request to the nearest edge location. If the content is already cached there, it is served immediately. If not, CloudFront fetches it from the configured origin (such as an Amazon S3 bucket or an EC2 instance), caches it at the edge, and serves it to the user.

CloudFront entered beta in 2008 and reached general availability in 2009. It operates across a global network of edge locations spanning multiple continents, including coverage across Europe, Asia-Pacific, South America, North America, and the Middle East.

What Can CloudFront Deliver?

CloudFront is not limited to static files. It is commonly used to deliver:

• Video and audio streams — both live and video-on-demand (VOD)
• Static web assets — JavaScript bundles, CSS, images, fonts
• Dynamic web applications — API responses, personalised HTML
• Large binary objects — software packages, datasets, model weights
• Business dashboards and SaaS applications — rich-media collaboration tools, analytics UIs

This flexibility makes CloudFront suitable for anything from a startup distributing marketing videos to a research institution sharing large scientific datasets with global partners.

How CloudFront Integrates with the AWS Ecosystem

One of CloudFront’s practical advantages for teams already on AWS is native integration with the broader service portfolio. Key integrations include:

• Amazon S3 — The most common origin type. S3 buckets feed static assets directly into the CloudFront distribution.
• Amazon EC2 / Elastic Load Balancing — Dynamic application servers act as custom origins, allowing CloudFront to front full web applications.
• AWS Shield Standard — Included at no additional cost with CloudFront. Provides automatic protection against common network-layer and transport-layer DDoS attacks.
• AWS WAF (Web Application Firewall) — Can be attached to a CloudFront distribution to filter malicious HTTP/S traffic at the edge before it reaches your origin.
• Lambda@Edge — Allows you to run Node.js or Python functions at CloudFront edge locations in response to viewer or origin request/response events. This enables use cases such as A/B testing, header manipulation, authentication checks, and URL rewrites without a round-trip to your origin.
• AWS Certificate Manager (ACM) — Provision and attach free TLS/SSL certificates to CloudFront distributions for HTTPS delivery.

All of this is managed through the same AWS Management Console, IAM policies, and CloudFormation or Terraform templates your team already uses, meaning there is no separate tooling to learn or maintain.

Real-World Usage: Who Uses CloudFront?

Several well-known services rely on CloudFront to meet demanding performance requirements:

• Amazon Prime Video — Amazon uses its own CloudFront infrastructure to deliver Prime Video streams globally, demonstrating the scale the service can support.
• Hulu — Uses CloudFront for video-on-demand delivery, ensuring movies and series start quickly and play without interruption regardless of viewer location.
• Slack — Although primarily known as a messaging platform, Slack transfers significant volumes of rich media — videos, images, and large business documents — and relies on CloudFront for consistent delivery performance.
• Major League Baseball and PBS — Both are cited as CloudFront customers, using it to serve video and web content to large, geographically distributed audiences.

Key Benefits of AWS CloudFront

Reduced Latency Through Edge Caching

By caching content at edge locations close to users, CloudFront reduces the physical and network distance a request must travel. This directly reduces time-to-first-byte (TTFB) and improves perceived load times for end users, particularly for large media files.

Origin Offload

Because CloudFront caches popular content at the edge, a significantly smaller proportion of requests hit your origin servers. This reduces compute costs, database pressure, and the risk of origin overload during traffic spikes.

Security at the Edge

CloudFront provides multiple layers of security without requiring separate infrastructure:

• HTTPS enforcement — Redirect HTTP to HTTPS or require HTTPS for all viewer connections.
• TLS termination at the edge — Encryption is terminated at the nearest edge location, and CloudFront can re-encrypt traffic to the origin.
• AWS Shield Standard — Automatic DDoS mitigation included at no extra cost.
• Geo-restriction — Block or allow content delivery to specific countries.
• Signed URLs and signed cookies — Restrict access to private content so only authorised users can retrieve it.

Operational Simplicity

CloudFront requires no physical infrastructure to provision or manage. A distribution can be created in minutes via the AWS Console, AWS CLI, or infrastructure-as-code tools. Capacity scales automatically — you do not need to pre-provision servers or predict peak traffic volumes.

Cost Model

CloudFront uses a pay-as-you-go pricing model based on the volume of data transferred out and the number of HTTP/S requests served. There are no minimum commitments for standard usage, which is particularly advantageous for startups and teams with variable traffic patterns. AWS also offers a CloudFront free tier for new accounts.

Getting Started with CloudFront

The basic steps to set up a CloudFront distribution are:

1. Choose an origin — An S3 bucket, an Application Load Balancer, an EC2 instance, or any publicly accessible HTTP server.
2. Create a distribution — In the AWS Console, navigate to CloudFront and create a new distribution. Specify your origin domain, cache behaviours, and default TTL values.
3. Configure HTTPS — Attach an ACM certificate or use the default CloudFront certificate.
4. Point your DNS — Use a CNAME record to map your custom domain (e.g., assets.example.com) to the CloudFront distribution domain (e.g., d1234abcd.cloudfront.net).
5. Validate and test — Use curl -I or browser developer tools to confirm the x-cache: Hit from cloudfront response header appears after the first request, indicating edge caching is working.

When Should You Use CloudFront?

CloudFront is a strong fit when:

• Your users are geographically distributed and latency to a single-region origin is measurable.
• You serve large media files (video, audio, software downloads) where caching dramatically reduces bandwidth costs.
• You need DDoS protection and TLS termination without operating separate infrastructure.
• You want to run edge logic (authentication, redirects, personalisation) close to users via Lambda@Edge or CloudFront Functions.
• Your application is already hosted on AWS and you want a CDN that integrates natively with S3, EC2, and your existing IAM policies.

If your user base is concentrated in a single region and your origin can handle the load, the cost and complexity of adding a CDN may not be justified. For most public-facing applications serving users in more than one geographic region, however, CloudFront provides a net improvement in both performance and security with relatively low operational overhead.

Cost optimisation is the discipline of achieving the required output at the lowest viable spend, using the right resources in the right configuration for each workload. In practice, it covers four broad areas:

• Cost-effective resources — selecting instance types and services that match workload requirements without over-provisioning.
• Matching supply with demand — scaling capacity up or down in response to actual load.
• Expenditure awareness — understanding where money is being spent and by whom.
• Optimising over time — continuously reviewing and improving cost posture as requirements change.

The eight techniques below map directly to these areas and are applicable to most AWS environments. They are not listed in order of priority — the right sequence depends on your specific workload and team maturity.

1. Attribute Provisioning

When you provision resources on AWS, it is easy to request more capacity than a workload actually needs. Attribute provisioning means continuously tuning the properties of your AWS managed services — compute, memory, storage, network throughput — so that they reflect current demand rather than a worst-case estimate made at launch time.

AWS exposes service attributes through the AWS Management Console, the AWS CLI, and the AWS SDKs. You can also pack multiple workloads onto a single resource to increase density and reduce idle capacity. Amazon CloudWatch is the primary tool for collecting and tracking the metrics you need to make informed provisioning decisions. Setting CloudWatch alarms on CPU, memory, and I/O utilisation gives you the data needed to right-provision resources proactively.

2. Right-Sizing

Right-sizing is the process of selecting the lowest-cost instance type or resource configuration that still meets the technical requirements of a workload. It is one of the highest-impact cost optimisation activities because many teams provision instances based on peak load projections or legacy assumptions rather than observed utilisation.

When conducting a right-sizing exercise, keep these principles in mind:

• Monitor end-user experience, not just infrastructure metrics. A downsized instance that causes latency regressions is not a successful optimisation.
• Weigh the cost of change against the savings. Some resizes require downtime or application restarts; factor that into your analysis.
• Choose an appropriate analysis window. Cover at least one full business cycle — daily, weekly, or monthly — to avoid optimising for an unrepresentative slice of traffic.

AWS services that support resizing include Amazon EC2 (compute), Amazon EBS (block storage volume types and sizes), and Amazon RDS (database instance classes). AWS also provides AWS Compute Optimizer, which analyses CloudWatch metrics and recommends right-sized instance types.

3. On-Demand Instances

On-demand instances let you pay for compute capacity by the hour or second with no long-term commitment. This flexibility makes them the right choice for:

• Short-term or unpredictable workloads where reserving capacity in advance would be wasteful.
• Development and testing environments where usage patterns are irregular.
• Applications being profiled before committing to a purchasing model.

On-demand is generally more expensive per unit of compute than Reserved or Spot instances, so it should be reserved for workloads that genuinely benefit from its flexibility. Once usage patterns become predictable, migrate those workloads to a more cost-effective purchasing option.

4. Spot Instances

AWS Spot instances provide access to spare EC2 compute capacity at a significantly reduced price compared to on-demand rates. The trade-off is that AWS can reclaim Spot instances with a two-minute warning when capacity is needed elsewhere, so workloads must be designed to tolerate interruption.

Spot instances are well suited to:

• Batch processing pipelines
• Scientific simulations and research workloads
• Image and video processing
• Financial modelling and analytics
• CI/CD test runners
• Big data processing with frameworks like Apache Spark or Hadoop

When using Spot instances, design for continuity: checkpoint progress regularly, use Spot Instance interruption notices to drain work gracefully, and diversify across multiple instance types and Availability Zones to reduce the probability of simultaneous reclamation. Mixing Spot and On-Demand capacity in an Auto Scaling group (using EC2 Fleet or Auto Scaling mixed instances policies) is a common production pattern.

5. Reserved Instances and Savings Plans

For predictable, steady-state workloads, committing to a one-year or three-year term in exchange for a discounted hourly rate is one of the most effective ways to reduce AWS spend. AWS offers two primary commitment models:

Reserved Instances (RIs)

• Standard Reserved Instances — provide a billing discount and optionally a capacity reservation when you specify an instance family, size, and Availability Zone. They offer the deepest discounts but the least flexibility.
• Convertible Reserved Instances — available on one-year or three-year terms; allow you to exchange the reservation for a different instance family, size, operating system, or tenancy during the term. Discounts are somewhat lower than Standard RIs but the flexibility is valuable as workloads evolve.

Savings Plans

AWS Savings Plans (Compute Savings Plans and EC2 Instance Savings Plans) offer a more flexible commitment model: you commit to a consistent amount of compute usage measured in dollars per hour, and the discount applies automatically across eligible services regardless of instance type, region, or operating system. For many teams, Savings Plans are simpler to manage than a portfolio of Reserved Instances.

6. Monitoring and Managed Services

Running and managing your own infrastructure on EC2 carries a hidden cost: the engineering time required to patch, scale, monitor, and maintain those servers. AWS managed services absorb much of that operational burden and often deliver lower cost per transaction at scale.

Examples include:

• Amazon RDS and Amazon Aurora — managed relational databases that handle patching, backups, and multi-AZ failover.
• Amazon DynamoDB — a fully managed NoSQL database with on-demand or provisioned capacity modes.
• AWS Lambda — serverless compute that charges only for the duration and memory of function invocations, eliminating idle compute costs.
• Amazon SQS, Amazon SNS, and Amazon SES — managed messaging and notification services that replace self-managed queues and email infrastructure.

Pair managed services with AWS Cost Explorer and AWS Budgets to track spending by service, team, or environment and receive alerts when costs exceed defined thresholds.

7. Optimise Data Transfer Costs

Data transfer is a frequently overlooked contributor to AWS costs. Charges apply for data leaving AWS to the internet, for cross-region transfers, and in some cases for cross-Availability-Zone traffic within a region. Strategies to minimise these costs include:

• Use Amazon CloudFront as a content delivery network (CDN) to cache and serve static and dynamic content close to end users. CloudFront’s data transfer pricing is lower than direct EC2-to-internet egress, and it reduces origin load.
• Colocate tightly coupled services in the same Availability Zone where possible to avoid cross-AZ data transfer charges.
• Use VPC endpoints for services such as S3 and DynamoDB to route traffic over the AWS backbone rather than over the public internet, which can eliminate some transfer charges.
• Compress data before transfer and use efficient serialisation formats to reduce the volume of data moving between services.

8. Turn Off or Schedule Unused Resources

One of the simplest and most impactful cost optimisation actions is stopping resources when they are not in use. Development, testing, and staging environments rarely need to run 24 hours a day, 7 days a week — yet many teams leave them running continuously.

Practical approaches include:

• Tag every resource with its environment (production, staging, dev) and owner so idle resources are easy to identify.
• Use AWS Instance Scheduler or Lambda-based automation to stop and start EC2 and RDS instances on a defined schedule — for example, running dev instances only during business hours on weekdays.
• Set idle timeouts for development databases and remove unused EBS volumes, unattached Elastic IPs, and idle load balancers — all of which accrue charges even when nothing is using them.
• Review and delete unused snapshots on a regular cadence; old EBS and RDS snapshots can accumulate significant storage costs over time.

Scheduling non-production resources to run only when needed can meaningfully reduce compute costs for those environments. The exact savings will depend on current utilisation patterns.

Bringing It All Together

AWS cost optimisation is not a one-time project — it is an ongoing engineering practice. Start by gaining visibility into where you are currently spending (Cost Explorer, AWS Cost and Usage Reports), then apply the techniques above in order of impact for your specific environment. Establish a regular review cadence — monthly at minimum — to catch cost regressions early and identify new optimisation opportunities as your architecture evolves.

Used together, these eight strategies can substantially reduce AWS spend without requiring a redesign of your core architecture. The biggest wins typically come from a combination of right-sizing, commitment-based pricing for stable workloads, and eliminating waste from resources left running unnecessarily.

Amazon EC2 Updates

EC2 Image Builder

EC2 Image Builder automates the creation and ongoing maintenance of OS images for Windows Server and Amazon Linux 2. Before this service, teams had to manually snapshot virtual machines to keep golden images current — a time-consuming and error-prone process. Image Builder handles this automatically for both on-premises and EC2-hosted workloads.

AWS Compute Optimizer

AWS Compute Optimizer applies machine learning to historical resource utilization data and recommends the most appropriate EC2 instance type for each workload. The goal is to replace manual instance sizing guesswork with data-driven recommendations, which can improve both performance and cost efficiency.

Graviton2-Powered Instances (Preview)

AWS previewed the sixth generation of its Arm-based EC2 instances, powered by the in-house Graviton2 processor. These instances are designed to offer improved performance and security compared to the fifth-generation M5, C5, and R5 families. The first instance type made available for preview was the M6g, which supports up to 64 vCPUs and 512 GiB of memory.

AWS Wavelength (Preview)

AWS Wavelength embeds AWS compute and storage capabilities directly within telecommunications providers’ 5G networks. The result is ultra-low latency for mobile and edge applications — relevant use cases include augmented reality, autonomous vehicles, and real-time analytics at the edge.

To use Wavelength, developers extend an existing VPC to include a Wavelength Zone, then provision standard AWS resources such as EC2 instances and CloudFormation stacks within that zone. At launch, Wavelength was announced in partnership with Verizon, with Vodafone, SK Telecom, and KDDI planned to follow in 2020.

Serverless Updates

Provisioned Concurrency for AWS Lambda

Cold starts are one of the most common complaints about AWS Lambda in latency-sensitive applications. Provisioned Concurrency addresses this by keeping a configurable number of Lambda execution environments initialized and ready to respond immediately. At launch, AWS priced this feature at $0.015 per GB-hour for provisioned concurrency and $0.035 per GB-hour for actual execution time.

EKS on AWS Fargate

Amazon Elastic Kubernetes Service (EKS) gained the ability to run Kubernetes pods on AWS Fargate, Amazon’s serverless compute engine for containers. Originally teased at re:Invent 2017, the capability finally shipped two years later. With this integration, teams can run Kubernetes workloads without provisioning or managing EC2 node groups — scheduling and infrastructure management are handled entirely through the EKS console.

Fargate Spot

Fargate Spot is the serverless equivalent of EC2 Spot Instances. It offers spare Fargate compute capacity at a discounted price for Amazon ECS tasks. Like EC2 Spot, Fargate Spot tasks can be interrupted when AWS reclaims that capacity, so the feature is best suited for fault-tolerant, interruptible workloads rather than mission-critical jobs.

Security Updates

AWS Nitro Enclaves (Preview)

Nitro Enclaves are isolated virtual machine environments that run attached to an EC2 instance. By isolating sensitive processing from the parent instance’s operating system and other software, enclaves reduce the attack surface for workloads that handle highly sensitive data — such as cryptographic key processing or personally identifiable information. The feature was in preview at the time of the announcement.

S3 Access Points

Managing access to shared S3 buckets through a single bucket policy becomes unwieldy as the number of consumers and use cases grows. S3 Access Points solve this by giving each application or team its own named endpoint — with a unique hostname and a dedicated IAM policy — that governs how that specific consumer can interact with data in the bucket. This makes per-application access control significantly easier to reason about and audit.

VPC Ingress Routing

VPC Ingress Routing lets teams route all inbound and outbound VPC traffic through a dedicated EC2 instance before it reaches production workloads. Security appliances, intrusion detection systems, or packet inspection tools running on that instance can inspect traffic in-line — bringing on-premises network security policy enforcement models into the AWS environment without significant architecture changes.

Developer Tools

Amazon CodeGuru (Preview)

Amazon CodeGuru is a managed service that uses machine learning to provide two distinct capabilities for improving code quality and application efficiency:

• CodeGuru Reviewer — Performs automated pull request reviews before code reaches production. It can identify potential problems such as unsanitized inputs, improper handling of sensitive data, and deviations from AWS best practices.
• CodeGuru Profiler — A lightweight runtime agent that continuously analyzes a running application, identifies CPU-intensive code paths and inefficient patterns, and surfaces actionable recommendations to improve performance and reduce compute costs.

At launch, CodeGuru supported integration with GitHub and AWS CodeCommit repositories. The service was available in preview at re:Invent 2019.

Summary

AWS re:Invent 2019 was dense with announcements. While Outposts and Braket earned most of the press coverage, the practical day-to-day impact for many engineering teams will come from services like Compute Optimizer, EC2 Image Builder, Provisioned Concurrency, and S3 Access Points. Review each capability against your current architecture to identify where they can reduce operational overhead, improve security posture, or lower your AWS bill.

Original reporting source: searchaws.techtarget.com

What Is AWS Elastic Beanstalk?

AWS Elastic Beanstalk is a Platform-as-a-Service (PaaS) offering from Amazon Web Services. You supply your application code and a few configuration choices; Elastic Beanstalk handles the rest — provisioning EC2 instances, configuring an Elastic Load Balancer, setting up autoscaling groups, deploying the application stack, and monitoring health. The underlying AWS resources remain visible and accessible, but you are not required to manage them directly.

In practice, Elastic Beanstalk sits between raw infrastructure services (like EC2 and S3) and a fully managed serverless environment. You retain full access to the resources it creates, which means experienced teams can still fine-tune the configuration while less infrastructure-focused teams can rely entirely on the managed defaults.

What Does Elastic Beanstalk Manage For You?

• Compute provisioning — Launches and configures Amazon EC2 instances appropriate for your application type.
• Load balancing — Sets up an Elastic Load Balancer to distribute incoming traffic across instances.
• Autoscaling — Adds or removes instances automatically based on demand, using configurable scaling triggers.
• Application deployment — Handles rolling deployments, blue/green deployments, and version management.
• Health monitoring — Continuously monitors instance and application health and replaces unhealthy instances.
• Logging — Aggregates application and server logs and makes them available through the console or CLI.
• Platform patching — Provides managed platform versions that receive OS and runtime updates.

Supported Languages and Platforms

Elastic Beanstalk supports a broad range of runtimes and web servers out of the box. Supported language platforms include:

• Java (with Apache Tomcat)
• Python
• Ruby
• PHP
• Node.js
• .NET on Windows Server
• Go
• Docker (single-container and multi-container)

Each platform is called a platform version and is maintained by AWS. You choose a platform when you create an environment and can upgrade to newer platform versions over time. Docker support is particularly useful when your stack does not match a natively supported runtime.

Core Concepts

Application

An application in Elastic Beanstalk is a logical container for your project. It holds all of the environments and application versions associated with that project.

Application Version

An application version is a specific, labeled iteration of your deployable code — typically a ZIP or WAR archive stored in Amazon S3. You can maintain multiple versions and deploy any of them to any environment at any time.

Environment

An environment is a running instance of an application version. Common patterns are to run separate environments for development, staging, and production. Each environment provisions its own set of AWS resources.

Environment Tier

Elastic Beanstalk supports two environment tiers:

• Web server tier — For HTTP/HTTPS applications. Includes a load balancer and autoscaling group.
• Worker tier — For background processing tasks. Pulls messages from an Amazon SQS queue and processes them without a public-facing load balancer.

How to Deploy an Application

There are three main ways to interact with Elastic Beanstalk:

1. AWS Management Console — A web-based dashboard where you can create environments, upload application versions, configure scaling rules, and view health metrics with no command-line knowledge required.
2. EB CLI — The Elastic Beanstalk Command Line Interface (eb) integrates with your local development workflow. Commands like eb init, eb create, and eb deploy let you manage environments from a terminal.
3. AWS SDKs and APIs — For programmatic and CI/CD pipeline integration, the Elastic Beanstalk API can be called directly from any language with an AWS SDK, or through tools like AWS CloudFormation and AWS CodePipeline.

Integration with Other AWS Services

Elastic Beanstalk is designed to work alongside the broader AWS ecosystem. Common integrations include:

• Amazon EC2 — Compute instances that run your application.
• Amazon S3 — Stores application version archives and log files.
• Amazon RDS — Managed relational databases that can be linked to an Elastic Beanstalk environment (though decoupling RDS from the environment lifecycle is a recommended production practice).
• Amazon CloudWatch — Metrics and alarms that drive autoscaling decisions and environment health monitoring.
• AWS IAM — Controls permissions for the service roles Elastic Beanstalk uses to manage resources on your behalf.
• Amazon VPC — Environments can be launched inside a Virtual Private Cloud for network isolation.

Key Benefits

Reduced Operational Overhead

Development teams without dedicated infrastructure engineers can deploy production-grade, scalable applications without becoming AWS architecture experts. Elastic Beanstalk handles the operational defaults while still exposing every underlying resource for teams that want deeper control.

Automatic Scaling

Autoscaling rules can be based on metrics such as CPU utilization or request count. When traffic spikes, new instances are added automatically; when demand drops, instances are terminated to control cost. Both the minimum and maximum instance counts are configurable.

No Additional Charge for the Service Itself

AWS does not charge extra for using Elastic Beanstalk. You pay only for the underlying AWS resources — EC2 instances, load balancers, data transfer, and so on — that your environments consume.

Flexibility and Escape Hatches

Unlike fully managed PaaS offerings that abstract away all infrastructure details, Elastic Beanstalk lets you SSH into instances, modify configuration files using .ebextensions, and replace managed components (such as the default nginx proxy) when you need to. This prevents lock-in to opinionated defaults.

When to Use Elastic Beanstalk — and When Not To

Elastic Beanstalk is a strong fit when:

• Your team wants to deploy quickly without building custom infrastructure automation.
• You are running a conventional web application or API in a supported runtime.
• You want managed autoscaling and health monitoring with minimal configuration.

It may not be the best fit when:

• You need fine-grained control over every aspect of the infrastructure (consider AWS CDK, CloudFormation, or Terraform instead).
• You are adopting a microservices architecture that maps more naturally onto Amazon ECS, Amazon EKS, or AWS Lambda.
• Your application has highly unusual runtime requirements that fall outside the supported platform list and cannot be containerized.

A Practical Example: Deploying a Python Flask App

The following walkthrough uses the EB CLI. It assumes you have already installed the CLI and configured AWS credentials.

# 1. Initialize a new Elastic Beanstalk application in your project directory
eb init my-flask-app --platform python-3.11 --region us-east-1

# 2. Create a new environment and deploy your code
eb create my-flask-env

# 3. Open the deployed application in a browser
eb open

# 4. Deploy an updated version of your code
eb deploy

# 5. View environment health and recent events
eb health
eb events

Elastic Beanstalk reads a Procfile or platform-specific configuration to determine how to start your application. For Python applications, a requirements.txt file in the project root is used to install dependencies automatically.

Summary

AWS Elastic Beanstalk bridges the gap between writing application code and running it reliably at scale. By automating provisioning, load balancing, autoscaling, health monitoring, and deployment, it frees engineering teams to concentrate on the product rather than the platform. It is not a replacement for container orchestration or Infrastructure-as-Code in complex architectures, but for teams that need a fast, manageable path from code to production on AWS, it remains a practical and cost-effective choice.

The Cloud Giant and the Open-Source Dilemma

Amazon Web Services (AWS) is the dominant force in cloud computing — by most estimates, roughly triple the size of its nearest competitor, Microsoft Azure, in terms of infrastructure market share at the time of this writing. Millions of people interact with AWS daily without knowing it: Netflix video streams, Apple iCloud storage, and countless enterprise applications all run on Amazon’s infrastructure.

That dominance has come with a specific and controversial playbook: identify popular open-source software, offer it as a fully managed AWS service, and leverage Amazon’s existing customer base, billing infrastructure, and sales teams to out-compete the company that built the software in the first place.

The Elasticsearch Case: A Defining Example

Elastic, a software company founded in Amsterdam, built Elasticsearch — a widely used open-source tool for searching and analysing large datasets. By 2015, Elastic had grown to around 100 employees and was already distributing its software through AWS’s marketplace.

In October 2015, Amazon announced its own managed Elasticsearch service — built directly on Elastic’s open-source code, which Amazon was legally permitted to use under its open-source licence. Within a year, Amazon was reportedly generating more revenue from Elastic’s software than Elastic itself, primarily because AWS made the tool trivially easy to use alongside its other services.

Elastic’s response evolved in two stages:

• Premium feature restrictions: Elastic began adding proprietary, paid features to its product and restricted how companies like Amazon could use them.
• Legal action: In September 2019, Elastic sued Amazon in federal court in California, alleging trademark infringement because Amazon had named its service “Amazon Elasticsearch Service” — using the exact trademark Elastic held. Amazon denied wrongdoing. The case was pending at the time of publication.

Amazon’s response to Elastic’s premium feature restrictions was to duplicate many of those features independently and offer them free of charge to AWS customers.

A Repeating Pattern: Strip-Mining Open Source

The Elasticsearch dispute is not an isolated incident. Industry observers and affected companies describe a consistent pattern that some have termed strip-mining software:

1. A startup builds and open-sources a useful software tool, often growing a community and a business around commercial support or premium features.
2. AWS packages the open-source version as a managed service, handling infrastructure, scaling, and operations for customers.
3. Because AWS customers can add the service with a single click — on the same bill, in the same console — they often choose the AWS-managed version over the startup’s own offering.
4. The startup loses revenue traction while Amazon profits from work it did not originate.

AWS’s template for this approach was established as early as 2009, when it launched a managed database service built on open-source database software rather than anything Amazon had created in-house. That service was an immediate commercial success.

MongoDB’s Licensing Counter-Move

MongoDB, a popular document-oriented database, attempted a structural defence. In 2018, MongoDB announced a new licence — the Server Side Public License (SSPL) — that required any company offering MongoDB as a managed cloud service to also open-source the entire stack used to provide that service. This was widely interpreted as a direct response to AWS’s practices, since AWS does not open-source its own service-delivery infrastructure.

AWS responded by releasing its own compatible database service — DocumentDB — built to replicate MongoDB’s API using an older version of MongoDB’s software that pre-dated the new licensing terms, meaning the SSPL did not apply.

The episode illustrated a core problem: by the time a startup changes its licence, Amazon can often pivot to an older, licence-compatible version of the code, or build a compatible alternative independently.

Why Startups Still Work with AWS Anyway

Despite the competitive threat, smaller software companies often feel they have no realistic alternative to partnering with AWS. The reasons are structural:

• Customer access: AWS’s sales force has relationships with millions of enterprises. Selling through the AWS Marketplace gives startups distribution they could not otherwise afford.
• Ecosystem lock-in: Customers who are already running workloads on AWS prefer solutions that integrate natively into the same console and billing system.
• Investor perception: Being available on AWS is often seen as a mark of legitimacy for early-stage startups.

The trade-offs, however, are significant. Startups selling through AWS Marketplace pay Amazon a percentage of their revenue. They often agree to restrictions on how they can market their products independently. Some also share customer and product data with Amazon as part of these arrangements — data that, critics argue, gives Amazon valuable intelligence about market demand.

The Convenience Moat: How AWS Buries Competitors

Beyond open-source adoption, AWS compounds its advantage through user experience design. Between 2014 and late 2019, AWS grew from roughly 30 services to approximately 175. Each new service is integrated into the same management console, the same IAM permissions model, and the same monthly bill.

Adding a native AWS service is a single click. Integrating a third-party competitor’s tool running on AWS requires additional configuration, separate billing, and more complex identity and networking setup. That friction disadvantages even technically superior third-party products.

AWS has also been accused of bundling discounts — making its own services cheaper when customers commit to a broader AWS spend — that make competing on price alone extremely difficult for independent software vendors.

Antitrust Scrutiny and Regulatory Interest

By late 2019, AWS’s practices had attracted attention from US regulators and lawmakers:

• The House Judiciary Committee, as part of its broader investigation into large technology platforms, sent Amazon a letter in September 2019 specifically asking about AWS’s competitive practices.
• The Federal Trade Commission (FTC), which was also investigating Amazon more broadly, was reported to have contacted AWS competitors as part of its inquiry.

Several affected companies were reportedly discussing whether to file formal antitrust complaints. The core allegation is that AWS uses its infrastructure monopoly — its control of the underlying computing layer — to distort competition in the software markets that run on top of that infrastructure.

AWS’s Business Context

It is worth noting the financial stakes. AWS generated approximately $25 billion in revenue in 2018 and was Amazon’s most profitable division by a significant margin, effectively subsidising Amazon’s lower-margin retail operations. Jeff Bezos described AWS as an idea “no one asked for” — it originated from Amazon’s internal need to standardise its own computing infrastructure in the early 2000s, after which Amazon recognised the same need existed across the broader market.

Companies such as Airbnb, Netflix, and General Electric became major AWS customers, renting computing capacity rather than running their own data centres. This scale gives AWS a cost and reliability advantage that is extremely difficult for any single startup to replicate.

What Open-Source Maintainers and Startups Can Do

The situation has prompted the open-source community to rethink licensing strategy. Options that have been explored include:

• Copyleft-style cloud licences (such as the SSPL) that require cloud providers to open-source their entire service stack — though these are contested and may affect OSI open-source classification.
• Open-core models that keep the most valuable enterprise features proprietary, while open-sourcing the core — though Amazon has demonstrated willingness to re-implement proprietary features independently.
• Avoiding open-source entirely for commercially sensitive components, as Todd Persen (quoted in original reporting) chose to do when founding a new company specifically to eliminate the risk of AWS adopting his work.
• Building deeper integrations with AWS that create customer switching costs — though this deepens reliance on the platform.

None of these approaches is a complete solution. Each involves trade-offs between community growth, commercial protection, and platform reach.

Key Takeaways for DevOps and Platform Engineers

• When evaluating managed AWS services built on open-source projects, be aware that the upstream project maintainer may receive no revenue from your usage.
• If you are building infrastructure tooling, your open-source licence choice has direct commercial consequences — particularly with respect to cloud provider adoption.
• The regulatory environment around cloud platform dominance is evolving; antitrust actions in this space could affect service availability, pricing, and API compatibility.
• Distribution through AWS Marketplace comes with real revenue-share and data-sharing costs that are not always prominently disclosed.

The broader debate about whether AWS’s practices constitute anticompetitive behaviour under US law remains unresolved. What is clear is that the open-source model — which powered much of the modern cloud ecosystem — is under structural pressure from the very platforms that benefited most from it.

AWS Opens Cloud Credits to Open Source Projects

AWS announced a promotional cloud credits program aimed at open source software (OSS) project maintainers, offering them AWS credits to use for infrastructure tasks directly related to their projects. The move was framed as part of a broader effort by AWS to deepen its engagement with the open source community alongside sponsorships, increased code contributions, and the open sourcing of some of its own internal technology.

According to an AWS blog post at the time, typical use cases for the credits include upstream and performance testing, CI/CD pipelines, and storage of build artifacts on AWS infrastructure. The stated goal was to relieve resource pressure on OSS projects so their maintainers could focus on expanding and innovating within their communities.

Who Qualifies and How the Program Works

To be eligible for AWS open source credits, project maintainers must meet the following criteria:

• Active AWS account with no outstanding balance.
• OSI-approved license: The project must be released under a license recognized by the Open Source Initiative (OSI), such as the GNU GPL or Apache License 2.0. AWS indicated exceptions could be made for projects using non-OSI licenses on a case-by-case basis.
• Specific credit request: Maintainers must request a defined amount of credits and provide a clear description of intended use.
• One-year validity: Credits are valid for twelve months from the time they are issued.

AWS stated it would select recipients partly based on alignment with Amazon’s leadership principles. Projects with maintainers from more than one organization, or those owned by nonprofits, were given preference in the selection process.

Representatives from AdoptOpenJDK and the Cloud Native Computing Foundation (CNCF), among others, publicly endorsed the program and stated they were already benefiting from it at the time of the announcement.

Context: AWS and the Open Source Community

The announcement came against a backdrop of growing friction between AWS and certain segments of the open source ecosystem. AWS had faced criticism—particularly from commercial open source vendors—for consuming and building managed services on top of popular OSS projects without proportional contributions back to those projects.

The tension had already escalated to legal action in at least one case. Elastic filed a trademark infringement lawsuit against AWS following AWS’s release of Open Distro for Elasticsearch. AWS argued the fork was necessary after Elastic began mixing proprietary code into what had been a fully open source project. That dispute would later lead to much broader licensing changes at Elastic.

Industry Reaction: Table Stakes or Genuine Progress?

Analysts and practitioners had divergent takes on the significance of the credits program.

“It’s actually pretty consistent with industry standard open source engagement to provide some level of free or heavily discounted services, so while it’s certainly an opportunity for AWS to try to build some goodwill, it is at the same time something like table stakes.”

— Stephen O’Grady, Analyst, RedMonk

Others were more skeptical about the timing of the initiative.

“They could have made this a program years ago. Open source groups have been reaching out to cloud vendors to donate resources for a long time. I think in the past it was a more human process. They’re making the process public for an obvious reason. … Is the timing suspect? You tell me.”

— Ryan Marsh, Serverless expert and DevOps trainer, TheStack.io

Unanswered Questions

At the time of the original reporting, several practical details about the program were not publicly addressed by AWS:

• The total budget AWS planned to commit to the credits program.
• The minimum or maximum credit amount any single project could receive.
• What would happen to credits if a project’s maintainer decided to commercialize the project after receiving credits.

These gaps left observers uncertain about the scale and long-term intent of the program.

How AWS Compared to Google at the Time

Even with the credits program, AWS had not matched the depth of Google’s commercial OSS engagement at that point. At Google Cloud Next in April 2019, Google announced expanded partnerships with seven commercial open source vendors—including MongoDB, Redis Labs, and DataStax—to create tightly integrated, fully managed services on Google Cloud Platform. Those deals included shared support and unified billing, going significantly further than credits alone.

The AWS credits program nonetheless represented a formal, structured acknowledgment of its obligations to the OSS ecosystem—a notable shift in posture even if industry veterans considered it overdue.

Key Takeaways for DevOps Teams and OSS Maintainers

• If you maintain an OSI-licensed open source project, check whether an AWS credits program is currently active and whether your project qualifies.
• Use cases like CI/CD, artifact storage, and performance testing are explicitly supported use cases for cloud credits across multiple providers—not just AWS.
• When evaluating cloud vendor support for OSS, look beyond credits to assess code contributions, governance participation, and partnership terms.
• The broader AWS-vs-OSS tension illustrated by the Elastic lawsuit has since reshaped open source licensing strategies across the industry, making license choice a strategic decision for project maintainers.

What Is Semantic Code Search?

Semantic code search is the task of retrieving relevant code snippets given a natural language query. Unlike traditional keyword search, it must bridge the gap between how developers describe intent in plain English and how that logic is expressed in source code. Standard information retrieval techniques are largely ineffective here because the vocabulary used in search queries and the vocabulary found in code rarely overlap. Compounding this, there was no substantial annotated dataset specifically designed for evaluating code search quality — a gap this project directly addresses.

The CodeSearchNet Corpus

The CodeSearchNet Corpus contains approximately 6 million functions extracted from open-source code across six programming languages:

• Go
• Java
• Python
• JavaScript
• PHP
• Ruby

A subset of approximately 2 million functions includes automatically generated query-like natural language descriptions derived from associated documentation. This pairing of code with natural language makes the corpus suitable for training and evaluating semantic search models. Each record also includes metadata indicating the original source location of the code.

How the Corpus Was Collected

The team sourced code exclusively from publicly available, non-forked GitHub repositories. To ensure quality, they used libraries.io to identify projects that are depended upon by at least one other project — a proxy for real-world relevance. Projects were then ranked by popularity using star and fork counts. Any repository without a permissive license that allowed redistribution of its contents was excluded.

For parsing and tokenizing functions across all six languages, the team used Tree-sitter, an incremental parser generator library capable of handling multiple programming languages. Only functions that had associated documentation were included in the training split for the CodeSearchNet Challenge, ensuring that natural language–code pairs were available for supervised learning.

The team released their data preprocessing pipeline publicly so others can replicate or extend the dataset collection process, lowering the barrier to applying machine learning techniques to source code at scale.

The CodeSearchNet Challenge

The CodeSearchNet Challenge is a formal evaluation benchmark consisting of 99 natural language queries and approximately 4,000 expert relevance annotations. The goal is to measure how well a model can retrieve relevant code from the CodeSearchNet Corpus for each query.

How the Queries Were Selected

Query selection followed a multi-step curation process:

1. Bing search logs: The team started with common code-related search queries that had high click-through rates on Bing.
2. StaQC dataset: These were supplemented with queries drawn from StaQC, a dataset of Stack Overflow question-code pairs.
3. Manual filtering: Queries that were purely technical keywords (e.g., API names or function signatures) were removed, leaving 99 natural language queries that describe developer intent.

How Relevance Annotations Were Collected

To generate annotation data, the team ran a standard Elasticsearch installation and their baseline models against the corpus, collecting the top 10 results per query. Data scientists, programmers, and machine learning researchers then manually annotated these results for relevance to each query. This human-labeled dataset forms the ground truth used for leaderboard evaluation.

Evaluation Protocol

To participate in the CodeSearchNet Challenge, a model must return a ranked list of results from the CodeSearchNet Corpus for each of the 99 predefined natural language queries. Results are then scored against the expert relevance annotations. The leaderboard, integrated with Weights & Biases, allows researchers to track and compare model performance transparently.

Why This Matters for the Developer Tools Ecosystem

CodeSearchNet fills a well-known gap in the code intelligence research community: the absence of a large, high-quality, multi-language benchmark for evaluating semantic code search. By releasing both the corpus and the evaluation harness, GitHub and Weights & Biases enable reproducible research and direct comparison across approaches — from classical information retrieval baselines to neural embedding models. The Tree-sitter-based preprocessing pipeline also gives practitioners a reusable starting point for building their own code-aware ML systems.

Original reporting via Packt Hub. The CodeSearchNet paper and dataset were published by the GitHub Next research team.

The Core Question: Declarative vs. Imperative Deployment

Before comparing the three services, it helps to understand the underlying deployment philosophy they share. You can provision AWS infrastructure using shell scripts full of AWS CLI commands—this is an imperative model where every step must be explicitly defined, error handling must be hand-coded, and scripts must be rewritten for each new application.

A more maintainable approach is declarative configuration: you describe the desired end state in a text file, and a deployment engine figures out how to reach that state, including error handling. All three services discussed here use this declarative model. The text files that describe your infrastructure can be stored in version control, enabling an infrastructure-as-code (IaC) workflow.

AWS CloudFormation: Deploying AWS Resources

CloudFormation is AWS’s foundational infrastructure-as-code service. You write a CloudFormation template—a JSON or YAML file—that declares which AWS resources your application needs. CloudFormation reads the template and provisions those resources in the correct order, handling dependencies automatically.

CloudFormation works well for:

• Deploying the AWS resource layer of an application (VPCs, subnets, security groups, RDS instances, S3 buckets, load balancers, etc.)
• Applications that are relatively static, where the entire infrastructure stack can be replaced on each update rather than patched in place
• Teams that need repeatable, version-controlled environment definitions

CloudFormation supports change sets, which let you preview proposed modifications to a running stack—such as adding or removing resources—before applying them. This gives operators an audit step before infrastructure changes take effect.

Key limitation: CloudFormation manages AWS resources at the provisioning level. It does not orchestrate configuration changes inside a running EC2 instance. For example, it cannot install new packages, update application configuration files, or restart services on an already-running server. Shell scripts can be embedded in CloudFormation user data, but their configuration capabilities are limited.

AWS OpsWorks: Configuration Management Inside EC2

AWS OpsWorks bridges the gap that CloudFormation leaves around in-instance configuration. OpsWorks provides managed versions of two popular configuration management tools—Chef and Puppet—as AWS services (OpsWorks for Chef Automate and OpsWorks for Puppet Enterprise). A third variant, OpsWorks Stacks, uses Chef cookbooks in a proprietary stack model.

OpsWorks handles:

• Initial deployment of applications onto EC2 instances, including OS-level configuration
• Ongoing configuration drift remediation—keeping instances in a defined state over time
• Application updates applied to running instances without replacing them
• Operating system package management, user accounts, file permissions, and service management

Use OpsWorks instead of CloudFormation alone when your application requires ongoing updates to the configuration of its EC2 instances—for example, rolling out new application versions, patching OS packages, or managing configuration files that change independently of the underlying infrastructure.

Combining CloudFormation and OpsWorks

For applications that use many AWS services and require detailed EC2 configuration management, CloudFormation and OpsWorks complement each other well. The recommended division of responsibility is:

In this model, CloudFormation bootstraps the OpsWorks stack as part of resource provisioning. Once EC2 instances are running, OpsWorks takes over to apply Chef recipes or Puppet manifests. When changes are needed, AWS resource changes go through CloudFormation change sets; application and OS changes go through OpsWorks. This separation of concerns keeps each tool focused on what it does best.

AWS Elastic Beanstalk: Managed Platform for Web Applications

Elastic Beanstalk takes a higher-level approach. Rather than requiring you to define individual AWS resources, Elastic Beanstalk is a platform-as-a-service (PaaS)-style provisioning service. You upload your application code, tell Elastic Beanstalk which platform to use (e.g., Python, Node.js, Java, Docker), and it provisions and configures the underlying infrastructure automatically.

Elastic Beanstalk automatically handles:

• Web server provisioning and configuration
• Auto Scaling group setup
• Load balancer configuration
• Health monitoring and alerting
• Log aggregation
• Rolling deployments and environment cloning

If your application is a self-contained web application or API and fits neatly within a supported Elastic Beanstalk platform, you may not need CloudFormation or OpsWorks at all. Elastic Beanstalk is the fastest path from code to a running, monitored web environment.

When to combine Elastic Beanstalk with CloudFormation: Most real-world applications need resources that Elastic Beanstalk does not manage on its own—a managed database (RDS), object storage (S3), a cache layer (ElastiCache), or a message queue (SQS). In these cases, use CloudFormation to define the full environment: let CloudFormation provision the Elastic Beanstalk environment alongside the supporting AWS resources. This gives you the convenience of Elastic Beanstalk for the application tier plus the full resource coverage of CloudFormation.

Decision Guide: Which Tool to Use

Use the following guidance to select the right tool or combination of tools for your use case:

• CloudFormation only — You need repeatable provisioning of AWS resources for a relatively static application, and you do not need to manage configuration inside EC2 instances beyond simple startup scripts.
• OpsWorks only (or OpsWorks + CloudFormation) — You have an application running on EC2 that requires ongoing configuration management, package updates, or application deployments to live instances. Add CloudFormation when you also have significant AWS resource dependencies.
• Elastic Beanstalk only — You are deploying a web application or API that fits a supported platform runtime and has minimal external AWS dependencies.
• Elastic Beanstalk + CloudFormation — Your web application needs Elastic Beanstalk’s managed platform conveniences but also depends on additional AWS resources such as RDS, S3, or ElastiCache that must be provisioned alongside it.

Summary Comparison

All three services support a declarative approach to infrastructure management and can be stored as code in version control. Choosing between them—or combining them—depends primarily on how much control you need over the underlying EC2 configuration, how complex your AWS resource dependencies are, and how quickly you need to get a web application running.

Full-text search can be both scary and exciting. Some popular databases such as MySql and Postgres are an amazing solution for storing data… but when it comes to full-text search performances, there’s no competition with ElasticSearch.

For those who don’t know, ElasticSearch is a search engine server built on top of Lucene with an amazing distributed-architecture support. According to db-engines.com, it is currently the most used search engine out there.

In this post, we are going to build a simple REST application called The Quotes Database which will allow us to store and search as many quotes as we want.

I’ve prepared a JSON file containing 5000+ quotes with their authors, we’ll be using it as our starting data for populating ElasticSearch.

Setting up Docker

First of all, we don’t want to install ElasticSearch on our machine. We’ll be using Docker for orchestrating both the Node.js server and the ES instance on a container, which will allow us to deploy a production-ready application with all the dependencies that it needs.

Let’s create a Dockerfile inside our project root folder:

FROM node:10.15.3-alpine

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install
RUN npm install -g pm2

COPY . ./

EXPOSE 3000
EXPOSE 9200

CMD npm run start

As you can see, we’re telling Docker that we’ll run a Node.js 10.15.3-alpine runtime. We’ll also create a new working directory under /usr/src/app, where we’ll copy both the package.json and the package-lock.json files. That way, Docker will be able to run npm install inside our WORKDIR, installing our needed dependencies.

We’ll also install PM2 by running RUN npm install -g pm2. The Node.js runtime is single-threaded, so if a process crashes, the entire app needs to be restarted… PM2 checks the Node.js process status and restarts it when the app goes down for any reason.

After installing PM2, we’ll copy our codebase inside our WORKDIR (COPY . ./), and we tell Docker to expose two ports, 3000, which will expose our RESTful service, and 9200, which exposes the ElasticSearch service (EXPOSE 3000 and EXPOSE 9200).

Last but not least, we tell Docker which command will start the Node.js app with npm run start.

Setting up docker-compose

Now you may be saying, ‘Great, I get it! But how do I handle the ElasticSearch instance inside Docker? I can’t find it in my Dockerfile!’… you’re right! That’s where docker-compose becomes useful. It allows us to orchestrate multiple Docker containers and to create a connection between them. So, let’s write down the docker-compose.yml file, which will be stored inside our project root directory:

version: '3.6'
services:
  api:
    image: node:10.15.3-alpine
    container_name: tqd-node
    build: .
    ports:
      - 3000:3000
    environment:
     - NODE_ENV=local
     - ES_HOST=elasticsearch
     - NODE_PORT=3000
     - ELASTIC_URL=http://elasticsearch:9200
    volumes:
      - .:/usr/src/app/quotes
    command: npm run start
    links:
        - elasticsearch
    depends_on:
        - elasticsearch
    networks: 
      - esnet
  elasticsearch:
    container_name: tqd-elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.0.1
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    logging:
      driver: none
    ports:
      - 9300:9300
      - 9200:9200
    networks: 
      - esnet
volumes:
  esdata:
networks:
  esnet:

This is a bit more complex than our Dockerfile, but let’s analyze it:

• We declare which version of docker-compose.yml file we’re using (3.6)
• We declare our services:
  • api, which is our Node.js app. Just like in our Dockerfile, it requires the node:10.15.3-alpine image. We also assign a name to this container tqd-node, and here we call the previously created Dockerfile using the build . command.
  • We need to expose the 3000 port so we write these statements as follows 3000:3000. It means that we’re mapping from the port 3000 (inside our container) to the port 3000 (accessible from our machine). Then we’ll set some environment variables. The value elasticsearch is a variable which refers to the elasticsearch service inside our docker-compose.yml file.
  • We also want to mount a volume /usr/src/app/quotes. That way, once we restart our container we’ll maintain our data without losing it.
  • Once more, we tell Docker which command we need to execute once the container starts up, then we set a link to the elasticsearch service. We also tell Docker to start the api service after the elasticsearch service has booted (using the depends_on directive).
  • Last but not least, we tell Docker to connect the api service under the esnet network. That’s because every container has its own network. That way, we say that api and elasticsearch services shares the same network so they will be able to call each other with the same ports.
  • elasticsearch, which is (as you may have guessed) our ES service. It’s configuration is pretty similar to the api service. We’ll just cut off its verbose logs setting the logging directive to driver: none .
• We also declare our volumes, where we store the ES data.
• And we declare our network, esnet.

Bootstrapping the Node.js app

Now we need to create our Node.js application, so let’s get started setting up our package.json file:

npm init -y

Now we need to install some dependencies:

npm i -s @elastic/elasticsearch body-parser cors dotenv express

Great! Our package.json file should look like this:

{
  "name": "nodejselastic",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@elastic/elasticsearch": "^7.3.0",
    "body-parser": "^1.19.0",
    "cors": "^2.8.5",
    "dotenv": "^8.0.0",
    "express": "^4.17.1"
  }
}

Let’s implement our ElasticSearch connector in Node.js. First, we need to create a new /src/elastic.js file:

const { Client } = require("@elastic/elasticsearch");
                   require("dotenv").config();

const elasticUrl = process.env.ELASTIC_URL || "http://localhost:9200";
const esclient   = new Client({ node: elasticUrl });
const index      = "quotes";
const type       = "quotes";

As you can see, here we set some constants that will be incredibly useful. First of all, we create a new connection to ElasticSearch using its official Node.js SDK, then we define an index ("quotes") and an index type("quotes" again, we’ll see later what they mean).

Now we need to create an index on ElasticSearch. You can think of “indexes” as a SQL “database” equivalent. ElasticSearch is a NoSQL database, which means that it has no tables — it just stores JSON documents. An index is a logical namespace which maps to one or more primary shards and can have zero or more replica shards. You can read more about ElasticSearch indexes here.

Let’s now define a function that will create our index:

async function createIndex(index) { 
  try {
    await esclient.indices.create({ index });
    console.log(`Created index ${index}`);
  } catch (err) {
    console.error(`An error occurred while creating the index ${index}:`);
    console.error(err);
  }
}

Now we need another function that creates the mapping for our quotes. The mapping defines the schema and types of our document:

async function setQuotesMapping () {
  try {
    const schema = {
      quote: {
        type: "text" 
      },
      author: {
        type: "text"
      }
    };

    await esclient.indices.putMapping({ 
      index, 
      type,
      include_type_name: true,
      body: { 
        properties: schema 
      } 
    })

    console.log("Quotes mapping created successfully");
  } catch (err) {
    console.error("An error occurred while setting the quotes mapping:");
    console.error(err);
  }
}

As you can see, we’re defining the schema for our documents, and we’re inserting it inside our index.

Now let’s consider that ElasticSearch is a huge system and can take some seconds to boot up. We can’t connect to ES until it is ready, so we need a function that checks when the ES server is ready:

function checkConnection() {
  return new Promise(async (resolve) => {
    console.log("Checking connection to ElasticSearch...");
    let isConnected = false;
    while (!isConnected) {
      try {
        await esclient.cluster.health({});
        console.log("Successfully connected to ElasticSearch");
        isConnected = true;
      // eslint-disable-next-line no-empty
      } catch (_) {
      }
    }
    resolve(true);
  });
}

As you can see, we’re returning a promise. That’s because by using async/await we’re able to stop the entire Node.js process until this promise resolves, and it won’t do it until it connects to ES. That way, we force our Node.js to wait for ES before booting up.

We’re done with ElasticSearch! Now let’s export our functions:

module.exports = {
  esclient,
  setQuotesMapping,
  checkConnection,
  createIndex,
  index,
  type
};

Great! Let’s see the entire elastic.js file:

const { Client } = require("@elastic/elasticsearch");
                   require("dotenv").config();
const elasticUrl = process.env.ELASTIC_URL || "http://localhost:9200";
const esclient   = new Client({ node: elasticUrl });
const index      = "quotes";
const type       = "quotes";
/**
 * @function createIndex
 * @returns {void}
 * @description Creates an index in ElasticSearch.
 */
async function createIndex(index) {
  try {
    await esclient.indices.create({ index });
    console.log(`Created index ${index}`);
  } catch (err) {
    console.error(`An error occurred while creating the index ${index}:`);
    console.error(err);
  }
}
/**
 * @function setQuotesMapping,
 * @returns {void}
 * @description Sets the quotes mapping to the database.
 */
async function setQuotesMapping () {
  try {
    const schema = {
      quote: {
        type: "text" 
      },
      author: {
        type: "text"
      }
    };
  
    await esclient.indices.putMapping({ 
      index, 
      type,
      include_type_name: true,
      body: { 
        properties: schema 
      } 
    })
    
    console.log("Quotes mapping created successfully");
  
  } catch (err) {
    console.error("An error occurred while setting the quotes mapping:");
    console.error(err);
  }
}
/**
 * @function checkConnection
 * @returns {Promise<Boolean>}
 * @description Checks if the client is connected to ElasticSearch
 */
function checkConnection() {
  return new Promise(async (resolve) => {
    console.log("Checking connection to ElasticSearch...");
    let isConnected = false;
    while (!isConnected) {
      try {
        await esclient.cluster.health({});
        console.log("Successfully connected to ElasticSearch");
        isConnected = true;
      // eslint-disable-next-line no-empty
      } catch (_) {
      }
    }
    resolve(true);
  });
}
module.exports = {
  esclient,
  setQuotesMapping,
  checkConnection,
  createIndex,
  index,
  type
};

Populating ElasticSearch with quotes

Now we need to populate our ES instance with our quotes. This may sound easy, but believe me, it can be tricky.

Let’s create a new file in /src/data/index.js:

const elastic = require("../elastic");
const quotes  = require("./quotes.json");

const esAction = {
  index: {
    _index: elastic.index,
    _type: elastic.type
  }
};

As you can see, we’re importing the elastic module we just created and our quotes from a JSON file stored in /src/data/quotes.json. We’re also creating an object called esAction, which will tell ES how to index a document once we insert it.

Now we need a script to populate our database. We also need to create an array of Object with the following structure:

[
  {
    index: {
      _index: elastic.index,
      _type:  elastic.type
    }
  },
  {
    author: "quote author",
    quote:  "quote"
  },
  ...
]

As you can see, for each quote we’re gonna insert, we need to set its mapping to ElasticSearch. So that’s what we’ll do:

async function populateDatabase() {
  const docs = [];
  for (const quote of quotes) {
    docs.push(esAction);
    docs.push(quote);
  }
  return elastic.esclient.bulk({ body: docs });
}

Great! Now let’s create our main file in /src/main.js and see how we’ll structure everything we’ve written so far:

const elastic = require("./elastic");
const data    = require("./data");
                require("dotenv").config();

(async function main() {

  const isElasticReady = await elastic.checkConnection();
  if (isElasticReady) {
    const elasticIndex = await elastic.esclient.indices.exists({index: elastic.index});

    if (!elasticIndex.body) {
      await elastic.createIndex(elastic.index);
      await elastic.setQuotesMapping();
      await data.populateDatabase()
    }
  }

})();

Let’s analyze the code above. We create a self-executing main function which will check for the ES connection. The code execution won’t proceed until ES is connected. When ES is ready, we’ll check if the quotes index exists. If it doesn’t, we’ll create it, we’ll set its mapping and will populate the database. Obviously, we’ll do so only the first time we start up our server!

Creating a RESTful API

Now we need to create our RESTful server. We’ll be using Express.js, which is the most popular Node.js framework out there for building servers.

We’ll start with the /src/server/index.js file:

const express      = require("express");
const cors         = require("cors");
const bodyParser   = require("body-parser");
const routes       = require("./routes");
                     require("dotenv").config();

const app  = express();
const port = process.env.NODE_PORT || 3000;

function start() {
  return  app.use(cors())
             .use(bodyParser.urlencoded({ extended: false }))
             .use(bodyParser.json())
             .use("/quotes",routes)
             .use((_req, res) => res.status(404).json({ success: false,error: "Route not found" }))
             .listen(port, () => console.log(`Server ready on port ${port}`));
}

module.exports = {
  start
};

As you can see, it is just a standard Express.js server, we won’t spend much time on that.

Let’s see our /src/server/routes/index.js file:

const express    = require("express");
const controller = require("../controllers");
const routes     = express.Router();

routes.route("/").get(controller.getQuotes);
routes.route("/new").post(controller.addQuote);

module.exports = routes;

We just create two endpoints:

1. GET /, will return a list of quotes matching our query string parameters.
2. POST /new/, will allow us to post a new quote storing it inside ElasticSearch.

So now let’s see our /src/server/controllers/index.js file:

const model = require("../models");

async function getQuotes(req, res) {
  const query  = req.query;
  if (!query.text) {
    res.status(422).json({
      error: true,
      data: "Missing required parameter: text"
    });
    return;
  }
  try {
    const result = await model.getQuotes(req.query);
    res.json({ success: true, data: result });
  } catch (err) {
    res.status(500).json({ success: false, error: "Unknown error."});
  }
}

async function addQuote(req, res) {
  const body = req.body;
  if (!body.quote || !body.author) {
    res.status(422).json({
      error: true,
      data: "Missing required parameter(s): 'body' or 'author'"
    });
    return;
  }
  try {
    const result = await model.insertNewQuote(body.quote, body.author);
    res.json({ 
      success: true, 
      data: {
        id:     result.body._id,
        author: body.author,
        quote:  body.quote
      } 
    });
  } catch (err) {
    res.status(500).json({ success: false, error: "Unknown error."});
  }
}
module.exports = {
  getQuotes,
  addQuote
};

Here we’re basically defining two functions:

1. getQuotes, which requires at least one querystring parameter – text
2. addQuote, which require two parameters – author and quote

The ElasticSearch interface is delegated to our /src/server/models/index.js. This structure helps us maintain a MVC-ish architecture.

Let’s see our model:

const { esclient, index, type } = require("../../elastic");

async function getQuotes(req) {
  const query = {
    query: {
      match: {
        quote: {
          query: req.text,
          operator: "and",
          fuzziness: "auto"
        }
      }
    }
  }

  const { body: { hits } } = await esclient.search({
    from:  req.page  || 0,
    size:  req.limit || 100,
    index: index, 
    type:  type,
    body:  query
  });

  const results = hits.total.value;
  const values  = hits.hits.map((hit) => {
    return {
      id:     hit._id,
      quote:  hit._source.quote,
      author: hit._source.author,
      score:  hit._score
    }
  });

  return {
    results,
    values
  }
}

As you can see, we’re composing our ElasticSearch query by selecting every quote that includes a given word or phrase.

Then, we spawn the query, setting both page and limit values we can pass them in a query string, for instance http://localhost:3000/quotes?text=love&page=1&limit=100. If these values are not passed via querystring, we’ll fall back to their default values.

ElasticSearch returns an incredibly big amount of data, but we just need four things:

1. Quote ID
2. The quote itself
3. Quote author
4. Score

The score represents how close the quote is to our search term. Once we have these values, we return them with the total results number, which may be useful when paginating the results on the front end.

Now we need to create the last function for our model insertNewQuote:

async function insertNewQuote(quote, author) {
  return esclient.index({
    index,
    type,
    body: {
      quote,
      author
    }
  })
}

This function is really simple, we just post the quote and the author to our index and return the query result back to the controller.

Now the complete /src/server/models/index.js file should look like this:

const { esclient, index, type } = require("../../elastic");

async function getQuotes(req) {
  const query = {
    query: {
      match: {
        quote: {
          query: req.text,
          operator: "and",
          fuzziness: "auto"
        }
      }
    }
  }

  const { body: { hits } } = await esclient.search({
    from:  req.page  || 0,
    size:  req.limit || 100,
    index: index, 
    type:  type,
    body:  query
  });

  const results = hits.total.value;

  const values  = hits.hits.map((hit) => {
    return {
      id:     hit._id,
      quote:  hit._source.quote,
      author: hit._source.author,
      score:  hit._score
    }
  });

  return {
    results,
    values
  }
}

async function insertNewQuote(quote, author) {
  return esclient.index({
    index,
    type,
    body: {
      quote,
      author
    }
  })
}

module.exports = {
  getQuotes,
  insertNewQuote
}

And we’re done! We just need to setup our start script inside out package.json file and we’re ready to go:

"scripts": {
  "start": "pm2-runtime start ./src/main.js --name node_app",
  "stop": "pm2-runtime stop node_app "
}

We also need to update our /src/main.js script in order to start up our Express.js server once ElasticSearch has been connected:

const elastic = require("./elastic");
const server  = require("./server");
const data    = require("./data");
                require("dotenv").config();

(async function main() {
  const isElasticReady = await elastic.checkConnection();

  if (isElasticReady) {
    const elasticIndex = await elastic.esclient.indices.exists({index: elastic.index});

    if (!elasticIndex.body) {
      await elastic.createIndex(elastic.index);
      await elastic.setQuotesMapping();
      await data.populateDatabase()
    }

    server.start();
  }
})();

Launching the application

We’re now ready to start our application using docker-compose!

Just run the following command:

docker-compose up

You’ll need to wait until Docker downloads both the ElasticSearch and Node.js images, then it will boot up your server and you’ll be ready to query against your REST endpoints!

Let’s test with a couple of cURL calls:

curl localhost:3000/quotes?text=love&limit=3

{
  "success": true,
  "data": {
    "results": 716,
    "values": [
      {
        "id": "JDE3kGwBuLHMiUvv1itT",
        "quote": "There is only one happiness in life, to love and be loved.",
        "author": "George Sand",
        "score": 6.7102118
      },
      {
        "id": "JjE3kGwBuLHMiUvv1itT",
        "quote": "Live through feeling and you will live through love. For feeling is the language of the soul, and feeling is truth.",
        "author": "Matt Zotti",
        "score": 6.2868223
      },
      {
        "id": "NTE3kGwBuLHMiUvv1iFO",
        "quote": "Genuine love should first be directed at oneself if we do not love ourselves, how can we love others?",
        "author": "Dalai Lama",
        "score": 5.236455
      }
    ]
  }
}

So as you can see, we decided to limit our results to 3, but there are over 713 quotes!

We can easily get the next three quotes by calling:

curl localhost:3000/quotes?text=love&limit=3&page=2

{
  "success": true,
  "data": {
    "results": 716,
    "values": [
      {
        "id": "SsyHkGwBrOFNsaVmePwE",
        "quote": "Forgiveness is choosing to love. It is the first skill of self-giving love.",
        "author": "Mohandas Gandhi",
        "score": 4.93597
      },
      {
        "id": "rDE3kGwBuLHMiUvv1idS",
        "quote": "Neither a lofty degree of intelligence nor imagination nor both together go to the making of genius. Love, love, love, that is the soul of genius.",
        "author": "Wolfgang Amadeus Mozart",
        "score": 4.7821507
      },
      {
        "id": "TjE3kGwBuLHMiUvv1h9K",
        "quote": "Speak low, if you speak love.",
        "author": "William Shakespeare",
        "score": 4.6697206
      }
    ]
  }
}

And what if you need to insert a new quote? Just call the /quotes/new endpoint!

curl --request POST \
     --url http://localhost:3000/quotes/new \
     --header 'content-type: application/json' \
     --data '{
        "author": "Michele Riva",
        "quote": "Using Docker and ElasticSearch is challenging, but totally worth it."
}'

and the response will be:

{
  "success": true,
  "data": {
    "id": "is2QkGwBrOFNsaVmFAi8",
    "author": "Michele Riva",
    "quote": "Using Docker and ElasticSearch is challenging, but totally worth it."
  }
}

Conclusion

Docker makes it incredibly easy to manage our dependencies and their deployment. From that point, we could host our application on Heroku, AWS ECS, Google Cloud Container or any other Docker-based service with ease, without struggling to setup our server with their super-complex configurations.

Next steps?

• Learn how to use Kubernetes in order to scale your container and orchestrate more ElasticSearch Instances!
• Create a new endpoint that will allow you to update an existing quote. Mistakes can happen!
• And what about deleting a quote? How would you implement that endpoint?
• It would be great to save your quotes with tags (for instances, quotes about love, health, art)… try to update your quotes index!

Software development is fun. With Docker, Node, and ElasticSearch it’s even better!

Plug: LogRocket, a DVR for web apps

LogRocket is a frontend logging tool that lets you replay problems as if they happened in your own browser. Instead of guessing why errors happen, or asking users for screenshots and log dumps, LogRocket lets you replay the session to quickly understand what went wrong. It works perfectly with any app, regardless of framework, and has plugins to log additional context from Redux, Vuex, and @ngrx/store.

In addition to logging Redux actions and state, LogRocket records console logs, JavaScript errors, stacktraces, network requests/responses with headers + bodies, browser metadata, and custom logs. It also instruments the DOM to record the HTML and CSS on the page, recreating pixel-perfect videos of even the most complex single-page apps.

Overview

Elastic — the company behind Elasticsearch and the Elastic Stack — announced the launch of Elasticsearch Service on Microsoft Azure. The move gives DevOps and platform engineering teams who have standardized on Azure a fully managed, officially supported path to run Elasticsearch and Kibana without managing the underlying infrastructure themselves.

Prior to this launch, Azure-based organizations had to self-manage Elasticsearch clusters on Azure VMs or use third-party offerings. The new service, hosted on Elastic Cloud, is operated directly by Elastic — meaning users get the same team that builds the software also supporting and operating it.

Key Capabilities at Launch

Elasticsearch Service on Azure launched as a public beta accessible to all existing Elasticsearch Service customers and new trial users. Highlights at launch included:

• One-click deployment of hosted Elasticsearch and Kibana directly from the Elastic Cloud console
• Access to the full Elastic Stack feature set: Elastic APM, SIEM, Maps, Canvas, and machine learning
• Single-click upgrades to new Elasticsearch releases with zero downtime
• Automatic security patches applied by Elastic’s operations team
• A free 14-day trial for new users
• Existing Elasticsearch Service customers could add Azure deployments within their existing accounts

At beta launch, the service was available in two Azure regions: East US 2 (Virginia) and West Europe (Netherlands). Elastic indicated plans to expand regions and move the service to general availability later in 2019, at which point mission-critical support tiers would also be added.

Deepening the Elastic and Microsoft Collaboration

This launch built on an existing partnership between Elastic and Microsoft. The two companies had previously collaborated on an Azure Resource Manager (ARM) template for the Elastic Stack, which many teams were already using to deploy self-managed clusters on Azure VMs.

Elastic’s broader Microsoft ecosystem support includes:

• .NET clients for Elasticsearch
• Winlogbeat for collecting Windows OS event logs and shipping them to Elasticsearch
• Azure Active Directory integration for securing Elasticsearch clusters
• .NET support in Elastic APM for application performance monitoring of .NET workloads

Shay Banon, founder and CEO of Elastic, commented: “The developer and open source focus that both companies share have made this integration a very natural fit. Microsoft’s commitment to choice is evident in their developer experience on Microsoft Azure, and mirrors our own.”

Scott Guthrie, EVP of Cloud + AI at Microsoft, added: “As customers adopt cloud services, having a solution for their most important needs such as search, logging, observability, and security of their critical applications, will be a key advantage. The focus on developer choice and managed services that both Microsoft and Elastic share benefit our mutual customers.”

Four Optimized Deployment Templates for Azure

Elastic’s engineering teams worked with Microsoft to benchmark and select optimal Azure VM types for common Elasticsearch workload profiles. The result was four deployment templates, each targeting a different performance and cost profile:

• High I/O: Designed for search-heavy or general-purpose use cases. Runs on L-series VMs equipped with local NVMe SSDs optimized for high read/write throughput.
• Hot/Warm: A tiered architecture suited to logging and time-series workloads. Combines NVMe SSD for the “hot” tier (fast, recent data) with HDD storage offering a 1:100 RAM-to-disk ratio for cost-effective long-term retention in the “warm” tier.
• High CPU: Targets compute-intensive workloads such as ingest pipelines, scripting, and aggregation-heavy queries. Provides double the CPU allocation relative to comparable memory.
• High Memory: A cost-effective option for search use cases that operate on lower data volumes but require larger JVM heaps and in-memory indices.

Choosing the right template at deployment time can significantly affect both performance and cloud spend, so understanding the expected query and ingestion patterns before deploying is worthwhile.

Beta Pricing Promotions

During the beta period, Elastic offered a promotional window in which data transfer costs (ingress, egress, and inter-node traffic) and snapshot storage costs (storage size and API requests) were waived. This made the beta period a low-risk time for teams to evaluate the service for production workloads before committing to full pricing.

Why This Matters for DevOps Teams

For DevOps and SRE teams already operating in Azure, the managed Elasticsearch Service eliminates significant operational overhead: cluster sizing, version upgrades, certificate management, snapshot scheduling, and monitoring of the Elasticsearch process itself all become Elastic’s responsibility rather than yours.

Teams that had been running Elasticsearch on self-managed Azure VMs — whether via the ARM template or custom Terraform configurations — gained a supported migration path to a fully managed topology without leaving the Azure ecosystem or switching cloud providers.

The addition of Azure also gave multi-cloud organizations the flexibility to co-locate Elasticsearch workloads on the same cloud provider as other Azure-native services (Azure Kubernetes Service, Azure Monitor, Azure Active Directory, etc.), reducing cross-cloud data transfer costs and latency.

Elastic Brings Managed Elasticsearch to GCP Tokyo

Elastic N.V. (NYSE: ESTC), the company behind Elasticsearch and the Elastic Stack, announced in August 2019 that Elasticsearch Service on Elastic Cloud became available in a fifth Google Cloud Platform (GCP) region: Asia Pacific (Tokyo). This made Tokyo the first GCP region for Elastic in Asia Pacific, complementing four other GCP regions the company already operated globally at the time of the announcement.

The expansion was driven by growing demand from Japanese enterprises and SMBs modernising their IT infrastructure — moving workloads off on-premises, traditional data centres and onto public cloud platforms.

“Google Cloud Platform (GCP) has a growing presence in Japan and we’re excited to partner with Google. This new GCP Tokyo region for Elasticsearch Service will give Japan customers a SaaS-first solution for their search, logging, APM, business analytics, infrastructure monitoring, security use cases and more.”

— Tom Kawasaki, Elastic Japan Country Manager

What Is Elasticsearch Service on Elastic Cloud?

Elasticsearch Service on Elastic Cloud is the official hosted Elasticsearch and Kibana offering, built and supported by Elastic themselves. As a fully managed service, it abstracts away cluster administration so teams can focus on their data and applications rather than operational overhead.

Key capabilities included at the time of this announcement:

• Elastic APM — Application performance monitoring integrated directly into the stack
• Elastic SIEM — Security information and event management for threat detection and response
• Elastic Maps — Geospatial visualisation within Kibana
• Canvas — Custom, live infographic-style presentations of data
• Machine learning — Anomaly detection and forecasting built into the platform
• Same-day version and security patch availability
• Zero-downtime upgrades
• Pre-configured hardware templates optimised for common use cases
• Direct support from Elastic

All of these features were made available immediately in the new Asia Pacific (Tokyo) region, consistent with Elastic’s other regional deployments.

Why the Tokyo Region Matters for Asia Pacific Customers

Running Elasticsearch clusters in-region reduces network latency for end users and applications hosted in Japan and nearby Asia Pacific markets. It also helps organisations meet data residency and compliance requirements that may mandate data remain within specific geographic boundaries.

Industries with particularly strong demand for low-latency search, real-time analytics, and log management — such as gaming, e-commerce, and financial services — stand to benefit directly from regional availability.

“As a GCP partner as well as an Elastic partner, we’re extremely excited to start using the Elasticsearch Service on the GCP Tokyo region. The gaming industry and other industries in Japan will greatly appreciate this announcement.”

— Yusuke Hasegawa, President and CEO, grasys Inc.

Deepening the Elastic–Google Cloud Partnership

The Tokyo launch was part of a broader, multi-year strategic partnership between Elastic and Google Cloud. At the time of the announcement, several additional integration milestones were either completed or in progress:

Next GCP Region: Sydney, Australia

Elastic announced that the next planned GCP region would be in Sydney, Australia, further expanding Asia Pacific coverage.

Native GCP Console Integration

Elasticsearch Service was being positioned as a Managed Partner Service (MPS) within the main GCP Console — the same interface where native Google services are provisioned. The planned integration would allow lifecycle operations such as creating, scaling, configuring, and reconfiguring Elasticsearch clusters using familiar GCP-style workflows, without leaving the GCP Console.

Integrated Billing Through GCP Marketplace

GCP customers were set to be able to provision Elasticsearch Service through the GCP Marketplace and have that spend count fully towards their existing Google Cloud committed spend. This simplifies procurement significantly for enterprises with pre-negotiated GCP contracts.

“Google Cloud and Elastic are both committed to the open source community and are excited to announce the expanded availability of Elastic Cloud in GCP’s Tokyo region, and unified billing, coming soon. This brings simplicity and speeds up the time to value realization for joint customers.”

— Masato Takahashi, Director, Partnerships, Google Cloud Japan

“New regions like Tokyo and Sydney, seamless access to the Elasticsearch Service through the Google Cloud Platform Console, and billing through Google Cloud, which is targeted for release this August, are all examples of our mutual investments to create a great user experience in the cloud.”

— Craig Griffin, Vice President of Business Development, Elastic

Getting Started with Elasticsearch Service on GCP Tokyo

The following guidance was provided at the time of the announcement:

• Existing Elasticsearch Service users: Log in to Elastic Cloud and select the Asia Pacific (Tokyo) GCP region when creating a new deployment.
• Migrating self-managed workloads: Elastic’s documentation and migration webinars provide guidance on moving from self-hosted Elasticsearch to the managed service.
• New users: A 14-day free trial was available, requiring only an email address to get started.

About Elastic

Elastic is a search company and the creator of the Elastic Stack — comprising Elasticsearch, Kibana, Beats, and Logstash. Elastic builds both self-managed and SaaS offerings designed to make data usable in real time and at scale across search, logging, security, and analytics use cases.

Elastic and associated marks are trademarks or registered trademarks of Elastic N.V. and its subsidiaries. All other company and product names may be trademarks of their respective owners.