Almost two years after unveiling its Inferentia high-performance machine learning inference chips, Amazon has almost completed a migration of the bulk of its Alexa text-to-speech ML inference workloads to Inf1 instances on its AWS EC2 platform. The dramatic move to infrastructure powered by the newer chips was made to gain significant cost savings and performance gains over the GPU-based instances on EC2 that were formerly being used, according to the company.

By migrating the workloads to Inf1 instances, the Alexa text-to-speech workloads experience 25% lower end-to-end latency, while cutting operating costs by 30% compared to the previously used GPU-based instances, Amazon announced on Thursday (Nov. 12). By reducing the latency, Alexa engineers are also able to use more complex algorithms to improve the Alexa experience for users.

The Amazon Inferentia chip design was unveiled at the AWS re:Invent conference in November 2018, while Inf1 inferences followed a year later with an announcement at re:Invent in 2019.

The Inferentia chips allow Inf1 instances to provide up to 30% higher throughput and up to 45% lower cost per inference compared to GPU-based G4 instances, which previously were the most economical AWS cloud instances for ML inference, according to Amazon.

“While training rightfully receives a lot of attention, inference actually accounts for the majority of complexity and the cost – for every dollar spent on training, up to nine are spent on inference – of running machine learning in production, which can limit much broader usage and stall innovation,” an AWS spokesperson told EnterpriseAI. “EC2 instances – virtual servers running in the AWS cloud – offer a broad range of instance types powered by a variety of processors to meet specific workload requirements. AWS offers EC2 instances based on the fastest processors from Intel, cost optimized instances with AMD processors, the most powerful GPU instances from Nvidia, as well as instances based on AWS Inferentia chips, a high performance machine learning inference chip designed by AWS.”

While the lower latency benefits Alexa engineers in their text-to-speech work, customers across a diverse set of industries are seeing similar benefits as they turn to machine learning to address common use cases such as personalized shopping recommendations, fraud detection in financial transactions, increasing customer engagement with chat bots and more, the AWS spokesperson said. “Many of these customers are evolving their use of machine learning from running experiments to scaling up production machine learning workloads where performance and efficiency really matter. Customers want high performance for their machine learning applications in order to deliver the best possible end user experience.”

That’s where the Inf1 instances provide answers, the spokesperson added. “With Inf1 instances, customers can run large scale machine learning inference to perform tasks like image recognition, speech recognition, natural language processing, personalization, and fraud detection at the lowest cost in the cloud.”

Inferencing is the science and practice of making predictions using a trained ML model. The Inferentia chips were developed by Israel-based Annapurna Labs, which was acquired by Amazon in 2015. Each Inferentia chip provides hundreds of TOPS (tera operations per second) of inference throughput. Hundreds of TOPS can be included in each chip and the chips can be banded together to harness thousands of TOPS.

The Inferentia chips are designed for larger workloads that consume entire GPUs or require lower latency. The chips provide hundreds of teraflops each and thousands of teraflops per Amazon EC2 instance for multiple frameworks, including TensorFlow, Apache MXNet, and PyTorch. They also are built for multiple data types, including INT-8 and mixed precision FP-16 and bfloat16. The Inferentia chips complement AWS’s Amazon Elastic Inference, which allows customers to avoid running workloads on whole Amazon EC2 P2 or P3 instances where there is relatively low utilization. Instead, developers can run on a smaller, general-purpose Amazon EC2 instance and use Elastic Inference to provision the right amount of GPU performance, resulting in up to 75% cost savings.

Despite the ongoing COVID-19 pandemic, companies around the world are continuing to seek IT workers with proven skills in open source software to fill job vacancies that consistently remain open due to a lack of qualified candidates.

That’s the conclusion of the 2020 Open Source Jobs Report from The Linux Foundation and learning platform vendor edX, which surveyed 175 hiring managers and more than 900 open source professionals to examine the demand and prospects for open source skills in the marketplace.

“Ninety-three percent of hiring managers are still having trouble finding open source talent,” Clyde Seepersad, the director of training and certification for The Linux Foundation, told EnterpriseAI. “Cloud is super-hot.”

The problem, said Seepersad, is that many job candidates continue to lack the specific open source skills that companies are desperately seeking.

The 18-page report is the latest update since the organization’s 2018 open source jobs report, when 87% of hiring managers said they were having a difficult time finding qualified open source talent for their companies. Some 37% of the hiring managers say they will hire more skilled IT professionals in the next six months.

DevOps Skills in Greater Demand

One area where skilled open source job candidates are needed in particular is in DevOps, where 65% of the companies surveyed said they are looking to hire more DevOps talent. DevOps was the most-sought-after job role mentioned in the study, surpassing software developers for the first time among respondents, according to The Linux Foundation. Software developers with open source skills are being sought by 59% of the respondents.

Also notable is that 52% of hiring managers are more likely to hire candidates with open source certifications, which is up from 47% two years ago.

Some 70% of the hiring managers said they are more likely to hire an IT pro today who has open source cloud skills in particular, which is up from 66% in 2018.

“We’re seeing signs that the long-rumored prominence of DevOps is really starting to happen among people who are getting serious about hiring DevOps talent and implementing DevOps philosophy and culture in their organizations,” said Seepersad. “What we’re seeing is that IT has been resilient and open source in particular [during the COVID-19 pandemic].”

Interestingly, the report also says that 80% of the surveyed employers now provide online training courses for employees to learn open-source software skills, allowing them to boost the skills of existing workers to help solve their job vacancy issues. That’s an increase from 66% two years ago, according to the report.

“More and more companies are realizing that acting on that upskilling is an important part of recruiting,” said Seepersad. “You don’t have to just bring in external candidates. There’s a viable path to growing skills in an organization to fill the evolving gaps. I suspect that’s going to become one of the big changes we’ll continue to see.”

Also important, he said, is that an increasing number of companies are looking at hiring remote workers around the globe, especially due to the pandemic, which can allow them to find qualified talent more flexibly.

More Companies Supporting Open Source Projects to Lure Talent

Some 63% of the business respondents said their organizations have begun to support open source projects with code or other resources just so they can more easily recruit IT workers with those software skills, the study reported. That’s an increase from the 48% who did this in 2018. About 74% of the employers surveyed said they are also now paying for employee open source certifications to encourage workers to bolster their skills. That’s up from 55% in 2018, 47% in 2017, and 34% in 2016.

For companies specializing in AI, machine learning and related technologies, finding talent with open source skills continues to grow in importance, said Seepersad.

“Everything that’s running in the cloud is running on Linux, including 99% of Amazon EC2, and the majority of Azure instances,” he said. “So, whether you knew it or not, everybody’s an open source technologist now.”

And in AI in particular, hundreds of different open source tool sets and frameworks have bloomed and are available for use, he said. “We’re all Linux admins now, because if you lift the covers it’s what’s underneath the cloud and everything. And we’re all Kubernetes people now because that’s how everybody orchestrates and instruments the cloud. And open source tools are what are powering the DevOps revolution, so whether that’s Puppet or Chef or Jenkins or Spinnaker, there’s a plethora of tools out there that are the new normal for how code gets developed and deployed.”

The group’s 2020 Open Source Jobs Report was conducted for the first time with the help of edX, an education and learning platform that was founded by Harvard and MIT.

The report includes data from more than 175 hiring managers at corporations, small and medium businesses (SMBs), government organizations, and staffing agencies across the globe, as well as responses from more than 900 open source professionals worldwide. Thirty-nine percent of the companies are based in North America, while 24% are based in Europe. Another 15% are in Asia, 7% are in Africa, 4% are in Mexico and Central America, 4% are in South America, 2% are in the Middle East and less than 1% each are in other regions. The respondents must have hired at least one open source professional in the last year or plan to hire open source professionals in 2020 to participate in the survey. Seventy-four percent of the 900 open source professionals who responded worked in those jobs for three years or more.