If you have been working from home during the recent lockdowns, you probably don’t feel like you are making many new connections, but when we come to look back on this time in a few years, it might prove to have been one of the most effective networking sessions of the past decade.

Nowhere is this more true than in the twin worlds of big data and DevOps. It seems that being forced to work from home, far from locking engineers in both disciplines deeper within their silos, has brought them out of their shells. In recent months, we’ve seen an outpouring of speculation on how DevOps and big data can work together, and even a number of impassioned manifestos on why it is time to bring DevOps ideas to big data.

In this article, we’ll look at why this crossover is suddenly so popular, how these two disciplines can engage in a symbiotic relationship with each other, and how they are already doing so.

The Pandemic and the Silo
For consumers, the experience of lockdown has accelerated internet usage trends. While some of the changes that have recently swept the online world were easy to guess, the speed at which they have taken hold has been astounding. It’s therefore no exaggeration to say, as the New York Times put it recently, that the virus has changed the way we use the internet.

These consumer trends are driving changes throughout the software development industry, forcing firms to adapt to rapid release, testing and update cycles.

One of the biggest changes in the way that developers collect data over the past decade has been the source of the data. This year, and largely due to the COVID-19 pandemic, we passed a notable watershed: Mobile devices and smartphones now account for more than half of all internet traffic, resulting in an increased data flow that boggles the mind and creates issues of its own. This increased data load from consumer mobile devices, rather than from corporate data acquisition systems, is forcing teams to work more closely together.

In the words of Bill Detwiler, editor in chief of TechRepublic, in a recent CBS interview: “The COVID-19 pandemic has accelerated the blending of data analytics and DevOps, meaning developers, data scientists, and product managers will need to work more closely together than ever before.” For most firms, the way in which these consumer trends are fed into the development life cycle is via analytics teams. This has meant that, during the pandemic, analytics teams were generating actionable insights at an increased rate, and this has been the driving force behind the newly intimate relationship between big data and DevOps staff.

Waterfall vs. DevOps
To understand the way in which big data and DevOps teams can best work together, it’s instructive to visualize the way in which developers, operational staff and data analytics agents work.

A decade ago, most software developers were using some form of waterfall development methodology. As the name suggests, this approach was essentially linear—the specifications for software were set in advance, and after this point multiple teams worked in tandem to deliver a continuous “stream” of software updates.

The DevOps framework changed that by encouraging communication and collaboration between operations and development staff. The idea, in short, was to ensure that software underwent continual testing—not just in terms of usability and security, but also against the requirements of the operations staff who were charged with maintaining it.

Now, we might be facing a shift of similar scale. Many firms have now adopted big data frameworks, but these systems generally have been seen as a way of harvesting and collating market insights. This was always a waste of potential for big data because as long as it was regarded as part of marketing, it’s utility in other parts of an organization were difficult to discern.

Not any longer, it seems. The changes caused by the pandemic—both in consumer behavior and in the way that organizations communicate internally—have given developers an appreciation of the value of the data that analytics teams hold.

Pioneers Show the Way Forward
It’s also worth noting that big data and DevOps teams working in tandem is not a utopian vision; rather, this way of working was already being explored within leading companies even before the pandemic.

In retrospect, in fact, it seems crazy that these two teams ever worked independently. If you’re a transportation company, the ability to track your loads on the road and the health and safety of the cargo that they’re carrying has become mission-critical. If you’re in the armed forces and using drones on the battlefield to conduct and report reconnaissance in real-time flyovers, the ability to adapt software parameters on-the-fly can be the difference between success and failure.

The military is, in fact, the place to look for examples of how big data and DevOps will shortly become one process. Big data in the military has long been more advanced than that in the civilian sector and is beginning to inform everything from IT predictive analytics to real-time usage analytics.

The Future of Big Data and DevOps
There remain, of course, significant challenges to be overcome before the promise of DevOps working with big data teams can be realized. Perhaps the most thorny of these is the fact that computing performance is crucial for big data in a way it hasn’t been for development: The latter teams may need to build capabilities to make use of the insights generated by big data teams.

The demand for fast order fulfilment and delivery services skyrocketed in the context of the global lockdowns, ushering in a new era of challenges and opportunities for the beleaguered real estate sector. Robotic micro-fulfilment warehouses are now rolling out globally and providing an efficient automated selection of items to enable faster delivery to consumers.

Micro-fulfilment centres are much smaller than warehouses, which typically need to be on the outskirts of cities. Instead, micro-fulfilment centres are compact and can be set up within cities, ensuring quicker service to a large segment of customers

Improving operational efficiency

In addition to the low cost per square metre, these types of robotic fulfilment centres significantly boost operational efficiency. They also need only a handful of employees at a time. An operator at a counter – one of just four “touchpoints” – assembles incoming orders from arriving bins, picks the items from them, then scans and bags them. Bags are lowered and robotically ferried to a second touchpoint, an outbound terminal.

Operated by robots and usually smaller than 1,000 square metres, micro fulfilment centres are emerging as the future of last-mile delivery logistics. The strategy, operated through software and executed via automation, powers the strategy known as micro fulfilment. It is aimed at speeding up the delivery of goods to consumers in cities through operations that pack large numbers of products into tight, urban spaces.

Far smaller than the typical big-box distribution centres found in most industrial parks, the sites are becoming increasingly attractive to retailers adjusting to the changes in consumer markets. Some have been testing these types of small fulfilment sites in recent years, but the rush to online shopping thanks to Covid-19 pandemic has accelerated moves to space-saving, automation-powered warehouses.

Adapting to alternative automated systems

By squeezing those operations into small urban warehouses, businesses hope to pare down delivery times so online orders reach their destination in hours, not days. Companies storing goods at micro-fulfilment sites should be able to get orders picked and packed in five minutes or less, with only a handful of workers.

Most micro-fulfilment operations are still being tested, but the strategy is good news for grocery chains and other retailers scrambling to meet rising e-commerce demand during the pandemic. The future market for automated grocery micro-fulfilment centres is estimated to be worth $1.2bn by 2024, according to market research firm Interact Analysis.

The key is to store products in spaces that only robots, not humans, can reach. There is no vertical airspace between bins and only about 20cm of space between stacks. Grocery installations typically include three different sections: one for room-temperature products, another for cooler items, and a manual picking area for fast-moving goods.

Grocery chain and retailer Walmart is among the companies testing various automated systems, but non-grocery retailers have been slower to embrace the concept. This is in spite of the fact that the automated alternative requires less space than traditional distribution, and reduces the number of steps for human workers.

Building faster ROI

For retailers working hard to meet the accelerated online demand, the technology offers a faster return on investment than larger robotics-equipped warehouses, while allowing retailers to fill orders more quickly than they could with human workers walking store aisles.

The market change brought on by the pandemic has not only escalated the necessity to fulfil orders in real-time. Automation is already replacing some of the labour-intensive processes either fully, or in part. For example, automated loading and unloading, package delivery, and even automated vehicles are already being tested and implemented in some cases.

We’re still pretty far off from seeing some of these things happen in South Africa, but distributors have to start looking at the big picture in respect of how they service their customers most efficiently in the next three to five years.