Developers Say Cloud Development Still Too Manual & Complex

TL;DR: We interviewed more than 30 developers to hear how organizations are handling cloud-native software development and delivery today. We learned a lot, and realized there’s still a long way to go.

Modern application development has come a long way thanks to DevOps practices. But front-line engineers still struggle with building apps for modern cloud-native infrastructure. They face immature processes, toil with too much manual work and needless complexity, and lack the specialty knowledge necessary to deliver software within the current set of common tools, platforms, and pipelines. This stresses their ability to deliver production ready code, containers, and systems quickly and efficiently. They are spending more and more time on DevOps-related tasks, and not enough time writing the application code that powers their businesses.

Part One of Four in a series about the current challenges in DevOps and ideas to solve them.

The DevOps revolution, still in its infancy, continues to evolve at a rapid pace as organizations search for ways to better leverage the cloud efficiently and effectively. New technologies emerge every day aimed at making application development faster, scalable, and more reliable. But with new tools and techniques come new challenges. The reality for even the best DevOps organizations is often just as fragmented, arcane, and ad hoc as it was in the pre-cloud era.

We interviewed more than thirty developers, many of them users of our open source project, DockerSlim, to hear how organizations are handling cloud-native software development and delivery, specifically focusing on the role of containers today.

While the developers we spoke with varied tremendously in experience, job role, industry, and technology preference, we heard several commonalities across all interviews. Nearly every developer we interviewed was using containers (Docker), container orchestration (Kubernetes), and CI/CD pipeline tools to build in the cloud. And nearly all were experiencing growing pains in their organization as they tried to find collaborative ways to leverage these technologies to speed deployments and reduce efforts across the engineering teams.

Here’s what they had to say:

Modern CI/CD processes are still often manual & inefficient

“What’s unclear to almost everyone is exactly how to do what I did. Nobody knows what they can clean out of a container, because most people don’t actually know what’s in them or why.”

This is how one DockerSlim user and cloud development industry veteran described his container optimization process at a major software infrastructure platform. Through cumbersome, mostly manual efforts, he is able to reduce his container sizes 10x, from approximately 300MB to just 30MB, but concedes he “has tribal knowledge” of DevOps that most others don’t.

Most organizations rely on dedicated engineering teams to build automated platforms and pipelines, with individual application developers tasked with making application- or service-specific infrastructure decisions more and more often. According to GitLab’s 2020 DevSecOps Report, 35% of developers surveyed define and create their own infrastructure, a role formerly owned exclusively by Ops teams.

The benefit of this, according to the developers we spoke with, is that engineers are finding new and efficient ways to work together, leveraging DevOps concepts and mentality to create the team cultures, practices and toolchains needed to reap the benefits of cloud infrastructure. The downside is there is often little coordination between teams on core infrastructure problems.

“Some of the infrastructure stuff is done for us, but a lot of it is manual,” said a product team developer working at a well-known analytics and monitoring platform. “And because of our silos, we all duplicate work — like way more than we should. Everyone has their own scripts for uploading Docker containers, for example.”

Despite plenty of efforts from large Infrastructure-as-a-Service (IaaS) companies to foster vertical integration, as well as efforts from specialized start-ups to create “the best tool for the job”, consistency in container management within an organization is still ad hoc, at best.

Technology managers chafe at the idea of developers repeating core tasks and solving problems individually. And this inefficiency becomes compounded when there are workarounds created to build local environments or pass integration tests en route to production. Most of the developers we interviewed struggled to replicate production environments locally on their machine without a lot of manual effort.

“The worst, honestly, is, like… anything, when you have a large microservices environment,” said one veteran of a cloud infrastructure company. “The staging environment never gets the same amount of love as the production environment. So there can be things that are broken there that you have to rerun.”

The out-of-the-box CI/CD environment is often too generic & customization is too complex

The delta is not only between the developer’s local environment and the production system, but also in the pipeline itself. We employ CI/CD systems to bridge the gap between staging and production, but developers admit that a lot of work and specialized knowledge goes into customizing CI/CD for their applications, and the inefficiencies extend into these systems.

The developers we interviewed described a DevOps environment that has a lot of useful tools, but can, at times, require significant time, customization and expertise to get right.

According to industry reports, the container services market has doubled in just the past two years, and is projected to grow another 50% by 2022. With so many new companies and services, not to mention emerging open-source projects, developers choose from a veritable buffet of pipeline tools and technologies.

Sometimes, it’s too much choice. When describing their development processes, the engineers we spoke with commonly described 15 to 25 different tools or technologies they were using, considering, or had used but already deprecated. Even more telling, the number of tools and tech being used varied little between developers describing themselves or their organizations as advanced DevOps practitioners or relative newbies.

“I have no problem editing the Dockerfile and running stuff locally,” said the cloud infrastructure engineer. “But if I were going to do anything interesting — like if I were going to build two network systems from scratch and run them containerized locally — I’d have to learn a lot about infrastructure, going through a lot of documentation.”

The problem is that there is a huge lack of documentation, and reliance on expert/tribal knowledge.

Simply put: There just isn’t one right way to do DevOps. Its application varies from project to project, problem to problem. This flexibility makes problem solving interesting and gives developers a ton of choice, but like the manual processes described above, can get in the way of scale, consistency, and efficiency. Paraphrasing what we heard often: “[Open source tools] are only free if your time has no value.”

Conclusion: It’s Time to Shift Left

The DevOps landscape is still emerging. At Slim.AI, we believe the next era of cloud-based application will come when any application developer can quickly and easily author, optimize, and ship infrastructure without massive bespoke delivery systems, myriad specialty knowledge, or messy ad-hoc processes.

To achieve this, we need to empower every developer in the organization to have tools and know-how to manage containers that suit their needs at every stage of the software development lifecycle. And we need to do better than overly-simple best practices documents, a daisy chain of highly specialized but barely compatible tools, or vertically integrated platform stacks that can do everything but are great at nothing.

In the following articles in this series, we describe our vision for fostering a “shift left” culture in your organization, putting the power of developer focused tools, streamlined workflows and great developer experience to work for developers, so they can create, build and deploy production-ready containers easily. We’ll outline why container management best practices don’t work, propose a new working model for the create-build-test-deploy software delivery workflow in the cloud, and lay out how exactly Slim.AI aims to solve these challenges.

We welcome your comments, always, and hope you are excited as we are to build a better future for application development.

Coming soon: Part 2 | The Problem with Container Best Practices

Slim.AI is a developer-first company. We like to keep things simple and transparent, and want you to be a part of that. We’re always looking to connect with people passionate about developing apps the way we do: Cloud first, ready for scale, and as efficiently as possible. If you’re interested in working with us -whether as a tester, design partner, or as a colleague or code contributor- join our community at Slim.AI, reach out to us at community@slim.ai and get the app here. We look forward to developing apps with you real soon!

Our domains: Slim.AI, Dockersl.im

Find us on Github, Twitter, LinkedIn, and join our Discord community.