Generative AI for DevOps with Kubiya

The Software Development Life Cycle has evolved dramatically over recent years, with DevOps emerging as a critical element. It’s estimated to reach a market size of $25 billion by 2028, and a recent survey suggests that it commands 80% of engineering budgets, according to Grove Ventures.

Yet, despite its significance, the practice is still considerably complex. Balancing speed with quality, orchestrating multi-layered deployments, and maintaining rigorous security standards are just a few of the hurdles encountered. Amidst these complexities, Generative AI is making a profound impact.

Within this dynamic landscape, Kubiya, an Israeli startup, has swiftly capitalized on the potential of Generative AI for DevOps. Founded in 2022 by Amit Eyal Govrin and Shaked Askayo, backed by VCs in Israel and Silicon Valley and with millions in their coffers, Kubiya developed an Agent Based, Large Language Model (LLM) powered DevOps assistant, 'Kubi', offering a comprehensive suite of engineering tools, enabling tasks such as IaC provisioning, triggering CI/CD jobs, managing Kubernetes namespace, and Jira ticket handling with unprecedented ease.

In my ongoing series of interviews with pioneers in Generative AI, I recently sat down with Mr. Govrin. We discussed their groundbreaking work with 'Agents', his early bets on LLMs in the context of Conversational AI and their ambitions to push the envelope further into the realm of autonomous agents for operations.

How has Generative AI helped in the implementation of DevOps?

"Generative AI gave way to a more efficient exchange between users and their engineering tools. The traditional one-sided human to machine interaction, which lacked any form of feedback and required substantial user effort, has evolved into a reciprocal communication medium. It allows for more intuitive and interactive experiences, significantly enhancing the way humans interact with machines, and more importantly, significantly changes the discussion around the time it takes to automate end-to-end DevOps tasks."

"At Kubiya we built a full-stack LLM solution layered with a suite of purpose-built, agent-actors to address every aspect of the enterprise use-cases. A user may interact with our agents on a general purpose question around their knowledge base or docs. In this instance Kubiya fine-tunes the embedding and uses RAG to infer answers to the questions complete with a summary and links, all while maintaining complete control of the access and permission levels of the user. This will typically give them an appetite to challenge the system with more advanced use cases such as granting permission to create a new development stack using Terraform for instance or create an IAM policy and attaching it to an S3 bucket. Finally a user will feel the confidence to ask the system to create a complex chain of actions using multiple tools, data sources and policies."

"We have seen several of our customers create automated tasks that sweep their Jira 'to-do' queue on a scheduled basis, try to solve a myriad of complex use-cases including Kubernetes troubleshooting, elevated permissions, GitHub PR merge and Terraform code, and then notate each ticket once finished so that it’s solved and moved to 'completed'. Some have even gone as far as having the agent assign follow-on steps to admins and notify the requesters on a channel once complete."

"It’s hard to give this use-case a name as it covers so much ground, but it’s an end-to-be automation of DevOps tasks that otherwise would have required multiple tools, knowledge sources and tasks not to mention involve multiple humans in the loop. This is the genesis of AI agents - offloading work from humans to free them to focus on their best skills."

Clarify the magnitude of the progression from using Conversational AI to Generative AI. How dramatic was that?

"Older Chatbot frameworks like Rasa and Botpress were intent-based pre-transformer versions of Conversational AI. They were fraught with challenges, weak in predicting user intentions and they required extensive training. An entire failed ecosystem was built around this, where the general consensus of the approach was that it delivered poor user experience and was far too costly to maintain."

"With the evolution of Generative AI, for the first time, LLMs are coming with near out-of-box intelligence, armed with advanced inferencing technologies and having the skills to chain together thought and reasoning that closely mimics human behavior. As for data and reducing hallucinations, by virtue of fine-tuning, these models that we are seeing are much more accurate, built from smaller data sets in a fraction of the time, and have the ability to orchestrate and provision the right agents, leveraging the right model for the right task, which makes Kubiya incredibly powerful at mimicking knowledge worker behavior on end-to-end task automation."

Which LLMs are you using? And how has that changed since you started using them?

"We started off in the early beta programs of GPT-3. Now, we're using a range of models between Hugging Face, Mistral, Cohere, Anthropic, and Code Llama. We're actually using the privately hosted versions of Azure OpenAI GPT and serving some of the open-source models ourselves. On top of that, we are continuously fine-tuning our models based on user input and enhancing those with synthetic data pipelines."

"To be honest, the industry is evolving so rapidly that there aren’t many cutting-edge techniques that we haven’t already experimented with as we realize that velocity is a competitive advantage and we are always looking for that next edge."

“One of the areas that we are most proud of is our ability to decouple the user experience from the LLM itself. We have developed the ability for customers to run everything locally including hosting their own models, vector databases, and every aspect of secrets management in order to maintain privacy and data sovereignty."

What are some of the latest features you've been able to add thanks to upgrades?

“The agent-actor framework model has been our most significant breakthrough. It provides users with an easy-on button experience, similar to spinning up OpenAI GPTs, only that it works on the customer hosted infrastructure and comes purpose-built with templates, flows, integrations, Terraform provider, and authentication for DevOps and platform engineering. To that end, engineers can bring in their own tools, scripts, integrations and interact with the system naturally and almost instantaneously, with very little configuration needed. You could actually be up and running and talking to your Kubernetes cluster within a few minutes using Kubiya’s Slack app, Kubi."

"We are introducing the concept of creating policies using natural language to grant access and authorize requests. Likewise, natural language is being used with Kubiya agents to create terraform modules, merge PR requests, troubleshoot unhealthy clusters and pretty much anything that a DevOps engineer would encounter on a day-to-day basis, without needing to rely on another human. This has the potential to completely change the coverage ratio between operators to developers in the domain of platform engineering, and needless to say, the productivity boost goes directly to the bottom line of the organization."

Can you elaborate about user engagements. How's that been changing as you've added more and more features and become more advanced?

"Most of our feature releases were engineered based on user feedback. We have seen customers start with siloed implementations of their operations teams around policy and CI/CD management and roll it out to engineering teams for secure infrastructure provisioning and just in time elevated permissions and access. Eventually they roll it out to the entire organization for a tier 1 and tier 2 support desk."

"Just to back this up with numbers, since launching earlier this year, we have seen upwards of 20-25 registrations per day, where some of the users are experimenting with side projects, while others are signing up for full scale POCs. Our ideal state is working with teams who have attempted a home grown solution and realize just how complex of an undertaking it can be so that by the time they speak to us, they are in awe at how advanced of a system we had built and how easy it is to get started."

In the DevOps and Generative AI for DevOps market, how does Kubiya stand apart from competitors, and how do you distinguish yourself?

"We’ve connected together two distinct worlds: one is a world of platform engineering where internal developer portals are the tools of choice, and the other is the world of advanced virtual assistants. We've layered in Generative AI throughout the entire tech stack, and rather than viewing ourselves as another tool in the DevOps toolchain, we are completely changing the way users interact with their tools by automating their day-to-day tasks via a simple chat conversation. Mind you, this has been our thesis all along and it's a bet that we took prior to the debut of ChatGPT; that the future of process automation will be dominated by natural language and not by code or low-code apps. Language is the new operating system."

"We do see competitors on one-off use-cases where one of our agents will replace the need for a point solution, however up to this point we have not seen anyone else come forward with a one-to-one platform of agent-actors that can do the breadth and depth of process automation in the world of platform engineering the way Kubiya does. I say this confidently because we’ve had the first mover advantage and the determination to keep developing towards the future state of autonomous agents as opposed to being stuck in the more near-term world of workflow automation."

"The productivity tools of the future will look a lot less like developer tools and a lot more like agents that act as digital teammates that are able to augment a good portion of the day-to-day tasks and undifferentiated heavy lifting of knowledge workers."

If the aim of what you're doing is to improve productivity of developers, can you put any kind of measure on that?

"Efficiency and time-saving can be measured by DORA metrics, with developers productivity being the central focus, but in my opinion, this largely misses the mark. It is still a fairly naive way of measuring total attribution to organizational productivity gains since so much of the value is lost in the process-laden exchange between humans working in silos. Just as an example, one of our customers was able to reduce their SLA to developers down from an average of three days down to a mere one hour, by virtue of automating their entire Jira ticket process. This is not a random anecdote, it’s literally a paradigm shift in how this organization now ships their code."

What happens to the developer in future and if they're using things like this does this mean there's going to be less of them? How do you see the developer role changing?

"Just like GitHub’s Copilot is a force multiplier for developer productivity when it comes to human-paired code generation, Kubiya brings a fresh look to human-paired operations. The developers who recognize the opportunity to level up their skills and embrace these tools the most will be even more valuable to their employer while those who see it as a threat to their jobs and don’t adjust will likely be correct."

"The knowledge worker displacement aspect of it will typically take the form of a lower level worker that does a lot of repetitive tasks that can be abstracted away into agent based automation. The key takeaway is that for workers to stay relevant in the workforce of the future they need to be adaptable to the changing times and become more specialized in what they do. This is the Industrial Revolution all over again in relation to assembly lines. This technological advancement didn't displace human labor, it simply forced the assembly worker to specialize in a specific area of the assembly line. Rather than having one person doing everything, now you have domain experts in each area.

"Pro tip: one of the most highly in demand careers of the future will be prompt engineering. These LLM ‘whisperers’ will control the keys to organizational efficiency moving forward."

Looking ahead, what’s in store for the immediate future, and the long term roadmap of Kubiya?

“We see a huge opportunity to expand the internal developer platforms to even more specialized areas, like SecOps, DevSecOps, and FinOps."

"I think there will be a day, not too far into the future, where everyone will be using autonomous agents to do a lot of their day-to-day work. With that NorthStar in mind, at Kubiya we are in position to become one of the world's first autonomous DevOps operators where your next DevOps hire will begin with a subscription."