DevOps Automation Spectrum: Human-in-the-Loop

With AI taking over and the rapid pace of agile software development combined with the rising complexity of technology stacks has led to new challenges in balancing speed and reliability. While the promises of end-to-end test automation and infrastructure-as-code hold appeal, the reality is that fully automated systems often fail to account for nuances, edge cases, and unexpected failures- it is yet to be realistically feasible. 

This is where the concept of human-in-the-loop (HTL) automation enters the picture – combining the speed and reliability of automation with human oversight and dynamic decision making at critical junctures in the development lifecycle. But, this is easier said than done, so we decided to help you with comprehensive insights 

Understand HTL with this article by Talent500:

Origins of Human-in-the-Loop

The term human-in-the-loop automation originated in the field of aviation, where automated flight systems had a human pilot oversight to account for unexpected scenarios.

(Credits)

This concept has now expanded to other domains like self-driving vehicles, IoT systems, and more recently to DevOps – where the rapid pace of software delivery and complexity of systems requires a tight and symbiotic human-machine collaboration.

Definition of Human-in-the-Loop

Human-in-the-loop refers to keeping human operators and decision makers integrated into workflows, processes, and systems that are otherwise automated. 

The goal is to combine the repetitive accuracy, speed, and scalability of automation with uniquely human capabilities like contextual reasoning, adaptable problem solving, coordination, and oversight.

Challenges with Fully Automated DevOps

While theoretically appealing, fully automated DevOps suffers from some key limitations in practice:

• Inability to dynamically adapt responses beyond programmed logic in unexpected scenarios
• Lack of specialized domain expertise needed to make nuanced judgements and decisions
• No capability for organic coordination across teams, departments and systems 
• Lack of business context on goals, risks and tradeoffs that guides human decision making 
• Inability to redirect responses and override recommendations based on emerging insights
• Difficulty detecting and accounting for upstream or downstream impacts across the value stream

Promise of Human-in-the-Loop  

By incorporating human oversight and control into automated pipelines, environments, and processes – we get the best of both worlds:

• Repeatable precision, speed, reliability and scale of automated tasks
• Adaptability, experience, and intuitive judgment of human operators
• Contextual decision making aligned with business objectives
• Institutional knowledge captured as codified runbooks and playbooks
• Improved system resilience through symbiotic collaboration between humans and automation

The end goal is to have systems that are scalable, flexible, and continuously self-improving over time.

Fundamentals of Human-in-the-Loop

Let’s deep dive into the fundamental concepts that underpin human-in-the-loop thinking:

Spectrum of Automation

Human-in-the-loop spans a wide spectrum – from fully manual to fully automated:

• Manual > Runbooks > Orchestration > Automatic Remediation > End-to-End Automation

(Credits)

It keeps humans integrated at critical points along this spectrum rather than relying solely on the extremes of full human control or automation.

Unique Capabilities of Humans vs Machines

Humans bring several unique capabilities that machines cannot replicate fully:

• Ability to dynamically adapt responses and decision making by leveraging experience and intuition. This is key to handling unknown unknowns.
• Subject matter expertise to make complex and nuanced judgements considering qualitative factors. Humans have “tribal knowledge”.
• Capability to direct and coordinate responses across teams, departments and systems. Vital for complex socio-technical systems.
• Contextual awareness about business goals, technical debt, risks and tradeoffs that guides responses. Machines lack an enterprise view.
• Ability to be flexible – to override, redirect, amend responses in real-time based on emerging insights. Automation follows static logic.

Whereas, automation excels at:

• Consistent and tireless execution of repetitive tasks without lapses in focus or vigilance.
• Ability to operate at high speeds, volumes and scale beyond human capabilities. Provides throughput and capacity.
• Precision and accuracy when operating within known parameters and scenarios. Minimizes errors.
• Codifying tribal knowledge into runbooks and playbooks. Makes knowledge transferable. 

Transitioning from Working Know-How to Codified Processes

A key aspect of human-in-the-loop automation is codifying specialized knowledge into machine executable and trainable processes:

• Identify expertise currently residing in siloed human heads – capture “tribal knowledge”
• Document critical decision points and steps followed based on experience and past incidents
• Build flexible interactive runbooks and playbooks around this knowledge 
• Continuously improve runbooks by integrating human feedback and lessons learned

This allows scaling of expertise while keeping human oversight on ambiguous and risky decisions.

Data-Driven Approach to Continuous Improvement

Human-in-the-loop also allows capturing granular data on human actions, decisions, and context during incidents. This facilitates continuous improvement:

• Move from post-mortems focused on accountability to using data for real improvements 
• Capture annotations, communications, thought processes from human responders
• Leverage data to identify gaps in runbooks, playbooks and processes
• Use data to provide recommendations and power robust search for future incidents
• Apply techniques like machine learning to glean insights over time  

By taking an analytical, data-driven approach, we can make the processes continuously self-improving.

Human-in-the-Loop for Incident Management

Incident response is a great practical use case that can benefit tremendously from incorporating human-in-the-loop principles. 

Flaws in Traditional Incident Management

Traditional incident management suffers from some key limitations:

• Manual, tribal knowledge makes responses inefficient and inconsistent
• Lack of documentation and structured data capture makes it hard to learn and improve
• Reliance on post-mortems for improvement provides hindsight but limited foresight
• Communication scattered across tools like email, chat makes insights hard to piece together
• Focus limited to immediate remediation rather than long-term resilience  

Key Benefits of Human-in-the-Loop for Incidents

By incorporating human oversight with automation, we can transform incident response:

• Faster resolution by automating repetitive tasks while humans handle unknowns
• Lower cognitive load by eliminating manual toil for responders
• Continuous improvement of processes through real-time human feedback
• Better cross-team coordination facilitated by automation guardrails 
• Increased flexibility to override automation recommendations with human judgment 
• Enhanced searchability by capturing human insights as structured data

Integrating Human and Machine Data for Insights

Human-in-the-loop allows capturing and integrating detailed data during incidents:

• Human actions taken, communications, and contextual insights 
• System events, log messages, telemetry, and alerts
• Cross-referencing human and machine events to reconstruct timelines 
• Applying analytics to glean trends and improvement opportunities

This provides a much more detailed picture compared to sparse retrospective summaries.

Example Incident Walkthrough

Let’s see an example walkthrough of how human-in-the-loop could work in practice during an incident:

1. Alert generated automatically by anomaly detection system on latency spike in API 
2. On-call engineer notified over SMS and in incident Slack channel
3. Runbook launched with several potential failure scenarios and remediation options
4. Engineer reviews options and overrides runbook to request a code rollout based on recent changes
5. Rollback automated through deployment pipeline – engineer verifies in monitoring
6. Post-incident, details like Slack conversations, log extractions, and engineer’s actions are automatically collected  
7. This data is parsed to identify improvements – e.g. updating runbook with potential code rollback step 

By taking an integrated approach, we can make the entire loop – detection, response, learning – more resilient.

Implementing Human-in-the-Loop Automation

Here are some best practices and tips for implementing human-in-the-loop automation:

Strategic Identification of Automation vs Human Points

Identify upfront where human oversight is most beneficial vs. full automation. Common cases:

• Anomalies, high risk scenarios, integration touch points 
• Need for coordination across teams or departments
• Business critical or data sensitive decisions
• Qualitative assessments involving tradeoffs and nuance

Developing Flexible and Interactive Playbooks

Develop runbooks and playbooks that provide guardrails without being overly restrictive:

• Document steps followed based on experience and past incidents
• Incorporate potential failure modes and recovery options
• Allow continuous human amendments based on feedback
• Make it interactive – enable engineers to execute remediation through runbook

Designing for Optimal Human Experience

Design systems and interfaces with seamless human collaboration in mind:  

• Facilitate sharing of insights and collaboration 
• Reduce communication fragmentation across tools
• Make runbooks and automation recommendations easily accessible
• Contextualize machine data for human interpretation 

Automating Repetitive Tasks Fully

Identify tasks that can be fully automated to free up humans:

• Alerting and notification workflows
• Documentation and reporting 
• Well-defined remediation procedures
• Lower risk scenario handling

Capturing Human Insights as Structured Data

Develop mechanisms to capture human interactions, decisions, and context:

• Communication across tools like Slack, email 
• Runbook usage – steps executed, overrides, amendments  
• Post-incident debriefs and timelines
• Track record of human decisions over time 

Adoption Roadmap and Lessons Learned

Like any process change, incorporating human-in-the-loop requires thoughtful change management. Here are some recommendations:

Gradual Rollout Plan

• Start with lower risk processes or incident sub-types
• Slowly expand scope based on lessons learned  
• Demonstrate value incrementally through metrics like MTTR

Training and Change Management

• Provide training on concept of human oversight over automation
• Involve engineers early when developing runbooks to build comfort  
• Gather feedback regularly and incorporate into iterations 
• Showcase early wins and celebrate champions

Key Lessons from Industry Implementations

• Balance standardization with customizability of playbooks
• Minimize false positives through tighter human feedback loops
• Use gradual rollout to build institutional muscle memory
• Incentivize knowledge sharing and continuous improvement 

Mistakes to Avoid

• Full automation of processes better suited for human judgment 
• Restrictive playbooks that provide little flexibility
• Unclear handoffs between automation and human tasks
• Lack of mechanisms to capture and learn from human insights

Key Takeaways and Future Outlook

Let’s recap the core concepts and benefits:

Summary of Core Concepts

• Human-in-the-loop retains human oversight over automated systems
• Combines strengths of automation and human cognition 
• Transitions tribal knowledge into codified and trainable processes
• Focuses on continuous improvement through granular data capture

Key Benefits and Applications

• Increased system resilience, flexibility and reliability
• Faster and more consistent incident response
• Lower cognitive load for human operators
• Continuous improvement of processes through feedback
• Enhanced organizational learning and maturation

Final Recommendations

• Identify automation vs human hand-off points judiciously 
• Develop playbooks iteratively with engineer involvement
• Capture human insights and amendments systematically
• Rollout gradually while demonstrating value increments  

Future Possibilities

Some future possibilities as human-in-the-loop practices mature:

• Incorporating predictive analytics and machine learning to guide human decisions
• Leveraging VR/AR to provide richer operational context to responders 
• Building knowledge management ecosystems for continuous learning
• Expanding scope to business processes beyond just technical domains

By incorporating human-in-the-loop principles, we can create truly collaborative environments where automation enhances humans while humans instill contextual reasoning and empathy into machines. The future lies in this symbiotic integration into humanized automation.

Looking for a remote DevOps opportunity that gives the right challenges, a good work-life, and a lucrative TC?

Sign Up on Talent500 to make your next big career move!