About the Role
Splunk is seeking a senior individual contributor to serve as the technical lead for AI agents that enhance technical seller productivity. This role involves setting direction through thought leadership and influencing field teams across the full agent lifecycle, from design and build to production maintenance and optimization. The immediate mission is to deliver 3-5 agents to streamline technical sellers' workflows, increasing customer-facing time and supporting revenue and customer-success goals.
Responsibilities
- Advise on the end-to-end design for agentic systems across platforms, contributing to reference designs, patterns, and standards.
- Define, document, and evangelize best practices for cost, performance, design, and integrations across agent platforms.
- Guide platform selection for agent development, balancing capability, cost, integration, security, and maintainability.
- Set technical direction for contractors building agents, specifying requirements and reviewing their work.
- Serve as the recognized technical authority for agentic systems, mentoring engineers and driving alignment with stakeholders.
- Increase technical sellers’ customer-facing time and improve agent capabilities, reliability, safety, and cost-effectiveness to advance revenue and customer-success goals.
- Own the delivery of an initial 3-5 production agents that streamline technical sellers’ daily workflows.
- Complete the last-mile build of agents, integrating tools, MCP servers/clients, data sources, and hardening them for production.
- Translate reference designs into working agentic systems with built-in evaluation and guardrails, defining retrieval (RAG) and tool-integration strategies.
- Establish comprehensive logging, monitoring, and alerting for agents in production, leading root-cause analysis for failures.
- Track resource consumption, define and monitor SLOs/SLIs, manage cost-per-task economics, and drive model-routing and efficiency decisions.
- Define boundaries for agent safety, budget adherence, and compliance with security and regulatory standards.
- Maintain a subset of production agents, resolving regressions, keeping them current, and refining prompts, tools, topologies, and guardrails using telemetry and evals.
- Validate changes in simulation before release and automate manual toil.
Requirements
- 3+ years of software, ML, or platform engineering experience.
- Demonstrated track record of leading technical direction through influence.
- Proven ability to act as a technical authority and force multiplier without formal management authority.
- Hands-on experience designing and building production agentic systems, including multi-agent orchestration and MCP integrations.
- Experience selecting among and building across multiple agent platforms.
- Strong tool and API integration skills, including designing RAG strategies.
- Experience building evaluation harnesses and offline/online evals.
- Experience validating agentic systems in simulation before release.
- Experience implementing guardrails, policy and permission models, and audit trails for autonomous systems.
- Familiarity with token and cost management across multiple LLM providers, including model routing and fallback strategies.
- Experience directing and reviewing the work of contractors, vendors, or peer engineers through technical review.
- Hands-on experience operating agents in production, including observability and monitoring with Splunk and Splunk Observability.
- Experience with Splunk products (Splunk Cloud/Enterprise, Splunk IT Service Intelligence, Splunk Observability, or equivalent) for telemetry, monitoring, and observability.
- Proficiency in cloud-native architectures, preferably AWS.
- Experience in programming/scripting languages such as Python and Bash.
- Understanding of distributed systems, microservices architecture, and API design.
- Strong strategic thinking and ability to translate business objectives into technical roadmaps.
- Excellent communication skills with ability to present technical concepts to non-technical stakeholders.
- Track record of delivering results in fast-paced, dynamic environments.
- Data-driven decision-making approach with focus on measurable outcomes.
- Commitment to Excellence: Dedication to delivering high-quality solutions, strengthening technical skills, and taking pride in continuous improvement.
- Adaptability / Flexibility: Growth mindset, openness to feedback, and ability to respond effectively to new challenges and rapid change.
- Collaboration: Strong team player attitude, excellent communication skills, and commitment to empowering colleagues.
- Ownership: Proactive approach to improvement, accountability for outcomes, and drive to deliver on commitments.
- Innovation: Passion for creative problem-solving, technical rigor, and pushing boundaries.
- Fun: Ability to embrace fun, celebrate success, and contribute to a positive and energetic culture.
Skills
- Software Engineering
- ML Engineering
- Platform Engineering
- Agentic Engineering
- Agentic Operations
- Technical Leadership
- Thought Leadership
- Influence
- Reference Design
- Standards Development
- Best Practices Definition
- Platform Selection
- Contractor Direction
- Technical Review
- Mentoring
- Stakeholder Alignment
- Agent Delivery
- Last-Mile Build
- Integration
- Agent Design
- Orchestration
- Model Context Protocol (MCP)
- Retrieval-Augmented Generation (RAG)
- Tool Integration
- API Integration
- Evaluation Harnesses
- Offline/Online Evals
- Simulation Validation
- Guardrails Implementation
- Policy and Permission Models
- Audit Trails
- Autonomous Systems
- Token Management
- Cost Management
- LLM Providers
- Model Routing
- Fallback Strategies
- Production Operations
- Observability
- Monitoring
- Splunk
- Splunk Observability (Splunk o11y)
- Cloud-Native Architectures
- AWS
- Python
- Bash
- Distributed Systems
- Microservices Architecture
- API Design
- Strategic Thinking
- Business Acumen
- Communication Skills
- Data-Driven Decision-Making
Location
- Central United States
- Eastern United States
Work Type
- Remote
Experience Level
- Senior
- 3+ years of software, ML, or platform engineering experience
Salary/Compensations
- $135,800.00 - $198,800.00 (U.S. and/or Canada)
- $168,800.00 - $277,400.00 (New York City Metro Area)
- $148,800.00 - $248,200.00 (Non-Metro New York state & Washington state)
Benefits
- Medical insurance
- Dental insurance
- Vision insurance
- 401(k) plan with Cisco matching contribution
- Paid parental leave
- Short-term disability coverage
- Long-term disability coverage
- Basic life insurance
- Cisco restricted stock units
- 10 paid holidays per full calendar year
- 1 floating holiday for non-exempt employees
- 1 paid day off for employee’s birthday
- Paid year-end holiday shutdown
- 4 paid days off for personal wellness
- 16 days of paid vacation time per full calendar year (non-exempt employees)
- Flexible vacation time off program (exempt employees)
- 80 hours of sick time off provided on hire date and each January 1st thereafter
- Up to 80 hours of unused sick time carried forward
- Additional paid time away for critical or emergency family issues
- Optional 10 paid days per full calendar year to volunteer
- Annual bonuses (non-sales roles)
- Performance-based incentive pay (sales roles)
About the Company
- At Cisco, we’re revolutionizing how data and infrastructure connect and protect organizations in the AI era – and beyond.
- We’ve been innovating fearlessly for 40 years to create solutions that power how humans and technology work together across the physical and digital worlds.
- Our solutions provide customers with unparalleled security, visibility, and insights across the entire digital footprint.
- Fueled by the depth and breadth of our technology, we experiment and create meaningful solutions.
- We work as a team, collaborating with empathy to make really big things happen on a global scale.
- Our impact is everywhere because our solutions are everywhere.
- We are Cisco, and our power starts with you.
