Job Title: Prompt / LLM Ops / AI Systems Engineer

Role Summary

We are seeking a versatile engineer to bridge the gap between creative prompt design and robust system

orchestration. You will own the design, evaluation, and production operation of our foundation model-based

systems. As an LLM Ops Specialist, you will be responsible for building the infrastructure that makes these systems

scalable, optimized and observable.

Core Responsibilities

System Architecture amp; Orchestration:

• Design end-to-end compound AI systems integrating multiple models, retrieval systems, and agentic
  
  

workflows.

• Implement Retrieval-Augmented Generation (RAG) pipelines with optimized chunking, vector
  
  

embeddings, and re-ranking strategies.

Prompt Engineering amp; Versioning:

• Craft structured, few-shot, and chain-of-thought prompts to reduce hallucinations and ensure predictable
  
  

outputs.

• Develop and manage a Prompt Registry to track versioning and A/B test results.
  
  

LLM Operations (LLM Ops)

• Observability: Instrument real-time monitoring for LLM-specific metrics, such as:
• Time to First Token (TTFT)
• Token throughput
• Drift detection
• Cost Control: Implement semantic caching and intelligent model routing to reduce API spend.
• CI/CD for AI: Build evaluation suites and methodologies to test for regressions on every deployment.
  
  

Core Tech Skills

• Python Proficiency
• Hands-on with Programmatic Prompting Frameworks
• Experience with Retrieval-Augmented Generation (RAG) Pipelines
• Experience with Structured Data Control and Integrations into other Software Systems
  
  

Mandatory Soft Skills

• Excellent Written and Spoken Communication Skills, to the Degree of Clinical Precision
• Excellent Tech-to-Business Translations
• Analytical Thinking pertaining to Problem Decomposition and Adversarial Thinking
• Collaborative Agility - Must have ability and willingness for Cross-Team Functionality and Faster Feedback
  
  

Integration Loop

• Constant Experimentation Mindset and High Degree of Learning Agility
  
  

This job is provided by Shine.com