Monitoring And Evaluation Specialist

About the Role We are seeking a skilled GenAI / LLM Observability & Evaluation Engineer to help ensure the reliability, performance, and quality of production AI agents and LLM-powered applications. The ideal candidate will have hands-on experience implementing LLM evaluation frameworks, distributed tracing, telemetry, monitoring dashboards, automated testing, and production reliability practices. Required Skills • Hands-on experience with LLMs, Generative AI, or Agentic AI applications. • Experience with LLM evaluation, agent testing, and performance analysis. • Strong Python programming and test automation skills. • Experience implementing observability, tracing, telemetry, and monitoring solutions. • Experience building metrics dashboards and analyzing application performance. • Understanding of SLOs, production operations, incident response, and operational readiness. Preferred Skills • LLM evaluation tools such as DeepEval, Ragas, Promptfoo, or equivalent. • AI observability platforms such as LangSmith, Langfuse, Arize Phoenix, or equivalent. • OpenTelemetry, OpenInference, Prometheus, Grafana, Datadog, or Splunk. • Experience with LangChain, LangGraph, Google ADK, or similar agent frameworks. • Familiarity with CI/CD pipelines, automated regression testing, and release quality gates.