Senior AI Engineer - APM Experiences
The opportunity Datadog’s APM Experiences team owns the core product experience for Application Performance Monitoring — including distributed tracing, service representation, and more. We’re building a new wave of AI-powered capabilities that help customers detect, resolve, and prevent performance issues faster.
What this role actually needs.
The opportunity Datadog’s APM Experiences team owns the core product experience for Application Performance Monitoring — including distributed tracing, service representation, and more. We’re building a new wave of AI-powered capabilities that help customers detect, resolve, and prevent performance issues faster. Responsibilities: - Shape AI experiences for APM. Design and ship LLM/agentic workflows that analyze traces, metrics, logs, and other telemetry to generate diagnoses, explanations, and guided fixes. - Own the full loop. Prototype quickly, define success metrics and evals, run experiments, iterate, and ultimately productionize for scale and reliability. - Build robust agent systems. Develop tools, retrieval and planning strategies, and guardrails; manage prompts/evals; design fallbacks and human‑in‑the‑loop paths. - Integrate with Datadog’s platform. Leverage surfaces like Trace Explorer, Service Catalog, monitors, and workflows to deliver end‑to‑end value in the APM UI. - Partner deeply. Collaborate with PM, Design, and partner teams to build cohesive experiences. - Raise the bar on engineering. Write performant, maintainable backend code, own services in production, and improve reliability for high‑throughput, low‑latency data systems. Requirements: - 4+ years building backend or real-time ML systems; you value simplicity, correctness, and performance - Proven experience delivering LLM/agent features to production (prompting, tooling, evals, safety/guardrails) - Comfortable owning user journeys, iterating from prototype → alpha → GA, and measuring impact with clear product metrics - You have demonstrated ability to use AI coding tools in day-to-day workflows and validate, critique, and refine AI-generated output - You’re motivated to push the boundaries of how AI can improve software engineering best practices and contribute to building AI-enabled products - Solid grasp of the ML lifecycle (task definition, dataset collection, modeling, evaluation, deployment, iteration) and statistics (experiment design, confidence intervals) Benefits: - Get to build tools for software engineers, just like yourself. And use the tools we build to accelerate our development. - Have a lot of influence on product direction and impact on the business. - Work with skilled, knowledgeable, and kind teammates who are happy to teach and learn. - Competitive global benefits. - Continuous professional development. Company context: Datadog operates a cloud observability and security platform with meaningful hiring across infrastructure, security, AI, and developer tooling.
Day-to-day expectations
Datadog lists these responsibilities for the Senior AI Engineer - APM Experiences role.
- Shape AI experiences for APM. Design and ship LLM/agentic workflows that analyze traces, metrics, logs, and other telemetry to generate diagnoses, explanations, and guided fixes.
- Own the full loop. Prototype quickly, define success metrics and evals, run experiments, iterate, and ultimately productionize for scale and reliability.
- Build robust agent systems. Develop tools, retrieval and planning strategies, and guardrails; manage prompts/evals; design fallbacks and human‑in‑the‑loop paths.
- Integrate with Datadog’s platform. Leverage surfaces like Trace Explorer, Service Catalog, monitors, and workflows to deliver end‑to‑end value in the APM UI.
- Partner deeply. Collaborate with PM, Design, and partner teams to build cohesive experiences.
- Raise the bar on engineering. Write performant, maintainable backend code, own services in production, and improve reliability for high‑throughput, low‑latency data systems.
What a strong candidate brings
These requirements are extracted from the source listing and normalized for UpJobz readers.
- 4+ years building backend or real-time ML systems; you value simplicity, correctness, and performance
- Proven experience delivering LLM/agent features to production (prompting, tooling, evals, safety/guardrails)
- Comfortable owning user journeys, iterating from prototype → alpha → GA, and measuring impact with clear product metrics
- You have demonstrated ability to use AI coding tools in day-to-day workflows and validate, critique, and refine AI-generated output
- You’re motivated to push the boundaries of how AI can improve software engineering best practices and contribute to building AI-enabled products
- Solid grasp of the ML lifecycle (task definition, dataset collection, modeling, evaluation, deployment, iteration) and statistics (experiment design, confidence intervals)
Why people would want this job
Datadog published these compensation, benefits, or working-context details with the role.
- Get to build tools for software engineers, just like yourself. And use the tools we build to accelerate our development.
- Have a lot of influence on product direction and impact on the business.
- Work with skilled, knowledgeable, and kind teammates who are happy to teach and learn.
- Competitive global benefits.
- Continuous professional development.
Why this listing is more than a copied job post.
Senior AI Engineer - APM Experiences is framed against UpJobz source checks, country scope, compensation visibility, and work-authorization signals so candidates can make a faster go/no-go decision.
United States tech market
United States roles on UpJobz are filtered for high-tech relevance, source freshness, and actionable employer detail before they are allowed into SEO surfaces.
Compensation read
The employer source does not expose a reliable salary range, so candidates should ask for compensation early instead of waiting until late-stage interviews.
Work authorization read
Current extracted signal: United States residents. UpJobz treats this as a search signal, not legal advice, and links visa-sensitive roles back to the relevant visa hub where possible.
Location read
On-site roles in New York City should be compared against commute, local salary bands, and nearby employer demand.
Browse similar jobs
Turn this listing into an application plan.
This is the first pass at the premium UpJobz layer: a fast brief that helps serious applicants move with more clarity.
Next moves
- Tailor your resume around ai and llm instead of sending a generic application.
- Use the first two bullets of your application to connect your background directly to senior ai engineer - apm experiences is a high-signal on-site role in new york city, and it is most realistic for united states residents.
- Open the role quickly if it fits and bookmark three similar jobs before you leave the page.
Interview themes
Watchouts
- Compensation is hidden, so get range clarity in the first recruiter conversation.
- Use united states residents as part of your positioning so the recruiter does not have to infer it.
- Show concrete examples of succeeding in on-site environments.
Keywords to match against your background
Use these terms to decide whether your resume, portfolio, and recent projects line up with the role.
Apply through the employer source
Open the source listing from careers.datadoghq.com, confirm the role is still active, then apply on the employer or ATS page.
Source: careers.datadoghq.com · Source ID: 7415266 · Confidence: 93/100 · Last checked: May 7, 2026
How UpJobz verifies job sourcesContinue browsing tech jobs