QA automation engineer

Role Overview

You will be the Quality Intelligence Architect of Exterview.

As a QA Automation Engineer (10+ Years) , you’ll design, implement, and own the automation and quality orchestration system that validates AI-driven candidate screening, live interviews, scoring agents, and feedback pipelines.

This is quality engineering at scale : validating probabilistic AI outputs, real-time interview flows, event-driven systems, and media-heavy workflows across thousands of concurrent interviews.

You’ll work closely with Backend, AI Engineering, Prompt Engineering, and Frontend teams to ensure quality is built into the system , not tested after the fact.

Execution is tracked via Linear (tasks), Notion (PRDs & test strategy), and GitHub (automation reviews) ensuring quality decisions are transparent and measurable.

Key Responsibilities

Automation Architecture

• Design and own a scalable, maintainable automation framework for:
• Web (Next.js)
• APIs (GraphQL via AppSync, REST)
• Event-driven and async workflows
• Move QA from test execution to quality architecture ownership.

AI & Probabilistic System Validation

• Validate:
• Resume parsing accuracy
• AI scoring consistency
• Interview logic stability
• Define assertion strategies for non-deterministic AI outputs.

Real-Time & Media Workflow Testing

• Test live interview systems :
• VideoSDK + Tavus video flows
• Real-time state updates
• Validate video upload, playback, retries, and failure handling.

API, Event & Integration Testing

• Build API-first automation for:
• GraphQL queries & mutations
• Lambda & microservice APIs
• Validate event-driven flows (notifications, interview state transitions).

Performance, Load & Reliability Testing

• Design load and stress tests for:
• Interview orchestration APIs
• Video playback & report generation
• Ensure system reliability under peak concurrency.

CI/CD & Quality Gates

• Integrate automation into GitHub Actions.
• Define release quality gates and production sign-off criteria.
• Track flaky tests, failure patterns, and quality metrics.

Cross-Team Quality Leadership

• Translate product requirements into test strategies.
• Partner with engineering to improve testability & observability.
• Mentor junior and mid-level QA engineers.

Success Metrics

First 90 Days

• Stable automation framework covering UI + API + AI workflows.
• Flaky test rate reduced to near zero.
• API-level load tests live for interview & video flows.
• Release quality gates enforced in CI/CD.

12 Months

• End-to-end automation coverage for all interview types.
• Predictable, fast release cycles with no QA bottlenecks.
• AI validation framework adopted as product standard.
• QA recognized as product quality owner , not gatekeeper.

Must-Haves

• 10+ years in QA Automation / SDET roles
• Strong experience designing automation frameworks , not just writing scripts
• Expertise in API testing (GraphQL & REST)
• Experience testing real-time and async systems
• Ability to validate AI-driven, non-deterministic outputs
• Strong understanding of CI/CD, release quality, and production readiness

Nice-to-Haves

• Experience testing AI/ML or agent-based systems
• Media or video workflow testing experience
• Startup or high-scale SaaS background
• Prior ownership of QA strategy for a product

Tech Stack Visibility

• Frontend: Next.js, TypeScript
• Automation: Cypress, Playwright, Jest
• Backend: Node.js, AWS Lambda, Serverless Framework
• APIs: GraphQL (AWS AppSync), REST
• Auth: AWS Cognito (OAuth, OTP)
• Data: MongoDB Atlas
• Media: VideoSDK, Tavus, Amazon S3
• Communication: Twilio (SMS, WhatsApp, IVR)
• Observability: Langfuse, internal monitoring

Assessment (PoC)

Objective

Validate your ability to design quality systems , not just write test cases.

Challenge Overview

Design and implement an automation strategy for an AI-driven interview flow.

The system should validate:

• Resume upload & parsing
• AI resume–job match scoring
• Interview scheduling
• Live interview execution
• Interview report generation

Functional Expectations

1\. Automation Design

• Cover:
• One UI flow
• One GraphQL API flow
• One real-time or async workflow
• Show clear separation between test layers.

2\. AI Validation Strategy

• Define how you:
• Assert AI scores
• Detect drift or instability
• Handle acceptable variance

3\. Performance & Reliability

• Include:
• One API load test
• Failure / retry validation
• Avoid direct S3 URL testing; use APIs only.

Deliverables

• Automation repo (clean structure)
• Test strategy README
• Example AI assertion logic
• CI pipeline config (GitHub Actions)
• ≤ 5 min Loom walkthrough explaining decisions