Distributed Systems & Platform Engineer

Job Purpose

To design, build, and operate the distributed computing and platform infrastructure that powers the organization’s AI, machine learning, generative AI, and data-intensive workloads.

The role will be responsible for developing highly scalable, reliable, secure and high-performance platforms for distributed model training, inference, experimentation and production AI services. The position will work across Kubernetes, cloud and/or bare-metal infrastructure, GPU compute, distributed systems, storage, networking, orchestration and platform automation.

The role combines deep software engineering with infrastructure and platform engineering, with a strong focus on creating self-service platforms that enable AI/ML engineering teams to deploy and operate workloads efficiently at scale. Current AI infrastructure roles increasingly emphasize Kubernetes control-plane expertise, GPU scheduling, distributed training/inference, multi-tenancy, observability and infrastructure-as-code.

Job Responsibilities

Distributed Systems Engineering

• Design and develop highly available, fault-tolerant and horizontally scalable distributed systems.
• Build services capable of operating reliably across multiple nodes, clusters, availability zones and cloud environments.
• Design mechanisms for distributed coordination, state management, caching, queuing, scheduling and workload orchestration.
• Develop systems capable of graceful degradation, automated recovery and failure isolation.
• Identify and resolve performance bottlenecks across compute, storage, networking and application layers.
• Design APIs, services and control planes that abstract complex infrastructure capabilities for internal users.
• Apply appropriate distributed-systems patterns including replication, sharding, partitioning, consensus, idempotency and eventual consistency.

AI/ML Infrastructure Platform

• Build and maintain the foundational platform supporting AI/ML model training, fine-tuning, experimentation and inference.
• Develop reusable infrastructure and services for AI workloads across development, testing and production environments.
• Enable distributed training and inference across GPU clusters.
• Develop mechanisms for job submission, scheduling, prioritisation, retries, checkpointing, logging and artifact management.
• Support model-serving platforms with scalable routing, autoscaling, deployment, versioning, canary releases and rollback capabilities.
• Build self-service capabilities that allow AI/ML engineers to provision and manage compute resources without extensive infrastructure intervention.
• Integrate AI workloads with model registries, experiment tracking, data platforms and CI/CD pipelines.

Kubernetes & Container Platforms

• Design, deploy and operate production-grade Kubernetes platforms for AI workloads.
• Develop Kubernetes operators, controllers, Custom Resource Definitions (CRDs) and automation.
• Understand and optimise Kubernetes control-plane components including API server, scheduler, controller manager, kubelet and etcd.
• Design multi-cluster and multi-tenant Kubernetes architectures.
• Implement workload scheduling, autoscaling, resource quotas and workload isolation.
• Manage cluster lifecycle, upgrades, configuration, capacity and resilience.
• Optimise Kubernetes for GPU-intensive and distributed workloads.
• Implement containerization standards and platform engineering best practices.

GPU & High-Performance Computing Infrastructure

• Design and operate GPU-enabled infrastructure supporting AI training and inference.
• Implement efficient GPU allocation, scheduling, sharing and utilisation strategies.
• Work with GPU technologies such as NVIDIA CUDA, GPU Operator, MIG, MPS and NCCL where applicable.
• Support GPU-aware scheduling, topology-aware workload placement and distributed GPU workloads.
• Optimise GPU utilisation, throughput, latency and cost.
• Work with cluster schedulers such as Kubernetes, Slurm, Ray, Kueue or Volcano where appropriate.
• Diagnose GPU, compute, memory, communication and networking bottlenecks.
• Support high-performance interconnects and technologies such as RDMA, InfiniBand and RoCE where applicable.

Platform Engineering & Developer Experience

• Build internal developer platforms, APIs, CLIs, SDKs and automation tools.
• Establish "golden paths" for AI/ML engineers to deploy, test and operate workloads.
• Provide reusable infrastructure components and platform services.
• Reduce infrastructure complexity and manual operational effort through automation.
• Develop self-service provisioning and deployment capabilities.
• Create platform documentation, technical standards and reusable engineering patterns.
• Treat internal engineering teams as platform customers and continuously improve platform usability and adoption.

Infrastructure as Code & Automation

• Develop and maintain Infrastructure as Code using Terraform, Pulumi or equivalent technologies.
• Automate infrastructure provisioning, configuration, deployment and lifecycle management.
• Implement GitOps practices for infrastructure and application deployment.
• Develop automated remediation and self-healing capabilities.
• Automate cluster provisioning, node management, GPU configuration and workload deployment.
• Build repeatable infrastructure patterns across cloud, on-premise and hybrid environments.

Cloud & Infrastructure Architecture

• Design scalable infrastructure across AWS, Microsoft Azure, Google Cloud and/or private infrastructure.
• Develop hybrid and multi-cloud infrastructure patterns where required.
• Design compute, storage and networking architectures optimised for AI workloads.
• Implement capacity planning and resource allocation strategies.
• Support cloud bursting and dynamic scaling for high-demand AI workloads.
• Optimise infrastructure cost without compromising performance, security or reliability.

Storage & Data Infrastructure

• Design high-throughput storage solutions for AI training datasets, model artefacts, checkpoints and logs.
• Work with object, block and distributed file storage systems.
• Implement caching and data-locality strategies to reduce training and inference bottlenecks.
• Support high-performance distributed storage environments.
• Implement appropriate data lifecycle, retention and quota-management mechanisms.
• Ensure storage systems meet required performance, availability and resilience objectives.

Networking

• Design and optimise networking for distributed AI workloads.
• Work with Kubernetes networking, CNI, service discovery, load balancing and ingress.
• Support high-throughput, low-latency communication between distributed compute nodes.
• Troubleshoot network performance and reliability issues.
• Support RDMA-enabled networking where required for distributed GPU workloads.
• Implement appropriate network segmentation, security policies and multi-tenant isolation.

Reliability, Observability & Performance

• Define and implement SLIs, SLOs and operational metrics for platform services.
• Build comprehensive monitoring, logging and distributed tracing.
• Monitor CPU, memory, GPU, storage, network and application-level performance.
• Develop dashboards, alerting and automated incident detection.
• Conduct performance benchmarking, capacity testing and scalability assessments.
• Participate in incident response and root-cause analysis.
• Build systems for fault detection, automated recovery and resilience testing.

Security & Multi-Tenancy

• Design secure, multi-tenant AI infrastructure platforms.
• Implement RBAC, namespaces, quotas, secrets management and workload isolation.
• Apply security controls across Kubernetes, cloud infrastructure, containers and APIs.
• Support identity and access management across platform services.
• Implement audit logging and appropriate security monitoring.
• Ensure infrastructure complies with organisational security and data-protection requirements.

Engineering Leadership & Collaboration

• Work closely with AI/ML engineers, data scientists, software engineers, DevOps/SRE, cybersecurity and enterprise architects.
• Translate AI/ML requirements into scalable infrastructure and platform solutions.
• Lead technical design discussions and architecture reviews.
• Conduct code reviews and promote engineering quality standards.
• Mentor engineers and contribute to technical capability development.
• Evaluate emerging infrastructure and AI technologies and recommend appropriate adoption.
• Contribute to engineering standards, technical roadmaps and platform strategy.

Key Deliverables / Expected Outcomes

The role holder will be expected to:

• Deliver a scalable and reliable AI infrastructure platform.
• Improve GPU and compute-resource utilisation.
• Reduce deployment and provisioning times for AI workloads.
• Enable self-service infrastructure capabilities for AI/ML engineering teams.
• Improve reliability and availability of distributed AI workloads.
• Reduce infrastructure-related operational toil through automation.
• Improve performance and efficiency of distributed training and inference.
• Establish effective monitoring, observability and incident-management capabilities.
• Maintain secure and appropriately isolated multi-tenant environments.
• Support rapid and reliable deployment of AI models into production.

Job Qualifications

• Bachelor's degree in Computer Science, Software Engineering, Computer Engineering, Information Technology or a related discipline.
• A Master's degree in Computer Science, Distributed Systems, AI/ML, Cloud Computing or a related field would be advantageous.
• Relevant professional certifications in cloud, Kubernetes or infrastructure engineering are desirable.

Professional Experience

• 3-5 years of software engineering, distributed systems, infrastructure, cloud or platform engineering experience, depending on seniority.
• Demonstrable experience designing and operating production-scale distributed systems.
• Strong production experience with Kubernetes and containerised environments.
• Experience building infrastructure platforms for AI/ML workloads is highly desirable.
• Experience with GPU infrastructure, distributed training or model serving is strongly preferred.
• Experience with cloud and/or large-scale bare-metal infrastructure.
• Experience developing production software in C++, Rust or equivalent systems-oriented languages.
• Experience with Infrastructure as Code and CI/CD automation.
• Experience troubleshooting complex production infrastructure across compute, storage and networking.