Dev Ops Engineer

Caltech is a world-renowned science and engineering institute that marshals some of the world's brightest minds and most innovative tools to address fundamental scientific questions. We thrive on finding and cultivating talented people who are passionate about what they do. Join us and be a part of the diverse Caltech community.

Job Summary

We are seeking a DevOps Engineer with experience managing large-scale, on-premises infrastructure deployments, to work on the Deep Synoptic Array (DSA), a world-leading radio telescope that is expected to commence construction in 2027. The array will use 1,650, 6.15-meter dishes to observe at radio wavelengths (0.7 - 2 GHz frequency range) to survey the skies 10x faster than any radio telescope current or planned. The telescope will conduct a broad range of research, including the study of the formation of galaxies, the search for cosmic explosions, and investigations into the nature of gravity.

The applicant would fulfill the role of Associate DevOps Engineer for the DSA project, working closely with and supporting the lead DevOps Engineer on high-impact infrastructure initiatives. This role will support the DSA Software Group as we progress through the final design stage to construction.

The role can be based at either the Caltech campus, the Owens Valley Radio Observatory in Big Pine, CA, Ely, NV, or hybrid/remote.

Essential Job Duties

The candidate will work closely with the DSA-2000 team in support of the following main duties:

• On-Premises Infrastructure Management: On-Premises Infrastructure Management: Co-lead the design, deployment, and management of the on-premises infrastructure for the DSA's processing and monitoring pipelines. This will involve provisioning and configuring hundreds to thousands of hosts with a variety of capabilities, including high performance GPU computing - whether through thousands of commercial GPUs or a handful of Nvidia NVL72 racks - and specialized signal processing hardware.
• Automation: Develop and implement automated processes for server provisioning, configuration management, and software testing, using tools like Ansible, Puppet, and Chef for configuration management alongside platforms like Foreman or MaaS.
• Clustering: Establish, maintain, and administer software deployment clusters like Proxmox, Kubernetes, and Slurm to provide high-level deployment targets for DSA software infrastructure.
• Networking: Maintain and optimize the configuration of routers and switches to support both the control traffic and high-bandwidth signal traffic of the DSA telescope.
• Packaging: Package DSA software for deployment to a variety of targets, including our production hardware and various testing platforms, using build tooling like UV, Cargo, and CMake alongside packaging and containerization solutions like Nix and Docker.
• CI Pipelines: Design and implement robust CI pipelines for the DSA software to ensure rapid feedback and consistent testing practices for proposed software changes.
• Deployment Orchestration: Develop and maintain deployment configurations for the DSA software packages, including Kubernetes manifests, Slurm jobs, and Ansible playbooks to ensure efficient deployments of updated DSA software.
• Monitoring and Logging: Establish and maintain comprehensive monitoring and logging systems to track infrastructure health and application performance.
• Security and Compliance: Support efforts to ensure compliance with relevant security standards and regulations.
• Collaboration: Collaborate closely with the Software Development team to understand their needs and provide ongoing support.

Basic Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience.
• 5+ years of professional experience in a DevOps, systems administration, or software infrastructure role.
• Extensive experience with Linux server administration.
• Deep experience with infrastructure as code (IaC) tools such as Terraform, Ansible, Puppet, or Chef.
• Proven experience with containerization technologies like Docker and orchestration platforms like Kubernetes.
• Experience designing and maintaining CI pipelines using tools like Jenkins, GitLab CI/CD, or similar solutions.
• Strong scripting and automation skills in Bash and Python.
• Experience with bare-metal provisioning tools such as MaaS, Foreman, or xCAT.
• Solid working knowledge of networking fundamentals (routers, switches) for control and data traffic.
• Strong communication and collaboration skills, with the ability to work independently and partner closely with a lead DevOps Engineer.

Preferred Qualifications

• Experience with clustering or job-scheduling platforms such as Proxmox or Slurm.
• Experience with build or packaging tools such as UV, Cargo, CMake, or Nix.
• Working knowledge of monitoring and logging tools like Prometheus, Grafana, or the ELK Stack.
• Experience with high-performance computing (HPC) systems.
• Working knowledge of timeseries databases like Prometheus or InfluxDB.

Required Documents

• Resume.
• Cover Letter.