Note: The job is a remote job and is open to candidates in USA. Stack AV is developing revolutionary AI and advanced autonomous systems for the trucking transportation industry. As a Senior Data Platform Engineer, you will design and operate high scale data systems that support large-scale autonomous systems development and ensure reliable, efficient, and easy-to-run critical services and pipelines.
Responsibilities
- Design and operate distributed storage systems for scheduling and executing large-scale batch workloads
- Build and maintain an open source, modern data platform
- Optimize utilization of storage resourcesImprove reliability and fault tolerance of large-scale storage systems and data platform components
- Collaborate with teams across the company to understand workload requirements and improve platform capabilities
- Contribute to platform tooling, automation, and CI/CD workflows
Skills
- 7+ years of experience building and operating distributed storage systems or modern data platforms
- Experience operating streaming platforms such as Kafka or Pulsar
- Fluent in Python, and SQL, with experience writing and maintaining highly available data applications using Trino and Apache Spark
- Knowledge of table formats (Iceberg, Delta Lake, Hudi, Xtable)
- Experience operating and optimizing at least one RDBMS (Postgres, MySQL)
- Strong debugging and problem-solving skills in complex distributed systems
- Ability to collaborate across teams and communicate technical concepts clearly
Company Overview