[C3F] 数据平台工程师
[C3F] Data Platform Engineer
Software Mind 为全球的公司提供具有影响力的技术解决方案。科技巨头和独角兽企业、变革性项目、新兴技术以及无限的机会——这些是我们的日常写照。我们组建跨职能的工程团队，他们拥有主人翁意识并追求卓越，因此我们一直在寻找那些为每个项目带来激情与创造力的优秀人才。我们的文化崇尚开放、尊重、坚韧与勇气，并将工作与乐趣相结合。
项目 – 你将参与的内容
我们的客户通过创新的解决方案和洞察力，帮助我们的客户管理风险并招聘最优秀的人才。他们的先进全球技术平台支持可完全扩展、可配置的筛选程序，满足全球33,000多家客户的独特需求。总部位于美国佐治亚州亚特兰大，他们在19个国家拥有国际化的员工队伍，约有5,500名员工。我们的合作伙伴每年在200多个国家和地区执行超过9300万次筛选。
职位 – 你将做出的贡献
· 使用 Kafka Connect 和 Debezium 作为平台核心，构建从 PostgreSQL 到 Azure 的变更数据捕获（CDC）管道
· 端到端配置、部署和扩展连接器 - 包括连接器设置、任务管理、偏移量、模式历史和快照策略
· 在 Kubernetes（AKS）上以有状态工作负载运行这些管道，涵盖配置、密钥、网络和资源调优
· 监控和排查生产环境中的平台问题：连接器失败、任务重新平衡、重启、吞吐量和背压、消息大小限制、重试和恢复
· 使用 Python 自动化平台 - 基于配置的新数据源接入、管道编排、监控和告警、恢复流程以及自动化测试
· 将 CDC 流集成到更广泛的 Azure 数据栈中：Event Hubs、ADLS、Azure PostgreSQL、ADF 和 Databricks
· 通过基础设施即代码管理平台基础设施，使环境可重复且更改可审查
· 对通过管道传输的敏感数据应用数据保护要求 - 包括脱敏、哈希处理、访问控制和保留策略
要求 – 你需要的经验
· 具备扎实的商业经验，作为数据或平台工程师，有实际操作流式处理或 CDC 管道的经验，而不仅仅是批量报告
· 实践性的 Kafka 知识 - 包括主题、分区、偏移量、消费者组和交付语义 - 包含
查看英文原文
Software Mind develops solutions that make an impact for companies around the globe. Tech giants & unicorns, transformative projects, emerging technologies and limitless opportunities – these are a few words that describe an average day for us. Building cross-functional engineering teams that take ownership and crave more means we’re always on the lookout for talented people who bring passion and creativity to every project. Our culture embraces openness, acts with respect, shows grit & guts and combines employment with enjoyment.
Project – the aim you’ll have
Our customer provides innovative solutions and insights that enable our clients to manage risk and hire the best talent. Their advanced global technology platform supports fully scalable, configurable screening programs that meet the unique needs of over 33,000 clients worldwide. Headquartered in Atlanta, GA, they have an internationally distributed workforce spanning 19 countries with about 5,500 employees. Our partner perform over 93 million screens annually in over 200 countries and territories.
Position – how you’ll contribute
· Build and operate change-data-capture pipelines from PostgreSQL into Azure, using Kafka Connect and Debezium as the core of the platform
· Configure, deploy and scale connectors end to end - connector setup, task management, offsets, schema history, and snapshot strategy
· Run these pipelines as stateful workloads on Kubernetes (AKS), covering configuration, secrets, networking and resource tuning
· Monitor and troubleshoot the platform in production: connector failures, task rebalances, restarts, throughput and backpressure, message-size limits, retries and recovery
· Automate the platform in Python - configuration-driven onboarding of new data sources, pipeline orchestration, monitoring and alerting, recovery workflows, and automated testing
· Integrate CDC streams with the wider Azure data stack: Event Hubs, ADLS, Azure PostgreSQL, ADF and Databricks
· Manage platform infrastructure as code, so environments are reproducible and changes are reviewable
· Apply data protection requirements to sensitive data flowing through the pipelines - masking, hashing, access control and retention
Expectations – the experience you need
· Solid commercial experience as a data or platform engineer, with hands-on work on streaming or CDC pipelines rather than batch reporting alone
· Practical Kafka knowledge - topics, partitions, offsets, consumer groups and delivery semantics - including at least one Kafka Connect deployment you ran yourself
· Strong SQL and PostgreSQL skills, with working knowledge of WAL, logical replication, replication slots and replication lag
· Working understanding of CDC concepts: initial snapshots, inserts, updates and deletes, event ordering, at-least-once delivery, and schema evolution
· Confident Python for automation and tooling - orchestration, monitoring, recovery scripts, and automated tests
· Hands-on experience with Azure data services, for example Event Hubs, ADLS or Azure PostgreSQL
· Comfortable working with Kubernetes as a user: deploying workloads, handling configuration and secrets, reading logs, debugging failing pods
· Ability to debug a running pipeline from metrics and logs - telling throughput problems from backpressure, retries or a genuine connector failure
Additional skills – the edge you have
· Production experience with Debezium specifically - snapshot strategies on large tables, schema history recovery, offset loss, and bringing connectors back after failure
· Experience operating stateful workloads on AKS: StatefulSets, stable worker identity, and resource tuning under load
· Infrastructure-as-code and CI/CD for data platform components (Terraform, Bicep or similar)
· Hands-on work with Databricks and ADF at production scale
· Experience implementing data protection controls for sensitive data - masking, hashing, access control and retention policies
Our offer – professional development, personal growth
· Flexible employment and remote work
· International projects with leading global clients
· International business trips
· Non-corporate atmosphere
· Language classes
· Internal & external training
· Private healthcare and insurance
· Multisport card
· Well-being initiatives