[C3A] Senior Data Platform Engineer (Microsoft Fabric)
软件思维开发出对全球公司产生影响的解决方案。科技巨头和独角兽，转型项目，新兴技术和无限机会这些是描述我们平均的一天的几句话。建立拥有更多资产并渴望多项工程团队意味着我们总是在寻找那些带来激情和创造力的才华横溢的人。我们的文化拥抱开放性，尊重行为，表现出勇气与胆量，并结合合作与乐趣。项目你将要有目标 客户端正在在微软织品上构建下一代风险分析和分数平台，这个角色拥有其奖牌数据架构的端到端。你将设计和建造铜/银/黄金达尔塔表，布火花功能管道，直湖语义模型和物理布局优化。这是我们项目中最具利用率的工程作用。这对第一阶段平台基础和第二阶段风险产品交付都很重要，与高级技术领导者，高级 DevOps 工程师以及中级开发人员一起工作。平台通过 SFTP 和 API 摄入客户的交易数据，使用我们的保留 BIN HMAC 衍生格式保存方案在摄入边界进行敏感 PAN 数据代币化，并根据我们内部标准规范将数据放到微软织物上的可信银中。你将建立通过 MERGE 填充可信银的管道，通过 OneLake 快捷方式消费它，并实现特定风险的黄金表，该表提供了自定义授权 DMN 规则和预分数的 Mastercard Brighterion 传输。角色如何贡献？· 设计和实施 PAN 安全的物理布局，统计配置，液体集群，故意保留设置，V-订单优化。· 为功能管道构建"织物火花笔记本书" （24h/7d/30d 窗口中滚动速度特性，商务级滚动功能，层次滚动功能，通过 pan_token 合并进行交叉交易）. · 作者每种味道版本银适配器 每种味道一个功能，以名称结合，测试与金色固定装置。·构建标准 MERGE DAG （铜 →规范银，在 Aperia 的内部标题 Fabrical v1.4 中使用 _natural_key）. · 建立对两个谷物 （事实_交易_风险，事实_商人_风险）的黄金材料管道，使用正确的分数来源盖章。
查看英文原文
Software Mind develops solutions that make an impact for companies around the globe. Tech giants & unicorns, transformative projects, emerging technologies and limitless opportunities – these are a few words that describe an average day for us. Building cross-functional engineering teams that take ownership and crave more means we’re always on the lookout for talented people who bring passion and creativity to every project. Our culture embraces openness, acts with respect, shows grit & guts and combines cooperation with enjoyment.
Project – the aim you’ll have
Client is building a next-generation risk analysis and scoring platform on Microsoft Fabric, and this role owns its medallion data architecture end-to-end. You'll design and build Bronze / Silver / Gold Delta tables, Fabric Spark feature pipelines, Direct Lake semantic models, and physical-layout optimizations. This is our highest-utilization engineering role on the project — central to both Phase 1 platform foundation and Phase 2 risk product delivery, working alongside a Senior Technical Lead, Senior DevOps engineer, and mid-level developer.
The platform ingests transaction and merchant data from clients via SFTP and API, tokenizes sensitive PAN data at the ingestion boundary using our reserved-BIN HMAC-derived format-preserving scheme, and lands data in canonical Silver on Microsoft Fabric per our internal canonical spec. You'll build the pipeline that populates canonical Silver via MERGE, consume it via OneLake shortcuts, and materialize risk-specific Gold tables that feed both custom-authored DMN rules and pre-scored Mastercard Brighterion feeds.
Role – how you’ll contribute
· Design and implement Bronze, canonical Silver, and Gold Delta table schemas with PAN-safe physical layout — statistics allowlist configuration, liquid clustering, deliberate retention settings, V-Order optimization.
· Build Fabric Spark notebooks for the feature pipeline (rolling velocity features across 24h / 7d / 30d windows, merchant-level rolling features, hierarchical rollups, cross-merchant features via pan_token joins).
· Author per-flavor-version Silver adapters — one function per flavor, unioned by name, tested against golden fixtures.
· Build the canonical MERGE DAG (Bronze → canonical Silver, idempotent on _natural_key per Aperia's canonical Fabric internal spec v1.4).
· Build the Gold materialization pipeline for both grains (fact_transaction_risk, fact_merchant_risk) with correct score-source stamping and idempotent merge semantics.
· Design and implement Direct Lake semantic models per reporting surface; performance-test against representative data volumes.
· Configure OneLake namespace layouts, workspace catalogs, and cross-workspace shortcut relationships with the residuals programme.
· Establish idempotent Delta write patterns using delta-rs from Python containers (segmented parallel writes with single atomic commit).
· Fabric capacity sizing (F-SKU selection, load testing, Reservation vs PAYG decisions).
· Partner with the DevOps engineer on Fabric IaC (Terraform capacity provisioning plus Fabric REST workspace bootstrap).
· Consult on ClickHouse contingency if Direct Lake latency proves insufficient for client-tier UIs.
Expectations – the experience you need
· 5+ years data engineering experience, with at least 2 years hands-on Spark development (PySpark preferred).
· Deep Delta Lake internals knowledge — transaction log format, checkpoints, retention settings (logRetentionDuration, deletedFileRetentionDuration), VACUUM, OPTIMIZE, deletion vectors, statistics collection.
· Parquet format expertise — footers, statistics allowlists, row-group sizing, dictionary encoding, column indexes.
· Microsoft Fabric hands-on experience — workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake, SQL analytics endpoint.
· SQL performance tuning skills for both OLAP (Fabric) and OLTP (PostgreSQL) workloads.
· Strong Python data engineering skills — pandas, PyArrow, delta-rs.
· Ability to communicate architecture clearly in writing and to work autonomously on well-scoped features.
Additional skills – the edge you have
· PCI-scoped data platform experience.
· Financial services or payments domain background.
· Power BI semantic model authoring (DAX).
· Familiarity with published data-standard governance patterns — logical model vs physical profile, SCD2 dimensions, natural-key idempotency, extension attribute stores.
· OneLake shortcuts and cross-workspace lakehouse patterns.
· ClickHouse or similar OLAP engine familiarity.
· Terraform for Azure data resources.
· Prior experience with fluid or schema-flexible ingestion patterns.
What we offer:
· Flexible and remote cooperation options
· International projects with leading global clients
· Travels related to international projects
· Non-corporate atmosphere
· Access* to language classes
· Access* to knowledge-sharing initiatives
· Access* to private healthcare and life insurance
· Access* to multisport card
*The availability and terms of individual benefits may vary depending on the chosen form of cooperation.