数据工程师（Elasticsearch + 数据仓库）
Data Engineer (Elasticsearch + Datawarehousing)
数据工程师（Elasticsearch + 数据仓库）
Particle41 正在寻找一位有才华且多面的数据工程师加入我们创新的团队。作为数据工程师，你将在设计、构建和维护强大的数据管道和基础设施方面发挥关键作用，以支持客户的数据显示需求。你将参与端到端的数据解决方案，与跨职能团队合作，确保高质量、可扩展且高效的数据交付。这是一个令人兴奋的机会，可以为有影响力项目做出贡献，解决复杂的数据挑战，并在一个支持性和动态的环境中提升你的技能。
在该职位中，你将：
软件开发
- 设计、实现并优化 Elasticsearch 集群，以实现高性能查询和数据检索。
- 构建和管理 Elasticsearch 索引，确保数据被高效存储、索引和查询。
- 构建和优化数据存储解决方案，如数据湖和数据仓库。
- 从各种内部和外部系统集成结构化和非结构化数据，以创建统一的分析视图。
- 通过严格的验证、清洗和转换过程，确保数据的准确性、一致性和完整性。
- 维护数据流程、工具和系统的全面文档，同时推广高效工作流程的最佳实践。
需求收集与分析
- 与产品经理和其他利益相关者合作，收集需求并将其转化为技术解决方案。
- 参与需求分析会议，以理解业务需求和用户需求。
- 在需求收集过程中提供技术见解和建议。
敏捷开发
- 参与敏捷开发流程，包括冲刺计划、每日站会和冲刺评审。
- 与敏捷团队紧密合作，按时且在范围内交付软件解决方案。
- 在快节奏的敏捷环境中适应变化的优先级和需求。
测试与调试
- 进行彻底的测试和调试，以确保应用程序的可靠性、安全性和性能。
- 编写单元测试并验证开发功能和单个元素的功能。
- 编写集成测试，以确保给定应用程序中的不同元素按预期运行并满足所需要求。
- 识别并解决软件缺陷、代码异味和性能瓶颈。
持续学习与创新
- 保持对新技术和趋势的了解，不断学习和探索创新方法。
查看英文原文
Data Engineer (Elasticsearch + Data warehousing)
Particle41 is seeking a talented and versatile Data Engineer to join our innovative team. As a Data Engineer, you will play a key role in designing, building, and maintaining robust data pipelines and infrastructure to support our clients' data needs. You will work on end-to-end data solutions, collaborating with cross-functional teams to ensure high-quality, scalable, and efficient data delivery. This is an exciting opportunity to contribute to impactful projects, solve complex data challenges, and grow your skills in a supportive and dynamic environment.
In This Role, You Will:Software Development
- Design, implement, and optimize Elasticsearch clusters for high-performance querying and data retrieval.
- Build and manage Elasticsearch indexes, ensuring data is stored, indexed, and queried efficiently.
- Build and optimize data storage solutions like data lakes and warehouses.
- Integrate structured and unstructured data from various internal and external systems to create a unified view for analysis.
- Ensure data accuracy, consistency, and completeness through rigorous validation, cleansing, and transformation processes.
- Maintain comprehensive documentation for data processes, tools, and systems while promoting best practices for efficient workflows.
Requirements Gathering and Analysis
- Collaborate with product managers, and other stakeholders to gather requirements and translate them into technical solutions.
- Participate in requirement analysis sessions to understand business needs and user requirements.
- Provide technical insights and recommendations during the requirements-gathering process.
Agile Development
- Participate in Agile development processes, including sprint planning, daily stand-ups, and sprint reviews.
- Work closely with Agile teams to deliver software solutions on time and within scope.
- Adapt to changing priorities and requirements in a fast-paced Agile environment.
Testing and Debugging
- Conduct thorough testing and debugging to ensure the reliability, security, and performance of applications.
- Write unit tests and validate the functionality of developed features and individual elements.
- Writing integration tests to ensure different elements within a given application function as intended and meet desired requirements.
- Identify and resolve software defects, code smells, and performance bottlenecks.
Continuous Learning and Innovation
- Stay updated with the latest technologies and trends in full-stack development.
- Propose innovative solutions to improve the performance, security, scalability, and maintainability of applications.
- Continuously seek opportunities to optimize and refactor existing codebase for better efficiency.
- Stay up-to-date with cloud platforms such as AWS, Azure, or Google Cloud Platform.
Collaboration
- Collaborate effectively with cross-functional teams, including testers, and product managers.
- Foster a collaborative and inclusive work environment where ideas are shared and valued.
Skills and Experience We Value:
- Bachelor's degree in Computer Science, Engineering, or related field.
- Proven experience as a Data Engineer, with a minimum of 3 years of experience.
- Proficiency in Elasticsearch and Python programming language is a must.
- Experience with database technologies such as SQL (e.g., MySQL, PostgreSQL) and NoSQL (e.g., MongoDB) databases.
- Strong understanding of Programming Libraries/Frameworks and technologies such as Flask, API frameworks, Data warehousing/lakehouse, Principles, Database, ORM, Data analysis, Databricks, Pandas, Spark, PySpark, Machine learning, OpenCV, Scikit-learn.
- Utilize Java to build and enhance backend systems, particularly for integration with Elasticsearch and databases.
- Develop APIs, microservices, and automation scripts as needed.
- Utilities & Tools: logging, requests, subprocess, regex, pytest
- ELK stack, Redis, distributed task queues
- Strong understanding of data warehousing/lakehousing principles and concurrent/parallel processing concepts.
- Familiarity with at least one cloud data engineering stack (Azure, AWS, or GCP) and the ability to quickly learn and adapt to new ETL/ELT tools across various cloud providers.
- Familiarity with version control systems like Git and collaborative development workflows.
- Competence in working on Linux OS and creating shell scripts.
- Solid understanding of software engineering principles, design patterns, and best practices.
- Excellent problem-solving and analytical skills, with a keen attention to detail.
- Effective communication skills, both written and verbal, and the ability to collaborate in a team environment.
- Adaptability and willingness to learn new technologies and tools as needed.
About Particle41Our core values of Empowering, Leadership, Innovation, Teamwork, and Excellence drive everything we do to achieve the ultimate outcomes for our clients. Empowering Leadership for Innovation in Teamwork with Excellence ( ELITE )
- E - Empowering: Enabling individuals to reach their full potential.
- L - Leadership: Taking initiative and guiding each other toward success.
- I - Innovation: Embracing creativity and new ideas to stay ahead.
- T - Teamwork: Collaborating with empathy to achieve common goals.
- E - Excellence: Striving for the highest quality in everything we do.
We seek team members who embody these values and are committed to contributing to our mission. Particle41 welcomes individuals from all backgrounds who are committed to our mission and values. We provide equal employment opportunities to all employees and applicants, ensuring that hiring and employment decisions are based on merit and qualifications without discrimination based on race, color, religion, caste, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, local, or international laws. This policy applies to all aspects of employment and hiring.
We appreciate your interest and encourage applicants from these regions to apply. If you need any assistance during the application or interview process, please feel free to reach out to us at .
Originally posted on Himalayas