AI Data Engineer

MOTOR Information Systems, an operating group of Hearst, is actively seeking a team of enthusiastic, creative, and driven technologists.

People with a proven track record, who are willing to experiment with new ideas, invest time in them, fail-fast, and move on if they don't work out.

We want team members who have shown a consistent interest in continuous learning, especially in Generative AI, who are aware of, and follow the latest and best generative AI technologies and trends, can learn by themselves, are self-motivated and value self-directed initiative with technology and AI exploration. 

Summary

As a Data Engineer, you will design, develop, and maintain data pipelines, data warehouses, and data platforms that support the analytics and business intelligence needs of our clients.

You will work with cutting-edge technologies and tools, such as Spark, Kafka, AWS, Azure, and Elastic, to handle large-scale and complex data challenges.

You will also collaborate with data scientists, analysts, and stakeholders to ensure data quality, reliability, and usability.

Main Responsibilities


* Build automated pipelines to extract and process data from a variety of legacy platforms (predominantly SQL Server), e.g., in stored procedures, Glue processing, etc.


* Implement data-related business logic on modern data platforms, such as AWS Glue, Databricks, Snowflake, Elastic, and Azure Cognitive Search using best practices and industry standards.


* Optimize and monitor the performance, reliability, and security of data systems and processes.


* Integrate and transform data from various sources and formats, such as structured, unstructured, streaming, and batch.


* Develop and maintain data quality checks, tests, and documentation.


* Support data analysis, reporting, and visualization using tools such as SQL, Python, R, Tableau, and Power BI


* Research and evaluate new data technologies and trends to improve data solutions and existing capabilities.

Qualifications and Skills


* Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or a related field


* At least 5 years of experience in data engineering or a similar role (previous DBA experience is a plus)


* Expert in SQL, including a knowledge of efficient query and schema design, DDL, and use of stored procedures


* Proficient in at least one programming language, such as Python, Scala, or Java


* Experience with data modeling, data warehousing, and ETL processes


* Experience building production systems with more modern ETL and data systems, such as AWS Glue, Databricks, Snowflake, Elastic, and Azure Cognitive Search


* Experience deploying data infrastructure on public cloud platforms (AWS, Azure, or GCP)


* Experience with big data frameworks and tools, such as Spark, Hadoop, Kafka, Hive, and Airflow


* Familiarity with pandas, polars, and similar data analysis libraries


* Strong knowledge of data quality,...