Responsibilities

• Build ingestion pipelines from multiple sources to raw data storage (e.g., S3) using micro-batch, batch, and streaming processes.
• Implement initial data quality checks, logging, and monitoring frameworks for data pipeline operations.
• Transform raw data into data products by applying business logic and requirements.
• Create and maintain several data feeds to external systems using micro-batch and streaming patterns.
• Review, suggest, and propose improvements for the data platform created by the team.
• Optimize performance of large-scale distributed data systems to ensure reliability and scalability.
• Collaborate with cross-functional teams to deploy pipelines and integrate data solutions into production environments.

Requirements

• 3+ years of experience as a data engineer in consumer finance or similar industries involving consumer loans, collections, servicing, optional products, or insurance sales.
• Strong background in math, statistics, computer science, data science, or a related discipline.
• Expertise in programming with one or more languages such as Snowflake, Java, Scala, Python, or C#.
• Production experience working with HDFS, YARN, Hive, Spark, Kafka, Oozie/Airflow, and containerization tools like Docker/Kubernetes.
• Proficiency in data mining and programming tools (e.g., SAS, SQL, R, Python) and database technologies such as PostgreSQL, Redshift, Snowflake, or Greenplum.
• Familiarity with business intelligence and analytical tools, technologies, and techniques for data insights.
• Showcase of experience deploying data pipelines on cloud platforms including AWS, with storage and processing tools.
• Flexibility to learn new tools and technologies, with a proven ability to adopt emerging solutions into workflows.
• Strong communication skills for articulating technical concepts to non-technical audiences effectively.
• Understanding of performance optimization for large-scale distributed systems and Big Data infrastructure.
• Fluent in English, with a proficiency level of B2 or higher.

Nice to have

• AWS certification demonstrating expertise in cloud technologies.
• Familiarity with Spark Streaming, Kafka Streaming, or Kafka Connect for real-time data processing.
• Experience with the ELK Stack (Elasticsearch, Logstash, Kibana) for monitoring and analytics.
• Knowledge of NoSQL databases such as Cassandra or MongoDB for large-scale data storage solutions.
• Understanding of CI/CD tools like Jenkins and GitLab, as well as tools such as Jira and Confluence for project management and collaboration.

We offer

• International projects with top brands.
• Work with global teams of highly skilled, diverse peers.
• Healthcare benefits.
• Employee financial programs.
• Paid time off and sick leave.
• Upskilling, reskilling and certification courses.
• Unlimited access to the LinkedIn Learning library and 22,000+ courses.
• Global career opportunities.
• Volunteer and community involvement opportunities.
• EPAM Employee Groups.
• Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn.