Data Engineer
Permanent Role
Amazing opportunity to join a global leading IT solutions provider and we are known for the diverse expertise and innovative thinking with breakthrough technologies to progress further.
As a Data Engineer, you will be responsible for designing, developing, and maintaining the data infrastructure and pipelines required to support data-driven decision-making processes within the organization. Your role will involve collaborating with cross-functional teams, including data scientists, analysts, and software engineers, to ensure the efficient and reliable flow of data from various sources to data repositories and analytical systems. You will play a vital role in optimizing data workflows, ensuring data quality, and implementing scalable solutions for data processing and storage.
Roles & Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETL processes to extract, transform, and load data from various sources into data warehouses, data lakes, or other storage systems.
- Collaborate with stakeholders to understand data requirements and translate them into technical specifications and data models.
- Implement data integration and data transformation solutions using programming languages (e.g., Python, SQL, Java) and big data processing frameworks (e.g., Apache Spark, Hadoop).
- Perform data cleansing, validation, and enrichment to ensure data quality and consistency.
- Monitor and optimize data pipelines to ensure efficient and reliable data flow, identifying and resolving performance bottlenecks.
- Collaborate with data scientists and analysts to support their data needs and ensure the availability of clean, reliable, and timely data for analysis and reporting.
- Develop and maintain data documentation, including data dictionaries, data lineage, and data governance policies.
- Implement and enforce data security and privacy measures to ensure compliance with relevant regulations (e.g., GDPR).
- Stay up-to-date with emerging trends and technologies in data engineering and recommend improvements to existing data infrastructure and processes.
- Participate in code reviews, provide constructive feedback, and adhere to best practices for software development and data engineering.
TECHNICAL EXPERIENCE:
- Bachelor's or master's degree in computer science, software engineering, data science, or a related field.
- Proven experience as a Data Engineer or in a similar role, with a focus on data integration, data transformation, and data pipeline development.
- Strong programming skills in languages such as Python, SQL, Java and familiarity with big data processing frameworks (e.g., Apache Spark, Hadoop).
- Experience with cloud-based data platforms and services (e.g., AWS, Google Cloud Platform, Azure).
- Experience working with Visualisation tools like Elastic Search (ELK stack) or Grafana
- Proficiency in data modeling and understanding of relational databases, data warehousing concepts, and dimensional modeling.
- Familiarity with data governance principles and best practices for data quality, security, and privacy.
- Strong problem-solving and analytical skills, with the ability to analyze complex data challenges and develop effective solutions.
- Excellent communication skills and the ability to collaborate effectively with cross-functional teams.
- Self-motivated and able to work independently with minimal supervision, as well as in a team-oriented and fast-paced environment.
- Familiarity with version control systems (e.g., Git) and experience with agile software development methodologies is a plus.