Data Engineer (Porto Ou Lisboa, 3y-6y)

🚀 About the Role

We are looking for a Senior GCP Data Engineer to support a strategic data transformation and cloud migration initiative.

In this role, you will help migrate databases, ETL processes, and analytical workloads from SQL Server to Google Cloud Platform , designing a modern, scalable, and cost-efficient data ecosystem powered by BigQuery, Cloud Composer, and Python .

You will work closely with Database Administrators, Data Engineers, Analysts, Data Scientists, and business stakeholders to understand the existing environment, rebuild critical data flows, and ensure a smooth transition to the new cloud platform.

This is an excellent opportunity for someone who combines strong data engineering expertise with hands-on cloud migration experience and enjoys solving complex data, performance, and integration challenges.

Your Mission

As a Senior GCP Data Engineer, you will:

• Contribute to the migration of data and ETL processes from SQL Server to GCP.
• Design and build scalable data pipelines using modern cloud technologies.
• Develop optimized data models and queries for BigQuery.
• Automate migration, transformation, validation, and integration activities using Python.
• Guarantee data quality, integrity, performance, and operational reliability throughout the migration.
• Help establish engineering standards and best practices for the new GCP data platform.
• Support teams and end users during the transition to the cloud environment.

Key Responsibilities

Data & Database Migration

• Contribute to the design and execution of the migration strategy from SQL Server to Google Cloud Platform .
• Analyze existing databases, ETL workflows, stored procedures, dependencies, and data structures.
• Develop migration scripts and pipelines to transfer and transform data into BigQuery.
• Rebuild or adapt existing SQL Server processes for a cloud-native data architecture.
• Support migration planning, technical assessments, testing, reconciliation, and production deployment.
• Ensure data consistency and integrity before, during, and after migration.
• Identify migration risks, technical dependencies, and potential performance issues.

GCP Data Pipeline Development

• Design, develop, and maintain robust ETL and ELT pipelines on Google Cloud Platform.
• Replace or modernize existing data flows using scalable cloud services.
• Build reusable components and standardized pipeline patterns.
• Ensure pipelines are reliable, maintainable, observable, and performance-oriented.
• Implement appropriate retry, recovery, logging, and error-handling mechanisms.
• Support both batch processing and data integration use cases.

Workflow Orchestration

• Create, schedule, monitor, and maintain workflows using Google Cloud Composer .
• Design and manage DAGs for migration, transformation, and integration processes.
• Define dependencies, execution schedules, retries, alerts, and failure-handling processes.
• Troubleshoot orchestration issues and optimize workflow execution.
• Ensure reliable operationalization of pipelines across development, testing, and production environments.

BigQuery Engineering & Optimization

• Design and implement scalable data structures in BigQuery.
• Develop and optimize complex SQL queries for analytical and transformation workloads.
• Adapt T-SQL logic and existing SQL Server processes to BigQuery SQL.
• Apply BigQuery optimization techniques, including:Partitioning
• Clustering
• Query optimization
• Storage optimization
• Cost-efficient processing
• Monitor query performance and resource consumption.
• Contribute to the design of cloud data models aligned with business and analytical requirements.

Python Development & Automation

• Develop Python scripts for:Data migration
• Data transformation
• Data reconciliation
• Test data preparation
• Process automation
• Integration with GCP services and APIs
• Build reusable and maintainable Python components.
• Automate repetitive migration and operational activities.
• Support the development of validation and monitoring utilities.
• Apply software engineering best practices, including code reviews, version control, testing, and documentation.

Data Quality & Validation

• Implement data quality controls throughout the migration lifecycle.
• Develop automated reconciliation processes between source and target systems.
• Validate data completeness, accuracy, consistency, and integrity.
• Define tests for migrated tables, transformations, pipelines, and business rules.
• Investigate data discrepancies and coordinate their resolution.
• Ensure that the new platform produces reliable and trusted data for its users.

Monitoring, Support & Troubleshooting

• Monitor data pipelines, scheduled workflows, and production processes.
• Investigate and resolve pipeline failures, data issues, and performance bottlenecks.
• Support debugging and incident resolution during the migration and after production deployment.
• Perform root cause analysis and implement sustainable corrective measures.
• Improve platform observability, alerts, operational procedures, and support documentation.

Collaboration & Stakeholder Support

• Work closely with SQL Server administrators and existing technical teams to understand the current ecosystem.
• Collaborate with Data Analysts, Data Scientists, and business teams to gather data requirements.
• Ensure that the new GCP platform meets functional, analytical, and operational expectations.
• Participate in technical design sessions, code reviews, and testing activities.
• Communicate migration progress, risks, dependencies, and technical decisions clearly.
• Support knowledge transfer and the adoption of the new cloud data platform.

Continuous Improvement

• Stay informed about GCP data engineering capabilities and cloud migration practices.
• Identify opportunities to improve performance, scalability, cost efficiency, and maintainability.
• Propose new tools, patterns, and automation opportunities for the data platform.
• Contribute to engineering standards, documentation, and reusable development practices.
• Help establish a culture of technical excellence and continuous improvement.

✅ What We’re Looking For

Mandatory Requirements

• Strong professional experience in Data Engineering .
• Advanced knowledge of Google Cloud Platform , particularly its data services.
• Significant hands-on experience with BigQuery , including:Data modeling
• Schema design
• Complex SQL development
• Partitioning and clustering
• Query performance optimization
• Cost management
• Strong experience with Google Cloud Composer .
• Hands-on experience creating, scheduling, monitoring, and debugging DAGs.
• Excellent SQL skills and the ability to adapt existing SQL workloads to BigQuery.
• Advanced Python skills for data engineering, automation, and GCP integration.
• Experience designing and building ETL or ELT pipelines.
• Strong understanding of database and data migration methodologies.
• Ability to analyze legacy or on-premises environments and redesign them for the cloud.
• Strong analytical and troubleshooting capabilities.
• Good English communication skills, with a minimum B2 level .

🛠 Technical Stack

Google Cloud Platform

• Google Cloud Platform
• BigQuery
• Cloud Composer
• Cloud Storage
• Dataflow
• GCP APIs
• Database Migration Service

Data Engineering

• ETL
• ELT
• Batch Data Processing
• Data Integration
• Pipeline Orchestration
• Data Migration
• Data Reconciliation

Development

• Python
• SQL
• T-SQL
• Automation Scripting
• API Integration

Databases & Data Warehousing

• Microsoft SQL Server
• BigQuery
• Relational Databases
• Cloud Data Warehouses
• Data Modeling
• Data Vault

Performance & Cost Optimization

• BigQuery Partitioning
• BigQuery Clustering
• Query Optimization
• Storage Optimization
• Cloud Cost Management

Quality & Delivery

• Data Quality
• Data Validation
• Automated Testing
• Code Reviews
• Version Control
• CI/CD Practices
• Technical Documentation

⭐ Nice to Have

• Direct experience with Microsoft SQL Server , including:SQL Server architecture
• T-SQL
• Stored procedures
• ETL dependencies
• Basic database administration
• Experience with additional GCP services such as:Cloud Storage
• Dataflow
• Database Migration Service
• Experience using cloud-specific database migration tools.
• Knowledge of Data Warehousing principles and dimensional modeling.
• Experience with Data Vault methodology.
• Knowledge of Power BI or other data visualization tools.
• Experience with source control and collaborative development workflows.
• Familiarity with CI/CD pipelines for data engineering solutions.
• Previous involvement in large-scale on-premises to cloud migration programs.