Data Engineer

Job Title: Data Engineer (Python, AWS, ETL)

Experience: 6+ years

Location: PAN India (Remote / Work From Home)

Employment Type: Full-time

Preferred domain: Real Estate

Key Responsibilities:

Develop and optimize ETL workflows using Python, Pandas, and PySpark.

Design and implement SQL queries for data extraction, transformation, and optimization.

Work with JSON and REST APIs for data integration and automation.

Manage and optimize Amazon S3 storage, including partitioning and lifecycle policies.

Utilize AWS Athena for SQL-based querying, performance tuning, and cost optimization.

Develop and maintain AWS Lambda functions for serverless processing.

Manage databases using Amazon RDS and Amazon DynamoDB, ensuring performance and scalability.

Orchestrate workflows with AWS Step Functions for efficient automation.

Implement Infrastructure as Code (IaC) using AWS CloudFormation for resource provisioning.

Set up AWS Data Pipelines for CI/CD deployment of data workflows.

Required Skills:

Programming & Scripting: Python (ETL, Automation, REST API Integration).

Databases: SQL (Athena / RDS), Query Optimization, Schema Design.

Big Data & Processing: Pandas, PySpark (Data Transformation, Aggregation).

Cloud & Storage: AWS (S3, Athena, RDS, DynamoDB, Step Functions, CloudFormation, Lambda, Data Pipelines).

Good to Have Skills:

Experience with Azure services such as Table Storage, AI Search, Cognitive Services, Functions, Service Bus, and Storage.

Qualifications:

Bachelor’s degree in Data Science, Statistics, Computer Science, or a related field.

6+ years of experience in data engineering, ETL, and cloud-based data processing.

Azure DE

Primary Responsibilities -

• Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
• Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure Create data models for analytics purposes
• Utilizing Azure Data Factory or comparable technologies, create and maintain ETL (Extract, Transform, Load) operations
• Use Azure Data Factory and Databricks to assemble large, complex data sets
• Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
• Ensure data security and compliance
• Collaborate with data engineers, and other stakeholders to understand requirements and translate them into scalable and reliable data platform architectures

Required skills:

• Blend of technical expertise, analytical problem-solving, and collaboration with cross-functional teams
• Azure DevOps
• Apache Spark, Python
• SQL proficiency
• Azure Databricks knowledge
• Big data technologies

The DEs should be well versed in coding, spark core and data ingestion using Azure. Moreover, they need to be decent in terms of communication skills. They should also have core Azure DE skills and coding skills (pyspark, python and SQL).

Out of the 7 open demands, 5 of The Azure Data Engineers should have minimum 5 years of relevant Data Engineering experience.

·      The Objective:

You will play a crucial role in designing, implementing, and maintaining our data infrastructure, run tests and update the systems

·      Job function and requirements

o  Expert in Python, Pandas and Numpy with knowledge of Python web Framework such as Django and Flask.

o  Able to integrate multiple data sources and databases into one system.

o  Basic understanding of frontend technologies like HTML, CSS, JavaScript.

o  Able to build data pipelines.

o  Strong unit test and debugging skills.

o  Understanding of fundamental design principles behind a scalable application

o  Good understanding of RDBMS databases among Mysql or Postgresql.

o  Able to analyze and transform raw data.

·      About us

Mitibase helps companies find warm prospects every month that are most relevant, and then helps their team to act on those with automation. We do so by automatically tracking key accounts and contacts for job changes and relationships triggers and surfaces them as warm leads in your sales pipeline.

EXP-Developer-4 to 12 years

         Must have low-level design and development skills.  Should able to design a solution for given use cases. 

• Agile delivery-  Person must able to show design and code on a daily basis
• Must be an experienced PySpark developer and  Scala coding.   Primary skill is PySpark
• Must have experience in designing job orchestration, sequence, metadata design, Audit trail, dynamic parameter passing and error/exception handling
• Good experience with unit, integration and UAT support
• Able to design and code reusable components and functions
• Should able to review design, code & provide review comments with justification
• Zeal to learn new tool/technologies and adoption
• Good to have experience with Devops and CICD