Big Data Engineer at Affine · Remote only · 3 - 8 years · Profitable · Remote only · Posted 11 Dec 2024
Scala
Amazon Web Services (AWS)Role Objective:
Big Data Engineer will be responsible for expanding and optimizing our data and database architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building. The Data Engineer will support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products
Roles & Responsibilities:
- Sound knowledge in Spark architecture and distributed computing and Spark streaming.
- Proficient in Spark – including RDD and Data frames core functions, troubleshooting and performance tuning.
- Good understanding in object-oriented concepts and hands on experience on Scala with excellent programming logic and technique.
- Good in functional programming and OOPS concept on Scala
- Good experience in SQL – should be able to write complex queries.
- Managing the team of Associates and Senior Associates and ensuring the utilization is maintained across the project.
- Able to mentor new members for onboarding to the project.
- Understand the client requirement and able to design, develop from scratch and deliver.
- AWS cloud experience would be preferable.
- Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services - DynamoDB, RedShift, Kinesis, Lambda, S3, etc. (preferred)
- Hands on experience utilizing AWS Management Tools (CloudWatch, CloudTrail) to proactively monitor large and complex deployments (preferred)
- Experience in analyzing, re-architecting, and re-platforming on-premises data warehouses to data platforms on AWS (preferred)
- Leading the client calls to flag off any delays, blockers, escalations and collate all the requirements.
- Managing project timing, client expectations and meeting deadlines.
- Should have played project and team management roles.
- Facilitate meetings within the team on regular basis.
- Understand business requirement and analyze different approaches and plan deliverables and milestones for the project.
- Optimization, maintenance, and support of pipelines.
- Strong analytical and logical skills.
- Ability to comfortably tackling new challenges and learn
External Skills And Expertise
Must have Skills:
- Scala
- Spark
- SQL (Intermediate to advanced level)
- Spark Streaming
- AWS preferable/Any cloud
- Kafka /Kinesis/Any streaming services
- Object-Oriented Programming
- Hive, ETL/ELT design experience
- CICD experience (ETL pipeline deployment)
Good to Have Skills:
- AWS Certification
- Git/similar version control tool
- Knowledge in CI/CD, Microservices
Shubham Vishwakarma
Full Stack Developer - Averlon
About Affine
About
Affine is a pioneering consulting firm that specializes in AI-driven enterprise transformation. With a robust focus on Generative AI, we deliver cutting-edge solutions that redefine industry standards.
Our comprehensive capabilities span the entire analytical value chain, from data to insights and transformation. We provide Cloud Advisory and Assessment, Cloud Migration, Data Lake Design and Development, Big Data Powered Advanced Analytics, AI & Deep Learning Solutions, and Deployment Consulting with Architecture Design.
Company social profiles
Similar jobs (10)
PythonScala🚨 HIRING | Data Engineer – Amazon EMR 🔥
We are looking for experienced Data Engineers with strong hands-on expertise in Amazon EMR, Python, and Scala.
📌 JOB DETAILS:
💼 Experience: 6+ Years
💰 CTC: Up to 30 LPA
📍 Location: Bangalore
🔑 MANDATORY SKILLS:
• Amazon EMR
• Python
• Scala
• Data Engineering
📋 KEY REQUIREMENTS:
• 6+ years of relevant experience in Data Engineering.
• Strong hands-on experience with Amazon EMR.
• Strong programming skills in Python and Scala.
• Experience with large-scale data processing and data engineering solutions.
• Strong problem-solving and communication skills.
📩 Interested candidates can share their updated resumes:
📧 pavithra.e @vysystems. com
📞 8 9 2 5 8 0 2 4 5 7
🔄 Know someone who would be a good fit? Tag them or share this post!
#Hiring #DataEngineer #DataEngineering #AmazonEMR #Python #Scala #AWSJobs #BangaloreJobs #ITJobs #TechHiring #HiringNow
PythonScalaWe are hiring experienced Data Engineers for a permanent opportunity in Bangalore.
Job Description
Role: Data Engineer
Location: Bangalore
Experience: 6+ Years
Key Responsibilities:
- Design, develop, and maintain scalable data processing pipelines using Amazon EMR.
- Develop data engineering solutions using Python and Scala.
- Process and transform large volumes of structured and unstructured data.
- Build and optimize distributed data processing workflows for performance and scalability.
- Work with big data technologies and cloud-based data processing environments.
- Perform data integration, validation, transformation, and quality checks.
- Troubleshoot data pipeline issues and optimize processing performance.
- Collaborate with data architects, analysts, and development teams to deliver reliable data solutions.
Required Skills:
- Strong hands-on experience with Amazon EMR.
- Proficiency in Python and Scala.
- Experience in big data processing and data pipeline development.
- Good understanding of distributed computing and data transformation techniques.
- Experience in performance tuning and troubleshooting data processing jobs.
- Strong analytical, problem-solving, and communication skills.
PythonJob Title : Senior Data Engineer – Databricks
Experience : 14 to 20 Years
Location : HSR Layout, Bangalore
Work Mode : Hybrid – 3 Days WFO
Shift : 11:30 AM – 07:30 PM IST
Positions : 2
Notice Period : Immediate Joiners Only
Interview : 1 Technical Round + 2 Client Rounds
Role Overview :
We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.
The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.
Must-Have Skills :
- 14 to 20 years of Data Engineering experience
- Databricks & Apache Spark / PySpark
- Python & SQL
- AWS Cloud
- Lakehouse Architecture
- ETL / ELT & Distributed Data Processing
- Batch & Streaming Pipelines
- Data Pipeline Optimization & Data Modeling
- CDC & Incremental Processing
- Git, CI/CD & Testing
- Data Quality, Monitoring & Observability
- Technical Leadership & Stakeholder Management
Key Responsibilities :
- Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
- Own data products from design through production.
- Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
- Optimize pipelines for performance, scalability, reliability, and cost.
- Design scalable data architectures and data models.
- Implement data quality, monitoring, lineage, and CI/CD practices.
- Lead technical discussions and mentor engineering teams.
- Collaborate with business stakeholders, architects, product owners, and engineering teams.
- Remain hands-on while providing technical leadership.
Ideal Candidate :
A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.
🔴 Super Urgent : Only Bangalore-based immediate joiners.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Python1st virtual , 2nd round F2F
Python pyspark, SQL, data engineer
5+yrs
9+yrs
Bangalore/Hyderabad
immediate to 15days.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Pythonrole: data engineer
Python pyspark, SQL, data engineer
5+yrs
Bangalore/Hyderabad
immediate to 15days.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Responsibilities and JD
Job Description: We are looking for a Senior Developer with strong expertise in PySpark, Databricks, and Snowflake to build scalable data engineering solutions and enterprise data platforms.
Key Responsibilities:
- Design, develop, and maintain ETL/ELT pipelines using PySpark, Databricks, and Snowflake.
- Develop batch and real-time data processing solutions for structured and semi-structured data.
- Build and optimize Databricks notebooks, workflows, and Delta Lake solutions.
- Design and implement Snowflake databases, schemas, views, stored procedures, tasks, and streams.
- Develop scalable data models, data marts, and data warehouse solutions.
- Optimize PySpark jobs, Databricks workloads, and Snowflake queries for performance and cost efficiency.
- Implement data quality, validation, governance, and security controls.
- Collaborate with business stakeholders, architects, and cross-functional teams to deliver data solutions.
- Manage source control and CI/CD deployments using Git and Azure DevOps.
- Troubleshoot production issues, perform root cause analysis, and ensure pipeline reliability.
- Mentor junior team members and participate in code reviews and technical design discussions.
Required Skills: PySpark, Databricks, Snowflake, Python, SQL.
Experience: 5+ years of Data Engineering experience with strong hands-on expertise in PySpark, Databricks, and Snowflake.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Python1st virtual , 2nd round F2F -
Python pyspark, SQL, data engineer
5+yrs
Bang/hyderabad
immediate to 15days.
Java
Spring BootScalaHiring: Data Engineer – Java / Scala
Experience: 4–10 Years
Location: Bangalore
Mandatory Skills: Java, Scala, Spring Boot, Microservices, REST API, SQL, Git, Testing.
Job Description:
Looking for an experienced Data Engineer with strong Java/Scala development skills and experience in building scalable backend applications and data platforms.
Candidates should have hands-on experience with Spring Boot, Microservices, REST APIs, SQL, Git and Testing. Knowledge of data engineering, data architecture, Databricks, Snowflake and cloud platforms (AWS/Azure/GCP) is preferred.
JavaPythonWe are hiring for a Data Engineer & Java scala for a full-time opportunity in Bangalore.
Experience: 4–10 Years
Must-Have Skills:
Java / Scala / Python, Data Engineering, Databricks / Snowflake, and Cloud (AWS / Azure / GCP).
We are looking for strong software engineers with hands-on experience in data platform development, scalable backend programming, data integration, and cloud technologies.
Experience in Banking, Capital Markets, Market Risk, or Regulatory Reporting will be an added advantage.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
PythonAmazon Web Services (AWS)Data Architect – Databricks & AWS
- Strong experience in Data Architecture, Data Engineering, Databricks, Apache Spark/PySpark, Python, and Advanced SQL.
- Design and implement scalable ETL/ELT pipelines, data platforms, and Lakehouse architectures using Medallion Architecture.
- Experience with Databricks, Databricks Workflows, Unity Catalog OR Databricks Jobs
- Strong knowledge of Delta Lake, Databricks Workflows, Delta Live Tables (DLT), and dimensional data modeling.
- Hands-on experience with AWS services such as S3, Glue, IAM, Lambda, and CloudWatch.
- Experience with Apache Airflow, Data Warehouse concepts, Git/CI-CD, performance optimization, and data quality.
- Good to have exposure to Kafka/Structured Streaming, Unity Catalog, and modern data governance.
Follow CutshortShubham Vishwakarma
Full Stack Developer - Averlon