Data Engineer (Pyspark Developer)

About the Role

We are looking for a highly motivated Project Manager with a strong background in cloud technologies, big data ecosystems, and software development lifecycles to lead cross-functional teams in delivering high-impact projects. The ideal candidate will combine excellent project management skills with technical acumen in GCP, DevOps, and Python-based applications.

Key Responsibilities

• Lead end-to-end project planning, execution, and delivery, ensuring alignment with business goals and timelines.
• Create and maintain project documentation including detailed timelines, sprint boards, risk logs, and weekly status reports.
• Facilitate Agile ceremonies: daily stand-ups, sprint planning, retrospectives, and backlog grooming.
• Actively manage risks, scope changes, resource allocation, and project dependencies to ensure delivery without disruptions.
• Ensure compliance with QA processes and security/compliance standards throughout the SDLC.
• Collaborate with stakeholders and senior leadership to communicate progress, blockers, and key milestones.
• Provide mentorship and support to cross-functional team members to drive continuous improvement and team performance.
• Coordinate with clients and act as a key point of contact for requirement gathering, updates, and escalations.

Required Skills & Experience

Cloud & DevOps

• Proficient in Google Cloud Platform (GCP) services: Compute, Storage, Networking, IAM.
• Hands-on experience with cloud deployments and infrastructure as code.
• Strong working knowledge of CI/CD pipelines, Docker, Kubernetes, and Terraform (or similar tools).

Big Data & Data Engineering

• Experience with large-scale data processing using tools like PySpark, Hadoop, Hive, HDFS, and Spark Streaming (preferred).
• Proven experience in managing and optimizing big data pipelines and ensuring high performance.

Programming & Frameworks

• Strong proficiency in Python with experience in Django (REST APIs, ORM, deployment workflows).
• Familiarity with Git and version control best practices.
• Basic knowledge of Linux administration and shell scripting.

Nice to Have

• Knowledge or prior experience in the Media & Advertising domain.
• Experience in client-facing roles and handling stakeholder communications.
• Proven ability to manage technical teams (5–6 members).

Why Join Us?

• Work on cutting-edge cloud and data engineering projects
• Collaborate with a talented, fast-paced team
• Flexible work setup and culture of ownership
• Continuous learning and upskilling environment
• Inclusive health benefits included

Profile: AWS Data Engineer

Mode- Hybrid

Experience- 5+7 years

Locations - Bengaluru, Pune, Chennai, Mumbai, Gurugram

Roles and Responsibilities

• Design and maintain ETL pipelines using AWS Glue and Python/PySpark
• Optimize SQL queries for Redshift and Athena
• Develop Lambda functions for serverless data processing
• Configure AWS DMS for database migration and replication
• Implement infrastructure as code with CloudFormation
• Build optimized data models for performance
• Manage RDS databases and AWS service integrations
• Troubleshoot and improve data processing efficiency
• Gather requirements from business stakeholders
• Implement data quality checks and validation
• Document data pipelines and architecture
• Monitor workflows and implement alerting
• Keep current with AWS services and best practices

Required Technical Expertise:

• Python/PySpark for data processing
• AWS Glue for ETL operations
• Redshift and Athena for data querying
• AWS Lambda and serverless architecture
• AWS DMS and RDS management
• CloudFormation for infrastructure
• SQL optimization and performance tuning

Job Description

Job Title: Data Engineer

Location: Hyderabad, India

Job Type: Full Time

Experience: 5 – 8 Years

Working Model: On-Site (No remote or work-from-home options available)

Work Schedule: Mountain Time Zone (3:00 PM to 11:00 PM IST)

Role Overview

The Data Engineer will be responsible for designing and implementing scalable backend systems, leveraging Python and PySpark to build high-performance solutions. The role requires a proactive and detail-orientated individual who can solve complex data engineering challenges while collaborating with cross-functional teams to deliver quality results.

Key Responsibilities

• Develop and maintain backend systems using Python and PySpark.
• Optimise and enhance system performance for large-scale data processing.
• Collaborate with cross-functional teams to define requirements and deliver solutions.
• Debug, troubleshoot, and resolve system issues and bottlenecks.
• Follow coding best practices to ensure code quality and maintainability.
• Utilise tools like Palantir Foundry for data management workflows (good to have).

Qualifications

• Strong proficiency in Python backend development.
• Hands-on experience with PySpark for data engineering.
• Excellent problem-solving skills and attention to detail.
• Good communication skills for effective team collaboration.
• Experience with Palantir Foundry or similar platforms is a plus.

Preferred Skills

• Experience with large-scale data processing and pipeline development.
• Familiarity with agile methodologies and development tools.
• Ability to optimise and streamline backend processes effectively.
• 
  

Job description

Strong experience with Laravel Framework

Experience in databases like MySQL

Experience in HTML, JavaScript, CSS, Angular/ReactJS/VUEJS, JQuery, Bootstrap, JSON

Requirement

Minimum 3+ years of Laravel Experience - building database-driven applications

Knowledge of Event Driven Architecture, Microservices, etc

Basic knowledge of Dev-ops - AWS, Github- Familiarity

About Us

Welthungerhilfe (WHH) is an international aid agency headquartered in Germany with its vision for “Zero Hunger by 2030”, the second sustainable development goal set by the UN. It has thousands of programs fighting hunger in more than 30 countries. (https://www.welthungerhilfe.org/">https://www.welthungerhilfe.org)  (https://www.childgrowthmonitor.org/">https://www.childgrowthmonitor.org).

Child Growth Monitor (CGM) is Welthungerhilfe’s digital innovation project leveraging AI to fight child malnutrition. Around 200 million children around the world are suffering from malnutrition. Malnutrition contributes to one-third of deaths of children under the age of five. In the fight against malnutrition, detection is the first important step, but this is not an easy task: Measuring children with traditional methods is still a complex, slow, and expensive task, frequently resulting in poor data and wrong assessments of the child’s health. Early detection of malnutrition becomes the key to initiating treatment, minimizing the risk of complications,

and saving lives by reducing the chances of death to a great extent.

Our solution to measure children replaces hardware (bulky measuring boards and physical scales) with off-the-shelf cell phones and AI. CGM uses augmented reality-enabled smartphones to record 3D scans of children. Artificial Intelligence is used to predict their height, weight, and middle upper-arm circumference needed to know the nutritional status of the children.

What you would do to help us save millions of children's lives

We are looking for you as an experienced Sr Software Engineer who works collaboratively with the dedicated team of tech specialists (AI, ML & Data Scientists) distributed globally have a shared responsibility for ensuring high quality and

Scalable health care product in an agile approach.

This posting concerns a full-time assignment working for Welthungerhilfe’s CGM in a freelance capacity but Full-time employment (payroll) is scheduled for the end of 2022 latest. We are in the process of setting up CGM as a registered social business from mostly January 2023 and moving all the consultants on CGM payrolls.

Responsibilities

• Designing, developing features, refinement/improving and implementing the CGM product in a secure, well-tested, and performant way.
• De-bugging of the current issues
• Collaborate with Product Management and other stakeholders within Engineering (Frontend, UX, etc.) to maintain a high bar for quality in a fast-paced, iterative environment.
• Advocate for improvements to product quality, security, and performance.
• Solve technical problems of moderate scope and complexity.
• Craft code that meets our internal standards for style, maintainability, and best practices for a high-scale Mobile app environment. Maintain and advocate for these standards through code review.
• Provide mentorship for Junior and Intermediate Engineers on your team to help them grow in their technical responsibilities and remove blockers to their autonomy.

 Requirements and skills

• Comfort working in a highly agile, intensely iterative software development process.
• Demonstrated capacity to clearly and concisely communicate about complex technical, architectural, and/or organizational problems and propose thorough iterative solutions.
• Experience with performance and optimization problems and a demonstrated ability to both diagnose and prevent these problems.
• Extensive experience in core Python (3-7yrs)
• Should have worked on Azure or any Cloud and Azure data factory or any similar data platforms
• Experience using Flask/Flutter framework
• Experience working on mobile applications/Web applications (writing flask framework)
• Knowledge of DevOps Pipeline
• Good experience in SQL database
• Working knowledge in Data engineering (ETL pipelines Result Generation Module)
• Basic understanding of front-end technologies such as Vue JS
• Working knowledge in creating predictive models for AI and ML-based features
• Tech /BE in Computer Science with a total of 7 years of software development/Programming experience.

Desired skills

• Good to have an experience in Kubernetes (opensource system)
• Good to have experience working on healthcare mobile apps (predictive analytics and image diagnostics)
• Experience owning a project from concept to production, including proposal, discussion, and execution.
• Proficient understanding of code versioning tools

Soft/Human Skills

• Positive and solution-oriented mindset.
• Excellent communication & interpersonal skills
• Self-motivated and self-managing, with excellent organizational skills.
• Ability to thrive in a fully remote organization

What you need to have:

• Identify bad code practices in Scala and write new analyzers to detect them.
• Improve the coverage of automatically fixable issues.
• Ensure fewer false-positives are reported by the analyzer.
• Work on the internal tools that support analysis runtimes.
• Contribute to open-source static analysis tools.

We’re looking for someone who has:

• Strong foundational knowledge in Computer Science.
• At least 3 years of professional software development experience in Scala and Java.
• Understanding of the nuances of execution of the source code (AST, data flow graphs, etc).
• Familiarity with Scala best practices followed in the industry.
• Native experience with Linux/Unix environment.
• A focus on delivering high-quality code through strong testing practices.

AutoScheduler is looking for a remote senior software developer to join our talented team. The ideal candidate is a self-starter who is interested in constant learning. We want this person to join our dynamic team as we take an established software and re-develop it from scratch as a part of a new start up.

We are looking for a Backend C++ developer responsible for maintaining and developing algorithmic C++ for new and existing customers. Your primary responsibilities will be to design and develop applications and services, and to coordinate with the rest of the team working on different layers of the infrastructure. Therefore, a commitment to collaborative problem solving, robust design, and quality product is essential.

• Work with non-technical personnel to translate business requirements into stories and epics
• Translate application storyboards and use cases into functional applications / features
• Design, build, and maintain efficient, reusable, and reliable code
• Ensure the best possible performance, quality, and responsiveness of applications
• Identify bottlenecks and bugs, and devise solutions to these problems
• Help maintain code quality, organization, and automation
• Write and maintain unit/functional/integration tests

• Bachelor’s degree with STEM concentration + 2 years’ work experience, or equivalent work experience
• Fluent in English
• Demonstrably proficient in C/C++ and the ability to parse C/C++ code
• Proficiency in Node.js and Javascript
• Basic understanding of Common Language Runtime (CLR), its limitations, weaknesses, and workarounds
• Working knowledge of at least two other programming languages (e.g. Go and Python)
• Proficient understanding of modern distributed code versioning tools (like ‘git’)
• Thorough understanding of SQL, and in-depth experience with at least one RDBMS (e.g. PostgreSQL or Microsoft SQL Server)
• Experience with automated testing frameworks and unit tests
• Solid understanding of object-oriented programming and principles
• Knowledge of functional programming principals / concepts

• Experience building applications with C++
• Experience building cross-platform applications with Scala
• Experience with functional programming
• Familiarity with concurrency patterns in Scala
• Experience building distributed systems and/or decoupled microservices
• Experience building software using cloud-based services (in any cloud platform)
• Experience using “gRPC” and Protocol Buffers (“protobuf”) and/or experience creating services that exchange non-JSON data over non-HTTP protocols
• Familiarity with continuous integration
• Familiarity with Docker / containerization