Lead II - SE - AWS, Apache Spark (PySpark/Scala), Apache Kafka
at Global digital transformation solutions provider.
Amazon Web Services (AWS)
Scala
Jenkins
Data Analytics
Data Science
Java
MongoDBJOB DETAILS:
* Job Title: Lead II - Software Engineering - AWS, Apache Spark (PySpark/Scala), Apache Kafka
* Industry: Global digital transformation solutions provider
* Salary: Best in Industry
* Experience: 5-8 years
* Location: Hyderabad
Job Summary
We are seeking a skilled Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms. The role involves working with large-scale batch and real-time data processing systems, collaborating with cross-functional teams, and ensuring data reliability, security, and performance across the data lifecycle.
Key Responsibilities
ETL Pipeline Development & Optimization
- Design, develop, and maintain complex end-to-end ETL pipelines for large-scale data ingestion and processing.
- Optimize data pipelines for performance, scalability, fault tolerance, and reliability.
Big Data Processing
- Develop and optimize batch and real-time data processing solutions using Apache Spark (PySpark/Scala) and Apache Kafka.
- Ensure fault-tolerant, scalable, and high-performance data processing systems.
Cloud Infrastructure Development
- Build and manage scalable, cloud-native data infrastructure on AWS.
- Design resilient and cost-efficient data pipelines adaptable to varying data volume and formats.
Real-Time & Batch Data Integration
- Enable seamless ingestion and processing of real-time streaming and batch data sources (e.g., AWS MSK).
- Ensure consistency, data quality, and a unified view across multiple data sources and formats.
Data Analysis & Insights
- Partner with business teams and data scientists to understand data requirements.
- Perform in-depth data analysis to identify trends, patterns, and anomalies.
- Deliver high-quality datasets and present actionable insights to stakeholders.
CI/CD & Automation
- Implement and maintain CI/CD pipelines using Jenkins or similar tools.
- Automate testing, deployment, and monitoring to ensure smooth production releases.
Data Security & Compliance
- Collaborate with security teams to ensure compliance with organizational and regulatory standards (e.g., GDPR, HIPAA).
- Implement data governance practices ensuring data integrity, security, and traceability.
Troubleshooting & Performance Tuning
- Identify and resolve performance bottlenecks in data pipelines.
- Apply best practices for monitoring, tuning, and optimizing data ingestion and storage.
Collaboration & Cross-Functional Work
- Work closely with engineers, data scientists, product managers, and business stakeholders.
- Participate in agile ceremonies, sprint planning, and architectural discussions.
Skills & Qualifications
Mandatory (Must-Have) Skills
- AWS Expertise
- Hands-on experience with AWS Big Data services such as EMR, Managed Apache Airflow, Glue, S3, DMS, MSK, and EC2.
- Strong understanding of cloud-native data architectures.
- Big Data Technologies
- Proficiency in PySpark or Scala Spark and SQL for large-scale data transformation and analysis.
- Experience with Apache Spark and Apache Kafka in production environments.
- Data Frameworks
- Strong knowledge of Spark DataFrames and Datasets.
- ETL Pipeline Development
- Proven experience in building scalable and reliable ETL pipelines for both batch and real-time data processing.
- Database Modeling & Data Warehousing
- Expertise in designing scalable data models for OLAP and OLTP systems.
- Data Analysis & Insights
- Ability to perform complex data analysis and extract actionable business insights.
- Strong analytical and problem-solving skills with a data-driven mindset.
- CI/CD & Automation
- Basic to intermediate experience with CI/CD pipelines using Jenkins or similar tools.
- Familiarity with automated testing and deployment workflows.
Good-to-Have (Preferred) Skills
- Knowledge of Java for data processing applications.
- Experience with NoSQL databases (e.g., DynamoDB, Cassandra, MongoDB).
- Familiarity with data governance frameworks and compliance tooling.
- Experience with monitoring and observability tools such as AWS CloudWatch, Splunk, or Dynatrace.
- Exposure to cost optimization strategies for large-scale cloud data platforms.
Skills: big data, scala spark, apache spark, ETL pipeline development
******
Notice period - 0 to 15 days only
Job stability is mandatory
Location: Hyderabad
Note: If a candidate is a short joiner, based in Hyderabad, and fits within the approved budget, we will proceed with an offer
F2F Interview: 14th Feb 2026
3 days in office, Hybrid model.
Shubham Vishwakarma
Full Stack Developer - Averlon
Similar jobs
Python
GitJob Title : Python Developer - Web3 (Mandatory) & Trading Bot Creation (Optional)
Experience : 2+ Years
Location : Noida (On-site)
Working Days : 6 Days WFO (Monday to Friday - WFO & Saturday - WFH)
Job Type : Full-time
Mandatory Skills : Python, Web3 (web3.py/ethers), smart contract interaction, real-time APIs/WebSockets, Git/Docker, security handling.
Responsibilities :
- Build and optimize Web3-based applications & integrations using Python.
- Interact with smart contracts and manage on-chain/off-chain data flows.
- Ensure secure key management, scalability, and performance.
- (Optional) Develop and enhance automated trading bots & strategies.
Required Skills :
- Strong experience in Python development.
- Proficiency in Web3 (web3.py/ethers) and smart contract interaction.
- Hands-on with real-time APIs, WebSockets, Git/Docker.
- Knowledge of security handling & key management.
- (Optional) Trading bot development, CEX/DEX APIs, backtesting (pandas/numpy).
ScalaJob Title: Senior Data Engineer
No. of Positions: 3
Employment Type: Full-Time, Permanent
Location: Remote (Pan India)
Shift Timings: 10:00 AM – 7:00 PM IST
Experience Required: Minimum 3+ Years
Mandatory Skills: Scala & PySpark
Role Overview
We are looking for an experienced Senior Data Engineer to design, build, and optimize scalable data pipelines and architectures. The ideal candidate should have hands-on experience working with Big Data technologies, distributed systems, and ETL pipelines. You will work closely with cross-functional teams including Data Analysts, Data Scientists, and Software Engineers to ensure efficient data flow and reliable data infrastructure.
Key Responsibilities
- Design and build scalable data pipelines for extraction, transformation, and loading (ETL) from various data sources.
- Enhance internal processes by automating tasks, optimizing data workflows, and improving infrastructure performance.
- Collaborate with Product, Engineering, Data, and Business teams to understand data needs and provide solutions.
- Work closely with machine learning and analytics teams to support advanced data modeling and innovation.
- Ensure systems are highly reliable, maintainable, and optimized for performance.
Required Qualifications & Skills
- Bachelor’s degree in Computer Science, Engineering, or related field.
- 3+ years of hands-on experience in Data Engineering.
- Strong experience with Apache Spark, with solid understanding of distributed data processing.
- Proficiency in Scala and PySpark is mandatory.
- Strong SQL skills and experience working with relational and non-relational data.
- Experience with cloud-based data platforms (preferably Databricks).
- Good understanding of Delta Lake architecture, Parquet, JSON, CSV, and related data file formats.
- Comfortable working in Linux/macOS environments with scripting capabilities.
- Ability to work in an Agile environment and deliver independently.
- Good communication and collaboration skills.
- Knowledge of Machine Learning concepts is an added advantage.
Reporting
- This role will report to the CEO or a designated Team Lead.
Benefits & Work Environment
- Remote work flexibility across India.
- Encouraging and diverse work culture.
- Paid leaves, holidays, performance incentives, and learning opportunities.
- Supportive environment that promotes personal and professional growth.
Role Review-
Our client's property Sales team is responsible for selling properties available in our inventory. As a Sales Associate / Sr. Sales Associate, you’re the First Point of Contact and the face of our client with respect to Sales. You’ll work closely with the Sales Manager to give ground insights and help the sales scale. You’ll be directly responsible for Sales for one or more micro-markets in the city of Bangalore. Your role will be a combination of customer conversation and contextual field presence.
WHAT WILL YOU DO?
● Communicating effectively with customers, proactively reaching out to potential clients, and diligently following up on leads.
● Establishing and nurturing relationships with Channel Partners (brokers) to cultivate customer leads.
● Profoundly comprehending the unique needs of customers and identifying lucrative sales
opportunities.
● Providing comprehensive responses to customer inquiries and promptly delivering
supplementary information.
● Keeping abreast of product and service updates, ensuring in-depth product knowledge.
● Methodically managing and updating a comprehensive customer database.
● Effectively elucidating and demonstrating the distinctive features of our products and services.
● Staying well-informed about competing products and services, and effectively positioning.
● Skillfully executing upselling strategies for our products and services.
● Conducting rigorous research to qualify new leads.
● Demonstrating a stellar track record in closing sales and consistently achieving sales targets.
WHAT DO YOU NEED TO BRING TO THE TABLE?
● 1-5 years of experience in a similar role within the Real Estate, Hospitality, Automobile, or
Insurance sectors. This is preferable but is not a barrier.
● A fervent enthusiasm for working in a target-oriented environment.
● Exceptional negotiation skills.
● Outstanding customer service skills.
● Excellent verbal communication skills in English; proficiency in additional languages is a
valuable asset.
● A keen understanding of timelines and a proactive approach to meeting deadlines.
● A proactive and tenacious attitude, coupled with a relentless pursuit of objectives.
● Strong organizational skills and the ability to manage multiple tasks concurrently.
● A remarkable memory; at any point in time, you will be driving multiple conversations where context is the king or queen!
● This is a 6-day work-week role. Weekends will be working since maximum action in Real
Estate happens on Saturday/Sundays
Key Responsibilities:
- Lead the end-to-end solution design and technical architecture for SAP projects and connected enterprise systems.
- Translate business and functional requirements into scalable, efficient, and maintainable SAP technical solutions.
- Configure SAP modules in line with defined business processes and project goals.
- Collaborate with stakeholders across business and IT to ensure alignment on solution design and implementation.
- Prepare and maintain high-quality technical documentation, including solution blueprints, configuration guides, and test cases.
- Oversee and support the testing lifecycle: unit testing, system integration testing, and user acceptance testing.
- Ensure integration with non-SAP systems follows enterprise standards and architecture best practices.
- Provide technical leadership to development teams and engage in solution reviews and risk assessments.
Required Skills:
- 8-12 years of IT experience, with at least 4-6 years as a Solution/Technical Architect in SAP environments.
- Strong hands-on expertise in SAP (S/4HANA, ECC, SAP Fiori, BTP, ABAP, or other modules as required).
- Proven ability to design integrations between SAP and third-party systems.
- Excellent skills in solution documentation, stakeholder communication, and problem-solving.
- Exposure to Agile/Scrum or hybrid delivery methodologies.
- SAP certifications (technical or functional) are highly desirable.
Good to Have:
- Experience with cloud platforms (Azure, AWS, GCP) and SAP on cloud.
- Knowledge of emerging SAP technologies like SAP BTP, AI/ML in SAP, RPA, etc.
- Experience working in large-scale global rollouts or transformation programs.
Location: Coimbatore
Work Type: Full-Time
Joining Period: Immediate, 20 Days Notice Period
Vue.js
AngularJS (1.x)
Angular (2+)
React.js
JavascriptCompany Profile
Quantiphi is an award-winning Applied AI and Big Data software and services company, driven by a deep desire to solve transformational problems at the heart of businesses. Our signature approach combines groundbreaking machine-learning research with disciplined cloud and data-engineering practices to create breakthrough impact at unprecedented speed.
Some company highlights:
- Quantiphi has seen 2.5x growth YoY since its inception in 2013.
- Winner of the "Machine Learning Partner of the Year" award from Google for two consecutive years - 2017 and 2018.
- Winner of the "Social Impact Partner of the Year" award from Google for 2019.
- Headquartered in Boston, with 700+ data science professionals across different offices.
For more details, visit: our http://www.quantiphi.com/">Website or our https://www.linkedin.com/company/quantiphi/">LinkedIn Page
Job Description
Role: Associate Tech Architect / Tech Architect – ReactJS +Python+AWS
Experience Level: 7-13 Years
Work location: Mumbai & Bangalore
We are looking for an experienced full stack developer( ReactJS and Python ) who can help create dynamic software applications for our clients with their skill set. In this role, you will be responsible for gathering requirements from clients and accordingly write and test scalable code, and develop front end and back-end components.
Technologies worked on:
ReactJS, Python, AWS
Requirement Description:
- Full Stack developer with experience in ReactJS, Python, API Gateway, Fargate and ECS
- Well-experienced in working with tools like Git, Maven, JFrog
- Should have a solid understanding of object-oriented programming (OOP)
- Well-experienced to perform Unit Testing and Integration Testing and have good experience in Agile based development approach
- Expertise in developing enterprise-level web applications and REST and SOAP APIs using MicroServices, with demonstrable production-scale experience
- Demonstrate strong design and programming skills using JSON, Web Services, XML, XSLT, PL/SQL in Unix and Windows environments
- Strong background working with Linux/UNIX environments and strong Shell scripting experience
- Working knowledge with SQL or No SQL databases
- Understand Architecture Requirements and ensure effective design, development, validation, and support activities
- Understanding of core AWS services, uses, and basic AWS architecture best practices
- Proficiency in developing, deploying, and debugging cloud-based applications using AWS
- Ability to use the AWS service APIs, AWS CLI, and SDKs to write applications
- Ability to identify key features of AWS services
- Identify bottlenecks and bugs, and recommend solutions by comparing the advantages and disadvantages of custom development
- Should contribute to team meetings, troubleshooting development and production problems across multiple environments and operating platforms
- Execute strong collaboration and communication skills within distributed project teams
- Continuously discover, evaluate, and implement new technologies to maximize development efficiency
jQuery
HTML/CSS
BootstrapJavascriptMust have experience on HTML5,CSS3,Jquery,Bootstrap,JS
Excellent analytical and multitasking skills
An ability to perform well in a fast-paced environment
Google AnalyticsOur client is a 5-year-old travel and hostel space providing platform, a Youth Hotel chain, where the spaces are designed for work and leisure. Their mission is to provide quality spaces for youths looking for hostel options and to other travelers across locations in India. They provide clean and hygienic dorms and private rooms along with different facilities like Wi-fi, travel helpdesk, fully stocked kitchens, lockers and laundry services, etc.
They are currently providing 800 beds in 13 locations, and plan to add another 1600 in the next few months, with the aim to add at least a lac bed in the near future. The spaces offered by them are highly rated by their customers and are preferred over other competitors for the location and services offered within a decent budget. Invested by some very well known names across Asia, it provides their team with a work culture that thrives on modern yet social and explorative values.
What you will do:
- Producing periodic reports for management and further analysing the data
- Assisting in the formulation of strategies to increase campaign efficiency
- Launching optimized online adverts through Google AdWords, Facebook to generate leads
- Optimizing ads to increase conversions and reduce CPL
- Suggesting and recommending changes in creatives and banners to improve results
- Providing ideas and collaborating with designers to improve ad experience
- Coordinating with various team members to achieve set targets
- Creating plans and objectives for campaigns
What you need to have:
- Proven experience in SEO, Email marketing and performance marketing is a must
- Understanding of digital marketing concepts and practices
- Knowledge of web analytics tool like Google Analytics
- Good content writing and communication skills
- Knowledge about ad platforms like Facebook & Google Ads
- Knowledge of MS office and Excel (Vlookup/ Pivot/ Text-to-Column/ Deduplication/ Filter/ Sorting)
- Planning, problem solving and decision-making skills
- Time management and project management skills
- Basic understanding of SEO and landing page
- Knowledge of Digital Marketing, Google Adwords, Google Analytics, Optimization and SEO management
We are scouting for an experienced candidate who has worked in Brand Activation and has an experience in managing brand campaigns effectively to promote products to generate brand recall.
The candidate must have a proven credential in managing the vendors and executing events/activations meticulously to avoid any hiccups.
We expect the candidate to also have deep understanding about the VM, In-branch branding collaterals, material and media types to efficiently implement the branch branding and manage the marketing collaterals requirements for internal stake holders.
Experience skills
- 6-8 years of experience in brand, activation, VM, negotiation, vendor management in CPG / FMCG / FMCD / BFSI.
- Excellent knowledge of media types, printing and vendor network.
- Understanding of conceptualization and execution of branding activities.
- Proven track record in brand campaign strategies
- In-depth knowledge of market catchment.
Key Responsibilities
- Developing the regional activation strategy for the company in line with the company objectives.
- Planning and implementing regional promotional campaigns in alignment with the sales team.
- Manage and execute lead generation programs.
- Stakeholder alignment on activation setting & budget sharing/planning basis Calendar.
- Co-ordinate new branch launches.
- Planning and managing small, medium and large format branch level channel partner engagement periodically in alignment with the stakeholder.
- Create value proposition for the product portfolios, on immediate and long-term basis
- Derive the regions budget for all marketing activation initiatives basis brand and business objectives in conjunction with the Brand Activation Lead.
- Scan the competitive and industry environment for potential consumer insights or marketing opportunities.
- Ensure brand compliance in all forms of communication.
- Leverage data analytics and martech tools to analyze sales, marketing, customer service, and competition trends.
Key performance Indication
On Time & In full Delivery Of:
- Ensure Budget compliance - Allocation vs actual spends, Billing Management & Vendor Management.
- Ground Activations, Catchment Visibility & Joint Business Activations - Planning, executing and monitoring on ground initiatives in line with Business objective. Initiatives focused in and around branch / micro branch catchment.
- Channel partner engagement, Branding Initiatives, Contests & R&R Meets - Planning and managing small, medium, large format and branch level channel partner engagement. End to end management of events.
- Large-scale activations – Planning, budgeting, Execution and monitoring all large-scale activations.
- Activity Efficiency - Ensure achievement of Conversion per lead generation activity for all business.
- Stakeholder Management – Ensuring stakeholder alignment and meeting on goal setting for all business.
Core Competency
- Execution & Delivery - Takes ownership providing timely and result oriented solutions with operational excellence
- Accountability - Responsible for the tasks assigned under the ambit of the key responsibilities of the role and additionally for any dependencies with cross functional teams
- Team - Work & Collaboration - Fosters a culture of trust, collaboration and support
- Effective Communication - Communicates timely and responsively with clarity and effect (both verbal and written). Seeks and provides feedback
Python
DjangoReact.js
FlaskJob Type: Full Time
Location: Permanent Remote / Bangalore / Thane
Skills: PYTHON, Django, Reactjs, Vue, Flask
Job Description
Are you passionate enough to be a crucial part of a highly analytical and scalable user engagement platform?
Are you ready learn new technologies and willing to step out of your comfort zone to explore and learn new skills?
If so, this is an opportunity for you to join a high-functioning team and make your mark on our organization!
The Impact you will create in the Job:
- Build campaign generation services which can send app notifications at a speed of 10 million a minute.
- Leverage the power of Kubernetes to maintain clusters running inside VPC's across the world.
- Dashboards to show Real time key performance indicators to clients.
- Develop complex user segmentation engines which creates segments on Terabytes of data within few seconds.
- Use AWS products extensively across our services such as Kinesis Data Streams, Lambda, SQS etc. that form an integral part of our scalable and robust infrastructure.
- As an individual contributor, you will tackle some of our broadest technical challenges that requires deep technical knowledge, hands-on software development, technical leadership, and seamless collaboration with all functions.
- You will envision and develop features that are highly reliable and fault tolerant to deliver a superior customer experience.
- You will get an opportunity to work with some of the best talent in the country while learning from various highly-functional teams in the company.
- Plan and execute the roadmap with the engineering team to build and deliver next-generation systems.
- You will also collaborate with cross-functional teams to meet deliverables throughout the software development lifecycle.
- Identify and improvise areas of improvement through data insights and research.
What we look for?
- 4-7 years’ experience in full stack Python development
- Very strong analytic and problem-solving skills
- Solid understanding of a relevant and globally supported framework- both front-end and back-end, if necessary, for example - Angular, React, Vue, or Django/Flask.
- Solid understanding of engineering best practices, continuous integration, and incremental delivery.
- Proven experience on design and implementation of the overall web and database architecture and updated knowledge on the latest practices and associated versions.
- Ensuring the entire stack is designed and built for speed and scalability.
- Familiarity with RESTful API design.
- Knowledge of continuous integration and deployment.
- Familiarity with Server management and deployment on AWS/GCE.
- Sound knowledge of CSS pre-processors, bundlers, and associated languages/syntaxes/libraries like Sass, Less, and webpack
- Experience implementing testing platforms and unit tests.
What’s in it for you?
- Immense growth, continuous learning and deliver the best to the top-notch brands
- Work with some of the most innovative brains
- Opportunity to explore your entrepreneurial mind-set
- Open culture where your creative bug gets activated.
If this sounds like a company you would like to be a part of, and a role you would thrive in, please don’t hold back from applying! We need your unique perspective for our continued innovation and success!
So let’s converse! Our inquisitive nature is all keen to know more about you.
Follow CutshortShubham Vishwakarma
Full Stack Developer - Averlon