Hadoop Developer

Location: Pune

Required Skills : Scala, Python, Data Engineering, AWS, Cassandra/AstraDB, Athena, EMR, Spark/Snowflake

• Build campaign generation services which can send app notifications at a speed of 10 million a minute
• Dashboards to show Real time key performance indicators to clients
• Develop complex user segmentation engines which creates segments on Terabytes of data within few seconds
• Building highly available & horizontally scalable platform services for ever growing data
• Use cloud based services like AWS Lambda for blazing fast throughput & auto scalability
• Work on complex analytics on terabytes of data like building Cohorts, Funnels, User path analysis, Recency Frequency & Monetary analysis at blazing speed
• You will build backend services and APIs to create scalable engineering systems.
• As an individual contributor, you will tackle some of our broadest technical challenges that requires deep technical knowledge, hands-on software development and seamless collaboration with all functions.
• You will envision and develop features that are highly reliable and fault tolerant to deliver a superior customer experience.
• Collaborating various highly-functional teams in the company to meet deliverables throughout the software development lifecycle.
• Identify and improvise areas of improvement through data insights and research.

• 2-5 years of experience in backend development and must have worked on Java/shell/Perl/python scripting.
• Solid understanding of engineering best practices, continuous integration, and incremental delivery.
• Strong analytical skills, debugging and troubleshooting skills, product line analysis.
• Follower of agile methodology (Sprint planning, working on JIRA, retrospective etc).
• Proficiency in usage of tools like Docker, Maven, Jenkins and knowledge on frameworks in Java like spring, spring boot, hibernate, JPA.
• Ability to design application modules using various concepts like object oriented, multi-threading, synchronization, caching, fault tolerance, sockets, various IPCs, database interfaces etc.
• Hands on experience on Redis, MySQL and streaming technologies like Kafka producer consumers and NoSQL databases like mongo dB/Cassandra.
• Knowledge about versioning like Git and deployment processes like CICD.

Job Details

- Job Title: Lead Software Engineer - Java, Python, API Development

- Industry: Global digital transformation solutions provider

- Domain - Information technology (IT)

- Experience Required: 8-10 years

- Employment Type: Full Time

- Job Location: Pune & Trivandrum/ Thiruvananthapuram

- CTC Range: Best in Industry

Job Description

Job Summary

We are seeking a Lead Software Engineer with strong hands-on expertise in Java and Python to design, build, and optimize scalable backend applications and APIs. The ideal candidate will bring deep experience in cloud technologies, large-scale data processing, and leading the design of high-performance, reliable backend systems.

Key Responsibilities

• Design, develop, and maintain backend services and APIs using Java and Python
• Build and optimize Java-based APIs for large-scale data processing
• Ensure high performance, scalability, and reliability of backend systems
• Architect and manage backend services on cloud platforms (AWS, GCP, or Azure)
• Collaborate with cross-functional teams to deliver production-ready solutions
• Lead technical design discussions and guide best practices

Requirements

• 8+ years of experience in backend software development
• Strong proficiency in Java and Python
• Proven experience building scalable APIs and data-driven applications
• Hands-on experience with cloud services and distributed systems
• Solid understanding of databases, microservices, and API performance optimization

Nice to Have

• Experience with Spring Boot, Flask, or FastAPI
• Familiarity with Docker, Kubernetes, and CI/CD pipelines
• Exposure to Kafka, Spark, or other big data tools

Skills

Java, Python, API Development, Data Processing, AWS Backend

Skills: Java, API development, Data Processing, AWS backend, Python,

Must-Haves

Java (8+ years), Python (8+ years), API Development (8+ years), Cloud Services (AWS/GCP/Azure), Database & Microservices

8+ years of experience in backend software development

Strong proficiency in Java and Python

Proven experience building scalable APIs and data-driven applications

Hands-on experience with cloud services and distributed systems

Solid understanding of databases, microservices, and API performance optimization

Mandatory Skills: Java API AND AWS

******

Notice period - 0 to 15 days only

Job stability is mandatory

Location: Pune, Trivandrum

Requirements

• Extensive and expert programming experience in at least one general programming language (e. g.

Java, C, C++) & tech stack to write maintainable, scalable, unit-tested code.

• Experience with multi-threading and concurrency programming.

• Extensive experience in object oriented design skills, knowledge of design patterns, and a huge passion

and ability to design intuitive modules and class-level interfaces.

• Excellent coding skills - should be able to convert design into code fluently.

• Knowledge of Test Driven Development.

• Good understanding of databases (e. g. MySQL) and NoSQL (e. g. HBase, Elasticsearch, Aerospike etc).

• Strong desire to solve complex and interesting real world problems.

• Experience with full life cycle development in any programming language on a Linux platform.

• Go-getter attitude that reflects in energy and intent behind assigned tasks.

• Worked in a startup-like environment with high levels of ownership and commitment.

• BTech, MTech or Ph. D. in Computer Science or related technical discipline (or equivalent).

• Experience in building highly scalable business applications, which involve implementing large complex

business flows and dealing with huge amounts of data.

• 3+ years of experience in the art of writing code and solving problems on a large scale.

• Open communicator who shares thoughts and opinions frequently, listens intently, and takes

constructive feedback.

Your Opportunity

• Own and drive business features into tech requirements
• Design & develop large scale real time server side systems
• Quickly create quality prototypes
• Staying updated on emerging technologies
• Ensuring that all deliverables adhere to our world class standards
• Promote coding best practices
• Mentor and develop junior developers in the team

Required Experience:

• 4+ years of relevant experience as described below
• Excellent grasp of Core Java, Multi Threading and OO design patterns
• Experience with Scala, functional, reactive programming and Akka/Play is a plus
• Excellent understanding of data structures and algorithms
• Solid grasp of large scale distributed real time systems
• Prior experience on building a scalable and resilient micro service
• Solid understanding of relational databases, NoSQL databases and Caching systems
• Good understanding of Big Data technologies such as Spark, Hadoop is a plus
• Experience on one of AWS, Azure or GCP

Who you are :

• You have excellent and effective communication and collaborative skills
• You love problem solving
• You stay up to date with the latest technologies and then apply them in real life
• You love paying attention to detail
• You thrive in meeting tight deadlines and prioritising workloads
• Ability to collaborate across multiple functions

Education:

Bachelor’s degree in Engineering or equivalent experience within the field

Be Part Of Building The Future

Dremio is the Data Lake Engine company. Our mission is to reshape the world of analytics to deliver on the promise of data with a fundamentally new architecture, purpose-built for the exploding trend towards cloud data lake storage such as AWS S3 and Microsoft ADLS. We dramatically reduce and even eliminate the need for the complex and expensive workarounds that have been in use for decades, such as data warehouses (whether on-premise or cloud-native), structural data prep, ETL, cubes, and extracts. We do this by enabling lightning-fast queries directly against data lake storage, combined with full self-service for data users and full governance and control for IT. The results for enterprises are extremely compelling: 100X faster time to insight; 10X greater efficiency; zero data copies; and game-changing simplicity. And equally compelling is the market opportunity for Dremio, as we are well on our way to disrupting a $25BN+ market.

About the Role

The Dremio India team owns the DataLake Engine along with Cloud Infrastructure and services that power it. With focus on next generation data analytics supporting modern table formats like Iceberg, Deltalake, and open source initiatives such as Apache Arrow, Project Nessie and hybrid-cloud infrastructure, this team provides various opportunities to learn, deliver, and grow in career. We are looking for innovative minds with experience in leading and building high quality distributed systems at massive scale and solving complex problems.

Responsibilities & ownership

• Lead, build, deliver and ensure customer success of next-generation features related to scalability, reliability, robustness, usability, security, and performance of the product.
• Work on distributed systems for data processing with efficient protocols and communication, locking and consensus, schedulers, resource management, low latency access to distributed storage, auto scaling, and self healing.
• Understand and reason about concurrency and parallelization to deliver scalability and performance in a multithreaded and distributed environment.
• Lead the team to solve complex and unknown problems 
• Solve technical problems and customer issues with technical expertise
• Design and deliver architectures that run optimally on public clouds like  GCP, AWS, and Azure
• Mentor other team members for high quality and design 
• Collaborate with Product Management to deliver on customer requirements and innovation
• Collaborate with Support and field teams to ensure that customers are successful with Dremio

Requirements

• B.S./M.S/Equivalent in Computer Science or a related technical field or equivalent experience
• Fluency in Java/C++ with 8+ years of experience developing production-level software
• Strong foundation in data structures, algorithms, multi-threaded and asynchronous programming models, and their use in developing distributed and scalable systems
• 5+ years experience in developing complex and scalable distributed systems and delivering, deploying, and managing microservices successfully
• Hands-on experience  in query processing or optimization, distributed systems, concurrency control, data replication, code generation, networking, and storage systems
• Passion for quality, zero downtime upgrades, availability, resiliency, and uptime of the platform
• Passion for learning and delivering using latest technologies
• Ability to solve ambiguous, unexplored, and cross-team problems effectively
• Hands on experience of working projects on AWS, Azure, and Google Cloud Platform 
• Experience with containers and Kubernetes for orchestration and container management in private and public clouds (AWS, Azure, and Google Cloud) 
• Understanding of distributed file systems such as  S3, ADLS, or HDFS
• Excellent communication skills and affinity for collaboration and teamwork
• Ability to work individually and collaboratively with other team members
• Ability to scope and plan solution for  big problems and mentors others on the same
• Interested and motivated to be part of a fast-moving startup with a fun and accomplished team

Spark / Scala experience should be more than 2 years.

Combination with Java & Scala is fine or we are even fine with Big Data Developer with strong Core Java Concepts. - Scala / Spark Developer.

Strong proficiency Scala on Spark (Hadoop) - Scala + Java is also preferred

Complete SDLC process and Agile Methodology (Scrum)

Version control / Git