50+ Big data Jobs in Pune | Big data Job openings in Pune

True Hands-On Developer in Programming Languages like Java or Scala . Expertise in Apache Spark . Database modelling and working with any of the SQL or NoSQL Database is must. Working knowledge of Scripting languages like shell/python. Experience of working with Cloudera is Preferred Orchestration tools like Airflow or Oozie would be a value addition. Knowledge of Table formats like Delta or Iceberg is plus to have. Working experience of Version controls like Git, build tools like Maven is recommended. Having software development experience is good to have along with Data Engineering experience

AccioJob is conducting a Walk-In Hiring Drive with a reputed global IT consulting company at AccioJob Skill Centres for the position of Data Engineer, specifically for female candidates.

To Apply, Register and select your Slot here: https://go.acciojob.com/8p9ZXN

We will not consider your application if you do not register and select slot via the above link.

Required Skills: Python, Database(MYSQL), Big Data(Spark, Kafka)

Eligibility:

• Degree: B.Tech/BE
• Branch: CSE – AI & DS / AI & ML
• Graduation Year: 2024 & 2025

Note: Only Female Candidates can apply for this job opportunity

Work Details:

• Work Mode: Work From Office
• Work Location: Bangalore & Coimbatore
• CTC: 11.1 LPA

Evaluation Process:

• Round 1: Offline Assessment at AccioJob Skill Centre in Noida, Pune, Hyderabad.

• Further Rounds (for Shortlisted Candidates only)

1. HackerRank Online Assessment
2. Coding Pairing Interview
3. Technical Interview
4. Cultural Alignment Interview

Important Note: Please bring your laptop and earphones for the test.

Register here: https://go.acciojob.com/8p9ZXN

Dear Candidate,

We are urgently looking for a Release- Big data Engineer For Pune Location.

Experience : 5-8 yrs

Location : Pune

Skills: Big data Engineer , Release Engineer ,DevOps, Aws/Azure/GCP Cloud exp. ,

JD:

• Oversee the end-to-end release lifecycle, from planning to post-production monitoring. Coordinate with cross-functional teams (DBA, BizOps, DevOps, DNS).
• Partner with development teams to resolve technical challenges in deployment and automation test runs
• Work with shared services DBA teams for schema-based multi-tenancy designs and smooth migrations.
• Drive automation for batch deployments and DR exercises. YAML based micro service deployment using shell/Python/Go
• Provide oversight for Big Data toolsets for deployment (e.g., Spark, Hive, HBase) in private cloud and public cloud CDP environments
• Ensure high-quality releases with a focus on stability and long-term performance.
• Able to run the automation batch scripts and debug the deployment and functional aspects/ work with dev leads to resolve the release cycle issues.

Regards,

Minakshi Soni

Executive- Talent Acquisition

Rigel Networks

At least 5 years of experience in testing and developing automation tests.

A minimum of 3 years of experience writing tests in Python, with a preference for experience in designing automation frameworks.

Experience in developing automation for big data testing, including data ingestion, data processing, and data migration, is highly desirable.

Familiarity with Playwright or other browser application testing frameworks is a significant advantage.

Proficiency in object-oriented programming and principles is required.

Extensive knowledge of AWS services is essential.

Strong expertise in REST API testing and SQL is required.

A solid understanding of testing and development life cycle methodologies is necessary.

Knowledge of the financial industry and trading systems is a plus

Job Summary:

We are seeking a skilled Senior Data Engineer with expertise in application programming, big data technologies, and cloud services. This role involves solving complex problems, designing scalable systems, and working with advanced technologies to deliver innovative solutions.

Key Responsibilities:

• Develop and maintain scalable applications using OOP principles, data structures, and problem-solving skills.
• Build robust solutions using Java, Python, or Scala.
• Work with big data technologies like Apache Spark for large-scale data processing.
• Utilize AWS services, especially Amazon Redshift, for cloud-based solutions.
• Manage databases including SQL, NoSQL (e.g., MongoDB, Cassandra), with Snowflake as a plus.

Qualifications:

• 5+ years of experience in software development.
• Strong skills in OOPS, data structures, and problem-solving.
• Proficiency in Java, Python, or Scala.
• Experience with Spark, AWS (Redshift mandatory), and databases (SQL/NoSQL).
• Snowflake experience is good to have.

At DocNexus, we’re revolutionizing how life sciences companies search and generate insights. Our search platform unlocks powerful insights, and we're seeking a Customer Success Team Member with strong technical skills to help our customers harness its full potential.

What you’ll do:

• Customer Support: Troubleshoot and resolve customer queries, particularly around referral reports, data anomalies, and data generation using our platform.
• Data Queries (BigQuery/ClickHouse): Respond to customer requests for custom data queries, working with large datasets in BigQuery and ClickHouse to deliver precise insights.
• Onboarding & Training: Lead onboarding for new customers, guide teams on platform usage, and manage access requests.
• Listen & Improve: Collect and act on customer feedback to continuously improve the platform, collaborating with the product team to enhance functionality.
• Technical Documentation: Assist with technical resources and help create training materials for both internal and customer use.

What you bring:

• Strong Technical Skills: Proficient in querying with BigQuery and ClickHouse. Comfortable working with complex data, writing custom queries, and resolving technical issues.
• Customer-Focused: Excellent communication skills, able to translate technical data insights to non-technical users and provide solutions clearly and effectively.
• Problem-Solver: Strong analytical skills and a proactive mindset to address customer needs and overcome challenges in a fast-paced environment.
• Team Player: Work collaboratively with both internal teams and customers to ensure success.

If you're passionate about data, thrive in a technical environment, and are excited to support life sciences teams in their data-driven decision-making, we'd love to hear from you!

Skills:

Experience with Cassandra, including installing configuring and monitoring a Cassandra cluster.

Experience with Cassandra data modeling and CQL scripting. Experience with DataStax Enterprise Graph

Experience with both Windows and Linux Operating Systems. Knowledge of Microsoft .NET Framework (C#, NETCore).

Ability to perform effectively in a team-oriented environment

TVARIT GmbH develops and delivers solutions in the field of artificial intelligence (AI) for the Manufacturing, automotive, and process industries. With its software products, TVARIT makes it possible for its customers to make intelligent and well-founded decisions, e.g., in forward-looking Maintenance, increasing the OEE and predictive quality. We have renowned reference customers, competent technology, a good research team from renowned Universities, and the award of a renowned AI prize (e.g., EU Horizon 2020) which makes Tvarit one of the most innovative AI companies in Germany and Europe. 

We are looking for a self-motivated person with a positive "can-do" attitude and excellent oral and written communication skills in English. 

We are seeking a skilled and motivated Data Engineer from the manufacturing Industry with over two years of experience to join our team. As a data engineer, you will be responsible for designing, building, and maintaining the infrastructure required for the collection, storage, processing, and analysis of large and complex data sets. The ideal candidate will have a strong foundation in ETL pipelines and Python, with additional experience in Azure and Terraform being a plus. This role requires a proactive individual who can contribute to our data infrastructure and support our analytics and data science initiatives. 

Skills Required 

• Experience in the manufacturing industry (metal industry is a plus)  
• 2+ years of experience as a Data Engineer 
• Experience in data cleaning & structuring and data manipulation 
• ETL Pipelines: Proven experience in designing, building, and maintaining ETL pipelines. 
• Python: Strong proficiency in Python programming for data manipulation, transformation, and automation. 
• Experience in SQL and data structures  
• Knowledge in big data technologies such as Spark, Flink, Hadoop, Apache and NoSQL databases. 
• Knowledge of cloud technologies (at least one) such as AWS, Azure, and Google Cloud Platform. 
• Proficient in data management and data governance  
• Strong analytical and problem-solving skills. 
• Excellent communication and teamwork abilities. 

Nice To Have 

• Azure: Experience with Azure data services (e.g., Azure Data Factory, Azure Databricks, Azure SQL Database). 
• Terraform: Knowledge of Terraform for infrastructure as code (IaC) to manage cloud. 

TVARIT GmbH develops and delivers solutions in the field of artificial intelligence (AI) for the Manufacturing, automotive, and process industries. With its software products, TVARIT makes it possible for its customers to make intelligent and well-founded decisions, e.g., in forward-looking Maintenance, increasing the OEE and predictive quality. We have renowned reference customers, competent technology, a good research team from renowned Universities, and the award of a renowned AI prize (e.g., EU Horizon 2020) which makes TVARIT one of the most innovative AI companies in Germany and Europe.

We are looking for a self-motivated person with a positive "can-do" attitude and excellent oral and written communication skills in English.

We are seeking a skilled and motivated senior Data Engineer from the manufacturing Industry with over four years of experience to join our team. The Senior Data Engineer will oversee the department’s data infrastructure, including developing a data model, integrating large amounts of data from different systems, building & enhancing a data lake-house & subsequent analytics environment, and writing scripts to facilitate data analysis. The ideal candidate will have a strong foundation in ETL pipelines and Python, with additional experience in Azure and Terraform being a plus. This role requires a proactive individual who can contribute to our data infrastructure and support our analytics and data science initiatives.

Skills Required:

• Experience in the manufacturing industry (metal industry is a plus)
• 4+ years of experience as a Data Engineer
• Experience in data cleaning & structuring and data manipulation
• Architect and optimize complex data pipelines, leading the design and implementation of scalable data infrastructure, and ensuring data quality and reliability at scale
• ETL Pipelines: Proven experience in designing, building, and maintaining ETL pipelines.
• Python: Strong proficiency in Python programming for data manipulation, transformation, and automation.
• Experience in SQL and data structures
• Knowledge in big data technologies such as Spark, Flink, Hadoop, Apache, and NoSQL databases.
• Knowledge of cloud technologies (at least one) such as AWS, Azure, and Google Cloud Platform.
• Proficient in data management and data governance
• Strong analytical experience & skills that can extract actionable insights from raw data to help improve the business.
• Strong analytical and problem-solving skills.
• Excellent communication and teamwork abilities.

Nice To Have:

• Azure: Experience with Azure data services (e.g., Azure Data Factory, Azure Databricks, Azure SQL Database).
• Terraform: Knowledge of Terraform for infrastructure as code (IaC) to manage cloud.
• Bachelor’s degree in computer science, Information Technology, Engineering, or a related field from top-tier Indian Institutes of Information Technology (IIITs).
• Benefits And Perks
• A culture that fosters innovation, creativity, continuous learning, and resilience
• Progressive leave policy promoting work-life balance
• Mentorship opportunities with highly qualified internal resources and industry-driven programs
• Multicultural peer groups and supportive workplace policies
• Annual workcation program allowing you to work from various scenic locations
• Experience the unique environment of a dynamic start-up

Why should you join TVARIT ?

Working at TVARIT, a deep-tech German IT startup, offers a unique blend of innovation, collaboration, and growth opportunities. We seek individuals eager to adapt and thrive in a rapidly evolving environment.

If this opportunity excites you and aligns with your career aspirations, we encourage you to apply today!

• KSQL
• Data Engineering spectrum (Java/Spark)
• Spark Scala / Kafka Streaming
• Confluent Kafka components
• Basic understanding of Hadoop

Title: Cloud DevOps Engineer 

Role: Individual Contributor (4-8 yrs)  

Requirements: 

• Energetic self-starter, a fast learner, with a desire to work in a startup environment  
• Experience working with Public Clouds like AWS 

• Operating and Monitoring cloud infrastructure on AWS. 
• Primary focus on building, implementing and managing operational support 
• Design, Develop and Troubleshoot Automation scripts (Configuration/Infrastructure as code or others) for Managing Infrastructure. 
• Expert at one of the scripting languages – Python, shell, etc  
• Experience with Nginx/HAProxy, ELK Stack, Ansible, Terraform, Prometheus-Grafana stack, etc 

• Handling load monitoring, capacity planning, and services monitoring. 
• Proven experience With CICD Pipelines and Handling Database Upgrade Related Issues. 
• Good Understanding and experience in working with Containerized environments like Kubernetes and Datastores like Cassandra, Elasticsearch, MongoDB, etc

1. Data Engineer

 Required skill set: AWS GLUE, AWS LAMBDA, AWS SNS/SQS, AWS ATHENA, SPARK, SNOWFLAKE, PYTHON

Mandatory Requirements  

• Experience in AWS Glue
• Experience in Apache Parquet 
• Proficient in AWS S3 and data lake 
• Knowledge of Snowflake
• Understanding of file-based ingestion best practices.
• Scripting language - Python & pyspark 

CORE RESPONSIBILITIES 

• Create and manage cloud resources in AWS 
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 
• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
• Define process improvement opportunities to optimize data collection, insights and displays.
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 
• Identify and interpret trends and patterns from complex data sets 
• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 
• Key participant in regular Scrum ceremonies with the agile teams  
• Proficient at developing queries, writing reports and presenting findings 
• Mentor junior members and bring best industry practices 

 QUALIFICATIONS 

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 
• Strong background in math, statistics, computer science, data science or related discipline
• Advanced knowledge one of language: Java, Scala, Python, C# 
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  
• Proficient with
• Data mining/programming tools (e.g. SAS, SQL, R, Python)
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
• Data visualization (e.g. Tableau, Looker, MicroStrategy)
• Comfortable learning about and deploying new technologies and tools. 
• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 
• Good written and oral communication skills and ability to present results to non-technical audiences 
• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus:  

• AWS certification
• Spark Streaming 
• Kafka Streaming / Kafka Connect 
• ELK Stack 
• Cassandra / MongoDB 
• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Technical/Core skills

1. Minimum 3 yrs of exp in Informatica Big data Developer(BDM) in Hadoop environment.
2. Have knowledge of informatica Power exchange (PWX).
3. Minimum 3 yrs of exp in big data querying tool like Hive and Impala.
4. Ability to designing/development of complex mappings using informatica Big data Developer.
5. Create and manage Informatica power exchange and CDC real time implementation
6. Strong Unix knowledge skills for writing shell scripts and troubleshoot of existing scripts.
7. Good knowledge of big data platforms and its framework.
8. Good to have an experience in cloudera data platform (CDP)
9. Experience with building stream processing systems using Kafka and spark
10. Excellent SQL knowledge

Soft skills :

1. Ability to work independently 
2. Strong analytical and problem solving skills
3. Attitude of learning new technology
4. Regular interaction with vendors, partners and stakeholders

Job Title: Product Manager

Job Description

Bachelor or master’s degree in computer science or equivalent experience.
Worked as Product Owner before and took responsibility for a product or project delivery.
Well-versed with data warehouse modernization to Big Data and Cloud environments.
Good knowledge* of any of the Cloud (AWS/Azure/GCP) – Must Have
Practical experience with continuous integration and continuous delivery workflows.
Self-motivated with strong organizational/prioritization skills and ability to multi-task with close attention to detail.
Good communication skills
Experience in working within a distributed agile team
Experience in handling migration projects – Good to Have
 

*Data Ingestion, Processing, and Orchestration knowledge

Roles & Responsibilities

Responsible for coming up with innovative and novel ideas for the product.
Define product releases, features, and roadmap.
Collaborate with product teams on defining product objectives, including creating a product roadmap, delivery, market research, customer feedback, and stakeholder inputs.
Work with the Engineering teams to communicate release goals and be a part of the product lifecycle. Work closely with the UX and UI team to create the best user experience for the end customer.
Work with the Marketing team to define GTM activities.
Interface with Sales & Customer teams to identify customer needs and product gaps
Market and competition analysis activities.
Participate in the Agile ceremonies with the team, define epics, user stories, acceptance criteria
Ensure product usability from the end-user perspective

Mandatory Skills

Product Management, DWH, Big Data

4-5 years of overall experience in software development.
- Experience on Hadoop (Apache/Cloudera/Hortonworks) and/or other Map Reduce Platforms
- Experience on Hive, Pig, Sqoop, Flume and/or Mahout
- Experience on NO-SQL – HBase, Cassandra, MongoDB
- Hands on experience with Spark development,  Knowledge of Storm, Kafka, Scala
- Good knowledge of Java
- Good background of Configuration Management/Ticketing systems like Maven/Ant/JIRA etc.
- Knowledge around any Data Integration and/or EDW tools is plus
- Good to have knowledge of  using Python/Perl/Shell

Please note - Hbase hive and spark are must.

Company Profile:

Easebuzz is a payment solutions (fintech organisation) company which enables online merchants to accept, process and disburse payments through developer friendly APIs. We are focusing on building plug n play products including the payment infrastructure to solve complete business problems. Definitely a wonderful place where all the actions related to payments, lending, subscription, eKYC is happening at the same time.

We have been consistently profitable and are constantly developing new innovative products, as a result, we are able to grow 4x over the past year alone. We are well capitalised and have recently closed a fundraise of $4M in March, 2021 from prominent VC firms and angel investors. The company is based out of Pune and has a total strength of 180 employees. Easebuzz’s corporate culture is tied into the vision of building a workplace which breeds open communication and minimal bureaucracy. An equal opportunity employer, we welcome and encourage diversity in the workplace. One thing you can be sure of is that you will be surrounded by colleagues who are committed to helping each other grow.

Easebuzz Pvt. Ltd. has its presence in Pune, Bangalore, Gurugram.

Salary: As per company standards.

Designation: Data Engineering

Location: Pune

Experience with ETL, Data Modeling, and Data Architecture

Design, build and operationalize large scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with 3rd parties
- Spark, EMR, DynamoDB, RedShift, Kinesis, Lambda, Glue.

Experience with AWS cloud data lake for development of real-time or near real-time use cases

Experience with messaging systems such as Kafka/Kinesis for real time data ingestion and processing

Build data pipeline frameworks to automate high-volume and real-time data delivery

Create prototypes and proof-of-concepts for iterative development.

Experience with NoSQL databases, such as DynamoDB, MongoDB etc

Create and maintain optimal data pipeline architecture,

Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.

Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.

Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.

Keep our data separated and secure across national boundaries through multiple data centers and AWS regions.

Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.

Evangelize a very high standard of quality, reliability and performance for data models and algorithms that can be streamlined into the engineering and sciences workflow

Build and enhance data pipeline architecture by designing and implementing data ingestion solutions.

Employment Type

Full-time

Key Responsibilities : ( Data Developer Python, Spark)

Exp : 2 to 9 Yrs 

Development of data platforms, integration frameworks, processes, and code.

Develop and deliver APIs in Python or Scala for Business Intelligence applications build using a range of web languages

Develop comprehensive automated tests for features via end-to-end integration tests, performance tests, acceptance tests and unit tests.

Elaborate stories in a collaborative agile environment (SCRUM or Kanban)

Familiarity with cloud platforms like GCP, AWS or Azure.

Experience with large data volumes.

Familiarity with writing rest-based services.

Experience with distributed processing and systems

Experience with Hadoop / Spark toolsets

Experience with relational database management systems (RDBMS)

Experience with Data Flow development

Knowledge of Agile and associated development techniques including:

n 

• Strong Python Coding skills and OOP skills
• Should have worked on Big Data product Architecture
• Should have worked with any one of the SQL-based databases like MySQL, PostgreSQL and any one of
• NoSQL-based databases such as Cassandra, Elasticsearch etc.
• Hands on experience on frameworks like Spark RDD, DataFrame, Dataset
• Experience on development of ETL for data product
• Candidate should have working knowledge on performance optimization, optimal resource utilization, Parallelism and tuning of spark jobs
• Working knowledge on file formats: CSV, JSON, XML, PARQUET, ORC, AVRO
• Good to have working knowledge with any one of the Analytical Databases like Druid, MongoDB, Apache Hive etc.
• Experience to handle real-time data feeds (good to have working knowledge on Apache Kafka or similar tool)

• Python and Scala (Optional), Spark / PySpark, Parallel programming

Job Role : Associate Manager (Database Development)

Key Responsibilities:

• Optimizing performances of many stored procedures, SQL queries to deliver big amounts of data under a few seconds.
• Designing and developing numerous complex queries, views, functions, and stored procedures
• to work seamlessly with the Application/Development team’s data needs.
• Responsible for providing solutions to all data related needs to support existing and new
• applications.
• Creating scalable structures to cater to large user bases and manage high workloads
• Responsible in every step from the beginning stages of the projects from requirement gathering to implementation and maintenance.
• Developing custom stored procedures and packages to support new enhancement needs.
• Working with multiple teams to design, develop and deliver early warning systems.
• Reviewing query performance and optimizing code
• Writing queries used for front-end applications
• Designing and coding database tables to store the application data
• Data modelling to visualize database structure
• Working with application developers to create optimized queries
• Maintaining database performance by troubleshooting problems.
• Accomplishing platform upgrades and improvements by supervising system programming.
• Securing database by developing policies, procedures, and controls.
• Designing and managing deep statistical systems.

Desired Skills and Experience  :

• 7+ years of experience in database development
• Minimum 4+ years of experience in PostgreSQL is a must
• Experience and in-depth knowledge in PL/SQL
• Ability to come up with multiple possible ways of solving a problem and deciding on the most optimal approach for implementation that suits the work case the most
• Have knowledge of Database Administration and have the ability and experience of using the CLI tools for administration
• Experience in Big Data technologies is an added advantage
• Secondary platforms: MS SQL 2005/2008, Oracle, MySQL
• Ability to take ownership of tasks and flexibility to work individually or in team
• Ability to communicate with teams and clients across time zones and global regions
• Good communication and self-motivated
• Should have the ability to work under pressure
• Knowledge of NoSQL and Cloud Architecture will be an advantage

1. Sr. Data Engineer:

 Core Skills – Data Engineering, Big Data, Pyspark, Spark SQL and Python

Candidate with prior Palantir Cloud Foundry OR Clinical Trial Data Model background is preferred

Major accountabilities:

• Responsible for Data Engineering, Foundry Data Pipeline Creation, Foundry Analysis & Reporting, Slate Application development, re-usable code development & management and Integrating Internal or External System with Foundry for data ingestion with high quality.
• Have good understanding on Foundry Platform landscape and it’s capabilities
• Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
• Defines company data assets (data models), Pyspark, spark SQL, jobs to populate data models.
• Designs data integrations and data quality framework.
• Design & Implement integration with Internal, External Systems, F1 AWS platform using Foundry Data Connector or Magritte Agent
• Collaboration with data scientists, data analyst and technology teams to document and leverage their understanding of the Foundry integration with different data sources - Actively participate in agile work practices
• Coordinating with Quality Engineer to ensure the all quality controls, naming convention & best practices have been followed

Desired Candidate Profile :

• Strong data engineering background
• Experience with Clinical Data Model is preferred
• Experience in
• SQL Server ,Postgres, Cassandra, Hadoop, and Spark for distributed data storage and parallel computing
• Java and Groovy for our back-end applications and data integration tools
• Python for data processing and analysis
• Cloud infrastructure based on AWS EC2 and S3
• 7+ years IT experience, 2+ years’ experience in Palantir Foundry Platform, 4+ years’ experience in Big Data platform
• 5+ years of Python and Pyspark development experience
• Strong troubleshooting and problem solving skills
• BTech or master's degree in computer science or a related technical field
• Experience designing, building, and maintaining big data pipelines systems
• Hands-on experience on Palantir Foundry Platform and Foundry custom Apps development
• Able to design and implement data integration between Palantir Foundry and external Apps based on Foundry data connector framework
• Hands-on in programming languages primarily Python, R, Java, Unix shell scripts
• Hand-on experience in AWS / Azure cloud platform and stack
• Strong in API based architecture and concept, able to do quick PoC using API integration and development
• Knowledge of machine learning and AI
• Skill and comfort working in a rapidly changing environment with dynamic objectives and iteration with users.

 Demonstrated ability to continuously learn, work independently, and make decisions with minimal supervision

Summary
Our Kafka developer has a combination of technical skills, communication skills and business knowledge. The developer should be able to work on multiple medium to large projects. The successful candidate will have excellent technical skills of Apache/Confluent Kafka, Enterprise Data WareHouse preferable GCP BigQuery or any equivalent Cloud EDW and also will be able to take oral and written business requirements and develop efficient code to meet set deliverables.

Must Have Skills

• Participate in the development, enhancement and maintenance of data applications both as an individual contributor and as a lead.
• Leading in the identification, isolation, resolution and communication of problems within the production environment.
• Leading developer and applying technical skills Apache/Confluent Kafka (Preferred) AWS Kinesis (Optional), Cloud Enterprise Data Warehouse Google BigQuery (Preferred) or AWS RedShift or SnowFlakes (Optional)
• Design recommending best approach suited for data movement from different sources to Cloud EDW using Apache/Confluent Kafka
• Performs independent functional and technical analysis for major projects supporting several corporate initiatives.
• Communicate and Work with IT partners and user community with various levels from Sr Management to detailed developer to business SME for project definition .
• Works on multiple platforms and multiple projects concurrently.
• Performs code and unit testing for complex scope modules, and projects
• Provide expertise and hands on experience working on Kafka connect using schema registry in a very high volume environment (~900 Million messages)

• Provide expertise in Kafka brokers, zookeepers, KSQL, KStream and Kafka Control center.
• Provide expertise and hands on experience working on AvroConverters, JsonConverters, and StringConverters.
• Provide expertise and hands on experience working on Kafka connectors such as MQ connectors, Elastic Search connectors, JDBC connectors, File stream connector,  JMS source connectors, Tasks, Workers, converters, Transforms.
• Provide expertise and hands on experience on custom connectors using the Kafka core concepts and API.
• Working knowledge on Kafka Rest proxy.
• Ensure optimum performance, high availability and stability of solutions.
• Create topics, setup redundancy cluster, deploy monitoring tools, alerts and has good knowledge of best practices.
• Create stubs for producers, consumers and consumer groups for helping onboard applications from different languages/platforms.  Leverage Hadoop ecosystem knowledge to design, and develop capabilities to deliver our solutions using Spark, Scala, Python, Hive, Kafka and other things in the Hadoop ecosystem. 
• Use automation tools like provisioning using Jenkins, Udeploy or relevant technologies
• Ability to perform data related benchmarking, performance analysis and tuning.
• Strong skills in In-memory applications, Database Design, Data Integration.

• Drive and participate in requirements gathering workshops, estimation discussions, design meetings and status review meetings
• Participate and contribute in Solution Design and Solution Architecture for implementing Big Data Projects on-premise and on cloud
• Technical Hands on experience in design, coding, development and managing Large Hadoop implementation
• Proficient in SQL, Hive, PIG, Spark SQL, Shell Scripting, Kafka, Flume, Scoop with large Big Data and Data Warehousing projects with either Java, Python or Scala based Hadoop programming background
• Proficient with various development methodologies like waterfall, agile/scrum and iterative
• Good Interpersonal skills and excellent communication skills for US and UK based clients

• Core Java: advanced level competency, should have worked on projects with core Java development.

• Linux shell : advanced level competency, work experience with Linux shell scripting, knowledge and experience to use important shell commands

• Rdbms, SQL: advanced level competency, Should have expertise in SQL query language syntax, should be well versed with aggregations, joins of SQL query language.

• Data structures and problem solving: should have ability to use appropriate data structure.

• AWS cloud : Good to have experience with aws serverless toolset along with aws infra

• Data Engineering ecosystem : Good to have experience and knowledge of data engineering, ETL, data warehouse (any toolset)

• Hadoop, HDFS, YARN : Should have introduction to internal working of these toolsets

• HIVE, MapReduce, Spark: Good to have experience developing transformations using hive queries, MapReduce job implementation and Spark Job Implementation. Spark implementation in Scala will be plus point.

• Airflow, Oozie, Sqoop, Zookeeper, Kafka: Good to have knowledge about purpose and working of these technology toolsets. Working experience will be a plus point here.

Datametica is looking for talented SQL engineers who would get training & the opportunity to work on Cloud and Big Data Analytics.

Mandatory Skills:

• Strong in SQL development
• Hands-on at least one scripting language - preferably shell scripting
• Development experience in Data warehouse projects

Opportunities:

• Selected candidates will be provided training opportunities on one or more of the following: Google Cloud, AWS, DevOps Tools, Big Data technologies like Hadoop, Pig, Hive, Spark, Sqoop, Flume, and KafkaWould get a chance to be part of the enterprise-grade implementation of Cloud and Big Data systems
• Will play an active role in setting up the Modern data platform based on Cloud and Big Data
• Would be part of teams with rich experience in various aspects of distributed systems and computing

Job Description

Experience : 10+ Years

Location : Pune

Job Requirements:

• Minimum of 10+ years of experience with a proven record of increased responsibility
• Hands on experience in design, development and managing Big Data, Cloud, Data warehousing
• and Business Intelligence projects
• Experience of managing projects in Big Data, Cloud, Data warehousing, Business Intelligence
• Using open source or top of the line tools and technologies
• Good knowledge of Dimensional Modeling
• Experience of working with any ETL and BI Reporting tools
• Experience of managing medium to large projects, preferably on Big Data
• Proven experience in project planning, estimation, execution and implementation of medium to
• large projects
• Should be able to effectively communicate in English
• Strong management and leadership skills, with proven ability to develop and manage client
• relationships
• Proven problem-solving skills from both technical and managerial perspectives
• Attention to detail and a commitment to excellence and high standards
• Excellent interpersonal and communication skills, both verbal and written
• Position is remote with occasional travel to other offices, client sites, conventions, training
• locations, etc.
• Bachelor’s degree in Computer Science, Business\Economics, or a related field or demonstrated,
• equivalent/practical knowledge or experience

Job Responsibilities:

• Day to day project management, scrum and agile management including project planning, delivery
• and execution of Big Data and
• Primary Point of contact for customer related to all project engagements, delivery and project
• escalations
• Design right architecture and technology stack depending on business requirement on Cloud / Big
• Data and BI related technologies both some on-premise and on cloud
• Liaise with key stakeholders to define the Cloud / Big data solutions roadmap, prioritize the
• deliverables
• Responsible for end to end project delivery of Cloud / Big Data Solutions from project estimations,
• project planning, resourcing and monitoring perspective
• Drive and participate in requirements gathering workshops, estimation discussions, design
• meetings and status review meetings
• Support & assist the team in resolving issues during testing and when the system is in production
• Involved in the full customer lifecycle with a goal to make customers successful and increase
• revenue and retention
• Interface with the offshore engineering team to solve customer issues
• Develop programs that meet customer needs with respect to functionality, performance,
• scalability, reliability, schedule, principles and recognized industry standards
• Requirement analysis and documentation
• Manage day-to-day operational aspects of a project and scope
• Prepare for engagement reviews and quality assurance procedures
• Visit and/or host clients to strengthen business relationships

Primary responsibilities:

• Architect, Design and Build high performance Search systems for personalization, optimization, and targeting
• Designing systems with Solr, Akka, Cassandra, Kafka
• Algorithmic development with primary focus Machine Learning
• Working with rapid and innovative development methodologies like: Kanban, Continuous Integration and Daily deployments
• Participation in design and code reviews and recommend improvements
• Unit testing with JUnit, Performance testing and tuning
• Coordination with internal and external teams
• Mentoring junior engineers
• Participate in Product roadmap and Prioritization discussions and decisions
• Evangelize the solution with Professional services and Customer Success teams

ears of Exp: 3-6+ Years 
Skills: Scala, Python, Hive, Airflow, Spark

Languages: Java, Python, Shell Scripting

GCP: BigTable, DataProc,  BigQuery, GCS, Pubsub

OR
AWS: Athena, Glue, EMR, S3, Redshift

MongoDB, MySQL, Kafka

Platforms: Cloudera / Hortonworks
AdTech domain experience is a plus.
Job Type - Full Time 

Location:  Pune/Nagpur,Goa,Hyderabad/Bangalore

Job Requirements:

• 9 years and above of total experience preferably in bigdata space.
• Creating spark applications using Scala to process data.
• Experience in scheduling and troubleshooting/debugging Spark jobs in steps.
• Experience in spark job performance tuning and optimizations.
• Should have experience in processing data using Kafka/Pyhton.
• Individual should have experience and understanding in configuring Kafka topics to optimize the performance.
• Should be proficient in writing SQL queries to process data in Data Warehouse.
• Hands on experience in working with Linux commands to troubleshoot/debug issues and creating shell scripts to automate tasks.
• Experience on AWS services like EMR.

Role Summary/Purpose:

We are looking for a Developer/Senior Developers to be a part of building advanced analytical platform leveraging Big Data technologies and transform the legacy systems. This role is an exciting, fast-paced, constantly changing and challenging work environment, and will play an important role in resolving and influencing high-level decisions.

Requirements:

• The candidate must be a self-starter, who can work under general guidelines in a fast-spaced environment.
• Overall minimum of 4 to 8 year of software development experience and 2 years in Data Warehousing domain knowledge
• Must have 3 years of hands-on working knowledge on Big Data technologies such as Hadoop, Hive, Hbase, Spark, Kafka, Spark Streaming, SCALA etc…
• Excellent knowledge in SQL & Linux Shell scripting
• Bachelors/Master’s/Engineering Degree from a well-reputed university.
• Strong communication, Interpersonal, Learning and organizing skills matched with the ability to manage stress, Time, and People effectively
• Proven experience in co-ordination of many dependencies and multiple demanding stakeholders in a complex, large-scale deployment environment
• Ability to manage a diverse and challenging stakeholder community
• Diverse knowledge and experience of working on Agile Deliveries and Scrum teams.

Responsibilities

• Should works as a senior developer/individual contributor based on situations
• Should be part of SCRUM discussions and to take requirements
• Adhere to SCRUM timeline and deliver accordingly
• Participate in a team environment for the design, development and implementation
• Should take L3 activities on need basis
• Prepare Unit/SIT/UAT testcase and log the results
• Co-ordinate SIT and UAT Testing. Take feedbacks and provide necessary remediation/recommendation in time.
• Quality delivery and automation should be a top priority
• Co-ordinate change and deployment in time
• Should create healthy harmony within the team
• Owns interaction points with members of core team (e.g.BA team, Testing and business team) and any other relevant stakeholders

• 4-10 years of experience in software development.
• At least 2 years of relevant work experience on large scale Data applications.
• Strong coding experience in Java is mandatory
• Good aptitude, strong problem solving abilities, and analytical skills, ability to take ownership as appropriate
• Should be able to do coding, debugging, performance tuning and deploying the apps to Prod.
• Should have good working experience on
• o Hadoop ecosystem (HDFS, Hive, Yarn, File formats like Avro/Parquet)
• o Kafka
• o J2EE Frameworks (Spring/Hibernate/REST)
• o Spark Streaming or any other streaming technology.
• Strong coding experience in Java is mandatory
• Ability to work on the sprint stories to completion along with Unit test case coverage.
• Experience working in Agile Methodology
• Excellent communication and coordination skills
• Knowledgeable (and preferred hands on) - UNIX environments, different continuous integration tools.
• Must be able to integrate quickly into the team and work independently towards team goals

• Take the complete responsibility of the sprint stories' execution
• Be accountable for the delivery of the tasks in the defined timelines with good quality.
• Follow the processes for project execution and delivery.
• Follow agile methodology
• Work with the team lead closely and contribute to the smooth delivery of the project.
• Understand/define the architecture and discuss the pros-cons of the same with the team
• Involve in the brainstorming sessions and suggest improvements in the architecture/design.
• Work with other team leads to get the architecture/design reviewed.
• Work with the clients and counter-parts (in US) of the project.
• Keep all the stakeholders updated about the project/task status/risks/issues if there are any.

CANDIDATE WILL BE DEPLOYED IN A FINANCIAL CAPTIVE ORGANIZATION @ PUNE (KHARADI)

Below are the job Details :-

Experience 10 to 18 years

Mandatory skills –

• data migration,
• data flow

The ideal candidate for this role will have the below experience and qualifications:  

• Experience of building a range of Services in a Cloud Service provider (ideally GCP)  
• Hands-on design and development of Google Cloud Platform (GCP), across a wide range of GCP services including hands on experience of GCP storage & database technologies. 
• Hands-on experience in architecting, designing or implementing solutions on GCP, K8s, and other Google technologies. Security and Compliance, e.g. IAM and cloud compliance/auditing/monitoring tools 
• Desired Skills within the GCP stack - Cloud Run, GKE, Serverless, Cloud Functions, Vision API, DLP, Data Flow, Data Fusion 
• Prior experience of migrating on-prem applications to cloud environments. Knowledge and hands on experience on Stackdriver, pub-sub, VPC, Subnets, route tables, Load balancers, firewalls both for on premise and the GCP.  
• Integrate, configure, deploy and manage centrally provided common cloud services (e.g. IAM, networking, logging, Operating systems, Containers.)  
• Manage SDN in GCP Knowledge and experience of DevOps technologies around Continuous Integration & Delivery in GCP using Jenkins.  
• Hands on experience of Terraform, Kubernetes, Docker, Stackdriver, Terraform  
• Programming experience in one or more of the following languages: Python, Ruby, Java, JavaScript, Go, Groovy, Scala  
• Knowledge or experience in DevOps tooling such as Jenkins, Git, Ansible, Splunk, Jira or Confluence, AppD, Docker, Kubernetes  
• Act as a consultant and subject matter expert for internal teams to resolve technical deployment obstacles, improve product's vision. Ensure compliance with centrally defined Security 
• Financial experience is preferred 
• Ability to learn new technologies and rapidly prototype newer concepts 
• Top-down thinker, excellent communicator, and great problem solver

Exp:- 10  to 18 years

Location:- Pune

Candidate must have experience in below.

• GCP Data Platform
• Data Processing:- Data Flow, Data Prep, Data Fusion
• Data Storage:- Big Query, Cloud Sql,
• Pub Sub, GCS Bucket

Position Name: Software Developer

Required Experience: 3+ Years

Number of positions: 4

Qualifications: Master’s or Bachelor s degree in Engineering, Computer Science, or equivalent (BE/BTech or MS in Computer Science).

Key Skills: Python, Django, Ngnix, Linux, Sanic, Pandas, Numpy, Snowflake, SciPy, Data Visualization, RedShift, BigData, Charting

Compensation - As per industry standards.

Joining - Immediate joining is preferrable.

Required Skills:

• Strong Experience in Python and web frameworks like Django, Tornado and/or Flask
• Experience in data analytics using standard python libraries using Pandas, NumPy, MatPlotLib
• Conversant in implementing charts using charting libraries like Highcharts, d3.js, c3.js, dc.js and data Visualization tools like Plotly, GGPlot
• Handling and using large databases and Datawarehouse technologies like MongoDB, MySQL, BigData, Snowflake, Redshift.
• Experience in building APIs, Multi-threading for tasks on Linux platform
• Exposure to finance and capital markets will be added advantage. 
• Strong understanding of software design principles, algorithms, data structures, design patterns, and multithreading concepts.
• Worked on building highly-available distributed systems on cloud infrastructure or have had exposure to architectural pattern of a large, high-scale web application.
• Strong understanding of software design principles, algorithms, data structures, design patterns, and multithreading concepts.
• Basic understanding of front-end technologies, such as JavaScript, HTML5, and CSS3

Company Description:

Reval Analytical Services is a fully-owned subsidiary of Virtua Research Inc. US. It is a financial services technology company focused on consensus analytics, peer analytics and Web-enabled information delivery. The Company’s unique combination of investment research experience, modeling expertise, and software development capabilities enables it to provide industry-leading financial research tools and services for investors, analysts, and corporate management.

Website: http://www.virtuaresearch.com" target="_blank">www.virtuaresearch.com

• Must have 5-8 years of experience in handling data
• Must have the ability to interpret large amounts of data and to multi-task
• Must have strong knowledge of and experience with programming (Python), Linux/Bash scripting, databases(SQL, etc)
• Must have strong analytical and critical thinking to resolve business problems using data and tech
•  Must have domain familiarity and interest of – Cloud technologies (GCP/Azure Microsoft/ AWS Amazon), open-source technologies, Enterprise technologies
• Must have the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
• Must have good communication skills
• Working knowledge/exposure to ElasticSearch, PostgreSQL, Athena, PrestoDB, Jupyter Notebook