39+ Big data Jobs in Hyderabad | Big data Job openings in Hyderabad

Skills:

Experience with Cassandra, including installing configuring and monitoring a Cassandra cluster.

Experience with Cassandra data modeling and CQL scripting. Experience with DataStax Enterprise Graph

Experience with both Windows and Linux Operating Systems. Knowledge of Microsoft .NET Framework (C#, NETCore).

Ability to perform effectively in a team-oriented environment

The Sr. Analytics Engineer would provide technical expertise in needs identification, data modeling, data movement, and transformation mapping (source to target), automation and testing strategies, translating business needs into technical solutions with adherence to established data guidelines and approaches from a business unit or project perspective.

Understands and leverages best-fit technologies (e.g., traditional star schema structures, cloud, Hadoop, NoSQL, etc.) and approaches to address business and environmental challenges.

Provides data understanding and coordinates data-related activities with other data management groups such as master data management, data governance, and metadata management.

Actively participates with other consultants in problem-solving and approach development.

Responsibilities :

Provide a consultative approach with business users, asking questions to understand the business need and deriving the data flow, conceptual, logical, and physical data models based on those needs.

Perform data analysis to validate data models and to confirm the ability to meet business needs.

Assist with and support setting the data architecture direction, ensuring data architecture deliverables are developed, ensuring compliance to standards and guidelines, implementing the data architecture, and supporting technical developers at a project or business unit level.

Coordinate and consult with the Data Architect, project manager, client business staff, client technical staff and project developers in data architecture best practices and anything else that is data related at the project or business unit levels.

Work closely with Business Analysts and Solution Architects to design the data model satisfying the business needs and adhering to Enterprise Architecture.

Coordinate with Data Architects, Program Managers and participate in recurring meetings.

Help and mentor team members to understand the data model and subject areas.

Ensure that the team adheres to best practices and guidelines.

Requirements :

- Strong working knowledge of at least 3 years of Spark, Java/Scala/Pyspark, Kafka, Git, Unix / Linux, and ETL pipeline designing.

- Experience with Spark optimization/tuning/resource allocations

- Excellent understanding of IN memory distributed computing frameworks like Spark and its parameter tuning, writing optimized workflow sequences.

- Experience of relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Redshift, Bigquery, Cassandra, etc).

- Familiarity with Docker, Kubernetes, Azure Data Lake/Blob storage, AWS S3, Google Cloud storage, etc.

- Have a deep understanding of the various stacks and components of the Big Data ecosystem.

- Hands-on experience with Python is a huge plus

Responsibilities -

• Collaborate with the development team to understand data requirements and identify potential scalability issues.
• Design, develop, and implement scalable data pipelines and ETL processes to ingest, process, and analyse large - volumes of data from various sources.
• Optimize data models and database schemas to improve query performance and reduce latency.
• Monitor and troubleshoot the performance of our Cassandra database on Azure Cosmos DB, identifying bottlenecks and implementing optimizations as needed.
• Work with cross-functional teams to ensure data quality, integrity, and security.
• Stay up to date with emerging technologies and best practices in data engineering and distributed systems.

Qualifications & Requirements -

• Proven experience as a Data Engineer or similar role, with a focus on designing and optimizing large-scale data systems.
• Strong proficiency in working with NoSQL databases, particularly Cassandra.
• Experience with cloud-based data platforms, preferably Azure Cosmos DB.
• Solid understanding of Distributed Systems, Data modelling, Data Warehouse Designing, and ETL Processes.
• Detailed understanding of Software Development Life Cycle (SDLC) is required.
• Good to have knowledge on any visualization tool like Power BI, Tableau.
• Good to have knowledge on SAP landscape (SAP ECC, SLT, BW, HANA etc).
• Good to have experience on Data Migration Project.
• Knowledge of Supply Chain domain would be a plus.
• Familiarity with software architecture (data structures, data schemas, etc.)
• Familiarity with Python programming language is a plus.
• The ability to work in a dynamic, fast-paced, work environment.
• A passion for data and information with strong analytical, problem solving, and organizational skills.
• Self-motivated with the ability to work under minimal direction.
• Strong communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.

Role & Responsibilities

1. Create innovative architectures based on business requirements.
2. Design and develop cloud-based solutions for global enterprises.
3. Coach and nurture engineering teams through feedback, design reviews, and best practice input.
4. Lead cross-team projects, ensuring resolution of technical blockers.
5. Collaborate with internal engineering teams, global technology firms, and the open-source community.
6. Lead initiatives to learn and apply modern and advanced technologies.
7. Oversee the launch of innovative products in high-volume production environments.
8. Develop and maintain high-quality software applications using JS frameworks (React, NPM, Node.js etc.,).
9. Utilize design patterns for backend technologies and ensure strong coding skills.
10. Deploy and manage applications on AWS cloud services, including ECS (Fargate), Lambda, and load balancers. Work with Docker to containerize services.
11. Implement and follow CI/CD practices using GitLab for automated build, test, and deployment processes.
12. Collaborate with cross-functional teams to design technical solutions, ensuring adherence to Microservice Design patterns and Architecture.
13. Apply expertise in Authentication & Authorization protocols (e.g., JWT, OAuth), including certificate handling, to ensure robust application security.
14. Utilize databases such as Postgres, MySQL, Mongo and DynamoDB for efficient data storage and retrieval.
15. Demonstrate familiarity with Big Data technologies, including but not limited to:

- Apache Kafka for distributed event streaming.

- Apache Spark for large-scale data processing.

- Containers for scalable and portable deployments.

Technical Skills:

1. 7+ years of hands-on development experience with JS frameworks, specifically MERN.
2. Strong coding skills in backend technologies using various design patterns.
3. Strong UI development skills using React.
4. Expert in containerization using Docker.
5. Knowledge of cloud platforms, specifically OCI, and familiarity with serverless technology, services like ECS, Lambda, and load balancers.
6. Proficiency in CI/CD practices using GitLab or Bamboo.
7. Strong knowledge of Microservice Design patterns and Architecture.
8. Expertise in Authentication and authorization protocols like JWT, and OAuth including certificate handling.
9. Experience working with high stream media data.
10. Experience working with databases such as Postgres, MySQL, and DynamoDB.
11. Familiarity with Big Data technologies related to Kafka, PySpark, Apache Spark, Containers, etc.
12. Experience with container Orchestration tools like Kubernetes.

Role: Oracle DBA Developer

Location: Hyderabad

Required Experience: 8 + Years

Skills : DBA, Terraform, Ansible, Python, Shell Script, DevOps activities, Oracle DBA, SQL server, Cassandra, Oracle sql/plsql, MySQL/Oracle/MSSql/Mongo/Cassandra, Security measure configuration

cid:[email protected]

Roles and Responsibilities:

1. 8+ years of hands-on DBA experience in one or many of the following: SQL Server, Oracle, Cassandra

2. DBA experience in a SRE environment will be an advantage.

3. Experience in Automation/building databases by providing self-service tools. analyze and implement solutions for database administration (e.g., backups, performance tuning, Troubleshooting, Capacity planning)

4. Analyze solutions and implement best practices for cloud database and their components.

5. Build and enhance tooling, automation, and CI/CD workflows (Jenkins etc.) that provide safe self-service capabilities to th6. Implement proactive monitoring and alerting to detect issues before they impact users. Use a metrics-driven approach to identify and root cause performance and scalability bottlenecks in the system.

7. Work on automation of database infrastructure and help engineering succeed by providing self-service tools.

8. Write database documentation, including data standards, procedures, and definitions for the data dictionary (metadata)

9. Monitor database performance, control access permissions and privileges, capacity planning, implement changes and apply new patches and versions when required.

10. Recommend query and schema changes to optimize the performance of database queries.

11. Have experience with cloud-based environments (OCI, AWS, Azure) as well as On-Premises.

12. Have experience with cloud database such as SQL server, Oracle, Cassandra

13. Have experience with infrastructure automation and configuration management (Jira, Confluence, Ansible, Gitlab, Terraform)

14. Have excellent written and verbal English communication skills.

15. Planning, managing, and scaling of data stores to ensure a business’ complex data requirements are met and it can easily access its data in a fast, reliable, and safe manner.

16. Ensures the quality of orchestration and integration of tools needed to support daily operations by patching together existing infrastructure with cloud solutions and additional data infrastructures.

17. Data Security and protecting the data through rigorous testing of backup and recovery processes and frequently auditing well-regulated security procedures.

18. use software and tooling to automate manual tasks and enable engineers to move fast without the concern of losing data during their experiments.

19. service level objectives (SLOs), risk analysis to determine which problems to address and which problems to automate.

20. Bachelor's Degree in a technical discipline required.

21. DBA Certifications required: Oracle, SQLServer, Cassandra (2 or more)

21. Cloud, DevOps certifications will be an advantage.

Must have Skills:

Ø Oracle DBA with development

Ø SQL

Ø Devops tools

Ø Cassandra

About Quadratyx:

We are a product-centric insight & automation services company globally. We help the world’s organizations make better & faster decisions using the power of insight & intelligent automation. We build and operationalize their next-gen strategy, through Big Data, Artificial Intelligence, Machine Learning, Unstructured Data Processing and Advanced Analytics. Quadratyx can boast more extensive experience in data sciences & analytics than most other companies in India.

We firmly believe in Excellence Everywhere.

Job Description

Purpose of the Job/ Role:

• As a Technical Lead, your work is a combination of hands-on contribution, customer engagement and technical team management. Overall, you’ll design, architect, deploy and maintain big data solutions.

Key Requisites:

• Expertise in Data structures and algorithms.

• Technical management across the full life cycle of big data (Hadoop) projects from requirement gathering and analysis to platform selection, design of the architecture and deployment.

• Scaling of cloud-based infrastructure.

• Collaborating with business consultants, data scientists, engineers and developers to develop data solutions.

• Led and mentored a team of data engineers.

• Hands-on experience in test-driven development (TDD).

• Expertise in No SQL like Mongo, Cassandra etc, preferred Mongo and strong knowledge of relational databases.

• Good knowledge of Kafka and Spark Streaming internal architecture.

• Good knowledge of any Application Servers.

• Extensive knowledge of big data platforms like Hadoop; Hortonworks etc.

• Knowledge of data ingestion and integration on cloud services such as AWS; Google Cloud; Azure etc. 

Skills/ Competencies Required

Technical Skills

• Strong expertise (9 or more out of 10) in at least one modern programming language, like Python, or Java.

• Clear end-to-end experience in designing, programming, and implementing large software systems.

• Passion and analytical abilities to solve complex problems Soft Skills.

• Always speaking your mind freely.

• Communicating ideas clearly in talking and writing, integrity to never copy or plagiarize intellectual property of others.

• Exercising discretion and independent judgment where needed in performing duties; not needing micro-management, maintaining high professional standards.

Academic Qualifications & Experience Required

Required Educational Qualification & Relevant Experience

• Bachelor’s or Master’s in Computer Science, Computer Engineering, or related discipline from a well-known institute.

• Minimum 7 - 10 years of work experience as a developer in an IT organization (preferably Analytics / Big Data/ Data Science / AI background.

Roles and Responsibilities

Skills

Bigdata with cloud:

Experience : 5-10 years

Location : Hyderabad/Chennai

Notice period : 15-20 days Max

1.  Expertise in building AWS Data Engineering pipelines with AWS Glue -> Athena -> Quick sight

2.  Experience in developing lambda functions with AWS Lambda

3.  Expertise with Spark/PySpark – Candidate should be hands on with PySpark code and should be able to do transformations with Spark

4.  Should be able to code in Python and Scala.

5.  Snowflake experience will be a plus

1. Data Engineer

 Required skill set: AWS GLUE, AWS LAMBDA, AWS SNS/SQS, AWS ATHENA, SPARK, SNOWFLAKE, PYTHON

Mandatory Requirements  

• Experience in AWS Glue
• Experience in Apache Parquet 
• Proficient in AWS S3 and data lake 
• Knowledge of Snowflake
• Understanding of file-based ingestion best practices.
• Scripting language - Python & pyspark 

CORE RESPONSIBILITIES 

• Create and manage cloud resources in AWS 
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 
• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
• Define process improvement opportunities to optimize data collection, insights and displays.
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 
• Identify and interpret trends and patterns from complex data sets 
• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 
• Key participant in regular Scrum ceremonies with the agile teams  
• Proficient at developing queries, writing reports and presenting findings 
• Mentor junior members and bring best industry practices 

 QUALIFICATIONS 

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 
• Strong background in math, statistics, computer science, data science or related discipline
• Advanced knowledge one of language: Java, Scala, Python, C# 
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  
• Proficient with
• Data mining/programming tools (e.g. SAS, SQL, R, Python)
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
• Data visualization (e.g. Tableau, Looker, MicroStrategy)
• Comfortable learning about and deploying new technologies and tools. 
• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 
• Good written and oral communication skills and ability to present results to non-technical audiences 
• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus:  

• AWS certification
• Spark Streaming 
• Kafka Streaming / Kafka Connect 
• ELK Stack 
• Cassandra / MongoDB 
• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Job Title: Product Manager

Job Description

Bachelor or master’s degree in computer science or equivalent experience.
Worked as Product Owner before and took responsibility for a product or project delivery.
Well-versed with data warehouse modernization to Big Data and Cloud environments.
Good knowledge* of any of the Cloud (AWS/Azure/GCP) – Must Have
Practical experience with continuous integration and continuous delivery workflows.
Self-motivated with strong organizational/prioritization skills and ability to multi-task with close attention to detail.
Good communication skills
Experience in working within a distributed agile team
Experience in handling migration projects – Good to Have
 

*Data Ingestion, Processing, and Orchestration knowledge

Roles & Responsibilities

Responsible for coming up with innovative and novel ideas for the product.
Define product releases, features, and roadmap.
Collaborate with product teams on defining product objectives, including creating a product roadmap, delivery, market research, customer feedback, and stakeholder inputs.
Work with the Engineering teams to communicate release goals and be a part of the product lifecycle. Work closely with the UX and UI team to create the best user experience for the end customer.
Work with the Marketing team to define GTM activities.
Interface with Sales & Customer teams to identify customer needs and product gaps
Market and competition analysis activities.
Participate in the Agile ceremonies with the team, define epics, user stories, acceptance criteria
Ensure product usability from the end-user perspective

Mandatory Skills

Product Management, DWH, Big Data

Job Description

Mandatory Requirements 

• Experience in AWS Glue

• Experience in Apache Parquet 

• Proficient in AWS S3 and data lake 

• Knowledge of Snowflake

• Understanding of file-based ingestion best practices.

• Scripting language - Python & pyspark

CORE RESPONSIBILITIES

• Create and manage cloud resources in AWS 

• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 

• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 

• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 

• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.

• Define process improvement opportunities to optimize data collection, insights and displays.

• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 

• Identify and interpret trends and patterns from complex data sets 

• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 

• Key participant in regular Scrum ceremonies with the agile teams  

• Proficient at developing queries, writing reports and presenting findings 

• Mentor junior members and bring best industry practices.

QUALIFICATIONS

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 

• Strong background in math, statistics, computer science, data science or related discipline

• Advanced knowledge one of language: Java, Scala, Python, C# 

• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  

• Proficient with

• Data mining/programming tools (e.g. SAS, SQL, R, Python)

• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)

• Data visualization (e.g. Tableau, Looker, MicroStrategy)

• Comfortable learning about and deploying new technologies and tools. 

• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 

• Good written and oral communication skills and ability to present results to non-technical audiences 

• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus: 

• AWS certification

• Spark Streaming 

• Kafka Streaming / Kafka Connect 

• ELK Stack 

• Cassandra / MongoDB 

• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

4-5 years of overall experience in software development.
- Experience on Hadoop (Apache/Cloudera/Hortonworks) and/or other Map Reduce Platforms
- Experience on Hive, Pig, Sqoop, Flume and/or Mahout
- Experience on NO-SQL – HBase, Cassandra, MongoDB
- Hands on experience with Spark development,  Knowledge of Storm, Kafka, Scala
- Good knowledge of Java
- Good background of Configuration Management/Ticketing systems like Maven/Ant/JIRA etc.
- Knowledge around any Data Integration and/or EDW tools is plus
- Good to have knowledge of  using Python/Perl/Shell

Please note - Hbase hive and spark are must.

CORE RESPONSIBILITIES

• Create and manage cloud resources in AWS 
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 
• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
• Define process improvement opportunities to optimize data collection, insights and displays.
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 
• Identify and interpret trends and patterns from complex data sets 
• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 
• Key participant in regular Scrum ceremonies with the agile teams  
• Proficient at developing queries, writing reports and presenting findings 
• Mentor junior members and bring best industry practices 

QUALIFICATIONS

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 
• Strong background in math, statistics, computer science, data science or related discipline
• Advanced knowledge one of language: Java, Scala, Python, C# 
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  
• Proficient with
• Data mining/programming tools (e.g. SAS, SQL, R, Python)
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
• Data visualization (e.g. Tableau, Looker, MicroStrategy)
• Comfortable learning about and deploying new technologies and tools. 
• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 
• Good written and oral communication skills and ability to present results to non-technical audiences 
• Knowledge of business intelligence and analytical tools, technologies and techniques.

Mandatory Requirements 

• Experience in AWS Glue
• Experience in Apache Parquet 
• Proficient in AWS S3 and data lake 
• Knowledge of Snowflake
• Understanding of file-based ingestion best practices.
• Scripting language - Python & pyspark

• At least 4 to 7 years of relevant experience as Big Data Engineer
• Hands-on experience in Scala or Python
• Hands-on experience on major components in Hadoop Ecosystem like HDFS, Map Reduce, Hive, Impala.
• Strong programming experience in building applications/platform using Scala or Python.
• Experienced in implementing Spark RDD Transformations, actions to implement business analysis

Skills- Informatica with Big Data Management

1.Minimum 6 to 8 years of experience in informatica BDM development
2.Experience working on Spark/SQL
3.Develops informtica mapping/Sql 

Summary
Our Kafka developer has a combination of technical skills, communication skills and business knowledge. The developer should be able to work on multiple medium to large projects. The successful candidate will have excellent technical skills of Apache/Confluent Kafka, Enterprise Data WareHouse preferable GCP BigQuery or any equivalent Cloud EDW and also will be able to take oral and written business requirements and develop efficient code to meet set deliverables.

Must Have Skills

• Participate in the development, enhancement and maintenance of data applications both as an individual contributor and as a lead.
• Leading in the identification, isolation, resolution and communication of problems within the production environment.
• Leading developer and applying technical skills Apache/Confluent Kafka (Preferred) AWS Kinesis (Optional), Cloud Enterprise Data Warehouse Google BigQuery (Preferred) or AWS RedShift or SnowFlakes (Optional)
• Design recommending best approach suited for data movement from different sources to Cloud EDW using Apache/Confluent Kafka
• Performs independent functional and technical analysis for major projects supporting several corporate initiatives.
• Communicate and Work with IT partners and user community with various levels from Sr Management to detailed developer to business SME for project definition .
• Works on multiple platforms and multiple projects concurrently.
• Performs code and unit testing for complex scope modules, and projects
• Provide expertise and hands on experience working on Kafka connect using schema registry in a very high volume environment (~900 Million messages)

• Provide expertise in Kafka brokers, zookeepers, KSQL, KStream and Kafka Control center.
• Provide expertise and hands on experience working on AvroConverters, JsonConverters, and StringConverters.
• Provide expertise and hands on experience working on Kafka connectors such as MQ connectors, Elastic Search connectors, JDBC connectors, File stream connector,  JMS source connectors, Tasks, Workers, converters, Transforms.
• Provide expertise and hands on experience on custom connectors using the Kafka core concepts and API.
• Working knowledge on Kafka Rest proxy.
• Ensure optimum performance, high availability and stability of solutions.
• Create topics, setup redundancy cluster, deploy monitoring tools, alerts and has good knowledge of best practices.
• Create stubs for producers, consumers and consumer groups for helping onboard applications from different languages/platforms.  Leverage Hadoop ecosystem knowledge to design, and develop capabilities to deliver our solutions using Spark, Scala, Python, Hive, Kafka and other things in the Hadoop ecosystem. 
• Use automation tools like provisioning using Jenkins, Udeploy or relevant technologies
• Ability to perform data related benchmarking, performance analysis and tuning.
• Strong skills in In-memory applications, Database Design, Data Integration.

• Drive and participate in requirements gathering workshops, estimation discussions, design meetings and status review meetings
• Participate and contribute in Solution Design and Solution Architecture for implementing Big Data Projects on-premise and on cloud
• Technical Hands on experience in design, coding, development and managing Large Hadoop implementation
• Proficient in SQL, Hive, PIG, Spark SQL, Shell Scripting, Kafka, Flume, Scoop with large Big Data and Data Warehousing projects with either Java, Python or Scala based Hadoop programming background
• Proficient with various development methodologies like waterfall, agile/scrum and iterative
• Good Interpersonal skills and excellent communication skills for US and UK based clients

Location:  Pune/Nagpur,Goa,Hyderabad/Bangalore

Job Requirements:

• 9 years and above of total experience preferably in bigdata space.
• Creating spark applications using Scala to process data.
• Experience in scheduling and troubleshooting/debugging Spark jobs in steps.
• Experience in spark job performance tuning and optimizations.
• Should have experience in processing data using Kafka/Pyhton.
• Individual should have experience and understanding in configuring Kafka topics to optimize the performance.
• Should be proficient in writing SQL queries to process data in Data Warehouse.
• Hands on experience in working with Linux commands to troubleshoot/debug issues and creating shell scripts to automate tasks.
• Experience on AWS services like EMR.

SpringML is looking to hire a top-notch Senior  Data Engineer who is passionate about working with data and using the latest distributed framework to process large dataset. As an Associate Data Engineer, your primary role will be to design and build data pipelines. You will be focused on helping client projects on data integration, data prep and implementing machine learning on datasets. In this role, you will work on some of the latest technologies, collaborate with partners on early win, consultative approach with clients, interact daily with executive leadership, and help build a great company. Chosen team members will be part of the core team and play a critical role in scaling up our emerging practice.

RESPONSIBILITIES:

• Ability to work as a member of a team assigned to design and implement data integration solutions.
• Build Data pipelines using standard frameworks in Hadoop, Apache Beam and other open-source solutions.
• Learn quickly – ability to understand and rapidly comprehend new areas – functional and technical – and apply detailed and critical thinking to customer solutions.
• Propose design solutions and recommend best practices for large scale data analysis

SKILLS:

• B.tech  degree in computer science, mathematics or other relevant fields.
• 4+years of experience in ETL, Data Warehouse, Visualization and building data pipelines.
• Strong Programming skills – experience and expertise in one of the following: Java, Python, Scala, C.
• Proficient in big data/distributed computing frameworks such as Apache,Spark, Kafka,
• Experience with Agile implementation methodologies

• Total Experience of 7-10 years and should be interested in teaching and research
• 3+ years’ experience in data engineering which includes data ingestion, preparation, provisioning, automated testing, and quality checks.
• 3+ Hands-on experience in Big Data cloud platforms like AWS and GCP, Data Lakes and Data Warehouses
• 3+ years of Big Data and Analytics Technologies. Experience in SQL, writing code in spark engine using python, scala or java Language. Experience in Spark, Scala
• Experience in designing, building, and maintaining ETL systems
• Experience in data pipeline and workflow management tools like Airflow
• Application Development background along with knowledge of Analytics libraries, opensource Natural Language Processing, statistical and big data computing libraries
• Familiarity with Visualization and Reporting Tools like Tableau, Kibana.
• Should be good at storytelling in Technology

About the Role

The Dremio India team owns the DataLake Engine along with Cloud Infrastructure and services that power it. With focus on next generation data analytics supporting modern table formats like Iceberg, Deltalake, and open source initiatives such as Apache Arrow, Project Nessie and hybrid-cloud infrastructure, this team provides various opportunities to learn, deliver, and grow in career. We are looking for technical leaders with passion and experience in architecting and delivering high-quality distributed systems at massive scale.

Responsibilities & ownership

• Lead end-to-end delivery and customer success of next-generation features related to scalability, reliability, robustness, usability, security, and performance of the product
• Lead and mentor others about concurrency, parallelization to deliver scalability, performance and resource optimization in a multithreaded and distributed environment
• Propose and promote strategic company-wide tech investments taking care of business goals, customer requirements, and industry standards
• Lead the team to solve complex, unknown and ambiguous problems, and customer issues cutting across team and module boundaries with technical expertise, and influence others
• Review and influence designs of other team members 
• Design and deliver architectures that run optimally on public clouds like GCP, AWS, and Azure
• Partner with other leaders to nurture innovation and engineering excellence in the team
• Drive priorities with others to facilitate timely accomplishments of business objectives
• Perform RCA of customer issues and drive investments to avoid similar issues in future
• Collaborate with Product Management, Support, and field teams to ensure that customers are successful with Dremio
• Proactively suggest learning opportunities about new technology and skills, and be a role model for constant learning and growth

Requirements

• B.S./M.S/Equivalent in Computer Science or a related technical field or equivalent experience
• Fluency in Java/C++ with 15+ years of experience developing production-level software
• Strong foundation in data structures, algorithms, multi-threaded and asynchronous programming models and their use in developing distributed and scalable systems
• 8+ years experience in developing complex and scalable distributed systems and delivering, deploying, and managing microservices successfully
• Subject Matter Expert in one or more of query processing or optimization, distributed systems, concurrency, micro service based architectures, data replication, networking, storage systems
• Experience in taking company-wide initiatives, convincing stakeholders, and delivering them
• Expert in solving complex, unknown and ambiguous problems spanning across teams and taking initiative in planning and delivering them with high quality
• Ability to anticipate and propose plan/design changes based on changing requirements 
• Passion for quality, zero downtime upgrades, availability, resiliency, and uptime of the platform
• Passion for learning and delivering using latest technologies
• Hands-on experience of working projects on AWS, Azure, and GCP 
• Experience with containers and Kubernetes for orchestration and container management in private and public clouds (AWS, Azure,  and GCP) 
• Understanding of distributed file systems such as  S3, ADLS or HDFS
• Excellent communication skills and affinity for collaboration and teamwork

Qentelli is seeking a Solution Architect to untangle and redesign a huge granny old monolithic legacy system. Interesting part is that the new system should be commissioned module by module and legacy system should phase off accordingly. So your design will have a cutting edge future state and a transition state to get there. Implementation now is all Microsoft tech stack and will continue to be on newer Microsoft tech stack. Also there is a critical component of API management to be introduced into the solution. Performance and scalability will be at the center of your solution architecture. Data modelling is one thing that is of super high importance to know.

You’ll have a distributed team with onshore in the US and offshore in India. As a Solution Architect, you should be able to wear multiple hats of working with client on solutioning and getting it implemented by engineering and infrastructure teams that are both onshore and offshore. Right candidate will be awesome at fleshing out and documenting every finer detail of the solution, elaborate at communicating with your teams, disciplined at getting it implemented and passionate for client success.

TECHNOLOGIES YOU’LL NEED TO KNOW

Greetings from Qentelli Solutions Private Limited!

We are hiring for PostgreSQL Developer

Experience: 4 to 12 years

Job Location: Hyderabad

Job Description:

• Experience in RDBMS (PostgreSQL preferred), Database Backend development, Data Modelling, Performance Tuning, exposure to NoSQL DB, Kubernetes or Cloud (AWS/Azure/GCS)

Skillset for Developer-II:

• Experience on any Big Data Tools (Nifi, Kafka, Spark, sqoop, storm, snowflake), Database Backend development, Python, No SQL DB, API Exposure, cloud or Kubernetes exposure

Skillset for API Developer:

• API Development with extensive knowledge on any RDBMS (preferred PostgreSQL), exposure to cloud or Kubernetes

Indium Software is a niche technology solutions company with deep expertise in Digital , QA and Gaming. Indium helps customers in their Digital Transformation journey through a gamut of solutions that enhance business value.

With over 1000+ associates globally, Indium operates through offices in the US, UK and India

Visit http://www.indiumsoftware.com">www.indiumsoftware.com to know more.

Job Title: Analytics Data Engineer

What will you do:
The Data Engineer must be an expert in SQL development further providing support to the Data and Analytics in database design, data flow and analysis activities. The position of the Data Engineer also plays a key role in the development and deployment of innovative big data platforms for advanced analytics and data processing. The Data Engineer defines and builds the data pipelines that will enable faster, better, data-informed decision-making within the business.

We ask:

Extensive Experience with SQL and strong ability to process and analyse complex data

The candidate should also have an ability to design, build, and maintain the business’s ETL pipeline and data warehouse The candidate will also demonstrate expertise in data modelling and query performance tuning on SQL Server
Proficiency with analytics experience, especially funnel analysis, and have worked on analytical tools like Mixpanel, Amplitude, Thoughtspot, Google Analytics, and similar tools.

Should work on tools and frameworks required for building efficient and scalable data pipelines
Excellent at communicating and articulating ideas and an ability to influence others as well as drive towards a better solution continuously.
Experience working in python, Hive queries, spark, pysaprk, sparkSQL, presto

• Relate Metrics to product
• Programmatic Thinking
• Edge cases
• Good Communication
• Product functionality understanding

Perks & Benefits:
A dynamic, creative & intelligent team they will make you love being at work.
Autonomous and hands-on role to make an impact you will be joining at an exciting time of growth!

Flexible work hours and Attractive pay package and perks
An inclusive work environment that lets you work in the way that works best for you!

ETL Developer – Talend

Job Duties:

• ETL Developer is responsible for Design and Development of ETL Jobs which follow standards,

best practices and are maintainable, modular and reusable.

• Proficiency with Talend or Pentaho Data Integration / Kettle.
• ETL Developer will analyze and review complex object and data models and the metadata

repository in order to structure the processes and data for better management and efficient

access.

• Working on multiple projects, and delegating work to Junior Analysts to deliver projects on time.
• Training and mentoring Junior Analysts and building their proficiency in the ETL process.
• Preparing mapping document to extract, transform, and load data ensuring compatibility with

all tables and requirement specifications.

• Experience in ETL system design and development with Talend / Pentaho PDI is essential.
• Create quality rules in Talend.
• Tune Talend / Pentaho jobs for performance optimization.
• Write relational(sql) and multidimensional(mdx) database queries.
• Functional Knowledge of Talend Administration Center/ Pentaho data integrator, Job Servers &

Load balancing setup, and all its administrative functions.

• Develop, maintain, and enhance unit test suites to verify the accuracy of ETL processes,

dimensional data, OLAP cubes and various forms of BI content including reports, dashboards,

and analytical models.

• Exposure in Map Reduce components of Talend / Pentaho PDI.
• Comprehensive understanding and working knowledge in Data Warehouse loading, tuning, and

maintenance.

• Working knowledge of relational database theory and dimensional database models.
• Creating and deploying Talend / Pentaho custom components is an add-on advantage.
• Nice to have java knowledge.

Skills and Qualification:

• BE, B.Tech / MS Degree in Computer Science, Engineering or a related subject.
• Having an experience of 3+ years.
• Proficiency with Talend or Pentaho Data Integration / Kettle.
• Ability to work independently.
• Ability to handle a team.
• Good written and oral communication skills.