11+ HDFS Jobs in Bangalore (Bengaluru) | HDFS Job openings in Bangalore (Bengaluru)

As Conviva is expanding, we are building products providing deep insights into end-user experience for our customers.

Platform and TLB Team

The vision for the TLB team is to build data processing software that works on terabytes of streaming data in real-time. Engineer the next-gen Spark-like system for in-memory computation of large time-series datasets – both Spark-like backend infra and library-based programming model. Build a horizontally and vertically scalable system that analyses trillions of events per day within sub-second latencies. Utilize the latest and greatest big data technologies to build solutions for use cases across multiple verticals. Lead technology innovation and advancement that will have a big business impact for years to come. Be part of a worldwide team building software using the latest technologies and the best of software development tools and processes.

What You’ll Do

This is an individual contributor position. Expectations will be on the below lines:

• Design, build and maintain the stream processing, and time-series analysis system which is at the heart of Conviva’s products
• Responsible for the architecture of the Conviva platform
• Build features, enhancements, new services, and bug fixing in Scala and Java on a Jenkins-based pipeline to be deployed as Docker containers on Kubernetes
• Own the entire lifecycle of your microservice including early specs, design, technology choice, development, unit-testing, integration-testing, documentation, deployment, troubleshooting, enhancements, etc.
• Lead a team to develop a feature or parts of a product
• Adhere to the Agile model of software development to plan, estimate, and ship per business priority

What you need to succeed

• 5+ years of work experience in software development of data processing products.
• Engineering degree in software or equivalent from a premier institute.
• Excellent knowledge of fundamentals of Computer Science like algorithms and data structures. Hands-on with functional programming and know-how of its concepts
• Excellent programming and debugging skills on the JVM. Proficient in writing code in Scala/Java/Rust/Haskell/Erlang that is reliable, maintainable, secure, and performant
• Experience with big data technologies like Spark, Flink, Kafka, Druid, HDFS, etc.
• Deep understanding of distributed systems concepts and scalability challenges including multi-threading, concurrency, sharding, partitioning, etc.
• Experience/knowledge of Akka/Lagom framework and/or stream processing technologies like RxJava or Project Reactor will be a big plus. Knowledge of design patterns like event-streaming, CQRS and DDD to build large microservice architectures will be a big plus
• Excellent communication skills. Willingness to work under pressure. Hunger to learn and succeed. Comfortable with ambiguity. Comfortable with complexity

Underpinning the Conviva platform is a rich history of innovation. More than 60 patents represent award-winning technologies and standards, including first-of-its kind-innovations like time-state analytics and AI-automated data modeling, that surfaces actionable insights. By understanding real-world human experiences and having the ability to act within seconds of observation, our customers can solve business-critical issues and focus on growing their business ahead of the competition. Examples of the brands Conviva has helped fuel streaming growth for include: DAZN, Disney+, HBO, Hulu, NBCUniversal, Paramount+, Peacock, Sky, Sling TV, Univision and Warner Bros Discovery.  

Privately held, Conviva is headquartered in Silicon Valley, California with offices and people around the globe. For more information, visit us at www.conviva.com. Join us to help extend our leadership position in big data streaming analytics to new audiences and markets! 

Responsibilities :

• Provide Support Services to our Gold & Enterprise customers using our flagship product suits. This may include assistance provided during the engineering and operations of distributed systems as well as responses for mission-critical systems and production customers.
• Lead end-to-end delivery and customer success of next-generation features related to scalability, reliability, robustness, usability, security, and performance of the product
• Lead and mentor others about concurrency, parallelization to deliver scalability, performance, and resource optimization in a multithreaded and distributed environment
• Demonstrate the ability to actively listen to customers and show empathy to the customer’s business impact when they experience issues with our products

Requires Skills :

• 10+ years of Experience with a highly scalable, distributed, multi-node environment (100+ nodes)
• Hadoop operation including Zookeeper, HDFS, YARN, Hive, and related components like the Hive metastore, Cloudera Manager/Ambari, etc
• Authentication and security configuration and tuning (KNOX, LDAP, Kerberos, SSL/TLS, second priority: SSO/OAuth/OIDC, Ranger/Sentry)
• Java troubleshooting, e.g., collection and evaluation of jstacks, heap dumps
• Linux, NFS, Windows, including application installation, scripting, basic command line
• Docker and Kubernetes configuration and troubleshooting, including Helm charts, storage options, logging, and basic kubectl CLI
• Experience working with scripting languages (Bash, PowerShell, Python)
• Working knowledge of application, server, and network security management concepts
• Familiarity with virtual machine technologies
• Knowledge of databases like MySQL and PostgreSQL,
• Certification on any of the leading Cloud providers (AWS, Azure, GCP ) and/or Kubernetes is a big plus

You will join a team of highly skilled engineers who are responsible for delivering Acceldata’s support services. Our Site Reliability Engineers are trained to be active listeners and demonstrate empathy when customers encounter product issues. In our fun and collaborative environment  Site Reliability Engineers develop strong business, interpersonal and technical skills to deliver high-quality service to our valued customers.

When you arrive for your first day, we’ll want you to have:

• Solid skills in troubleshooting to repair failed products or processes on a machine or a system using a logical, systematic search for the source of a problem in order to solve it, and make the product or process operational again
• A strong ability to understand the feelings of our customers as we empathize with them on the issue at hand
• A strong desire to increase your product and technology skillset; increase- your confidence supporting our products so you can help our customers succeed

In this position you will…

• Provide Support Services to our Gold & Enterprise customers using our flagship Acceldata Pulse,Flow & Torch Product suits. This may include assistance provided during the engineering and operations of distributed systems as well as responses for mission-critical systems and production customers.
• Demonstrate the ability to actively listen to customers and show empathy to the customer’s business impact when they experience issues with our products
• Participate in the queue management and coordination process by owning customer escalations, managing the unassigned queue.
• Be involved with and work on other support related activities - Performing POC & assisting Onboarding deployments of Acceldata & Hadoop distribution products.
• Triage, diagnose and escalate customer inquiries when applicable during their engineering and operations efforts.
• Collaborate and share solutions with both customers and the Internal team.
• Investigate product related issues both for particular customers and for common trends that may arise
• Study and understand critical system components and large cluster operations
• Differentiate between issues that arise in operations, user code, or product
• Coordinate enhancement and feature requests with product management and Acceldata engineering team.
• Flexible in working in Shifts.
• Participate in a Rotational weekend on-call roster for critical support needs.
• Participate as a designated or dedicated engineer for specific customers. Aspects of this engagement translates to building long term successful relationships with customers, leading weekly status calls, and occasional visits to customer sites

In this position, you should have…

• A strong desire and aptitude to become a well-rounded support professional. Acceldata Support considers the service we deliver as our core product.
• A positive attitude towards feedback and continual improvement
• A willingness to give direct feedback to and partner with management to improve team operations
• A tenacity to bring calm and order to the often stressful situations of customer cases
• A mental capability to multi-task across many customer situations simultaneously
• Bachelor degree in Computer Science or Engineering or equivalent experience. Master’s degree is a plus
• At least 2+ years of experience with at least one of the following cloud platforms: Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), experience with managing and supporting a cloud infrastructure on any of the 3 platforms. Also knowledge on Kubernetes, Docker is a must.
• Strong troubleshooting skills (in example, TCP/IP, DNS, File system, Load balancing, database, Java)
• Excellent communication skills in English (written and verbal)
• Prior enterprise support experience in a technical environment strongly preferred

Strong Hands-on Experience Working With Or Supporting The Following

• 8-12 years of Experience with a highly-scalable, distributed, multi-node environment (50+ nodes)
• Hadoop operation including Zookeeper, HDFS, YARN, Hive, and related components like the Hive metastore, Cloudera Manager/Ambari, etc
• Authentication and security configuration and tuning (KNOX, LDAP, Kerberos, SSL/TLS, second priority: SSO/OAuth/OIDC, Ranger/Sentry)
• Java troubleshooting, e.g., collection and evaluation of jstacks, heap dumps

You might also have…

• Linux, NFS, Windows, including application installation, scripting, basic command line
• Docker and Kubernetes configuration and troubleshooting, including Helm charts, storage options, logging, and basic kubectl CLI

• Experience working with scripting languages (Bash, PowerShell, Python)
• Working knowledge of application, server, and network security management concepts
• Familiarity with virtual machine technologies
• Knowledge of databases like MySQL and PostgreSQL,
• Certification on any of the leading Cloud providers (AWS, Azure, GCP ) and/or Kubernetes is a big plus

The right person in this role has an opportunity to make a huge impact at Acceldata and add value to our future decisions. If this position has piqued your interest and you have what we described - we invite you to apply! An adventure in data awaits.

Learn more at https://www.acceldata.io/about-us">https://www.acceldata.io/about-us

Location: Bangalore

Function: Software Engineering → Backend Development

• Responsibilities
       - Responsible for implementation and ongoing administration of Hadoop
  infrastructure.
       - Aligning with the systems engineering team to propose and deploy new
  hardware and software environments required for Hadoop and to expand existing
  environments.
       - Working with data delivery teams to setup new Hadoop users. This job includes
  setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig
  and MapReduce access for the new users.
       - Cluster maintenance as well as creation and removal of nodes using tools like
  Ganglia, Nagios, Cloudera Manager Enterprise, Dell Open Manage and other tools
       - Performance tuning of Hadoop clusters and Hadoop MapReduce routines
       - Screen Hadoop cluster job performances and capacity planning
       - Monitor Hadoop cluster connectivity and security
       - Manage and review Hadoop log files.
       - File system management and monitoring.
       - Diligently teaming with the infrastructure, network, database, application and
  business intelligence teams to guarantee high data quality and availability
       - Collaboration with application teams to install operating system and Hadoop
  updates, patches, version upgrades when required.
      
  READ MORE OF THE JOB DESCRIPTION 
  Qualifications
  Qualifications
       - Bachelors Degree in Information Technology, Computer Science or other
  relevant fields
       - General operational expertise such as good troubleshooting skills,
  understanding of systems capacity, bottlenecks, basics of memory, CPU, OS,
  storage, and networks.
       - Hadoop skills like HBase, Hive, Pig, Mahout
       - Ability to deploy Hadoop cluster, add and remove nodes, keep track of jobs,
  monitor critical parts of the cluster, configure name node high availability, schedule
  and configure it and take backups.
       - Good knowledge of Linux as Hadoop runs on Linux.
       - Familiarity with open source configuration management and deployment tools
  such as Puppet or Chef and Linux scripting.
       Nice to Have
       - Knowledge of Troubleshooting Core Java Applications is a plus.

About the Role

The Dremio India team owns the DataLake Engine along with Cloud Infrastructure and services that power it. With focus on next generation data analytics supporting modern table formats like Iceberg, Deltalake, and open source initiatives such as Apache Arrow, Project Nessie and hybrid-cloud infrastructure, this team provides various opportunities to learn, deliver, and grow in career. We are looking for technical leaders with passion and experience in architecting and delivering high-quality distributed systems at massive scale.

Responsibilities & ownership

• Lead end-to-end delivery and customer success of next-generation features related to scalability, reliability, robustness, usability, security, and performance of the product
• Lead and mentor others about concurrency, parallelization to deliver scalability, performance and resource optimization in a multithreaded and distributed environment
• Propose and promote strategic company-wide tech investments taking care of business goals, customer requirements, and industry standards
• Lead the team to solve complex, unknown and ambiguous problems, and customer issues cutting across team and module boundaries with technical expertise, and influence others
• Review and influence designs of other team members 
• Design and deliver architectures that run optimally on public clouds like GCP, AWS, and Azure
• Partner with other leaders to nurture innovation and engineering excellence in the team
• Drive priorities with others to facilitate timely accomplishments of business objectives
• Perform RCA of customer issues and drive investments to avoid similar issues in future
• Collaborate with Product Management, Support, and field teams to ensure that customers are successful with Dremio
• Proactively suggest learning opportunities about new technology and skills, and be a role model for constant learning and growth

Requirements

• B.S./M.S/Equivalent in Computer Science or a related technical field or equivalent experience
• Fluency in Java/C++ with 15+ years of experience developing production-level software
• Strong foundation in data structures, algorithms, multi-threaded and asynchronous programming models and their use in developing distributed and scalable systems
• 8+ years experience in developing complex and scalable distributed systems and delivering, deploying, and managing microservices successfully
• Subject Matter Expert in one or more of query processing or optimization, distributed systems, concurrency, micro service based architectures, data replication, networking, storage systems
• Experience in taking company-wide initiatives, convincing stakeholders, and delivering them
• Expert in solving complex, unknown and ambiguous problems spanning across teams and taking initiative in planning and delivering them with high quality
• Ability to anticipate and propose plan/design changes based on changing requirements 
• Passion for quality, zero downtime upgrades, availability, resiliency, and uptime of the platform
• Passion for learning and delivering using latest technologies
• Hands-on experience of working projects on AWS, Azure, and GCP 
• Experience with containers and Kubernetes for orchestration and container management in private and public clouds (AWS, Azure,  and GCP) 
• Understanding of distributed file systems such as  S3, ADLS or HDFS
• Excellent communication skills and affinity for collaboration and teamwork

REQUIREMENT:

•  Previous experience of working in large scale data engineering
•  4+ years of experience working in data engineering and/or backend technologies with cloud experience (any) is mandatory.
•  Previous experience of architecting and designing backend for large scale data processing.
•  Familiarity and experience of working in different technologies related to data engineering – different database technologies, Hadoop, spark, storm, hive etc.
•  Hands-on and have the ability to contribute a key portion of data engineering backend.
•  Self-inspired and motivated to drive for exceptional results.
•  Familiarity and experience working with different stages of data engineering – data acquisition, data refining, large scale data processing, efficient data storage for business analysis.
•  Familiarity and experience working with different DB technologies and how to scale them.

RESPONSIBILITY:

•  End to end responsibility to come up with data engineering architecture, design, development and then implementation of it.
•  Build data engineering workflow for large scale data processing.
•  Discover opportunities in data acquisition.
•  Bring industry best practices for data engineering workflow.
•  Develop data set processes for data modelling, mining and production.
•  Take additional tech responsibilities for driving an initiative to completion
•  Recommend ways to improve data reliability, efficiency and quality
•  Goes out of their way to reduce complexity.
•  Humble and outgoing - engineering cheerleaders.