11+ Apache Pig Jobs in Bangalore (Bengaluru) | Apache Pig Job openings in Bangalore (Bengaluru)

• Bring in industry best practices around creating and maintaining robust data pipelines for complex data projects with/without AI component
  • programmatically ingesting data from several static and real-time sources (incl. web scraping)
  • rendering results through dynamic interfaces incl. web / mobile / dashboard with the ability to log usage and granular user feedbacks
  • performance tuning and optimal implementation of complex Python scripts (using SPARK), SQL (using stored procedures, HIVE), and NoSQL queries in a production environment

• Industrialize ML / DL solutions and deploy and manage production services; proactively handle data issues arising on live apps
• Perform ETL on large and complex datasets for AI applications - work closely with data scientists on performance optimization of large-scale ML/DL model training
• Build data tools to facilitate fast data cleaning and statistical analysis
• Ensure data architecture is secure and compliant
• Resolve issues escalated from Business and Functional areas on data quality, accuracy, and availability
• Work closely with APAC CDO and coordinate with a fully decentralized team across different locations in APAC and global HQ (Paris).

•  Expert in structured and unstructured data in traditional and Big data environments – Oracle / SQLserver, MongoDB, Hive / Pig, BigQuery, and Spark
• Have excellent knowledge of Python programming both in traditional and distributed models (PySpark)
• Expert in shell scripting and writing schedulers
• Hands-on experience with Cloud - deploying complex data solutions in hybrid cloud / on-premise environment both for data extraction/storage and computation
• Hands-on experience in deploying production apps using large volumes of data with state-of-the-art technologies like Dockers, Kubernetes, and Kafka
• Strong knowledge of data security best practices
• 5+ years experience in a data engineering role
• Science / Engineering graduate from a Tier-1 university in the country
• And most importantly, you must be a passionate coder who really cares about building apps that can help people do things better, smarter, and faster even when they sleep

Job Title: Data Engineer

Job Summary: As a Data Engineer, you will be responsible for designing, building, and maintaining the infrastructure and tools necessary for data collection, storage, processing, and analysis. You will work closely with data scientists and analysts to ensure that data is available, accessible, and in a format that can be easily consumed for business insights.

Responsibilities:

• Design, build, and maintain data pipelines to collect, store, and process data from various sources.
• Create and manage data warehousing and data lake solutions.
• Develop and maintain data processing and data integration tools.
• Collaborate with data scientists and analysts to design and implement data models and algorithms for data analysis.
• Optimize and scale existing data infrastructure to ensure it meets the needs of the business.
• Ensure data quality and integrity across all data sources.
• Develop and implement best practices for data governance, security, and privacy.
• Monitor data pipeline performance / Errors and troubleshoot issues as needed.
• Stay up-to-date with emerging data technologies and best practices.

Requirements:

Bachelor's degree in Computer Science, Information Systems, or a related field.

Experience with ETL tools like Matillion,SSIS,Informatica

Experience with SQL and relational databases such as SQL server, MySQL, PostgreSQL, or Oracle.

Experience in writing complex SQL queries

Strong programming skills in languages such as Python, Java, or Scala.

Experience with data modeling, data warehousing, and data integration.

Strong problem-solving skills and ability to work independently.

Excellent communication and collaboration skills.

Familiarity with big data technologies such as Hadoop, Spark, or Kafka.

Familiarity with data warehouse/Data lake technologies like Snowflake or Databricks

Familiarity with cloud computing platforms such as AWS, Azure, or GCP.

Familiarity with Reporting tools

Teamwork/ growth contribution

• Helping the team in taking the Interviews and identifying right candidates
• Adhering to timelines
• Intime status communication and upfront communication of any risks
• Tech, train, share knowledge with peers.
• Good Communication skills
• Proven abilities to take initiative and be innovative
• Analytical mind with a problem-solving aptitude

Good to have :

Master's degree in Computer Science, Information Systems, or a related field.

Experience with NoSQL databases such as MongoDB or Cassandra.

Familiarity with data visualization and business intelligence tools such as Tableau or Power BI.

Knowledge of machine learning and statistical modeling techniques.

If you are passionate about data and want to work with a dynamic team of data scientists and analysts, we encourage you to apply for this position.

• Partnering with internal business owners (product, marketing, edit, etc.) to understand needs and develop custom analysis to optimize for user engagement and retention
• Good understanding of the underlying business and workings of cross functional teams for successful execution
• Design and develop analyses based on business requirement needs and challenges.
• Leveraging statistical analysis on consumer research and data mining projects, including segmentation, clustering, factor analysis, multivariate regression, predictive modeling, etc.
• Providing statistical analysis on custom research projects and consult on A/B testing and other statistical analysis as needed. Other reports and custom analysis as required.
• Identify and use appropriate investigative and analytical technologies to interpret and verify results.
• Apply and learn a wide variety of tools and languages to achieve results
• Use best practices to develop statistical and/ or machine learning techniques to build models that address business needs.

• 2 - 4 years  of relevant experience in Data science.
• Preferred education: Bachelor's degree in a technical field or equivalent experience.
• Experience in advanced analytics, model building, statistical modeling, optimization, and machine learning algorithms.
• Machine Learning Algorithms: Crystal clear understanding, coding, implementation, error analysis, model tuning knowledge on Linear Regression, Logistic Regression, SVM, shallow Neural Networks, clustering, Decision Trees, Random forest, XGBoost, Recommender Systems, ARIMA and Anomaly Detection. Feature selection, hyper parameters tuning, model selection and error analysis, boosting and ensemble methods.
• Strong with programming languages like Python and data processing using SQL or equivalent and ability to experiment with newer open source tools.
• Experience in normalizing data to ensure it is homogeneous and consistently formatted to enable sorting, query and analysis.
• Experience designing, developing, implementing and maintaining a database and programs to manage data analysis efforts.
• Experience with big data and cloud computing viz. Spark, Hadoop (MapReduce, PIG, HIVE).
• Experience in risk and credit score domains preferred.

• 4+ years of experience Solid understanding of Python, Java and general software development skills (source code management, debugging, testing, deployment etc.).
• Experience in working with Solr and ElasticSearch Experience with NLP technologies & the handling of unstructured text Detailed understanding of text pre-processing and normalisation techniques such as tokenisation, lemmatisation, stemming, POS tagging etc.
• Prior experience in implementation of traditional ML solutions - classification, regression or clustering problem Expertise in text-analytics - Sentiment Analysis, Entity Extraction, Language modelling - and associated sequence learning models ( RNN, LSTM, GRU).
• Comfortable working with deep-learning libraries (eg. PyTorch)
• Candidate can even be a fresher with 1 or 2 years of experience IIIT, IIIT, Bits Pilani, top 5 local colleges are preferred colleges and universities.
• A Masters candidate in machine learning.
• Can source candidates from Mu Sigma and Manthan.

Job Description

Experience: 3+ yrs

We are looking for a MySQL DBA who will be responsible for ensuring the performance, availability, and security of clusters of MySQL instances. You will also be responsible for design of database, database architecture, orchestrating upgrades, backups, and provisioning of database instances. You will also work in tandem with the other teams, preparing documentations and specifications as required.

Responsibilities:

Database design and data architecture

Provision MySQL instances, both in clustered and non-clustered configurations

Ensure performance, security, and availability of databases

Prepare documentations and specifications

Handle common database procedures, such as upgrade, backup, recovery, migration, etc.

Profile server resource usage, optimize and tweak as necessary

Skills and Qualifications:

Proven expertise in database design and data architecture for large scale systems

Strong proficiency in MySQL database management

Decent experience with recent versions of MySQL

Understanding of MySQL's underlying storage engines, such as InnoDB and MyISAM

Experience with replication configuration in MySQL

Knowledge of de-facto standards and best practices in MySQL

Proficient in writing and optimizing SQL statements

Knowledge of MySQL features, such as its event scheduler

Ability to plan resource requirements from high level specifications

Familiarity with other SQL/NoSQL databases such as Cassandra, MongoDB, etc.

Knowledge of limitations in MySQL and their workarounds in contrast to other popular relational databases

The Data Engineer would be responsible for selecting and integrating Big Data tools and frameworks required. Would implement Data Ingestion & ETL/ELT processes

Required Experience, Skills and Qualifications:

• Hands on experience on Big Data tools/technologies like Spark,  Databricks, Map Reduce, Hive, HDFS.
• Expertise and excellent understanding of big data toolset such as Sqoop, Spark-streaming, Kafka, NiFi
• Proficiency in any of the programming language: Python/ Scala/  Java with 4+ years’ experience
• Experience in Cloud infrastructures like MS Azure, Data lake etc
• Good working knowledge in NoSQL DB (Mongo, HBase, Casandra)