Data Engineer

1. Excellent understanding of at least any one of the programming language .NET ,Python, Perl, and Java
2. Good understanding and hands on experience in Shell/Bash scripting, sonarqube, Terraform,
3. Experience with Continuous Integration and Continuous Deployment Pipelines
4. Experience in SVN, Maven, Git and Git workflows
5. Should be able to develop overall strategy for Build & Release management
6. Experience in working with container orchestration tools such as Docker and Kubernetes
7. Good knowledge in Devops Automation Tools like Chef, Ansible, Puppet, helm, splunk, maven, gradle & XL Deploy.etc
8. Managing stakeholders and external interfaces and Setting up tools and required infrastructure
9. Encouraging and building automated processes wherever possible
10. Awareness of critical concepts in DevOps and Agile principles
11. Experience in Cloud infrastructure like AWS, GCP or Azure. In AWS understanding on EC2, S3 & cloud
12. Strong knowledge and hands on experience in unix OS
13.Experience in network, server, application status monitoring and troubleshooting, Security.
14.Design, develop automation suite and integrate with continuous integration process through Jenkins
15. Possess good problem solving and debugging skills. Troubleshoot issues and coordinate with development team to streamline code deployment to generate build

About CarWale: CarWale's mission is to bring delight in car buying, we offer a bouquet of reliable tools and services to help car consumers decide on buying the right car, at the right price and from the right partner. CarWale has always strived to serve car buyers and owners in the most comprehensive and convenient way possible. We provide a platform where car buyers and owners can research, buy, sell and come together to discuss and talk about their cars.We aim to empower Indian consumers to make informed car buying and ownership decisions with exhaustive and un-biased information on cars through our expert reviews, owner reviews, detailed specifications and comparisons. We understand that a car is by and large the second-most expensive asset a consumer associates his lifestyle with! Together with CarTrade & BikeWale, we are the market leaders in the personal mobility media space.About the Team:We are a bunch of enthusiastic analysts assisting all business functions with their data needs. We deal with huge but diverse datasets to find relationships, patterns and meaningful insights. Our goal is to help drive growth across the organization by creating a data-driven culture.

We are looking for an experienced Data Scientist who likes to explore opportunities and know their way around data to build world class solutions making a real impact on the business. 

 

Skills / Requirements –

• 3-5 years of experience working on Data Science projects
• Experience doing statistical modelling of big data sets
• Expert in Python, R language with deep knowledge of ML packages
• Expert in fetching data from SQL
• Ability to present and explain data to management
• Knowledge of AWS would be beneficial
• Demonstrate Structural and Analytical thinking
• Ability to structure and execute data science project end to end

 

Education –

Bachelor’s degree in a quantitative field (Maths, Statistics, Computer Science). Masters will be preferred.

 

About LodgIQ

LodgIQ is led by a team of experienced hospitality technology experts, data scientists and product domain experts. Seed funded by Highgate Ventures, a venture capital platform focused on early stage technology investments in the hospitality industry and Trilantic Capital Partners, a global private equity firm, LodgIQ has made a significant investment in advanced machine learning platforms and data science.

Title : Data Scientist

• Apply Data Science and Machine Learning to a REAL-LIFE  problem  - “Predict Guest Arrivals and Determine Best Prices for Hotels”
• Apply advanced analytics in a BIG Data Environment – AWS, MongoDB, SKLearn
• Help scale up the product in a global offering across 100+  global markets

• Minimum 3 years of experience with advanced data analytic techniques, including data mining, machine learning, statistical analysis, and optimization.    Student projects are acceptable.
• At least 1 year of experience with Python / Numpy / Pandas / Scipy/ MatPlotLib / Scikit-Learn
•   Experience in working with massive data sets, including structured and unstructured with at least 1 prior engagement involving data gathering, data cleaning, data mining, and data visualization
•   Solid grasp over  optimization techniques
•   Master's  or PhD degree in Business Analytics.  Data science, Statistics or Mathematics
•   Ability to show a track record of solving large, complex problems

Job Description:

We are looking for a Big Data Engineer who have worked across the entire ETL stack. Someone who has ingested data in a batch and live stream format, transformed large volumes of daily and built Data-warehouse to store the transformed data and has integrated different visualization dashboards and applications with the data stores.    The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them.

Responsibilities:

• Develop, test, and implement data solutions based on functional / non-functional business requirements.
• You would be required to code in Scala and PySpark daily on Cloud as well as on-prem infrastructure
• Build Data Models to store the data in a most optimized manner
• Identify, design, and implement process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Implementing the ETL process and optimal data pipeline architecture
• Monitoring performance and advising any necessary infrastructure changes.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Proactively identify potential production issues and recommend and implement solutions
• Must be able to write quality code and build secure, highly available systems.
• Create design documents that describe the functionality, capacity, architecture, and process.
• Review peer-codes and pipelines before deploying to Production for optimization issues and code standards

Skill Sets:

• Good understanding of optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and ‘big data’ technologies.
• Proficient understanding of distributed computing principles
• Experience in working with batch processing/ real-time systems using various open-source technologies like NoSQL, Spark, Pig, Hive, Apache Airflow.
• Implemented complex projects dealing with the considerable data size (PB).
• Optimization techniques (performance, scalability, monitoring, etc.)
• Experience with integration of data from multiple data sources
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB, etc.,
• Knowledge of various ETL techniques and frameworks, such as Flume
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Creation of DAGs for data engineering
• Expert at Python /Scala programming, especially for data engineering/ ETL purposes

 

 

 

• Key responsibility is to design & develop a data pipeline for real-time data integration, processing, executing of the model (if required), and exposing output via MQ / API / No-SQL DB for consumption
• Provide technical expertise to design efficient data ingestion solutions to store & process unstructured data, such as Documents, audio, images, weblogs, etc
• Developing API services to provide data as a service
• Prototyping Solutions for complex data processing problems using AWS cloud-native solutions
• Implementing automated Audit & Quality assurance Checks in Data Pipeline
• Document & maintain data lineage from various sources to enable data governance
• Coordination with BIU, IT, and other stakeholders to provide best-in-class data pipeline solutions, exposing data via APIs, loading in down streams, No-SQL Databases, etc

Skills

• Programming experience using Python & SQL
• Extensive working experience in Data Engineering projects, using AWS Kinesys,  AWS S3, DynamoDB, EMR, Lambda, Athena, etc for event processing
• Experience & expertise in implementing complex data pipeline
• Strong Familiarity with AWS Toolset for Storage & Processing. Able to recommend the right tools/solutions available to address specific data processing problems
• Hands-on experience in Unstructured (Audio, Image, Documents, Weblogs, etc) Data processing.
• Good analytical skills with the ability to synthesize data to design and deliver meaningful information
• Know-how on any No-SQL DB (DynamoDB, MongoDB, CosmosDB, etc) will be an advantage.
• Ability to understand business functionality, processes, and flows
• Good combination of technical and interpersonal skills with strong written and verbal communication; detail-oriented with the ability to work independently

Functional knowledge

• Real-time Event Processing
• Data Governance & Quality assurance
• Containerized deployment
• Linux
• Unstructured Data Processing
• AWS Toolsets for Storage & Processing
• Data Security

 

Senior Big Data Engineer 

Note:   Notice Period : 45 days 

Banyan Data Services (BDS) is a US-based data-focused Company that specializes in comprehensive data solutions and services, headquartered in San Jose, California, USA. 

 

We are looking for a Senior Hadoop Bigdata Engineer who has expertise in solving complex data problems across a big data platform. You will be a part of our development team based out of Bangalore. This team focuses on the most innovative and emerging data infrastructure software and services to support highly scalable and available infrastructure. 

 

It's a once-in-a-lifetime opportunity to join our rocket ship startup run by a world-class executive team. We are looking for candidates that aspire to be a part of the cutting-edge solutions and services we offer that address next-gen data evolution challenges. 

 

 

Key Qualifications

 

·   5+ years of experience working with Java and Spring technologies

· At least 3 years of programming experience working with Spark on big data; including experience with data profiling and building transformations

· Knowledge of microservices architecture is plus 

· Experience with any NoSQL databases such as HBase, MongoDB, or Cassandra

· Experience with Kafka or any streaming tools

· Knowledge of Scala would be preferable

· Experience with agile application development 

· Exposure of any Cloud Technologies including containers and Kubernetes 

· Demonstrated experience of performing DevOps for platforms 

· Strong Skillsets in Data Structures & Algorithm in using efficient way of code complexity

· Exposure to Graph databases

· Passion for learning new technologies and the ability to do so quickly 

· A Bachelor's degree in a computer-related field or equivalent professional experience is required

 

Key Responsibilities

 

· Scope and deliver solutions with the ability to design solutions independently based on high-level architecture

· Design and develop the big data-focused micro-Services

· Involve in big data infrastructure, distributed systems, data modeling, and query processing

· Build software with cutting-edge technologies on cloud

· Willing to learn new technologies and research-orientated projects 

· Proven interpersonal skills while contributing to team effort by accomplishing related results as needed 

Technology Skills (Good to Have):

• We are looking for a Data Engineer to build the next-generation mobile applications for our world-class fintech product.
• The candidate will be responsible for expanding and optimising our data and data pipeline architecture, as well as optimising data flow and collection for cross-functional teams.
• The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimising data systems and building them from the ground up.
• Looking for a person with a strong ability to analyse and provide valuable insights to the product and business team to solve daily business problems.
• You should be able to work in a high-volume environment, have outstanding planning and organisational skills.

 

Qualifications for Data Engineer

 

• Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimising ‘big data’ data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets. Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Looking for a candidate with 2-3 years of experience in a Data Engineer role, who is a CS graduate or has an equivalent experience.

 

What we're looking for?

 

• Experience with big data tools: Hadoop, Spark, Kafka and other alternate tools.
• Experience with relational SQL and NoSQL databases, including MySql/Postgres and Mongodb.
• Experience with data pipeline and workflow management tools: Luigi, Airflow.
• Experience with AWS cloud services: EC2, EMR, RDS, Redshift.
• Experience with stream-processing systems: Storm, Spark-Streaming.
• Experience with object-oriented/object function scripting languages: Python, Java, Scala.

Looking out for Internship Candidates . Designation:- Intern/ Trainee Technology : .NET/JAVA/ Python/ AI/ ML Duration : 2-3 Months Job Location :Online Internship Joining :Immediately Job Type :Internship Job Description - MCA/M.Tech/ B.Tech/ BE who need 2-6 months internship project to be done. - Should be available to join us immediately. - Should be flexible to work on any Skills/ Technologies. - Ready to work in long working hours. - Must possess excellent analytical and logical skills. - Internship experience is provided from experts - Internship Certificate will be provided at the end of training. - The requirement is strictly for internship and not a permanent job - Stipend will be provided only based on the performance.