Data Engineer
at consulting & implementation services in the area of Oil & Gas, Mining and Manufacturing Industry
- Data Engineer
Required skill set: AWS GLUE, AWS LAMBDA, AWS SNS/SQS, AWS ATHENA, SPARK, SNOWFLAKE, PYTHON
Mandatory Requirements
- Experience in AWS Glue
- Experience in Apache Parquet
- Proficient in AWS S3 and data lake
- Knowledge of Snowflake
- Understanding of file-based ingestion best practices.
- Scripting language - Python & pyspark
CORE RESPONSIBILITIES
- Create and manage cloud resources in AWS
- Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies
- Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform
- Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations
- Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
- Define process improvement opportunities to optimize data collection, insights and displays.
- Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible
- Identify and interpret trends and patterns from complex data sets
- Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders.
- Key participant in regular Scrum ceremonies with the agile teams
- Proficient at developing queries, writing reports and presenting findings
- Mentor junior members and bring best industry practices
QUALIFICATIONS
- 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales)
- Strong background in math, statistics, computer science, data science or related discipline
- Advanced knowledge one of language: Java, Scala, Python, C#
- Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake
- Proficient with
- Data mining/programming tools (e.g. SAS, SQL, R, Python)
- Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
- Data visualization (e.g. Tableau, Looker, MicroStrategy)
- Comfortable learning about and deploying new technologies and tools.
- Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.
- Good written and oral communication skills and ability to present results to non-technical audiences
- Knowledge of business intelligence and analytical tools, technologies and techniques.
Familiarity and experience in the following is a plus:
- AWS certification
- Spark Streaming
- Kafka Streaming / Kafka Connect
- ELK Stack
- Cassandra / MongoDB
- CI/CD: Jenkins, GitLab, Jira, Confluence other related tools
Similar jobs
Apply only if you can join within 2 week of selection.
Ideal for candidates who would like to be part of the learning process across growing technologies and embark journey on a tech architect path. Very deep into technologies and eager to learn and adapt with technology advancement
Job Description:
• Proficient ASP.NET/.NET CORE, Web API, MVC, Entity framework and MS SQL/MySQL is mandatory.
• Experience with HTML5/CSS3, JavaScript/jQuery, Angular; web-service: SOAP, RESFUL, MVC API
• Willing to learn new technologies
• Have good knowledge or experience with Agile/SCRUM development method
• Detail oriented, self-motivated and disciplined, with ability to complete assigned tasks with minimal oversight in a professional and timely manner is a must
|| #5daysworking || #flexibleworkinghours || #employeeorientedpolicy || #attractivebonus
Our client is a rapid growth stage Edtech start-up focused on solving the teacher shortage crisis in the US education system using technology by bringing teachers to the classrooms and giving them access to the right tools to teach students in a highly engaging manner.
They have a development center in Mumbai and they are looking to scale up the teams in Pune and Bangalore as well.
What does the role entail:
● Lead User Stories and guide senior/software engineers in the development process
● Understand architecture principles, design patterns and implement them from architectural artifacts.
● Provide effort estimation of assigned work and be able to complete as per the estimations and timelines
● Write clean, maintainable, and efficient code including unit tests and integration tests to ensure software quality
● Collaborate with cross-functional teams to analyze requirements, design solutions, and implement new features
● Participate in design and code reviews
● Ability to resolve performance issues
● Mentor junior software engineers and help them grow their technical skills
Must Have Skills:
● B.E/B.Tech in computer science
● 8-11 years of experience
● Hands-on experience following skills
MS Technologies: NET CORE 5+, C#
Backend : MSSQL /MySQL/Postgres
● Good understanding of Cloud technologies like Azure/AWS.
● Good understanding of Design Principles, Design patterns and Microservices Architecture.
● Excellent problem-solving, critical thinking, and communication skills
● Worked in an Agile software development environment
Must understand full stack development frameworks including knowledge of building scalable APIs, interfaces, software components, schema design, availability, and latency preferably in a cloud environment.
● Worked in a fast-paced environment preferably in a tech driven startup
● Understanding of different types of architectures.
Tech Lead
Key goals and responsibilities
· Building reusable components and front-end libraries for future use
· Familiarity with databases (e.g. MySQL, MongoDB), web servers (e.g. Apache
· Ensure best coding practices to build efficiencies and scalability of the tech stack; to be implemented by the team.
· Ensure defect-free software deployment and maintenance thereof.
· Focus on critical trouble shooting issues; be obstacle remover for the team.
· Create and maintain project documentation and tech specs.
· Review the code periodically from technical and security point of view with due recommendations.
· Track team on performance and task management to deliver to timelines and quality.
· Thorough understanding of React.js, Vue Js and its core principles
· Familiarity with modern front-end build pipelines and tools
· A knack of benchmarking and optimization
· Familiarity with code versioning tools such as Git, SVN, and Mercurial
· Good understanding of quality JavaScript code (Object oriented and commonly used design patterns )
· Capable of debugging complex issues in JavaScript
· Understanding of JS debug tools and chrome dev tools
Your key qualifications & competencies:
· Understanding of web markup, including HTML5 and CSS3
· Familiarity with RESTful APIs
· Knowledge of modern authorization mechanisms, such as JSON Web Token
· Experience with data structure libraries (e.g., Immutable.js)
· At least 5+ years of hands on coding experience
· Strong understanding of OOPS Concepts and data structure
· Have practical knowledge/experience in JAVA , Microservices , Springboot , hibernate and MySQL database
· Have prior experience in scrum and agile methodologies
· Should have 5+ years of experience in modern JavaScript frameworks such as React.js /Vue/AngularJS /Knockout
· Must have experience on AWS, or Google cloud. Must have knowledge of GIT.
· Should have sound understanding of http and REST. Be a proponent of strict code quality standards using JSLint etc.
· Should have atleast 3+ years of leading teams and proven record of adherence to ambitious deadlines.
· Should have strong understanding of object oriented and functional programming in JavaScript.
Looking for Experienced Trainers with experience in Dotnet Training
Must be good in Fullstack-MERN
Azure knowledge is desired
Roles and
Responsibilities
Seeking AWS Cloud Engineer /Data Warehouse Developer for our Data CoE team to
help us in configure and develop new AWS environments for our Enterprise Data Lake,
migrate the on-premise traditional workloads to cloud. Must have a sound
understanding of BI best practices, relational structures, dimensional data modelling,
structured query language (SQL) skills, data warehouse and reporting techniques.
Extensive experience in providing AWS Cloud solutions to various business
use cases.
Creating star schema data models, performing ETLs and validating results with
business representatives
Supporting implemented BI solutions by: monitoring and tuning queries and
data loads, addressing user questions concerning data integrity, monitoring
performance and communicating functional and technical issues.
Job Description: -
This position is responsible for the successful delivery of business intelligence
information to the entire organization and is experienced in BI development and
implementations, data architecture and data warehousing.
Requisite Qualification
Essential
-
AWS Certified Database Specialty or -
AWS Certified Data Analytics
Preferred
Any other Data Engineer Certification
Requisite Experience
Essential 4 -7 yrs of experience
Preferred 2+ yrs of experience in ETL & data pipelines
Skills Required
Special Skills Required
AWS: S3, DMS, Redshift, EC2, VPC, Lambda, Delta Lake, CloudWatch etc.
Bigdata: Databricks, Spark, Glue and Athena
Expertise in Lake Formation, Python programming, Spark, Shell scripting
Minimum Bachelor’s degree with 5+ years of experience in designing, building,
and maintaining AWS data components
3+ years of experience in data component configuration, related roles and
access setup
Expertise in Python programming
Knowledge in all aspects of DevOps (source control, continuous integration,
deployments, etc.)
Comfortable working with DevOps: Jenkins, Bitbucket, CI/CD
Hands on ETL development experience, preferably using or SSIS
SQL Server experience required
Strong analytical skills to solve and model complex business requirements
Sound understanding of BI Best Practices/Methodologies, relational structures,
dimensional data modelling, structured query language (SQL) skills, data
warehouse and reporting techniques
Preferred Skills
Required
Experience working in the SCRUM Environment.
Experience in Administration (Windows/Unix/Network/
plus.
Experience in SQL Server, SSIS, SSAS, SSRS
Comfortable with creating data models and visualization using Power BI
Hands on experience in relational and multi-dimensional data modelling,
including multiple source systems from databases and flat files, and the use of
standard data modelling tools
Ability to collaborate on a team with infrastructure, BI report development and
business analyst resources, and clearly communicate solutions to both
technical and non-technical team members
- Design, build and maintain efficient, reusable, and reliable Java code.
- Providing technical oversight to the team, involved in design and code review.
- You would also be spending most of the time in development activities in varied technologies and should have a passion to write code.
- Should be able to create good technical documents.
- Translate application storyboards and use cases into functional applications.
- Ensure the best possible performance, quality, and responsiveness of the applications.
- Identify bottlenecks and bugs, and devise solutions to these problems - Help maintain code quality, organization, and automatization.
- Prepare the technical design of complex technology components as well as suggest the pros and cons of using a certain technology stack or component or design pattern versus another, to the clients.
- Team Management.
Required Skills and Qualifications
- Qualifications: BTECH/MTECH/MCA/MSc.
- Proficient in Java, with a good knowledge of its ecosystems with a knack for writing clean, readable Java code, writing reusable Java libraries along with knowledge of multithreading, concurrency patterns, and collections in Java.
- Solid understanding of object-oriented programming along with various design and architectural patterns.
- Hands on experience with JMS, JPA, Spring (MVC, Boot & Cloud preferred) & Hibernate.
- Familiarity with concepts of MVC, JDBC, and RESTful. Experience with Presentation Layer (JSP/Servlets), JS Frameworks (Angular, jQuery, react, etc.).
- Creating database schemas that represent and support business processes and experience with both external and embedded databases. Implementing automated testing platforms and unit tests.
- Proficient understanding of code versioning tools, such as Git, Ant, Maven, and Gradle; continuous integration.
- Knowledge in xml based mappings, SAML, rest clients, CAS authentication and Jetty.
- Knowledge of Apache Camel and Kafka and Drools Rule Engine is preferred.
exp - 3+years
Location - Pune , Bangalore and Trivandrum
Job description -
- Experience of Core Java, REST-API and Micro-Services development,
- Experience of Spring boot framework
- Strong Object-oriented concepts, Design Patterns
- Experience of SQL/NoSQL
- Well versed with Cloud Development technologies and Microservices architecture.
- CICD pipeline understanding, CICD tools such as Jenkin, Github etc.
- Experience designing data flows e.g. streaming, batch processing, parallel execution (Kafka, Kinesis, etc.)