Big Data Engineer

at Product Company Chennai based

Data pre-processing, data transformation, data analysis, and feature engineering, 
The candidate must have Expertise in ADF(Azure data factory), well versed with python.
Performance optimization of scripts (code) and Productionizing of code (SQL, Pandas, Python or PySpark, etc.)
Required skills:
Bachelors in - in Computer Science, Data Science, Computer Engineering, IT or equivalent
Fluency in Python (Pandas), PySpark, SQL, or similar
Azure data factory experience (min 12 months)
Able to write efficient code using traditional, OO concepts, modular programming following the SDLC process.
Experience in production optimization and end-to-end performance tracing (technical root cause analysis)
Ability to work independently with demonstrated experience in project or program management
Azure experience ability to translate data scientist code in Python and make it efficient (production) for cloud deployment

Designation: Specialist - Cloud Service Developer (ABL_SS_600)

Position description:

• The person would be primary responsible for developing solutions using AWS services. Ex: Fargate, Lambda, ECS, ALB, NLB, S3 etc.
• Apply advanced troubleshooting techniques to provide Solutions to issues pertaining to Service Availability, Performance, and Resiliency
• Monitor & Optimize the performance using AWS dashboards and logs
• Partner with Engineering leaders and peers in delivering technology solutions that meet the business requirements 
• Work with the cloud team in agile approach and develop cost optimized solutions

 

Primary Responsibilities:

• Develop solutions using AWS services includiing Fargate, Lambda, ECS, ALB, NLB, S3 etc.

 

Reporting Team

• Reporting Designation: Head - Big Data Engineering and Cloud Development (ABL_SS_414)
• Reporting Department: Application Development (2487)

Required Skills:

• AWS certification would be preferred
• Good understanding in Monitoring (Cloudwatch, alarms, logs, custom metrics, Trust SNS configuration)
• Good experience with Fargate, Lambda, ECS, ALB, NLB, S3, Glue, Aurora and other AWS services. 
• Preferred to have Knowledge on Storage (S3, Life cycle management, Event configuration)
• Good in data structure, programming in (pyspark / python / golang / Scala)

We are actively seeking a Senior Data Engineer experienced in building data pipelines and integrations from 3rd party data sources by writing custom automated ETL jobs using Python. The role will work in partnership with other members of the Business Analytics team to support the development and implementation of new and existing data warehouse solutions for our clients. This includes designing database import/export processes used to generate client data warehouse deliverables.

 

Job Description :- 5 to 11 years experience- Experienced any of the BI tools like PowerBI, QlikView.- Writing SQLqueries/functions/procedures on big data.- Apt in databases NO SQL : mongoDB/ Cassandra OR MySQL, PostgresSQL, SQLServer- Able to Analyze clean,massage, cleanse and organize raw (big) data. - Manage Security for data on AWS or Azure cloud platform - Create, validate and maintain optimal data pipelines, assemble large, complex data sets - Helps in Structuring for upstream / downstream processing

You will work on: 

We help many of our clients make sense of their large investments in data – be it building analytics solutions or machine learning applications. You will work on cutting edge cloud native technologies to crunch terabytes of data into meaningful insights. 

 

What you will do (Responsibilities):

• Collaborate with Data Scientists, Engineers and Product Management to transform raw data to often actionable and meaningful insights for the enterprise
• Work in small dynamic, product-oriented environment to deliver enterprise class products.
• Continuously improve software development practices work across the full stack.

 

What you bring (Skills):

 

• Experience in building modern cloud-native microservices based applications using state of the art frameworks in Java/Kotlin, Spring Boot, Hibernate and other JVM based frameworks.
• Experience developing elastic, cloud based, BigData applications with modern technologies like Apache Spark, BigQuery, Airflow, Beam, Kafka and ElasticSearch.               
• Ability to produce easily consumable RESTful APIs with strong living documentation and specification-by-example tests.

 

Great if you know (Skills):

• T-shaped skills are always preferred – so if you have the passion to work across the full stack spectrum – it is more than welcome.
• Exposure to infrastructure-based skills like Docker, Istio, Kubernetes is a plus
• Ability to work out right strategies of deployment for BigData systems.
• Collaborate with DevOps and Test Automation teams to build favorable developer experience in both build and CI/CD.
• Attitude to look at an application holistically and debug any component if required whether it be based on UI technology like JavaScript, HTML or an intricate Linux System based issue.

 

 
Advantage Cognologix:

•  Higher degree of autonomy, startup culture & small teams
•  Opportunities to become expert in emerging technologies
•  Remote working options for the right maturity level
•  Competitive salary & family benefits
•  Performance based career advancement

 

About Cognologix:

 

Cognologix helps companies disrupt by reimagining their business models and innovate like a Startup. We are at the forefront of digital disruption and take a business first approach to help meet our client’s strategic goals.

We are an Data focused organization helping our clients to deliver their next generation of products in the most efficient, modern and cloud-native way.

 

Job Responsibilities :  
As a Data Warehouse Engineer in our team, you should have a proven ability to deliver high-quality work on time and with minimal supervision.
Develops or modifies procedures to solve complex database design problems, including performance, scalability, security and integration issues for various clients (on-site and off-site).
Design, develop, test, and support the data warehouse solution.
Adapt best practices and industry standards, ensuring top quality deliverable''s and playing an integral role in cross-functional system integration.
Design and implement formal data warehouse testing strategies and plans including unit testing, functional testing, integration testing, performance testing, and validation testing.
Evaluate all existing hardware's and software's according to required standards and ability to configure the hardware clusters as per the scale of data.
Data integration using enterprise development tool-sets (e.g. ETL, MDM, Quality, CDC, Data Masking, Quality).
Maintain and develop all logical and physical data models for enterprise data warehouse (EDW).
Contributes to the long-term vision of the enterprise data warehouse (EDW) by delivering Agile solutions.
Interact with end users/clients and translate business language into technical requirements.
Acts independently to expose and resolve problems.  
Participate in data warehouse health monitoring and performance optimizations as well as quality documentation.

Job Requirements :  
2+ years experience working in software development & data warehouse development for enterprise analytics.
2+ years of working with Python with major experience in Red-shift as a must and exposure to other warehousing tools.
Deep expertise in data warehousing, dimensional modeling and the ability to bring best practices with regard to data management, ETL, API integrations, and data governance.
Experience working with data retrieval and manipulation tools for various data sources like Relational (MySQL, PostgreSQL, Oracle), Cloud-based storage.
Experience with analytic and reporting tools (Tableau, Power BI, SSRS, SSAS). Experience in AWS cloud stack (S3, Glue, Red-shift, Lake Formation).
Experience in various DevOps practices helping the client to deploy and scale the systems as per requirement.
Strong verbal and written communication skills with other developers and business clients.
Knowledge of Logistics and/or Transportation Domain is a plus.
Ability to handle/ingest very huge data sets (both real-time data and batched data) in an efficient manner.

ETL Developer – Talend

Job Duties:

• ETL Developer is responsible for Design and Development of ETL Jobs which follow standards,

best practices and are maintainable, modular and reusable.

• Proficiency with Talend or Pentaho Data Integration / Kettle.
• ETL Developer will analyze and review complex object and data models and the metadata

repository in order to structure the processes and data for better management and efficient

access.

• Working on multiple projects, and delegating work to Junior Analysts to deliver projects on time.
• Training and mentoring Junior Analysts and building their proficiency in the ETL process.
• Preparing mapping document to extract, transform, and load data ensuring compatibility with

all tables and requirement specifications.

• Experience in ETL system design and development with Talend / Pentaho PDI is essential.
• Create quality rules in Talend.
• Tune Talend / Pentaho jobs for performance optimization.
• Write relational(sql) and multidimensional(mdx) database queries.
• Functional Knowledge of Talend Administration Center/ Pentaho data integrator, Job Servers &

Load balancing setup, and all its administrative functions.

• Develop, maintain, and enhance unit test suites to verify the accuracy of ETL processes,

dimensional data, OLAP cubes and various forms of BI content including reports, dashboards,

and analytical models.

• Exposure in Map Reduce components of Talend / Pentaho PDI.
• Comprehensive understanding and working knowledge in Data Warehouse loading, tuning, and

maintenance.

• Working knowledge of relational database theory and dimensional database models.
• Creating and deploying Talend / Pentaho custom components is an add-on advantage.
• Nice to have java knowledge.

Skills and Qualification:

• BE, B.Tech / MS Degree in Computer Science, Engineering or a related subject.
• Having an experience of 3+ years.
• Proficiency with Talend or Pentaho Data Integration / Kettle.
• Ability to work independently.
• Ability to handle a team.
• Good written and oral communication skills.

• Looking for Big Data Engineer with 3+ years of experience. • Hands-on experience with MapReduce-based platforms, like Pig, Spark, Shark. • Hands-on experience with data pipeline tools like Kafka, Storm, Spark Streaming. • Store and query data with Sqoop, Hive, MySQL, HBase, Cassandra, MongoDB, Drill, Phoenix, and Presto. • Hands-on experience in managing Big Data on a cluster with HDFS and MapReduce. • Handle streaming data in real time with Kafka, Flume, Spark Streaming, Flink, and Storm. • Experience with Azure cloud, Cognitive Services, Databricks is preferred.