i

Big Data Engineer
at Product Company Chennai based

Apply to this job
i
Remote only
i
4 - 8 yrs
i
₹10L - ₹15L / yr
Skills
Big Data
Hadoop
Spark
Amazon Web Services (AWS)
kafka
Job description
  • Hands-on programming expertise in Java OR Python
  • Strong production experience with Spark (Minimum of 1-2 years)
  • Experience in data pipelines using Big Data technologies (Hadoop, Spark, Kafka, etc.,) on large scale unstructured data sets
  • Working experience and good understanding of public cloud environments (AWS OR Azure OR Google Cloud)
  • Experience with IAM policy and role management is a plus
Why apply to jobs via CutShort
i
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
i
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
i
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
2101133
Matches delivered
3712187
Network size
6212
Companies hiring
Similar jobs
i
Founded 2017  •  Product  •  0-20 employees  •  Raised funding
Big Data
PySpark
Hadoop
Apache Spark
Apache Hive
Python
Flask
i
Bengaluru (Bangalore)
i
3 - 6 yrs
i
₹11L - ₹25L / yr
Senior Big Data Engineer, Corridor Platforms Inc.

Corridor Platforms Inc., with offices in NJ, USA and Bangalore, India is looking to expand its technology team in India. The company is developing a fully automated Credit Risk management platform that leverages big data technologies, data science and latest front-end technologies. The product features prospecting, underwriting, loan selection, portfolio management and many other risk solutions. This risk management platform is primarily targeted for use by investors/mid-tier
and large-tier banks in funding consumer/institutional loans (helps manage risk and grow revenue). It comes with excellent UX and developed with latest technologies. New hires would experience and interact with leading experts in Credit Risk, Data science (ML/AI), Big Data and front-end full stack.
We are looking for creative and talented individuals to join a highly motivated team to
conceptualize, build and deploy risk decision platforms for consumer lending in a Big Data environment. An ideal candidate must have experience leading teams in E2E development and integration of Big Data Technologies, Big Data Warehousing, Advanced analytical platforms and production systems for deployment in batch and real-time. The individual must be action- oriented self-starter, process driven, maintain alignment with the team, abide by the company’s culture and follow industry-wide best practices.

Responsibilities
● Design and develop highly scalable, fault tolerant platform using python and big data technologies
● Work effectively in a small, adaptable, agile team with each member having complete independence to develop creative solutions
● Understand and enforce the right balance between innovation, experimentation and
delivering completed features to production in a timely manner
● Understand the core business and apply the technologies to provide clients with best-in-class big data analytical platform Must haves
● Must have deep experience developing products/platform leveraging Big Data
Technologies 
● 3+ years of experience programming enterprise solutions in Python (preferably) or
Java/Scala
● 2+ years of experience in Open Source Apache projects: Spark, Hive, or other Big Data technology

Nice to haves
● Experience with designing APIs (Flask) with RDBMS databases (MySQL, Postgres,
Oracle, etc).
● Experience in compilers and transliteration between languages and analytical platforms
● Experience in Business Insight and Dashboarding/Repor
● Experience in integrating with multiple analytical tools like Tableau, H2O, DataRobot,
Jupyter, etc.
●Experience with integrating into production decisioning systems like Provenir, Zoot,
FICO-Blaze, Experian-Powercurve, etc.
● Experience with cloud based
Read more
Job posted by
i
Niti Anand
Apply for job
i
Founded 2014  •  Products & Services  •  20-100 employees  •  Bootstrapped
Python
PySpark
Snow flake schema
SQL
PL/SQL
Microsoft Windows Azure
Amazon Web Services (AWS)
Data Warehouse (DWH)
i
Remote, Bengaluru (Bangalore)
i
3 - 7 yrs
i
₹5L - ₹10L / yr

Basic Qualifications

- Need to have a working knowledge of AWS Redshift.

- Minimum 1 year of designing and implementing a fully operational production-grade large-scale data solution on Snowflake Data Warehouse.

- 3 years of hands-on experience with building productized data ingestion and processing pipelines using Spark, Scala, Python

- 2 years of hands-on experience designing and implementing production-grade data warehousing solutions

- Expertise and excellent understanding of Snowflake Internals and integration of Snowflake with other data processing and reporting technologies

- Excellent presentation and communication skills, both written and verbal

- Ability to problem-solve and architect in an environment with unclear requirements

Read more
Job posted by
i
Vishal Sharma
Apply for job
i
at Service based company
Performance optimixation
oo concepts
SQL
Python
pandas
PySpark
Big Data
Data engineering
i
Remote only
i
3 - 8 yrs
i
₹8L - ₹13L / yr
Data pre-processing, data transformation, data analysis, and feature engineering, 
The candidate must have Expertise in ADF(Azure data factory), well versed with python.
Performance optimization of scripts (code) and Productionizing of code (SQL, Pandas, Python or PySpark, etc.)
Required skills:
Bachelors in - in Computer Science, Data Science, Computer Engineering, IT or equivalent
Fluency in Python (Pandas), PySpark, SQL, or similar
Azure data factory experience (min 12 months)
Able to write efficient code using traditional, OO concepts, modular programming following the SDLC process.
Experience in production optimization and end-to-end performance tracing (technical root cause analysis)
Ability to work independently with demonstrated experience in project or program management
Azure experience ability to translate data scientist code in Python and make it efficient (production) for cloud deployment
Read more
Job posted by
i
Sonali Kamani
Apply for job
i
Founded 1987  •  Product  •  500-1000 employees  •  Profitable
Amazon Web Services (AWS)
Python
Scala
Go Programming (Golang)
Java
AWS Lambda
ECS
NLB
Amazon S3
Apache Aurora
Spark
PySpark
Apache Kafka
Redis
Amazon VPC
athena
Amazon EMR
Serverless
Kubernetes
fargate
ALB
Glue
cloudwatch
container
i
Remote only
i
2 - 6 yrs
i
₹12L - ₹18L / yr

Designation: Specialist - Cloud Service Developer (ABL_SS_600)

Position description:

  • The person would be primary responsible for developing solutions using AWS services. Ex: Fargate, Lambda, ECS, ALB, NLB, S3 etc.
  • Apply advanced troubleshooting techniques to provide Solutions to issues pertaining to Service Availability, Performance, and Resiliency
  • Monitor & Optimize the performance using AWS dashboards and logs
  • Partner with Engineering leaders and peers in delivering technology solutions that meet the business requirements 
  • Work with the cloud team in agile approach and develop cost optimized solutions

 

Primary Responsibilities:

  • Develop solutions using AWS services includiing Fargate, Lambda, ECS, ALB, NLB, S3 etc.

 

Reporting Team

  • Reporting Designation: Head - Big Data Engineering and Cloud Development (ABL_SS_414)
  • Reporting Department: Application Development (2487)

Required Skills:

  • AWS certification would be preferred
  • Good understanding in Monitoring (Cloudwatch, alarms, logs, custom metrics, Trust SNS configuration)
  • Good experience with Fargate, Lambda, ECS, ALB, NLB, S3, Glue, Aurora and other AWS services. 
  • Preferred to have Knowledge on Storage (S3, Life cycle management, Event configuration)
  • Good in data structure, programming in (pyspark / python / golang / Scala)
Read more
Job posted by
i
Naim Punasiya
Apply for job
i
Founded 2016  •  Services  •  20-100 employees  •  Bootstrapped
ETL
Python
Amazon Web Services (AWS)
SQL
PostgreSQL
i
Remote, Bengaluru (Bangalore)
i
2 - 5 yrs
i
₹6L - ₹12L / yr

We are actively seeking a Senior Data Engineer experienced in building data pipelines and integrations from 3rd party data sources by writing custom automated ETL jobs using Python. The role will work in partnership with other members of the Business Analytics team to support the development and implementation of new and existing data warehouse solutions for our clients. This includes designing database import/export processes used to generate client data warehouse deliverables.

 

Requirements
  • 2+ Years experience as an ETL developer with strong data architecture knowledge around data warehousing concepts, SQL development and optimization, and operational support models.
  • Experience using Python to automate ETL/Data Processes jobs.
  • Design and develop ETL and data processing solutions using data integration tools, python scripts, and AWS / Azure / On-Premise Environment.
  • Experience / Willingness to learn AWS Glue / AWS Data Pipeline / Azure Data Factory for Data Integration.
  • Develop and create transformation queries, views, and stored procedures for ETL processes, and process automation.
  • Document data mappings, data dictionaries, processes, programs, and solutions as per established standards for data governance.
  • Work with the data analytics team to assess and troubleshoot potential data quality issues at key intake points such as validating control totals at intake and then upon transformation, and transparently build lessons learned into future data quality assessments
  • Solid experience with data modeling, business logic, and RESTful APIs.
  • Solid experience in the Linux environment.
  • Experience with NoSQL / PostgreSQL preferred
  • Experience working with databases such as MySQL, NoSQL, and Postgres, and enterprise-level connectivity experience (such as connecting over TLS and through proxies).
  • Experience with NGINX and SSL.
  • Performance tune data processes and SQL queries, and recommend and implement data process optimization and query tuning techniques.
Read more
Job posted by
i
Pavel Gupta
Apply for job
i
Founded 2019  •  Products & Services  •  20-100 employees  •  Bootstrapped
Data Analytics
Business Intelligence (BI)
Microsoft Business Intelligence (MSBI)
Google Analytics
MongoDB
Amazon Web Services (AWS)
i
Pune
i
- yrs
i
₹3L - ₹18L / yr
Job Description :- 5 to 11 years experience- Experienced any of the BI tools like PowerBI, QlikView.- Writing SQLqueries/functions/procedures on big data.- Apt in databases NO SQL : mongoDB/ Cassandra OR MySQL, PostgresSQL, SQLServer- Able to Analyze clean,massage, cleanse and organize raw (big) data. - Manage Security for data on AWS or Azure cloud platform - Create, validate and maintain optimal data pipelines, assemble large, complex data sets - Helps in Structuring for upstream / downstream processing
Read more
Job posted by
i
Pankaj G
Apply for job
i
Founded 2016  •  Product  •  20-100 employees  •  Bootstrapped
Big Data
Python
Elastic Search
Hadoop
Spark
Apache Kafka
i
Bengaluru (Bangalore)
i
2 - 4 yrs
i
₹3L - ₹4L / yr

You will work on: 

We help many of our clients make sense of their large investments in data – be it building analytics solutions or machine learning applications. You will work on cutting edge cloud native technologies to crunch terabytes of data into meaningful insights. 

 

What you will do (Responsibilities):

  • Collaborate with Data Scientists, Engineers and Product Management to transform raw data to often actionable and meaningful insights for the enterprise
  • Work in small dynamic, product-oriented environment to deliver enterprise class products.
  • Continuously improve software development practices work across the full stack.

 

What you bring (Skills):

 

  • Experience in building modern cloud-native microservices based applications using state of the art frameworks in Java/Kotlin, Spring Boot, Hibernate and other JVM based frameworks.
  • Experience developing elastic, cloud based, BigData applications with modern technologies like Apache Spark, BigQuery, Airflow, Beam, Kafka and ElasticSearch.               
  • Ability to produce easily consumable RESTful APIs with strong living documentation and specification-by-example tests.

 

Great if you know (Skills):

  • T-shaped skills are always preferred – so if you have the passion to work across the full stack spectrum – it is more than welcome.
  • Exposure to infrastructure-based skills like Docker, Istio, Kubernetes is a plus
  • Ability to work out right strategies of deployment for BigData systems.
  • Collaborate with DevOps and Test Automation teams to build favorable developer experience in both build and CI/CD.
  • Attitude to look at an application holistically and debug any component if required whether it be based on UI technology like JavaScript, HTML or an intricate Linux System based issue.

 

 
Advantage Cognologix:

  •  Higher degree of autonomy, startup culture & small teams
  •  Opportunities to become expert in emerging technologies
  •  Remote working options for the right maturity level
  •  Competitive salary & family benefits
  •  Performance based career advancement

 


About Cognologix:

 

Cognologix helps companies disrupt by reimagining their business models and innovate like a Startup. We are at the forefront of digital disruption and take a business first approach to help meet our client’s strategic goals.

We are an Data focused organization helping our clients to deliver their next generation of products in the most efficient, modern and cloud-native way.

 

Read more
Job posted by
i
Payal Jain
Apply for job
i
Founded 2018  •  Services  •  20-100 employees  •  Bootstrapped
Analytics
Datawarehousing
Python
Amazon Redshift
Amazon Web Services (AWS)
SQL server
Data engineering
i
Chandigarh, NCR (Delhi | Gurgaon | Noida)
i
- yrs
i
₹7L - ₹15L / yr
Job Responsibilities :  
As a Data Warehouse Engineer in our team, you should have a proven ability to deliver high-quality work on time and with minimal supervision.
Develops or modifies procedures to solve complex database design problems, including performance, scalability, security and integration issues for various clients (on-site and off-site).
Design, develop, test, and support the data warehouse solution.
Adapt best practices and industry standards, ensuring top quality deliverable''s and playing an integral role in cross-functional system integration.
Design and implement formal data warehouse testing strategies and plans including unit testing, functional testing, integration testing, performance testing, and validation testing.
Evaluate all existing hardware's and software's according to required standards and ability to configure the hardware clusters as per the scale of data.
Data integration using enterprise development tool-sets (e.g. ETL, MDM, Quality, CDC, Data Masking, Quality).
Maintain and develop all logical and physical data models for enterprise data warehouse (EDW).
Contributes to the long-term vision of the enterprise data warehouse (EDW) by delivering Agile solutions.
Interact with end users/clients and translate business language into technical requirements.
Acts independently to expose and resolve problems.  
Participate in data warehouse health monitoring and performance optimizations as well as quality documentation.

Job Requirements :  
2+ years experience working in software development & data warehouse development for enterprise analytics.
2+ years of working with Python with major experience in Red-shift as a must and exposure to other warehousing tools.
Deep expertise in data warehousing, dimensional modeling and the ability to bring best practices with regard to data management, ETL, API integrations, and data governance.
Experience working with data retrieval and manipulation tools for various data sources like Relational (MySQL, PostgreSQL, Oracle), Cloud-based storage.
Experience with analytic and reporting tools (Tableau, Power BI, SSRS, SSAS). Experience in AWS cloud stack (S3, Glue, Red-shift, Lake Formation).
Experience in various DevOps practices helping the client to deploy and scale the systems as per requirement.
Strong verbal and written communication skills with other developers and business clients.
Knowledge of Logistics and/or Transportation Domain is a plus.
Ability to handle/ingest very huge data sets (both real-time data and batched data) in an efficient manner.
Read more
Job posted by
i
PS Dhillon
Apply for job
i
Founded 2012  •  Products & Services  •  20-100 employees  •  Profitable
TAC
PL/SQL
ETL
Relational Database (RDBMS)
MySQL
Big Data
i
Hyderabad
i
- yrs
i
₹3L - ₹8L / yr

ETL Developer – Talend

Job Duties:

  • ETL Developer is responsible for Design and Development of ETL Jobs which follow standards,

best practices and are maintainable, modular and reusable.

  • Proficiency with Talend or Pentaho Data Integration / Kettle.
  • ETL Developer will analyze and review complex object and data models and the metadata

repository in order to structure the processes and data for better management and efficient

access.

  • Working on multiple projects, and delegating work to Junior Analysts to deliver projects on time.
  • Training and mentoring Junior Analysts and building their proficiency in the ETL process.
  • Preparing mapping document to extract, transform, and load data ensuring compatibility with

all tables and requirement specifications.

  • Experience in ETL system design and development with Talend / Pentaho PDI is essential.
  • Create quality rules in Talend.
  • Tune Talend / Pentaho jobs for performance optimization.
  • Write relational(sql) and multidimensional(mdx) database queries.
  • Functional Knowledge of Talend Administration Center/ Pentaho data integrator, Job Servers &

Load balancing setup, and all its administrative functions.

  • Develop, maintain, and enhance unit test suites to verify the accuracy of ETL processes,

dimensional data, OLAP cubes and various forms of BI content including reports, dashboards,

and analytical models.

  • Exposure in Map Reduce components of Talend / Pentaho PDI.
  • Comprehensive understanding and working knowledge in Data Warehouse loading, tuning, and

maintenance.

  • Working knowledge of relational database theory and dimensional database models.
  • Creating and deploying Talend / Pentaho custom components is an add-on advantage.
  • Nice to have java knowledge.

Skills and Qualification:

  • BE, B.Tech / MS Degree in Computer Science, Engineering or a related subject.
  • Having an experience of 3+ years.
  • Proficiency with Talend or Pentaho Data Integration / Kettle.
  • Ability to work independently.
  • Ability to handle a team.
  • Good written and oral communication skills.
Read more
Job posted by
i
Niyotee Gupta
Apply for job
i
Founded 2011  •  Services  •  100-1000 employees  •  Profitable
HDFS
Apache Sqoop
Apache Flume
Apache HBase
Hadoop
Apache Kafka
Big Data
i
Chennai
i
- yrs
i
₹1L - ₹6L / yr
• Looking for Big Data Engineer with 3+ years of experience. • Hands-on experience with MapReduce-based platforms, like Pig, Spark, Shark. • Hands-on experience with data pipeline tools like Kafka, Storm, Spark Streaming. • Store and query data with Sqoop, Hive, MySQL, HBase, Cassandra, MongoDB, Drill, Phoenix, and Presto. • Hands-on experience in managing Big Data on a cluster with HDFS and MapReduce. • Handle streaming data in real time with Kafka, Flume, Spark Streaming, Flink, and Storm. • Experience with Azure cloud, Cognitive Services, Databricks is preferred.
Read more
Job posted by
i
John Richardson
Apply for job
Did not find a job you were looking for?
i
Search
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on CutShort.
iiiii
Want to apply for this role at Product Company Chennai based?
i
Apply for this job
Why apply via CutShort?
Connect with actual hiring teams and get their fast response. No spam.