Data Engineer

Sigmoid works with a variety of clients from start-ups to fortune 500 companies. We are looking for a detailed oriented self-starter to assist our engineering and analytics teams in various roles as a Software Development Engineer.

This position will be a part of a growing team working towards building world class large scale Big Data architectures. This individual should have a sound understanding of programming principles, experience in programming in Java, Python or similar languages and can expect to

spend a majority of their time coding.

Location - Bengaluru and Hyderabad

Responsibilities:

● Good development practices

○ Hands on coder with good experience in programming languages like Java or

Python.

○ Hands-on experience on the Big Data stack like PySpark, Hbase, Hadoop, Mapreduce and ElasticSearch.

○ Good understanding of programming principles and development practices like checkin policy, unit testing, code deployment

○ Self starter to be able to grasp new concepts and technology and translate them into large scale engineering developments

○ Excellent experience in Application development and support, integration development and data management.

● Align Sigmoid with key Client initiatives

○ Interface daily with customers across leading Fortune 500 companies to understand strategic requirements

● Stay up-to-date on the latest technology to ensure the greatest ROI for customer &Sigmoid

○ Hands on coder with good understanding on enterprise level code

○ Design and implement APIs, abstractions and integration patterns to solve challenging distributed computing problems

○ Experience in defining technical requirements, data extraction, data

transformation, automating jobs, productionizing jobs, and exploring new big data technologies within a Parallel Processing environment

● Culture

○ Must be a strategic thinker with the ability to think unconventional /

out:of:box.

○ Analytical and data driven orientation.

○ Raw intellect, talent and energy are critical.

○ Entrepreneurial and Agile : understands the demands of a private, high growth company.

○ Ability to be both a leader and hands on "doer".

Qualifications: -

- Years of track record of relevant work experience and a computer Science or related technical discipline is required

- Experience with functional and object-oriented programming, Java must.

- hand-On knowledge in Map Reduce, Hadoop, PySpark, Hbase and ElasticSearch.

- Effective communication skills (both written and verbal)

- Ability to collaborate with a diverse set of engineers, data scientists and product managers

- Comfort in a fast-paced start-up environment

Preferred Qualification:

- Technical knowledge in Map Reduce, Hadoop & GCS Stack a plus.

- Experience in agile methodology

- Experience with database modeling and development, data mining and warehousing.

- Experience in architecture and delivery of Enterprise scale applications and capable in developing framework, design patterns etc. Should be able to understand and tackle technical challenges, propose comprehensive solutions and guide junior staff

- Experience working with large, complex data sets from a variety of sources

Job Description

Mandatory Requirements 

• Experience in AWS Glue

• Experience in Apache Parquet 

• Proficient in AWS S3 and data lake 

• Knowledge of Snowflake

• Understanding of file-based ingestion best practices.

• Scripting language - Python & pyspark

CORE RESPONSIBILITIES

• Create and manage cloud resources in AWS 

• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 

• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 

• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 

• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.

• Define process improvement opportunities to optimize data collection, insights and displays.

• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 

• Identify and interpret trends and patterns from complex data sets 

• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 

• Key participant in regular Scrum ceremonies with the agile teams  

• Proficient at developing queries, writing reports and presenting findings 

• Mentor junior members and bring best industry practices.

QUALIFICATIONS

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 

• Strong background in math, statistics, computer science, data science or related discipline

• Advanced knowledge one of language: Java, Scala, Python, C# 

• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  

• Proficient with

• Data mining/programming tools (e.g. SAS, SQL, R, Python)

• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)

• Data visualization (e.g. Tableau, Looker, MicroStrategy)

• Comfortable learning about and deploying new technologies and tools. 

• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 

• Good written and oral communication skills and ability to present results to non-technical audiences 

• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus: 

• AWS certification

• Spark Streaming 

• Kafka Streaming / Kafka Connect 

• ELK Stack 

• Cassandra / MongoDB 

• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Job Description

1. Solid technical skills with a proven and successful history working with data at scale and empowering organizations through data
2. Big data processing frameworks: Spark, Scala, Hadoop, Hive, Kafka, EMR with Python
3. Advanced experience and hands-on architecture and administration experience on big data platforms

What we look for: 

We are looking for an associate who will be doing data crunching from various sources and finding the key points from the data. Also help us to improve/build new pipelines as per the requests. Also, this associate will be helping us to visualize the data if required and find flaws in our existing algorithms. 

Responsibilities: 

• Work with multiple stakeholders to gather the requirements of data or analysis and take action on them. 
• Write new data pipelines and maintain the existing pipelines. 
• Person will be gathering data from various DB’s and will be finding the required metrics out of it. 

Required Skills: 

• Experience with python and Libraries like Pandas,and Numpy. 
• Experience in SQL and understanding of NoSQL DB’s. 
• Hands-on experience in Data engineering. 
• Must have good analytical skills and knowledge of statistics. 
• Understanding of Data Science concepts. 
• Bachelor degree in Computer Science or related field. 
• Problem-solving skills and ability to work under pressure. 

Nice to have: 

• Experience in MongoDB or any NoSql DB. 
• Experience in ElasticSearch. 
• Knowledge of Tableau, Power BI or any other visualization tool.

• Hands-on programming expertise in Java OR Python
• Strong production experience with Spark (Minimum of 1-2 years)
• Experience in data pipelines using Big Data technologies (Hadoop, Spark, Kafka, etc.,) on large scale unstructured data sets
• Working experience and good understanding of public cloud environments (AWS OR Azure OR Google Cloud)
• Experience with IAM policy and role management is a plus

Responsibilities

• Installing and configuring Informatica components, including high availability; managing server activations and de-activations for all environments; ensuring that all systems and procedures adhere to organizational best practices
• Day to day administration of the Informatica Suite of services (PowerCenter, IDS, Metadata, Glossary and Analyst).
• Informatica capacity planning and on-going monitoring (e.g. CPU, Memory, etc.) to proactively increase capacity as needed.
• Manage backup and security of Data Integration Infrastructure.
• Design, develop, and maintain all data warehouse, data marts, and ETL functions for the organization as a part of an infrastructure team.
• Consult with users, management, vendors, and technicians to assess computing needs and system requirements.
• Develop and interpret organizational goals, policies, and procedures.
• Evaluate the organization's technology use and needs and recommend improvements, such as software upgrades.
• Prepare and review operational reports or project progress reports.
• Assist in the daily operations of the Architecture Team , analyzing workflow, establishing priorities, developing standards, and setting deadlines.
• Work with vendors to manage support SLA’s and influence vendor product roadmap
• Provide leadership and guidance in technical meetings, define standards and assist/provide status updates
• Work with cross functional operations teams such as systems, storage and network to design technology stacks.

Preferred Qualifications

• Minimum 6+ years’ experience as Informatica Engineer and Developer role
• Minimum of 5+ years’ experience in an ETL environment as a developer.
• Minimum of 5+ years of experience in SQL coding and understanding of databases
• Proficiency in Python
• Proficiency in command line troubleshooting
• Proficiency in writing code in Perl/Shell scripting languages
• Understanding of Java and concepts of Object-oriented programming
• Good understanding of systems, networking, and storage
• Strong knowledge of scalability and high availability

Data Platform engineering at Uber is looking for a strong Technical Lead (Level 5a Engineer) who has built high quality platforms and services that can operate at scale. 5a Engineer at Uber exhibits following qualities: 

• Demonstrate tech expertise › Demonstrate technical skills to go very deep or broad in solving classes of problems or creating broadly leverageable solutions. 
• Execute large scale projects › Define, plan and execute complex and impactful projects. You communicate the vision to peers and stakeholders.
• Collaborate across teams › Domain resource to engineers outside your team and help them leverage the right solutions. Facilitate technical discussions and drive to a consensus.
• Coach engineers › Coach and mentor less experienced engineers and deeply invest in their learning and success. You give and solicit feedback, both positive and negative, to others you work with to help improve the entire team.
• Tech leadership › Lead the effort to define the best practices in your immediate team, and help the broader organization establish better technical or business processes.

What You’ll Do

• Build a scalable, reliable, operable and performant data analytics platform for Uber’s engineers, data scientists, products and operations teams.
• Work alongside the pioneers of big data systems such as Hive, Yarn, Spark, Presto, Kafka, Flink to build out a highly reliable, performant, easy to use software system for Uber’s planet scale of data. 
• Become proficient of multi-tenancy, resource isolation, abuse prevention, self-serve debuggability aspects of a high performant, large scale, service while building these capabilities for Uber's engineers and operation folks.

What You’ll Need

• 7+ years experience in building large scale products, data platforms, distributed systems in a high caliber environment.
• Architecture: Identify and solve major architectural problems by going deep in your field or broad across different teams. Extend, improve, or, when needed, build solutions to address architectural gaps or technical debt.
• Software Engineering/Programming: Create frameworks and abstractions that are reliable and reusable. advanced knowledge of at least one programming language, and are happy to learn more. Our core languages are Java, Python, Go, and Scala.
• Data Engineering: Expertise in one of the big data analytics technologies we currently use such as Apache Hadoop (HDFS and YARN), Apache Hive, Impala, Drill, Spark, Tez, Presto, Calcite, Parquet, Arrow etc. Under the hood experience with similar systems such as Vertica, Apache Impala, Drill, Google Borg, Google BigQuery, Amazon EMR, Amazon RedShift, Docker, Kubernetes, Mesos etc.
• Execution & Results: You tackle large technical projects/problems that are not clearly defined. You anticipate roadblocks and have strategies to de-risk timelines. You orchestrate work that spans multiple teams and keep your stakeholders informed.
• A team player: You believe that you can achieve more on a team that the whole is greater than the sum of its parts. You rely on others’ candid feedback for continuous improvement.
• Business acumen: You understand requirements beyond the written word. Whether you’re working on an API used by other developers, an internal tool consumed by our operation teams, or a feature used by millions of customers, your attention to details leads to a delightful user experience.

Indium Software is a niche technology solutions company with deep expertise in Digital , QA and Gaming. Indium helps customers in their Digital Transformation journey through a gamut of solutions that enhance business value.

With over 1000+ associates globally, Indium operates through offices in the US, UK and India

Visit http://www.indiumsoftware.com">www.indiumsoftware.com to know more.

Job Title: Analytics Data Engineer

What will you do:
The Data Engineer must be an expert in SQL development further providing support to the Data and Analytics in database design, data flow and analysis activities. The position of the Data Engineer also plays a key role in the development and deployment of innovative big data platforms for advanced analytics and data processing. The Data Engineer defines and builds the data pipelines that will enable faster, better, data-informed decision-making within the business.

We ask:

Extensive Experience with SQL and strong ability to process and analyse complex data

The candidate should also have an ability to design, build, and maintain the business’s ETL pipeline and data warehouse The candidate will also demonstrate expertise in data modelling and query performance tuning on SQL Server
Proficiency with analytics experience, especially funnel analysis, and have worked on analytical tools like Mixpanel, Amplitude, Thoughtspot, Google Analytics, and similar tools.

Should work on tools and frameworks required for building efficient and scalable data pipelines
Excellent at communicating and articulating ideas and an ability to influence others as well as drive towards a better solution continuously.
Experience working in python, Hive queries, spark, pysaprk, sparkSQL, presto

• Relate Metrics to product
• Programmatic Thinking
• Edge cases
• Good Communication
• Product functionality understanding

Perks & Benefits:
A dynamic, creative & intelligent team they will make you love being at work.
Autonomous and hands-on role to make an impact you will be joining at an exciting time of growth!

Flexible work hours and Attractive pay package and perks
An inclusive work environment that lets you work in the way that works best for you!