Principal Data Engineer (Real World Data)

Lightning Job By Cutshort⚡

As part of this feature, you can expect status updates about your application and replies within 72 hours (once the screening questions are answered)

Roles & Responsibilities

Basic Qualifications:

● The position requires a four-year degree from an accredited college or university.

● Three years of data engineering / AWS Architecture and security experience.

Top candidates will also have:

Proven/Strong understanding and/or experience in many of the following:-

● Experience designing Scalable AWS architecture.

● Ability to create modern data pipelines and data processing using AWS PAAS components (Glue, etc.) or open source tools (Spark, Hbase, Hive, etc.).

● Ability to develop SQL structures that support high volumes and scalability using

RDBMS such as SQL Server, MySQL, Aurora, etc.

● Ability to model and design modern data structures, SQL/NoSQL databases, Data Lakes, Cloud Data Warehouse

● Experience in creating Network Architecture for secured scalable solution.

● Experience with Message brokers such as Kinesis, Kafka, Rabbitmq, AWS SQS, AWS SNS, and Apache ActiveMQ. Hands-on experience on AWS serverless architectures such as Glue,Lamda, Redshift etc.

● Working knowledge of Load balancers, AWS shield, AWS guard, VPC, Subnets, Network gateway Route53 etc.

● Knowledge of building Disaster management systems and security logs notification system

● Knowledge of building scalable microservice architectures with AWS.

● To create a framework for monthly security checks and wide knowledge on AWS services

● Deploying software using CI/CD tools such CircleCI, Jenkins, etc.

● ML/ AI model deployment and production maintainanace experience is mandatory.

● Experience with API tools such as REST, Swagger, Postman and Assertible.

● Versioning management tools such as github, bitbucket, GitLab.

● Debugging and maintaining software in Linux or Unix platforms.

● Test driven development

● Experience building transactional databases.

● Python, PySpark programming experience .

● Must experience engineering solutions in AWS.

● Working AWS experience, AWS certification is required prior to hiring

● Working in Agile Framework/Kanban Framework

● Must demonstrate solid knowledge of computer science fundamentals like data structures & algorithms.

● Passion for technology and an eagerness to contribute to a team-oriented environment.

● Demonstrated leadership on medium to large-scale projects impacting strategic priorities.

● Bachelor’s degree in Computer science or Electrical engineering or related field is required

The Data Engineering team is one of the core technology teams of Lumiq.ai and is responsible for creating all the Data related products and platforms which scale for any amount of data, users, and processing. The team also interacts with our customers to work out solutions, create technical architectures and deliver the products and solutions.

If you are someone who is always pondering how to make things better, how technologies can interact, how various tools, technologies, and concepts can help a customer or how a customer can use our products, then Lumiq is the place of opportunities.

Who are you?

• Enthusiast is your middle name. You know what’s new in Big Data technologies and how things are moving
• Apache is your toolbox and you have been a contributor to open source projects or have discussed the problems with the community on several occasions
• You use cloud for more than just provisioning a Virtual Machine
• Vim is friendly to you and you know how to exit Nano
• You check logs before screaming about an error
• You are a solid engineer who writes modular code and commits in GIT
• You are a doer who doesn’t say “no” without first understanding
• You understand the value of documentation of your work
• You are familiar with Machine Learning Ecosystem and how you can help your fellow Data Scientists to explore data and create production-ready ML pipelines

Experience

• At least 2 years of Data Engineering Experience
• Have interacted with Customers

Must Have Skills

• Amazon Web Services (AWS) - EMR, Glue, S3, RDS, EC2, Lambda, SQS, SES
• Apache Spark
• Python
• Scala
• PostgreSQL
• Git
• Linux

Good to have Skills

• Apache NiFi
• Apache Kafka
• Apache Hive
• Docker
• Amazon Certification

Data Analyst

Job Description

Are you passionate about handling large & complex data problems, want to make an impact and have the desire to work on ground-breaking big data technologies? Then we are looking for you.

At Amagi, great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring passion and dedication to your job and there's no telling what you could accomplish. Would you like to work in a fast-paced environment where your technical abilities will be challenged on a day-to-day basis? If so, Amagi’s Data Engineering and Business Intelligence team is looking for passionate, detail-oriented, technical savvy, energetic team members who like to think outside the box.

Amagi’s Data warehouse team deals with petabytes of data catering to a wide variety of real-time, near real-time and batch analytical solutions. These solutions are an integral part of business functions such as Sales/Revenue, Operations, Finance, Marketing and Engineering, enabling critical business decisions. Designing, developing, scaling and running these big data technologies using native technologies of AWS and GCP are a core part of our daily job.

• Experience in building highly cost optimised data analytics solutions
• Experience in designing and building dimensional data models to improve accessibility, efficiency and quality of data
• Experience (hands on) in building high quality ETL applications, data pipelines and analytics solutions ensuring data privacy and regulatory compliance.
• Experience in working with AWS or GCP
• Experience with relational and NoSQL databases
• Experience to full stack web development (Preferably Python)
• Expertise with data visualisation systems such as Tableau and Quick Sight
• Proficiency in writing advanced SQL queries with expertise in performance tuning handling large data volumes
• Familiarity with ML/AÍ technologies is a plus
• Demonstrate strong understanding of development processes and agile methodologies
• Strong analytical and communication skills. Should be self-driven, highly motivated and ability to learn quickly

Data Analytics is at the core of our work, and you will have the opportunity to:

• Design Data-warehousing solutions on Amazon S3 with Athena, Redshift, GCP Bigtable etc
• Lead quick prototypes by integrating data from multiple sources
• Do advanced Business Analytics through ad-hoc SQL queries
• Work on Sales Finance reporting solutions using tableau, HTML5, React applications

We build amazing experiences and create depth in knowledge for our internal teams and our leadership. Our team is a friendly bunch of people that help each other grow and have a passion for technology, R&D, modern tools and data science.

Our work relies on deep understanding of the company needs and an ability to go through vast amounts of internal data such as sales, KPIs, forecasts, Inventory etc. One of the key expectations of this role would be to do data analytics, building data lakes, end to end reporting solutions etc. If you have a passion for cost optimised analytics and data engineering and are eager to learn advanced data analytics at a large scale, this might just be the job for you..

A bachelor’s/master’s degree in Computer Science with 5 to 7 years of experience and previous experience in data engineering is a plus.

Mactores is a trusted leader among businesses in providing modern data platform solutions. Since 2008, Mactores have been enabling businesses to accelerate their value through automation by providing End-to-End Data Solutions that are automated, agile, and secure. We collaborate with customers to strategize, navigate, and accelerate an ideal path forward with a digital transformation via assessments, migration, or modernization.

We are looking for a DataOps Engineer with expertise while operating a data lake. Amazon S3, Amazon EMR, and Apache Airflow for workflow management are used to build the data lake.

You have experience of building and running data lake platforms on AWS. You have exposure to operating PySpark-based ETL Jobs in Apache Airflow and Amazon EMR. Expertise in monitoring services like Amazon CloudWatch.

If you love solving problems using yo, professional services background, usual and fun office environment that actively steers clear of rigid "corporate" culture, focuses on productivity and creativity, and allows you to be part of a world-class team while still being yourself.

What you will do?

• Operate the current data lake deployed on AWS with Amazon S3, Amazon EMR, and Apache Airflow
• Debug and fix production issues in PySpark.
• Determine the RCA (Root cause analysis) for production issues.
• Collaborate with product teams for L3/L4 production issues in PySpark.
• Contribute to enhancing the ETL efficiency
• Build CloudWatch dashboards for optimizing the operational efficiencies
• Handle escalation tickets from L1 Monitoring engineers
• Assign the tickets to L1 engineers based on their expertise

What are we looking for?

• AWS data Ops engineer.
• Overall 5+ years of exp in the software industry Exp in developing architecture data applications using python or scala, Airflow, and Kafka on AWS Data platform Experience and expertise.
• Must have set up or led the project to enable Data Ops on AWS or any other cloud data platform.
• Strong data engineering experience on Cloud platform, preferably AWS.
• Experience with data pipelines designed for reuse and use parameterization.
• Experience of pipelines was designed to solve common ETL problems.
• Understanding or experience on various AWS services can be codified for enabling DataOps like Amazon EMR, Apache Airflow.
• Experience in building data pipelines using CI/CD infrastructure.
• Understanding of Infrastructure as code for DataOps ennoblement.
• Ability to work with ambiguity and create quick PoCs.

You will be preferred if

• Expertise in Amazon EMR, Apache Airflow, Terraform, CloudWatch
• Exposure to MLOps using Amazon Sagemaker is a plus.
• AWS Solutions Architect Professional or Associate Level Certificate
• AWS DevOps Professional Certificate

Life at Mactores

We care about creating a culture that makes a real difference in the lives of every Mactorian. Our 10 Core Leadership Principles that honor Decision-making, Leadership, Collaboration, and Curiosity drive how we work.

1. Be one step ahead

2. Deliver the best

3. Be bold

4. Pay attention to the detail

5. Enjoy the challenge

6. Be curious and take action

7. Take leadership

8. Own it

9. Deliver value

10. Be collaborative

We would like you to read more details about the work culture on https://mactores.com/careers 

The Path to Joining the Mactores Team

At Mactores, our recruitment process is structured around three distinct stages:

Pre-Employment Assessment: 

You will be invited to participate in a series of pre-employment evaluations to assess your technical proficiency and suitability for the role.

Managerial Interview: The hiring manager will engage with you in multiple discussions, lasting anywhere from 30 minutes to an hour, to assess your technical skills, hands-on experience, leadership potential, and communication abilities.

HR Discussion: During this 30-minute session, you'll have the opportunity to discuss the offer and next steps with a member of the HR team.

At Mactores, we are committed to providing equal opportunities in all of our employment practices, and we do not discriminate based on race, religion, gender, national origin, age, disability, marital status, military status, genetic information, or any other category protected by federal, state, and local laws. This policy extends to all aspects of the employment relationship, including recruitment, compensation, promotions, transfers, disciplinary action, layoff, training, and social and recreational programs. All employment decisions will be made in compliance with these principles.

Big data Developer

Exp: 3yrs to 7 yrs.
Job Location: Hyderabad
Notice: Immediate / within 30 days

1. Expertise in building AWS Data Engineering pipelines with AWS Glue -> Athena -> Quick sight
2. Experience in developing lambda functions with AWS Lambda
3. Expertise with Spark/PySpark Candidate should be hands on with PySpark code and should be able to do transformations with Spark
4. Should be able to code in Python and Scala.
5. Snowflake experience will be a plus

We can start keeping Hadoop and Hive requirements as good to have or understanding of is enough rather than keeping it as a desirable requirement.

Role Description:

• You will be part of the data delivery team and will have the opportunity to develop a deep understanding of the domain/function.
• You will design and drive the work plan for the optimization/automation and standardization of the processes incorporating best practices to achieve efficiency gains.
• You will run data engineering pipelines, link raw client data with data model, conduct data assessment, perform data quality checks, and transform data using ETL tools.
• You will perform data transformations, modeling, and validation activities, as well as configure applications to the client context. You will also develop scripts to validate, transform, and load raw data using programming languages such as Python and / or PySpark.
• In this role, you will determine database structural requirements by analyzing client operations, applications, and programming.
• You will develop cross-site relationships to enhance idea generation, and manage stakeholders.
• Lastly, you will collaborate with the team to support ongoing business processes by delivering high-quality end products on-time and perform quality checks wherever required.

Responsibilities:

* 3+ years of Data Engineering Experience - Design, develop, deliver and maintain data infrastructures.

* SQL Specialist – Strong knowledge and Seasoned experience with SQL Queries

* Languages: Python

* Good communicator, shows initiative, works well with stakeholders.

* Experience working closely with Data Analysts and provide the data they need and guide them on the issues.

* Solid ETL experience and Hadoop/Hive/Pyspark/Presto/ SparkSQL

* Solid communication and articulation skills

* Able to handle stakeholders independently with less interventions of reporting manager.

* Develop strategies to solve problems in logical yet creative ways.

* Create custom reports and presentations accompanied by strong data visualization and storytelling

We would be excited if you have:

* Excellent communication and interpersonal skills

* Ability to meet deadlines and manage project delivery

* Excellent report-writing and presentation skills

* Critical thinking and problem-solving capabilities

Datametica is Hiring for Datastage Developer

• Must have 3 to 8 years of experience in ETL Design and Development using IBM Datastage Components.
• Should have extensive knowledge in Unix shell scripting.
• Understanding of DW principles (Fact, Dimension tables, Dimensional Modelling and Data warehousing concepts).
• Research, development, document and modification of ETL processes as per data architecture and modeling requirements.
• Ensure appropriate documentation for all new development and modifications of the ETL processes and jobs.
• Should be good in writing complex SQL queries.

About Us!

A global Leader in the Data Warehouse Migration and Modernization to the Cloud, we empower businesses by migrating their Data/Workload/ETL/Analytics to the Cloud by leveraging Automation.

We have expertise in transforming legacy Teradata, Oracle, Hadoop, Netezza, Vertica, Greenplum along with ETLs like Informatica, Datastage, AbInitio & others, to cloud-based data warehousing with other capabilities in data engineering, advanced analytics solutions, data management, data lake and cloud optimization.

Datametica is a key partner of the major cloud service providers - Google, Microsoft, Amazon, Snowflake.

We have our own products!

Eagle – Data warehouse Assessment & Migration Planning Product

Raven – Automated Workload Conversion Product

Pelican - Automated Data Validation Product, which helps automate and accelerate data migration to the cloud.

Why join us!

Datametica is a place to innovate, bring new ideas to live and learn new things. We believe in building a culture of innovation, growth and belonging. Our people and their dedication over these years are the key factors in achieving our success.

Benefits we Provide!

Working with Highly Technical and Passionate, mission-driven people

Subsidized Meals & Snacks

Flexible Schedule

Approachable leadership

Access to various learning tools and programs

Pet Friendly

Certification Reimbursement Policy

Check out more about us on our website below!

www.datametica.com