
Data Engineer
at consulting & implementation services in the area of Oil & Gas, Mining and Manufacturing Industry


- Data Engineer
Required skill set: AWS GLUE, AWS LAMBDA, AWS SNS/SQS, AWS ATHENA, SPARK, SNOWFLAKE, PYTHON
Mandatory Requirements
- Experience in AWS Glue
- Experience in Apache Parquet
- Proficient in AWS S3 and data lake
- Knowledge of Snowflake
- Understanding of file-based ingestion best practices.
- Scripting language - Python & pyspark
CORE RESPONSIBILITIES
- Create and manage cloud resources in AWS
- Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies
- Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform
- Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations
- Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
- Define process improvement opportunities to optimize data collection, insights and displays.
- Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible
- Identify and interpret trends and patterns from complex data sets
- Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders.
- Key participant in regular Scrum ceremonies with the agile teams
- Proficient at developing queries, writing reports and presenting findings
- Mentor junior members and bring best industry practices
QUALIFICATIONS
- 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales)
- Strong background in math, statistics, computer science, data science or related discipline
- Advanced knowledge one of language: Java, Scala, Python, C#
- Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake
- Proficient with
- Data mining/programming tools (e.g. SAS, SQL, R, Python)
- Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
- Data visualization (e.g. Tableau, Looker, MicroStrategy)
- Comfortable learning about and deploying new technologies and tools.
- Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.
- Good written and oral communication skills and ability to present results to non-technical audiences
- Knowledge of business intelligence and analytical tools, technologies and techniques.
Familiarity and experience in the following is a plus:
- AWS certification
- Spark Streaming
- Kafka Streaming / Kafka Connect
- ELK Stack
- Cassandra / MongoDB
- CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Similar jobs

PHP Developer Responsibilities:
- Conducting analysis of website and application requirements.
- Writing back-end code and building efficient PHP modules.
- Developing back-end portals with an optimized database.
- Troubleshooting application and code issues.
- Integrating data storage solutions.
- Responding to integration requests from front-end developers.
- Finalizing back-end features and testing web applications.
- Updating and altering application features to enhance performance.
PHP Developer Requirements:
- Bachelor’s degree in computer science or a similar field.
- Knowledge of PHP web frameworks including Laravel, MySQL, API Development, OAuth, JWT, Git.
- Understanding of object-oriented PHP programming.
- Previous experience creating scalable applications.
- Proficient with code versioning tools including Git.
- Familiarity with MySQL databases.
- Ability to project manage.
- Good problem-solving skills.
Company link: https://www.catalystmi.com/" target="_blank">https://www.catalystmi.com/
Job Location: Mumbai (Hybrid)
Job Description
About the Role:
It is the product that sells, not just the idea. We are looking for a top-notch Sr. Product Designer with deep understanding of design principles and experience to back it up. Help build a brand and product that is loved by its users. Mainly focusing building the design system , of which components will also be exposed to clients as UI Kit. Experience with design systems and leading a project is highly appreciated. You will wear multiple hats and also help across branding, landing page, product discussion and much more. You have an active say in the direction of our product. We want someone who is passionate about working in this type of environment! We are looking for designers with a keen eye to details, user focused approach, familiarity with latest design processes and ability to bring concepts to life. You will be responsible for ensuring the alignment of web design and user experience requirements, optimizing web pages for maximum efficiency and maintaining brand consistency across all web pages, among other duties. Additionally, you will be required to work in teams alongside Frontend Developers to ensure all elements of web creation are consistent.
Who you are:
Experience with building a Design System and Component based designs.
3-5+ years of professional product design experience, bonus points if it was for SaaS platforms.
Expertise in product design (UI, UX, and Research) and modern tooling e.g. Figma, Miro, etc.
Ability to work well in fast-paced teams that ship often.
Experience in high growth startups and building minimum viable products.
A solid design foundation and aesthetic delivering designs/documents that are simple, informative, and filled with craftsmanship and strong attention of detail.
Basic technical expertise to communicate with developers.
Business know-how: Understanding of business metrics and the ability to translate company goals and objectives into digital experiences.
Good aesthetic sense, creative thinking with attention to detail.
Hands-on experience with conceptualizing design projects, coordinating with creative teams and working on its execution.
You're able to demonstrate an history of shipping simple, beautiful software experiences. Ability to juggle multiple projects and priorities and effectively deliver in a fast-paced, dynamic agile environment.
Team player that enjoys collaborating cross functionally and working with a great team of designers and engineers, who genuinely care about design and build with each other.
Familiarity with design tokens is a bonus
Ability to document design work. What you'll do:
Build the design language and system at dyte along with other designers that can be exposed to clients too.
Work on pages using the design system & create UX experiences that seem flawless.
Help in setting up design processes
Customer driven product shaping
Work closely with product managers and engineers to define product specs and iterate on experiences as they are delivered.
Help maintain, improve, and scale our current product experience.
Collaborate with other designers and product managers to provide input on the long term vision of the product experience.
Create user flows to effectively communicate interaction and design ideas
Coordinate between marketing and design teams to ensure smooth flow of work
Ensure the products created are user-friendly and convey brand consistency with all projects.
Our Perks:
1. Competitive Salary 2. ESOPs 3. Medical Insurances 4. Friendly leaves policy 5. 100% Remote Option
JOB DESCRIPTION
Role: Backend Architect
Amagi Background
Amagi is a SaaS platform that lets every content owner deliver their content to consumers anytime, anywhere, to any device. Amagi helps bring entertainment to hundreds of millions of consumers leading the transformation in media consumption. We believe in a connected ecosystem bringing content owners, distribution platforms, consumers and advertisers together to create great experiences.
Amagi grew by 136% last year and is on its way to double itself again this year. The market leader in FAST (Free Ad-supported Streaming TV), it delivers more than 500 media brands to 1500+ end points and is growing exponentially.
Amagi is looking for Software Architects with 8-12 yrs experience in high level & low level design and development of highly scalable software products.
Role reporting into: Director, Streaming TV - Distribution, Analytics & Monetization
Location: Bangalore, India
Key responsibilities include (but are not limited to):
- You will lead a team of talented engineers to build the B2B products and services, as well as necessary frameworks, libraries and primitives to scale & connect our offerings together.
- Architect, design, implement and maintain highly scalable backend systems that can scale to millions of concurrent users in production.
- Derive next generation tools and technologies to support hundreds of high impactful customers on monetization products in production
- Collaborate with the product, business, design and engineering functions to be on top of your team’s deliverables & milestones.
- Timely delivery of highly reliable and scalable engineering architecture, and high quality, maintainable and operationally excellent code for your team.
- Track & monitor team & individual performance, building continuous improvement into the process
- Drive design discussions and code reviews
- Identify and resolve performance and scalability issues
- Hire the top talent for your team and mentor the team members.
Person Specifications
- 8-12 years of software development experience with demonstrated ability to design, build, and deliver complex, successful engineering systems or products efficiently and with high quality.
- Good understanding of technologies and architecture in a highly scalable and available set up.
- Good understanding & expertise with highly transactional, large relational and complex systems
- Strong object-oriented design skills, and an uncanny ability to design intuitive module and class-level interfaces.
- Working knowledge of one or more backend technologies from among Java, Ruby-on-Rails, Python, Golang.
- Well versed with development workflows for distributed systems deployed on the cloud (preferably AWS)
- Establishing and maintaining unit, integration and system test mechanisms in staging, pre-production and production environments
- Generate qualified opportunities for the company by rigorously prospecting and researching in the given region/ market/ industry. Information about MEDDIC is a plus
- Being the face of CM.com’s and introducing the company and the product to the prospects so as to generate funnel.
- Write creative communication (emails/ Inmails) to grab the attention of prospects.
- Ideate and innovate methods to improve SDR metrics

- Do you like the idea of playing Master Shifu for product development teams? Or, what organizations think of it as playing an Engineering Manager.
- Do you live by the mantra of “buck stops with me”. Seriously, we also don’t know who Buck is and we are also looking for him.
- Have you led a product team of 15+ members in the past to successful outcomes? Okay, we would love to talk even if you unfortunately ran into a failure.
- Do you speak both binary and Humanary? Python , dialect, to talk to geeks and the Humanary, English or Hindi dialect, to talk to Clients.
- Do you practice Agility? No, we don’t mean asanas from Baba Ramdev but things like Scrum or Kanban
- Do you practice Curiosity over Complacency? I..e are you a lean mean learning machine?
- Can you envision a solution architecture after hearing the product's vision and user stories? Do things like scalability, performability, flexibility, maintainability give you goosebumps of joy?
- Do you have an eye for good user experience?
Did you say Yes to all of them? My My! Love you! I’m sure your current team loves you too. But, if for whatever reason, you are looking for a new home then we would love to talk to you. We are a certified Great Place to Work, which means it’s a place where people love working with each other. And, you would too.
Send me an email, will ya?

- Power BI Report and Dashboard development.
- Building Analysis Services reporting models.
- Developing visual reports, KPI scorecards, and dashboards using Power BI desktop.
- Connecting data sources, importing data, and transforming data for Business intelligence.
- Analytical thinking for translating data into informative reports and visuals.
- Should have an edge over making DAX queries in Power BI desktop.
- Expert in using advanced-level calculations on the data set.
- SQL Server with SSAS is must.
- Very good communication skills must be able to discuss the requirements effectively with the client teams, and with internal teams.

Hiring Full Stack Developers at Krishivan Technologies Pvt Ltd
Job Overview:
We are urgently looking for a Full Stack Developer with 1-3 years of experience to produce scalable software solutions. You'll be part of a cross-functional team that's responsible for the full software development life cycle, from conception to deployment.
As a Full Stack Developer, you should be comfortable around both front-end and back-end coding languages, development frameworks and third-party libraries. You should also be a team player with a knack for visual design and utility.
About Krishivan:
Krishivan Technologies is an AgroTech start-up that aims to partner with grassroots farmers to improve their livelihoods through smart collaboration between ‘Agriculture and Technology’. With us, farmers have exclusive access to Agri-solutions and Agri-advice at their fingertips.
We bridge the gap between farming challenges, expert knowledge, shortage of high-quality agri-in puts, sustainable financial aid, equipped machinery and infrastructure to enhance the production a nd income ability of farmers with the integration of technology from pre-harvest to post-harvest.
Roles and responsibilities:
● Participate in requirements analysis
● Building interactive consumer data from multiple systems and REST entirely abstract to the UI through a Node.js backend
● Ensure the best possible performance, quality, and responsiveness of applications
● Test-Driven Development
● Address and improve any technical issue.
● Support the implementation of developed software solutions
Requirements and skills:
● Strong understanding of object-oriented programming
● Knowledge of NodeJS, ExpressJS
● React.js/ Next.js / Node.js / Mongo DB
● Javascript, Jquery, HTML, CSS, Bootstrap
● Understanding of Agile methodologies
● Hands-on experience with JavaScript Development on both client and server-side
● Proficient understanding of code versioning tools.
Good to have:
● AWS and/or Azure Services
● Git Setup
● Experience with cloud technologies
● Familiar with various design and architectural patterns
VOIP Developer
- Knowledge on PSTN lines
- Good understanding of TCP/IP protocol stack and practical knowledge of LAN / WAN
- Good knowledge of value-added features such as Voicemail, ACD, IVRS, CTI, etc
- Familiarity with SIP / SDP protocol / Softphones
- Familiarity with OpenSIPS, Kamalio
- Familiarity with Freeswitch
- Familiarity with MySQL
- Knowledge of Linux / server administration
- Familiarity with C language
- Firewall, Security
- Knowledge of WebRTC
- Standard features of Enterprise Communication Servers (PBXs)
- Problem Troubleshooting and debugging techniques (both Hardware and Software)
Must Have Skills:
- Good experience in Pyspark - Including Dataframe core functions and Spark SQL
- Good experience in SQL DBs - Be able to write queries including fair complexity.
- Should have excellent experience in Big Data programming for data transformation and aggregations
- Good at ELT architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.
- Good customer communication.
- Good Analytical skills
Technology Skills (Good to Have):
- Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub.
- Experience in migrating on-premise data warehouses to data platforms on AZURE cloud.
- Designing and implementing data engineering, ingestion, and transformation functions
- Azure Synapse or Azure SQL data warehouse
- Spark on Azure is available in HD insights and data bricks
- Proven hands-on Software Development experience
- Proven working experience in Java development
- Hands on experience in designing and developing applications using Java EE platforms
- Object Oriented analysis and design using common design patterns.

