Data Engineer

Qualifications & Experience:

▪ 2 - 4 years overall experience in ETLs, data pipeline, Data Warehouse development and database design

▪ Software solution development using Hadoop Technologies such as MapReduce, Hive, Spark, Kafka, Yarn/Mesos etc.

▪ Expert in SQL, worked on advanced SQL for at least 2+ years

▪ Good development skills in Java, Python or other languages

▪ Experience with EMR, S3

▪ Knowledge and exposure to BI applications, e.g. Tableau, Qlikview

▪ Comfortable working in an agile environment

Publicis Sapient Overview:

The Senior Associate People Senior Associate L1 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution 

.

Job Summary:

As Senior Associate L2 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution

The role requires a hands-on technologist who has strong programming background like Java / Scala / Python, should have experience in Data Ingestion, Integration and data Wrangling, Computation, Analytics pipelines and exposure to Hadoop ecosystem components. You are also required to have hands-on knowledge on at least one of AWS, GCP, Azure cloud platforms.

Role & Responsibilities:

Your role is focused on Design, Development and delivery of solutions involving:

• Data Integration, Processing & Governance

• Data Storage and Computation Frameworks, Performance Optimizations

• Analytics & Visualizations

• Infrastructure & Cloud Computing

• Data Management Platforms

• Implement scalable architectural models for data processing and storage

• Build functionality for data ingestion from multiple heterogeneous sources in batch & real-time mode

• Build functionality for data analytics, search and aggregation

Experience Guidelines:

Mandatory Experience and Competencies:

# Competency

1.Overall 5+ years of IT experience with 3+ years in Data related technologies

2.Minimum 2.5 years of experience in Big Data technologies and working exposure in at least one cloud platform on related data services (AWS / Azure / GCP)

3.Hands-on experience with the Hadoop stack – HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data pipeline.

4.Strong experience in at least of the programming language Java, Scala, Python. Java preferable

5.Hands-on working knowledge of NoSQL and MPP data platforms like Hbase, MongoDb, Cassandra, AWS Redshift, Azure SQLDW, GCP BigQuery etc

6.Well-versed and working knowledge with data platform related services on at least 1 cloud platform, IAM and data security

Preferred Experience and Knowledge (Good to Have):

# Competency

1.Good knowledge of traditional ETL tools (Informatica, Talend, etc) and database technologies (Oracle, MySQL, SQL Server, Postgres) with hands on experience

2.Knowledge on data governance processes (security, lineage, catalog) and tools like Collibra, Alation etc

3.Knowledge on distributed messaging frameworks like ActiveMQ / RabbiMQ / Solace, search & indexing and Micro services architectures

4.Performance tuning and optimization of data pipelines

5.CI/CD – Infra provisioning on cloud, auto build & deployment pipelines, code quality

6.Cloud data specialty and other related Big data technology certifications

Personal Attributes:

• Strong written and verbal communication skills

• Articulation skills

• Good team player

• Self-starter who requires minimal oversight

• Ability to prioritize and manage multiple tasks

• Process orientation and the ability to define and set up processes

• Your responsibilities:
• Build, improve and extend NLP capabilities
• Research and evaluate different approaches to NLP problems
• Must be able to write code that is well designed, produce deliverable results
• Write code that scales and can be deployed to production

• Fundamentals of statistical methods is a must
• Experience in named entity recognition, POS Tagging, Lemmatization, vector representations of textual data and neural networks - RNN, LSTM
• A solid foundation in Python, data structures, algorithms, and general software development skills.
• Ability to apply machine learning to problems that deal with language
• Engineering ability to build robustly scalable pipelines
• Ability to work in a multi-disciplinary team with a strong product focus

We are a rapidly expanding global technology partner, that is looking for a highly skilled Senior (Python) Data Engineer to join their exceptional Technology and Development team. The role is in Kolkata. If you are passionate about demonstrating your expertise and thrive on collaborating with a group of talented engineers, then this role was made for you!

At the heart of technology innovation, our client specializes in delivering cutting-edge solutions to clients across a wide array of sectors. With a strategic focus on finance, banking, and corporate verticals, they have earned a stellar reputation for their commitment to excellence in every project they undertake.

We are searching for a senior engineer to strengthen their global projects team. They seek an experienced Senior Data Engineer with a strong background in building Extract, Transform, Load (ETL) processes and a deep understanding of AWS serverless cloud environments.

As a vital member of the data engineering team, you will play a critical role in designing, developing, and maintaining data pipelines that facilitate data ingestion, transformation, and storage for our organization.

Your expertise will contribute to the foundation of our data infrastructure, enabling data-driven decision-making and analytics.

Key Responsibilities:

• ETL Pipeline Development: Design, develop, and maintain ETL processes using Python, AWS Glue, or other serverless technologies to ingest data from various sources (databases, APIs, files), transform it into a usable format, and load it into data warehouses or data lakes.
• AWS Serverless Expertise: Leverage AWS services such as AWS Lambda, AWS Step Functions, AWS Glue, AWS S3, and AWS Redshift to build serverless data pipelines that are scalable, reliable, and cost-effective.
• Data Modeling: Collaborate with data scientists and analysts to understand data requirements and design appropriate data models, ensuring data is structured optimally for analytical purposes.
• Data Quality Assurance: Implement data validation and quality checks within ETL pipelines to ensure data accuracy, completeness, and consistency.
• Performance Optimization: Continuously optimize ETL processes for efficiency, performance, and scalability, monitoring and troubleshooting any bottlenecks or issues that may arise.
• Documentation: Maintain comprehensive documentation of ETL processes, data lineage, and system architecture to ensure knowledge sharing and compliance with best practices.
• Security and Compliance: Implement data security measures, encryption, and compliance standards (e.g., GDPR, HIPAA) as required for sensitive data handling.
• Monitoring and Logging: Set up monitoring, alerting, and logging systems to proactively identify and resolve data pipeline issues.
• Collaboration: Work closely with cross-functional teams, including data scientists, data analysts, software engineers, and business stakeholders, to understand data requirements and deliver solutions.
• Continuous Learning: Stay current with industry trends, emerging technologies, and best practices in data engineering and cloud computing and apply them to enhance existing processes.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Data Science, or a related field.
• Proven experience as a Data Engineer with a focus on ETL pipeline development.
• Strong proficiency in Python programming.
• In-depth knowledge of AWS serverless technologies and services.
• Familiarity with data warehousing concepts and tools (e.g., Redshift, Snowflake).
• Experience with version control systems (e.g., Git).
• Strong SQL skills for data extraction and transformation.
• Excellent problem-solving and troubleshooting abilities.
• Ability to work independently and collaboratively in a team environment.
• Effective communication skills for articulating technical concepts to non-technical stakeholders.
• Certifications such as AWS Certified Data Analytics - Specialty or AWS Certified DevOps Engineer are a plus.

Preferred Experience:

• Knowledge of data orchestration and workflow management tools
• Familiarity with data visualization tools (e.g., Tableau, Power BI).
• Previous experience in industries with strict data compliance requirements (e.g., insurance, finance) is beneficial.

What You Can Expect:

- Innovation Abounds: Join a company that constantly pushes the boundaries of technology and encourages creative thinking. Your ideas and expertise will be valued and put to work in pioneering solutions.

- Collaborative Excellence: Be part of a team of engineers who are as passionate and skilled as you are. Together, you'll tackle challenging projects, learn from each other, and achieve remarkable results.

- Global Impact: Contribute to projects with a global reach and make a tangible difference. Your work will shape the future of technology in finance, banking, and corporate sectors.

They offer an exciting and professional environment with great career and growth opportunities. Their office is located in the heart of Salt Lake Sector V, offering a terrific workspace that's both accessible and inspiring. Their team members enjoy a professional work environment with regular team outings. Joining the team means becoming part of a vibrant and dynamic team where your skills will be valued, your creativity will be nurtured, and your contributions will make a difference. In this role, you can work alongside some of the brightest minds in the industry.

If you're ready to take your career to the next level and be part of a dynamic team that's driving innovation on a global scale, we want to hear from you.

Apply today for more information about this exciting opportunity.

Onsite Location: Kolkata, India (Salt Lake Sector V)

Designation: business analyst

Company name - promilo.com (sawara solutions pvt ltd)

Experience – 2 - 8 yrs.

Location: Bangalore

Mode – full time / work from office

About us:

Promilo is India’s 1st innovative platform which “pay to browse”

It is a b2b SaaS start-up that enables to accelerate  the business appointment funnel of the

Companies. We’re an SaaS based advertising platform that connects both users & advertisers. Users will be able to book an online appointment based on their interests with the advertiser, without compromising their data privacy and get rewarded for sharing their data and time. We’re registered and recognized by start-up India, start-up Karnataka & MSME companies. Also, the top 100 Google AppScale academy start-up

Job description:

We are looking for an experienced business analyst to join our team. The ideal candidate will have 2-8 years of experience in web & mobile user & client data analyst for start-ups, with a strong passion to help start-ups and a proven track record to bring the strong business insight to improve the sales, marketing, user, client, ui, ux of the organisation.

Responsibilities

• Requirement gathering and analyzing
• Conduct gap analysis, assess scope & suggest solutions
• Responsible for technical proposal writing and time and cost analysis for web and mobile application development
• Preparing rfp/rfq
• Would be involved in presales activities
• Work as liaison between client and technical team
• Create wireframe | prototype | feature list | srs | brd & flow diagrams as per the client's requirement
• High it literacy proven use of web and associated technologies (excel, power point, google apps).
• Previous experience with data visualization tools (tableau, power bi, etc.), is strongly preferred.
• Cleanse and curate sourced data into standardized reporting templates
• Create, document, validate and ensure delivery of ad hoc, daily weekly, and monthly reports to internal stakeholders
• Create, validate and deliver tracking links for the marketing department
• Assist in the creation, qa, validation and reporting of a/b and multivariate tests
• Proactively monitor the marketing kpis and ua data on a daily basis
• Analyze marketing ua performance and conduct deep dive analysis to answer hypotheses and questions posed by the team
• Gather, transform, and analyze digital marketing data, including paid media, search, social, website, and conversion funnel analytics
• Analyze marketing data searching for top of funnel growth opportunities
• Analyze product data searching for insights to increase app engagement, conversion, and retention
• Analyze ltv/cac drivers to support overall business growth
• Partner with marketing, product, and growth teams
• Present findings to stakeholders and make recommendations for spend targets and campaign strategies
• Pov on ios 14 and upcoming android privacy changes and we can navigate tracking in light of these changes
• Pov on transition to skan 4.0
• Working knowledge of statistical techniques (regression, k-means clustering, pca)
• Experience with lift studies and marketing mix modeling working experience with python, r, & dbt
• Experience at a small company
• Experience with a subscription business
• Analyze website and mobile app data on traffic sources and patterns. Provide insight on data trends and anomalies, making recommendations where appropriate to improve business performance.

Qualification

• Master's or bachelor degree in computer science
• Well-versed with its technologies
• 2+ years of business analysis or project analysis experience
• Tech-savvy with proficiency in Microsoft office, google apps, and other web and mobile applications
• Excellent written and verbal communication skills
• Self-motivated, flexible, and comfortable with a fast-paced startup environment
• Advanced experience with Excel, google sheets including an understanding of visualizations, pivot tables, vlookup, and other key functions
• Experience with adobe analytics, google analytics, tableau, SQL, and data grid is a plus.
• Strong analytical and problem-solving skills, with clear attention to detail
• Ability to prioritize and work under tight deadlines
• Fast learner, able to master new concepts, theories, ideas, and processes with ease
• Experience creating user acquisition reports and dashboards
• Deep understanding of mobile attribution, cohort analysis, customer segmentation, and ltv modeling
• Experience pulling data and creating databases using an API of at least one of these ad platforms; Facebook, Snapchat, TikTok, google ads, applovin
• Experience with the architecture and deployment of mobile tracking solutions, including SDK integration, ad platforms APIs, server-postbacks, and mmp providers such as Appsflyer, adjust, kochava

If you are data driven individual with a passion for start-ups and have experience in business analytics, we encourage you to apply for this position. We offer a competitive salary package, flexible working hours, and a supportive work environment that fosters growth and development.

Data Engineer JD:

• Designing, developing, constructing, installing, testing and maintaining the complete data management & processing systems.
• Building highly scalable, robust, fault-tolerant, & secure user data platform adhering to data protection laws.
• Taking care of the complete ETL (Extract, Transform & Load) process.
• Ensuring architecture is planned in such a way that it meets all the business requirements.
• Exploring new ways of using existing data, to provide more insights out of it.
• Proposing ways to improve data quality, reliability & efficiency of the whole system.
• Creating data models to reduce system complexity and hence increase efficiency & reduce cost.
• Introducing new data management tools & technologies into the existing system to make it more efficient.
• Setting up monitoring and alarming on data pipeline jobs to detect failures and anomalies

What do we expect from you?

• BS/MS in Computer Science or equivalent experience
• 5 years of recent experience in Big Data Engineering.
• Good experience in working with Hadoop and Big Data technologies like HDFS, Pig, Hive, Zookeeper, Storm, Spark, Airflow and NoSQL systems
• Excellent programming and debugging skills in Java or Python.
• Apache spark, python, hands on experience in deploying ML models
• Has worked on streaming and realtime pipelines
• Experience with Apache Kafka or has worked with any of Spark Streaming, Flume or Storm

Focus Area:

• 3+ years of industry experience in administering (including setting up, managing, monitoring) data processing pipelines (both streaming and batch) using frameworks such as Kafka, ELK Stack, Fluentd and streaming databases like druid
• Strong industry expertise with containerization technologies including kubernetes, docker-compose
• 2+ years of industry in experience in developing scalable data ingestion processes and ETLs
• Experience with cloud platform services such as AWS, Azure or GCP especially with EKS, Managed Kafka
• Experience with scripting languages. Python experience highly desirable.
• 2+ Industry experience in python
• Experience with popular modern web frameworks such as Spring boot, Play framework, or Django
• Demonstrated expertise of building cloud native applications
• Experience in administering (including setting up, managing, monitoring) data processing pipelines (both streaming and batch) using frameworks such as Kafka, ELK Stack, Fluentd
• Experience in API development using Swagger
• Strong expertise with containerization technologies including kubernetes, docker-compose
• Experience with cloud platform services such as AWS, Azure or GCP.
• Implementing automated testing platforms and unit tests
• Proficient understanding of code versioning tools, such as Git
• Familiarity with continuous integration, Jenkins

• Design and Implement Large scale data processing pipelines using Kafka, Fluentd and Druid
• Assist in dev ops operations
• Develop data ingestion processes and ETLs
• Design and Implement APIs
• Assist in dev ops operations
• Identify performance bottlenecks and bugs, and devise solutions to these problems
• Help maintain code quality, organization, and documentation
• Communicate with stakeholders regarding various aspects of solution.
• Mentor team members on best practices