Content creation

Description

Come Join Us

Experience.com - We make every experience matter more

Position: Senior GCP Data Engineer

Job Location: Chennai (Base Location) / Remote

Employment Type: Full Time

Summary of Position

A Senior Data Engineer is a professional who specializes in preparing big data infrastructure for analytical or operational uses. He/She is responsible for develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity. They collaborate with data scientists and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organisation.

Responsibilities:

• Collaborate with cross-functional teams to define, prioritize, and execute data engineering initiatives aligned with business objectives.
• Design and implement scalable, reliable, and secure data solutions by industry best practices and compliance requirements.
• Drive the adoption of cloud-native technologies and architectural patterns to optimize the performance, cost, and reliability of data pipelines and analytics solutions.
• Mentor and lead a team of Data Engineers.
• Demonstrate a drive to learn and master new technologies and techniques.
• Apply strong problem-solving skills with an emphasis on building data-driven or AI-enhanced products.
• Coordinate with ML/AI and engineering teams to understand data requirements.

Experience & Skills:

• 8+ years of Strong experience in ETL and ELT data from various sources in Data Warehouses
• 8+ years of experience in Python, Pandas, Numpy, and SciPy.
• 5+ years of Experience in GCP 
• 5+ years of Experience in BigQuery, PySpark, and Pub/Sub
• 5+ years of Experience working with and creating data architectures.
• Certified in Google Cloud Professional Data Engineer.
• Advanced proficiency in Google Cloud services such as Dataflow, Dataproc, Dataprep, Data Studio, and Cloud Composer.
• Proficient in writing complex Spark (PySpark) User Defined Functions (UDFs), Spark SQL, and HiveQL.
• Good understanding of Elastic search.
• Experience in assessing and ensuring data quality, data testing, and addressing data quality issues.
• Excellent understanding of Spark architecture and underlying frameworks including storage management.
• Solid background in database design and development, database administration, and software engineering across full life cycles.
• Experience with NoSQL data stores like MongoDB, DocumentDB, and DynamoDB.
• Knowledge of data governance principles and practices, including data lineage, metadata management, and access control mechanisms.
• Experience in implementing and optimizing data security controls, encryption, and compliance measures in GCP environments.
• Ability to troubleshoot complex issues, perform root cause analysis, and implement effective solutions in a timely manner.
• Proficiency in data visualization tools such as Tableau, Looker, or Data Studio to create insightful dashboards and reports for business users.
• Strong communication and interpersonal skills to effectively collaborate with technical and non-technical stakeholders, articulate complex concepts, and drive consensus.
• Experience with agile methodologies and project management tools like Jira or Asana for sprint planning, backlog grooming, and task tracking.

About ThoughtClan Technologies

ThoughtClan is a niche, technology-focused, 100+ people strong software company that works on building complex enterprise-scale web and mobile-oriented digitalization and data science-related projects. They are in IT Services as well as Product Development space. They focus on applying technology to enable businesses to function better. ThoughtClan is a team of highly specialized technical folks and is growing rapidly.

They have expertise in developing projects related to:

1. Data Science — including Image Analytics, Video Analytics, Building AI/ML-based Prediction Models, etc.
2. Blockchain — based Cryptocurrency and NFT projects.
3. Enterprise-Scale Greenfield Web and Mobile Application Development, Integration, eCommerce, Marketing, and Content Management projects.

We are looking for a Data Scientist to join our fast-growing team.

The candidate must have:

• 3–4 years’ experience in Data Modeling in Python and AI/ML.
• Hands-on experience with Machine Learning and Deep Learning techniques and tools. Tools: RAG, LLMs, Agentic AI, Langchain, Langgraph, PyTorch, OpenCV, Pandas, Scikit Learn, CrewAI, Autogen or AI chatbots. Proven ability to use/create algorithms and run simulations. Experience: Minimum 1–1.5 years or 2 projects. A technical understanding of Microservice Architectures is a plus.
• Good knowledge of Azure Platform for deployment.
• Good knowledge of web frameworks such as Flask.
• Hands-on knowledge on a NoSQL database (Maria DB, Mongo DB, etc.).
• Experience in Visualization of Data using tools like D3.js, Plotly, Power BI, and Tableau. Experience in visualizing large data is a plus.
• Experience using a variety of data mining/data analysis methods with the ability to drive business results using data-based insights and work with large data sets.
• Comfortable working with a wide range of stakeholders and functional teams.
• Good designing skills and communication skills.
• Good knowledge of front-end technologies (HTML, CSS, etc.) would be an advantage.

Job Description:

Production Assurance and Maintenance Activities

1. Production Assurance L2 and L3 activities (Incident reporting and management, Incident resolution and RCA, Real time communication, Assigning defect severity and triaging, On-call Support, UAT support and deployment)

2. Application Maintenance (Software maintenance, Disaster recovery Planning and support, Business Continuity planning and support, Application documentation, Release Management activities, Configuration management, Change Control documentation, Metrics reporting etc

Development Activities

1. Business Requirements (BRD Reviews, Impact analysis and high level estimation)

2. High Level Design

3. Low Level design

4. Build (development, Unit Testing, Code reviews and static performance reports)

5. User Acceptance Testing Support (Defect resolution etc)

6. Performance testing Support and issue resolution

7. Implementation and Control (Implementation support and Post implementation support)

8. Configuration Management

9. DOcumentation and Reporting

About the role:

SDET 2 [5-8 years’ experience]

Here’s what will be expected out of you*:

➢ Understanding the features & requirements and design end to end test cases

➢ Performing manual-functional testing

➢ Automating UI test cases as per requirements

➢ Automating API test cases using existing Karate Framework *Job duties may change at any time due to business needs.

What we want:

➢ Good knowledge of Core Java programming skills.

➢ Expertise in UI automation using Selenium web-driver with BDD framework.

➢ Expertise in functional testing to identify critical, negative & edge cases.

➢ Knowledge in API automation using rest-assured or Karate API

➢ Hands-on experience on jenkins, git, SQL

➢ Very good understanding of microservice architecture, AWS services

➢ Very strong experience in mobile automation testing using Appium would be an added advantage.

Location: Bangalore/Mumbai

Develop server-side applications using NestJS and NodeJS.

. Should have great hands-on knowledge of TypeOrm.

• Design and implement APIs, web services, and database models.

• Collaborate with front-end developers and other team members to integrate user-facing elements with server-side logic.

• Optimize application performance for maximum speed and scalability.

• Implement security and data protection measures.

• Conduct thorough testing and debugging of applications.

• Troubleshoot and resolve issues, ensuring high application reliability.

• Stay up-to-date with emerging technologies and industry trends, and apply them to projects as necessary.

• Collaborate with team members, participate in code reviews, and provide constructive feedbacks.

• Follow best practices and coding standards to ensure high-quality and maintainable code..

Requirements:

• Bachelor's degree in Computer Science...