Bits In Glass

We are seeking a highly skilled Senior Data Engineer with expertise in Databricks, Python, Scala, Azure Synapse, and Azure Data Factory to join our data engineering team. The team is responsible for ingesting data from multiple sources, making it accessible to internal stakeholders, and enabling seamless data exchange across internal and external systems.

You will play a key role in enhancing and scaling our Enterprise Data Platform (EDP) hosted on Azure and built using modern technologies such as Databricks, Synapse, Azure Data Factory (ADF), ADLS Gen2, Azure DevOps, and CI/CD pipelines.

Responsibilities

• Design, develop, optimize, and maintain scalable data architectures and pipelines aligned with ETL principles and business goals.
• Collaborate across teams to build simple, functional, and scalable data solutions.
• Troubleshoot and resolve complex data issues to support business insights and organizational objectives.
• Build and maintain data products to support company-wide usage.
• Advise, mentor, and coach data and analytics professionals on standards and best practices.
• Promote reusability, scalability, operational efficiency, and knowledge-sharing within the team.
• Develop comprehensive documentation for data engineering standards, processes, and capabilities.
• Participate in design and code reviews.
• Partner with business analysts and solution architects on enterprise-level technical architectures.
• Write high-quality, efficient, and maintainable code.

Technical Qualifications

• 5–8 years of progressive data engineering experience.
• Strong expertise in Databricks, Python, Scala, and Microsoft Azure services including Synapse & Azure Data Factory (ADF).
• Hands-on experience with data pipelines across multiple source & target systems (Databricks, Synapse, SQL Server, Data Lake, SQL/NoSQL sources, and file-based systems).
• Experience with design patterns, code refactoring, CI/CD, and building scalable data applications.
• Experience developing batch ETL pipelines; real-time streaming experience is a plus.
• Solid understanding of data warehousing, ETL, dimensional modeling, data governance, and handling both structured and unstructured data.
• Deep understanding of Synapse and SQL Server, including T-SQL and stored procedures.
• Proven experience working effectively with cross-functional teams in dynamic environments.
• Experience extracting, processing, and analyzing large / complex datasets.
• Strong background in root cause analysis for data and process issues.
• Advanced SQL proficiency and working knowledge of a variety of database technologies.
• Knowledge of Boomi is an added advantage.

Core Skills & Competencies

• Excellent analytical and problem-solving abilities.
• Strong communication and cross-team collaboration skills.
• Self-driven with the ability to make decisions independently.
• Innovative mindset and passion for building quality data solutions.
• Ability to understand operational systems, identify gaps, and propose improvements.
• Experience with large-scale data ingestion and engineering.
• Knowledge of CI/CD pipelines (preferred).
• Understanding of Python and parallel processing frameworks (MapReduce, Spark, Scala).
• Familiarity with Agile development methodologies.

Education

• Bachelor’s degree in Computer Science, Information Technology, MIS, or an equivalent field.

As a Data Engineer, you will be an integral part of our team, working on data pipelines, data warehousing, and data integration for various analytics and AI use cases. You will collaborate closely with Delivery Managers, ML Engineers and other stakeholders to ensure seamless data flow and accessibility. Your expertise will be crucial in enabling data-driven decision-making for our clients. To thrive in this role, you need to be a quick learner, get excited about innovation and be on the constant lookout to master new technologies as they come up in the Data, AI & Cloud teams.   

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL processes to support downstream analytics and AI applications.
• Collaborate with ML Engineers to integrate data solutions into machine learning models and workflows.
• Work closely with clients to understand their data requirements and deliver tailored data solutions.
• Ensure data quality, integrity, and security across all projects.
• Optimize and manage data storage solutions in cloud environments (AWS, Azure, GCP).
• Utilize Databricks for data processing and analytics tasks, leveraging its capabilities to enhance data workflows.
• Monitor the performance of data pipelines, identify bottlenecks or failures, and implement improvements to enhance efficiency and reliability.
• Implement best practices for data engineering, including documentation, testing, and version control.
• Troubleshoot and resolve data-related issues in a timely manner.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
• 3 to 5 years of experience as a Data Engineer or in a similar role.
• Strong proficiency in SQL, Python, and other relevant programming languages.
• Hands-on experience with Databricks and its ecosystem.
• Familiarity with major cloud environments (AWS, Azure, GCP) and their data services.
• Experience with data warehousing solutions like Snowflake, Redshift, or BigQuery.
• Comfortable working with a variety of SQL, NoSQL and graph databases like PostgreSQL and MongoDB;
• Knowledge of data integration tools.
• Understanding of data modelling, data architecture, and database design.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills.

Highly Desirable Skills

• Experience with real-time data processing frameworks (e.g., Apache Kafka, Spark Streaming).
• Knowledge of data visualisation tools (e.g., Tableau, Power BI).
• Familiarity with machine learning concepts and frameworks.
• Experience working in a client-facing role.

As a Google Cloud Infrastructure / DevOps Engineer, you will design, implement, and maintain cloud infrastructure while enabling efficient development operations. This role bridges development and operations, with a strong focus on automation, scalability, reliability, and collaboration. You will work closely with cross-functional teams to optimize systems and enhance CI/CD pipelines.

Key Responsibilities:

Cloud Infrastructure Management

• Manage and monitor Google Cloud Platform (GCP) services and components.
• Ensure high availability, scalability, and security of cloud resources.

CI/CD Pipeline Implementation

• Design and implement automated pipelines for application releases.
• Build and maintain CI/CD workflows.
• Collaborate with developers to streamline deployment processes.
• Automate testing, deployment, and rollback procedures.

Infrastructure as Code (IaC)

• Use Terraform (or similar tools) to define and manage infrastructure.
• Maintain version-controlled infrastructure code.
• Ensure environment consistency across dev, staging, and production.

Monitoring & Troubleshooting

• Monitor system performance, resource usage, and application health.
• Troubleshoot cloud infrastructure and deployment pipeline issues.
• Implement proactive monitoring and alerting.

Security & Compliance

• Apply cloud security best practices.
• Ensure compliance with industry standards and internal policies.
• Collaborate with security teams to address vulnerabilities.

Collaboration & Documentation

• Work closely with development, operations, and QA teams.
• Document architecture, processes, and configurations.
• Share knowledge and best practices with the team.

Qualifications:

Education

• Bachelor’s degree in Computer Science, Information Technology, or a related field.

Experience

• Minimum 3 years of industry experience.
• At least 1 year designing and managing production systems on GCP.
• Familiarity with GCP services (Compute Engine, GKE, Cloud Storage, etc.).
• Exposure to Docker, Kubernetes, and microservices architecture.

Skills

• Proficiency in Python or Bash for automation.
• Strong understanding of DevOps principles.
• Knowledge of Jenkins or other CI/CD tools.
• Experience with GKE for container orchestration.
• Familiarity with event streaming platforms (Kafka, Google Cloud Pub/Sub).

About the Company: Bits In Glass – India

Industry Leader

• Established for 20+ years with global operations in the US, Canada, UK, and India.
• In 2021, Bits In Glass joined hands with Crochet Technologies, strengthening global delivery capabilities.
• Offices in Pune, Hyderabad, and Chandigarh.
• Specialized Pega Partner since 2017, ranked among the top 30 Pega partners globally.
• Long-standing sponsor of the annual PegaWorld event.
• Elite Appian partner since 2008 with deep industry expertise.
• Dedicated global Pega Center of Excellence (CoE) supporting customers and development teams worldwide.

Employee Benefits

• Career Growth: Clear pathways for advancement and professional development.
• Challenging Projects: Work on innovative, high-impact global projects.
• Global Exposure: Collaborate with international teams and clients.
• Flexible Work Arrangements: Supporting work-life balance.
• Comprehensive Benefits: Competitive compensation, health insurance, paid time off.
• Learning Opportunities: Upskill on AI-enabled Pega solutions, data engineering, integrations, cloud migration, and more.

Company Culture

• Collaborative Environment: Strong focus on teamwork, innovation, and knowledge sharing.
• Inclusive Workplace: Diverse and respectful workplace culture.
• Continuous Learning: Encourages certifications, learning programs, and internal knowledge sessions.

Core Values

• Integrity: Ethical practices and transparency.
• Excellence: Commitment to high-quality work.
• Client-Centric Approach: Delivering solutions tailored to client needs.