Data Engineer at Inviz Ai Solutions Private Limited · Bengaluru (Bangalore) · 4 - 8 years · Profitable · Posted 28 Mar 2023
Scala
Python
JavaInViz is Bangalore Based Startup helping Enterprises simplifying the Search and Discovery experiences for both their end customers as well as their internal users. We use state-of-the-art technologies in Computer Vision, Natural Language Processing, Text Mining, and other ML techniques to extract information/concepts from data of different formats- text, images, videos and make them easily discoverable through simple human-friendly touchpoints.
TSDE - Data
Data Engineer:
- Should have total 3-6 Yrs of experience in Data Engineering.
- Person should have experience in coding data pipeline on GCP.
- Prior experience on Hadoop systems is ideal as candidate may not have total GCP experience.
- Strong on programming languages like Scala, Python, Java.
- Good understanding of various data storage formats and it’s advantages.
- Should have exposure on GCP tools to develop end to end data pipeline for various scenarios (including ingesting data from traditional data bases as well as integration of API based data sources).
- Should have Business mindset to understand data and how it will be used for BI and Analytics purposes.
- Data Engineer Certification preferred
|
Experience in Working with GCP tools like |
|
|
Store : CloudSQL , Cloud Storage, Cloud Bigtable, Bigquery, Cloud Spanner, Cloud DataStore |
|
|
Ingest : Stackdriver, Pub/Sub, AppEngine, Kubernete Engine, Kafka, DataPrep , Micro services |
|
|
Schedule : Cloud Composer |
|
|
Processing: Cloud Dataproc, Cloud Dataflow, Cloud Dataprep |
|
|
CI/CD - Bitbucket+Jenkinjs / Gitlab |
|
|
Atlassian Suite |
|
|
|
.
Shubham Vishwakarma
Full Stack Developer - Averlon
Similar jobs (10)
PythonRole Overview
We are looking for a GCP Data Engineer with 10+ years of experience to design, develop, and optimize scalable cloud-based data solutions. The ideal candidate will have strong hands-on expertise in GCP, BigQuery, and advanced SQL, with experience building data pipelines and working with large-scale datasets.
Key Responsibilities
- Design and develop scalable data pipelines and ETL/ELT processes on GCP.
- Build, optimize, and maintain data solutions using Google BigQuery.
- Develop complex SQL queries for data transformation, aggregation, and analysis.
- Design efficient data models and optimize pipelines for performance, scalability, and cost.
- Integrate data from multiple sources and ensure data quality, reliability, and availability.
- Troubleshoot pipeline and data issues and drive continuous improvement.
- Collaborate with data architects, analysts, application teams, and business stakeholders.
- Follow best practices for cloud security, data governance, testing, and documentation.
Required Skills
- 8+ years of Data Engineering experience
- Strong hands-on experience with GCP, Django, and MongoDB
- Extensive experience with BigQuery
- Advanced SQL skills
- Strong understanding of ETL/ELT and data pipeline development
- Data modeling and data warehousing experience
- Experience handling large-scale datasets and performance optimization
- Strong problem-solving and communication skills
Good to Have
- GCP services such as Cloud Storage, Dataflow, Pub/Sub, Cloud Composer, or Cloud Functions
- Python or other data engineering languages
- Experience with data governance and security
- Agile development experience
Job Summary
We are seeking a highly skilled GCP Data Engineer with strong expertise in Google Cloud Platform (GCP), Python, ETL, and modern data engineering technologies. The ideal candidate should have hands-on experience designing and building scalable data pipelines using BigQuery, Dataflow, Pub/Sub, Airflow, and modern data lake technologies such as Apache Iceberg or Delta Lake.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines on Google Cloud Platform.
- Build and optimize data processing workflows using Python and Google Cloud Dataflow (Apache Beam).
- Develop and manage large-scale analytical data models in BigQuery.
- Implement event-driven data ingestion using Google Cloud Pub/Sub.
- Create, schedule, and monitor workflows using Apache Airflow and Autosys.
- Design and implement modern data lake architectures using Apache Iceberg or Delta Lake.
- Optimize query performance, storage, and compute costs in GCP.
- Ensure data quality, governance, security, and compliance across data platforms.
- Collaborate with Data Scientists, Analysts, and Application teams to deliver scalable data solutions.
- Troubleshoot production issues and continuously improve pipeline reliability and performance.
Mandatory Skills
- Strong hands-on experience with Google Cloud Platform (GCP).
- Proficiency in Python programming.
- Experience in designing and implementing ETL/ELT pipelines.
- Strong knowledge of BigQuery.
- Experience with Google Cloud Dataflow (Apache Beam).
- Experience with Google Cloud Pub/Sub.
- Hands-on experience with Apache Airflow.
- Experience in job scheduling using Autosys.
- Experience with modern table formats such as Apache Iceberg or Delta Lake.
- Strong SQL and data modeling skills.
Preferred Skills
- Experience with Cloud Storage, Dataproc, Cloud Composer, and Cloud Functions.
- Knowledge of CI/CD pipelines and DevOps practices.
- Experience with Docker and Kubernetes.
- Familiarity with Git and Agile/Scrum methodologies.
- Knowledge of data warehousing and dimensional modeling.
- Exposure to streaming and real-time data processing.
Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
- 4–8+ years of experience in Data Engineering with hands-on expertise in GCP technologies.
Required Experience
- Strong experience in developing enterprise-grade data pipelines using Python and GCP.
- Hands-on experience with BigQuery, Dataflow, Pub/Sub, and Airflow.
- Experience scheduling and monitoring batch workflows using Autosys.
- Experience implementing modern data lake architectures using Apache Iceberg or Delta Lake.
- Strong understanding of ETL best practices, performance tuning, and data optimization.
- Excellent analytical, troubleshooting, and problem-solving skills.
Mandatory Skills
- Google Cloud Platform (GCP)
- Python
- ETL
- BigQuery
- Autosys
- Apache Airflow
- Google Cloud Pub/Sub
- Google Cloud Dataflow (Apache Beam)
- Apache Iceberg / Delta Lake
- SQL & Data Modeling
PythonExperience: 5+ Years
Employment Type: Full-Time
Role Overview
We are looking for an experienced GCP Data Engineer with 5+ years of experience in data engineering and strong hands-on expertise in Google BigQuery, Google Cloud Storage (GCS), Airflow/Cloud Composer, Python, and Vertex AI. The candidate should be capable of designing, developing, and maintaining scalable data pipelines and cloud-based data solutions on Google Cloud Platform.
Key Skills – Mandatory
- BigQuery – Strong hands-on experience in data warehousing, SQL, optimization, and performance tuning.
- Google Cloud Storage (GCS) – Experience with data storage, file management, and integration with data pipelines.
- Airflow / Cloud Composer – Experience in developing, scheduling, monitoring, and managing data workflows.
- Python – Strong programming skills for data engineering, ETL/ELT development, automation, and pipeline implementation.
- Vertex AI – Experience working with ML/AI workflows, model integration, or data pipelines supporting AI/ML solutions.
Good to Have / Added Advantage
- Dataproc – Experience with distributed data processing and Spark-based workloads.
- Cloud Data Fusion – Experience in building and managing data integration pipelines.
- Cloud Run – Understanding of deploying and running containerized applications/services on GCP.
- Experience with ETL/ELT processes and data pipeline development.
- Knowledge of GCP data architecture and cloud-native services.
- Experience in data quality, validation, monitoring, and troubleshooting.
Responsibilities
- Design, develop, and maintain scalable GCP-based data pipelines.
- Build and optimize data solutions using BigQuery and Cloud Storage.
- Develop and manage workflows using Airflow / Cloud Composer.
- Write efficient and reusable Python code for data processing and automation.
- Support Vertex AI integrations and AI/ML data workflows.
- Monitor pipeline performance and troubleshoot data processing issues.
- Work with cross-functional teams to understand data requirements and deliver reliable solutions.
- Implement best practices for data security, quality, scalability, and performance.
You must have :
- 5+ years of overall experience in Data Engineering.
- Strong hands-on experience with BigQuery, GCS, Airflow/Cloud Composer, Python, and Vertex AI.
- Strong understanding of data engineering concepts, ETL/ELT, data pipelines, and cloud technologies.
- Dataproc, Data Fusion, and Cloud Run experience will be an added advantage.
PythonJob Summary
Role Overview
We are looking for an experienced Data Engineer with strong expertise in Python, ETL, Advanced SQL, CI/CD, DevOps, and Data Analytics. The ideal candidate should have hands-on experience designing and developing scalable data pipelines, transforming large datasets, and supporting data-driven applications.
Experience with Google Cloud Platform (GCP) will be an added advantage.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Python and SQL.
- Develop complex and optimized SQL queries, stored procedures, and data transformations.
- Build and maintain reliable data integration workflows across multiple data sources.
- Perform data cleansing, validation, transformation, and quality checks.
- Analyze data and provide insights to support business and technical requirements.
- Implement and maintain CI/CD pipelines for data engineering applications.
- Work with DevOps practices and tools to automate deployments, monitoring, and infrastructure processes.
- Troubleshoot data pipeline failures, performance issues, and production incidents.
- Optimize data processing workflows for performance, scalability, and reliability.
- Collaborate with Data Analysts, Data Scientists, Developers, and other stakeholders.
- Follow best practices for version control, testing, documentation, and deployment.
- Contribute to cloud-based data engineering initiatives, preferably on GCP.
Required Skills
- 5–7 years of hands-on experience in Data Engineering.
- Strong programming skills in Python.
- Strong expertise in Advanced SQL and database concepts.
- Hands-on experience with ETL/ELT processes and data pipelines.
- Good understanding of Data Warehousing and Data Modeling concepts.
- Experience with CI/CD practices and tools.
- Strong understanding of DevOps principles, automation, and deployment processes.
- Strong data analytics and problem-solving skills.
- Experience working with large datasets and performance optimization.
- Good understanding of Git/version control and software development best practices.
Good to Have
- Hands-on experience with Google Cloud Platform (GCP).
- Exposure to GCP data services such as BigQuery, Cloud Storage, Dataflow, Composer, or Pub/Sub.
- Experience with containerization/orchestration technologies such as Docker/Kubernetes.
- Experience with workflow orchestration tools such as Airflow.
- Knowledge of cloud-based data architecture and distributed data processing.
Preferred Candidate Profile
- Strong analytical and problem-solving abilities.
- Good communication and stakeholder management skills.
- Ability to work independently as well as in a collaborative team environment.
- Strong ownership of data pipelines and production systems.
- Candidates who can join at short notice are preferred.
Mandatory Skills
Data Engineer, Python , ETL, GCP, Advanced SQL, Strong Data Analytics skills, CICD, Devops
PythonExperience: 6+ years overall Data Engineering experience.
Must-have — candidates should have hands-on experience in ALL of these:
- GCP (Google Cloud Platform) – strong hands-on experience
- Python – data engineering/ETL development
- SQL – advanced SQL, query optimization, data transformation
- BigQuery – strong hands-on experience with development, optimization and data warehousing
- Data Engineering / ETL – building and maintaining data pipelines
- GCP data services – preferably Cloud Storage, Dataflow, Pub/Sub, Composer/Airflow, etc.
- Data warehousing / dimensional modeling
PythonJob Summary: GCP Data Engineering Lead
Experience: 9+ Years
Location: Bangalore / Hyderabad
Notice Period: Immediate to 15 Days
Key Skills:
- Strong experience in GCP Data Engineering
- Proven Technical Lead / Lead experience
- Strong programming skills in Python
- Hands-on experience with PySpark
- Strong expertise in SQL / PL-SQL
- Good understanding of GCP data services and data engineering concepts
- Experience in designing and developing scalable data pipelines
- Strong problem-solving and technical leadership skills
Roles & Responsibilities:
- Lead the design and development of scalable GCP data engineering solutions.
- Develop and optimize data pipelines using Python, PySpark and SQL/PL-SQL.
- Design data processing solutions and ensure performance and scalability.
- Provide technical leadership, conduct code reviews, and mentor team members.
- Collaborate with business and technical teams to understand requirements and deliver data solutions.
- Troubleshoot issues and ensure quality across the data engineering lifecycle.
KubernetesScalaDescription
We are looking for Senior Data Engineers to join our AdTech team and build scalable, high-performance data platforms that power advertising insights and analytics. The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Spark and Scala.
You will work on designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.
Key Responsibilities
- Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
- Build and optimize distributed data applications using Spark and Scala.
- Develop reliable, high-performance data pipelines for batch and streaming workloads.
- Work with large datasets to ensure data quality, consistency, and performance.
- Collaborate with engineering, product, and analytics teams to deliver robust data solutions.
- Optimize data workflows for scalability, reliability, and cost efficiency.
- Deploy and manage data workloads in cloud and containerized environments.
- Troubleshoot production issues and continuously improve platform performance.
Requirements
Candidates who demonstrate:
- 5+ years of experience in Data Engineering or Big Data Engineering.
- Strong hands-on experience with Apache Spark and Scala.
- Experience building and maintaining ETL pipelines.
- Familiarity with Google Cloud Storage (GCS).
- Experience with Kubernetes (K8s).
- Strong SQL skills and understanding of distributed data processing.
- Excellent debugging, problem-solving, and performance optimization skills.
- Strong communication and collaboration skills.
Good to Have
- Experience with AWS and cloud-native data services.
- Familiarity with streaming technologies such as Kafka.
- Experience working on large-scale data platforms or AdTech systems.
- Exposure to orchestration tools such as Airflow.
Benefits
- Best-in-class salary: We hire strong talent and compensate accordingly.
- Proximity Talks: Meet and learn from designers, engineers, product leaders, and AI practitioners.
- Continuous learning: Work with a world-class team and stay close to the latest in AI, engineering, and product development.
- High-impact work: Build AI-first systems and products used at scale by global clients.
About Us
Proximity is the trusted technology, design, and consulting partner for some of the biggest Sports, Media, and Entertainment companies in the world. We’re headquartered in San Francisco and have offices in Palo Alto, Dubai, Mumbai, and Bangalore.
Since 2019, Proximity has built high-impact, scalable products used by millions of users every day. Today, we are a global team of engineers, designers, product managers, and experts solving complex problems and building cutting-edge technology at scale.
Senior Data Engineer – Ab Initio | GCP | Spark | Agentic AI
Location: Bangalore
Experience: 5+ Years
Role: Senior Data Engineer
Work Mode: Bangalore
Job Summary
We are looking for an experienced Senior Data Engineer with strong expertise in Ab Initio, GCP, Apache Spark, and Agentic AI. The ideal candidate will have hands-on experience designing and developing scalable data engineering solutions, building data pipelines, and working with modern cloud and AI technologies.
The candidate should be comfortable working across traditional enterprise data platforms and emerging Generative AI / Agentic AI solutions.
Key Responsibilities
- Design, develop, and maintain scalable and high-performance data pipelines using Ab Initio, Spark, and GCP services.
- Develop and optimize complex ETL/ELT workflows using Ab Initio.
- Build and maintain data processing solutions using Apache Spark / PySpark.
- Develop cloud-based data solutions on Google Cloud Platform (GCP).
- Work with GCP data services such as BigQuery, Cloud Storage, Dataflow, Dataproc, Pub/Sub, or equivalent services.
- Perform data integration, transformation, cleansing, and validation.
- Optimize data pipelines for performance, scalability, reliability, and cost.
- Collaborate with data architects,
Python
LeadershipGCP Data Engineering Lead
Experience: 9+ Years
Lead Experience: 2+ Years
Location: Bangalore / Hyderabad
Key Skills:
- Strong experience in GCP Data Engineering and BigQuery
- Hands-on experience with Oracle Exadata / PL-SQL
- Strong knowledge of PySpark / Scala and Python
- Experience with GoldenGate, Kafka and CDC
- Hands-on experience with Apache Airflow
- Good experience in CI/CD and DevOps practices
- Experience with Terraform / Infrastructure as Code
- Exposure to AI/LLM technologies and GenAI solutions
- Strong understanding of data architecture, ETL/ELT and data pipelines
Roles & Responsibilities:
- Lead the design and development of scalable GCP data engineering solutions.
- Design and implement batch and real-time data pipelines using BigQuery, PySpark, Kafka/CDC and Airflow.
- Work with Oracle Exadata/PL-SQL and GoldenGate for data integration and migration.
- Implement CI/CD pipelines and infrastructure automation using Terraform.
- Explore and integrate AI/LLM capabilities into data engineering solutions.
- Lead technical discussions, code reviews, solution design and mentor team members.
- Collaborate with business and technical teams to deliver high-quality data solutions.
Python🚨 Hiring: GCP Data Engineer
We are looking for experienced GCP Data Engineers to join our team!
🔹 Experience: 9+ Years
🔹 Relevant Experience: 4+ Years in GCP Data Engineering
🔹 Required Skills: GCP, Oracle PL/SQL, Python, PySpark
🔹 Location: Bangalore / Hyderabad
🔹 Notice Period: Immediate to 10 Days Preferred
Key Skills:
🔹 Strong hands-on experience in GCP Data Engineering
🔹 Good experience with PySpark & Python
🔹 Strong knowledge of Oracle PL/SQL
🔹 Experience in data processing, ETL, and data pipelines
🔹 Good understanding of cloud-based data engineering
📩 Interested candidates can share their updated CV via DM.
#Hiring #GCPDataEngineer #GCP #DataEngineering #PySpark #Python #OraclePLSQL #DataEngineer #BangaloreJobs #HyderabadJobs #ImmediateJoiner #TechJobs #ITJobs #HiringNow
Follow CutshortShubham Vishwakarma
Full Stack Developer - Averlon