Data Engineer

Company Description

TECHSOPHY specializes in productizing solutions based on new technology, focusing on emerging platforms of BPM & ECM, Low Code, AI (ML/RPA/NLP). Founded in 2009, TECHSOPHY operates with headquarters in California, USA, and regional offices in Dubai, UAE, and an offshore innovation center in Hyderabad, India.

Qualifications

• Solid Fundamentals and exceptional problem-solving skills
• Solid and fluent understanding of algorithms and data structures
• Proficiency in Scala + Spark
• Proficiency in Scala + Play framework
• Experience Range: 4 to 7 Years

Requirement:

Some or all of them – because we believe intelligent people can pick up whatever they need in a short period of time. You just need to prove that you can:

• Excellent programming skills and knowledge of Java / Scala
• Excellent software design, problem-solving, and debugging skills
• Experience with modern Big Data technologies such as Spark, NoSQL, Cassandra, Kafka, MapReduce, and the Hadoop ecosystem is a must-have
• Experience with data analytics and the ability to mine data to obtain insights are much appreciated

4 - 10 years of experience in designing and buildingarchitecting highly resilient data platforms 

∙Strong knowledge of data engineering, architecture and data modeling 

∙Experience in platforms like Databricks and Snowflake 

∙Experience on building applications on cloud (AWS or Azure or Google Cloud) 

∙Strong analytical and problem-solving skills 

∙Prior experience in developing data or computation intensive (e.g. grid based) backend applications is an 

advantage 

∙OOP design skills with an understanding or at least personal interest towards the concepts of Functional 

Programming  

∙Willingness to understand and enhance other people’s code, being able to work in an environment where 

developers will oversee and work on wider components also dealing with older “legacy” code 

∙Strong programming skills (Java/ Scala / Python) skills with the willingness to pick up the other language if not 

already mastered at a sufficient level is important 

∙Spring knowledge is an advantage, but in general willingness to learn, work with and even enhance in-house 

developed frameworks is a must 

∙Prior experience in working with Git, Bitbucket, Jenkins, working with PR-s, using JIRA, following the Scrum Agile 

methodology is an advantage 

∙Prior knowledge of financial products is an advantage 

∙Bachelors or Masters in any relevant field of IT/Engineering area is an advantage  

Must have experience in Java

Must have experience in Spark

Must have experience in ETL coding

Strong expertise in coding

Company: Wissen Technology

Position: Databricks Engineer

Experience: 6-10 Years

Location: Bengaluru/Mumbai

Employment Type: Full-time

About Wissen Technology

Wissen Technology is a global technology services company focused on delivering innovative software engineering, data, and technology solutions to leading enterprises. The company works with clients across industries to build scalable, high-performance technology platforms and solve complex business and technology challenges.

With a strong focus on engineering excellence, innovation, and collaboration, Wissen Technology brings together skilled technology professionals across areas such as software engineering, data engineering, cloud, analytics, and digital transformation.

At Wissen Technology, employees have the opportunity to work on challenging technology projects, collaborate with experienced engineering teams, and contribute to solutions that create measurable business impact.

Key Responsibilities

• Design, develop, and maintain scalable data processing applications using Python, PySpark, and Spark.
• Develop and optimize data pipelines and workflows on Databricks.
• Collaborate with data engineers, data scientists, business stakeholders, and other technical teams to understand requirements and deliver high-quality solutions.
• Ensure data integrity, quality, performance, and reliability across data processing pipelines.
• Write clean, maintainable, scalable, and efficient code following established coding standards and best practices.
• Perform data analysis and implement appropriate data validation and quality checks.
• Monitor, troubleshoot, and optimize performance issues across data workflows and pipelines.
• Work with relational databases and develop efficient SQL queries for data extraction and transformation.
• Participate in code reviews, testing, deployment, and continuous improvement of data engineering solutions.
• Use Git/version control and follow established software development and deployment practices.

Required Skills & Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, or a related field.
• Proven experience as a Databricks Developer, Data Engineer, or similar role.
• Strong hands-on expertise in Apache Spark and PySpark.
• Strong programming skills in Python; experience with Scala is an advantage.
• Strong proficiency in SQL and hands-on experience with relational databases.
• Practical experience developing and optimizing data pipelines and data processing applications.
• Familiarity with Git and version control systems.
• Strong understanding of data engineering concepts, data transformation, and data validation.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.

Description

We are looking for Senior Data Engineers to join our AdTech team and build scalable, high-performance data platforms that power advertising insights and analytics. The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Spark and Scala.

You will work on designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
• Build and optimize distributed data applications using Spark and Scala.
• Develop reliable, high-performance data pipelines for batch and streaming workloads.
• Work with large datasets to ensure data quality, consistency, and performance.
• Collaborate with engineering, product, and analytics teams to deliver robust data solutions.
• Optimize data workflows for scalability, reliability, and cost efficiency.
• Deploy and manage data workloads in cloud and containerized environments.
• Troubleshoot production issues and continuously improve platform performance.

Requirements

Candidates who demonstrate:

• 5+ years of experience in Data Engineering or Big Data Engineering.
• Strong hands-on experience with Apache Spark and Scala.
• Experience building and maintaining ETL pipelines.
• Familiarity with Google Cloud Storage (GCS).
• Experience with Kubernetes (K8s).
• Strong SQL skills and understanding of distributed data processing.
• Excellent debugging, problem-solving, and performance optimization skills.
• Strong communication and collaboration skills.

Good to Have

• Experience with AWS and cloud-native data services.
• Familiarity with streaming technologies such as Kafka.
• Experience working on large-scale data platforms or AdTech systems.
• Exposure to orchestration tools such as Airflow.

Benefits

• Best-in-class salary: We hire strong talent and compensate accordingly.
• Proximity Talks: Meet and learn from designers, engineers, product leaders, and AI practitioners.
• Continuous learning: Work with a world-class team and stay close to the latest in AI, engineering, and product development.
• High-impact work: Build AI-first systems and products used at scale by global clients.

About Us

Proximity is the trusted technology, design, and consulting partner for some of the biggest Sports, Media, and Entertainment companies in the world. We’re headquartered in San Francisco and have offices in Palo Alto, Dubai, Mumbai, and Bangalore.

Since 2019, Proximity has built high-impact, scalable products used by millions of users every day. Today, we are a global team of engineers, designers, product managers, and experts solving complex problems and building cutting-edge technology at scale.

Description

We are looking for Senior Data Engineers to join our Data Platform team and build scalable, high-performance data platforms that power data processing, analytics, and downstream applications.

The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Apache Spark and Python Scala.

You will be responsible for designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL and data processing pipelines for large-scale datasets.
• Build and optimize distributed data applications using Apache Spark and Python Scala.
• Develop reliable, high-performance data pipelines for batch and streaming workloads.
• Design and manage data workflows using Apache Airflow.
• Build and operate data workloads on AWS, with strong usage of Amazon S3 for large-scale data storage.
• Work with large datasets to ensure data quality, consistency, reliability, and performance.
• Collaborate with engineering, product, analytics, and other platform teams to deliver robust data solutions.
• Optimize data workflows for scalability, reliability, performance, and cost efficiency.
• Troubleshoot production issues, identify bottlenecks, and continuously improve platform performance.

Requirements

Candidates who demonstrate:

• 5+ years of experience in Data Engineering, Big Data Engineering, or a similar role.
• Strong hands-on experience with Apache Spark and Scala.
• Experience designing, building, and maintaining large-scale ETL pipelines.
• Strong hands-on experience with AWS, particularly Amazon S3.
• Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
• Strong SQL skills and a solid understanding of distributed data processing concepts.
• Experience working with batch and/or streaming data pipelines.
• Excellent debugging, problem-solving, and performance optimization skills.
• Strong communication and collaboration skills.

Good to Have

• Experience with Databricks and the broader Databricks data platform.
• Familiarity with streaming technologies such as Apache Kafka.
• Experience working on large-scale data platforms handling high-volume data workloads.
• Exposure to additional AWS data services and cloud-native data architectures.

Join our product development team at Planview as a Senior Software Engineer I and become a pivotal force on the Viz Core Team. This role offers the unique opportunity to shape and lead the development of data-processing pipelines and APIs that are at the heart of our software solutions. These solutions are designed to streamline and enhance the efficiency of software delivery, resonating deeply with software engineers who strive to build better and faster.

At Planview Viz, which is powered by the innovative “Flow Framework,” you will tackle complex data challenges and develop scalable solutions within an AWS cloud environment. Your efforts will be crucial in revolutionizing how businesses harness and interpret vast amounts of workflow data, transforming it into actionable insights that propel organizational efficiency and effectiveness.

Responsibilities (What you'll do)

• Create and refine powerful data-processing architectures that integrate seamlessly with a diverse array of external tools, enhancing the way software is delivered across industries.
• Drive operational excellence by critically analyzing problems, defining requirements, and devising robust solutions that push the boundaries of technology.
• Lead and inspire a team of talented engineers, promoting a culture of ownership, meticulous attention to quality, and proactive problem-solving.
• Stay at the cutting edge of technology by updating and expanding your team’s knowledge of cloud architectures, data processing, and advanced analytics, ensuring that you and your team remain leaders in technological innovation.
• Make a direct impact on the efficiency and effectiveness of software delivery worldwide through innovation and leadership.

Qualifications (What you'll bring)

Who We’re Looking For

The ideal candidate is a seasoned professional in cloud software development with a strong foundation in data processing and scalable software systems. You thrive in collaborative environments and are passionate about advancing cloud technology and data architecture to new heights. You are a leader who enjoys mentoring, guiding, and inspiring others.

Preferred Qualifications

Skills, Knowledge, and Expertise

• A degree in Computer Science, Engineering, or a related field.
• 6+ years of experience with a modern programming language, with a focus on back-end systems and cloud-based technologies.
• Strong capability in architecting and designing robust, scalable software systems.
• Proficiency in AWS or another popular cloud platform.

Additional Qualifications

• Experience with big-data technologies such as Spark and Apache Kafka.
• Experience with Java (version 17), Scala, or other JVM programming languages.
• Experience with tools such as Amazon Redshift and MongoDB.
• Experience with continuous integration and continuous deployment tools such as GitHub, Jenkins, Travis CI, or similar platforms.
• Proficiency in test-driven development.
• Experience with containers and orchestration tools such as Kubernetes.
• Experience working on remote or distributed teams and projects.

About the Role

We are looking for a Forward Deployed Engineer with strong hands-on experience in Databricks and AI-assisted software development using Cursor. The role is suited for an engineer who can work directly with clients and internal teams to understand business problems, rapidly build solutions, and take them from prototype to production.

The ideal candidate should have strong expertise in Python, SQL, Databricks, PySpark, data engineering, APIs, and modern AI-assisted development workflows, along with excellent problem-solving and client-facing skills.

Key Responsibilities

Forward Deployed Engineering

• Work directly with clients and stakeholders to understand business and technical requirements.
• Translate business problems into scalable technical and data solutions.
• Rapidly prototype, test, iterate, and productionize solutions.
• Collaborate with engineering, data, AI, and delivery teams to implement customer solutions.
• Troubleshoot production issues and continuously improve deployed solutions.
• Act as a technical bridge between clients and internal engineering teams.

Databricks & Data Engineering

• Design and develop scalable data solutions using Databricks, PySpark, Python, and SQL.
• Build and optimize ETL/ELT and data processing pipelines.
• Work with Databricks Lakehouse, Delta Lake, Unity Catalog, and Databricks Workflows.
• Develop data ingestion and transformation pipelines for structured and semi-structured data.
• Optimize Databricks workloads for performance, scalability, reliability, and cost.
• Integrate Databricks with databases, APIs, cloud platforms, and enterprise applications.

Cursor & AI-Assisted Development

• Use Cursor and AI-assisted development workflows to accelerate software development, debugging, refactoring, and documentation.
• Effectively use AI coding assistants to understand existing codebases and develop new features.
• Apply appropriate engineering judgment to review, validate, test, and secure AI-generated code.
• Use AI-assisted development for rapid prototyping and proof-of-concept development.
• Work with modern AI/LLM APIs and tools where required for customer solutions.
• Stay current with emerging AI-assisted software engineering practices.

Production & Deployment

• Develop production-ready applications, APIs, and data pipelines.
• Work with Git, CI/CD, APIs, containers, and cloud environments.
• Monitor application and pipeline performance and resolve production issues.
• Ensure solutions meet requirements for scalability, security, reliability, and maintainability.
• Collaborate with Data Scientists and ML Engineers to integrate AI/ML capabilities into production systems.

Required Skills & Experience

• 4+ years of experience in Software Engineering, Data Engineering, AI Engineering, or a related field.
• Strong hands-on experience with Databricks.
• Strong proficiency in Python, PySpark, and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Experience building production-grade data pipelines.
• Hands-on experience with Cursor or similar AI-powered coding assistants.
• Strong understanding of REST APIs and system integrations.
• Experience working with cloud platforms such as AWS, Azure, or GCP.
• Strong debugging, problem-solving, and analytical skills.
• Excellent communication and client-facing abilities.

Preferred Skills

• Experience with Databricks Unity Catalog, Workflows, and MLflow.
• Experience with Generative AI / LLM applications.
• Knowledge of Claude, OpenAI, Azure OpenAI, or other LLM platforms.
• Experience with RAG, vector databases, embeddings, or AI agents.
• Experience with Docker, Kubernetes, and CI/CD.
• Experience in a consulting, customer-facing engineering, or professional services environment.
• Exposure to Agile/Scrum methodologies.

Key Competencies

• Strong problem-solving and ownership mindset
• Ability to work in ambiguous and fast-paced environments.
• Strong client/stakeholder management skills.
• Ability to understand business requirements and convert them into technical solutions.
• Strong communication and presentation skills.
• Ability to rapidly learn new technologies and tools.
• Comfortable working with AI-assisted development while maintaining high engineering standards.

Education

• Bachelor's or master’s degree in computer science, Information Technology, Engineering, Data Science, or a related discipline.

Company Name – Wissen Technology

Group of companies in India – Wissen Technology & Wissen Infotech

Work Location – Whitefield, Bangalore

Website and Company profile:

www.wissen.com

LinkedIn Page:

https://www.linkedin.com/company/wissen-technology/

While you may already know about Wissen and the company history, here is a quick rundown for you.

About Wissen Technology:

·    The Wissen Group was founded in the year 2000. Wissen Technology, a part of Wissen Group, was established in the year 2015.

·    Wissen Technology is a specialized technology company that delivers high-end consulting for organizations in the Banking & Finance, Telecom, and Healthcare domains. We help clients build world class products.

·    Our workforce has highly skilled professionals, with leadership and senior management executives who have graduated from Ivy League Universities like Wharton, MIT, IITs, IIMs, and NITs and with rich work experience in some of the biggest companies in the world.

·    Wissen Technology has grown its revenues by 400% in these five years without any external funding or investments.

·    Globally present with offices US, India, UK, Australia, Mexico, and Canada.

·    We offer an array of services including Application Development, Artificial Intelligence & Machine Learning, Big Data & Analytics, Visualization & Business Intelligence, Robotic Process Automation, Cloud, Mobility, Agile & DevOps, Quality Assurance & Test Automation.

·    Wissen Technology has been certified as a Great Place to Work®.

·    Wissen Technology has been voted as the Top 20 AI/ML vendor by CIO Insider in 2020.

·    Over the years, Wissen Group has successfully delivered $650 million worth of projects for more than 20 of the Fortune 500 companies.

·    We have served client across sectors like Banking, Telecom, Healthcare, Manufacturing, and Energy. They include likes of Morgan Stanley, Goldman Sachs, MSCI, StateStreet, Flipkart, Swiggy, Trafigura, GE to name a few.

About Role :

Key Responsibilities

• Build and maintain data transformation pipelines using java Spark
• Develop and optimize large-scale/CPU intensive data processing using Apache Spark
• Orchestrate workflows using Airflow
• Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
• Support schema evolution, backfills, and incremental processing
• Ensure pipelines meet SLAs for freshness, reliability, and performance
• Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

• Strong hands-on experience with 
• HBase
• Apache Spark
• Experience with HBase or similar lakehouse query engines
• Airflow 
• Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)
• Proficiency in Java
• Experience with Git-based development and CI/CD

Nice-to-Have Skills

• OpenTable format/Iceberg ,Apache Arrow
• CDC-based analytics pipelines
• Cloud platforms (AWS)
• Kubernetes-based data platforms

We are looking for a skilled Python & PySpark Developer with strong expertise in Big Data technologies, Spark, SQL/PL-SQL, and REST API development using Flask or Django. The ideal candidate should have experience building scalable data pipelines, processing large datasets, developing APIs, and working with distributed computing frameworks.

Key Responsibilities

• Develop, optimize, and maintain scalable data pipelines using PySpark and Apache Spark.
• Design, develop, and optimize complex SQL and PL/SQL queries, stored procedures, functions, and database objects.
• Build and maintain RESTful APIs using Flask or Django.
• Develop robust Python applications for data engineering and backend services.
• Process and analyze large-scale datasets using Big Data technologies.
• Optimize Spark jobs for performance, scalability, and reliability.
• Integrate APIs with internal and external systems.
• Collaborate with cross-functional teams including Data Engineers, Data Scientists, and Application Developers.
• Troubleshoot production issues and implement performance improvements.
• Follow coding standards, version control, and CI/CD best practices.

Mandatory Skills

• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Strong SQL coding skills.
• Experience with PL/SQL development.
• Experience in Big Data ecosystem.
• REST API development using Flask or Django.
• Experience in developing and consuming Python APIs.
• Knowledge of data processing, ETL, and distributed computing.
• Experience with Git/version control.

Preferred Skills

• Experience with Hadoop ecosystem (Hive, HDFS, YARN).
• Exposure to cloud platforms such as AWS, Azure, or GCP.
• Knowledge of Airflow or other workflow orchestration tools.
• Experience with Docker and Kubernetes.
• Familiarity with Kafka or other streaming technologies.
• Understanding of CI/CD pipelines.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• 4–8+ years of experience in Python and Big Data development (can be adjusted based on the role).

Required Experience

• Strong hands-on experience in Python, PySpark, and Apache Spark.
• Extensive experience writing optimized SQL and PL/SQL code.
• Experience developing REST APIs using Flask or Django.
• Experience working with large-scale data processing and ETL pipelines.
• Strong analytical, debugging, and problem-solving skills.

Mandatory Skills: Python, PySpark, SQL Coding, Apache Spark, Big Data, Flask/Django (REST API), PL/SQL, Python APIs.