Data Engineer at DemandMatrix · Remote only · 9 - 12 years · ₹25L - ₹30L / yr · Profitable · Remote only · Posted 31 Jan 2021
Only a solid grounding in computer engineering, Unix, data structures and algorithms would enable you to meet this challenge. 7+ years of experience architecting, developing, releasing, and maintaining large-scale big data platforms on AWS or GCP Understanding of how Big Data tech and NoSQL stores like MongoDB, HBase/HDFS, ElasticSearch synergize to power applications in analytics, AI and knowledge graphs Understandingof how data processing models, data location patterns, disk IO, network IO, shuffling affect large scale text processing - feature extraction, searching etc Expertise with a variety of data processing systems, including streaming, event, and batch (Spark, Hadoop/MapReduce) 5+ years proficiency in configuring and deploying applications on Linux-based systems 5+ years of experience Spark - especially Pyspark for transforming large non-structured text data, creating highly optimized pipelines Experience with RDBMS, ETL techniques and frameworks (Sqoop, Flume) and big data querying tools (Pig, Hive) Stickler of world class best practices, uncompromising on the quality of engineering, understand standards and reference architectures and deep in Unix philosophy with appreciation of big data design patterns, orthogonal code design and functional computation models |

About DemandMatrix
About
Connect with the team
Similar jobs (10)

Role overview
The client is building a multimodal AI platform that processes multi-hour video, audio and text to generate structured insights, narratives and highlight workflows for broadcasters and media organisations.
We are seeking a Backend / Platform Engineer to design and build high-throughput media pipelines, robust APIs, and model-serving infrastructure that connect our AI engine (video perception + multimodal reasoning) to real products and customer environments.
This is not a CRUD‑only backend role.
You will work on:
- long‑running jobs
- distributed processing
- GPU inference orchestration
- storage for embeddings and metadata
- integration with AI models
- reliability and observability at scale
Key responsibilities
Media ingestion & processing pipelines
- Design and implement ingestion pipelines for multi‑hour video and audio content.
- Build microservices for frame extraction, audio processing, transcription integration and metadata generation.
- Handle long‑running, asynchronous jobs using queues, workers and robust retry strategies.
- Integrate with FFmpeg or similar tools for transcoding, segmenting and preparing media for AI models.
API & platform architecture
- Design and implement REST/gRPC APIs that expose AI model outputs (perception, multimodal alignment, narratives) to frontend and external systems.
- Define clear contracts for internal services and external integrations.
- Implement authentication, authorisation and rate‑limiting for platform endpoints.
- Ensure backward‑compatible API evolution as the product matures.
Model‑serving & AI integration
- Integrate with AI inference services (video models, multimodal models, LLM/VLM) running on GPUs or specialised infrastructure.
- Design request/response flows that handle large payloads, streaming outputs and structured results.
- Optimise throughput and latency for inference pipelines, including batching, caching and concurrency control.
- Collaborate closely with AI engineers to productionise models and debug end‑to‑end behaviour.
Storage, data models & performance
- Design data models to store embeddings, timelines, metadata, scene/shot boundaries, and narrative units.
- Work with appropriate storage technologies (SQL/NoSQL, object storage, search indices) based on access patterns.
- Implement indexing and query strategies for fast retrieval of segments, highlights and multimodal insights.
- Optimise performance for large datasets and high‑volume workloads.
Reliability, observability & operations
- Implement logging, metrics and tracing across services for debugging and monitoring.
- Set up health checks, circuit breakers and graceful degradation for critical services.
- Work with CI/CD pipelines to ensure safe, repeatable deployments.
- Collaborate on Kubernetes‑based deployments (or equivalent orchestration) for scaling services.
Requirements (must‑have)
Experience:
- 4–8 years in backend or platform engineering.
- At least 3 years working on distributed systems, high‑throughput services or complex pipelines (not just simple CRUD apps).
Languages & frameworks:
- Strong proficiency in Python or Node.js (one primary, both are a plus).
- Experience with at least one modern backend framework (FastAPI, Flask, Express, NestJS, etc.).
Distributed systems & pipelines:
- Hands‑on experience with queues and workers (e.g. Celery, RabbitMQ, Kafka, SQS, etc.).
- Experience building asynchronous, long‑running job pipelines.
- Understanding of idempotency, retries, backoff, and failure handling.
APIs & integration:
- Strong experience designing and implementing REST APIs (gRPC is a plus).
- Experience integrating with external services and handling network‑level failures.
Cloud & infrastructure:
- Experience deploying services on AWS, GCP or Azure (EC2/Compute Engine, S3/GCS, IAM, networking basics).
- Experience with Docker; exposure to Kubernetes is a strong plus.
Data & storage:
- Experience with SQL and at least one NoSQL store.
- Ability to design schemas and data models for performance and maintainability.
Engineering quality:
- Strong debugging skills across services and environments.
- Experience with unit/integration tests for backend systems.
- Clear, structured communication in English.
Nice‑to‑have
- Experience with media/video processing (FFmpeg, transcoding, segmenting).
- Experience with AI/ML model integration (serving models, handling inference requests).
- Experience with search/retrieval systems (e.g. Elasticsearch, vector databases).
- Experience with observability stacks (Prometheus, Grafana, OpenTelemetry).
- Experience working with remote teams across time zones.
What we are explicitly NOT looking for
To reduce noise and mismatches, we are not looking for:
- Pure CRUD‑only backend developers with no pipeline or distributed systems experience.
- Engineers who have only worked on small, single‑service apps without scale or complexity.
- Candidates who cannot explain trade‑offs in architecture, data modelling and reliability.
- Candidates who are uncomfortable with ownership of subsystems end‑to‑end.
Why join us
- Work on real, complex problems at the intersection of media, AI and distributed systems.
- Collaborate with senior AI engineers working on perception, multimodal fusion and narrative reasoning.
- Build the core platform that turns AI models into a usable product for broadcasters and media organisations.
- Operate with high ownership, clear expectations and direct access to the CTO.
SE/SSE - Backend (Java)
JAVA: Software Engineer / Senior Software Engineer
Job Overview: RevSure.AI is building a world-class team of Engineers with a mandate to architect, design, build, scale, and maintain our cutting-edge Revops platform and derive insights from the customer data. If you are a great JavaScript developer (or, for that matter, in any major programming language) with experience in building SaaS web applications and looking for an opportunity to build world-class products using cutting-edge technologies, please read on -
Responsibilities:
● Contribute to the growth and development of our product by writing and maintaining code, focusing on quality and maintainability. We believe that our teams own the code they build.
● Share your knowledge, both inside and outside your own team.
● Design, implement, test, and monitor valuable solutions that achieve the teamʼs goals while keeping a smooth delivery flow and balancing tradeoffs between scope, time, and effort.
● Continuously improve your and your teamʼs way of working, expanding the boundaries of your teamʼs autonomy and self-organisation.
Requirements:
Must-Have:
● Good interpersonal and communication skills. An analytical mind and an eye for detail
● Experience of 4-7 years in building large applications, preferably from the heavy data processing & analytics side
● Experience in architecting, designing, and building end-to-end System Design of multi tenant SAAS applications
● Experience architecting interactive configuration experiences around DAGs, configuration-heavy reporting feature, interactive data pipeline design interfaces, and charting.
● Experience in Java, Spring Boot, and GCP
● In-depth understanding of design patterns, OOPs, and Functional programming.
● Passionate programmer focused on Backend Programming.
Good to have:
● Working knowledge of GCP managed services like Kubernetes, PubSub
Please feel free to share your resume to chandana at revsure.ai
**Based on your experience, the title could vary. We care more about mindset , ownership, and craft than just titles.
Strong Full stack/Backend engineer profile
Mandatory (Experience): Must have 2+ years of hands-on experience as a full stack developer (backend-heavy)
Mandatory (Backend Skills): Must have 1.5+ strong experience in Python, building REST APIs, and microservices-based architectures
Mandatory (Frontend Skills): Must have hands-on experience with modern frontend frameworks (React or Vue) and JavaScript, HTML, and CSS
Mandatory (Database Skills): Must have solid experience working with relational and NoSQL databases such as MySQL, MongoDB, and Redis
Mandatory (Cloud & Infra): Must have hands-on experience with AWS services including EC2, ELB, AutoScaling, S3, RDS, CloudFront, and SNS
Mandatory (DevOps & Infra): Must have working experience with Linux environments, Apache, CI/CD pipelines, and application monitoring
Mandatory (CS Fundamentals): Must have strong fundamentals in Data Structures, Algorithms, OS concepts, and system design
Mandatory (Company) : Product companies (B2B SaaS preferred)
Job Title: Full Stack/ Frontend Developer
Location: Chennai, India
Duration: Full-Time
Position Overview
We are seeking an experienced and motivated Full Stack/ Frontend Developer to join our innovative team. If you are passionate about coding, problem-solving, and building intuitive user interfaces, we would love to hear from you! In this role, you’ll work closely with back-end developers and UX/UI designers to deliver exceptional web applications for the higher education sector.
Key Responsibilities
● Develop and maintain interactive, responsive, and user-friendly front-end applications using Java, JavaScript, and modern frameworks.
● Collaborate with back-end developers to integrate front-end functionality with back-end services.
● Work closely with UX/UI designers to ensure user interfaces are visually appealing, functional, and consistent.
● Participate in code reviews and uphold high-quality coding standards. ● Troubleshoot and resolve front-end issues and optimize performance. ● Stay up-to-date with the latest trends in front-end development.
Required Skills & Experience
● 4+ years of experience in Java-based front-end development.
● Strong experience with HTML, CSS, JavaScript, and front-end frameworks. ● Hands-on experience with Spring, Hibernate, Thymeleaf, Velocity, jQuery, and Java.
● Familiarity with RESTful APIs and front-end integration practices.
● Experience in responsive design and cross-browser compatibility.
● Strong problem-solving skills and attention to detail.
● Ability to collaborate effectively in a team environment with strong communication skills. Preferred Skills
● Experience with modern JavaScript frameworks (React, jQuery plugins, DataTables, etc.).
● Familiarity with version control systems like Git.
● Knowledge of Agile development methodologies.
● Prior experience in higher education technology or education-related software.
Note: Experience with Thymeleaf and Velocity is good to have. Tapestry can be ignored as it is legacy.
About Hudson Data
At Hudson Data, we view AI as both an art and a science. Our cross-functional teams — spanning business leaders, data scientists, and engineers — blend AI/ML and Big Data technologies to solve real-world business challenges. We harness predictive analytics to uncover new revenue opportunities, optimize operational efficiency, and enable data-driven transformation for our clients.
Beyond traditional AI/ML consulting, we actively collaborate with academic and industry partners to stay at the forefront of innovation. Alongside delivering projects for Fortune 500 clients, we also develop proprietary AI/ML products addressing diverse industry challenges.
Headquartered in New Delhi, India, with an office in New York, USA, Hudson Data operates globally, driving excellence in data science, analytics, and artificial intelligence.
⸻
About the Role
We are seeking a Data Analyst & Modeling Specialist with a passion for leveraging AI, machine learning, and cloud analytics to improve business processes, enhance decision-making, and drive innovation. You’ll play a key role in transforming raw data into insights, building predictive models, and delivering data-driven strategies that have real business impact.
⸻
Key Responsibilities
1. Data Collection & Management
• Gather and integrate data from multiple sources including databases, APIs, spreadsheets, and cloud warehouses.
• Design and maintain ETL pipelines ensuring data accuracy, scalability, and availability.
• Utilize any major cloud platform (Google Cloud, AWS, or Azure) for data storage, processing, and analytics workflows.
• Collaborate with engineering teams to define data governance, lineage, and security standards.
2. Data Cleaning & Preprocessing
• Clean, transform, and organize large datasets using Python (pandas, NumPy) and SQL.
• Handle missing data, duplicates, and outliers while ensuring consistency and quality.
• Automate data preparation using Linux scripting, Airflow, or cloud-native schedulers.
3. Data Analysis & Insights
• Perform exploratory data analysis (EDA) to identify key trends, correlations, and drivers.
• Apply statistical techniques such as regression, time-series analysis, and hypothesis testing.
• Use Excel (including pivot tables) and BI tools (Tableau, Power BI, Looker, or Google Data Studio) to develop insightful reports and dashboards.
• Present findings and recommendations to cross-functional stakeholders in a clear and actionable manner.
4. Predictive Modeling & Machine Learning
• Build and optimize predictive and classification models using scikit-learn, XGBoost, LightGBM, TensorFlow, Keras, and H2O.ai.
• Perform feature engineering, model tuning, and cross-validation for performance optimization.
• Deploy and manage ML models using Vertex AI (GCP), AWS SageMaker, or Azure ML Studio.
• Continuously monitor, evaluate, and retrain models to ensure business relevance.
5. Reporting & Visualization
• Develop interactive dashboards and automated reports for performance tracking.
• Use pivot tables, KPIs, and data visualizations to simplify complex analytical findings.
• Communicate insights effectively through clear data storytelling.
6. Collaboration & Communication
• Partner with business, engineering, and product teams to define analytical goals and success metrics.
• Translate complex data and model results into actionable insights for decision-makers.
• Advocate for data-driven culture and support data literacy across teams.
7. Continuous Improvement & Innovation
• Stay current with emerging trends in AI, ML, data visualization, and cloud technologies.
• Identify opportunities for process optimization, automation, and innovation.
• Contribute to internal R&D and AI product development initiatives.
⸻
Required Skills & Qualifications
Technical Skills
• Programming: Proficient in Python (pandas, NumPy, scikit-learn, XGBoost, LightGBM, TensorFlow, Keras, H2O.ai).
• Databases & Querying: Advanced SQL skills; experience with BigQuery, Redshift, or Azure Synapse is a plus.
• Cloud Expertise: Hands-on experience with one or more major platforms — Google Cloud, AWS, or Azure.
• Visualization & Reporting: Skilled in Tableau, Power BI, Looker, or Excel (pivot tables, data modeling).
• Data Engineering: Familiarity with ETL tools (Airflow, dbt, or similar).
• Operating Systems: Strong proficiency with Linux/Unix for scripting and automation.
Soft Skills
• Strong analytical, problem-solving, and critical-thinking abilities.
• Excellent communication and presentation skills, including data storytelling.
• Curiosity and creativity in exploring and interpreting data.
• Collaborative mindset, capable of working in cross-functional and fast-paced environments.
⸻
Education & Certifications
• Bachelor’s degree in Data Science, Computer Science, Statistics, Mathematics, or a related field.
• Master’s degree in Data Analytics, Machine Learning, or Business Intelligence preferred.
• Relevant certifications are highly valued:
• Google Cloud Professional Data Engineer
• AWS Certified Data Analytics – Specialty
• Microsoft Certified: Azure Data Scientist Associate
• TensorFlow Developer Certificate
⸻
Why Join Hudson Data
At Hudson Data, you’ll be part of a dynamic, innovative, and globally connected team that uses cutting-edge tools — from AI and ML frameworks to cloud-based analytics platforms — to solve meaningful problems. You’ll have the opportunity to grow, experiment, and make a tangible impact in a culture that values creativity, precision, and collaboration.
Key Responsibilities:
• Salesforce Administration:
o Manage user setup, roles, profiles, security settings, and permissions.
o Configure and maintain custom objects, record types, fields, page layouts, workflows, process builders, and validation rules.
o Perform regular data management tasks including data import, export, deduplication, and cleaning.
o Ensure platform stability and performance with regular system maintenance.
• Salesforce Development:
o Design and implement custom solutions using Apex, Visualforce, Lightning Components (LWC and AURA), and SOQL
o Develop and integrate APIs to connect Salesforce with other systems.
o Automate processes with Flow Builder, Process Builder, and custom code.
o Optimize Salesforce instance through custom development for better performance and scalability.
• System Enhancements:
o Identify opportunities for automation, efficiency, and improved data quality.
o Implement third-party apps from AppExchange as required.
o Stay up-to-date with Salesforce releases and ensure smooth implementation of new features.
#JobDescription:
Are you a talented and motivated Drupal Developer with a passion for creating amazing digital experiences? We are seeking a skilled Drupal Developer to join our dynamic team. As a Drupal Developer you will have the opportunity to work on exciting projects, collaborate with a talented team, and grow your career in web development.
#KeyResponsibilities:
-> Develop and maintain Drupal-based websites and web applications.
-> Collaborate with cross-functional teams to design and implement web solutions.
-> Write clean, maintainable, and efficient PHP code.
-> Customize and extend Drupal modules and themes.
->Ensure the performance, security, and scalability of Drupal websites.
->Debug and troubleshoot issues as they arise.
->Stay up-to-date with the latest Drupal and web development trends and technologies.
#Qualifications:
-> Bachelor's degree in Computer Science or a related field (preferred).
->1-3 years of professional experience in Drupal development.
->Strong proficiency in PHP, HTML, and JavaScript.
->Experience working with Drupal 8 or 9.
->Knowledge of Drupal theming and site building.
->Familiarity with front-end frameworks and libraries (e.g., React, Vue.js) is a plus.
->Excellent problem-solving and debugging skills.
->Ability to work collaboratively in a team environment.
->Strong communication and interpersonal skills.
->Openness to learning and adapting to new technologies.
#mandatory #skills to have: PHP , HTML, CSS, JavaScript and Drupal
If you are looking for a change this is what we can promise you:
You will have challenging problems to solve
You will have flexibility and autonomy to solve problems and deliver solutions
We will provide a highly collaborative environment with skilled and super friendly teammates
We will fully support you in developing software the right way
We won't burden you with useless policies and procedures
We will provide you the tools you need to do your job right
If that sounds attractive please apply! We'd love to talk to you.
Here are some more details about the position:
You will be a senior level engineer that is part of an agile development team. You will be hands-on writing code and providing technical leadership through training, code reviews, mentoring and coaching. You will work with a project manager, software development engineers, user experience and interface designers, quality assurance engineers and customer stakeholders to deliver high quality, scalable and maintainable software solutions. We expect you to be responsible for the architecture of the applications and making sure the highest quality standards are kept.
Required Qualifications
4+ years experience developing web applications
Expert in building web applications and REST APIs
Experience in Go
Experience in React or other frontend framework
AWS experience - EC2, RDS, etc.
SQL database (Postgres, MySQL, etc.)
Data and System Architecture
Responsibilities
Implement customer requirements by creating high quality source code, database structures and other aspects of software application solutions.
Provide leadership to a team of software engineers and QA engineers to deliver commitments on time with a positive attitude.
Architect solutions to ensure that patterns for scalability, maintainability and performance are developed.
Ensure that the process for software development, testing and deployment is followed.
Ensure that user stories (requirements) have sufficient detail to be implemented properly.
Ensure that the team has sufficient understanding of user stories to provide reliable estimates.
Present training to team members on processes and technical topics.
Work with customer stakeholders and the project manager as needed to build requirements and understanding of software features.
Perform code reviews to ensure that code standards and proper architectural patterns are followed.
Attend daily meetings status meeting with the team.
Make suggestions regarding improvements to the software.
Help software engineers and QA engineers to develop their careers and skills.
Be an expert in the technology stack of the software project.
Desired Qualifications
Serverless technologies
IoC / Dependency Injection
Continuous Integration
Continuous delivery
Git
SourceTree and Bitbucket
Scrum
Project management software - Jira, Rally or similar
What You Will Do:
- Your primary focus will be to implement responsive & high-quality UI and reusable code for CoinSwitch platform, with a particular focus on performance.
- You will help translate requirements and mockups into fully functioning features using frontend technologies.
- You will ensure the technical feasibility of UI/UX designs while maintaining high quality standards.
- You will collaborate timely with designers, product managers and own the front-end layer.
- You will constantly look for optimizing applications to handle high scale.
- You will keep yourself updated with the latest web technologies coming up in the market and suggest ways to implement it in our system.
What You Should Have:
- You should have 5+ years of extensive experience in Javascript.
- You should have 3+ years of experience in ReactNative based Android and IOS apps.
- You have experience writing clean, modular, object- oriented and optimized scripts.
- You are aware of Javascript design patterns and can architect a fairly complex UI.
- You have experience building a product using a mainstream library or framework (ReactNative ecosystem preferred).
- You keep pace with the ever-changing specs of cutting edge front-end technologies.
Good to Have:
- Prior experience working with fin-tech companies










