
Data Engineer
Mandatory Requirements
- Experience in AWS Glue
- Experience in Apache Parquet
- Proficient in AWS S3 and data lake
- Knowledge of Snowflake
- Understanding of file-based ingestion best practices.
- Scripting language - Python & pyspark
CORE RESPONSIBILITIES
- Create and manage cloud resources in AWS
- Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies
- Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform
- Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations
- Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
- Define process improvement opportunities to optimize data collection, insights and displays.
- Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible
- Identify and interpret trends and patterns from complex data sets
- Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders.
- Key participant in regular Scrum ceremonies with the agile teams
- Proficient at developing queries, writing reports and presenting findings
- Mentor junior members and bring best industry practices
QUALIFICATIONS
- 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales)
- Strong background in math, statistics, computer science, data science or related discipline
- Advanced knowledge one of language: Java, Scala, Python, C#
- Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake
- Proficient with
- Data mining/programming tools (e.g. SAS, SQL, R, Python)
- Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
- Data visualization (e.g. Tableau, Looker, MicroStrategy)
- Comfortable learning about and deploying new technologies and tools.
- Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.
- Good written and oral communication skills and ability to present results to non-technical audiences
- Knowledge of business intelligence and analytical tools, technologies and techniques.
Familiarity and experience in the following is a plus:
- AWS certification
- Spark Streaming
- Kafka Streaming / Kafka Connect
- ELK Stack
- Cassandra / MongoDB
- CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Similar jobs
DocNexus is revolutionizing the global medical affairs & commercial ecosystem with search. We provide a next-generation data platform that simplifies searching through millions of insights, publications, clinical trials, payments, and social media data within seconds to identify healthcare professionals (HCPs), products, manufacturers, and healthcare systems. Leveraging AI-powered Knowledge Graphs, DocNexus assists life science organizations in finding the right key opinion leaders (KOL/DOLs) who play a crucial role in developing and bringing life-saving pharmaceutical products and medical devices to market. Backed by industry leaders such as Techstars, JP Morgan, Mass Challenge, and recognized as one of the Top 200 Most Innovative Startups by TechCrunch Disrupt, we are committed to transforming healthcare insights. We are seeking a skilled and passionate DevOps Engineer to join our dynamic team and contribute to the efficient development, deployment, and maintenance of our platform.
We are looking for a visionary Sr. Full Stack Engineering Lead who is passionate about building and leading our technology department. The ideal candidate will have a solid technical background and experience in leading a team to drive innovation and growth. As Engineering Lead, you will oversee the development and dissemination of technology for external customers, vendors, and other clients to help improve and increase business.
Leadership and Strategy:
- Lead the engineering team and make strategic decisions regarding the technology stack, project management, and resource allocation.
- Establish the company’s technical vision and lead all aspects of technological development.
Development:
- Develop and maintain the front-end and back-end of web applications.
- Ensure the performance, quality, and responsiveness of applications.
- Collaborate with a team to define, design, and ship new features.
Maintenance and Optimization:
- Maintain code integrity and organization.
- Identify and correct bottlenecks and fix bugs.
- Continually work on optimizing the performance of different applications.
Security: Ensure the security of the web applications by integrating security best practices.
- Regularly update the system to protect against vulnerabilities.
Innovation:
- Research and implement new technologies and frameworks that can improve the performance and user experience of the platform.
- Stay informed on emerging technologies and trends that can potentially impact the company's products and services.
Collaboration and Communication:
- Work closely with other departments to understand their needs and translate them into technical solutions.
- Communicate technology strategy to partners, management, investors, and employees.
Project Management:
- Oversee and support project planning, deadlines, and progress.
- Ensure that the technology standards and best practices are maintained across the organization.
Mentoring and Team Building:
- Foster a culture of innovation and excellence within the technology team.
- Mentor and guide the professional and technical development of team members.
Front-End Development:
- HTML/CSS: For structuring and styling the web pages.
- JavaScript/TypeScript: Core scripting language, along with frameworks like Angular, React, or Vue.js for dynamic and responsive user interfaces.
Back-End Development:
- Python: Using frameworks like Django or Flask for server-side logic.
- Node.js: JavaScript runtime environment for building scalable network applications.
- Ruby on Rails: A server-side web application framework written in Ruby.
Database Management:
- SQL Databases: MySQL, PostgreSQL for structured data storage.
- NoSQL Databases: MongoDB, Cassandra for unstructured data or specific use cases.
Server Management:
- Nginx or Apache: For server and reverse proxy functionalities.
- Docker: For containerizing applications and ensuring consistency across multiple development and release cycles.
- Kubernetes: For automating deployment, scaling, and operations of application containers.
DevOps and Continuous Integration/Continuous Deployment (CI/CD):
- Git: For version control.
- Jenkins, Travis CI, or CircleCI: For continuous integration and deployment.
- Ansible, Chef, or Puppet: For configuration management.
Cloud Services:
- AWS: For various cloud services like computing, database storage, content delivery, etc.
- Serverless Frameworks: Such as AWS Lambda or Google Cloud Functions for running code without provisioning or managing servers.
Security:
- OAuth, JWT: For secure authentication mechanisms.
- SSL/TLS: For secure data transmission.
- Various Encryption Techniques: To safeguard sensitive data.
Performance Monitoring and Testing:
- Selenium, Jest, or Mocha: For automated testing.
- New Relic or Datadog: For performance monitoring.
Data Science and Analytics:
- Python Libraries: NumPy, Pandas, or SciPy for data manipulation and analysis.
- Machine Learning Frameworks: TensorFlow, PyTorch for implementing machine learning models.
Other Technologies:
- GraphQL: For querying and manipulating data efficiently.
- WebSockets: For real-time bi-directional communication between web clients and servers.
Key Responsibilities:
- Rewrite existing APIs in NodeJS.
- Remodel the APIs into Micro services-based architecture.
- Implement a caching layer wherever possible.
- Optimize the API for high performance and scalability.
- Write unit tests for API Testing.
- Automate the code testing and deployment process.
Skills Required:
- At least 3 years of experience developing Backends using NodeJS — should be well versed with its asynchronous nature & event loop, and know its quirks and workarounds.
- Excellent hands-on experience using MySQL or any other SQL Database.
- Good knowledge of MongoDB or any other NoSQL Database.
- Good knowledge of Redis, its data types, and their use cases.
- Experience with graph-based databases like GraphQL and Neo4j.
- Experience developing and deploying REST APIs.
- Good knowledge of Unit Testing and available Test Frameworks.
- Good understanding of advanced JS libraries and frameworks.
- Experience with Web sockets, Service Workers, and Web Push Notifications.
- Familiar with NodeJS profiling tools.
- Proficient understanding of code versioning tools such as Git.
- Good knowledge of creating and maintaining DevOps infrastructure on cloud platforms.
- Should be a fast learner and a go-getter — without any fear of trying out new things Preferences.
- Experience building a large-scale social or location-based app.
- Minimum 2.5 years of experience as a Python Developer.
- Minimum 2.5 years of experience in any framework like Django/Flask/Fast API
- Minimum 2.5 years of experience in SQL/ Postgress
- Minimum 2.5 years of experience in Git/Gitlab/Bit-Bucket
- Minimum 2+ years of experience in deployment (CICD with Jenkins)
- Minimum 2.5 years of experience in any cloud like AWS/GCP/Azure
Responsibilities:
- Understanding requirements from the product manager and translating to technical specifications
- DB schema development
- API development (structuring REST endpoints + real-time endpoints like socket.io integration)
- Integrating 3rd party services for payments, email, whatsapp, and other such services
- Optimizing our current architecture & processes for scale (we expect usage to go up)
- Guiding junior developers on best coding practices
Requirements:
- Experience in at least one Python-based Web Framework like Flask or Django
- Experience in SQL databases & API design
- Experience with Docker
- Hands on experience in full lifecycle development and deployment of a platform/system
- Being a self-starter with the ability to thrive in ambiguous and high pressure scenarios working towards a common goal with other teams
- Bachelor’s degree in computer science, engineering, or a related field.
- At-least more than 1+ Years of Experience in a similar role would be desired.
- Excellent technical, diagnostic, and troubleshooting skills.
- Strong leadership skills to drive good coding and design practices across multiple engineering teams
- Willingness to build professional relationships with staff and clients.
- Excellent communication, motivational, and interpersonal skills.
- Expertise in architecting, building, and maintaining ultra-low latency, cost-efficient systems in cloud environments
- Excellent track record in modernizing cloud-based applications using micro services, containers, and other architectures
- Experience and working knowledge in building large-scale, data-intensive Text search applications using Solr / Elastic Search is a Must
- Experience and working knowledge of AI/ML/ML Ops with respect to building large-scale, data-intensive applications would be a Plus
- Experience and working knowledge in building Java based Spring Boot micro-services integrated with any messaging framework
- Driven, Highly-motivated and passionate towards development & innovation
- Determining project requirements and developing work schedules for the team.
- Delegating tasks and achieving daily, weekly, and monthly goals.
- Liaising with team members, management, and clients to ensure projects are completed to standard.
- Identifying risks and forming contingency plans as soon as possible.
- Analyzing existing operations and scheduling training sessions and meetings to discuss improvements.
- Keeping up-to-date with industry trends and developments.
- Updating work schedules and performing troubleshooting as required.
- Motivating staff and creating a space where they can ask questions and voice their concerns.
- Being transparent with the team about challenges, failures, and successes.
- Writing progress reports and delivering presentations to the relevant stakeholders.
- Extremely hands-on in delivering Development and R&D tasks
- Design, plan and perform dev-analysis to determine effort estimates on every sprint for the team
- Running Demo's, ensuring thorough documentation of the features built
- Identify & Plan upgrades to technologies & frameworks from time-to-time
📌 Knowledge of development technologies including HTML, JavaScript, JQuery, AJAX, and CSS.
📌 Knowledge of Microsoft.NET/ C#, If not Ready to learn
📌 Comprehensive understanding of object-oriented and service-oriented application development techniques and theories.
📌 Internally motivated, able to work proficiently in a team environment.
📌 The candidate should be ready to learn multiple technologies.
- B2B
- B2C( International IT Sales )
- Knowledge of products and technologies and market conditions.
- Excellent communications











