Spark Scala Developer

Location: Pune

Required Skills : Scala, Python, Data Engineering, AWS, Cassandra/AstraDB, Athena, EMR, Spark/Snowflake

Azure DE

Primary Responsibilities -

• Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
• Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure Create data models for analytics purposes
• Utilizing Azure Data Factory or comparable technologies, create and maintain ETL (Extract, Transform, Load) operations
• Use Azure Data Factory and Databricks to assemble large, complex data sets
• Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
• Ensure data security and compliance
• Collaborate with data engineers, and other stakeholders to understand requirements and translate them into scalable and reliable data platform architectures

Required skills:

• Blend of technical expertise, analytical problem-solving, and collaboration with cross-functional teams
• Azure DevOps
• Apache Spark, Python
• SQL proficiency
• Azure Databricks knowledge
• Big data technologies

The DEs should be well versed in coding, spark core and data ingestion using Azure. Moreover, they need to be decent in terms of communication skills. They should also have core Azure DE skills and coding skills (pyspark, python and SQL).

Out of the 7 open demands, 5 of The Azure Data Engineers should have minimum 5 years of relevant Data Engineering experience.

Software Developer

Roles and Responsibilities

• Apply knowledge set to fetch data from multiple online sources, cleanse it and build APIs on top of it
• Develop a deep understanding of our vast data sources on the web and know exactly how, when, and which data to scrap, parse and store
• We're looking for people who will naturally take ownership of data products and who can bring a project all the way from a fast prototype to production.

Desired Candidate Profile

• At Least 1-2 years of experience
• Strong coding experience in Python (knowledge of Javascripts is a plus)
• Strong knowledge of scraping frameworks in Python (Request, Beautiful Soup)
• Experience with SQL and NoSQL databases
• Knowledge in Version Control tools like GIT.
• Good understanding and hands-on with scheduling and managing tasks with cron.

Nice to have:

• Experience of having worked with elastic search
• Experience with multi-processing, multi-threading, and AWS/Azure is a plus
• Experience with web crawling is a plus
• Deploy server/related components to staging, live environments.
• Experience with cloud environments like AWS,etc as well as cloud solutions like Docker,Lambda, etc
• Experience in DevOps and related practices to improve development lifecycle, continuous delivery with high quality is an advantage.
• Compile and analyze data, processes, and codes to troubleshoot problems and identify areas for improvement

• Augmenting, improving, redesigning, and/or re-implementing Dolat's low-latency/high-throughput production trading environment, which collects data from and disseminates orders to exchanges around the world
• Optimizing this platform by using network and systems programming, as well as other advanced techniques
• Developing systems that provide easy access to historical market data and trading simulations
• Building risk-management and performance-tracking tools
• Shaping the future of Dolat through regular interviewing and infrequent campus recruiting trips
• Implementing domain-optimized data structures
• Learn and internalize the theories behind current trading system
• Participate in the design, architecture and implementation of automated trading systems
• Take ownership of system from design through implementation

EXP-Developer-4 to 12 years

         Must have low-level design and development skills.  Should able to design a solution for given use cases. 

• Agile delivery-  Person must able to show design and code on a daily basis
• Must be an experienced PySpark developer and  Scala coding.   Primary skill is PySpark
• Must have experience in designing job orchestration, sequence, metadata design, Audit trail, dynamic parameter passing and error/exception handling
• Good experience with unit, integration and UAT support
• Able to design and code reusable components and functions
• Should able to review design, code & provide review comments with justification
• Zeal to learn new tool/technologies and adoption
• Good to have experience with Devops and CICD