Data Engineer

Bachelor’s degree in computer science or other technical/scientific discipline

● Solid experience with cloud environments like Azure, GCP, AWS 

● Proficient with CI/CD tools such as CircleCI, Travis, Jenkins, Ansible, Terraform, Puppet, Chef 

● Strong expertise in Serverless architecture, Kubernetes cluster, and service mesh. 

● Strong hands-on experience with monitoring tools such as Prometheus 

● Ability to program with one or more high-level languages, such as Python, Java, Ruby, and JavaScript 

● Familiar with Git 

● Experience with performance monitoring and optimization. Ideally worked on large scale e-commerce projects 

The Sr. Analytics Engineer would provide technical expertise in needs identification, data modeling, data movement, and transformation mapping (source to target), automation and testing strategies, translating business needs into technical solutions with adherence to established data guidelines and approaches from a business unit or project perspective.

Understands and leverages best-fit technologies (e.g., traditional star schema structures, cloud, Hadoop, NoSQL, etc.) and approaches to address business and environmental challenges.

Provides data understanding and coordinates data-related activities with other data management groups such as master data management, data governance, and metadata management.

Actively participates with other consultants in problem-solving and approach development.

Responsibilities :

Provide a consultative approach with business users, asking questions to understand the business need and deriving the data flow, conceptual, logical, and physical data models based on those needs.

Perform data analysis to validate data models and to confirm the ability to meet business needs.

Assist with and support setting the data architecture direction, ensuring data architecture deliverables are developed, ensuring compliance to standards and guidelines, implementing the data architecture, and supporting technical developers at a project or business unit level.

Coordinate and consult with the Data Architect, project manager, client business staff, client technical staff and project developers in data architecture best practices and anything else that is data related at the project or business unit levels.

Work closely with Business Analysts and Solution Architects to design the data model satisfying the business needs and adhering to Enterprise Architecture.

Coordinate with Data Architects, Program Managers and participate in recurring meetings.

Help and mentor team members to understand the data model and subject areas.

Ensure that the team adheres to best practices and guidelines.

Requirements :

- Strong working knowledge of at least 3 years of Spark, Java/Scala/Pyspark, Kafka, Git, Unix / Linux, and ETL pipeline designing.

- Experience with Spark optimization/tuning/resource allocations

- Excellent understanding of IN memory distributed computing frameworks like Spark and its parameter tuning, writing optimized workflow sequences.

- Experience of relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Redshift, Bigquery, Cassandra, etc).

- Familiarity with Docker, Kubernetes, Azure Data Lake/Blob storage, AWS S3, Google Cloud storage, etc.

- Have a deep understanding of the various stacks and components of the Big Data ecosystem.

- Hands-on experience with Python is a huge plus

·      Core responsibilities to include analyze business requirements and designs for accuracy and completeness. Develops and maintains relevant product.

·      BlueYonder is seeking a Senior/Principal Architect in the Data Services department (under Luminate Platform ) to act as one of key technology leaders to build and manage BlueYonder’ s technology assets in the Data Platform and Services.

·      This individual will act as a trusted technical advisor and strategic thought leader to the Data Services department. The successful candidate will have the opportunity to lead, participate, guide, and mentor other people in the team on architecture and design in a hands-on manner. You are responsible for technical direction of Data Platform. This position reports to the Global Head, Data Services and will be based in Bangalore, India.

·      Core responsibilities to include Architecting and designing (along with counterparts and distinguished Architects) a ground up cloud native (we use Azure) SaaS product in Order management and micro-fulfillment

·      The team currently comprises of 60+ global associates across US, India (COE) and UK and is expected to grow rapidly. The incumbent will need to have leadership qualities to also mentor junior and mid-level software associates in our team. This person will lead the Data platform architecture – Streaming, Bulk with Snowflake/Elastic Search/other tools

Our current technical environment:

·      Software: Java, Springboot, Gradle, GIT, Hibernate, Rest API, OAuth , Snowflake

·      • Application Architecture: Scalable, Resilient, event driven, secure multi-tenant Microservices architecture

·      • Cloud Architecture: MS Azure (ARM templates, AKS, HD insight, Application gateway, Virtue Networks, Event Hub, Azure AD)

·      Frameworks/Others: Kubernetes, Kafka, Elasticsearch, Spark, NOSQL, RDBMS, Springboot, Gradle GIT, Ignite

Experience:

The candidate should have about 2+ years of experience with design and development in Java/Scala. Experience in algorithm, data-structure, database and distributed System is mandatory.

Required Skills:

Mandatory: -

1. Core Java or Scala
2. Experience in Big Data, Spark
3. Extensive experience in developing spark job. Should possess good Oops knowledge and be aware of enterprise application design patterns.
4. Should have the ability to analyze, design, develop and test complexity of spark job.
5. Working knowledge of Unix/Linux.
6. Hands on experience in Spark, creating RDD, applying operation - transformation-action

Good To have: -

1. Python
2. Spark streaming
3. Py Spark
4. Azure/AWS Cloud Knowledge of Data Storage and Compute side

Your Opportunity

• Own and drive business features into tech requirements
• Design & develop large scale real time server side systems
• Quickly create quality prototypes
• Staying updated on emerging technologies
• Ensuring that all deliverables adhere to our world class standards
• Promote coding best practices
• Mentor and develop junior developers in the team

Required Experience:

• 4+ years of relevant experience as described below
• Excellent grasp of Core Java, Multi Threading and OO design patterns
• Experience with Scala, functional, reactive programming and Akka/Play is a plus
• Excellent understanding of data structures and algorithms
• Solid grasp of large scale distributed real time systems
• Prior experience on building a scalable and resilient micro service
• Solid understanding of relational databases, NoSQL databases and Caching systems
• Good understanding of Big Data technologies such as Spark, Hadoop is a plus
• Experience on one of AWS, Azure or GCP

Who you are :

• You have excellent and effective communication and collaborative skills
• You love problem solving
• You stay up to date with the latest technologies and then apply them in real life
• You love paying attention to detail
• You thrive in meeting tight deadlines and prioritising workloads
• Ability to collaborate across multiple functions

Education:

Bachelor’s degree in Engineering or equivalent experience within the field

Spark / Scala experience should be more than 2 years.

Combination with Java & Scala is fine or we are even fine with Big Data Developer with strong Core Java Concepts. - Scala / Spark Developer.

Strong proficiency Scala on Spark (Hadoop) - Scala + Java is also preferred

Complete SDLC process and Agile Methodology (Scrum)

Version control / Git