Big Data Architect

Job Title: Senior Data Engineer

Experience: 8Yrs to 11Yrs

Location: Remote

Notice: Immediate or Max 1Month

Role: Permanent Role

Skill set: Google Cloud Platform, Big Query, Java, Python Programming Language, Airflow, Data flow, Apache Beam.

Experience required:

5 years of experience in software design and development with 4 years of experience in the data engineering field is preferred.

2 years of Hands-on experience in GCP cloud data implementation suites such as Big Query, Pub Sub, Data Flow/Apache Beam, Airflow/Composer, Cloud Storage, etc.

Strong experience and understanding of very large-scale data architecture, solutions, and operationalization of data warehouses, data lakes, and analytics platforms.

Mandatory 1 year of software development skills using Java or Python.

Extensive hands-on experience working with data using SQL and Python.

Must Have: GCP, Big Query, Airflow, Data flow, Python, Java.

GCP knowledge must

Java as programming language(preferred)

Big Query, Pub-Sub, Data Flow/Apache Beam, Airflow/Composer, Cloud Storage,

Python

Communication should be good.

Experience : 3 to 7 Years
Number of Positions : 20

Job Location : Hyderabad

Notice : 30 Days

1. Expertise in building AWS Data Engineering pipelines with AWS Glue -> Athena -> Quick sight

2. Experience in developing lambda functions with AWS Lambda

3. Expertise with Spark/PySpark – Candidate should be hands on with PySpark code and should be able to do transformations with Spark

4. Should be able to code in Python and Scala.

5. Snowflake experience will be a plus

Hadoop and Hive requirements as good to have or understanding of is enough.

We are looking for an outstanding Big Data Engineer with experience setting up and maintaining Data Warehouse and Data Lakes for an Organization. This role would closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Roles and Responsibilities:

• Develop and maintain scalable data pipelines and build out new integrations and processes required for optimal extraction, transformation, and loading of data from a wide variety of data sources using 'Big Data' technologies.
• Develop programs in Scala and Python as part of data cleaning and processing.
• Assemble large, complex data sets that meet functional / non-functional business requirements and fostering data-driven decision making across the organization.  
• Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
• Implement processes and systems to validate data, monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
• Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Provide high operational excellence guaranteeing high availability and platform stability.
• Closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Skills:

• Experience with Big Data pipeline, Big Data analytics, Data warehousing.
• Experience with SQL/No-SQL, schema design and dimensional data modeling.
• Strong understanding of Hadoop Architecture, HDFS ecosystem and eexperience with Big Data technology stack such as HBase, Hadoop, Hive, MapReduce.
• Experience in designing systems that process structured as well as unstructured data at large scale.
• Experience in AWS/Spark/Java/Scala/Python development.
• Should have Strong skills in PySpark (Python & SPARK). Ability to create, manage and manipulate Spark Dataframes. Expertise in Spark query tuning and performance optimization.
• Experience in developing efficient software code/frameworks for multiple use cases leveraging Python and big data technologies.
• Prior exposure to streaming data sources such as Kafka.
• Should have knowledge on Shell Scripting and Python scripting.
• High proficiency in database skills (e.g., Complex SQL), for data preparation, cleaning, and data wrangling/munging, with the ability to write advanced queries and create stored procedures.
• Experience with NoSQL databases such as Cassandra / MongoDB.
• Solid experience in all phases of Software Development Lifecycle - plan, design, develop, test, release, maintain and support, decommission.
• Experience with DevOps tools (GitHub, Travis CI, and JIRA) and methodologies (Lean, Agile, Scrum, Test Driven Development).
• Experience building and deploying applications on on-premise and cloud-based infrastructure.
• Having a good understanding of machine learning landscape and concepts. 

Qualifications and Experience:

Engineering and post graduate candidates, preferably in Computer Science, from premier institutions with proven work experience as a Big Data Engineer or a similar role for 3-5 years.

Certifications:

Good to have at least one of the Certifications listed here:

    AZ 900 - Azure Fundamentals

    DP 200, DP 201, DP 203, AZ 204 - Data Engineering

    AZ 400 - Devops Certification

SpringML is looking to hire a top-notch Senior  Data Engineer who is passionate about working with data and using the latest distributed framework to process large dataset. As an Associate Data Engineer, your primary role will be to design and build data pipelines. You will be focused on helping client projects on data integration, data prep and implementing machine learning on datasets. In this role, you will work on some of the latest technologies, collaborate with partners on early win, consultative approach with clients, interact daily with executive leadership, and help build a great company. Chosen team members will be part of the core team and play a critical role in scaling up our emerging practice.

RESPONSIBILITIES:

• Ability to work as a member of a team assigned to design and implement data integration solutions.
• Build Data pipelines using standard frameworks in Hadoop, Apache Beam and other open-source solutions.
• Learn quickly – ability to understand and rapidly comprehend new areas – functional and technical – and apply detailed and critical thinking to customer solutions.
• Propose design solutions and recommend best practices for large scale data analysis

SKILLS:

• B.tech  degree in computer science, mathematics or other relevant fields.
• 4+years of experience in ETL, Data Warehouse, Visualization and building data pipelines.
• Strong Programming skills – experience and expertise in one of the following: Java, Python, Scala, C.
• Proficient in big data/distributed computing frameworks such as Apache,Spark, Kafka,
• Experience with Agile implementation methodologies

We are looking for someone with 6+ years of relevant experience working on problems in NLP or Computer Vision with a Master's degree (PhD preferred).

We are looking for BE/BTech graduates (2018/2019 pass out) who want to build their career as Data Engineer covering technologies like Hadoop, NoSQL, RDBMS, Spark, Kafka, Hive, ETL, MDM & Data Quality. You should be willing to learn, explore, experiment, develop POCs/Solutions using these technologies with guidance and support from highly experienced Industry Leaders. You should be passionate about your work and willing to go extra mile to achieve results.

We are looking for candidates who believe in commitment and in building strong relationships. We need people who are passionate about solving problems through software and are flexible.

Required Experience, Skills and Qualifications

Passionate to learn and explore new technologies

Any RDBMS experience (SQL Server/Oracle/MySQL)

Any ETL tool experience (Informatica/Talend/Kettle/SSIS)

Understanding of Big Data technologies

Good Communication Skills

Excellent Mathematical / Logical / Reasoning Skills