11+ Amazon VPC Jobs in Hyderabad | Amazon VPC Job openings in Hyderabad

Technical Skills:

• Ability to understand and translate business requirements into design.
• Proficient in AWS infrastructure components such as S3, IAM, VPC, EC2, and Redshift.
• Experience in creating ETL jobs using Python/PySpark.
• Proficiency in creating AWS Lambda functions for event-based jobs.
• Knowledge of automating ETL processes using AWS Step Functions.
• Competence in building data warehouses and loading data into them.

Responsibilities:

• Understand business requirements and translate them into design.
• Assess AWS infrastructure needs for development work.
• Develop ETL jobs using Python/PySpark to meet requirements.
• Implement AWS Lambda for event-based tasks.
• Automate ETL processes using AWS Step Functions.
• Build data warehouses and manage data loading.
• Engage with customers and stakeholders to articulate the benefits of proposed solutions and frameworks.

Key Responsibilities:

● Architect and Design Data Systems: Design and develop scalable data pipelines and systems to manage large volumes of structured and unstructured data.

● DataPipeline Development: Develop scalable end-to-end ETL (Extract, Transform, Load) data pipelines to ingest data from various diverse sources, perform transformations, and load data into target systems to meet both functional and non-functional requirements.

● DataModeling and Optimization: Develop and optimize data models (including relational Data) for efficient storage and retrieval, ensuring data quality and integrity.

● Technical Oversight: Manage technical scope, ensuring alignment with business objectives and delivering high-quality solutions. Provided required technical guidance and oversight to various cross-functional teams and team members on databases like PostgreSQL/ Vector DB and various queries, functions, and datasets.

● Configuration Management: Manage the configurations for various tools and databases like postgresDB, vectorDB, etc. for both on-premises and cloud platforms for optimized performance and redundancy

● Performance Tuning: Monitor and optimize data pipelines, SQL/queries, functions, procedures, and triggers for performance and scalability.

● Stakeholder Collaboration: Work closely with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and deliver solutions ensuring alignment with organizational goals.

● Promoting Best Practices: Define and promote reusable, scalable, and maintainable solutions, emphasizing software engineering best practices and continuous improvement.

● Communication: Communicate effectively at all levels about the importance of solution design, conducting technical walk-throughs to ensure a clear understanding of system architecture.

● Continuous Improvement: Work with data and analytics experts to enhance functionality in data systems and products, driving improvements and growth within the data team.

● CloudExperience: Experience of working with AWS RDS databases is preferred.

Required Experience, Skills & Qualifications:

● Experience: 5-7 years in High-Performance Data Products or Data Systems as a Data Architect/Engineer.

● Technical Skills: Proficiency in orchestration tools like Airflow. Proficiency in SQL and relational databases (e.g., PostgreSQL, Aurora) and NoSQL databases (e.g., MongoDB, Cassandra).

● Knowledge of Vector databases like pgvector etc.

● Hands-on experience with cloud platforms (e.g., AWS, Azure, GCP) and containerization technologies (e.g., Docker, Kubernetes).

● Experience with agile development methodologies (e.g., Scrum, Kanban).

● Software Engineering: Proficient in software engineering best practices, unit testing, integration testing, and tools like Git and Docker. ● Security and Compliance: Familiarity with security compliances and design practices.

● Communication: Exceptional interpersonal, analytical, and communication skills. Ability to explain and discuss concepts with colleagues and teams effectively.

● CI/CDPipeline: Fully adhere to and promote an entire CI/CD pipeline. ● APIDevelopment: Familiarity with API development and data formats like JSON/XML.

Job Description

Mandatory Requirements 

• Experience in AWS Glue

• Experience in Apache Parquet 

• Proficient in AWS S3 and data lake 

• Knowledge of Snowflake

• Understanding of file-based ingestion best practices.

• Scripting language - Python & pyspark

CORE RESPONSIBILITIES

• Create and manage cloud resources in AWS 

• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 

• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 

• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 

• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.

• Define process improvement opportunities to optimize data collection, insights and displays.

• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 

• Identify and interpret trends and patterns from complex data sets 

• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 

• Key participant in regular Scrum ceremonies with the agile teams  

• Proficient at developing queries, writing reports and presenting findings 

• Mentor junior members and bring best industry practices.

QUALIFICATIONS

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 

• Strong background in math, statistics, computer science, data science or related discipline

• Advanced knowledge one of language: Java, Scala, Python, C# 

• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  

• Proficient with

• Data mining/programming tools (e.g. SAS, SQL, R, Python)

• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)

• Data visualization (e.g. Tableau, Looker, MicroStrategy)

• Comfortable learning about and deploying new technologies and tools. 

• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 

• Good written and oral communication skills and ability to present results to non-technical audiences 

• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus: 

• AWS certification

• Spark Streaming 

• Kafka Streaming / Kafka Connect 

• ELK Stack 

• Cassandra / MongoDB 

• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Skills and requirements

• Experience analyzing complex and varied data in a commercial or academic setting.
• Desire to solve new and complex problems every day.
• Excellent ability to communicate scientific results to both technical and non-technical team members.

Desirable

• A degree in a numerically focused discipline such as, Maths, Physics, Chemistry, Engineering or Biological Sciences..
• Hands on experience on Python, Pyspark, SQL
• Hands on experience on building End to End Data Pipelines.
• Hands on Experience on Azure Data Factory, Azure Data Bricks, Data Lake - added advantage
• Hands on Experience in building data pipelines.
• Experience with Bigdata Tools, Hadoop, Hive, Sqoop, Spark, SparkSQL
• Experience with SQL or NoSQL databases for the purposes of data retrieval and management.
• Experience in data warehousing and business intelligence tools, techniques and technology, as well as experience in diving deep on data analysis or technical issues to come up with effective solutions.
• BS degree in math, statistics, computer science or equivalent technical field.
• Experience in data mining structured and unstructured data (SQL, ETL, data warehouse, Machine Learning etc.) in a business environment with large-scale, complex data sets.
• Proven ability to look at solutions in unconventional ways. Sees opportunities to innovate and can lead the way.
• Willing to learn and work on Data Science, ML, AI.

Responsibilities Description:

Responsible for the development and implementation of machine learning algorithms and techniques to solve business problems and optimize member experiences. Primary duties may include are but not limited to: Design machine learning projects to address specific business problems determined by consultation with business partners. Work with data-sets of varying degrees of size and complexity including both structured and unstructured data. Piping and processing massive data-streams in distributed computing environments such as Hadoop to facilitate analysis. Implements batch and real-time model scoring to drive actions. Develops machine learning algorithms to build customized solutions that go beyond standard industry tools and lead to innovative solutions. Develop sophisticated visualization of analysis output for business users.

Experience Requirements:

BS/MA/MS/PhD in Statistics, Computer Science, Mathematics, Machine Learning, Econometrics, Physics, Biostatistics or related Quantitative disciplines. 2-4 years of experience in predictive analytics and advanced expertise with software such as Python, or any combination of education and experience which would provide an equivalent background. Experience in the healthcare sector. Experience in Deep Learning strongly preferred.

Required Technical Skill Set:

• Full cycle of building machine learning solutions,

o   Understanding of wide range of algorithms and their corresponding problems to solve

o   Data preparation and analysis

o   Model training and validation

o   Model application to the problem

• Experience using the full open source programming tools and utilities
• Experience in working in end-to-end data science project implementation.
• 2+ years of experience with development and deployment of Machine Learning applications
• 2+ years of experience with NLP approaches in a production setting
• Experience in building models using bagging and boosting algorithms
• Exposure/experience in building Deep Learning models for NLP/Computer Vision use cases preferred
• Ability to write efficient code with good understanding of core Data Structures/algorithms is critical
• Strong python skills following software engineering best practices
• Experience in using code versioning tools like GIT, bit bucket
• Experience in working in Agile projects
• Comfort & familiarity with SQL and Hadoop ecosystem of tools including spark
• Experience managing big data with efficient query program good to have
• Good to have experience in training ML models in tools like Sage Maker, Kubeflow etc.
• Good to have experience in frameworks to depict interpretability of models using libraries like Lime, Shap etc.
• Experience with Health care sector is preferred
• MS/M.Tech or PhD is a plus

Required skill

• Around 6- 8.5 years of experience and around 4+ years in AI / Machine learning space
• Extensive experience in designing large scale machine learning solution for the ML use case,  large scale deployments and establishing continues automated improvement / retraining framework.
• Strong experience in Python and Java is required.
• Hands on experience on Scikit-learn, Pandas, NLTK
• Experience in Handling of Timeseries data and associated techniques like Prophet, LSTM
• Experience in Regression, Clustering, classification algorithms
• Extensive experience in buildings traditional Machine Learning SVM, XGBoost, Decision tree and Deep Neural Network models like RNN, Feedforward is required.
• Experience in AutoML like TPOT or other
• Must have strong hands on experience in Deep learning frameworks like Keras, TensorFlow or PyTorch 
• Knowledge of Capsule Network or reinforcement learning, SageMaker is a desirable skill
• Understanding of Financial domain is desirable skill

 Responsibilities 

• Design and implementation of solutions for ML Use cases
• Productionize System and Maintain those
• Lead and implement data acquisition process for ML work
• Learn new methods and model quickly and utilize those in solving use cases