Python Developer

• You're proficient in AI/Machine learning latest technologies  
• You're proficient in GPT-3 based algorithms 
• You have a passion for writing code as well as understanding and crafting the ways systems interact
• You believe in the benefits of agile processes and shipping code often
• You are pragmatic and work to coalesce requirements into reasonable solutions that provide value

• Deploy well-tested, maintainable and scalable software solutions
• Take end-to-end ownership of the technology stack and product 
• Collaborate with other engineers to architect scalable technical solutions
• Embrace and improve our standards and processes to reduce friction and unlock efficiency

Required Experience

· 3+ years of relevant technical experience as a data analyst role

· Intermediate / expert skills with SQL and basic statistics

· Experience in Advance SQL

· Python programming- Added advantage

· Strong problem solving and structuring skills

· Automation in connecting various sources to the data and representing it through various dashboards

· Excellent with Numbers and communicate data points through various reports/templates

· Ability to communicate effectively internally and outside Data Analytics team

· Proactively take up work responsibilities and take adhocs as and when needed

· Ability and desire to take ownership of and initiative for analysis; from requirements clarification to deliverable

· Strong technical communication skills; both written and verbal

· Ability to understand and articulate the "big picture" and simplify complex ideas

· Ability to identify and learn applicable new techniques independently as needed

· Must have worked with various Databases (Relational and Non-Relational) and ETL processes

· Must have experience in handling large volume and data and adhere to optimization and performance standards

· Should have the ability to analyse and provide relationship views of the data from different angles

· Must have excellent Communication skills (written and oral).

· Knowing Data Science is an added advantage

Required Skills

MYSQL, Advanced Excel, Tableau, Reporting and dashboards, MS office, VBA, Analytical skills

Preferred Experience

· Strong understanding of relational database MY SQL etc.

· Prior experience working remotely full-time

· Prior Experience working in Advance SQL

· Experience with one or more BI tools, such as Superset, Tableau etc.

· High level of logical and mathematical ability in Problem Solving

Internshala is a dot com business with the heart of dot org.

We are a technology company on a mission to equip students with relevant skills & practical exposure through internships, fresher jobs, and online trainings. Imagine a world full of freedom and possibilities. A world where you can discover your passion and turn it into your career. A world where your practical skills matter more than your university degree. A world where you do not have to wait till 21 to taste your first work experience (and get a rude shock that it is nothing like you had imagined it to be). A world where you graduate fully assured, fully confident, and fully prepared to stake a claim on your place in the world.

At Internshala, we are making this dream a reality!

👩🏻‍💻 Your responsibilities would include-

• Designing, implementing, testing, deploying, and maintaining stable, secure, and scalable data engineering solutions and pipelines in support of data and analytics projects, including integrating new sources of data into our central data warehouse, and moving data out to applications and affiliates
• Developing analytical tools and programs that can help in Analyzing and organizing raw data
• Evaluating business needs and objectives
• Conducting complex data analysis and report on results
• Collaborating with data scientists and architects on several projects
• Maintaining reliability of the system and being on-call for mission-critical systems
• Performing infrastructure cost analysis and optimization
• Generating architecture recommendations and the ability to implement them
• Designing, building, and maintaining data architecture and warehousing using AWS services.
• ETL optimization, designing, coding, and tuning big data processes using Apache Spark, R, Python, C#, and/or similar technologies.
• Disaster recovery planning and implementation when it comes to ETL and data-related services
• Define actionable KPIs and configure monitoring/alerting

🍒 You will get-

• A chance to build and lead an awesome team working on one of the best recruitment and online trainings products in the world that impact millions of lives for the better
• Awesome colleagues & a great work environment
• Loads of autonomy and freedom in your work

💯 You fit the bill if-

• You have the zeal to build something from scratch
• You have experience in Data engineering and infrastructure work for analytical and machine learning processes.
• You have experience in a Linux environment and familiarity with writing shell scripts using Python or any other scripting language
• You have 3-5 years of experience as a Data Engineer or similar software engineering role

• 5+ years of industry experience in administering (including setting up, managing, monitoring) data processing pipelines (both streaming and batch) using frameworks such as Kafka Streams, Py Spark, and streaming databases like druid or equivalent like Hive
• Strong industry expertise with containerization technologies including kubernetes (EKS/AKS), Kubeflow
• Experience with cloud platform services such as AWS, Azure or GCP especially with EKS, Managed Kafka
• 5+ Industry experience in python
• Experience with popular modern web frameworks such as Spring boot, Play framework, or Django
• Experience with scripting languages. Python experience highly desirable. Experience in API development using Swagger
• Implementing automated testing platforms and unit tests
• Proficient understanding of code versioning tools, such as Git
• Familiarity with continuous integration, Jenkins

Responsibilities

• Architect, Design and Implement Large scale data processing pipelines using Kafka Streams, PySpark, Fluentd and Druid
• Create custom Operators for Kubernetes, Kubeflow
• Develop data ingestion processes and ETLs
• Assist in dev ops operations
• Design and Implement APIs
• Identify performance bottlenecks and bugs, and devise solutions to these problems
• Help maintain code quality, organization, and documentation
• Communicate with stakeholders regarding various aspects of solution.
• Mentor team members on best practices

Job Description:

We are looking for a Big Data Engineer who have worked across the entire ETL stack. Someone who has ingested data in a batch and live stream format, transformed large volumes of daily and built Data-warehouse to store the transformed data and has integrated different visualization dashboards and applications with the data stores.    The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them.

Responsibilities:

• Develop, test, and implement data solutions based on functional / non-functional business requirements.
• You would be required to code in Scala and PySpark daily on Cloud as well as on-prem infrastructure
• Build Data Models to store the data in a most optimized manner
• Identify, design, and implement process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Implementing the ETL process and optimal data pipeline architecture
• Monitoring performance and advising any necessary infrastructure changes.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Proactively identify potential production issues and recommend and implement solutions
• Must be able to write quality code and build secure, highly available systems.
• Create design documents that describe the functionality, capacity, architecture, and process.
• Review peer-codes and pipelines before deploying to Production for optimization issues and code standards

Skill Sets:

• Good understanding of optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and ‘big data’ technologies.
• Proficient understanding of distributed computing principles
• Experience in working with batch processing/ real-time systems using various open-source technologies like NoSQL, Spark, Pig, Hive, Apache Airflow.
• Implemented complex projects dealing with the considerable data size (PB).
• Optimization techniques (performance, scalability, monitoring, etc.)
• Experience with integration of data from multiple data sources
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB, etc.,
• Knowledge of various ETL techniques and frameworks, such as Flume
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Creation of DAGs for data engineering
• Expert at Python /Scala programming, especially for data engineering/ ETL purposes

Responsibilities

• Installing and configuring Informatica components, including high availability; managing server activations and de-activations for all environments; ensuring that all systems and procedures adhere to organizational best practices
• Day to day administration of the Informatica Suite of services (PowerCenter, IDS, Metadata, Glossary and Analyst).
• Informatica capacity planning and on-going monitoring (e.g. CPU, Memory, etc.) to proactively increase capacity as needed.
• Manage backup and security of Data Integration Infrastructure.
• Design, develop, and maintain all data warehouse, data marts, and ETL functions for the organization as a part of an infrastructure team.
• Consult with users, management, vendors, and technicians to assess computing needs and system requirements.
• Develop and interpret organizational goals, policies, and procedures.
• Evaluate the organization's technology use and needs and recommend improvements, such as software upgrades.
• Prepare and review operational reports or project progress reports.
• Assist in the daily operations of the Architecture Team , analyzing workflow, establishing priorities, developing standards, and setting deadlines.
• Work with vendors to manage support SLA’s and influence vendor product roadmap
• Provide leadership and guidance in technical meetings, define standards and assist/provide status updates
• Work with cross functional operations teams such as systems, storage and network to design technology stacks.

Preferred Qualifications

• Minimum 6+ years’ experience as Informatica Engineer and Developer role
• Minimum of 5+ years’ experience in an ETL environment as a developer.
• Minimum of 5+ years of experience in SQL coding and understanding of databases
• Proficiency in Python
• Proficiency in command line troubleshooting
• Proficiency in writing code in Perl/Shell scripting languages
• Understanding of Java and concepts of Object-oriented programming
• Good understanding of systems, networking, and storage
• Strong knowledge of scalability and high availability