Data Engineer-Backend

Responsibilities

● Able contribute to the gathering of functional requirements, developing technical
specifications, and project & test planning
● Demonstrating technical expertise, and solving challenging programming and design
problems
● Roughly 80% hands-on coding
● Generate technical documentation and PowerPoint presentations to communicate
architectural and design options, and educate development teams and business users
● Resolve defects/bugs during QA testing, pre-production, production, and post-release
patches
● Work cross-functionally with various bidgely teams including: product management,
QA/QE, various product lines, and/or business units to drive forward results

Requirements
● BS/MS in computer science or equivalent work experience
● 2-4 years’ experience designing and developing applications in Data Engineering
● Hands-on experience with Big data Eco Systems.
● Hadoop,Hdfs,Map Reduce,YARN,AWS Cloud, EMR, S3, Spark, Cassandra, Kafka,
Zookeeper
● Expertise with any of the following Object-Oriented Languages (OOD): Java/J2EE,Scala,
Python
● Strong leadership experience: Leading meetings, presenting if required
● Excellent communication skills: Demonstrated ability to explain complex technical
issues to both technical and non-technical audiences
● Expertise in the Software design/architecture process
● Expertise with unit testing & Test-Driven Development (TDD)
● Experience on Cloud or AWS is preferable
● Have a good understanding and ability to develop software, prototypes, or proofs of
concepts (POC's) for various Data Engineering requirements.

We are looking for an exceptionally talented Lead data engineer who has exposure in implementing AWS services to build data pipelines, api integration and designing data warehouse. Candidate with both hands-on and leadership capabilities will be ideal for this position.

 

Qualification: At least a bachelor’s degree in Science, Engineering, Applied Mathematics. Preferred Masters degree

 

Job Responsibilities:

• Total 6+ years of experience as a Data Engineer and 2+ years of experience in managing a team

• Have minimum 3 years of AWS Cloud experience.

• Well versed in languages such as Python, PySpark, SQL, NodeJS etc

• Has extensive experience in the real-timeSpark ecosystem and has worked on both real time and batch processing

• Have experience in AWS Glue, EMR, DMS, Lambda, S3, DynamoDB, Step functions, Airflow, RDS, Aurora etc.

• Experience with modern Database systems such as Redshift, Presto, Hive etc.

• Worked on building data lakes in the past on S3 or Apache Hudi

• Solid understanding of Data Warehousing Concepts

• Good to have experience on tools such as Kafka or Kinesis

• Good to have AWS Developer Associate or Solutions Architect Associate Certification

• Have experience in managing a team

Key  Responsibilities :

• Development of proprietary processes and procedures designed to process various data streams around critical databases in the org
• Manage technical resources around data technologies, including relational databases, NO SQL DBs, business intelligence databases, scripting languages, big data tools and technologies, visualization tools.
• Creation of a project plan including timelines and critical milestones to success in support of the project
• Identification of the vital skill sets/staff required to complete the project
• Identification of crucial sources of the data needed to achieve the objective.

 

Skill Requirement :

• Experience with data pipeline processes and tools
• Well versed in the Data domains (Data Warehousing, Data Governance, MDM, Data Quality, Data Catalog, Analytics, BI, Operational Data Store, Metadata, Unstructured Data, ETL, ESB)
• Experience with an existing ETL tool e.g Informatica and Ab initio etc
• Deep understanding of big data systems like Hadoop, Spark, YARN, Hive, Ranger, Ambari
• Deep knowledge of Qlik ecosystems like  Qlikview, Qliksense, and Nprinting
• Python, or a similar programming language
• Exposure to data science and machine learning
• Comfort working in a fast-paced environment

Soft attributes :

• Independence: Must have the ability to work on his/her own without constant direction or supervision. He/she must be self-motivated and possess a strong work ethic to strive to put forth extra effort continually
• Creativity: Must be able to generate imaginative, innovative solutions that meet the needs of the organization. You must be a strategic thinker/solution seller and should be able to think of integrated solutions (with field force apps, customer apps, CCT solutions etc.). Hence, it would be best to approach each unique situation/challenge in different ways using the same tools.
• Resilience: Must remain effective in high-pressure situations, using both positive and negative outcomes as an incentive to move forward toward fulfilling commitments to achieving personal and team goals.

Technical Knowledge (Must Have)

• Strong experience in SQL / HiveQL/ AWS Athena,
• Strong expertise in the development of data pipelines (snaplogic is preferred).
• Design, Development, Deployment and administration of data processing applications.
• Good Exposure towards AWS and Azure Cloud computing environments.
• Knowledge around BigData, AWS Cloud Architecture, Best practices, Securities, Governance, Metadata Management, Data Quality etc.
• Data extraction through various firm sources (RDBMS, Unstructured Data Sources) and load to datalake with all best practices.
• Knowledge in Python
• Good knowledge in NoSQL technologies (Neo4J/ MongoDB)
• Experience/knowledge in SnapLogic (ETL Technologies)
• Working knowledge on Unix (AIX, Linux), shell scripting
• Experience/knowledge in Data Modeling. Database Development
• Experience/knowledge creation of reports and dashboards in Tableau/ PowerBI

Bigdata with cloud:

 

Experience : 5-10 years

 

Location : Hyderabad/Chennai

 

Notice period : 15-20 days Max

 

1.  Expertise in building AWS Data Engineering pipelines with AWS Glue -> Athena -> Quick sight

2.  Experience in developing lambda functions with AWS Lambda

3.  Expertise with Spark/PySpark – Candidate should be hands on with PySpark code and should be able to do transformations with Spark

4.  Should be able to code in Python and Scala.

5.  Snowflake experience will be a plus

Job Description:

We are looking for a Big Data Engineer who have worked across the entire ETL stack. Someone who has ingested data in a batch and live stream format, transformed large volumes of daily and built Data-warehouse to store the transformed data and has integrated different visualization dashboards and applications with the data stores.    The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them.

Responsibilities:

• Develop, test, and implement data solutions based on functional / non-functional business requirements.
• You would be required to code in Scala and PySpark daily on Cloud as well as on-prem infrastructure
• Build Data Models to store the data in a most optimized manner
• Identify, design, and implement process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Implementing the ETL process and optimal data pipeline architecture
• Monitoring performance and advising any necessary infrastructure changes.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Proactively identify potential production issues and recommend and implement solutions
• Must be able to write quality code and build secure, highly available systems.
• Create design documents that describe the functionality, capacity, architecture, and process.
• Review peer-codes and pipelines before deploying to Production for optimization issues and code standards

Skill Sets:

• Good understanding of optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and ‘big data’ technologies.
• Proficient understanding of distributed computing principles
• Experience in working with batch processing/ real-time systems using various open-source technologies like NoSQL, Spark, Pig, Hive, Apache Airflow.
• Implemented complex projects dealing with the considerable data size (PB).
• Optimization techniques (performance, scalability, monitoring, etc.)
• Experience with integration of data from multiple data sources
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB, etc.,
• Knowledge of various ETL techniques and frameworks, such as Flume
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Creation of DAGs for data engineering
• Expert at Python /Scala programming, especially for data engineering/ ETL purposes

 

 

 

Recko Inc. is looking for data engineers to join our kick-ass engineering team. We are looking for smart, dynamic individuals to connect all the pieces of the data ecosystem.

 

What are we looking for:

1. 3+  years of development experience in at least one of MySQL, Oracle, PostgreSQL or MSSQL and experience in working with Big Data technologies like Big Data frameworks/platforms/data stores like Hadoop, HDFS, Spark, Oozie, Hue, EMR, Scala, Hive, Glue, Kerberos etc.

2. Strong experience setting up data warehouses, data modeling, data wrangling and dataflow architecture on the cloud

3. 2+ experience with public cloud services such as AWS, Azure, or GCP and languages like Java/ Python etc

4. 2+ years of development experience in Amazon Redshift, Google Bigquery or Azure data warehouse platforms preferred

5. Knowledge of statistical analysis tools like R, SAS etc 

6. Familiarity with any data visualization software

7. A growth mindset and passionate about building things from the ground up and most importantly, you should be fun to work with

As a data engineer at Recko, you will:

1. Create and maintain optimal data pipeline architecture,

2. Assemble large, complex data sets that meet functional / non-functional business requirements.

3. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

4. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.

5. Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.

6. Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.

7. Keep our data separated and secure across national boundaries through multiple data centers and AWS regions.

8. Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.

9. Work with data and analytics experts to strive for greater functionality in our data systems.

 

About Recko: 

Recko was founded in 2017 to organise the world’s transactional information and provide intelligent applications to finance and product teams to make sense of the vast amount of data available. With the proliferation of digital transactions over the past two decades, Enterprises, Banks and Financial institutions are finding it difficult to keep a track on the money flowing across their systems. With the Recko Platform, businesses can build, integrate and adapt innovative and complex financial use cases within the organization and  across external payment ecosystems with agility, confidence and at scale.  . Today, customer-obsessed brands such as Deliveroo, Meesho, Grofers, Dunzo, Acommerce, etc use Recko so their finance teams can optimize resources with automation and prioritize growth over repetitive and time-consuming tasks around day-to-day operations. 

 

Recko is a Series A funded startup, backed by marquee investors like Vertex Ventures, Prime Venture Partners and Locus Ventures. Traditionally enterprise software is always built around functionality. We believe software is an extension of one’s capability, and it should be delightful and fun to use.

 

Working at Recko: 

We believe that great companies are built by amazing people. At Recko, We are a group of young Engineers, Product Managers, Analysts and Business folks who are on a mission to bring consumer tech DNA to enterprise fintech applications. The current team at Recko is 60+ members strong with stellar experience across fintech, e-commerce, digital domains at companies like Flipkart, PhonePe, Ola Money, Belong, Razorpay, Grofers, Jio, Oracle etc. We are growing aggressively across verticals.

Desired Skill, Experience, Qualifications, and Certifications:
• 5+ years’ experience developing and maintaining modern ingestion pipeline using
technologies like Spark, Apache Nifi etc).
• 2+ years’ experience with Healthcare Payors (focusing on Membership, Enrollment, Eligibility,
• Claims, Clinical)
• Hands on experience on AWS Cloud and its Native components like S3, Athena, Redshift &
• Jupyter Notebooks
• Strong in Spark Scala & Python pipelines (ETL & Streaming)
• Strong experience in metadata management tools like AWS Glue
• String experience in coding with languages like Java, Python
• Worked on designing ETL & streaming pipelines in Spark Scala / Python
• Good experience in Requirements gathering, Design & Development
• Working with cross-functional teams to meet strategic goals.
• Experience in high volume data environments
• Critical thinking and excellent verbal and written communication skills
• Strong problem-solving and analytical abilities, should be able to work and delivery
individually
• Good-to-have AWS Developer certified, Scala coding experience, Postman-API and Apache
Airflow or similar schedulers experience
• Nice-to-have experience in healthcare messaging standards like HL7, CCDA, EDI, 834, 835, 837
• Good communication skills

Strong exposure in ETL / Big Data / Talend / Hadoop / Spark / Hive / Pig

To be considered as a candidate for a Senior Data Engineer position, a person must have a proven track record of architecting data solutions on current and advanced technical platforms. They must have leadership abilities to lead a team providing data centric solutions with best practices and modern technologies in mind. They look to build collaborative relationships across all levels of the business and the IT organization. They possess analytic and problem-solving skills and have the ability to research and provide appropriate guidance for synthesizing complex information and extract business value. Have the intellectual curiosity and ability to deliver solutions with creativity and quality. Effectively work with business and customers to obtain business value for the requested work. Able to communicate technical results to both technical and non-technical users using effective story telling techniques and visualizations. Demonstrated ability to perform high quality work with innovation both independently and collaboratively.

• Design secure solutions in line with the business strategy and security requirements • Contribute to the enterprise security architecture through developing Strategies, Reference Architectures, Roadmaps, Architectural Principles, Technology Standards, Security Non-Functional Requirements, Architectural Decisions and Design Patterns. • Deliver cyber security architectural artifacts such as High Level Designs and Solution Blueprints. • Ensure the enforcement of security requirements in solution architecture • Contribute to educating other architects and engineering teams in designing and implementing secure solutions Technologies The candidate should have knowledge and experience in designing and implementing the following technologies and related domains • Cloud security • Identity and Access Management • Encryption, Masking and Key Management • Data Classification, Data Privacy and Data Leakage Prevention • Infrastructure security (Network/Servers/Virtualization) • Application Security • Endpoint Security • SIEM and Log Management • Forward and Reverse Proxy • Big Data Security • IoT Security • SAP Security (Preferred) Architecture Skills • Solid experience in developing security solution architecture • Solid experience and knowledge in TOGAF and SABSA or other Enterprise Architecture frameworks. • Strong experience in developing architectural artifacts including reference architectures, roadmaps, architectural principles, technology standards, security non-functional requirements, architectural decisions and design patterns • Strong experience in documenting existing, transition and target architectures. Cyber Security Skills • Solid experience in performing security risk assessments and controls implementation • Strong experience in designing and implementing security controls by utilizing the technologies mentioned in the technologies section above • Strong knowledge in offensive and defensive aspects of cybesecurity with solid understanding of attack techniques and abuse cases. • Strong knowledge and implementation experience of cyber security standards, frameworks and regulations such as ISO27001, NIST CSF, CSA CCM, PCI-DSS, GDPR