Data Engineer
Mandatory Requirements
- Experience in AWS Glue
- Experience in Apache Parquet
- Proficient in AWS S3 and data lake
- Knowledge of Snowflake
- Understanding of file-based ingestion best practices.
- Scripting language - Python & pyspark
CORE RESPONSIBILITIES
- Create and manage cloud resources in AWS
- Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies
- Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform
- Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations
- Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
- Define process improvement opportunities to optimize data collection, insights and displays.
- Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible
- Identify and interpret trends and patterns from complex data sets
- Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders.
- Key participant in regular Scrum ceremonies with the agile teams
- Proficient at developing queries, writing reports and presenting findings
- Mentor junior members and bring best industry practices
QUALIFICATIONS
- 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales)
- Strong background in math, statistics, computer science, data science or related discipline
- Advanced knowledge one of language: Java, Scala, Python, C#
- Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake
- Proficient with
- Data mining/programming tools (e.g. SAS, SQL, R, Python)
- Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
- Data visualization (e.g. Tableau, Looker, MicroStrategy)
- Comfortable learning about and deploying new technologies and tools.
- Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.
- Good written and oral communication skills and ability to present results to non-technical audiences
- Knowledge of business intelligence and analytical tools, technologies and techniques.
Familiarity and experience in the following is a plus:
- AWS certification
- Spark Streaming
- Kafka Streaming / Kafka Connect
- ELK Stack
- Cassandra / MongoDB
- CI/CD: Jenkins, GitLab, Jira, Confluence other related tools
Similar jobs
This is a full-time on-site role for a Marketing Strategist at PRODBIZ SOLUTIONS PVT LTD located in Hyderabad. The Marketing Strategist will be responsible for market research, developing marketing strategies, conducting research, and communication tasks on a day-to-day basis.
Java Fullstack Developer
Experience and in - depth knowledge of Java/J2EE, Spring Boot, Spring and Hibernate, Angular framework, HQL.
Experience integrating with Restful web services
Familiar with various design and architectural patterns.
Write well designed testable, efficient code, and ensure that the designs comply with specifications.
A team member who can contribute in all phases of the development lifecycle.
Able to Plan, Estimate the effort based on requirements.
Skill for writing reusable Java libraries, Spring boot, Angular.
Design and development of Spring based services
Design, build, and maintain efficient, reusable, and reliable Java code.
Development of spring boot applications and related concepts Knowledge
Excellent core java skills with preferably in Java 8 new features understanding
Strong in Collections, Generics, IO, Threads, and Functional Programming.
REST and SOAP API understanding, knowledge and experience.
Able to write REST API endpoint using Spring MVC Framework.
Good knowledge of Microservices is mandator
Industry
- IT Services and IT Consulting
Key Skills: Sales, negotiation, client handling, good communication, team handling
About Company:
Rent2stays.com, We are a start-up taken birth to solve the challenges faced by Owners, Tenants, Builders, Owners who stay remote, Owners who need services not only through digital platforms but someone who can be owner’s representative.
We understand the Real Estate rental space because we don’t know anything else apart from Rental space. In nutshell, we breathe Real Estate Rental space. So all our solutions, services are long term to protect your asset. To get good returns and to get asset appreciation.
Our services are two-fold:
We sign up with owners to manage their properties, rent we will transfer, security Deposit we will transfer, Demand Generation we will take care. We will arrange visit to tenants, move ins, move out will be taken care by us. Owners need not to return security deposit every time when there is change in tenants, so its complete End to End Rental management solutions
Roles & Responsibilities:
- Maintaining a relationship with existing Property Owners (Residential) & Tenants.
- Finding out properties across Bangalore for building acquisition to rent out the same for a prospective tenant.
- Manage and maintain a pipeline of interested prospects.
- Negotiate pricing with customers, and suppliers in some cases.
- Able to handle client relationships & closing the business deal.
- Able to work in target driven environment.
- Should handle apartment sales / renting out properties across Bangalore.
- Able to handle team & train on sales process
- Responsible for sales document preparation.
- Keep records of sales, revenue, invoices etc.
- Conduct market research to identify new markets and customer needs and should be closing the business deal.
- Developing goals for the team, assigning the target and ensuring they are met.
Responsibilities:
• Designing Hive/HCatalog data model includes creating table definitions, file formats, compression techniques for Structured & Semi-structured data processing
• Implementing Spark processing based ETL frameworks
• Implementing Big data pipeline for Data Ingestion, Storage, Processing & Consumption
• Modifying the Informatica-Teradata & Unix based data pipeline
• Enhancing the Talend-Hive/Spark & Unix based data pipelines
• Develop and Deploy Scala/Python based Spark Jobs for ETL processing
• Strong SQL & DWH concepts.
Preferred Background:
• Function as integrator between business needs and technology solutions, helping to create technology solutions to meet clients’ business needs
• Lead project efforts in defining scope, planning, executing, and reporting to stakeholders on strategic initiatives
• Understanding of EDW system of business and creating High level design document and low level implementation document
• Understanding of Big Data Lake system of business and creating High level design document and low level implementation document
• Designing Big data pipeline for Data Ingestion, Storage, Processing & Consumption
§ Experience of developing web based enterprise wide applications in .NET4.5 (C#, ASP.NET), WCF, WPF, AJAX etc
§ Understanding of enterprise application development and deployment.
§ Good understanding OOPS and Design Patterns and experience of tools like Enterprise Architect.
§ Good knowledge of RDBMS design, programming and DBA concepts and experience of working in SQL Server and Oracle databases.
§ Good Design, Coding and Testing skills.
§ Knowledge and exposure to Service Oriented Architecture, Enterprise Service Bus and application Integration (using middleware).
§ Experience of working in electricity utility domain – preferably in EA/AMR/AMI/EMS/IT System Integration/ERP (for Utilities) projects
§ Mobility based software development, GPRS, GSM Knowledge of BI tools implementation.
§ Experience in mobility based applications will be an added advantage.
We are looking for a Full Stack Developer to produce scalable software solutions. You’ll be part of a cross-functional team that’s responsible for the full software development life cycle, from conception to deployment.
As a Full Stack Developer, you should be comfortable around both front-end and back-end coding languages, development frameworks and third-party libraries. You should also be a team player with a knack for visual design and utility.
The Incident Response Senior Principal Analyst leads a team of experts with diverse skill sets across areas such as Security Operations Center (SOC), Forensics, and other applicable technical Subject Matter Expert (SME) resources. The IR Senior Principal Analyst is specifically tasked with managing all aspects of an Incident Response engagement to include incident validation, monitoring, containment, log analysis, system forensic analysis, and reporting. The IR Senior Principal Analyst is also responsible for developing and sustaining strong relationships with our clients, and client’s counsel to ensure the engagement’s objectives and expectations are met and executed successfully as documented in the statement of work. The incumbent of this role should display a strong foundation of technical expertise in Cybersecurity, Incident Response, and Digital Forensics to successfully execute the responsibilities associated with this role.
ROLES AND RESPONSIBILITIES
- Supports the management of the technical aspects from client setup and kickoff to supporting the reporting process.
- Co-leads project scoping calls to accurately collect information from the client concerning the incident to include but not be limited to the client’s environment, size, technology, and security threats. Responsible for capturing all client’s expectations and objectives throughout the engagement to ensure successful engagement delivery.
- Organize and maintain an inventory of requests sent to the client to include at a minimum public IP ranges, requested information (including systems for collection), collected logs, systems Skadi or full systems, and any other requested made of the client by Arete or counsel.
- Works directly with the client and other Arete team members to preserve and collect artifacts for forensic analysis.
- Engages in communications with the TA for negotiation and recovery of decryption keys or manages the ransomware specialist team.
- Ensures deadlines are met and timely update meetings are established with client and counsel.
- Responsible for quality control over the budget of engagement and proactively identifying the need for addendums for engagements. Discusses with counsel before provided addendum.
- The main point of contact who manages and participates in all communications with the client and client’s counsel during the engagement. Assists with the development of communications.
- Supports the management and coordination of all technical efforts for the IR engagement to drive the process forward through; tool deployment, ransomware decryption, restoration, and recovery efforts, system rebuilds, system, application, and network administration tasks.
- Coordinates with the Ransom Specialist when ransom negotiations are needed. Ensures updates regarding ransom status are delivered to the client and counsel in a timely fashion.
- Manages and coordinates the onsite efforts with the Onsite Lead or team ensuring they understand and can execute the objectives for the onsite work. Additional responsibilities with onsite efforts include ensuring communications are frequent and getting the daily onsite update communicating these back to the IR Director and/or IR Ops Associate for their Tiger Team.
- Co-manages restoration team when engaged with the client for recovery of systems, data collection, and SentinelOne (S1) deployment.
- Partners with the Forensic Lead to coordinate additional data collection requests pertinent to the investigation.
- Communicates in tandem with the Forensic Lead relevant findings to the client during the investigation.
- Designs and executes a strategy to install S1 and live response data within the SLAs set by Arete.
- Manage the SOC for accurate reporting of S1 metrics from threats to checked-in systems based on the need from the client.
- Follows up with the SOC Lead on SentinelOne alerts and encourages/coordinates client participation with the product.
- Organizes the updates for client and counsel and acts as the "quarterback" for leading update calls when prompted by counsel; maintains an organized and methodical approach for providing updates from negotiations, system restoration, data collection forensics, and closeout. Accountable for final report review, ensuring the report is accurate, professional, and meets the objective of client counsel.
- Can troubleshoot instability issues within infected operating systems and stabilize the system for continued recovery.
- Cross trains across the IR services within SOC, IR Lead, Forensics, and Restoration.
- Supports peers and IR Directors within the engagement lifecycle. Familiarizes oneself with the negotiation tactics and communications with threat actors.
- Other duties as assigned.
DISCLAIMER
The above statements are intended to describe the general nature and level of work being performed. They are not intended to be an exhaustive list of all responsibilities, duties, and skills required personnel so classified.
SKILLS AND KNOWLEDGE
- Experience delivering consulting engagements in a fast-paced environment
- Experience leading scoping calls
- Strong background and practical hands-on experience with Windows or Linux System and Network Administration, Security DevOps, Incident Response and Digital Forensics, or Security Engineering
- Practical experience performing in a functional role including but not limited to one or more of the following disciplines: computer forensics, Incident Response, data analytics, Security Operations, and Engineering, Digital Investigations
- Knowledgeable of collection methodologies and tools.
- Comfortable working within various OS including Windows, Linux, and OSX
- Organized communications and notes
- Communicates clearly and concisely
- Generally knowledgeable of the multiple services that comprise an IR investigation
- In-depth knowledge of the ransom negotiation process and details it accordingly to clients
- Must have 5-8 years of experience in handling data
- Must have the ability to interpret large amounts of data and to multi-task
- Must have strong knowledge of and experience with programming (Python), Linux/Bash scripting, databases(SQL, etc)
- Must have strong analytical and critical thinking to resolve business problems using data and tech
- Must have domain familiarity and interest of – Cloud technologies (GCP/Azure Microsoft/ AWS Amazon), open-source technologies, Enterprise technologies
- Must have the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
- Must have good communication skills
- Working knowledge/exposure to ElasticSearch, PostgreSQL, Athena, PrestoDB, Jupyter Notebook