Data Engineer

Senior Software Engineer - Data 

Job Description:

We are looking for a tech savvy Data Engineer to join our growing data team. The hire will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up. The Data Engineer will support our software developers, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. The hire must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. 

Data Engineer Job Responsibilities:

• Develop and maintain scalable data pipelines and build out new API integrations to support continuing increases in data volume and complexity.
• Implement processes and systems to monitor data accuracy, ensuring 100% data availability for key stakeholders and business processes that depend on it.
• Write unit/integration tests and document work.
• Perform data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
• Design data integrations and reporting framework.
• Work with stakeholders including the Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs
• Design and evaluate open source and vendor tools for data lineage.
• Work closely with all business units and engineering teams to develop strategy for long term data platform architecture.

Data Engineer Qualifications / Skills:

• 3+ years of Java development experience
• Experience with or knowledge of Agile Software Development methodologies
• Excellent problem solving and troubleshooting skills
• Process oriented with great documentation skills
• Experience with big data technologies like Kafka, BigQuery, etc
• Experience with AWS cloud services: EC2, RDS, etc
• Experience with message queuing, stream-processing systems

Education, Experience and Licensing Requirements:

• Degree in Computer Science, IT, or similar field; a Master’s is a plus
• 3+ years of hands on development experience
• 3+ years of SQL experience (No-SQL experience is a plus)
• 3+ years of experience with schema design and dimensional data modeling
• Experience designing, building and maintaining data processing systems

Technical Knowledge (Must Have)

• Strong experience in SQL / HiveQL/ AWS Athena,
• Strong expertise in the development of data pipelines (snaplogic is preferred).
• Design, Development, Deployment and administration of data processing applications.
• Good Exposure towards AWS and Azure Cloud computing environments.
• Knowledge around BigData, AWS Cloud Architecture, Best practices, Securities, Governance, Metadata Management, Data Quality etc.
• Data extraction through various firm sources (RDBMS, Unstructured Data Sources) and load to datalake with all best practices.
• Knowledge in Python
• Good knowledge in NoSQL technologies (Neo4J/ MongoDB)
• Experience/knowledge in SnapLogic (ETL Technologies)
• Working knowledge on Unix (AIX, Linux), shell scripting
• Experience/knowledge in Data Modeling. Database Development
• Experience/knowledge creation of reports and dashboards in Tableau/ PowerBI

Job Responsibilities:-

• Develop robust, scalable and maintainable machine learning models to answer business problems against large data sets.
• Build methods for document clustering, topic modeling, text classification, named entity recognition, sentiment analysis, and POS tagging.
• Perform elements of data cleaning, feature selection and feature engineering and organize experiments in conjunction with best practices.
• Benchmark, apply, and test algorithms against success metrics. Interpret the results in terms of relating those metrics to the business process.
• Work with development teams to ensure models can be implemented as part of a delivered solution replicable across many clients.
• Knowledge of Machine Learning, NLP, Document Classification, Topic Modeling and Information Extraction with a proven track record of applying them to real problems.
• Experience working with big data systems and big data concepts.
• Ability to provide clear and concise communication both with other technical teams and non-technical domain specialists.
• Strong team player; ability to provide both a strong individual contribution but also work as a team and contribute to wider goals is a must in this dynamic environment.
• Experience with noisy and/or unstructured textual data.

knowledge graph and NLP including summarization, topic modelling etc

• Strong coding ability with statistical analysis tools in Python or R, and general software development skills (source code management, debugging, testing, deployment, etc.)
• Working knowledge of various text mining algorithms and their use-cases such as keyword extraction, PLSA, LDA, HMM, CRF, deep learning & recurrent ANN, word2vec/doc2vec, Bayesian modeling.
• Strong understanding of text pre-processing and normalization techniques, such as tokenization,
• POS tagging and parsing and how they work at a low level.
• Excellent problem solving skills.
• Strong verbal and written communication skills
• Masters or higher in data mining or machine learning; or equivalent practical analytics / modelling experience
• Practical experience in using NLP related techniques and algorithms
• Experience in open source coding and communities desirable.

Able to containerize Models and associated modules and work in a Microservices environment

Technical/Core skills

1. Minimum 3 yrs of exp in Informatica Big data Developer(BDM) in Hadoop environment.
2. Have knowledge of informatica Power exchange (PWX).
3. Minimum 3 yrs of exp in big data querying tool like Hive and Impala.
4. Ability to designing/development of complex mappings using informatica Big data Developer.
5. Create and manage Informatica power exchange and CDC real time implementation
6. Strong Unix knowledge skills for writing shell scripts and troubleshoot of existing scripts.
7. Good knowledge of big data platforms and its framework.
8. Good to have an experience in cloudera data platform (CDP)
9. Experience with building stream processing systems using Kafka and spark
10. Excellent SQL knowledge

Soft skills :

1. Ability to work independently 
2. Strong analytical and problem solving skills
3. Attitude of learning new technology
4. Regular interaction with vendors, partners and stakeholders

• 6+ months of proven experience as a Data Scientist or Data Analyst
• Understanding of machine-learning and operations research
• Extensive knowledge of R, SQL and Excel
• Analytical mind and business acumen
• Strong Statistical understanding
• Problem-solving aptitude
• BSc/BA in Computer Science, Engineering or relevant field; graduate degree in Data Science or other quantitative field is preferred

Responsibilities

• Build and mentor the computer vision team at TransPacks
• Drive to productionize algorithms (industrial level) developed through hard-core research
• Own the design, development, testing, deployment, and craftsmanship of the team’s infrastructure and systems capable of handling massive amounts of requests with high reliability and scalability
• Leverage the deep and broad technical expertise to mentor engineers and provide leadership on resolving complex technology issues
• Entrepreneurial and out-of-box thinking essential for a technology startup
• Guide the team for unit-test code for robustness, including edge cases, usability, and general reliability

Eligibility

• Tech in Computer Science and Engineering/Electronics/Electrical Engineering, with demonstrated interest in Image Processing/Computer vision (courses, projects etc) and 6-8 years of experience
• Tech in Computer Science and Engineering/Electronics/Electrical Engineering, with demonstrated interest in Image Processing/Computer vision (Thesis work) and 4-7 years of experience
• D in Computer Science and Engineering/Electronics/Electrical Engineering, with demonstrated interest in Image Processing/Computer vision (Ph. D. Dissertation) and inclination to working in Industry to provide innovative solutions to practical problems

Requirements

• In-depth understanding of image processing algorithms, pattern recognition methods, and rule-based classifiers
• Experience in feature extraction, object recognition and tracking, image registration, noise reduction, image calibration, and correction
• Ability to understand, optimize and debug imaging algorithms
• Understating and experience in openCV library
• Fundamental understanding of mathematical techniques involved in ML and DL schemas (Instance-based methods, Boosting methods, PGM, Neural Networks etc.)
• Thorough understanding of state-of-the-art DL concepts (Sequence modeling, Attention, Convolution etc.) along with knack to imagine new schemas that work for the given data.
• Understanding of engineering principles and a clear understanding of data structures and algorithms
• Experience in writing production level codes using either C++ or Java
• Experience with technologies/libraries such as python pandas, numpy, scipy
• Experience with tensorflow and scikit.