J&F

Role Summary

We are hiring a Data Engineer / ML Data Pipeline Engineer to build and operate the data backbone of the Enterprise AI platform:

What You'll Own

• Ingestion & ETL/ELT pipelines for heterogeneous project folders (PDF drawings, SVG files, IFC models, BBS.json bar-bending-schedule data, Excel exports, and AI agent output JSON).
• AWS-based data architecture: S3 raw/staging/curated/outputs structuring, partitioning, versioning, and lifecycle management; querying via Athena/Glue and warehousing via Redshift or Snowflake as needed.
• Data validation frameworks: GUID cross-referencing between SVG and BBS data, schema enforcement, duplicate/orphan detection, reference integrity checks, and structured validation reporting.
• Agent run logging & observability: designing the database schema and pipelines that track every AI agent run (inputs, outputs, status, errors, cost, retries, reviewer feedback).
• AI Factory monitoring dashboards: operational dashboards (failure rates, retries, latency, data quality) and business dashboards (throughput, cost per run, rework rate) for Power BI/QuickSight or equivalent.
• ML data pipeline support: dataset preparation, labeling/annotation workflows, human-in-the-loop review tooling, and dataset versioning for models that classify or QC drawing issues.
• APIs: designing and building FastAPI/Flask endpoints to trigger validation runs and expose agent processing status to internal tools.
• Data quality & testing discipline: idempotent pipelines, quarantine/reject handling, regression and reconciliation testing, and root-cause debugging when pipelines or query performance degrade in production.

Key Skills — Non-Negotiable (Must-Have, Strong Level)

• Python — production-grade scripting: file/folder handling, JSON/schema processing, clean error handling, not just notebook-level scripting.
• SQL — strong hands-on ability, including GROUP BY/HAVING for duplicate detection, window functions, and daily aggregate/rate calculations (e.g., success-rate queries).
• AWS S3 data handling — practical experience structuring buckets for raw/staging/curated data, versioning, and avoiding overwrite issues at scale.
• Data validation — demonstrable experience building validation logic (set comparisons, duplicate/missing detection, structured pass/fail reporting), not just "I write assertions."
• ETL/ELT pipeline design — end-to-end ownership of at least one pipeline: source → transform → storage → validation → monitoring → business outcome, with clear articulation of what they personally built.
• Query/warehouse engine judgment — working knowledge of when to use Athena vs. Redshift vs. Snowflake (or equivalent), partitioning, clustering, sort/distribution keys, and storage format trade-offs (Parquet vs. JSON vs. CSV).

Key Skills — Good to Have

• Dashboarding — Power BI / QuickSight (or equivalent) fact/dimension table design, KPI cards, drill-downs; medium-to-strong level is a plus but trainable.
• FastAPI / Flask — building real endpoints with request/response schemas and basic error handling; especially valuable for validation-trigger and agent-status APIs.
• ML data pipeline experience — dataset labeling, annotation platform design, train/test/validation splitting, dataset versioning; strong on the pipeline/data side rather than model training itself.
• Human-in-the-loop / review tooling — experience building or contributing to browser-based labeling/review platforms (session persistence, label schema, export formats).
• Large-scale metadata querying — experience making file discovery fast across large volumes (1,000+ projects, thousands of files each) via metadata index tables, event-based ingestion, or catalog tools like AWS Glue.

Role overview:

We are hiring one Senior Backend Engineer to take end-to-end ownership of our serverless backend — a hands-on IC role for someone both technically excellent and comfortable being one of the few people the entire backend depends on. You'll own the services across several Node.js and Python repositories, work directly with the founders and product team, and set the technical bar for reliability, security, and performance.

Key responsibilities

• Design, build, and operate AWS Lambda services across our HCM/workforce, project-management, commercial/revenue, permissions, and document domains — each comprising dozens of functions.
• Own the multi-tenant PostgreSQL data layer — schema design, query performance, and the permission/relationship model — end to end.
• Maintain and evolve the request path — API Gateway → custom Lambda authorizer → VPC-bound Lambda → private databases — including the runtime IAM/credential model that scopes every request.
• Safeguard tenant isolation and security across a per-company Cognito authentication model.
• Build and maintain integrations with external construction data environments (Asite, Autodesk Construction Cloud), including large-scale document synchronization.
• Optimize performance and reliability to keep latency-sensitive endpoints well within platform limits under growing load.
• Raise the engineering bar — testing, observability, CI/CD, and modernization of legacy components.
• Debug and resolve production incidents to root cause, and put safeguards in place so they don't recur.
• Document decisions and designs and collaborate with the frontend (Angular) and product teams.

Challenges you'll solve.

We prefer to be candid — these are the problems that make this role genuinely interesting:

Latency under a hard ceiling

API Gateway terminates any request beyond ~29 seconds regardless of the Lambda's own timeout — yet much of our value comes from heavy cross-project reporting. You'll keep p95 latency within budget through set-based SQL, pagination, streaming, and asynchronous processing.

Least-privilege, per-request security

A shared custom authorizer mints short-lived, request-scoped credentials via sts:AssumeRole under a strict 2,048-character inline session-policy limit. You'll design permission models that stay within that budget and reason about IAM precisely.

Graph-shaped data, relational store

The permission and relationship model is inherently graph-like, but lives in PostgreSQL — you'll model it with recursive queries, careful indexing, and set-based traversal rather than reaching for a separate graph engine.

Watertight multi-tenancy

One Cognito pool per company and tenant-scoped access throughout — isolation is a first-order concern.

VPC-bound serverless

Lambdas run inside a VPC to reach private databases; you'll manage cold starts, connection lifecycles, and pool limits.

Resilient external integrations

Syncing large document sets from third-party APIs (including SOAP/XML) demands backpressure, deduplication, retries, and graceful partial-failure handling.

Compute-heavy workloads

Server-side PDF generation, image processing, and multi-currency handling within Lambda's memory and time constraints.

The stack.

Runtime — Node.js, Python, AWS Lambda

AWS services — -1 API Gateway, Lambda, Cognito, STS / IAM, Secrets Manager, S3 CloudWatch, VPC, EC2

Infrastructure & CI/CD- AWS SAM, CodePipeline → CodeBuild Shared Data —PostgreSQL

Qualifications.

• 5+ years building and operating production backend systems.
• Deep expertise in Node.js and JavaScript — the asynchronous model, event loop, and memory behavior — plus solid working proficiency in Python and its production behavior.
• Strong hands-on AWS experience, ideally serverless (Lambda, API Gateway, IAM/STS, VPC, Secrets Manager, CloudWatch) — able to reason about IAM policies, not just apply them.
• Advanced SQL and relational data modeling — set-based query design and a working understanding of why N+1 patterns cause production issues.
• Proven production-debugging ability — root-cause analysis in distributed systems from logs and first principles.
• Strong ownership, sound judgment, and clear written communication — able to make good decisions with incomplete information and explain trade-offs to non-engineers.

Interview Process:

Introductory call-Mutual fit and role overview.

Technical deep-dive- A walkthrough of a challenging production problem you have owned.

Practical exercise -A realistic backend task, or a walkthrough of your own representative code.

System design- Collaborative design on a real scenario.

Final conversation- Values, ownership, compensation, and offer.

KEY RESPONSIBILITIES:

•Build agents with persistent context & memory

•Design self-learning feedback loops

•Implement RAG pipelines for domain knowledge

•Manage conversation state & orchestration

•Integrate with LLM APIs (OpenAI, Claude, open-source)

Iterate fast — ship daily, measure weekly

MUST-HAVE SKILLS

•Python / TypeScript proficiency

•LangChain, CrewAI, AutoGen or custom frameworks

•Experience with vector DBs (Pinecone, Weaviate, Qdrant)

•Prompt engineering & evaluation pipelines

•Understanding of agent architectures (ReAct, tool-use)

Git, CI/CD, containerization basics