Terrabase

Experience: 5+ years production software engineering, with 2+ years working directly on LLM or agent systems in production. 

Location: Remote 

To streamline and fast-track screening, please submit your details here (if you haven’t already): https://airtable.com/appbtkr4odapnb5I6/pagqo91lKv3VJg3GT/form 

We’ll review your responses as part of the initial screening process. Please make sure you complete and submit all details through the form to be considered for the next stage. Submissions outside the form may not be considered.

Why This Role Matters

Terrabase builds agent infrastructure that enterprise customers rely on daily for SQL generation, forecasting, data analysis, and artifact delivery. Our orchestration layer routes between specialized sub-agents, manages typed handoff contracts, runs structured eval suites, and enforces correctness across every turn.

This is not a research-prototype role. You will build and evolve agent architecture, but always in service of making the system observable, typed, evaluated, recoverable, and boringly reliable in production.

What You Will Do

Own the harness architecture and middleware stack. Our LangGraph orchestrator routes between sub-agents through a layered middleware stack: file upload handling, source resolution, local context, workspace sync, state hydration, aggregation barriers, and typed handoff contracts. You will extend this stack, enforce its contracts in code, and keep it operational as routing logic and agent surfaces evolve.

Maintain typed contracts and boundaries. Agent handoffs at Terrabase carry typed contracts with barrier conditions and retry predicates. You will design these contracts, enforce them with strict typing, manage backward compatibility when contracts change, and write the contract tests that prevent silent regressions.

Own the eval suites. We run structured eval suites across routing decisions, context-resolution accuracy, multi-turn coherence, visual reference alignment, and artifact correctness. You will extend coverage, write new evals where gaps exist, and build CI gates that block releases when regressions are detected. A routing change or prompt change with no eval coverage does not ship.

Triage production failures and close the loop. When an agent turn fails in production, you will trace it in LangSmith, identify the failure class, and convert it into a durable regression test. You will own the release gates, keep prompts and runtime contracts in sync, manage feature flag rollout risk, and remove dead paths as the system evolves.

Own SQL and artifact correctness. Our agents generate SQL over customer schemas and produce structured artifacts (reports, dashboards, data sheets) under a strict schema contract. You will own the correctness layer: source grounding, schema-aware validation, provenance surfaces, and the eval infrastructure that catches generated artifact failures before they reach customers.

Build and maintain HITL workflows. Human-in-the-loop checkpoints let users intervene, redirect, or approve mid-chain. You will design these workflows, enforce their resumable state contracts, and ensure they degrade gracefully when interrupted.

Instrument for traceability. You will extend LangSmith tracing coverage, add structured span annotations, and build the tooling that lets us diagnose a bad agent turn from production trace data alone, without requiring a local reproduction.

What We Are Looking For

• 5+ years production software engineering, with strong Python fundamentals
• 2+ years working hands-on with LLM-based systems: agent loops, tool use, context management, or inference pipelines
• Experience with LangGraph, LangChain, OpenAI/Anthropic tool-use systems, or equivalent multi-step agent/runtime orchestration
• Practical eval engineering: you have built or extended eval harnesses, written automated test cases for agent behavior, and treated evaluation as an ongoing engineering discipline
• Strong engineering hygiene: strict typing, small interfaces, contract tests, clear schema migrations, and CI discipline
• Ability to debug from production traces and artifacts, not only local reproductions
• Comfort working across prompts, Python runtime code, TypeScript product surfaces, data systems, and eval infrastructure
• Systems thinking: you design for observability, recovery, and state management, not just the happy path
• Maintenance ownership mindset: you triage, close loops, and leave systems more debuggable than you found them
• Pragmatic judgment: you can distinguish between reliability-critical infrastructure and speculative abstraction

Bonus Points

• HITL workflow design: checkpoints, approvals, mid-chain interrupts, resumable state
• Context engineering depth: chunking strategies, retrieval-augmented generation, semantic routing, re-ranking
• Experience with LangSmith, Weights and Biases, or similar trace and evaluation platforms
• Prior work shipping agent systems to enterprise customers where SQL or data correctness is a hard requirement
• Experience with mypy, Pydantic contracts, or strict typing disciplines in a production Python codebase

Life at Terrabase

We are a sharp, focused, fully remote team building agent infrastructure that enterprise customers trust with their data. You will work directly alongside the engineer who designed this harness, with broad ownership, generous compute budgets, and a culture that treats reliability as a product requirement, not a research topic.

Terrabase is an equal-opportunity employer. We celebrate diversity and are committed to building an inclusive environment for every team member.

Experience: 6+ years building and operating production ML systems that drive commercial decisions at scale. 

Location: Remote 

To streamline and fast-track screening, please submit your details here (if you haven’t already): https://airtable.com/appbtkr4odapnb5I6/pag05ROZwgz5AaLDG/form 

We’ll review your responses as part of the initial screening process. Please make sure you complete and submit all details through the form to be considered for the next stage. Submissions outside the form may not be considered.

Why This Role Matters

Terrabase builds decisioning infrastructure for enterprise customers: ranked recommendations, scoring pipelines, and policy-governed outputs that drive real commercial action. Our ML systems do not live in notebooks. They run multi-stage evaluation harnesses, apply structured governance rules, backtest against historical outcomes, and ship ranked outputs that customers act on daily.

This role owns the decisioning system end to end. That means the models, the eval harness, the policy layer, the production services, and the technical roadmap for where all of it goes next.

What You Will Do

Own the decisioning and ranking pipeline. Design, extend, and operate the end-to-end system: candidate generation in DuckDB, multi-stage scoring with LightGBM and AutoGluon, post-score policy application, and final ranked output delivery. You understand each layer well enough to debug latency, correctness, and coverage problems quickly, and to design the next version.

Lead the evaluation harness. Our eval pipeline runs multiple gates before any output ships: data health checks, specification validation, business rules enforcement, resolution checks, LLM-as-judge scoring, backtest against historical outcomes, and final output validation. You will own this harness, extend it as the system grows, and ensure every model or pipeline change is measurable and reproducible before it reaches a customer.

Apply policy logic with rigor. Our ML systems operate under structured governance rules that determine which offers apply to which customer segments, under what conditions. You will implement, test, and audit these rules in code, not configure them in a spreadsheet. Every exclusion must be traceable and explainable.

Engineer features that move metrics. Identify and build the behavioral signals, engagement indicators, contract features, and value-band attributes that improve model performance. Close the loop from feature hypothesis through offline evaluation to production monitoring. Own the data contracts between upstream sources and the scoring pipeline.

Build and maintain the production pipeline and service layer. The decisioning system is not a batch notebook. You will write and operate the Python pipeline and service layer that wraps model inference, handles edge cases, versions model artifacts, and connects to downstream consumers. You own CI, test coverage, reproducible training runs, monitoring, and production incidents.

Drive technical direction. Write design documents, lead code review, and set the engineering standard for the decisioning system. Help define the roadmap: what gets built, in what order, and why. Mentor contributors who work alongside you on this system.

Work forward-deployed. You will engage directly with customer stakeholders to understand business context, interpret model outputs, and translate commercial requirements into system constraints. You are accountable for customer delivery, not just model accuracy.

What We Are Looking For

• 6+ years building and operating production ML systems, not prototypes or research work
• Strong Python skills across the full ML lifecycle: data pipelines, feature engineering, model training, inference services, and monitoring
• Production experience with gradient boosting models (LightGBM, XGBoost)
• Hands-on with DuckDB or similar in-process analytical engines for large-scale data processing
• Evaluation discipline: held-out metrics, backtesting against historical data, multi-gate eval pipelines, LLM-as-judge patterns
• Experience applying business rules, policy overrides, or constraint layers on top of model outputs
• Engineering fundamentals: CI pipelines, data contracts, versioned artifacts, test coverage, incident response
• Technical leadership: design docs, code review, roadmap input, mentoring
• Comfort with forward-deployed work: you can run a meeting with a non-technical stakeholder and turn the output into a system requirement
• Comfort inheriting an existing production codebase, improving its structure, and raising reliability without rewriting everything from scratch

Bonus Points

• Experience with next-best-offer engines, customer-level targeting, or recommendation systems at scale
• Experience with AutoML frameworks (AutoGluon or similar) in a production scoring pipeline
• Thompson sampling, multi-armed bandits, or portfolio-level optimization experience
• Exposure to structured data from telecoms, financial services, or retail sectors
• Prior work owning a decisioning or ranking system as the technical lead

Life at Terrabase

We are a sharp, focused, fully remote team that ships to real enterprise customers weekly. You will own a system that drives measurable commercial outcomes, with high autonomy, generous cloud budgets, and a culture that prizes rigor over hype.

Terrabase is an equal-opportunity employer. We celebrate diversity and are committed to building an inclusive environment for every team member.

Experience: 5+ years building and operating production-grade Python services. 

Location: Remote 

To streamline and fast-track screening, please submit your details here (if you haven’t already): https://airtable.com/appbtkr4odapnb5I6/pag8eyxvIdQ5YQCku/form

We’ll review your responses as part of the initial screening process. Please make sure you complete and submit all details through the form to be considered for the next stage. Submissions outside the form may not be considered.

Why This Role Matters

Every insight Terrabase delivers travels through a Python service you will own. Our platform powers real-time agent workflows, multi-connector data pipelines, sandboxed execution, and versioned artifact delivery, all streaming live to enterprise customers. Reliable async workers, low-latency APIs, and precise observability are not nice-to-haves here. They decide whether customers trust the system.

Your mission: keep this engine reliable and scale it as we grow.

What You Will Do

Own the FastAPI platform. Design, extend, and operate the core services powering agent orchestration, connector management, schema resolution, streaming chat, and sandboxed execution. Async handlers, SSE and WebSocket support, Pydantic v2 validation, SQLAlchemy with Alembic migrations against PostgreSQL.

Build and scale async workers. Operate Celery workers backed by Redis and RabbitMQ for schema fetching, task routing, stuck-task detection, and real-time notifications. Understand failure modes at the worker level, not just the API level.

Own the context layer pipeline. Build and operate the ingestion pipeline that processes enterprise documents, extracts and ranks business concepts, and builds the structured knowledge layer that agents reason over. This covers connector integrations, chunking strategies, and the data contracts between upstream sources and the agent layer.

Manage data connections at scale. Build and harden runtime connectors to Snowflake, DuckDB, Databricks, BigQuery, and other warehouse and SaaS sources. Handle encrypted credentials, OAuth flows, and live schema discovery. Make connections stay alive, fail cleanly, and recover fast.

Instrument everything. Own the observability stack: Prometheus and Grafana, structured logging with correlation IDs, OpenTelemetry tracing, health endpoints. P99 latency and error budgets are yours to define and defend.

Ship and operate on AWS. Docker-based deployments, Nginx, Terraform, GitHub Actions CI/CD. Write runbooks and post-mortems anyone can use to debug at 2am. Harden secrets management and SOC 2 logging.

Collaborate across teams. The platform serves LangGraph-based agent workflows and React frontends. Design API contracts that enable sub-second streaming responses and zero-downtime releases.

What We Are Looking For

• 5+ years building and operating production Python services
• Strong bias for ownership: you identify problems, propose fixes, and drive them to closure without supervision
• Deep FastAPI expertise: async handlers, dependency injection, middleware, SSE streaming, WebSocket
• Solid Celery and Redis knowledge: retry logic, task routing, idempotency, worker failure recovery
• Hands-on with Docker, Linux, and AWS deployment
• Experience with Terraform or equivalent infrastructure-as-code tooling
• Production observability mindset: Prometheus, Grafana, structured logging, distributed tracing, alerting
• Proficient with type hints, pytest, and modern Python packaging
• PostgreSQL, SQLAlchemy, and Alembic in production
• Clear communicator: your design docs and PRs show first-principles thinking

Bonus Points

• Experience with Snowflake, DuckDB, or Databricks connector patterns
• Prior work integrating LangGraph or LangChain workflows into a production API layer
• Exposure to document processing pipelines, chunking, retrieval, or knowledge graph construction
• Contributions to open-source backend or infrastructure tooling
• Experience operating under SOC 2 or equivalent compliance requirements

Life at Terrabase

Sharp, fully remote team shipping to enterprise customers weekly. Real ownership, generous cloud budgets, and a culture that prizes reliability over ceremony.

Terrabase is an equal-opportunity employer. We celebrate diversity and are committed to building an inclusive environment for every team member.