travissmith
engineering/README.md

AI engineering

Fourteen implemented modules covering retrieval, LLM interfaces, MCP, agents, evaluation and fine-tuning. Each case pairs the implementation with its recorded result.

How an AI application fits together 14 fundamentals
LLM inference and provider interfaces

An existing language model receives a context window and predicts output tokens. Provider interfaces let applications choose local models, route requests, and handle fallback without tying the application to one provider.

The portfolio demonstrates inference with existing models; foundation-model pretraining is not demonstrated.

Tokens, grammars, and structured output

Tokenization turns text into the pieces a model predicts. Grammar constraints restrict legal next tokens, while validators check whether the completed output matches an expected structure.

Structural validity does not establish factual accuracy. Recorded grammar results use bounded task and grammar subsets.

Prompting and context assembly

Applications assemble instructions, examples, retrieved evidence, and tool results within a finite context window. Clear context and explicit handling of missing evidence shape what a model can answer.

Supplying evidence or reference speaker labels does not independently establish answer truth or autonomous speaker diarization.

Embeddings and vector indexes

Embedding models turn content into numerical vectors. A vector index searches those vectors by distance, with filters and recall measurements determining how well it finds relevant candidates.

The SIFT vector benchmark measures index behavior on image descriptors; it does not measure language-retrieval accuracy.

Chunking and source metadata

Chunking splits documents into retrievable units while preserving source identifiers and useful structure. Fixed, structural, and parent-child strategies trade retrieval precision against the context available for an answer.

The recorded document experiments use a synthetic corpus and do not establish one universally best chunking strategy.

RAG, citations, and abstention

Retrieval-augmented generation finds source passages, supplies them to a model, and links the answer to evidence. An answerable-versus-unanswerable evaluation checks whether the system answers or abstains appropriately.

A valid citation tag identifies a supplied source; it does not prove that the source supports every associated claim.

Hybrid and graph retrieval

Hybrid search combines lexical matches with vector similarity. Graph retrieval adds entity relationships and provenance, allowing retrieval experiments to compare different ways of finding supporting evidence.

Recorded reranking and graph variants include regressions. Relationship provenance does not establish the truth of an extracted relationship.

MCP and tool contracts

The Model Context Protocol exposes discoverable tools with structured arguments and results. Servers can wrap retrieval, queries, files, or deterministic engines while validating calls and recording refusals.

MCP provides a tool interface, not model intelligence. Passing fixture calls does not establish an agent can complete a real task correctly.

Agents and workflow orchestration

Agent workflows coordinate planning, tool calls, dependencies, retries, persisted state, and budgets. Traces show how the workflow progressed, while outcome grading checks whether it solved the task.

Workflow completion differs from correctness. Recorded retrieval-loop budgets do not form a hard ceiling on all subsequent synthesis work.

SQL, metadata, and persistent state

Schemas give stored data meaning. Query validation and read-only execution control access, while persistent indexes and workflow state support recovery and scoped memory across requests.

Local Knowledge Store has unfinished knowledge-base and concurrent-store paths. Its implemented memory mechanisms do not establish a complete privacy guarantee.

Evaluations and calibrated judgments

Evaluations compare outputs with defined ground truth, separate held-out examples, measure agreement, and inspect regressions. Judge calibration helps distinguish a persuasive model assessment from a reliable score.

Constructed tasks, model judges, and historical sports evaluations have different evidence conditions; their scores are not interchangeable public benchmarks.

LoRA and model adaptation

Low-rank adaptation trains a small set of additional parameters on an existing model. The pipeline covers data curation, completion masking, adapter training, merging, GGUF export, and local serving.

The recorded CPU training run is partial. Its small synthetic evaluation does not establish a general improvement or foundation-model training from scratch.

Caching, routing, cost, and latency

Exact and semantic caches reuse eligible results within explicit scopes and versions. Routing, fallback, and telemetry help compare model calls, response time, and correctness across configurations.

Fewer model calls do not necessarily mean faster responses. Recorded cache experiments include additional embedding latency and notional cost estimates.

Independent operations and recovery

Standalone entry points, persisted state, logs, and detached launchers let software run independently of an assistant session. Local models and explicit degraded modes keep provider availability from becoming the application lifecycle.

Launcher and recovery implementations are source evidence. Installed scheduler configuration, current uptime, and reboot recovery are not verified by the source review.

14 modules

Document RAG

AI01 · Retrieval & data

Markdown ingestion, multiple ways to split documents, metadata, combined keyword and meaning-based search, surrounding context, model answers, citation checks and abstention. Local Ollama and OpenAI-compatible adapters; separate scripted test backend.

→

Agentic RAG

AI02 · Retrieval & data

Planning, repeated retrieval, sufficiency checks, query reformulation, synthesis, routing, evidence windows and traces.

→

Graph RAG

AI03 · Retrieval & data

Ontology-validated triples, entity resolution, graph persistence/community summaries, local/global queries and provenance.

→

Hybrid Search

AI04 · Retrieval & data

Keyword search, exact vector matching, metadata filtering and combined rankings; real embedding and reranking models; saved indexes and comparisons of individual components.

→

Vector Search

AI05 · Retrieval & data

Rust HNSW/flat search, SQ8/PQ vector compression, metadata filters, WAL/snapshots/recovery mechanisms and Python bindings.

→

MCP Servers

AI06 · Tools & workflows

Four Python MCP servers and 11 tools for document search, SQL, filesystem and web access; SDK stdio harness; schemas, restrictions and audit chains. Additional TypeScript server code.

→

Agent Workflows

AI07 · Tools & workflows

Dependency scheduling, persistent state/events, model/tool loop, schema validation, repair, concurrency and traces.

→

Model Routing

AI08 · Models & output

Provider interfaces, model routing/pins, fallback, circuit state, budgets/rate limits, caching, streaming and observability.

→

Text-to-SQL

AI09 · Tools & workflows

Question/schema linking, few-shot memory, model-generated SQL, AST policies, read-only execution, bounded repair and masking.

→

Structured Output

AI10 · Models & output

Tokenization/vocabulary, grammar compiler, legal-token masking, local Transformers inference and external output validation.

→

Model Evaluation

AI11 · Evaluation & optimization

Reference answers, question difficulty tiers, test contamination checks, model and judge adapters, grading rubrics, agreement checks, sampled regression comparisons and saved checkpoints.

→

Semantic Caching

AI12 · Evaluation & optimization

Scoped/versioned keys, exact/semantic lookup, number/negation guards, persistence, expiry and invalidation.

→

Skill Routing

AI13 · Evaluation & optimization

Skill validation/loading, calibrated routing and abstention, paired task experiments and rubric evaluation.

→

LoRA Fine-Tuning

AI14 · Models & output

Data curation/decontamination, completion-only masking, real LoRA training, adapter merge, GGUF export, local serving and evaluation.

→