Personal Project | 2025-2026

AskJustin

A public AI agent trained on my real work stories, so hiring managers can ask questions a resume cannot answer.

Solo project · 6 weeks at ~5 hrs/week · Live at ask.justinschaeffer.com

The Problem

A resume gives you the outline of a career. It does not tell you how someone thinks, handles ambiguity, or actually builds. Two people feel that gap, and each makes it worse for the other.

The hiring manager reads resumes in 30 seconds. Judgment, ambiguity-tolerance, actual craft: none of it survives pattern-matching. By the time a real conversation happens, the manager has already decided whether to be curious or defensive.

The candidate (me) has been job-searching since September 2025. Strong roles evaporate at the recruiter screen, before anyone with domain judgment looks at the work. Every application is a bet that a resume is a good enough proxy for me. It is not.

What I knew going in came from two sources. First, direct experience: I had run dozens of applications and could see where signal was leaking. Second, a market observation: hiring managers were fatigued by AI-generated applications and newly skeptical of surface polish. I did not need user research to confirm the problem. I was the user, and my sample was every conversation I had had that quarter.

Key reframe

Building an AI product end-to-end IS the portfolio piece for a PM role. The product demonstrates the skills it claims to show. That reframe collapsed two projects into one: the job-search tool and the portfolio case study became the same object.

Try it: ask.justinschaeffer.com →

What I Explored and What I Cut

The two most important design decisions on this project are things I did not build. Both were tempting. Both would have looked more sophisticated. Both would have made the product worse.

Rejected: Persistent chatbot memory

The obvious 2026 pattern is a chatbot that remembers who you are, greets you back, threads context across sessions. It looks modern. It reads as product sophistication.

I cut it for three reasons. Cost: persistent memory means longer context on every turn, which breaks the sub-$1/month economics and widens the abuse surface on a public endpoint tied to my personal API key. Fit: the audience is hiring managers doing a 3-to-8-turn evaluation, not returning users building a relationship. Optimizing for retention is optimizing for the wrong verb. Trust: a memory-ful agent quietly collecting behavioral data on hiring managers is a bad look for a candidate. Ephemeral is the honest default.

Kept: 6-question session cap

Visible as a turn counter. It communicates "focused evaluation, not endless chat," and it caps cost exposure without introducing a paywall.

Rejected: Letting the agent speculate

An LLM will happily answer any question. When a hiring manager asks about a project the corpus does not cover, the default behavior is to synthesize something plausible from adjacent context.

I cut it hard. A single fabricated PM story blows up the entire premise. The agent's only defensible claim is that it speaks for me. The moment it starts inventing, that claim is false, and the case-study argument I am making with this product collapses.

Kept: Explicit "I don't know" fallback

An "I do not have that in my context, contact Justin directly" response, engineered into the system prompt with concrete anti-patterns rather than abstract rules. "I don't know" became a first-class response, treated in the eval rubric as a passing behavior when appropriate.

What connects the two rejections

Both were about resisting the pull of "more capable-looking is better." The product's real value is a narrow honesty contract with the hiring manager. Every feature that would have muddied that contract got cut, even the ones that would have improved the demo.

Constraints

Time: 5 hours per week. Real ceiling. Every scope decision had to survive an "is this worth 20% of a week's build budget" test.
Cost: Near-zero. Target under $1/month at ~20 queries/day. Ruled out most vector DB options, heavier models on the default path, and forced prompt caching from day one.
Timeline: 6-8 weeks to a usable V1. The job search was live. A portfolio piece that lands after the search ends is worthless.
PM, not career engineer. Pushed the stack toward things I could reason about end-to-end, and away from architectures I would need a specialist to explain in an interview.
Public endpoint, personal API key. Cost-attack surface from launch minute one. Multi-tier rate limiting had to ship in V1.
Portfolio-worthy quality bar. Every technical and product decision had to be one I could defend in an interview conversation.
Privacy of source material. No proprietary employer docs. Derivative story-format only, with a hard split between interview-prep originals and the shipped corpus.

Architecture and Stack

Framework Next.js 16 (App Router) on Vercel Hobby. Streaming LLM support is native. Deploy is a git push.

Generation Claude Haiku 4.5. Retrieval + synthesis over a curated corpus does not need Sonnet-level reasoning. Sonnet triples per-query cost for gains I could not reliably measure.

Embeddings Voyage AI voyage-3.5-lite. Anthropic does not ship an embeddings API. Voyage is the partner path, with a free tier that covers this project's lifetime.

Retrieval Flat-file cosine similarity, embeddings bundled with the app. Zero infra, zero cold-start. A vector DB is unjustified at 60 chunks and would be a resume-line choice, not an engineering one.

Streaming Vercel AI SDK. First token in under a second changes the perceived latency entirely.

Rate Limiting Upstash Redis: per-IP, per-session, and global daily cap.

Query Logging Supabase, RLS-locked, IP-hashed with a salt. Fire-and-forget. Real signal source for corpus iteration.

What I Designed

Solo project. Every design decision is mine.

The corpus. Roughly 60 short work stories in STAR format, one per file, metadata inline. Chunking by narrative unit rather than fixed token count. Two independent boundaries enforced at ingestion: privacy and factual accuracy.
The grounding contract. System prompt engineered with concrete anti-patterns, not abstract rules. "If the retrieved chunks do not contain a specific dollar figure, do not produce one; say 'I don't have that in my context.'"
The three starter chips. They set the register of the conversation, demonstrate the corpus's range in one glance, and lower activation energy for the first turn.
The disclaimer copy. One sentence, visible below the composer at all times. Sets the honesty contract before the first token.
The "How I built this" reveal. Progressive disclosure. Both audiences (just-ask vs. want-the-build-details) are first-class.
The session cap and its framing. Six turns per session, shown as a counter, framed as "focused evaluation" rather than "you have been rate limited."

What I did not do personally: I used Claude Code as a coding assistant throughout the build. The judgment work was mine. The typing was assisted.

Post-Launch Results

Honest framing: site is soft-launched with noindex on. Intentionally not discoverable through search. Small sample by design.

Grounding drift (the failure mode being watched)

Pre-launch drift instance: "Tell me about a build-vs-buy decision" surfaced a coherent story with specific numbers ($400K in-house build cost, 2 engineers, 12-month timeline, $150K subscription, 4-month payback). Every number was fabricated at the story-bank layer. Root cause: plausible figures got baked in when the story bank was written. Fix: full corpus fact-audit before embed, and a policy that story-bank inheritance is not verification.

Corpus gaps from real logs

Logged queries across 10 days

Thin-retrieval gaps identified

Post-launch fabrications detected

Three questions hit thin retrieval: compensation expectations, handling stakeholder resistance, working in controversial industries. These gaps were only visible from real usage, not from reading the corpus. Validated that query logging is a first-class product-signal source.

What worked

Honesty contract holds. No fabrication detected post-launch.
Soft-launch bet was correct. Production system to test against without exposing gaps to the target audience.
Eval harness catches things missed by eye. Example: "no contractions" rule over-applied to possessive apostrophe-s.

What I'd Do Differently

Distribution is the actual unsolved problem. The agent only helps if I clear the ATS and recruiter screen first. Would design the distribution model in from day one.
Build the eval harness in Phase 1, not Phase 2. Shipped product first, built eval framework after. Right for launch speed, wrong for iteration speed.
Wider fact-audit up front. Systematic pass at ingestion (cite the source for every number in a comment) would have been faster than the incident-driven audit.