JobHunter

A volume problem with an honesty problem sitting inside it.

Job searching in Melbourne's data and AI market in 2026 is a volume problem. Two hundred-plus new listings a day across multiple boards. A hundred-plus applicants per role. The quality of the match is often hard to tell from the description alone. The default workflow — manual review plus a one-size-fits-all CV — wastes time on bad matches and undersells the good ones.

Underneath the volume problem is an honesty problem. A single LLM asked "is this job a good fit for Harry?" will find reasons to say yes. Language models, by default, are cooperative. The moments in a job search where you most need a cold second opinion are the moments an LLM will cheerfully supply the hottest first opinion.

I wanted a system that read every relevant listing daily, scored them with honesty rather than optimistic enthusiasm, produced tailored application materials only for the genuinely promising ones, learned from my apply/skip decisions over time, and cost less than a coffee per day to operate. That last constraint shaped the design more than any other.

Five stages. Cheap filters first, expensive reasoning last.

A few details worth naming:

• The evidence bank. A Notion database with structured career bullets — each with a STAR story, technologies, strengths, and metadata about when to use them. A keyword match against the scored job pulls the most relevant evidence into the prompt for material generation. CVs are composed, not templated.
• Lane-aware classification. Jobs get classified into one of three career lanes (applied ML, analytics, data PM) before scoring, using a deterministic keyword-based classifier (zero LLM cost). Each lane has its own positioning, CV base, and priority skills — so the material generation runs with lane-specific context.
• The Devil's Advocate prompt. Not "score this job negatively" — that produces low-quality adversarial text. The real prompt asks: "Find the three strongest reasons this candidate would not make it past a phone screen for this role. Be specific." That reframing produces useful criticism instead of performative pessimism.

The evaluation story, honestly. My production eval signal is the feedback loop — apply rate, skip rate by source, skip reason patterns. When the skip rate climbs above baseline, something drifted: either the scorer, my preferences, or the market. That's weaker than a proper regression test on a golden set, but it's the honest version of continuous eval for a single-user system. For a team-scale system, I'd invest in the golden set and per-prompt regression testing — this is the place I'd expect to learn more than I teach.

Cost shape is a product decision.

Two product-level lessons worth carrying forward.

Cost shape is a first-class design constraint, not an operational concern. The difference between a system that costs $1/day and $15/day isn't money — it's whether you actually use it. The $1/day system runs every day. The $15/day one gets turned off after a week. When I'm thinking about AI product features, "what does this cost per use at scale" is now one of the first questions, not one of the last.

Honest eval is worth engineering for. The Devil's Advocate pattern is structurally simple — a third agent, a specific prompt, a consensus score — but it prevents the category of failure where the model produces CVs for jobs that will never call back. Anyone can throw an LLM at a problem. Knowing when the LLM is lying to you is the harder craft, and it's where data science training pays off in AI product work.