What we were testing ?
Repeated professional work produces a large amount of useful behavioural signal :
- When we correct an answer, we implicitly reveal preferences.
- Repeated requests expose reasoning patterns and habits.
- Recurring frameworks show how we tend to approach similar problems.
- Unresolved discussions create future context.
However, none of this automatically becomes useful knowledge simply because the conversation has been stored. This then led us to another question:
Could completed work be analysed and distilled into reusable learning without turning the whole conversation history into permanent memory?
What we observed ?
Summarizing conversations was not enough, since it could only tell us what a conversation was about.
What we really needed was to identify patterns in how the work was performed.
The first extraction model was designed around observations such as:
ARGUMENTATION
FRAMEWORK
CORRECTION
VOCABULARY
THEME
HEURISTIC
DECISION
OPEN_THREAD
CLIENT_NOTE
While a correction can reveal a preference, we figured out that a repeated reasoning pattern can reveal a heuristic, and that a recurring framework can reveal a working habit.
The useful lesson was that these signals should not automatically be promoted into permanent instructions. They should instead help shape how future prompts and responses behave when the pattern is strong enough to justify it.
What we changed
This part of the experiment became an actual Active Learning Layer as the implementation moved beyond individual conversation summaries and started analyzing the archive itself.
The working mechanism became:
Recent work history
↓
Pattern detection
↓
Cluster recurring behaviours
↓
Assess whether the pattern is significant
↓
Create knowledge | update knowledge | create rule | flag gap | ignore
↓
Record the action
↓
Reuse validated learning in future prompts
In practice, the pattern miner analyses up to 200 user messages over a 90-day window.
- A first pass uses a lightweight model to group recurring messages and behaviours into clusters.
- A second model analyses the most significant clusters and determines whether the signal deserves an action.
The possible outcomes are deliberately limited:
create knowledge
append existing knowledge
create prompt rule
flag knowledge gap
do nothing
This matters because the system is not instructed to learn something from every interaction, as only sufficiently strong and recurring signals should produce a persistent change.
When a learned rule is created, it becomes part of a dedicated preference layer injected into future prompts.
So the implemented loop became less:
conversation → memory
and more:
conversation history
→ recurring signal
→ assessed pattern
→ controlled action
→ audit trail
→ future behaviour
What we retained ?
Three distinctions became important.
A - History is not learning.
Stored conversations remain raw material until something repeatable is identified.
B - Pattern extraction is not truth.
A recurring behaviour can justify a candidate rule, but it still needs thresholds, scope and the possibility to be reversed.
C - Learning must remain inspectable.
If the system changes future behaviour based on previous work, we need to know what changed and why.
This last point turned out to be particularly important because instead of making the agent appear to "learn" invisibly, we made learning behave more like controlled configuration generated from observed work patterns.
Takeaway
At application level, learning does not necessarily require retraining a foundation model.
A useful learning loop can start by identifying recurring signals in actual work, converting only the strongest ones into explicit rules or knowledge, and keeping those changes visible and reversible.