From Messy Documents to Governed Knowledge: What Our Hackathon Revealed About AI Agents

Most of an organization's knowledge lives outside structured systems: slide decks, meeting notes, contracts, feedback forms, and old project folders. This knowledge was the focus of a recent Datashift hackathon. We brought consultants together with a practical assignment: take content from previous editions of our annual bootcamp and turn it into structured, searchable, governed knowledge to help shape the next edition.

On paper, the task sounded simple. Feed the material to AI agents, let them scan and classify it, use the output to design the next curriculum faster. In practice, the exercise exposed a harder question: how do you make AI useful when the information it needs is messy and fragmented?

The capability was there, but the quality was not

The agents could store the material and find documents on a given topic. The harder challenge was classifying that material meaningfully and generating output actually useful for curriculum design. Without the right context, an agent may answer a question using an outdated document, reuse a slide that no longer reflects the company's position, or combine correct fragments into a response that sounds convincing but does not match how the organization works. The work does not disappear. It moves into review, correction, and risk management.

The missing piece: a semantic layer

Access to documents is not enough. Agents need a semantic layer that tells them how to interpret information, how to store it, and how to use it. That means defining the concepts that matter: learning objectives, trainers, target audiences, curriculum versions, and the rules that govern them. Which content is approved? Which is outdated, sensitive, or still under review?

This layer is used twice: when agents process the archive, and when they reason over that structured knowledge to generate new output. Documents stop being isolated files. They become part of an active knowledge base, connected to business context and governed by clear rules.

What we learned

Models and prompts are not enough. Agents need structure when they store information and that same structure when they reason over it later. Organizations do not need systems that generate fast answers from messy data. They need systems that turn scattered content into governed knowledge and use that knowledge to support better decisions and reliable output.