Menu

Managed properly, it could accelerate innovation across the business

By Femke van Engen, project lead, and Olivier Lamers, data scientist at Rewire.

Let’s play make-believe.

You manage enthusiastic non-technical teams who interact regularly with technical teams. For example, to improve workflows, flesh out new ideas, or analyse data. The problem is that there is only so much your technical teams can handle.

Still, you'd like to improve collaboration between technical and non-technical teams and, more broadly, accelerate your organisation's innovation capabilities.

Well, vibe coding might be part of the answer.

But what, exactly, is vibe coding?

What vibe coding means

Put simply, vibe coding means describing what you want in plain natural language, and an AI tool turns it into working software in minutes. No coding skills needed. No briefing an engineer who sends you back to the drawing board until you’ve figured all the unknowns. No begging a tech lead whose next free slot is in six weeks.

The result? An idea that would have taken a few sprints to render visible can become a working prototype in less than an hour.

Still on the fence? Check out this web app that was put together by a senior exec in one of our recent vibe coding workshop. She had never coded before, yet built a Property Selector in under an hour — a desktop app that automatically scores, filters, clusters and assigns scraped real estate listings into optimised scout-visit schedules, helping investment analysts identify the best opportunities and plan field visits more efficiently.

The best part: no particular setup was needed. Just a Lovable account and her laptop.

Figure 1. From vague idea to detailed prototype in under an hour.

Why vibe coding accelerates innovation

The practical benefits of translating an idea into a working prototype that fast are straightforward:

  • Speed to output. An idea becomes something your non-technical teams can show and react to in minutes instead of weeks. The distance between thinking and doing collapses.
  • Better conversations. A working prototype beats a slide deck or a written brief. Fuzzy ideas become concrete. Requirements and variables are better defined. Your non-technical teams are better prepared and the translation gap from the person who understands the business problem to the person who can render it in code is vastly reduced.
  • Lower cost of being wrong. If the concept does not hold up under scrutiny, your non-technical team finds out in an afternoon, not after a development sprint.


What vibe coding won't do

Ok. We’ve convinced you since you made it this far. There are, however, a couple of caveats.

First of all, the constraint now is conceptual, rather than technical: the quality of what you generate depends almost entirely on the quality of your thinking before you start. Define what you want (and don’t want!) clearly and a single well-constructed prompt will produce something impressive. Let’s be very clear: the best results tend to come from spending roughly 80% of the available time on definition and scoping, and 20% on generation. The AI is not a substitute for thinking.

Second, do not rush to make your IT team redundant. Vibe coding is great to build prototypes running on a laptop that overheats when you have more than 3 browser tabs open. But don’t expect to put this in production within hours. There are simply too many issues that need to be addressed first: cybersecurity, data sovereignty, governance, integration with legacy systems, to name a few. So stay friends with your technical team, and use vibe coding to improve the quality of the discussions between your technical and non-technical teams.

The main takeaway

Vibe coding won't replace good thinking, or your technical teams. It will show you and your business teams, whether an idea is worth their time, faster than ever.

The best prototypes start with the best thinking.

You don't need to know or write code. You need to describe what you want in terms that are clear to an AI. That skill alone will let you and your teams turn ideas into working prototypes, improve discussions with technical teams, and accelerate innovation.

Find out more

The financial industry often regards regulation as a constraint. Agentic AI is turning it into an advantage.

Across banks, insurers, and asset servicers today, a similar picture emerges. On the one side, AI initiatives running in parallel without ever being absorbed into daily operations. On the other, initiatives that reliably create value. The difference between the two groups lies in the approach.

Let's focus on asset servicing in particular, because the industry tends to assume regulation is what holds it back. The opposite is closer to the truth: Fund administration, transfer agency, and the middle office already run as a sequence of controlled steps with defined owners, defined inputs, and a check at every gate. That structure is precisely the one in which AI, and agentic AI in particular, does its best work.

When AI really delivers in asset servicing

Before examining what separates success from failure, it is worth starting with what actually works. Across our projects, three factors consistently drive AI value.

  • First, a clear goal inside the process. For example, "cut the time from a complete subscription pack to a cleared investor onboarding by half, so the transfer agency team can absorb a fund launch without adding headcount" is a goal a team can tackle immediately, and one against which success can be measured. On the other hand, "use AI in onboarding" will fail on both counts.
  • Second, the right data, understandable and reliable. Asset servicing sits on rich, structured data: investor records, subscription and redemption flows, fund accounting ledgers, custody positions, prospectuses, etc. Here the regulated firm has a quiet advantage, because reporting obligations and audit requirements have often already forced a level of data discipline that less regulated industries lack.
  • Third, results where the decision is made. In asset servicing the point of decision usually sits at a control gate: the oversight check before a NAV is released, the four-eyes review on a payment, the compliance sign-off on a new investor. Good AI does prepares the decision inside the tool the reviewer already uses, surfaces the exceptions that need a human eye, and leaves a trace of how it got there.

Where these three elements come together, impact emerges that is measurable and scalable. And it is the pattern behind the AI work that actually takes hold in regulated operations.

The structure is the advantage

It helps to be precise about why asset servicing is such fertile ground. A fund's operational life runs as a chain of distinct stages, each with its own rules, its own data, and its own responsible owner. Investor onboarding and the KYC and AML checks that gate it. Subscription and redemption processing. NAV calculation and the oversight that validates it. Reconciliation across custody and accounting. Regulatory and investor reporting at the end. Between every stage sits a handover and, almost always, a control.

This is the same architecture that multi-agent AI systems are built around. Specialized components do focused work within clear boundaries, and they hand off to one another at defined points. A single, monolithic AI asked to do everything across this chain tends to drift and to hallucinate. Specialized agents with narrow remits and clean handover points behave far more predictably, which is exactly what a regulated process requires.

The regulatory apparatus that asset servicers sometimes experience as friction is, in this light, scaffolding. The four-eyes principle is human-in-the-loop, written into the operating model long before anyone spoke of agents. AIFMD and MiFID II define who is accountable for what. DORA, which has applied to financial entities across the EU since January 2025, has pushed firms to map their operational and ICT dependencies in detail, which is the same map an agent needs to act safely. The EU AI Act, now phasing in, asks for exactly the documentation, oversight, and traceability that a well-run fund administrator already produces for its regulators. Under the CSSF and its peer supervisors, governance is not a feature to be bolted on. It is already in the building.

The constraint, then, is coordination: getting the right specialized capability to the right step, with a clean handover and a human at the gate. That is an architecture problem, and it is one this industry has been solving by hand for decades.

Figure 1

Every rule the asset servicing industry already follows is a guardrail agents need.

Together: the control stack agentic AI requires.

Tap or click a milestone for the detail behind it.

Each rule that once felt like a constraint installed a guardrail: accountability, audit trail, human oversight, a mapped environment, delegation control. That is the exact control set agentic AI needs, and the regulated firm already operates it.

Four patterns that mark successful delivery

However different mandates and maturity levels may be, the firms making visible progress attend to the same four themes.

1.  Enable the right people, not only the technologists

A common assumption is that AI competence is mainly a matter of hiring more data scientists. In practice the larger leverage comes from the people who truly know the work: the fund accountants, the transfer agency officers, the oversight and quality managers who can tell a genuine exception from a benign one. They carry the context that makes an AI initiative take off. They know how a redemption will be processed, which investor data can be trusted, and where a NAV error occurs. When these people are deliberately enabled, in short, practice-oriented sessions on real cases, a technical experiment becomes a working tool. The reliable pattern is a tandem: domain expert and AI specialist working side by side, domain knowledge flowing into requirements, AI results flowing back into the process. The time from idea to first productive version typically halves.

2.  Focus on a few, genuinely viable use cases

The portfolios that work are deliberately lean. Three to five use cases with visible leverage on core processes, each with a named owner and clear success criteria. Equally important is the discipline of finishing things. When a pilot has exhausted its potential, it is closed deliberately, and it makes room for the next. An illustrative picture from operations: rather than spreading effort across onboarding, reconciliation, and reporting at once, a successful team concentrates on one defined step, for example pre-checking incoming subscription documents against the legal requirements and flagging what is missing before it reaches the reviewer. The benefit shows up less in a single headline number than in a team that is noticeably relieved and a queue that clears more consistently.

3.  Anchor AI directly where decisions are made

The last mile decides between AI introduced and AI in use. The firms that are consistent here build AI into the workflow: a flag inside the existing oversight tool rather than another screen, clear accountability for who acts on it, and a defined escalation for when the model is unsure. AI then becomes not an add-on but an extra pair of eyes and hands at the gate. An illustrative picture from oversight: a NAV review assistant recalculates key positions in parallel, compares them against the administrator's output, and presents only the discrepancies that exceed the NAV error threshold, with the underlying data attached. The reviewer still releases the NAV, as always. They simply see sooner where to look.

4.  Treat AI as a product, not a project

Software projects have an end. AI systems do not. Successful firms give their AI systems the attention they give any production capability: a responsible owner, an operating concept, regular quality checks, versioning, and cost control. The effect is that the system stays reliable as inputs and processes change, and trust in it grows steadily. This is familiar territory for a regulated firm, because model validation, ongoing monitoring, and documented oversight are already part of the vocabulary.

Agentic AI as the natural next step

These four patterns lead naturally to where the technology is heading. The most useful way to picture agentic AI in asset servicing is a small set of specialized agents working in parallel within human decision gates. The point is that this division of labour decouples the depth of analysis from the number of people available to do it. Routine, synthesis, and scale move to the agents. The human contribution concentrates where it has always created the most value, in judgment at the gate.

Figure 2

The fund lifecycle is a chain of decision gates

Agents prepare. A person decides at every gate. Nothing passes without an audit trail.

Gate 1

Investor onboarding (KYC / AML)

Agent prepares

Reads onboarding and KYC documents, carries out Adverse Media Screening, screens against sanctions and PEP lists, flags missing or inconsistent items.

Human decides

Compliance officer (MLRO) approves the investor.

Pre-check incoming subscription packs and flag gaps before they reach the officer.

Gate 2

Subscription and redemption

Agent prepares

Validates dealing instructions against fund rules and cut-offs, checks cash, isolates exceptions.

Human decides

Transfer agency officer authorises the transaction.

Auto-validate dealing instructions and surface only the exceptions.

Gate 3

Reconciliation (custody and accounting)

Agent prepares

Matches positions and cash across custodian and accounting records, classifies breaks, proposes a resolution.

Human decides

Reconciliation analyst confirms the match before NAV.

Auto-reconciliation with explained, pre-classified breaks.

Gate 4

NAV calculation and oversight

Agent prepares

Recalculates key positions in parallel, compares against the administrator's output, flags NAV errors with the data attached.

Human decides

Oversight or fund accountant releases the NAV (four-eyes).

A NAV review assistant that shows the reviewer where to look first.

Gate 5

Regulatory and investor reporting

Agent prepares

Compiles data across systems, drafts filings and investor reports, checks against templates.

Human decides

Reporting lead reviews and signs off.

Assemble and pre-validate regulatory report packs.

Between every stage sits a control. Agents do the preparation, a person decides at the gate, and nothing passes without an audit trail. This is the decision-gate architecture, drawn straight from the way fund administration already works.

Notice the feedback loop: regulation shapes the architecture of the AI system; the system, in turn, strengthens compliance — producing a more complete audit trail, faster.

It's not the tools. It's the capability

The firms that stay ahead build capability rather than acquire tools. Teams that understand and steer AI. Data products that are maintained and trusted. Systems that run in everyday operations instead of a test environment. And an operating logic that secures the value tomorrow as well. The most regulated corner of finance does not need to wait for the rest of the industry to work out how to govern AI. It already knows. The handover points, the four-eyes checks, the audit trail, and the documented oversight that can feel like weight are, for agentic AI, the blueprint.

Footnotes

* A further milestone reinforces the onboarding gate: the EU's 2024 anti-money-laundering package. Its single, directly applicable Anti-Money Laundering Regulation (AMLR) takes effect across the EU on 10 July 2027, and a central supervisor, AMLA, has been operational in Frankfurt since July 2025. In agent terms it sharpens a guardrail the chart already shows: verified, structured client due-diligence data and one harmonised standard for what must be checked before an investor passes.


About the authors

Dr. Philipp Diesinger is a data science executive with over 15 years of global experience driving AI-powered transformation across industries. He has led high-impact initiatives at Boehringer Ingelheim, BCG, and Rewire, delivering measurable value through advanced analytics, GenAI, and data strategy at scale.

Dorthe van Waarden leads organizations in creating tangible impact by developing and implementing advanced algorithmic systems. She is passionate about combining the power of humans with the potential of AI to achieve the best outcomes and doing this in a responsible way. Dorthe holds a MSc in Mathematics from the University of Amsterdam.

Agents reward organisations that ask harder questions first.

The organisations getting compounding value from agentic AI share one trait: they invested in getting the foundations right before scaling.

We've built agent systems across financial services, energy, telecom, and public sector, from early proofs of concept to production systems running at scale. We've seen what separates the deployments that deliver from the ones that don't.

Let's find the answers for your situation

Organisations that govern their knowledge are about to pull ahead of their competition

By Marcel Mol and Job van Zijl, Rewire.

Companies looking to scale agentic systems quickly run into a limitation: AI is only as intelligent as the knowledge it can draw upon. While LLMs let organisations open up sources that were previously hard to utilize, such as PDFs and Word documents, the fact is that businesses aren’t documented the way they need to be to fully leverage today’s technology. 

Three problems arise:

  • Knowledge is tacit, internalized by employees and never written down.
  • Knowledge is imperfect, with out-of-date information, discrepancies between sources, and so on.
  • Knowledge is missing.

To solve these problems, we introduce the knowledge engine methodology, underpinned by a broader vision: the codified enterprise. This is how we define the winning organisations of tomorrow — where knowledge compounds rather than churns, and where agents can be deployed reliably at scale.

How the codified enterprise develops asymmetric competitive advantage

Today, organisational expertise is siloed by design: it belongs to individuals and teams, rather than the organisation as a whole. People do most of the work, and AI augments specific processes as and when needed.

Agentic AI flips is set to flip the equation around, with agents doing most of the operational work, while people oversee strategy, creativity, and edge cases. But this is only possible with a governed knowledge layer that agents and humans can query and maintain. Knowledge — rather than churning out of the door whenever someone leaves — becomes a compounding asset.

Think about what this means operationally: the codified enterprise can onboard a new agent in hours rather than weeks, because the knowledge it needs exists in structured form. It can scale a process across markets with improved institutional memory. It can respond to regulatory change by updating a knowledge graph and propagating the change systematically, rather than hoping that the right people update the right documents and that those documents get read.

The non-codified competitor, meanwhile, is still dependent on specific individuals, still recreating knowledge from scratch, still debugging agents that behave inconsistently because their prompts are patching over gaps in an unstructured information environment.

Building tomorrow’s codified enterprise

Building a codified enterprise starts with addressing three challenges:

  • Making all knowledge machine-readable. The issue here is that most business knowledge lives in people’s heads, unstructured documents, or process tooling that agents cannot access. None of it is governed or current.
  • Knowledge extraction taking human effort. Even where documentation exists, it is incomplete and ungoverned. The decision rules that experienced people apply intuitively — when to escalate a claim, how to handle an exception — are rarely written down. Structuring this for autonomous agent use still requires people to validate and sharpen what gets captured.
  • Knowledge is never finished. Processes change. Thresholds shift. Regulations evolve. A knowledge system not designed to update continuously will decay — and agents grounded in stale knowledge become unreliable.

The knowledge engine methodology uses process documentation as input, and allows expert validation of the acquired knowledge. But there’s more to it. As a video is worth more than a thousand words, check out the video demo below. You’ll see how the knowledge engine in action, as well as:

  • How knowledge can be extracted following an expert-defined ‘knowledge skeleton’, applied to building a knowledge graph from internal policy documents
  • How to the extracted knowledge can be visualized, and gaps and contradictions can be made visible
  • How AI can help closing these gaps by suggesting fixes, which can be approved or changed by experts

Watch the video

You are currently viewing a placeholder content from YouTube. To access the actual content, click the button below. Please note that doing so will share data with third-party providers.

More Information


Curious about what a knowledge engine could do for your organisation? Get in touch!

The biopharma industry already operates like a multi-agent system, just without the agents.

Recent AI agent performance data is revealing: on focused tasks (two-hour sprints), top agents score four times higher than human experts. But stretch the same agent to a 32-hour task, and humans outperform it two-to-one. Agents achieve near-perfect success on tasks that take a human under four minutes; below 10% success on tasks that run over four hours.

The problem is structural. Making the agent smarter will not fix it, but redesigning the task will.

Source: Kwa et al. (2026). Measuring AI Ability to Complete Long Software Task. https://arxiv.org/pdf/2503.14499

Why the monolith fails, especially in pharma

The temptation to build one generalist agent and load it with context across regulatory, scientific, commercial, and clinical domains is inefficient. A generalist agent trying to manage regulatory compliance and synthesise scientific evidence and model reimbursement pathways will simultaneously hallucinate across domains won't understand deeply enough. It might be 90% right on average, but you can never know which 10% is wrong. In biopharma, that's an unacceptable risk.

When a generalist agent fails, everything fails: no isolation, no fallback. One flawed reasoning step cascades through the entire chain. Software engineering learned this lesson in the 2000s. One giant codebase, impossible to debug, impossible to scale. The answer was microservices: independent modules, independent deployment, independent validation. The same logic now applies to agentic AI in biopharma.

The orchestration answer: specialised agents with human decision gates

If intelligence comes from the model, correctness comes from the architecture. To achieve this, this is what the architecture should looks like:

  • Small, specialised agents, each excellent at one narrow task. A regulatory agent that only checks compliance. A science agent that only reviews the evidence. A market access agent that only models reimbursement. Each can be validated independently, tested against known outcomes, retrained without touching the rest of the system. Need to add a new domain? Add a new agent.
  • Validation gates between agents. If the regulatory agent flags a problem, the pipeline stops before downstream agents waste cycles on an invalid pathway. Guardrails must be structural, as we explain here.
  • Humans at the right moments: not reviewing every output, but sitting at decision gates between phases. The architecture defines where human judgement matters most, rather than distributing it thinly across every step.

Now, most industries adopting agentic AI have to build governance frameworks from scratch: who validates what, where humans sign off, how to create audit trails. But here's the kicker: Biopharma already has this.

The decision-gate architecture — specialised agents operating within human checkpoints — maps directly onto how pharma already works. Regulatory affairs, medical affairs, clinical development, market access: each function already has defined review cycles, sign-off requirements, and clear accountability structures. That is the multi-agent orchestration model, without the agents.

The vast sums of money that pharma spends on regulatory compliance, usually framed as overhead, is actually a head start: the governance infrastructure that other industries are scrambling to build, pharma already operates.

What this looks like in practice

Take medical content creation. A single piece of promotional content currently moves through approximately seven sequential review cycles across medical, legal, regulatory, and localisation. Each reviewer waits for the previous one. Average cycle: six to eight weeks.

Under an orchestrated multi-agent model, a drafting agent produces the initial version. Medical accuracy agents and regulatory compliance agents run in parallel. Localisation agents adapt for local markets simultaneously. Humans approve at two defined gates: after the initial draft and after the final review. The cycle collapses from weeks to days, while the quality controls remain intact.

The same logic applies across the pipeline: pharmacovigilance monitoring, regulatory submission drafting, protocol design, site selection, KOL mapping. In each case, the opportunity is not to replace human judgement, but to remove the coordination overhead that delays it.

The gap will be measured in pipeline years

Enterprise application of agentic systems is growing at nearly 50% compound annually, and the pattern is becoming the dominant architecture of enterprise AI. For pharma, the gap between organisations that get the orchestration layer right and those still experimenting with generalist agents will very soon be measured in years of pipeline advantage.

Philipp Diesinger leads the pharma Practice Area at Rewire.


Agents reward organisations that ask harder questions first.

The organisations getting compounding value from agentic AI share one trait: they invested in getting the foundations right before scaling.

We've built agent systems across financial services, energy, telecom, and public sector, from early proofs of concept to production systems running at scale. We've seen what separates the deployments that deliver from the ones that don't.

Let's find the answers for your situation

Insights and debates from a day dedicated to transforming AI experimentation into measurable business impact.

Rewire LIVE started with a vision: to create an interactive event where participants from the business community could openly share their experiences with AI. Last week in Amsterdam, business leaders and innovators gathered to make that vision a reality.

The result exceeded expectations.

Through debates, workshops, and keynotes, attendees collectively explored what it takes to move from experimentation to meaningful impact with AI.

The next stage in the evolution of AI: introverted thinking

The day opened with a thought-provoking keynote by Wouter Huygen, Rewire CEO, who explored the paradox at the heart of Generative AI: it’s both incredibly smart and... surprisingly stupid.

Figure 1. ChatGPT, waxing digital.

So... “Are we building on shaky foundations and quicksand?”

His answer: no. Despite the limitations of current models, there is a massive opportunity overhang. With emerging toolkits and rapid innovation, today’s GenAI represents years of transformative potential.

Perhaps one of the most surprising insights was the emergence of “introverted thinking” in reasoning models – enabling transformers to refine their internal states before producing an answer. This leads to higher performance on complex reasoning tasks with smaller, more energy-efficient models. As Wouter noted, the bitter lesson of AI has, in the end, often tasted quite sweet.

Successful AI adoption: it’s all about speed and leadership

Moderated by Joe van Burik, tech editor at BNR Nieuwsradio, participants were invited to respond to a series of questions that revealed how they view AI adoption in corporate environments. Key insights included:

  • 87% disagreed with the idea that AI is “just hype.”
  • Most agreed that AI adoption should start with leadership, ideally from the boardroom down.
  • The consensus was that it’s better to be a fast follower than a slow pioneer. Speed consistently emerged as a key success factor.

Participants then debated a range of topics, from build vs. buy strategies to regulation and productivity gains.

Figure 2. Joe submitting the audience to intense questioning.

The discussion concluded with a deceptively simple question:

What do you hope AI will bring to society?

We’ll leave you to ponder the resulting word cloud (below).

Figure 3. Every cloud has a silver lining.

Data & AI in action with hands-on workshops

The day featured four practical workshops, each tackling a different dimension of AI transformation:

  • From GenAI demo to production, by Simon Koolstra & Mirte Pruppers (Rewire). Participants began with a GenAI demo and collaborated to solve the challenges that typically emerge when moving from demo to production. The exercise highlighted a hard truth: many can build demos, but scaling is where most fail.
  • Reimagine your business with data, by Freek Gulden & Nanne van’t Klooster (Rewire). This session challenged participants to reimagine their businesses once data is viewed as a value stream rather than operational exhaust.
  • Decentralized data leadership, by Daniëlle Bourgondië (PGGM Investments). Daniëlle led a lively discussion on what it takes for business domains to take ownership of their data and drive AI strategies. As she aptly put it: “This is not optimization. It’s reimagination.”
  • Roles & responsibilities in digital transformation, by Sam van Kesteren (Royal Terberg Group) & Ties Cabo (Rewire). This workshop explored how to align business and technical roles & responsibilities to overcome some of the trickiest barriers to large-scale transformation.

Figure 4. Working out GenAI best practices: One team. One dream.

Real-world lessons from the corporate world

The day wouldn't have been complete without guest speakers detailing how AI is transforming their organizations from the inside out.

  • Maarten Kramer, Chief Data Officer at a.s.r., demonstrated how agentic AI is reshaping complex insurance processes, starting with personal injury claims involving over two million documents annually. Their approach: treat AI agents like employees, with training, performance management, and oversight.
  • Tim Prins, Program Director of Autonomous Operations at KPN, outlined how KPN aims to deliver the best customer service in Europe while halving costs. Agentic AI plays a central role in this ambitious transformation.
  • Finally, Winifred Andriessen (VP of AI Excellence Center at KPN), Maarten Kramer (Chief Data Officer at a.s.r.), and Laura Brandwacht (Partner at Rewire) joined a panel discussion on AI’s impact on the workforce. They explored how AI is reshaping roles, the need for reskilling and upskilling, the enduring importance of human skills, and the leadership and culture required to make AI a success. The conversation underscored that AI transformation is as much about people and leadership as it is about technology.

Figure 5. Not just talk: the people who are transforming businesses with AI.

Key takeaway

The day concluded with drinks and conversations went well into the evening. Reflecting on the event, two themes stood out. First, we’re at a pivotal moment in history where the playbooks for leveraging AI are still being written. Second, collaboration is key. Joining forces is the best way to reimagine your business.

How business leaders are turning AI ambition into operational reality

Last week we had the honor of welcoming industry leaders and researchers to Rewire LIVE 2025 in Frankfurt — for a day of practical insights on scaling data and AI, from vision to value at scale. With a packed agenda of keynotes, case studies, and mastermind sessions, the event zeroed in on one of the toughest challenges in enterprise AI today: how to bridge the last mile — turning promising pilots into production-grade, scalable, and trusted AI systems.

We know the answer but not the question

Rewire partner Christoph Sporleder thus opened the day by challenging common strategic pitfalls in AI adoption. Whether taking an exploratory, foundational, or holistic approach, many organizations struggle to operationalize at scale due to mismatched expectations, over-engineering, or governance complexities. His conclusion on breaking it down, balancing the transformation dimensions and going the full mile towards scale were picked up in the the powerful keynote of Simone Menne, Lufthansa’s former first female CFO. She reminded us that the real challenge of AI isn’t finding answers — it’s asking the right questions. Drawing from The Hitchhiker’s Guide to the Galaxy and her experience in leadership, Simone Menne spoke to the cultural and psychological hurdles that slow AI adoption. To overcome this we need to confront fear, foster curiosity, and prioritize education, governance, and collaboration across sectors to close the gap between potential and execution.

Dr. Peter Bärnreuther from Munich Re walked us through the rapidly growing domain of AI risks, from legal liabilities and IP issues to model drift and data bias. With over 200 AI-related lawsuits already filed globally, the urgency for transparent, testable, and auditable AI systems is mounting, and Munich Re is leading the way in also providing insurance products for AI. Nikhil Srinidhi, Partner at Rewire, illustrated the growing chasm between AI ambition and data readiness. In his talk, “Mind the Data Gap,” he highlighted the widening divide caused by poor data quality, fragmented ownership, and an expanding AI vendor landscape. His takeaway: scaling GenAI means scaling data maturity — and doing it across architecture, culture, and capability. Shannon Kehoe of QuantPi went on to emphasize that trust in AI starts with predictability and transparency. She introduced a model-agnostic platform that tests AI solutions across dimensions and presents results clearly for diverse stakeholders — from data scientists to regulators. Loïc Tilman of Elia Group shared what it takes to scale AI within a critical infrastructure operator. From evolving legacy systems to upskilling teams and ensuring sovereignty in the IT infrastructure, his talk made it clear: even in high-stakes environments, digital, data and AI transformation isn’t a luxury — it’s a necessity.

AI masterminds: defining real solutions to real challenges

The event also included interactive sessions, where Lu Yu of Novo Nordisk, Dr. Pierre Fischer of Roche and Dr. Martin Paschmann of Douglas shared real-world case studies. Participants dissected challenges ranging from siloed systems to frontline adoption. Solutions focused on data culture, internal champions, and aligning science with business needs. In one mastermind session, participants unpacked how to define the value of data products — from use-case fit and quality to cloud cost savings, time-to-insight, and user satisfaction. A key insight: value must be seen through multiple lenses — technical, financial, and strategic — and agreed upon by data, finance, and business stakeholders.

Creating a blueprint for AI Transformation

Our strategies elevate the performance of your AI applications from marginal or tactical results to unequivocal, transformational successes.

Discover our Data & AI Strategy

Panel discussion: scaling AI responsibly

The day concluded with a rich panel featuring Eberhard Schnebel (Goethe University Frankfurt / Commerzbank), Dr. Stefan Rose (University of Cologne), and Paola Daniore (EPFL Lausanne), moderated by Damien Deighan, Editor of the Data and AI magazine. Topics included the privacy paradox between academia and industry; the social psychology of AI interaction in teams; Trust-building through transparency and governance; The ethical design of emotionally resonant AI systems. Their message: AI adoption is about more than models. It’s about culture, behavior, and values — and how we ensure they evolve together.

Final thoughts

Rewire LIVE 2025 reinforced a shared truth: next to data and technology, AI is a leadership challenge. The ideas, stories, and solutions shared in Frankfurt revealed a deep appetite for actionable, scalable, and human-centered approaches to Data & AI.

We’re grateful to all speakers, participants, and partners who made the day a success. Let’s keep building — together.


Want to learn more about upcoming Rewire events? Visit the Rewire events page here.

A handful of principles will set you up for success when bringing GenAI to enterprise data.

As highlighted in our 2025 Data Management Trends report, the rapid evolution of LLMs is accelerating the shift of GenAI models and agentic systems from experimental pilots to mission-critical applications. Building a proof of concept that taps a single data source or uses a few prompts is relatively simple. But delivering real business value requires these models to access consistent, high-quality data—something often trapped in silos, buried in legacy systems, or updated at irregular intervals.

To scale from pilot to production, organizations need a more structured approach—one grounded in solid data management practices. That means integrating with enterprise data, enforcing strong governance, and ensuring continuous maintenance. Without this foundation, organizations risk stalling after just one or two operationalized GenAI use cases, unable to scale or accommodate new GenAI initiatives, due to persistent data bottlenecks. The good news is that those early GenAI implementation efforts are the ideal moment to confront your data challenges head-on and embed sustainable practices from the start.

How, then, can data management principles help you build GenAI solutions that scale with confidence?

In the sections that follow, we’ll look at the nature of GenAI data and the likely challenges it poses in your first value cases. We’ll then explore how the right data management practices can help you address these challenges—ensuring your GenAI efforts are both scalable and future-proof.

Understanding the nature of GenAI data

GenAI data introduces a new layer of complexity that sets it apart from traditional data pipelines, where raw data typically flows into analytics or reporting systems. Rather than dealing solely with structured, tabular records in a centralized data warehouse, GenAI applications must handle large volumes of both structured and unstructured content—think documents, transcripts, and other text-based assets—often spread across disparate systems.

To enable accurate retrieval and provide context to the model, unstructured data first be made searchable. This typically involves chunking the content and converting it into vector embeddings or exposing it via a Model Context Protocol (MCP) server, depending on the storage provider (more on that later). Additional context is increasingly delivered through metadata, knowledge graphs, and prompt engineering, all of which are becoming essential assets in GenAI architectures. Moreover, GenAI output often extends beyond plain text. Models may generate structured results—such as function calls, tool triggers, or formatted prompts—that require their own governance and tracking. Even subtle variations in how text is chunked or labeled can significantly influence model performance.

Because GenAI taps into a whole new class of data assets, organizations must treat it differently from standard BI or operational datasets. This introduces new data management requirements to ensure that models can reliably retrieve, generate, and act on timely information—without drifting out of sync with rapidly changing data. By addressing these nuances from the outset, organizations can avoid performance degradation, build trust, and ensure their GenAI solutions scale effectively while staying aligned with business goals.

Start smart: consider data representability

Selecting your first GenAI value case to operationalize is a critical step—one that sets the tone for everything that follows. While quick experiments are useful for validating a model’s capabilities, the real challenge lies in integrating that model into your enterprise architecture in a way that delivers sustained, scalable value.

Choosing the right value case requires more than just identifying an attractive opportunity. It involves a clear-eyed assessment of expected impact, feasibility, and data readiness. One often overlooked but highly valuable criterion is data representability: how well does the use case surface the kinds of data challenges and complexities you’re likely to encounter in future GenAI initiatives?

Your first operationalized value case is, in effect, a strategic lens into your data landscape. It should help you identify bottlenecks, highlight areas for improvement, and stress-test the foundations needed to support long-term success. Avoid the temptation to tackle an overly ambitious or complex use case out of the gate—that can easily derail early momentum. Instead, aim for a focused, representative scenario that delivers clear early wins while also informing the path to sustainable, scalable GenAI adoption.

Don’t get stuck: Answer these critical data questions

The targeted approach described above delivers the early tangible win needed to get stakeholder buy-in—and, more critically, it exposes underlying data issues that might otherwise go unnoticed. Here are typical data requirements for your first GenAI use case, along with the key challenge they create and core questions you’ll need to answer.

RequirementChallengeKey questions
Integrated informationPreviously untapped information (for example, pdf documents), is now a key data source and must be integrated with your systems.* Where is all this data physically stored? Who maintains ownership of each source?

* Do you have the necessary permissions to access them?

* Does the data storage tool provide a MCP server to interact with the data?
Searchable knowledge systemSearchable documents must be chunked, embedded, and updated in a vector database—or exposed via an MCP server—creating new data pipeline and infrastructure demands.* Can you find a reliable embedding model for capturing the nuances in your business?

* How frequently must you re-embed to handle updates?

* Are the associated costs manageable at scale?
Context on how data is interconnectedReasoning capabilities require explicitly defined relationships—typically managed through a knowledge graph.* Is your data described well enough to construct such a graph?

* Who will ensure these relationships and definitions stay accurate over time?
Retrieval from relational databases by an LLMModels need rich, well-documented metadata to understand and accurately query relational data.* Are vital metadata fields consistent and detailed?

* Do you have the schemas and documentation required for the model to navigate these databases effectively?
Defining prompts, output structures and tool interfacePrompts, interfaces, and outputs must be reusable, versioned, and traceable to scale reliably and maintain trust.* How do you ensure these prompts and interfaces are reusable?

* Can you track changes for auditability?

* How do you log their usage to troubleshoot issues?
Evaluate model outputs over timeUser feedback loops are essential to monitor performance, but collecting, storing, and acting on feedback systematically is lacking.* How do you store feedback?

* How do you detect dips in accuracy or relevance?

* Can you trace poor results back to specific data gaps?
Access to multiple data sources for the LLMGenAI creates data leakage risks (e.g. prompt injection), requiring access controls to secure model interactions.* Which guardrails or role-based controls prevent unauthorized data exposure?

* How do you ensure malicious prompts don’t compromise the agent?

Look beyond your first pilot: apply data product thinking -from day one

A common mistake is solving data challenges only within the narrow context of the initial pilot—resulting in quick fixes that don’t scale. As you tackle more GenAI projects, inconsistencies in data structures, metadata, or security can quickly undermine progress. Instead, adopt a reusable data strategy from day one—one that aligns with your broader data management framework.

Treat core assets like vector databases, knowledge graphs, relational metadata, and prompt libraries as data products. That means assigning ownership, defining quality standards, providing documentation, and identifying clear consumers—so each GenAI use case builds on a solid, scalable foundation.

Core data product ingredients for GenAI

Below, we outline the key principles for making that happen, along with examples, roles, and how to handle incremental growth.

1. Discoverability

Implement a comprehensive data catalogue that documents what data exists, how it's structured, who owns it, and when it's updated. This catalogue should work in conjunction with—but not necessarily be derived from—any knowledge graphs built for specific GenAI use cases. While knowledge graphs excel at modelling domain-specific relationships, your data catalogue needs broader enterprise-wide coverage. Include the metadata and examples needed for generating SQL queries, as well as references for your vector database. (This is something we find particularly useful, for example when developing AI agents that autonomously generate SQL queries to search through databases. Curated, detailed metadata, including table schemas, column definitions, and relationships allow the agents to understand the structure and semantics of the databases.) Define a scheduled process to keep these inventories current, to ensure that new GenAI projects can easily discover existing datasets or embeddings, while avoiding unnecessary re-parsing or re-embedding of data that is already available.

2. Quality standards

This includes a range of processes. From reviewing PDF documents to remove or clarify checkboxes and handwritten text to checking chunk-level completeness for embeddings, consistent naming conventions, or standardized templates for prompt outputs. Involve domain experts to clarify which fields or checks are critical, and align those standards in your data catalogue. By enforcing requirements at the source, you prevent silent inconsistencies (like missing metadata or mismatched chunk sizes) from undermining your GenAI solutions later on.

3. Accessibility

GenAI models require consistent, reliable ways to retrieve information across your enterprise landscape—yet building a custom integration for every data source quickly becomes unsustainable. As an example, at a client a separate data repository was created—with its own API and security—to pull data from siloed systems. This is where the MCP comes in handy: by specifying a standard interface for models to access external data and tools, it removes much of the overhead associated with custom connectors. Not all vendors support MCP but adoption is growing (OpenAI for example provides it). So keep that in mind and avoid rushing into building custom connectors for every system. In sum, focus on building a solid RAG foundation for your most critical, large-scale enterprise data—where the return on investment is clear. For simpler or lower-priority use cases, it may be better to wait for MCP-ready solutions with built-in search, rather than rushing into one-off integrations. This balanced approach lets you meet immediate needs while staying flexible as MCP adoption expands. (We’ll dive deeper into MCP in our next blog post.)

4. Auditability

Log every step—when doing at least make sure to include knowledge graph updates to prompts, reasoning, and final outputs—so you can pinpoint which data, transformations, or model versions led to a given response. This is especially vital in GenAI, where subtle drifts (like outdated embeddings or older prompt templates) often go unnoticed until user trust is already compromised. From a compliance perspective, this is critical when GenAI outputs influence business decisions. Organizations may need to demonstrate exactly which responses were generated, what processing logic was applied, and which data was used at specific points in time—particularly in regulated industries or during audits. Ensure each logged event links back to a clear version ID for both models and data, and store these audit trails securely with appropriate retention policies to meet regulatory, privacy, and governance requirements.

5. Security & access control

Ensure appropriate role- and attribute-based access is enforced on all data sources accessed by your model, rather than granting blanket privileges. Align access with the end-user's identity and context, preventing the model from inadvertently returning unauthorized information. Be particularly vigilant about prompt injection attacks—where carefully crafted inputs manipulate models into bypassing security controls or revealing sensitive information. This remains one of GenAI's most challenging security vulnerabilities, as traditional safeguards can be circumvented through indirect methods. Consider implementing real-time checks, especially in autonomous agent scenarios, alongside an "LLM-as-a-judge" layer that intercepts and sanitizes prompts that exceed a user's permissions or exhibit malicious patterns. While no solution offers perfect protection against these threats, understanding the risk vectors and implementing defence-in-depth strategies significantly reduces your exposure. Resources like academic papers and technical analyses on prompt injection techniques can help your team stay informed about this evolving threat landscape.

6. LLMOps

A shared, organization-wide LLMOps pipeline and toolkit gives every GenAI project a stable, reusable foundation—covering everything from LLM vendor switching and re-embedding to common connectors and tool interfaces. Centralizing these resources eliminates ad-hoc solutions, promotes collaboration, and ensures proper lifecycle management. The result: teams can launch GenAI initiatives faster, apply consistent best practices, and still stay flexible enough to experiment with different models and toolchains.

Transform Generative AI potential into performance with robust data management

Whether you win or lose in your market may soon rely on having the best Generative AI capability - and the data management setup to support it. We can help you with this.

Explore our data management services

The takeaway: combine your GenAI implementation with proper data management

Implementing GenAI models and agentic systems isn’t just about prompting an LLM. It’s a data challenge and an architectural exercise. By starting with a high-impact use case, you secure immediate value and insights, while robust data management practices ensure you stay in control—maintaining quality and trust as AI spreads across your organization. By adopting these principles in your very first GenAI initiative, you avoid the trap of quick-win pilots that ultimately buckle under growing data complexity. Instead, each new use case taps into a cohesive, well-governed ecosystem—keeping your AI agents current, consistent, and secure as you scale.

Those who excel at both GenAI innovation and data management will be the ones to transform scattered, siloed information into a unified, GenAI-ready asset—yielding genuine enterprise value on a foundation built to last. If you haven’t already established data management standards and principles, don’t wait any longer—take a look at our blog posts on data management fundamentals part 1 and 2.

If you’d like to learn more about building a robust GenAI foundation, feel free to reach out to us.


This article was written by Daan Manneke, Data & AI Engineer at Rewire and Frejanne Ruoff, Principal at Rewire.

The race to building competitive advantage just took an interesting turn. Here is what you need to know.

What is model distillation?

In chemistry, distillation is a purification process where a liquid mixture is heated until components with different boiling points separate. The result is often a more concentrated, refined substance. This elegant concentration technique has been used for centuries to create everything from perfumes to whiskey, preserving essential qualities while removing unnecessary bulk.

Model distillation in AI follows a remarkably similar principle; knowledge is extracted from a large and complex AI model (the “teacher”) and transferred to a smaller, more efficient model (the “student”). At its core, model distillation is about teaching the student to act like the teacher. Instead of training a small model with traditionally labeled data, response-based knowledge distillation allows the student model to mimic the intelligence of the teacher (i.e., output token probability distributions, for the data scientists among us).

The result is a refined, lightweight model that captures a concentrated essence of its larger counterpart's capabilities within its domain of expertise.

Why model distillation is everywhere

While DeepSeek did not invent model distillation, their breakthrough in early 2025, which we discussed before, catapulted cost efficiency for GenAI models into the frontstage, proving that smaller teams with limited resources could compete at the cutting edge of AI development.

The latest generation Large Language Models (LLMs) like GPT-4.5, DeepSeek V3, and Claude 3.7 are incredibly powerful—but also incredibly expensive to run. Model distillation offers the process of training a smaller model to mimic a larger one, delivering near state-of-the-art performance in a narrow domain at a fraction of the cost.

DeepSeek was a game changer in the pricing of large scale models. Model distillation extends beyond that: it enables AI capabilities where even the cheapest full-scale LLMs are too expensive or impractical to run.

For example, researchers from the University of Washington created their own reasoning model in merely 26 minutes for less than $50. InHand Networks successfully distilled models on their edge AI computers to provide low-power real-time inference capabilities. This revolution has forced even industry giants like OpenAI to reconsider their closed-source approach.

Why it is a game-changer for businesses

The value of model distillation extends far beyond reducing expenses—it unlocks new possibilities for AI implementation that were not feasible previously. Here's how it's changing the game:

Better unit economics. Running small, distilled models requires significantly less computational resources, translating directly to lower operational costs for high-volume AI applications.

✅ Enhanced inference speed. Smaller models process data more quickly, leading to faster response times and improved user experience in time-sensitive applications.

Near state-of-the-art performance. For well-defined use cases, a distilled model can perform remarkably close to full-size LLMs while being more efficient and cost-effective.

Smaller bottleneck on data. The requirement on your data shifts from curating and labeling large datasets, to a smaller curated set of examples with teacher-generated answers, significantly reducing annotation costs and time.

Not a silver bullet: the downsides you should consider

Model distillation is not without hurdles. Here’s what you need to plan for:

Upfront development cost. Building and training your own distilled model takes time and effort from experts in your team.

Teacher performance limitation. The small model can only be as good as the large model it learns from. Furthermore, some large proprietary model providers like OpenAI and Anthropic put restrictions on the output of their models, hindering their use for model distillation.

No automatic updates. When improved foundation models are released, you need to repeat the process of distillation to get access to these improvements in distilled format.

Model distillation in action: where it works best

Distillation works great when you need cost-effective, scalable AI in a controlled environment. Good use cases include: 

🔹 Edge devices & on-device solutions. Frontier models do not fit on consumer hardware or smaller edge devices. (Edge devices—e.g., phones, sensors, or cameras—process data locally and on the spot instead of sending it to a distant data center.) If low-latency or portability is a requirement, you may want to embed the required intelligence in a smaller package.

🔹 Personalized content generation at scale. Think online learning platforms that generate exercises tailored to their students’ needs, or marketing tools that generate personalized e-mail content.

🔹 Domain-specific chatbots. Customer support bots trained for highly specific industries without relying on expensive full-scale models for all interactions.

🔹Specialized parts of agentic workflows. Agents use notoriously many tokens. Hence, splitting your solution up in specialized modules with distilled models can reduce costs and latency significantly, especially at scale.

However, don’t use model distillation for deep research or complex, open-ended reasoning tasks. If your AI needs to push the boundaries of intelligence or perform highly creative problem-solving, a distilled model won’t cut it; inherently, it will remain inferior to its teacher.

The bottom line: does distillation fit your business needs?

Model distillation is transforming from an academic idea into a competitive advantage for businesses. It democratizes advanced AI capabilities while dramatically reducing costs. For businesses facing the dual pressures of innovation and efficiency, distillation offers a middle ground: near state-of-the-art performance without the steep price tag.

Just be mindful that it’s not a silver bullet. Many use cases will still require full-scale LLMs, which offer the benefit of broader and more general knowledge and capabilities.

If your AI roadmap has well-defined use cases in a specific domain that can afford a short-term investment for long-term cost benefits, it may well be worth it to explore this approach.

Decorative background iconography - Generative AI consulting services for enterprises

Turning Generative AI potential into bottom line impact

Our strategies enable you to harness generative AI, moving beyond marginal or tactical gains to achieve transformational success.

Explore our Generative AI services

This article was written by Jacco Broere, Data & AI Engineer at Rewire and Simon Koolstra, Principal at Rewire.

Unlocking true intelligence with memory, strategic planning, and transparent performance management

The rise of intelligent AI agents marks a shift from traditional task automation toward more adaptive, decision-making systems. While many AI-powered workflows today rely on predefined rules and deterministic processes, true agentic AI goes beyond fixed automation—it incorporates memory, reasoning, and self-improvement to adapt and autonomously achieve specific goals.

The previous agentic AI blog post explained the building blocks (memory, planning, tools and actions), use cases, and potential for autonomy. It positioned AI agents not just as tools for automating repetitive tasks but as systems capable of enhancing workflow efficiency and tackling complex problems with some degree of independent decision-making. In this blog post, we explore how these agents can be seamlessly integrated into real-world scenarios, striking a balance between cognitive science theory and the practical realities of human-agent interaction, with a particular focus on the memory and planning building blocks, alongside the addition of performance management.

Moving from models to real-world AI agents

Scaling AI agents presents challenges. While they can improve efficiency, their success depends on integration with existing systems. Use cases like personal assistants and content generators demonstrate clear value, whereas others struggle with reliability and adaptability.

Current LLM-based workflow automation relies on knowledge—whether through large reasoning models or knowledge bases. However, these agents often lack persistent memory, meaning they re-solve the same issues repeatedly. Without storing and leveraging past experiences, they remain reactive rather than truly intelligent. To bridge this gap, AI agents need:

  • Memory – Retaining and applying past experiences.
  • Ability to plan – Setting goals, adapting strategies, and managing complexity.
  • Transparent performance management – Ensuring alignment, oversight, and trust.

These elements go beyond the building blocks of tools and actions, which have already been widely discussed in AI agent design. Here we focus on memory, planning, and performance management, as they represent critical design choices to move toward AI agents that are not just reactive task automators, but intelligent, adaptable decision-makers capable of handling more sophisticated tasks in real-world scenarios. Let’s start by exploring what intelligence in that sense even means.

Memory: the key to true intelligence

True intelligence goes beyond automation. An AI agent must not only process information but also learn from past experiences to improve over time. Without memory, an AI agent would remain static, unable to adapt or evolve. By integrating memory, reasoning, and learning, an intelligent AI agent moves beyond simply performing predefined tasks. In our previous blog post on AI agents, we explained that memory, planning, tools and actions are the building blocks of agents. Now, let's examine how memory plays a crucial role in enhancing an AI agent's capabilities. In cognitive science, memory is divided into:

  • Semantic memory – A structured database of facts, rules, and language that provides foundational knowledge.
  • Episodic memory – Past experiences that inform future decisions and allow for adaptation.

The interplay between semantic and episodic memory enables self-improvement: experiences enrich knowledge, while knowledge structures experiences. When agents lack episodic memory, they struggle with contextual awareness and must rely solely on predefined rules or external prompting to function effectively. To obtain intelligent agents, episodic memory is therefore crucial. By organizing past interactions into meaningful units (through chunking), agents can recall relevant solutions, compare them to new situations, and refine their approach. This form of memory actively supports an agent’s ability to reflect on past actions and outcomes.

To illustrate this, let's consider an example from supply chain management, as depicted in the image below. An AI agent that tracks delivery data, inventory, and demand can improve logistics by learning from past experiences. If the agent identifies patterns, such as delays during certain weather conditions or peak seasons, it can proactively adjust shipping schedules and notify relevant stakeholders. Without memory, the agent would simply repeat tasks without optimizing them, leading to inefficiencies and missed opportunities for improvement.

Figure 1 - Memory in AI Agents - An example from supply chain management

Ability to plan: the foundation for autonomous decision-making

Intelligent AI agents must dynamically plan and break complex problems into manageable tasks—mirroring the analytical nature of the human mind. Unlike rule-based automation, these agents should be able to assess different strategies, evaluate potential outcomes, and adjust their approach based on real-time feedback. Planning allows an agent to remain flexible, ensuring it can pivot when conditions change rather than blindly following predefined sequences.

LLMs serve as the reasoning engines of AI agents, showcasing increasingly advanced cognitive abilities. However, they struggle with long-term memory and sustained focus—much like the human mind under information overload. This limitation poses challenges in designing AI agents that must retain context across extended interactions or tackle complex problem-solving tasks.

A critical design question is whether an agent should retain plans internally or offload them to an external tool. Keeping plans within an LLM provides full information access but may be limited by context constraints. For example, an AI managing a real-time chat-based customer support system could benefit from internal memory to dynamically adapt to an ongoing conversation, keeping track of the customer's previous questions and preferences without relying on external systems. This allows the agent to provide personalized responses without the delay of querying an external database. On the other hand, external tools lighten the cognitive load but can introduce rigidity if not well-integrated. For instance, an AI-powered weather application might be better off using an external tool to retrieve up-to-date weather data rather than relying on its internal model, which could become outdated or too complex to manage. This allows the system to focus on processing and presenting the information without overloading its internal resources. A balanced approach ensures adaptability without overloading the agent’s working memory. Ultimately, the necessity of such a tool depends on the LLM's ability to retrieve, retain, and adjust information—an advanced reasoning model might even eliminate the need for external tools.

For example, an AI-powered financial advisor might need to balance long-term investment strategies with short-term market fluctuations. If it relies too heavily on immediate context, it might make impulsive decisions based on temporary trends. On the other hand, if it solely adheres to a rigid external planning framework, it might fail to adapt to new opportunities. The ideal approach blends both—leveraging structured knowledge while maintaining the ability to dynamically reassess and adjust strategies.

Transparent performance management: balancing efficiency and trust

Human-AI agent interaction is shaped by the trade-off between efficiency and trust: the more autonomous an AI agent becomes, the more it can streamline operations and reduce human workload—yet the less transparent its decision-making may feel. In scenarios where tasks are low-risk and repetitive, full automation makes sense as errors have minimal impact, and efficiency gains outweigh the downsides. However, in high-stakes environments like financial trading or medical diagnosis, the costs of a wrong decision are simply too high. Transparent performance management is thus essential.

The challenge is that AI agents, while improving, are still fallible, inheriting issues like hallucinations and biases from LLMs. AI must operate within defined trust thresholds—where automation is reliable enough to act independently yet remains accountable. Rather than requiring continuous human oversight, performance management should focus on designing mechanisms that allow AI agents to function autonomously while ensuring reliability. This involves self-monitoring, self-correction, and explainability.

Mechanisms like agent self-critique mitigate these issues by enabling agents to evaluate their own decisions before execution. Also known as LLM-as-a-judge, self-critique involves sending both input and output to a separate LLM entity that is unaware of the entire agentic workflow, assessing whether the response logically follows from the input. For instance, an LLM can check its output for consistency by sending both its input and response to a separate validation model, which then determines whether the response aligns with the provided information. This process helps catch hallucinations, biases, and inconsistencies before decisions are finalized, improving the reliability of autonomous AI agents.

In the early stages of AI agent deployment, human experts play a crucial role in shaping and refining performance management processes. However, as these agents evolve, the goal is to reduce direct human intervention while maintaining oversight through structured performance metrics. Instead of requiring constant check-ins, AI agents should be designed to self-monitor and adapt, ensuring alignment with objectives without excessive human involvement. By incorporating mechanisms for self-assessment, AI agents can achieve greater autonomy while maintaining accountability. The ultimate aim is to develop fully autonomous agents that balance efficiency with transparency—operating independently while ensuring performance remains reliable.

For example, consider an AI agent managing IT system maintenance in a large enterprise. Such an agent monitors server performance, security threats, and software updates. Instead of relying on human intervention for every decision, it can autonomously detect anomalies, apply minor patches, and optimize system configurations based on historical performance data. However, major decisions—such as deploying a company-wide software update—may still require validation through transparent reporting mechanisms. If the AI agent consistently demonstrates accuracy in its assessments and risk predictions, human involvement can gradually decrease, ensuring both operational efficiency and system integrity.

Transform potential into performance with GenAI

Whether you win or lose in your market may soon rely on having the best GenAI capability and the data foundations to support it. We can help you build this.

Explore our Generative AI services

What’s next for AI agents?

AI agents are evolving beyond simple automation. To become truly intelligent, they must adapt, plan, and learn from experience. Without memory, an agent is static. Without planning, it lacks direction. Without transparent performance management, it risks unreliability.

By integrating memory, planning, and performance management, AI agents can move beyond task execution toward strategic problem-solving. Future AI will not merely automate processes but will actively contribute to decision-making, helping organizations navigate complexity with greater precision and efficiency.

The future belongs to AI that doesn’t just execute tasks but remembers, adapts, and improves. An agent without intelligence is merely automation with an attitude.

Sources

Greenberg DL, Verfaellie M. "Interdependence of episodic and semantic memory: evidence from neuropsychology." J Int Neuropsychol Soc. 2010;16(5):748-753. doi:10.1017/S1355617710000676. Link.

"Does AI Remember? The Role of Memory in Agentic Workflows." (2025) Link.

"RULER: What's the Real Context Size of Your Long-Context Language Models?" (2024). arXiv:2404.06654


This article was written by Gijs Smeets, Data Scientist at Rewire and Mirte Pruppers, Data Scientist at Rewire.

An introduction to the world of LLM output quality evaluation: the challenges and how to overcome them in a structured manner

Perhaps you’ve been experimenting with GenAI for some time now, but how do you determine when the output quality of your Large Language Model (LLM) is sufficient for deployment? Of course, your solution needs to meet its objectives and deliver reliable results. But how can you evaluate this effectively?

In contrast to LLMs, assessing machine learning models is often a relatively straightforward process: metrics like Area Under the Curve for classification or Mean Absolute Percentage Error for regression give you valuable insights in the performance of your model. On the other hand, evaluating LLMs is another ball game, since GenAI generates unstructured, subjective outputs – in the form of texts, images, or videos - that often lack a definitive "correct" answer. This means that you’re not just assessing whether the model produces accurate outputs; you also need to consider, for example, relevance and writing style.

For many LLM-based solutions (except those with very specific tasks, like text translation), the LLM is just one piece of the puzzle. LLM-based systems are typically complex, since they often involve multi-step pipelines, such as retrieval-augmented generation (RAG) or agent-based decision systems, where each component has its own dependencies and performance considerations.

In addition, system performance (latency, cost, scalability) and responsible GenAI (bias, fairness, safety) add more layers of complexity. LLMs operate in ever-changing contexts, interacting with evolving data, APIs, and user queries. Maintaining consistent performance requires constant monitoring and adaptation.

With so many moving parts, figuring out where to start can feel overwhelming. In this article, we’ll purposefully over-simplify things by answering the question: “How can you evaluate the quality of your LLM output?”. First, we explain what areas you should consider in the evaluation of the output. Then, we’ll discuss the methods needed to evaluate output. Finally, to make it concrete, we bring everything together in an example.

What are the evaluation criteria of LLM output quality?

High-quality outputs build trust and improve user experience, while poor-quality responses can mislead users and foster misinformation. The start of building an  evaluation (eval) is to start with the end-goal of the model. The next step is to define the quality criteria to be evaluated. Typically these are:

  • Correctness: Are the claims generated by the model factually accurate?
  • Relevance: Is the information relevant to the given prompt? Is all required information provided -by the end user, or in the training data- to adequately offer an answer to the given prompt?
  • Robustness: Does the model consistently handle variations and challenges in input, such as typos, unfamiliar question formulations, or types of prompts that the model was not specifically instructed for?
  • Instruction and restriction adherence: Does the model comply with predefined restrictions or is it easily manipulated to jailbreak the rules?
  • Writing style: Does the tone, grammar, and phrasing align with the intended audience and use case?

How to test the quality of LLM outputs?

Now that we’ve identified what to test, let’s explore how to test. A structured approach involves defining clear requirements for each evaluation criterion listed in the previous section. There are two aspects to this: references to steer your LLM towards the desired output and the methods to test LLM output quality.

1. References for evaluation

In LLM-based solutions, the desired output is referred to as the golden standard, which contain reference answers for a set of input prompts. Moreover, you can provide task-specific guidelines such as model restrictions and evaluate how well the solution adheres to those guidelines.

While using a golden standard and task-specific guidelines can effectively guide your model towards the desired direction, it often requires a significant time investment and may not always be feasible. Alternatively, performance can also be assessed through open-ended evaluation. For example, you can use another LLM to assess relevance, execute generated code to verify its validity, or test the model on an intelligence benchmark.

2. Methods for assessing output quality

Selecting the right method depends on factors like scalability, interpretability, and the evaluation requirement being measured. In this section we explore several methods, and assess their strengths and limitations.

2.1. LLM-as-a-judge

An LLM isn’t just a text generator—it can also assess the outputs of another LLM. By assessing outputs against predefined criteria, LLMs provide an automated and scalable evaluation method.

Let’s demonstrate this with an example. For example, ask the famous question, "How many r's are in strawberry?" to ChatGPT's 4o mini model. It responds with, "The word 'strawberry' contains 1 'r'.", which is obviously incorrect. With the LLM-as-a-judge method, we would like the evaluating LLM (in this case, also 4o mini) to recognize and flag this mistake. In this example, there is a golden reference answer “There are three 'r’s'  in 'strawberry'.”, which can be used to evaluate the correctness of the answer.

Indeed, the evaluating LLM appropriately recognizes that the answer is incorrect.

The example shows that LLMs can evaluate outputs consistently and at scale due to their ability to quickly assess several criteria. On the other hand, LLMs may struggle to understand complex, context-dependent nuances or subjective cases. Moreover, LLMs may strengthen biases within the training data and can be costly to use as an evaluation tool.

2.2. Similarity metrics for texts

When a golden reference answer is available, similarity metrics provide scalable and objective assessments of LLM performance. Famous examples are NLP metrics like BLEU and ROUGE, or more advanced embedding-based metrics like cosine similarity and BERTScore. These methods provide quantitative insights in measuring the overlap in words and sentence structure without the computational burden of running full-scale LLMs. This can be beneficial when outcomes must closely align with provided references – for example in the case of summarization or translation.

While automated metrics provide fast, repeatable, and scalable evaluations, they can fall short on interpretability and often fail to capture deeper semantic meaning and factual accuracy. As a result, they are best used in combination with human evaluation or other evaluation methods.

2.3. Human evaluation

Human evaluation provides a strong evaluation method due to its flexibility. In early stages of model development, it is used to thoroughly evaluate errors such as hallucinations, reasoning flaws, and grammar mistakes to provide insights into model limitations. As the model improves through iterative development, groups of evaluators can systematically score outputs on correctness, coherence, fluency, and relevance. To reduce workload and enable real-time human evaluation after deployment, pairwise comparison can be used. Here, two outputs are compared to determine which performs better for the same prompt. This is in fact implemented in ChatGPT.

It is recommended to use both experts as non-experts in human evaluation of your LLM. Experts can validate the model’s approach based on their expertise. On the other hand, non-experts play a crucial role in identifying unexpected behaviors and offering fresh perspectives on real-world system usage.

While human evaluation offers deep, context-aware insights and flexibility, it is resource- and time-intensive. Moreover, comparing different examiners can lead to inconsistent evaluations when they are not aligned.

2.4. Benchmarks

Lastly, there are standardized benchmarks that offer an approach to assess the general intelligence of LLMs. These benchmarks evaluate models on various capabilities, such as general knowledge (SQuAD), natural language understanding (SuperGLUE), and factual consistency (TruthfulQA). To maximize their relevance, it’s important to select benchmarks that closely align with your domain or use case. Since these benchmarks test broad abilities, they are often used to identify an initial model for prototyping. However, standardized benchmarks can provide a skewed perspective due to their lack of alignment with your specific use case.

2.5. Task specific evaluation

Depending on the task, other evaluation methods are appropriate. For instance, when testing a categorization LLM, accuracy can be measured using a predefined test set alongside a simple equality check (of the predicted category vs. actual category). Similarly, the structure of outputs can be tested by counting line-breaks; certain headers and/or the presence of certain keywords can also be checked. Although these technique are not easily generalizable across different use cases, they offer a precise and efficient way to verify model performance.

Putting things together: writing an eval to measure LLM summarization performance

Consider a scenario where you're developing an LLM-powered summarization feature designed to condense large volumes of information into three structured sections. To ensure high-quality performance, we evaluate the model for each of our five evaluation criteria. For each criterion, we identify a key question that guides the evaluation. This question helps define the precise metric needed and determines the appropriate method for calculating it.

CriteriumKey questionMetricHow
CorrectnessIs the summary free from hallucinations?Number of statements in summary that can be verified based on source text* Use an LLM-as-a-judge to check if each statement can be answered based on the source texts
* Use human evaluation to verify correctness of outputs
RelevanceIs the summary complete?Number of key elements present with respect to a reference guideline or golden standard summaryCross-reference statements in summaries with LLM-as-a-judge and measure the overlap
Is the summary concise?Number of irrelevant statements with respect to golden standard Length of summary* Cross-reference statements in summaries with LLM-as-a-judge and measure the overlap
* Count the number of words of generated summaries
RobustnessIs the model prone to noise in the input text?Similarity of summary generated for original text with respect to summary generated for text with noise such as typo’s and inserted irrelevant informationCompare statements with LLM-as-a-judge, or compare textual similarity with ROUGE or BERTscore
Instruction & restriction adherenceDoes the summary comply with required structure?Presence of three structured sectionsCount number of line breaks and check presence of headers
Writing styleIs the writing style professional, fluent and free of grammatical errors?Rating of tone-of-voice, fluency and grammar* Ask LLM-as-a-judge to rate fluency and professionality and mark grammatical errors
* Rate writing style with human evaluation
Overarching: alignment with golden standardDo generated summaries align with golden standard summaries?Textual similarity with respect to golden standard summaryCalculate similarity with ROUGE or BERTscore

The table shows that the proposed evaluation strategy leverages multiple tools and combines reference-based and reference-free assessments to ensure a well-rounded analysis. And so we ensure that our summarization model is accurate, robust, and aligned with real-world needs. This multi-layered approach provides a scalable and flexible way to evaluate LLM performance in diverse applications.

Final thoughts

Managing LLM output quality is challenging, yet crucial to build robust and reliable applications. To ensure success, here are a few tips:

  • Proactively define the evaluation criteria. Establish clear quality standards before model deployment to ensure a consistent assessment framework.
  • Automate when feasible. While human evaluation is essential for subjective aspects, automate structured tests for efficiency and consistency.
  • Leverage GenAI to broaden your evaluation. Use LLMs to generate diverse test prompts, simulate user queries, and assess robustness against variations like typos or multi-language inputs.
  • Avoid reinventing the wheel. There are already various evaluation frameworks available on the internet (for instance, DeepEval). These frameworks provide structured methodologies that combine multiple evaluation techniques.

Achieving high-quality output is only the beginning. Generative AI systems require continuous oversight to address challenges that arise after deployment. User interactions can introduce unpredictable edge cases, exposing the gap between simulated scenarios and real-world usage. In addition, updates to models and datasets can impact performance, making continuous evaluation crucial to ensure long-term success. At Rewire, we specialize in helping organizations navigate the complexities of GenAI, offering expert guidance to achieve robust performance management and deployment success. Ready to take your GenAI solution to the next level? Let’s make it happen!


This article was written by Gerben Rijpkema, Data Scientist at Rewire, and Renske Zijm, Data Scientist at Rewire.

Decorative background iconography - Generative AI consulting services for enterprises

Turning Generative AI potential into bottom line impact

Our strategies enable you to harness generative AI, moving beyond marginal or tactical gains to achieve transformational success.

Explore our Generative AI services