The Association Intelligence Podcast by Betty:AI

0:00 20:48

[Thomas Altman] (0:05 - 0:10)

Today's big question, how did we make the Association Intelligence Podcast and why?

[Rob Barnes] (0:10 - 1:06)

We are Thomas Altman and Rob Barnes and this is episode zero of the Association Intelligence Podcast. Thomas, I'm so glad that you came up with the idea of doing an episode zero because it reminds me of the conversations we were having before we started writing the Association Knowledge Cycle book, which is now nearly three years ago. But also because I think that, and I've said this to everyone that's bought the book or taken a copy of the book, I said, please just go to the epilogue.

My favorite part of the book is the epilogue that you put together, which is about how we wrote the book. Because it was such an interesting experiment, seeing that we wanted to experiment with the interplay between our human endeavor, the thoughts, the ideas that we wanted to share and how we used AI tools to help us ultimately write the book.

[Rob Barnes] (1:06 - 1:26)

And over the 12 months it took us to write the book, how dramatically the AI that we were using to do this changed in capability and intelligence and insights and pushback, like that whole experiment. And I feel like this is exactly the same experience again, but in a completely unique way.

[Thomas Altman] (1:26 - 2:50)

Yeah, I think it's really, really cool. So the idea here is we've now recorded all the episodes for this podcast. So kind of set the table a little bit because hopefully this is the first one you're listening to.

What this season of the Association Intelligence Podcast has is a series of 12 questions, right? 12 questions that we either are asked a lot and deserve really, good questions, or that we think aren't asked enough, but are really, really important. And kind of, we went through this whole process of creating this really cool, awesome podcast for you guys, and then took a step back and my thought went to the same place to the epilogue of our book, the Association Knowledge Cycle.

And wanted to give you guys a little peek behind the scenes, like how did this all come together? Why is this episode just humans, but the rest are going to be humans and AI kind of back and forth? And how can, I think more importantly, how can you use sort of this experience to inform your own ideas around AI and how you can employ it at your association, right?

I think that that's really, really important. I think all of it is grounded, for those of you who have not kind of, are not familiar with the Association Knowledge Cycle concept that we've been playing with for a few years now. The idea here is that, one, associations are the organizations best situated to benefit from generative AI.

We fully believe that, we've believed that since before ChatGPT launched.

[Thomas Altman] (2:51 - 3:26)

And what we, the way we see that happening is kind of through this interplay around intellectual property, which associations are the owners of, the creators of, the guardians of for your industry, right, or your profession. And the interaction between members and the public with that intellectual property, with that knowledge. And what generative AI gets us when done correctly and deployed correctly, is the ability to see those interactions at scale, across all your members, across time, and then learn from those interactions in a way that allows you to create the next wave of knowledge.

[Thomas Altman] (3:27 - 4:10)

And as we've been building Betty and deploying Betty across kind of almost 200 different associations now, we've been good about that first part, right? We've gotten to the point where we've got, we've scaled the conversations and people can see them, but we haven't quite executed on that last, that last mile of taking those and creating something useful. So this podcast is kind of us putting our own words into action, looking at how people interact with our version of Betty, and turning that into something that we believe is kind of more broadly useful across the entire association ecosystem to help understand sort of good and bad and sort of beyond use cases for AI.

So that's kind of where this, this all came together.

[Rob Barnes] (4:10 - 7:00)

I remember when we first started scoping out, like, how did we want this to kind of take shape? And I was reflecting on how the nature of the conversations I'm having about Betty and AI, the application, the insights, those conversations have markedly changed over the last 12 months because of where the maturity around conversations, but also where things like AI fatigue are really starting to, to hit association executives. And so we wanted to go back a little bit and listen.

And so the first part of the first step was, well, let's build a repository of knowledge around what the conversations are in the association sector about AI. And that repository of knowledge would then become, you know, part of the source, you know, the major part of the source of how we would produce, you know, the episodes. And that's how we arrived at the, you know, the big questions of the day, even that experiment alone, and how long and how purposeful we stepped through what the listening tool that we built, how it was following that process, right?

You know, we wanted, like, there's amazing consultants in the association industry. There's amazing other podcasts, some of the, you know, 14, 15 best podcasts already around the association industry. There is books, there's newsletters, there is, you know, ASAE's Association Now, there's associations for associations.

You know, we had the AI tools listening to all of this over a period of time. It wasn't just a one and done. We didn't just accept the first outputs that came.

We were like, okay, that's super interesting. That doesn't sound right. Like, tell us where you got that from and where's that?

Like, we really kind of interrogated that knowledge base pretty critically. And even that in itself is an experiment that, you know, any association can think about what is being said about our industry or our profession out there broadly on the internet from reputable sources, and what do we do with that? And so, yeah, this view of that broad kind of listening, but really narrowing in on some key questions that are still being asked, that are being debated.

Then, okay, we're going to turn this into something that is usable, was always where this was going to go. We did it with the book. We had a thought.

We had a methodology. We had research. We produced a book.

Now, what are we going to produce that's also showing that interplay, that interoperability between us as humans and AI can produce something that's really meaningful? So, I think the hosts are synthetic, but the questions aren't. That's a really great way to frame where we ended up, right?

[Thomas Atlman] (7:01 - 10:49)

So, yeah, I think that's the point. So, what we want to do in this episode zero is kind of pull back the curtain a little bit, tell you how we came to this, how this was made. We want to inject that transparency, that human element back into it, keep this pretty unpolished for that purpose.

So, you know, it is more human. And the way we did this, the way we came across this is we started kind of with the conversations, right? We did this listening.

And then we had this repository of kind of the conversations that are being had, our own external research. And we said, what do we, we didn't even know we wanted to do a podcast at this point. We kind of went to the AI and said, what are some ideas here, right?

And one of the ones that popped out was, hey, what about a podcast, a serial style podcast where we ask and answer the most important questions per episode and the light bulb and all our heads went off. Like, that's a great idea. I don't, I don't know that any of us would have thought about that on our own.

And the way we made this, I think is important. As I think if you're not careful, if you guys are thinking about trying to do something like this yourself, if you're not intentional about how you employ these tools, you will end up with AI slop, right? We do not want, this is not slop.

Hopefully it's not slop. That is not our goal here. And the reason we're confident it's not is because we didn't just throw stuff at AI and copy and paste it and send it on.

We were very active in the way that we interacted with the AI on the front end, right? We were very intentional. Our own ideas were what initiated this process when we started kind of creating the podcast themselves.

We listened to the podcast, right? When we used a notebook LM to create these, it's got a very distinct style. We listened to the podcast.

We read through the transcripts. We were very critical, right? Most of these episodes went through six, seven, eight iterations where a couple interesting things happened.

On the first end, sometimes it was just not what we wanted. Most of the time, the first episode was just not it. Maybe it was mostly it, but there was stuff about it that was like, oh, no, I don't like that.

So we'd go back and kind of rewrite kind of our instructions back to the AI and have this dialogue back and forth. But the other thing, and I think maybe the more important thing that happened, so one, we were governed here. We advocate governed AI.

We absolutely governed what we did here. The other thing that happened, though, that I thought maybe was more interesting was that dialogue itself, like when I was getting frustrated that it wasn't doing what I wanted it to do, it forced me to do some self-reflection, right? It forced me to think about what actually I wanted, right?

It actually forced me, because I wasn't just accepting what AI did and pasting it, as being critical of the output. I was being critical of my own ideas, and it made me crystallize kind of my own thoughts about this in a way that I don't think I ever would have if I wasn't using AI to challenge myself. I frame it like that on purpose.

It's not AI was challenging me. AI was a tool that I used to challenge myself to think more deeply about the concept. And what we wanted to do here was kind of, one, open up that, you know, pull back that curtain and show you guys, you know, how we did this.

But then, two, as we did it, you'll find the structure of all our episodes starts with myself and Rob having a little bit of a back and forth, throwing it over to the Notebook LM produced podcast, and then an outro where we talk about it, right? We want to cap in both of those with the human experience and the way we interact with it. And what we're hoping is you can kind of see how that interaction happens, and then you yourself can contribute that too.

Hopefully, this forces you to be a little bit more critical, more thoughtful, disagree with, you know, both the AI and with us, right? But do so in a way that has kind of concrete, crystallized actions behind it. And that's kind of our goal here, and that's why we did this and how we did it.

[Rob Barnes] (10:50 - 14:54)

Yeah, the point that you made there, like most people who know me know that I'm relatively self-aware, but self-reflection hasn't necessarily been the strong suit. And you're quite right. At this point in the Betty journey, you know, we're two and a half years into this company, and we've done almost 200, you know, AI projects with different associations around the world.

And I didn't want to use that as a, I didn't want to make any assumptions about what would be meaningful or purposeful in this exercise, just because of that experience. And so you're right, on a far more rudimentary level than you, but my frustrations with what the AI was outputting was making me stop and think about, is it just assuming a bunch of things because the only thing that I've given it is my learned experience, I'm not asking it to explore outside of that. And so you're quite right, the process of, you know, interrogating it or sharing my frustrations with it or asking it to come back and challenge me to answer questions that I wouldn't have thought about, in and of itself created, again, some knowledge assets that became super useful in working out, well, what am I learning from these episodes each time?

You know, so actually going and reading a transcript and listening to an AI-produced podcast about stuff that we've researched is, I mean, it's a lot of fun, if nothing else, and particularly in the Notebook LM format. But it's actually insightful to look back and go, I would never have gleaned that insight from the same research just by myself, I just wouldn't have. And so being able to go and say, yeah, our format that we wanted is intentional, the human experience up front, us talking about what we, a preview about the episode, what we liked about it, where we had some challenges with it perhaps, then the outro, you know, the end of it, like where we're recapping it, but also saying, you know what, this was one of those episodes where we didn't necessarily agree with exactly what came out, nor do we have to agree, it just became the point of conversation and the point of learning, which every listener is going to interpret for themselves.

And that in itself is another important point, that you can go through this exercise to produce an asset, if you like, and in this case for us, it's 12 podcast episodes on the big questions that are being talked about in associations right now. But the learning is in the learning, you know, it's like it's learning how to learn from these things, the process by which we produced it, how we then fell into this really nice routine of previewing it and critiquing it afterwards, the number of times we went through that, but in really quick time, again, because the AI allowed us to do the fun stuff, which was to dig into the things that we learned and liked or didn't like.

And then also, you know, just making sure there's teachable takeaways, you know, from each of this, right, we want to make sure that association executives who do invest the time to listen to us and listen to what we've produced, understand the very clear distinction between what we're talking about, what the AI has produced, but also the companion asset, which again was a great idea. Let's have something that they can just take away and use, even if they don't ever listen to the podcast ever again, there is a document here that they can download and go, I can actually use this today in my job, then kind of our work is done, you know, and the amount of time we've invested to pull this together for everybody is well worth it if just one person says, well, I used this checklist or I got this, you know, I asked these questions in my staff meeting, it's like, okay, great, you know, that's exactly what we wanted to do it for. It's still the most fun thing to talk about how we did it and why we chose some of the tools as it is actually listening to the quality of the content that was produced from the research that we did.

[Thomas Atlman] (14:55 - 16:07)

I want to lean in that the companion asset, which I think is a really important part of this. So each episode has a companion asset, a workbook that allows you to kind of engage in that dialogue, you know, take it offline and think deeply about what you may have learned in each episode. And I want to call that out, one, it is a tangible benefit to you as you go through this that you can kind of hold on and reflect on and, you know, share around as you need to.

And I think you'll get a lot of value from that. I want to call that out too, because at Betty, we are all about this idea of governed AI, right? This is where we see the value for associations in that the AI is governed, it's traceable, it's auditable, it's coachable, right?

And we talk about that throughout the 12 questions. So the podcast itself is about that, but I want to call this out too, is that it's also an example of it. So the way that those companion assets were produced was through a process of governed AI, right?

It wasn't just like this idea of AI swap. We didn't just create stuff. We know where the sources came from.

We traceable to it. We understand how it was made. It's auditable.

When it was wrong, we coached it so that the next time it was right.

[Thomas Altman] (16:07 - 16:16)

And we've got a legacy, a repository of that coaching that then will inform kind of future versions of this experiment, whatever that might be, right?

[Thomas Altman] (16:16 - 16:36)

We're can kind of now expand out our knowledge asset creation. So yes, we are all about governed AI, but those companion assets are examples of how that is put into action. So kind of leading by example in this situation or just showing kind of eating our own, no, drinking our own champagne, almost said eating our own dog food, but we drink our own champagne at Betty.

[Rob Barnes] (16:36 - 18:39)

That's right. Yeah. And I think there's a great, you know, for us, it was a great experiment with some of the different tools that are available to us right now and, you know, how we produce those assets and, you know, even how we produce the name of the podcast in the first place and kind of iterate it on all of these sorts of things.

It can be a lot of fun with that kind of disciplined approach to the application of the applications as well. So the episodes as well are interchangeable. Like you don't have to follow it from one to 12.

You grab a topic, grab a question that's of interest to you right now, dig into that particular one. So in any order, I think it's going to be, you know, useful for people. And I think while voice has become very topical, I don't necessarily think it's that important, but it's very topical at the moment, like giving AI a voice.

The podcast was just a natural extension of that exploration because, you know, NotebookLM has had this audio capability for a good 12 months now and it's pretty cool. And so like, okay, what does our research sound like if we gave it to that? And I think that's the part that's like, you know, the sources are the important point.

It wasn't the hosts. It wasn't what the hosts sounded like. It had much nicer voices for podcasting than we do probably anyway, Thomas.

But because we governed the sources and we governed the way we wanted it to be produced, the hosts followed that. And by the time we got to the publishable versions of the episodes, we are stand behind. We are very confident that this is not AI slop, that this is rooted in genuine research and genuine questions and then capped off with our own experiences, you know, at the start and the end.

[Thomas Altman] (18:41 - 19:15)

So what I think we want you to do is, yeah, jump in wherever you need to. This is not, you know, you don't have to go from one through 12. Find the topic that's interest to you and then engage in the dialogue, whatever that means to you, whether that's through the companion asset, whether that's just sort of chatting with your colleagues at work or wherever you may be.

Engage with the dialogue yourself and kind of try to crystallize your own thoughts about AI. And if you disagree with us, that's great. That's what we, that's what you want.

You can disagree with us and the AI and have your own idea, but we ask that you share it. So find a way to share that back with us and we will love to learn from you guys.

[Rob Barnes] (19:15 - 20:42)

Yeah, I feel like that's actually a little, that is a little permission slip for everybody that's listening to the podcast at any point, is if you want to take the companion asset and load it into your AI tool of choice, if you're a co-work, if you're on a chat GPT, if you like perplexity, load it in there and have it do some critical analysis for you and see what it comes back with. And if you start getting some advice as to, like, this is a crackpot idea, this one, whatever, we would love to, like, book time with us at any point. If you want to have a conversation with us about the things that we learn from these big questions, just book time.

We can talk, you know, we can talk about this stuff under wet cement with a mouth full of marbles, I think, as my mother used to say. So we're open to those conversations, truly, because every time we have one of these conversations, we learn so much more about how these issues are impacting associations. And that's our wheelhouse.

That's where we, that's where we work. That's the work that we love every day. And, you know, producing the podcast is just another contribution, I think, to the body of knowledge for the associations industry that we know and love, that we hope people will have a bit of fun with and hopefully do some of their own experimenting after this.

Thanks, Thomas. Thank you, everyone, for listening to the Association Intelligence Podcast. This was episode zero. Join us on the first episode where we discuss this question: "Are we ready for AI?"

What this helps your team answer

How to use AI to research a topic without ending up with "AI slop" - what a rigorous, governed process for turning AI research into a real deliverable looks like.

An example model for pairing human judgment with AI output.

How to make AI-assisted work feel human and trustworthy instead of generic or robotic.

0:00 31:28

[Thomas Altman] (0:10 - 0:13)

Today's big question, are we ready for AI?

[Rob Barnes] (0:14 - 1:04)

We are Thomas Altman and Thomas Altman, and this is Episode 1 of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the AI Content Governance Readiness Checklist. Yes, that's a lot of words, but it's a very important asset.

It's linked in the show notes, so grab it now if you want to follow along, or download it after and put it to work. Today, we are digging into the first big question that we discovered online, and it's really one of the things that we hear a lot in conversation with associations, this idea of are we ready for AI and what that even means. Thomas, when you were doing the research for pulling together these big questions for associations, how did this resonate with you as one of the big things that we needed to start with?

[Thomas Altman] (1:05 - 1:51)

Yeah, so this was, to me, the reason why we're starting with this one is it's the biggest one, right? I think it's the one where there's the most hesitation. And what we've been finding is that this hesitation a lot of times is unfounded or turned on its head, but then, too, it's the most value-destroying hesitation you can have when approaching this question.

So one of the things we wanted to do through this episode was to kind of flip the script a little bit, kind of show you how you can get started with AI in a way that's going to be super valuable to you. And it all starts with this idea, the two dreadful words I know I hear a lot, Rob, I think you hear it every day, is digital transformation, and digital transformation specifically being a stumbling block. Rob, do you ever hear that?

Do you ever hear people say, oh, we're in the middle of a digital transformation?

[Rob Barnes] (1:52 - 3:17)

And it's funny how quickly in the last year or two that the meaning of that has changed dramatically. And yet it's still being treated the same. You know, like depending on what level of association executive we're talking to, yeah, digital transformation is either a really cool strategic excuse not to deal with a software vendor, a particular vendor, or literally that's the funded mandate from the board, is to undertake this digital transformation, and it needs to cut across user experience online, the tech stack itself is going to be completely either consolidated or expanded depending on which administration is in at the time. And I also hear that we need to get our house in order first.

You know, this concept that association's knowledge is so disparately housed that until you get it all consolidated or collaborated into one kind of place, you can't move forward with AI. And that's really interesting to me, this idea that it's fragmented, but it's the right content. You know, it might be incomplete, but only incomplete from what we know about our own body of knowledge.

You're right. It's such a challenging environment, and it is holding a lot of associations back from moving forward.

[Thomas Altman] (3:18 - 5:56)

So I think flipping the script a little bit, it's not get our house in order, a digital transformation so that we can start AI. What we've seen, and we've done nearly 200 successful implementations of AI projects for associations now, it's the other way around. Get AI in order so that you can get your house in order.

The process of getting AI set up is quicker time to value, and it informs a better strategy, a more future-looking strategy around kind of how you can do that digital transformation. And the way we want you to think about this, the thing that I think leads to people getting the most of value out of AI first, and then a digital transformation later, is when you think about what's the value that one, an association serves as related to its intellectual property, right? How can AI enable kind of those gaps, right, in getting that value from the intellectual property?

And then how can you inform that through whatever your strategy is going forward? So what we recommend is instead of thinking about, like, how do I do these crazy taxonomies, or how do I kind of reorganize my website, or I've got to get another new AMS, it's been five years since our last AMS, so we've got to get a new one, right? All that other stuff, I'd say, not so important.

More focused in, what's that end value? If what you're doing as an association is creating valuable intellectual property that will be consumed by your members, its general public, or whoever your constituencies are, and the problem is people are either not able to find it, not able to use it, not able to incorporate it in their day-to-day workflow so that they get value from this content. You can put AI first, but you need to do, the way you get your house in order is not digitally.

It's not through technology. It's through understanding your members, understanding what value they get, understanding where that content already lives. You don't have to reorganize or put it all in a single sort of data warehouse.

It can live in disparate places, in different kind of platforms, but as long as you know where it is and you can point the AI to that with the goal of helping people consume it, getting that process up and running, one, it shortens your time to value. You get more value in a shorter time frame by going that route first. Two, it informs the digital strategy going forward.

You're going to learn so much by doing that process first that once you finally get to kind of reorganize your website, you'll know what you want to do there. That's a lesson we've learned over and over again, and it's the biggest hurdle we've seen organizations like yours have trouble in overcoming before they get started. There's this fear of movement, and by taking that barrier down, we see people have a lot of success, both with AI and beyond.

[Rob Barnes] (5:57 - 6:36)

What I liked about this episode and the challenge that lives there is this idea that the biggest hidden ROI isn't really about member satisfaction. It's about an organization agreeing what the official answers are, what are the responses that the organization should be giving to a request to an AI, and is that really the silver lining? I think that's where this episode goes, and I really enjoy it is what is the silver lining that comes from all of this work, and then the interface with the members.

So I'm really looking forward to digging into it.

[Thomas Altman] (6:37 - 7:03)

Before we hand this over to our AI podcast host, a quick reminder that the following is generated using Notebook LM, but with heavy involvement from the Betty team to ensure that the content generated is valuable. For a deeper dive on our approach, we recorded a full episode where we discuss how and why we made this. If you're interested in understanding how this all came together, we recommend you listen to episode zero.

And with that, over to our AI hosts.

[AI Host 1] (7:07 - 7:14)

Right now, thousands of organizations are deliberately delaying artificial intelligence.

[AI Host 2] (7:14 - 7:15)

Yeah, they really are.

[AI Host 1] (7:15 - 7:30)

And they're doing it for what sounds like the most responsible, mature reason in the world. If you spend any time in leadership meetings, you've probably heard someone sigh and say, we can't deploy AI yet because we're in the middle of a massive digital transformation.

[AI Host 2] (7:31 - 7:32)

Right, the classic stall.

[AI Host 1] (7:33 - 7:47)

Exactly. It's this looming cloud. The assumption is that you simply must wait for the new website to be built or the new content management system to launch or that endless taxonomy overhaul to be perfectly finalized before you can even think about AI.

[AI Host 2] (7:47 - 7:54)

I mean, it sounds incredibly logical on the surface, right? You don't put new furniture into a house while you're still pouring the foundation.

[AI Host 1] (7:54 - 7:54)

Makes sense.

[AI Host 2] (7:54 - 8:05)

But based on the sources we're diving into today, that assumption is actually strategically backward. Waiting until your digital transformation is quote-unquote done is the wrong play.

[AI Host 1] (8:05 - 8:08)

Because the AI changes the transformation itself.

[AI Host 2] (8:08 - 8:26)

Exactly. Governed AI fundamentally changes the requirements of the very transformation you're trying to finish. See, that completely flips the standard boardroom logic.

Because the ultimate question everyone asks is, are we ready for AI? But what you're saying is that the premise of that question is just flawed from the start.

[AI Host 1] (8:26 - 8:49)

It is flawed. If governed AI is going to become a primary interface to your organization's knowledge, bolting it on at the very end of a website rebuild makes no sense. It alters how you need to think about natural language access, how you connect different sources, and how you establish access rules.

It really needs to come along for the ride.

[AI Host 2] (8:49 - 8:50)

Not just an afterthought.

[AI Host 1] (8:50 - 8:59)

Exactly. And we should ground this right away for you listening, this isn't just theory. These insights are drawn from patterns across nearly 200 association AI implementations.

[AI Host 2] (8:59 - 9:00)

Wow, 200.

[AI Host 1] (9:00 - 9:08)

Yeah. We're looking at the recurring hurdles and the actual successful pivots leaders make when dealing with vast amounts of member or staff knowledge.

[AI Host 2] (9:09 - 9:23)

That context is vital. And as you think about your own organization's digital ecosystem, which I'm guessing has a few messy corners, we actually have a highly practical tool linked in the show notes for you. Yeah.

That's the AI content and governance readiness checklist.

[AI Host 1] (9:23 - 9:38)

Exactly. Keep that handy because we're going to cover exactly how to apply it during this deep dive. But let's start with the paralysis itself.

Why do leaders feel they have to wait for the perfect website or system before bringing in AI?

[AI Host 2] (9:39 - 9:48)

Well, it stems from what we can call multi-platform fear. Most leaders know their institutional knowledge isn't sitting in one pristine database. It's totally scattered.

[AI Host 1] (9:48 - 9:49)

Oh, yeah. Everywhere.

[AI Host 2] (9:49 - 10:08)

You have data in your membership database, your AMS. You have articles in the website's CMS, courses in the learning platform, the LMS. Plus, like, decades of trapped knowledge in old PDFs and recorded committee meetings. Right. So the assumption becomes we have to clean all this up and move it into one centralized master system before an AI can read it.

[AI Host 1] (10:08 - 10:10)

Which sounds like a five-year project, honestly.

[AI Host 2] (10:10 - 10:22)

Minimum. And it creates a false binary. Leaders think the only options are either to pause everything and spend years moving files into one repository or to just recklessly connect an AI to everything and see what happens.

[AI Host 1] (10:22 - 10:24)

Upload everything and pray.

[AI Host 2] (10:24 - 10:34)

Yeah. But neither is true. Governed AI doesn't need you to build a single new filing cabinet.

Technologically, it functions more like a highly secure smart librarian.

[AI Host 1] (10:35 - 10:36)

OK, a smart librarian. I like that.

[AI Host 2] (10:36 - 10:46)

Yeah. So through API connections and secure indexing, it maps out where the information lives across your existing platforms without physically moving the files.

[AI Host 1] (10:46 - 10:48)

Oh, so the files stay where they are.

[AI Host 2] (10:48 - 10:55)

Exactly. It just needs to be told where the approved sources are and what the access rules are for those specific sources.

[AI Host 1] (10:55 - 11:07)

I want to make sure I understand the mechanism here, though. If the files are in completely different formats, say an old Word document on a shared drive, a modern Web page and, I don't know, a transcribed video file.

[AI Host 2] (11:07 - 11:07)

Yeah.

[AI Host 1] (11:07 - 11:11)

The AI doesn't need us to standardize all of those into one file type first.

[AI Host 2] (11:11 - 11:19)

Not at all. Modern AI parsing and indexing systems can ingest text from wildly different formats and convert them into mathematical representations. They're called vectors.

[AI Host 1] (11:20 - 11:20)

Vectors.

[AI Host 2] (11:20 - 11:34)

Yeah. So the AI isn't reading a PDF and a Web page differently. It's reading the underlying concepts mapped out in those vectors.

As long as it has a secure key to the room where that PDF lives, it can retrieve the answer.

[AI Host 1] (11:34 - 11:42)

So waiting to use AI because your files are scattered is basically like refusing to hire a brilliant researcher just because your books are in three different rooms.

[AI Host 2] (11:43 - 11:44)

That's a great way to put it.

[AI Host 1] (11:44 - 11:58)

The researcher doesn't care as long as they have the keys to the rooms. But wait, if the AI is just indexing concepts across all these different systems, what happens to the, like, hundreds of hours organizations spend building website navigation?

[AI Host 2] (11:58 - 11:59)

The taxonomy projects?

[AI Host 1] (11:59 - 12:03)

Yeah. The endless debates over taxonomy and drop-down menus.

[AI Host 2] (12:03 - 12:23)

Well, this is where Govern AI changes the priority of your digital transformation. It drastically reduces your dependence on perfect site navigation. Really?

Yeah. Think about the mental load we currently put on a user to find, let's say, a 2022 policy update. They have to reverse engineer the mental model of the person who designed the website.

[AI Host 1] (12:23 - 12:27)

They have to guess if it's under resources or advocacy or maybe about us.

[AI Host 2] (12:28 - 12:29)

Right. And if they guess wrong.

[AI Host 1] (12:30 - 12:31)

They assume the document doesn't exist.

[AI Host 2] (12:31 - 12:37)

Precisely. But AI doesn't navigate by clicking folders. It operates on natural language intent.

[AI Host 1] (12:37 - 12:39)

Meaning they just ask for what they want.

[AI Host 2] (12:39 - 12:55)

Yes. A member can just type, what changed in the safety policy a few years ago regarding site inspections? The AI understands the semantic intent of that question, the relationship between safety, inspections, and recent history, and fetches the relevant approved source directly.

[AI Host 1] (12:55 - 12:56)

That's huge.

[AI Host 2] (12:56 - 13:02)

It is. And even better, deploying Govern AI early actually accelerates your taxonomy project.

[AI Host 1] (13:02 - 13:06)

Wait, how so? By telling you what people are actually looking for.

[AI Host 2] (13:06 - 13:19)

Exactly. It produces real-time signal. Instead of guessing how members navigate, you see exactly what natural language shorthand they use, how they group topics in their minds, and where your current structure is failing to deliver.

[AI Host 1] (13:19 - 13:36)

So you use the AI's data to build a better website map. Let me push back on that, though. Because if the AI can perfectly read intent and deliver the right document, a web developer might hear this and think, great, we don't need a structured website anymore.

Just put a search bar on a blank page.

[AI Host 2] (13:36 - 13:46)

No, no, no. Structure still matters immensely. Members still want to browse, they still want curated experiences, and you definitely still need an organized digital presence.

[AI Host 1] (13:46 - 13:47)

Okay, good to clarify.

[AI Host 2] (13:47 - 14:02)

What AI does is remove the urgency of that structure being absolutely perfect before you can deliver value. You don't have to freeze your AI rollout while committees debate a sitemap for six months. The AI bridges the gap.

[AI Host 1] (14:02 - 14:16)

That shifts the pressure in a really healthy way. So if we accept that we don't need perfect navigation or a single master database to begin, we have to look at the content itself. Right.

Because the fear isn't just that the content is scattered, it's that the content is messy.

[AI Host 2] (14:16 - 14:32)

And let's be honest, everyone's content is messy. The key is understanding what kind of messy you are dealing with. The implementation patterns give us a very practical framework for this.

Instead of trying to clean everything at once, you triage your existing content into three distinct buckets.

[AI Host 1] (14:33 - 14:44)

Okay, let's unpack this. It's like deciding what to pack when you're moving houses, right? So if the goal is to figure out what's ready for AI without a massive cleanup, I imagine the first bucket has to be the content we actually trust.

[AI Host 2] (14:44 - 14:48)

That's it. The first bucket is content that is fragmented but accurate.

[AI Host 1] (14:49 - 14:50)

Fragmented but accurate.

[AI Host 2] (14:50 - 15:01)

Yeah. The knowledge might be spread out across recent journal articles, formal policy PDFs, and official conference transcripts. The storage is fragmented, but the source truth is highly trusted by your organization.

[AI Host 1] (15:02 - 15:03)

So what's the action for that bucket?

[AI Host 2] (15:04 - 15:11)

For this bucket, the action is simple. It is ready for launch or a strong pilot. You don't need to reorganize it.

You just let the governed AI index it.

[AI Host 1] (15:11 - 15:28)

That makes sense. It's like your good silverware scattered in different drawers, just pack it. But what happens when we have a topic where the documentation isn't entirely comprehensive, where maybe the official handbook has the basics, but the real nuance lives in the heads of your senior subject matter experts?

[AI Host 2] (15:28 - 15:45)

Ah, that brings us to bucket two. Incomplete or uneven. You have valuable knowledge, but the coverage is spotty.

If you put an AI purely on this content, it might give surface-level answers because it lacks the deeper context your staff usually provides in person.

[AI Host 1] (15:45 - 15:49)

So do we hold off on AI for that content until the staff writes everything down?

[AI Host 2] (15:49 - 15:55)

No. You run a narrow controlled pilot. You use the AI system as a diagnostic tool.

[AI Host 1] (15:55 - 15:56)

Oh, to see what it can't answer.

[AI Host 2] (15:56 - 16:09)

Exactly. When the AI struggles to answer a complex question from a test group, the system's data highlights the exact gap. You expect to coach the system here, and you use the AI to figure out exactly which documents your experts actually need to write.

[AI Host 1] (16:09 - 16:23)

Rather than having them just guess what's missing. So it's like a puzzle missing a few pieces. Pack it, but note it.

Which brings us to the third bucket. If bucket one is ready to go, and bucket two needs a pilot, bucket three has to be the liabilities.

[AI Host 2] (16:23 - 16:43)

Yes. Bucket three is contradictory, obsolete, permission confused, or high risk. Oh, bad stuff.

The bad stuff. We are talking about stale technical standards that were superseded five years ago, but never deleted. Or like conflicting compliance policies that could create legal risk if interpreted incorrectly.

[AI Host 1] (16:43 - 16:44)

So what do we do with this?

[AI Host 2] (16:44 - 16:56)

The action here is a hard stop. Do not scale this broadly to your members. You must clean this content, establish authoritative ownership, and set strict guardrails before AI is ever allowed to surface it.

[AI Host 1] (16:56 - 17:01)

So it's not about feeding everything into the machine. Bucket three is actual garbage don't pack the garbage.

[AI Host 2] (17:01 - 17:03)

Exactly. Don't pack the garbage.

[AI Host 1] (17:03 - 17:13)

And that garbage, the high risk, obsolete content, really naturally leads to the most critical part of governed AI, right? Making sure the system understands what it's not allowed to share.

[AI Host 2] (17:13 - 17:15)

Yes, the governance guardrails.

[AI Host 1] (17:15 - 17:29)

Let's talk about permissions, because a lot of organizations have deep archives on shared network drives or, you know, internal SharePoint sites. If an organization just plugs a generic AI search tool into their internal servers to index everything, what actually happens?

[AI Host 2] (17:30 - 17:34)

It creates a massive exposure problem. And it usually involves stale permissions.

[AI Host 1] (17:34 - 17:36)

Stale permissions, like what?

[AI Host 2] (17:36 - 17:48)

Well, think about how organizations typically work. Six years ago, an HR team drafts a sensitive internal policy. They put it in a shared folder.

Over time, people move on, folder permissions get inherited or changed.

[AI Host 1] (17:48 - 17:48)

Right.

[AI Host 2] (17:48 - 17:56)

And suddenly that draft is technically accessible to a wider group of staff, but no one realizes it because it's buried 30 folders deep.

[AI Host 1] (17:57 - 18:01)

So it's protected by obscurity. It was so hard to find that the broken permissions didn't matter.

[AI Host 2] (18:01 - 18:06)

Exactly. But an AI doesn't experience obscurity. It just reads the permissions.

[AI Host 1] (18:06 - 18:06)

Oh, wow.

[AI Host 2] (18:06 - 18:21)

If you ask a generic AI a question about that HR policy, it will instantly retrieve that half-baked sensitive draft because it perfectly matches the semantic intent of your prompt. And the system technically has access to it. The AI is just doing its job.

[AI Host 1] (18:21 - 18:22)

That is terrifying.

[AI Host 2] (18:22 - 18:33)

It is. Which is why the safest first move is to draw a hard boundary. When you start, you point the governed AI only at approved, external, or clearly member-facing content.

[AI Host 1] (18:33 - 18:35)

You just leave the internal stuff out.

[AI Host 2] (18:35 - 18:43)

You leave the internal drafts, and the messy staff network drives completely out of scope until you have audited who actually owns what.

[AI Host 1] (18:43 - 18:52)

Okay, that covers the internal permissions. But here's where it gets really interesting for me. What about public-facing content that just changes over time?

[AI Host 2] (18:53 - 18:53)

Versioning.

[AI Host 1] (18:53 - 19:05)

Yeah, versioning. Let's say we have an official safety standard from 2015 that is fully approved and public. But then we published a totally updated safety standard in 2025.

Both are public. Both are technically approved.

[AI Host 2] (19:06 - 19:06)

Yeah.

[AI Host 1] (19:06 - 19:11)

How does the AI know how to handle a question about safety standards? Doesn't it just look at the date?

[AI Host 2] (19:11 - 19:32)

You would hope so, but a generic generative model doesn't inherently understand version control unless it's explicitly governed. Yeah. Generative AI is designed to synthesize information.

Left to its own devices, it might pull a foundational paragraph from the 2015 standard and combine it with a new bullet point from the 2025 standard because both documents conceptually answer the prompt.

[AI Host 1] (19:32 - 19:36)

So it synthesizes them into a Franken-standard that doesn't actually exist in the real world.

[AI Host 2] (19:36 - 19:58)

Precisely. And in technical or compliance-driven fields, a synthesized Franken-standard is incredibly dangerous. We can imagine.

To prevent that, the system requires explicit current source rules. You have to apply source labels. Now, the 2015 standard is still valuable, right?

Members often need to compare the new regulation against the old one so you don't delete the 2015 document.

[AI Host 1] (19:58 - 19:59)

Use governance.

[AI Host 2] (19:59 - 20:13)

Exactly. The governance layer explicitly instructs the AI, treat the 2025 document as the authoritative truth for all current queries. Only reference the 2015 document if the user explicitly asks for historical context.

[AI Host 1] (20:14 - 20:17)

The AI cannot be left to infer that nuance on its own.

[AI Host 2] (20:17 - 20:18)

No, it can't.

[AI Host 1] (20:18 - 20:32)

That completely demystifies the idea that AI is just a plug-and-play magic wand. It requires human guidance to set those rules, which implies that rolling out AI isn't just an IT project. It's an ongoing operational practice for the staff.

[AI Host 2] (20:33 - 20:46)

It requires an active process called coaching. An AI model, out of the box, does not know the unique culture or the specific dialect of your organization. Your staff and your subject matter experts have to teach it over time.

[AI Host 1] (20:46 - 20:49)

What does coaching actually look like in practice, though?

[AI Host 2] (20:49 - 20:58)

It's about teaching the AI your organizational shorthand, abbreviations, retrieval cues. For instance, your members might constantly ask about the big show.

[AI Host 1] (20:58 - 21:00)

Oh, like a nickname for an event.

[AI Host 2] (21:00 - 21:15)

Right. The AI doesn't know what that means. A staff member coaches the system by mapping that phrase to the official annual global convention.

You are taking the tacit knowledge that usually only lives in the heads of your senior staff and embedding it into the AI's understanding.

[AI Host 1] (21:15 - 21:24)

That makes the system increasingly valuable over time because it's learning the specific language of the community. But, you know, coaching requires knowing what the members are actually asking. How do we see that?

[AI Host 2] (21:25 - 21:32)

Through usage data, which is often referred to as insights. This is where Governed AI becomes a strategic engine for the whole organization.

[AI Host 1] (21:33 - 21:33)

How so?

[AI Host 2] (21:33 - 21:49)

Well, every time a member asks a natural language question, the system attempts to answer it using the approved sources. When you review the insights data, you aren't just looking at search volume. You are looking for the queries where the AI struggled to generate a high confidence answer.

[AI Host 1] (21:50 - 21:56)

Because if the AI couldn't answer it based on our approved documents, it means our approved documents are missing the information.

[AI Host 2] (21:56 - 22:09)

Exactly. It turns real member questions into a literal roadmap for your content team. Unanswered questions show you exactly what standards you need to update, what blog posts you need to write, or what training modules you need to develop next.

[AI Host 1] (22:09 - 22:11)

That's incredibly powerful.

[AI Host 2] (22:11 - 22:21)

It is. We see this dynamic play out quietly with tools like Betty, which is a really practical example of this kind of governed, source-bound, multi-tier AI.

[AI Host 1] (22:21 - 22:21)

Right.

[AI Host 2] (22:21 - 22:29)

A system like Betty is coachable, and it's designed to activate the knowledge you have approved and then use insights to reveal the gaps you didn't know existed.

[AI Host 1] (22:30 - 22:33)

But importantly, a system like that doesn't write the new standard for you.

[AI Host 2] (22:33 - 22:41)

No, it doesn't. It doesn't magically fix your bad guidance, it doesn't repair your broken SharePoint permissions, and it absolutely does not replace human governance.

[AI Host 1] (22:41 - 22:43)

It's just activating what you have.

[AI Host 2] (22:43 - 22:51)

Yes. It relies entirely on the parameters you set. Which brings us to what we should call the honest beat of this entire readiness conversation.

[AI Host 1] (22:51 - 22:53)

The honest beat. I like that. The reality check.

[AI Host 2] (22:53 - 23:14)

Yes. Here's the candid truth. If an organization has a high-stakes, highly sensitive topic, but no one on staff can confidently identify what the actual authoritative source document is, or who owns the access rights to it, that topic is simply not ready for broad, member-facing AI.

[AI Host 1] (23:14 - 23:25)

Wow. So what does this all mean for you, the listener? It means AI isn't a magic wand that cleans your messy room.

It is simply a very bright flashlight that shows you exactly where the mess is.

[AI Host 2] (23:25 - 23:26)

That's exactly it.

[AI Host 1] (23:26 - 23:31)

And if the mess is a liability, you don't shine a spotlight on it for your members. You fix the source first.

[AI Host 2] (23:32 - 23:40)

It illuminates the gaps. That is the core of readiness. It's not about waiting for a pristine digital ecosystem.

It's about knowing your sources.

[AI Host 1] (23:40 - 23:58)

We've covered a lot of ground here in this deep dive. We started with the stall, that fear of the incomplete digital transformation. We walked through how AI actually acts like a smart librarian, indexing across platforms without moving data, and how it navigates by intent rather than rigid website structures.

[AI Host 2] (23:59 - 23:59)

Right.

[AI Host 1] (23:59 - 24:11)

We looked at sorting content into those three actionable buckets, the dangers of stale permissions, Frankenstandards, and the necessity of coaching. So how does a listener actually start this tomorrow without feeling overwhelmed?

[AI Host 2] (24:11 - 24:13)

You do a readiness pass, and you start small.

[AI Host 1] (24:13 - 24:14)

Okay.

[AI Host 2] (24:14 - 24:23)

Pick one single use case for your organization. Don't try to solve the whole website, just one use case. First, list the authoritative sources for that specific topic.

[AI Host 1] (24:23 - 24:24)

Just the trusted ones. Right.

[AI Host 2] (24:24 - 24:28)

Second, name the actual human content owners who are responsible for those sources.

[AI Host 1] (24:28 - 24:28)

Right.

[AI Host 2] (24:28 - 24:35)

Third, set the access tier. Clearly define exactly who gets to see this information. Is it public?

Is it for members only?

[AI Host 1] (24:35 - 24:39)

And then flag any of the versioning or permission risks we discussed.

[AI Host 2] (24:39 - 24:56)

Yes. And once you have that use case mapped out, look at the content and assign it to a bucket. Is it bucket one, fragmented but accurate and ready to launch?

Is it bucket two, incomplete and ready for a pilot? Or is it bucket three, and you need to clean it before moving forward?

[AI Host 1] (24:56 - 25:04)

It takes something that feels like an impossible multi-year IT overhaul and turns it into a targeted content strategy.

[AI Host 2] (25:04 - 25:05)

It makes it manageable.

[AI Host 1] (25:05 - 25:18)

And to make that strategy tangible, you can go to the show notes right now and grab the AI content and governance readiness checklist. It walks you through that exact sorting and planning process so you can take the safest next step for your association.

[AI Host 2] (25:18 - 25:48)

It gives you the map. And, you know, I want to leave you with one final thought to mull over. We spend so much time talking about the ROI of artificial intelligence in terms of member satisfaction or search efficiency.

But what if the greatest immediate ROI of AI isn't actually about giving your members perfect answers on day one? What if the greatest value is that implementing governed AI finally forces your organization to sit down, look at your scattered files, and agree on what your official answers actually are?

[AI Host 1] (25:48 - 26:00)

That is something to chew on. Forcing organizational alignment as the hidden ROI of AI. Love that.

We will leave it there for today. Grab the checklist in the show notes, start sorting those buckets, and thanks for joining us on this deep dive. We'll see you next time.

[Rob Barnes] (26:07 - 26:41)

So the big question today was, are we ready for AI? And I thought it was really interesting, Thomas, that the research pulled out this kind of risk use case around a document that was buried deep in, you know, and I think it's called it buried by obscurity. But all of a sudden it was resurfaced again, just because of the capability of generative AI and some of the challenges, you know, getting ready.

Do we even have to worry about being more cautious because of things like that potentially happening?

[Thomas Altman] (26:42 - 29:06)

Yeah, I think to me, that was super interesting, too. It wasn't something I was expecting, but it made me think. And one of the things I liked about it was this idea of understanding sort of the buckets of content that you can get started with first, right?

So this idea of if it's in SharePoint, a lot of times if you've got a legacy sort of Microsoft system that was set up in the 90s, and you've gone through like 10 different IT directors in the interim, who knows who has permissions or access to what? And really what's preventing people from accessing stuff they shouldn't access is the fact that they don't even know that they can. But AI would be able to, right?

So that's a genuine, real concern. And it's something that I really caution people around when they're thinking about adopting a solution like Copilot, right? Not that Copilot's bad, but like, it's something you need to think about.

And to me, that is definitely like a risk. It's something you don't want to think too much about. So then flipping that around, where is there content that is de-risked?

So I liked that the AI duo kind of dialed in on, it was around like the seven, seven and a half minute mark, where they started talking about like these different buckets, right? And the idea that you've got some content that I think you already are pretty confident in, right? So that might be your, you know, if you've got journal articles that you publish that went through peer review, if you've got stuff that's published on your website, you know, around upcoming events or things that just need to be correct and have gone through some process where you trust it.

Or if you've got standards or a handbook or a textbook, right? These are kind of sources of truth that have been vetted, right? So understanding what those vetted pieces of content are, they're more trustworthy, and you probably know who the audience is, you know, who's going to use it, you know how they want to use it, right?

So we're starting to build these sort of defendable use cases here, you know, the content, it's trustworthy, you know, who should have access to it and how they are intended to use it. And then you internally probably already have a set of rules that govern not just access, but you know what it can be used for. And once you have those three things, an audience, the sources, and the governing rules, that is a kind of a clean bucket that you then can use AI for.

So kind of carving those out and starting with those while you clean up your SharePoint, right? Those kind of permissions.

[Rob Barnes] (29:06 - 31:28)

And we hear this quite often, that sense of, what I love about associations is even through the natural onboarding of new teams over time, people who work there now know what the trusted sources of information is almost immediately. They know that the policy documentation that exists is the current truth, the journal articles, you know, the white papers, these sorts of things. They might not know the full extent of the library of all of that, but they know that that's the stuff that everybody can trust and that the members, the profession trusts in them sort of straight away.

So, you know, when they started talking about what AI isn't going to do, you know, in terms of fixing bad guidance or repairing, you know, the broken permissions of who's got access to what to do what, that human oversight, and I think, you know, we've always talked about the human in the loop, uh, is still an incredibly important part and a much more focused function.

And that’s what’s useful about today’s, um, companion asset that’s available for everyone to download, it’s really look at what is the purpose for implementing AI. You know, instead of just taking the board’s recommendation, which is “We need to be doing AI,” whatever that really means, look for the purpose first. And don’t be so concerned about the full, kind of, audit. You know, it’s not about creating a single clearing house necessarily. AI is going to help you get to that sort of path, but form follows function has always been a fun sort of thing for me. If you decide on the function that you need to solve, the form that the solution takes, in some cases, becomes kind of self-evident. I think the readiness checklist that we’ve prepared for everyone today is something that everyone is going to get a benefit from following this.

So, I would say, one more time before you go, everybody, download the companion asset today. It’s the fastest way to actually put today’s episode to work for you.

Thanks for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question: “What do we tell the board?”

What this helps your team answer

Why you should get AI in order first, and let it inform your digital transformation strategy.
How to triage content into readiness buckets before launch: fragmented but accurate, incomplete/uneven, and contradictory/obsolete, or permission-confused.
Two big governance risks to guard against: stale permissions and outdated versioning. Both require explicit source labeling, access rules, and ongoing human coaching.

Companion Asset

AI Content Governance Readiness Checklist

0:00 30:41

[Thomas Altman] (0:05 – 0:09)

Today's big question, what do we tell the board?

[Rob Barnes] (0:09 – 1:27)

We are Thomas Altman and Rob Barnes, and this is episode two of the Association Intelligence Podcast. Before we get into today's episode, this episode's companion asset is the board brief, Governed AI for Association.

It's linked in the show notes, grab it now if you want to follow along, or download it afterwards and put it to work. Thomas, this is an excellent question, and I'm glad it came up in the research that we did for Association Intelligence Podcast, because oftentimes, and I think even what do we tell the board? I'm hearing a lot is like what the board has told us, you know, from association executives.

It's like, as we talked about in the last episode, it's like, we have to do AI, but without any kind of real framework for any of that, and, you know, bringing a board a solution without any of the, I guess the preambles, like, I think about it from an alignment perspective, again, purpose, the purpose of the organization is not to become a software company and start developing its own AI. You know, do we see organizations tie themselves in knots now at a board level because of this question around, we have to be doing AI?

[Thomas Altman] (1:27 - 3:31)

Who doesn't love, who doesn't love the board meetings, right? Who doesn't love getting up and having to, you know, stand in front of a group or do your presentation, right? And to me, anytime I'm in front of a board, and we've done several of these, right?

It's the ultimate herding cats experience, right? You've got a bunch of different people from a bunch of different backgrounds, all who do care about the well-being of the association, but they bring their own biases or their own experience, right? So, you've got sort of the person who's obsessed with AI, and they want to be the AI guy, and they kind of get too into the weeds, and they lose sight, or you have kind of the, the person's a bit maybe more fearful of AI, and right, they're like, well, we don't want to do this, or we don't want to do that.

And how do you get all of these people on the same page, so that you're delivering value back to your members at the end of the day, right, that you're executing on the mission of your association, the best way you can with the technology that we have. And that's kind of the goal of this episode. And then sort of where people ask, like, I got to prep for my board meeting, what I need to do.

The thing that I thought was interesting is calling out that it's not really the AI that you want to focus on. It's the mission of the association, right? You don't want to bring a demo, because demos, or you don't want to bring just a shiny demo, and then that becomes a tax service, like people just start picking apart, like, oh, should this button be blue or red, right?

You don't want to get to that level. Or you don't want to start going in the weeds around sort of technical jargon, right, because, or like your favorite model, this model or that model, like, that doesn't matter. At the end of the day, what I think the thread that we were able to pull on with this, and kind of it's been at least my experience, I think Rob, yours as well, you want to be able to deliver a kind of a finely crafted, this is advancing our mission in this way.

It's protecting us from these particular concerns that are fairly common throughout any board meeting and they're valid, right? And this is the way we're going to do it. So focus on what you're delivering, how you're protecting your downside, and the value you're creating at the end of the day.

And Rob, I know you've been in many board meetings, I don't know what your take is.

[Rob Barnes] (3:31 - 5:53)

Yeah, the board meeting that has the self-anointed AI expert on the board, because they happen to do it in their job or for their company or wherever, it's that I've seen boards be derailed by that level of enthusiasm, because that individual has forgotten their role as a steward of the organization. And got into the exciting kind of new tool. And that also has an impact on the staff that are trying to navigate alignment.

And this is what this episode, I think, really brings to the fore quickly is this concept of alignment. When you are thinking about AI that's to solve a problem and create value, value needs to be aligned to strategic purpose. And this is where we want to keep our board is on the strategy.

They've agreed to three to five years strategic plan. They're still very common. So when we're looking at value, align it to the things that the board has already agreed to and remind them at this opportunity to say, hey, this is the stewardship that you've already agreed to.

This is the strategy going forward. This is an opportunity for us to, and you're right, it's less, hopefully at that point, it's kind of less about the tool itself. The boards that we talk to, they've got, they understand their fiduciary risks.

They're looking to protect the organization from any potential threats that seem to be well publicized around AI these days. But if we can bring that back to, again, the blueprint, this is the opportunity. This is the value.

This is the alignment to strategy. Then it does get us a little bit closer to that. And there are some great guardrails.

You know, I think in the episode it talks about some really interesting guardrails that everybody can kind of learn from. And, you know, that's part of the companion asset as well. You know, we see some of this stuff go sideways pretty quickly with boards and with an association, but we're also seeing ones that have got a little bit of courage mixed in with a real understanding of their stewardship, start to take great strides very quickly and get great value and realize the impact.

And ultimately that's what we want, right? Is we want the association to have impact, right?

[Thomas Altman] (5:52 - 6:18)

And with that, over to our AI hosts.

[AI Host 2] (6:24 - 6:25)

What do we tell the board?

[AI Host 1] (6:25 - 6:32)

Oh, man, that is, I think that's the single most critical question hanging over executive leadership right now.

[AI Host 2] (6:32 - 6:40)

Yeah, absolutely. I mean, if you're listening to this deep dive, chances are you are an association CEO or, you know, a COO, maybe a senior leader.

[AI Host 1] (6:41 - 6:42)

Staring down an upcoming board meeting.

[AI Host 2] (6:42 - 6:50)

Exactly. You know, artificial intelligence is a priority. Your members are asking about it.

Your staff is like already experimenting with it on their phones.

[AI Host 1] (6:50 - 6:52)

Yeah, on their personal devices, for sure.

[AI Host 2] (6:52 - 7:09)

And you are expected to present this coherent path forward to your governing body. But there's this massive gap between knowing you need a strategy and actually standing in front of a fiduciary board without the whole meeting just, you know, turning into a complete derailment.

[AI Host 1] (7:09 - 7:14)

Which happens a lot. And the approach we're outlining today really relies on what we called the demo versus blueprint dynamic.

[AI Host 2] (7:15 - 7:16)

Right. Break that down for us.

[AI Host 1] (7:16 - 7:30)

Well, the absolute worst maneuver an executive can make is walking into that boardroom, pulling up a screen and just asking the board to, like, approve AI based on some flashy generic technology demonstration.

[AI Host 2] (7:30 - 7:32)

Because that just screams risk to them.

[AI Host 1] (7:32 - 7:41)

Exactly. So we're going to map out the alternative. We're going to detail why you need to bring your board a one page blueprint instead of a pitch.

[AI Host 2] (7:41 - 7:55)

A blueprint that details what concrete value, specific risk controls, internal ownership and really strict rollout boundaries. Got it. Now, before we get into the actual mechanics of that, I do want to give you all a tool right out of the gate.

[AI Host 1] (7:56 - 7:57)

Oh, yeah. The companion asset.

[AI Host 2] (7:57 - 8:08)

Right. We have a companion asset for this deep dive called the board brief. Consider it like your show notes tool, specifically designed to help you structure this exact internal board conversation.

[AI Host 1] (8:08 - 8:09)

It's super helpful.

[AI Host 2] (8:09 - 8:19)

It really is. We'll circle back to it later. But just know that you don't need to furiously take notes or memorize the framework we're about to build.

That board brief is your practical next step.

[AI Host 1] (8:19 - 8:27)

Perfect. So before we can discuss how to present the right blueprint, we really have to look at what the board is actually apprehensive about.

[AI Host 2] (8:27 - 8:28)

You know, what's keeping them up at night.

[AI Host 1] (8:28 - 8:39)

Right. Looking at patterns from nearly 200 association AI implementations, the mapping shows us that board anxiety generally clusters around a few very specific areas.

[AI Host 2] (8:39 - 8:40)

Like what?

[AI Host 1] (8:40 - 8:45)

Content scope, intellectual property leakage, multi-tier access, traceability.

[AI Host 2] (8:46 - 8:47)

Traceability is a big one.

[AI Host 1] (8:47 - 8:55)

Huge. Plus answer quality, member trust, reputation risk. And honestly, who actually owns the review process?

[AI Host 2] (8:55 - 9:04)

OK, let me play devil's advocate here for a second, because if I'm an executive who's really pushing for modernization, I might find those fears a bit overblown.

[AI Host 1] (9:04 - 9:05)

Really? How so?

[AI Host 2] (9:05 - 9:18)

Well, I mean, if an association's PDFs and articles are already sitting on their public facing website, hasn't a system like ChatGPT already scraped and read them anyway? Like, why is the board acting like this is some sudden new threat to our intellectual property?

[AI Host 1] (9:18 - 9:31)

OK, I hear that a lot, but there is a profound mechanical difference between a public search engine scraping your public marketing copy and an association taking its proprietary multi-million dollar member research.

[AI Host 2] (9:31 - 9:33)

Oh, you mean the stuff hidden behind the paywall.

[AI Host 1] (9:33 - 9:38)

Exactly. Taking those paid assets and just plugging them into an unmanaged third party AI model.

[AI Host 2] (9:38 - 9:39)

Right. That makes sense.

[AI Host 1] (9:39 - 9:53)

Board members have a fiduciary duty to protect the organization's assets. So when they ask if a new AI tool is going to absorb their peer review journals and just feed them to the open Internet, I mean, they're asking a fundamental governance question.

[AI Host 2] (9:54 - 9:55)

They want to know if the pipe flows both ways.

[AI Host 1] (9:55 - 10:07)

Exactly. They need to know the pipe is sealed. But the common mistake we see leaders make is misinterpreting this apprehension as like a lack of technical understanding.

[AI Host 2] (10:08 - 10:09)

So they try to overcompensate?

[AI Host 1] (10:09 - 10:19)

Yes. The leader walks into the boardroom and tries to overcompensate by explaining the plumbing. They start throwing around terms like large language models, our RAG, vector databases.

[AI Host 2] (10:19 - 10:20)

Just trying to sound cutting edge.

[AI Host 1] (10:20 - 10:22)

Right. And it completely backfires.

[AI Host 2] (10:23 - 10:32)

Wait, before we move past that, for the executive listening who keeps hearing AAG from their IT department, what actually is that? Let's just define it quickly so we know what concept we're supposed to be avoiding in the boardroom.

[AI Host 1] (10:32 - 10:50)

Sure. So RAG stands for retrieval augmented generation. Instead of asking an AI to answer a question from its general memory, which is, by the way, how you get hallucinations and totally made up facts, RAG forces the AI to run a search through your specific documents first.

[AI Host 2] (10:50 - 10:52)

Oh, OK. So it has to look it up.

[AI Host 1] (10:52 - 11:02)

Yeah. It retrieves your document, reads it, and then generates an answer based strictly on what it just read. Think of it as changing the AI's task from a closed book exam to an open book exam.

[AI Host 2] (11:02 - 11:22)

Where the only book allowed is your official archives. That makes total sense. But to your point, pitching the mechanics of retrieval augmented generation or showing some flashy demo to a board without a governance model.

I mean, it's like trying to sell someone a sports car by only showing them the top speed right when they're asking to see the safety crash ratings.

[AI Host 1] (11:22 - 11:31)

That is a perfect analogy. And treating the phrase we have an AI policy as if it is the same thing as having a governed system. That's another major pitfall.

[AI Host 2] (11:31 - 11:34)

Yeah. A policy on a piece of paper isn't a system.

[AI Host 1] (11:34 - 11:45)

Right. Leading with hype or competitor fear, like saying we have to do this because rival associations are instead of leading with architecture. Well, that guarantees the board will pump the brakes.

[AI Host 2] (11:45 - 11:50)

But safety ratings alone don't sell the car either. Nobody buys a Volvo just to leave it parked in the driveway.

[AI Host 1] (11:50 - 11:51)

Fair point.

[AI Host 2] (11:51 - 11:59)

We need governance. Sure. But governance is simply the condition that makes it safe to unlock the actual value sitting in the association's archives.

[AI Host 1] (11:59 - 12:10)

Right. The board conversation cannot be a referendum on AI enthusiasm. If the boardroom devolves into a philosophical debate about whether AI is good or bad for humanity, the initiative is dead.

[AI Host 2] (12:11 - 12:11)

Oh, totally dead.

[AI Host 1] (12:11 - 12:22)

The conversation must be entirely about value activation. Because associations are sitting on a staggering amount of intellectual property that they have already paid to create.

[AI Host 2] (12:22 - 12:37)

Yeah. We are talking about decades of conference recordings, webinars, certification guides, industry standards, peer reviewed research, FAQs, not to mention the subject matter expertise that's just trapped in the heads and like the email outboxes of senior staff.

[AI Host 1] (12:37 - 12:49)

Exactly. And much of that value is completely inert right now. It's trapped in a PDF on some clunky legacy portal or buried in an MP4 video file from an annual conference three years ago.

[AI Host 2] (12:49 - 12:59)

Right. If a member needs to know the association's stance on a highly specific regulatory change, they are definitely not going to sit down and watch a two hour conference video from 2022.

[AI Host 1] (12:59 - 13:04)

No way. Not just to find the three minute segment where a panelist happened to discuss it.

[AI Host 2] (13:04 - 13:15)

Right. A recorded annual conference session shouldn't be a one time asset. A member shouldn't have had to be sitting in a specific hotel ballroom on a specific Tuesday two years ago to get the value of that insight.

[AI Host 1] (13:15 - 13:23)

Beautifully said. And what a governed source bound knowledge assistant does mechanically is multiply that existing value by making it findable and evergreen.

[AI Host 2] (13:23 - 13:25)

It brings it back to life.

[AI Host 1] (13:25 - 13:39)

Yes. It transforms an association's static dormant archives into an active on demand mentor. The AI itself is just a translation layer.

The actual product is your historical knowledge delivered at the exact moment a member experiences a problem.

[AI Host 2] (13:39 - 13:50)

OK. So if the goal is to safely unlock that trapped value without, you know, triggering the board's risk aversion, we really need to talk about how to construct that one page blueprint.

[AI Host 1] (13:50 - 13:51)

The blueprint is everything.

[AI Host 2] (13:52 - 14:02)

You mentioned five specific boundaries that define the exact shape of the box we are asking the board to approve. Let's look at the mechanisms behind those. The first one is the source boundary.

Right.

[AI Host 1] (14:02 - 14:18)

The source boundary dictates exactly what content the system is mechanically permitted to read. You present a hard line to the board. This system will only pull from our approved 2023 and 2024 conference transcripts, our peer reviewed journals and our official policy handbook.

[AI Host 2] (14:18 - 14:21)

And it is completely disconnected from the open Internet.

[AI Host 1] (14:21 - 14:28)

Exactly. That immediately lowers the temperature regarding intellectual property leakage. You are defining the universe of knowledge.

[AI Host 2] (14:28 - 14:47)

But what about the access boundary? Because how does a system actually differentiate between what a non-member is allowed to see versus a premium paying member? Like if I'm on the board, I am terrified that an AI is going to accidentally summarize our $500 certification study guide for a free website visitor.

[AI Host 1] (14:47 - 14:55)

Oh, absolutely. And that's why the access boundary requires the AI to integrate with your existing association management system or AMS. OK.

[AI Host 2] (14:56 - 14:56)

How does that work?

[AI Host 1] (14:56 - 15:10)

Well, when the user logs in, the AMS passes a token to the AI saying, hey, this person is a tier one premium member. The AI then searches the vector database, but it only retrieves documents tagged for tier one.

[AI Host 2] (15:10 - 15:11)

Oh, wow.

[AI Host 1] (15:11 - 15:18)

So if a non-member asks the exact same question, the system physically cannot see the premium study guide. It respects the entitlement structures you have already built.

[AI Host 2] (15:18 - 15:24)

That is super smart. OK, let's push on the third boundary, because I think this is where implementations actually die.

[AI Host 1] (15:24 - 15:25)

The answer and traceability boundary.

[AI Host 2] (15:25 - 15:33)

Right. Because even if you restrict the sources and lock down the access tiers, large language models are just notorious for trying to be overly helpful.

[AI Host 1] (15:33 - 15:34)

They want to please you.

[AI Host 2] (15:34 - 15:51)

Right. They combine two correct facts into one massive, embarrassing hallucination. If I am a board member, my nightmare is our official AI giving a member legally disastrous advice.

How do you prevent them from shutting the project down the very first time it makes a mistake?

[AI Host 1] (15:51 - 16:02)

You architect the system so it is fundamentally incapable of returning an answer without providing a literal clickable footnote back the specific paragraph it pulled from.

[AI Host 2] (16:02 - 16:11)

So instead of thinking of the AI as a college graduate who read all your files and is trying to summarize them from memory, it's more like an ultra fast intern with a highlighter.

[AI Host 1] (16:11 - 16:22)

I love that. Yes, an intern with a highlighter. It isn't generating original thoughts.

It is only allowed to fetch and highlight the exact paragraph in your archives. It has to show its work.

[AI Host 2] (16:22 - 16:24)

So mechanically, how does it do that?

[AI Host 1] (16:24 - 16:33)

The system maps the user's query to an exact digital coordinate in your database, a vector point. It finds the paragraph sitting at that exact coordinate and retrieves it.

[AI Host 2] (16:33 - 16:34)

And if it can't find a match?

[AI Host 1] (16:34 - 16:49)

If it cannot find a match within the approved source boundary, it is programmed to say, I don't know, but here is a human you can contact. Traceability gives the board the accountability they require. Proving the system is only fetching your approved truth.

[AI Host 2] (16:49 - 17:02)

OK, but what happens when the intern highlights the wrong paragraph? Or, you know, what if the source document it fetches is technically correct based on a 2019 policy, but functionally wrong because the industry moved on?

[AI Host 1] (17:03 - 17:08)

And that brings us to the human coaching boundary. AI is not a set it and forget it technology.

[AI Host 2] (17:08 - 17:09)

No, definitely not.

[AI Host 1] (17:10 - 17:17)

The board needs to know whose desk the buck stops at, which specific staff members or subject matter experts own the coaching and review process.

[AI Host 2] (17:17 - 17:19)

So there has to be a way to course correct.

[AI Host 1] (17:19 - 17:31)

Exactly. If an answer needs to be adjusted, there must be a dashboard. Where human staff member can look at the query, see the document the AI fetched and manually correct the pathway for the future.

You are showing the board who holds the steering wheel.

[AI Host 2] (17:31 - 17:43)

Which naturally leads to the final piece of the blueprint, the rollout and success boundary. I imagine you never walk into a board meeting and announce a plan to launch an AI to 50,000 members globally on a Tuesday.

[AI Host 1] (17:44 - 17:53)

Oh, please do not do that. A phased pilot is universally more credible than a sweeping, untethered transformation plan. You contain the blast radius.

[AI Host 2] (17:53 - 17:54)

Start small.

[AI Host 1] (17:54 - 18:06)

Right. You propose rolling the system out exclusively to the internal membership team for the first 30 days to help them answer phone queries faster. Or you run a beta test strictly with your top tier committee members.

[AI Host 2] (18:06 - 18:25)

You tell the board exactly who gets it first and what metrics will determine if it expands. Exactly. So we have built this highly governed, deeply accountable box.

But how do we measure if it is actually worth the investment without promising the board guaranteed financial outcomes? Because if we over promise ROI in year one, we are setting ourselves up for failure.

[AI Host 1] (18:26 - 18:42)

We definitely frame the rollout as a series of measurable hypotheses. You are bringing the board pads to test. And looking at the implementation patterns across those 200 associations, we see three primary hypotheses that organizations test during a pilot phase.

[AI Host 2] (18:42 - 18:43)

Let's hear them.

[AI Host 1] (18:43 - 18:45)

First is a member return behavior hypothesis.

[AI Host 2] (18:45 - 18:53)

OK, so how does the AI mechanically shift a member from visiting the website once a month to, say, once a week?

[AI Host 1] (18:53 - 18:57)

Think about it. Traditional association search bars are often incredibly frustrating.

[AI Host 2] (18:57 - 18:58)

Oh, the worst.

[AI Host 1] (18:58 - 19:07)

A member types in a query about a complex compliance issue and the search bar returns 40 unhelpful links to massive 100 page PDF documents.

[AI Host 2] (19:07 - 19:12)

And then the member has to download them and use CTRL plus F just to find their answer. Right.

[AI Host 1] (19:12 - 19:25)

But if the new system provides an instant plain language summary with a direct citation to the exact page they need, all that friction vanishes. So the hypothesis is, if we remove the friction of finding knowledge, does the frequency of member engagement increase?

[AI Host 2] (19:26 - 19:27)

I'd bet it does. What's the second hypothesis?

[AI Host 1] (19:28 - 19:36)

The second is the non-member preview or the join path hypothesis. This seems like it could fundamentally alter how marketing departments operate.

[AI Host 2] (19:37 - 19:38)

Oh, interesting. How so?

[AI Host 1] (19:38 - 20:02)

Consider the psychological trigger of the access boundary we discussed earlier. A non-member lands on your site and asks a highly technical industry question. The system provides a high level two sentence summary.

But the deep dive citations, the actual mechanics of the solution. Are blurred out. Yes, they're blurred out or locked behind a prompt that says log in or join to read the full standard.

[AI Host 2] (20:02 - 20:16)

Oh, that is the ultimate show don't tell for membership value. You aren't just sending them some generic marketing email claiming you have the best resources. You are proving you have the exact answer to the problem that you're trying to solve right now and offering them the key to unlock it.

[AI Host 1] (20:16 - 20:25)

Exactly. It's incredibly powerful. And the third hypothesis might actually be the most operationally transformative.

The intelligence and content gap hypothesis.

[AI Host 2] (20:26 - 20:29)

Moving from search bar zero to actual query intelligence.

[AI Host 1] (20:30 - 20:45)

Exactly. When a member uses a traditional search bar and finds nothing, they just leave. The association never knows what they were even looking for.

But with a conversational system, you capture the exact natural language questions your industry is asking.

[AI Host 2] (20:45 - 21:04)

So if hundreds of members suddenly query the system about a new international supply chain regulation and your system returns empty handed because your association hasn't written a policy on it yet, you now have real time intelligence. You are literally crowdsourcing your editorial calendar based on immediate member needs rather than just guessing what topics to cover at the next annual conference.

[AI Host 1] (21:05 - 21:19)

Bingo. A practical example of this structural approach in the real world is a system like Betty. It is designed around these exact patterns, a governed, source bound, multi-tier coached and traceable association AI.

[AI Host 2] (21:19 - 21:20)

Right.

[AI Host 1] (21:20 - 21:30)

And when we look at implementations like the one at ISA, the International Society of Automation, what is really notable is how they built their deployment team. They didn't just treat it as an isolated IT project.

[AI Host 2] (21:30 - 21:30)

Which is so common.

[AI Host 1] (21:30 - 21:37)

Right. Instead, they assembled a cross-functional group involving governance, membership, sponsorship and IT.

[AI Host 2] (21:38 - 21:53)

And having a board liaison involved directly in the pilot group is just an incredibly strategic move because then the board is no longer sitting in judgment of some black box technology at the end of the quarter. They are active participants in tuning the governance model.

[AI Host 1] (21:54 - 22:03)

It demonstrates profound organizational maturity. However, any executive presenting this blueprint must deliver an honest caveat to the board.

[AI Host 2] (22:03 - 22:04)

There's always a catch.

[AI Host 1] (22:04 - 22:15)

Always. Even with the perfect blueprint, the most robust governance model and a brilliant cross-functional team, we have to set realistic expectations about what source bound AI absolutely cannot do.

[AI Host 2] (22:15 - 22:26)

Right. Because if the association's source material is an unorganized mess full of contradictory policies from different decades, the AI is not going to magically organize it into a cohesive strategy.

[AI Host 1] (22:26 - 22:52)

Exactly. Source bound AI is a mirror. It does not fix bad source content.

If your PDS from 2018 directly contradicts your web copy from 2024, the AI will simply fetch both and expose that conflict with incredible speed. Yeah. It will not resolve internal policy disputes.

It does not remove bias that is already present in your historical source material. And it absolutely does not solve existing organizational governance problems.

[AI Host 2] (22:53 - 22:54)

Garbage in, garbage fetched.

[AI Host 1] (22:55 - 22:55)

Precisely.

[AI Host 2] (22:56 - 23:13)

But honestly, I would position that as another point in favor of the blueprint approach. An executive can stand in front of the board and say, look, this system is going to shine a harsh light on where our archives are messy and where our content is weak. And we welcome that.

We want to know where the gaps are so we can clean them up.

[AI Host 1] (23:14 - 23:29)

That is a great spin. It turns a potential operational negative into a strategic positive. It manages expectations beautifully.

The board won't expect a flawless omniscient oracle. They will expect a tool that retrieves what the association has built exactly as it was built.

[AI Host 2] (23:29 - 23:44)

We have covered a tremendous amount of ground today regarding the mechanics of governance, boundaries and hypotheses. The core takeaway for the listener preparing for that upcoming meeting is clear. Do not walk into the boardroom and ask your governing body to approve AI.

[AI Host 1] (23:44 - 23:44)

We don't do.

[AI Host 2] (23:45 - 23:54)

If you do, you are asking them to underwrite an unlimited, unbounded concept and their fiduciary risk aversion will shut the initiative down immediately.

[AI Host 1] (23:55 - 23:57)

Bring them a one page architecture and value brief.

[AI Host 2] (23:57 - 23:57)

Yes.

[AI Host 1] (23:57 - 24:06)

Show them the bounded box. Show them exactly what historical knowledge assets will be activated, what the system can access and what it physically cannot access.

[AI Host 2] (24:06 - 24:12)

Show them the traceability mechanism and define where the initial rollout is intentionally restricted.

[AI Host 1] (24:12 - 24:12)

Yes.

[AI Host 2] (24:13 - 24:25)

And to help you translate the strategy into a physical document, you can actually slide across the boardroom table. I want to remind you to grab the board brief companion asset we mentioned at the start of this deep dive.

[AI Host 1] (24:25 - 24:26)

It's basically a cheat sheet.

[AI Host 2] (24:27 - 24:33)

It really is. It's structured specifically to help you build out this exact one page conversation. It is your immediate practical next step.

[AI Host 1] (24:34 - 24:56)

And as you prepare that brief and define the value of unlocking your archives, consider the long term mechanical shift this creates. Once a member can instantly query your association's entire history of standards, webinars and peer reviewed research at the exact moment they face a crisis at their desk. How does that fundamentally shift the core value proposition of an association?

[AI Host 2] (24:56 - 24:57)

That's a huge question.

[AI Host 1] (24:57 - 25:05)

It is moving away from just being about networking and annual events to actually be an indispensable daily operational partner. That's the real transformation.

[Rob Barnes] (25:11 - 25:19)

Super interesting. The big question today, what do we tell the board? Thomas, what did you take away from that?

[Thomas Atlman] (25:20 - 27:23)

So I like this one and I like the way it kind of it drove to what I think was a pretty conclusive ending, right? That the idea that you want to approach the board with sort of this value brief, this this way to unlock kind of your archives, consider the long term mechanical shift. I think that was the the idea here.

How does your value proposition as an association change and have a brief ready for the board in that matter? So the idea to me is thinking along those terms of. Kind of what are what are we actually asking the board to approve?

Like you don't you're not just blindly saying we're asking the board to let us use AI. We're asking the board to let us use AI so that we can fill in the blank. Right.

What is that so that we can and then thinking in those terms like who are who are the people you're using it for? Like frame it more as value towards your member, advancing your association's ambition, your three to five year plan concretely using this tool. Right.

And it's not you want to shift the conversation away from the tool and more about the value and thinking in those terms, thinking the value breeding thing, how it contributes to the larger strategic mission that the board is there to oversee and then thinking about the guardrails, sort of how you want to protect your your downsides. Right. So if questions are.

Does this protect our IP or are we giving away our IP to be trained on some other model and we actually lose control? Right. Understanding what that is, that's a very common concern that I think gets addressed.

The other one is who has access to it by letting people use AI? Are we kind of undercutting our value proposition? Right.

That's another concern that I think you need to have ready to address there. Like do non-members get access to paid materials and how do we govern that? So once you identify those guardrails, you can create a governing strategy around that and including that as part of your brief, I think that makes it much more powerful and it makes it something the board is excited to approve instead of worried about.

[Rob Barnes] (27:23 - 29:10)

It's super interesting to me that the research that picked out International Society for Automation part of this, because I felt like, you know, when we look at the five boundaries that are shown in the companion asset for today's episode, I reflect back on the conversations I was having with the team at ISA and they had a very good sense of these boundaries like right from the get-go and which probably speaks to the success of MIMO, award-winning MIMO, I might add, now because they really did have a good sense of what was the, you know, what's the source, what's the approved content that we wanted to make accessible in ways that wasn't possible before, you know, who can get access to it, how they had learned that, you know, that automation engineers, professionals around the world were wanting, you know, the responses that they needed from this content source, you know, these assets themselves, and always this really strong sense of the human in the loop, you know, they have this cross-functional team that own MIMO and the way MIMO performs, it's not just an IT thing, it's not just a membership or governance or marketing thing, it's very much some cross-ownership, and so that human in the loop element was there right from the get-go, and I think that the, well, I know that the way they have sort of reported upstream and downstream, you know, the way the board have taken ownership of MIMO's value and alignment to strategy has really helped keep them not just, that's thriving, like it's growing in its capability and its use because of that strategic alignment, the board owns that now.

[Thomas Atlman] (29:11 – 30:09)

And I think, like it ended on this note, it really stuck with me, this podcast, where it says you want to move away from being sort of about networking and annual events and more into an indispensable daily operational partner, right? So the idea here is that because you've invested this time in identifying the value and helping people unlock kind of value that's already there, like get access to stuff, you aren't this punctuated, like at our annual conference, people are engaged, but then we don't see them again until the next annual conference, or people stay alongside with you as an operational partner, the association then gets embedded in the workflows and the daily lives of its members. And kind of reframing the value around that, which I think is something ISA did really really really well. And kind of presenting that as the end goal back to your board, I think makes it much more of an exciting prospect and let’s you get to yes much more easily than kind of getting dragged into the weeds around what model do you want to use.

[Rob Barnes] (30:10 – 30:41)

So one more time, before you go, thanks everyone for listening today. Download the Board Brief: Governed AI for Associations from the show notes. It’s the fastest way to actually put today’s episode to work. Thank you for listening to the Association Intelligence Podcast. Join us next week to answer this question: “Why not just use ChatGPT or Copilot?”

What this helps your team answer

How to frame AI adoption without triggering board risk aversion.
Which outcomes matter to governance-minded leaders, and what boundaries (source, access, traceability, human ownership, rollout scope) need to be defined to earn board confidence and approval.
How to set realistic, measurable expectations for a pilot instead of overpromising ROI, and what you should track to prove its working.

Companion Asset

Board AI Briefing One-Pager

Tool Selection

Why Not Just Use ChatGPT or Copilot?

A practical breakdown of when to use ChatGPT, Copilot, or governed AI — separating personal productivity, Microsoft ecosystem work, and the official answers your association has to stand behind.

36:07

0:00 36:07

00:00.000 --> 00:08.140
Today's big question, why not just use ChatGPT or Copilot?

00:08.140 --> 00:13.380
We are Thomas Altman and Rob Barnes and this is Episode 3 of the Association Intelligence

00:13.380 --> 00:16.020
Podcast.

00:16.020 --> 00:22.320
Before we get into it, this episode's companion asset is the ChatGPT, Copilot or Governed

00:22.320 --> 00:25.160
AI Use Case Decision Guide.

00:25.160 --> 00:29.640
It's linked in the show notes, so grab it now if you want to follow along or download

00:29.640 --> 00:32.120
it after and put it to work.

00:32.120 --> 00:40.440
Thomas, one of the most fun episodes I think, simply because the analogies that came out

00:40.440 --> 00:48.960
of the research and the interaction with Betty and the AI tools, it's just incredibly fun.

00:48.960 --> 00:56.740
Talk us through the custom GPT as a treehouse bit because this really resonates and I want

00:56.740 --> 01:02.820
to prep everybody that's about to listen to the call to stay on track with this.

01:02.820 --> 01:06.300
This is one of those things where this whole project of creating this podcast has been

01:06.300 --> 01:13.300
so eye-opening for me because it's this interaction with human talking to AI, the dialogue, us

01:13.300 --> 01:15.660
kind of debating and going back and forth.

01:15.660 --> 01:19.420
And then something comes out of that that's not just from the AI or just from us, it's

01:19.420 --> 01:20.580
something more.

01:20.580 --> 01:26.420
And the anecdotes or the metaphors it comes up with are really helpful.

01:26.420 --> 01:29.300
They help me sort of crystallize it in this idea.

01:29.300 --> 01:32.580
So the topic today is like, why not use chat GPT or copilot?

01:32.580 --> 01:36.380
Why do you need sort of a governed solution for associations?

01:36.380 --> 01:41.560
The idea of a treehouse being a custom GPT is so perfect.

01:41.560 --> 01:46.220
The idea that you can build a treehouse and you can go to the backyard and sort of do

01:46.220 --> 01:49.980
it over the weekend and it's going to be cool and you'll learn a lot by doing it.

01:49.980 --> 01:52.100
You'll kind of learn how to build stuff.

01:52.100 --> 01:54.900
But are you going to move your corporate headquarters into that treehouse?

01:54.900 --> 01:55.900
There's no plumbing.

01:55.900 --> 01:56.900
There's no AC.

01:56.900 --> 01:57.900
You're not going to live there.

01:57.900 --> 01:59.660
You can't like put a bed in it.

01:59.660 --> 02:06.860
There's so much more that goes from a cool project that is cool and is fun and is learning.

02:06.860 --> 02:11.100
And we're not saying don't build the treehouse, but there's a difference between that and

02:11.100 --> 02:15.820
something kind of built for being a corporate headquarters.

02:15.820 --> 02:18.820
And just when it came up with that, when I heard it's like a treehouse, it's like, how

02:18.820 --> 02:21.020
did that even come into play?

02:21.020 --> 02:23.300
It's so smart.

02:23.300 --> 02:27.620
I think the perspective that I'm hearing a lot in the conversations I'm having every

02:27.620 --> 02:35.980
week is this, again, this idea of yes, and right.

02:35.980 --> 02:40.700
The Jobs to be Done framework that we use at Betty, it really plays into this nicely.

02:40.700 --> 02:43.820
It's like chat GPT, yes.

02:43.820 --> 02:44.820
Individual productivity.

02:44.820 --> 02:49.980
Obviously, we use called co-work instead of chat GPT, but we use codecs for a bunch of

02:49.980 --> 02:50.980
things as well.

02:50.980 --> 02:53.140
Like that stuff, yes.

02:53.980 --> 02:54.980
And co-pilot.

02:54.980 --> 02:59.020
If you're a Microsoft office shop and you came into the Microsoft office in the 80s

02:59.020 --> 03:01.300
and 90s and you're still there, great.

03:01.300 --> 03:07.780
It's an ecosystem specific kind of environment across applications.

03:07.780 --> 03:13.300
Governed AI is around what do you do when you need to scale capability and skill and

03:13.300 --> 03:18.260
your knowledge assets to external parties.

03:18.260 --> 03:21.980
So I'm often having conversations with Association staff.

03:21.980 --> 03:25.580
And when they ask this question about why can't we just use an enterprise license of

03:25.580 --> 03:31.060
GPT, it's like you're thinking about it like a staff member instead of thinking about it

03:31.060 --> 03:41.580
in terms of your whole Association's mandate and the audiences that it serves.

03:41.580 --> 03:48.020
And then there are some real risks with playing the internal productivity or my internal ecosystem.

03:48.020 --> 03:51.700
When you start to try and think that you can scale that outside and into the, I'll call

03:51.740 --> 03:56.740
it the public domain, but external to the staffing environment, there's some risks still

03:56.740 --> 03:59.420
associated with that which come up in this episode, right?

03:59.420 --> 04:00.900
Yeah, that's exactly the point.

04:00.900 --> 04:04.260
So yeah, co-pilot's great for its purpose.

04:04.260 --> 04:09.340
Chat GPT or Claude co-work, great for their purpose.

04:09.340 --> 04:13.300
Not all those purposes line up perfectly with the Association use case, right?

04:13.300 --> 04:20.080
So as you're going through this episode, kind of listen for those, listen for this fun imagery

04:20.080 --> 04:21.520
that it comes up with, right?

04:21.520 --> 04:26.920
So on the co-pilot side, think about how you're inheriting the permission structure that was

04:26.920 --> 04:32.480
set up maybe by three or four IT directors ago, where the reason that the intern isn't

04:32.480 --> 04:38.800
accessing sort of the CEO's compensation package might not be because they're not allowed to.

04:38.800 --> 04:41.720
It might just be because they don't know they can, right?

04:41.720 --> 04:45.600
And the second you install co-pilot or you use co-pilot, all those permissions get carried

04:45.600 --> 04:49.320
forward and the intern could just say, hey, what's the CEO's compensation package, right?

04:49.320 --> 04:52.800
Those kinds of things happen all the time, right?

04:52.800 --> 04:57.320
And the imagery there was turning on the fluorescent lights in the dive bar at 2 AM and you get

04:57.320 --> 05:01.160
to see all the screw stuff you just didn't know you were sitting in, right?

05:01.160 --> 05:05.920
So those ideas or kind of when is a custom GPT appropriate?

05:05.920 --> 05:07.880
Because it can be in some cases, right?

05:07.880 --> 05:08.880
And when is it not?

05:08.880 --> 05:14.080
Like how is a custom GPT trained on, you know, three or four documents different from scaling

05:14.080 --> 05:18.960
that to over a hundred thousand documents, which many associations do.

05:18.960 --> 05:23.400
And what happens when you ask the question and it doesn't find the right document?

05:23.400 --> 05:24.600
Can you coach GPT on that?

05:24.600 --> 05:25.600
No, you can't.

05:25.600 --> 05:27.440
A governed AI solution, you can, right?

05:27.440 --> 05:31.480
These are the differences that I want people to keep in mind, not so that you always choose

05:31.480 --> 05:37.000
the governed solution, but so that these can co-exist for the use case that they're appropriate

05:37.000 --> 05:38.000
for.

05:38.000 --> 05:39.800
Enjoy this episode, everybody.

05:39.800 --> 05:41.520
It's super interesting.

05:41.520 --> 05:44.720
And we hand over to the AI podcasts host.

05:44.720 --> 05:50.040
A quick reminder that the following is generated using notebook LM with heavy involvement from

05:50.040 --> 05:54.560
the Betty team to ensure that the content generated is valuable.

05:54.560 --> 05:59.400
For a deeper dive on our approach, please go back and listen to episode zero.

05:59.400 --> 06:08.080
And now over to the AI hosts.

06:08.080 --> 06:14.080
Listen a brand new hire asking, you know, your organization's AI to summarize a recent

06:14.080 --> 06:16.080
departmental meeting.

06:16.080 --> 06:21.520
And instead of just giving them the minutes, the tool spits out the CEOs unredacted compensation

06:21.520 --> 06:22.520
package.

06:22.520 --> 06:23.520
Oh, wow.

06:23.520 --> 06:24.520
Right.

06:24.520 --> 06:27.620
And like maybe a list of internal restructuring plans from five years ago.

06:27.620 --> 06:30.880
And here is the wildest part of that whole scenario.

06:30.880 --> 06:32.520
The AI didn't malfunction.

06:32.520 --> 06:33.520
Right.

06:33.520 --> 06:34.520
It didn't go rogue.

06:34.520 --> 06:35.520
Exactly.

06:35.520 --> 06:36.520
It didn't break.

06:36.520 --> 06:40.720
It was actually Microsoft co-pilot working flawlessly doing exactly what it was designed

06:40.720 --> 06:41.720
to do.

06:41.720 --> 06:47.560
It is a it's a pretty sobering reality check, honestly, because it forces us to look at

06:47.560 --> 06:50.840
the tools we already have from a completely different angle.

06:50.840 --> 06:54.840
Which is exactly what we are doing today, because if you're an association leader or

06:54.840 --> 06:58.240
say a tech director, you are looking at your budget right now.

06:58.240 --> 06:59.240
Yeah, you're seeing it everywhere.

06:59.240 --> 07:00.240
Right.

07:00.240 --> 07:03.260
You're seeing AI integrated into literally every piece of software you own.

07:03.260 --> 07:05.960
So you are inevitably asking a very, very practical question.

07:05.960 --> 07:08.800
Why not just use chat, GPT or co-pilot for everything?

07:08.800 --> 07:10.600
It's the most natural question to ask.

07:10.600 --> 07:11.640
It really is.

07:11.640 --> 07:16.440
So today in this deep dive, we are answering that question by looking at the hard data.

07:16.440 --> 07:22.960
We're pulling from documentation that analyzes patterns from nearly 200 association AI implementations.

07:22.960 --> 07:26.280
Which is a huge sample size for this specific sector.

07:26.280 --> 07:27.280
Exactly.

07:27.280 --> 07:28.600
So we aren't talking about theories here.

07:28.600 --> 07:32.560
We are looking at how this actually plays out in the wild.

07:32.560 --> 07:36.700
But okay, let's unpack this, because I think we should establish a ground rule for this

07:36.700 --> 07:37.860
deep dive right up front.

07:37.860 --> 07:39.220
Yeah, I think that's crucial.

07:39.220 --> 07:43.380
Our approach today is firmly rooted in, well, what we call a yes and philosophy.

07:43.380 --> 07:44.380
A yes and.

07:44.380 --> 07:45.380
Okay, explain that.

07:45.380 --> 07:51.100
So the takeaway here is absolutely not that chat GPT or Microsoft co-pilot are bad tools

07:51.100 --> 07:52.420
or that you shouldn't use them.

07:52.420 --> 07:55.060
I mean, they're remarkable engineering achievement.

07:55.060 --> 07:56.060
Truly incredible.

07:56.060 --> 07:57.060
Right.

07:57.060 --> 08:01.380
So the mission is really to understand how to use generic AI where it fits perfectly

08:01.380 --> 08:06.640
and to avoid the trap of confusing, you know, a personal productivity tool with a system

08:06.640 --> 08:10.780
capable of delivering an official governed answer on behalf of your association.

08:10.780 --> 08:15.320
That distinction is so important because it's just so tempting to consolidate everything.

08:15.320 --> 08:16.320
Right.

08:16.320 --> 08:23.800
I mean, if you were already paying for Microsoft E3 or E5 licenses, co-pilot is likely sitting

08:23.800 --> 08:25.240
right there in your tech stack.

08:25.240 --> 08:27.000
Yeah, you're already paying for it.

08:27.000 --> 08:28.000
Exactly.

08:28.000 --> 08:32.100
So staff are definitely already figuring out how to prompt chat GPT to write their emails

08:32.100 --> 08:33.480
faster.

08:33.480 --> 08:37.720
So the natural instinct for any leader is to maximize the investment you've already

08:37.720 --> 08:40.640
made rather than going out and buying something brand new.

08:40.640 --> 08:41.640
Right.

08:41.640 --> 08:46.120
I mean, sunk costs and existing vendor relationships are incredibly powerful drivers.

08:46.120 --> 08:52.640
But expecting one single tool to manage every AI workflow in an organization is, well, it's

08:52.640 --> 08:54.880
like expecting a Swiss army knife to build a house.

08:54.880 --> 08:56.040
Oh, I like that analogy.

08:56.040 --> 08:57.040
Right.

08:57.080 --> 08:58.840
You can have a screw, you can saw a really small branch.

08:58.840 --> 08:59.840
Yeah.

08:59.840 --> 09:01.720
You definitely wouldn't use it to pour a concrete foundation.

09:01.720 --> 09:07.660
And when we look at the patterns from nearly 200 association AI implementations, the organizations

09:07.660 --> 09:10.600
that succeed are the ones that separate the jobs to be done.

09:10.600 --> 09:11.760
The jobs to be done.

09:11.760 --> 09:12.760
Exactly.

09:12.760 --> 09:15.120
They separate them into three very distinct categories.

09:15.120 --> 09:19.360
Okay, let's lay those out because I think having a mental map of these three jobs is

09:19.360 --> 09:21.840
really going to anchor the rest of our conversation today.

09:21.840 --> 09:22.840
Sure.

09:22.840 --> 09:25.720
So the first category is individual productivity.

09:25.720 --> 09:27.800
And this is really where chat GPT shines.

09:27.800 --> 09:29.760
Right, like writing a draft or something.

09:29.760 --> 09:30.760
Exactly.

09:30.760 --> 09:32.640
It is your ultimate brainstorming partner.

09:32.640 --> 09:35.000
It's your drafter, your summarizer.

09:35.000 --> 09:38.520
Then the second category is ecosystem specific work.

09:38.520 --> 09:40.080
And that is co-pilot.

09:40.080 --> 09:41.720
Because it lives inside Microsoft.

09:41.720 --> 09:42.720
You got it.

09:42.720 --> 09:47.120
It is highly specialized for your Microsoft 365 environment.

09:47.120 --> 09:52.640
So synthesizing your emails, drafting Word documents, summarizing Teams meetings.

09:52.640 --> 09:53.640
That's its lane.

09:53.640 --> 09:54.640
Okay, so that's two.

09:54.640 --> 09:55.640
What's the third?

09:55.640 --> 09:58.840
The third category is entirely separate and that is governed AI.

09:58.840 --> 10:03.240
This is a layer designed exclusively for official approved association knowledge access.

10:03.240 --> 10:05.040
Okay, meaning what exactly?

10:05.040 --> 10:09.120
Meaning this is the space where accuracy and authorization are just non-negotiable.

10:09.120 --> 10:10.120
It's for official answers.

10:10.120 --> 10:14.400
Okay, so we have the individual drafter, the Microsoft assistant, and then the official

10:14.400 --> 10:16.080
spokesperson.

10:16.080 --> 10:19.280
And before we tear into the mechanics of why these are so fundamentally different, I do

10:19.280 --> 10:22.200
want to point you to a highly practical asset.

10:22.200 --> 10:24.000
We've actually linked in the show notes for you.

10:24.000 --> 10:25.600
Yeah, this is a great resource.

10:25.600 --> 10:26.600
It really is.

10:26.600 --> 10:30.440
It's called the chat GPT co-pilot or governed AI.

10:30.440 --> 10:32.000
Use case decision guide.

10:32.000 --> 10:37.440
It maps out these exact boundaries we're talking about and helps you evaluate your own internal

10:37.440 --> 10:38.440
projects against them.

10:38.440 --> 10:43.320
So you can open that up right now actually and use it to follow along with what we're

10:43.320 --> 10:44.760
about to unpack.

10:44.760 --> 10:49.760
And having that guide as a visual companion is so helpful, especially as we dive into

10:49.760 --> 10:52.680
probably the most complicated tool on that list, which is co-pilot.

10:52.680 --> 10:53.820
Oh, absolutely.

10:53.820 --> 10:58.460
Because co-pilot is native to the tools most associations use every single day.

10:58.460 --> 11:03.100
So it is almost always the first place leaders look when they want to deploy AI organization

11:03.100 --> 11:04.100
wide.

11:04.100 --> 11:05.100
Naturally.

11:05.100 --> 11:09.140
And Microsoft pushes it heavily as this seamless, secure integration.

11:09.140 --> 11:10.140
Right.

11:10.140 --> 11:14.780
But let's go back to that scenario I opened with, the intern pulling the CEO's compensation

11:14.780 --> 11:15.780
package.

11:15.780 --> 11:21.020
Like, how does a supposedly obedient tool create such a massive security headache?

11:21.460 --> 11:25.060
Well to understand that we really have to look at the underlying mechanics of it.

11:25.060 --> 11:28.600
So co-pilot relies on the Microsoft Graph API.

11:28.600 --> 11:30.820
It does not ingest the public internet.

11:30.820 --> 11:33.340
It ingest your organization's internal data.

11:33.340 --> 11:34.340
Okay.

11:34.340 --> 11:35.660
So it only knows what's inside the walls.

11:35.660 --> 11:36.660
Exactly.

11:36.660 --> 11:39.980
But the critical feature of co-pilot is that it acts as a perfect mirror.

11:39.980 --> 11:41.700
Oh, perfect mirror.

11:41.700 --> 11:42.700
Yes.

11:42.700 --> 11:47.100
It flawlessly honors the permissions that are currently set within your SharePoint,

11:47.100 --> 11:49.500
your OneDrive folders, and your team's channels.

11:49.500 --> 11:50.500
Right.

11:50.500 --> 11:54.280
And if it is properly restricted to just the HR department, co-pilot won't show it to

11:54.280 --> 11:55.620
a marketing intern.

11:55.620 --> 11:56.620
In theory, yes.

11:56.620 --> 11:58.260
Ah, in theory.

11:58.260 --> 11:59.260
Exactly.

11:59.260 --> 12:03.260
Because here's where the operational reality completely collides with the technical design.

12:03.260 --> 12:04.260
Think about it.

12:04.260 --> 12:09.340
How many associations actually have pristine permission hygiene going back a decade?

12:09.340 --> 12:11.220
Oh, none.

12:11.220 --> 12:12.220
Literally none.

12:12.220 --> 12:13.220
Right.

12:13.220 --> 12:17.900
Think about, you know, a sensitive HR policy that was maybe accidentally saved to a company-wide

12:17.900 --> 12:20.460
SharePoint drive back in like 2018.

12:20.460 --> 12:21.780
It's just been sitting there.

12:21.780 --> 12:22.780
For years.

12:22.780 --> 12:23.780
Yeah.

12:23.780 --> 12:25.900
Because that document was protected by obscurity.

12:25.900 --> 12:30.660
Like no human was ever going to spend a random Tuesday afternoon clicking through eight layers

12:30.660 --> 12:33.500
of badly named subfolders to go find it.

12:33.500 --> 12:34.500
Right.

12:34.500 --> 12:35.940
So wait, here's where it gets really interesting.

12:35.940 --> 12:40.660
You're saying that co-pilot doing its job safely and correctly is exactly what creates

12:40.660 --> 12:45.680
a discoverability risk if our own internal digital hygiene is a mess.

12:45.680 --> 12:47.020
That is exactly what I'm saying.

12:47.020 --> 12:49.220
AI essentially eliminates obscurity.

12:49.220 --> 12:50.220
Wow.

12:50.460 --> 12:51.940
Co-pilot doesn't get tired.

12:51.940 --> 12:55.500
Doesn't get frustrated by your terrible folder naming conventions.

12:55.500 --> 12:59.680
So if an employee asks a broad question and co-pilot has the technical permission to see

12:59.680 --> 13:05.340
that misplaced 2018 HR document, it just grabs it, it instantly retrieves it and synthesizes

13:05.340 --> 13:06.820
it right into the answer.

13:06.820 --> 13:09.500
The discoverability risk just skyrockets.

13:09.500 --> 13:11.740
So co-pilot isn't broken in this scenario.

13:11.740 --> 13:15.260
It is holding up a mirror to the reality of your data governance.

13:15.260 --> 13:19.220
It's like it's like turning on the fluorescent overhead lights in a dive bar at two point

13:19.220 --> 13:20.220
a.m.

13:20.220 --> 13:21.220
Yeah.

13:21.220 --> 13:24.820
Suddenly you see all the dirt and the grime that you were perfectly happy ignoring in

13:24.820 --> 13:25.820
the dark.

13:25.820 --> 13:29.900
That is a very apt if slightly unsettling way to put it.

13:29.900 --> 13:30.900
But yeah.

13:30.900 --> 13:35.500
And the implication for association leaders here is that permission hygiene is an absolute

13:35.500 --> 13:36.500
prerequisite.

13:36.500 --> 13:37.620
You have to clean house first.

13:37.620 --> 13:38.620
You have to.

13:38.620 --> 13:42.340
You cannot simply turn co-pilot on for all staff and just let it run if you haven't

13:42.340 --> 13:47.020
done the unglamorous manual work of cleaning up your legacy access controls.

13:47.100 --> 13:49.700
OK. But let's follow the chain reaction here.

13:49.700 --> 13:53.820
Let's say an I.T. director realizes their SharePoint is an absolute nightmare.

13:53.820 --> 13:55.620
They do the responsible thing, right?

13:55.620 --> 13:59.020
They lock co-pilot down until they can audit all those permissions.

13:59.020 --> 14:00.020
Which could take months.

14:00.020 --> 14:01.020
Exactly.

14:01.020 --> 14:05.120
But the staff still have jobs to do and they know AI can help them do those jobs twice

14:05.120 --> 14:06.240
as fast.

14:06.240 --> 14:11.100
So if the official internal tool is locked behind a massive I.T. audit, staff aren't

14:11.100 --> 14:12.540
going to just sit on their hands.

14:12.540 --> 14:13.540
Definitely not.

14:13.540 --> 14:19.740
They're going to open a new browser tab and use a generic external tool like ChatGPT.

14:19.740 --> 14:23.480
And that introduces the massive challenge of shadow AI.

14:23.480 --> 14:26.940
The source material highlights this as a very recurring pattern.

14:26.940 --> 14:28.380
Shadow AI.

14:28.380 --> 14:29.780
Meaning unofficial use.

14:29.780 --> 14:30.780
Right.

14:30.780 --> 14:33.700
When official tools are too slow or they're too locked down or they just simply don't

14:33.700 --> 14:37.560
exist yet, people will find a workaround to be productive.

14:37.560 --> 14:40.540
And the thing is, the intent is almost always good.

14:40.540 --> 14:42.900
They are literally just trying to be efficient.

14:42.900 --> 14:43.900
Right.

14:43.900 --> 14:46.940
Nobody's maliciously trying to leak company secrets on purpose.

14:46.940 --> 14:51.460
They just want to write a member newsletter without staring at a blank page for two hours.

14:51.460 --> 14:52.460
Exactly.

14:52.460 --> 14:57.340
But from an organizational standpoint, this shadow AI use creates a cascading series of

14:57.340 --> 14:58.340
liabilities.

14:58.340 --> 15:02.740
You know, you have staff feeding proprietary membership strategies or intellectual property

15:02.740 --> 15:04.140
directly into public models.

15:04.140 --> 15:05.260
Oh, yeah.

15:05.260 --> 15:06.260
Which is bad.

15:06.260 --> 15:07.260
Very bad.

15:07.260 --> 15:11.620
And you have potential privacy breaches if member data is included in those prompts.

15:11.620 --> 15:16.300
But perhaps most critically for an association, you completely lose your single source of

15:16.300 --> 15:17.300
truth.

15:17.300 --> 15:18.300
Okay.

15:18.300 --> 15:20.140
I want to drill into that source of truth problem for a second.

15:20.140 --> 15:24.660
Because if I have, say, three different staff members in the membership department and they

15:24.660 --> 15:30.220
all ask a generic AI model, a complicated question about a lap stem or grace period.

15:30.220 --> 15:31.220
Right.

15:31.220 --> 15:32.220
A specific policy question.

15:32.220 --> 15:33.220
Yeah.

15:33.220 --> 15:37.580
They might get three slightly different, completely unapproved answers just based on how they word

15:37.580 --> 15:38.580
at the prompt.

15:38.580 --> 15:39.580
Right.

15:39.580 --> 15:40.580
Exactly.

15:40.580 --> 15:44.700
And the staff members, you know, copies that hallucinated answer and emails it to a dues

15:44.700 --> 15:45.700
paying member.

15:45.700 --> 15:46.700
Oh, boy.

15:46.700 --> 15:47.700
Yeah.

15:47.700 --> 15:51.380
The association is now legally and reputationally on the hook for a policy that doesn't actually

15:51.380 --> 15:53.180
exist.

15:53.180 --> 15:58.140
You've essentially outsourced your institutional voice to an algorithm that you don't control.

15:58.140 --> 16:02.020
So I can already hear an objection forming from some of our more technically inclined

16:02.020 --> 16:03.180
listeners right now.

16:03.180 --> 16:04.180
Let's hear it.

16:04.180 --> 16:07.580
They might say, well, we know the public chat GPT model is too broad.

16:07.580 --> 16:09.620
So we paid for the enterprise tier.

16:09.620 --> 16:16.340
We created a custom GPT and we literally uploaded the PDFs of our 2024 bylaws and our

16:16.340 --> 16:18.140
certification requirements.

16:18.140 --> 16:19.940
The walled garden approach.

16:19.940 --> 16:20.940
Exactly.

16:20.940 --> 16:22.420
We built a walled garden.

16:22.420 --> 16:24.020
Doesn't that solve the problem?

16:24.020 --> 16:25.020
Doesn't that make it governed?

16:25.020 --> 16:28.300
It is the most common pivot we see in the implementations.

16:28.300 --> 16:29.400
It really is.

16:29.400 --> 16:32.660
But the documentation is incredibly clear on this.

16:32.660 --> 16:38.000
Building a custom GPT or a quick prototype is a phenomenal way for a team to learn.

16:38.000 --> 16:39.000
To learn how it works.

16:39.000 --> 16:40.000
Right.

16:40.000 --> 16:45.360
It teaches staff how models interact with documents and how prompting works in general.

16:45.360 --> 16:49.060
But a custom GPT is fundamentally an illusion of governance.

16:49.060 --> 16:53.160
It is simply not sufficient for a governed member facing official use case.

16:53.160 --> 16:55.000
See, I'm going to push back on that a little bit.

16:55.000 --> 16:57.400
If I explicitly upload the PDF, right?

16:57.400 --> 17:02.100
And I literally tell the system only use this PDF, why isn't it governed?

17:02.100 --> 17:03.960
Because of the underlying architecture.

17:03.960 --> 17:08.880
When you upload a PDF to a custom GPT, the system essentially reads it as flat text.

17:08.900 --> 17:14.060
It lacks the metadata and really the architectural rigor required to manage complex access.

17:14.060 --> 17:15.060
Okay, I'm following.

17:15.060 --> 17:16.060
Give me an example.

17:16.060 --> 17:17.060
Let's use your bylaws example.

17:17.060 --> 17:22.520
Suppose a board member and a student member both ask that custom GPT a question.

17:22.520 --> 17:26.240
The custom GPT does not know the difference between those two users.

17:26.240 --> 17:28.160
It cannot enforce audience tiers.

17:28.160 --> 17:29.160
Ah, okay.

17:29.160 --> 17:31.780
So it can't say, oh, you only have student level clearance.

17:31.780 --> 17:35.960
So I'm going to redact this specific financial clause from my answer.

17:35.960 --> 17:36.960
Exactly.

17:36.960 --> 17:38.140
It just gives everything to everyone.

17:38.140 --> 17:44.080
And furthermore, a custom GPT rarely has built in auditable source traceability down

17:44.080 --> 17:47.140
to the specific paragraph level.

17:47.140 --> 17:52.320
Nor does it have an interface for human subject matter experts to intercept and correct weak

17:52.320 --> 17:53.880
answers over time.

17:53.880 --> 17:55.120
So no review loops.

17:55.120 --> 17:56.120
Right.

17:56.120 --> 18:00.940
Unless you are intentionally designing and building robust source scoping, access controls,

18:00.940 --> 18:05.480
traceability layers, review loops, and ownership protocols, which is a ton of work.

18:05.480 --> 18:06.480
Yeah.

18:06.480 --> 18:09.260
GPT is really just a slightly more specialized shadow tool.

18:09.260 --> 18:10.260
Wow.

18:10.260 --> 18:12.740
It's kind of like building a really impressive tree house in your backyard.

18:12.740 --> 18:13.740
A tree house.

18:13.740 --> 18:14.740
Yeah.

18:14.740 --> 18:15.740
Like it was super fun to build.

18:15.740 --> 18:18.760
It works perfectly for a weekend hangout, but you would never move your association's

18:18.760 --> 18:20.500
actual corporate headquarters into it.

18:20.500 --> 18:21.740
I mean, it doesn't have plumbing.

18:21.740 --> 18:23.960
It doesn't meet commercial fire codes.

18:23.960 --> 18:28.700
And the structural integrity basically relies on whoever happened to nail the boards together.

18:28.700 --> 18:30.380
That is a brilliant way to look at it.

18:30.380 --> 18:35.540
The infrastructure of governance is just entirely missing in the tree house, which really brings

18:35.540 --> 18:36.960
us to the alternative.

18:36.960 --> 18:37.960
Right.

18:37.960 --> 18:43.120
If Copilot is a mirror reflecting our internal messes and custom GBTs are just tree houses

18:43.120 --> 18:47.920
lacking infrastructure, what does the actual corporate headquarters look like?

18:47.920 --> 18:53.360
Like what does a tool built specifically for official answers actually entail?

18:53.360 --> 18:56.360
So now we need to define governed association AI.

18:56.360 --> 18:59.320
What are the actual mechanics that separated from generic models?

18:59.320 --> 19:00.320
Yeah.

19:00.320 --> 19:01.320
Break that down for us.

19:01.320 --> 19:05.320
A governed AI layer is architecturally different from the ground up.

19:05.340 --> 19:10.980
Instead of relying on a broad internet scrape or even flat file uploads, it relies exclusively

19:10.980 --> 19:13.500
on a controlled database of approved sources.

19:13.500 --> 19:14.500
Only the good stuff.

19:14.500 --> 19:15.500
Right.

19:15.500 --> 19:19.860
And it vectorizes that data, which basically means it maps the relationships between concepts.

19:19.860 --> 19:24.060
But critically, it tags every single piece of data with metadata.

19:24.060 --> 19:25.060
Okay.

19:25.060 --> 19:26.260
So it's not just reading the text.

19:26.260 --> 19:28.660
It's reading the rules attached to the text.

19:28.660 --> 19:29.660
Exactly.

19:29.660 --> 19:33.060
And the sources point to a specific implementation pattern for this.

19:33.060 --> 19:38.880
They often refer to a system named Betty, but, and this is important, it is vital to

19:38.880 --> 19:41.400
frame tools like Betty correctly.

19:41.400 --> 19:42.400
Meaning what?

19:42.400 --> 19:46.760
Meaning governed AI is not a magic bullet that magically cleans all your messy data

19:46.760 --> 19:47.760
for you.

19:47.760 --> 19:48.760
Oh.

19:48.760 --> 19:51.080
And it absolutely does not replace your core infrastructure.

19:51.080 --> 19:52.360
Oh, that's huge.

19:52.360 --> 19:53.360
Yeah.

19:53.360 --> 19:57.040
It does not replace your association management system, your AMS, where all your transaction

19:57.040 --> 19:58.040
data lives.

19:58.040 --> 20:01.640
It does not replace your learning management system, your LMS.

20:01.640 --> 20:04.300
It certainly doesn't replace your actual human staff.

20:04.300 --> 20:05.300
Okay.

20:05.300 --> 20:09.100
So if it isn't replacing the AMS or the LMS, where does it actually sit?

20:09.100 --> 20:12.920
It sits as a governed knowledge layer on top of your approved content.

20:12.920 --> 20:15.920
So mechanically, it features strict audience tiers.

20:15.920 --> 20:20.140
It cross-references the user asking the question with the permissions attached to the data.

20:20.140 --> 20:21.140
Okay.

20:21.140 --> 20:22.140
So it knows who is asking.

20:22.140 --> 20:23.140
Yes.

20:23.140 --> 20:25.540
Ensuring the board member gets the full financial breakdown and the student member just gets

20:25.540 --> 20:26.980
the public summary.

20:26.980 --> 20:29.540
And it provides complete unbreakable source traceability.

20:29.540 --> 20:30.540
Right.

20:30.540 --> 20:31.540
The show your work test.

20:31.540 --> 20:32.540
Exactly.

20:32.540 --> 20:37.440
Every single claim it generates is directly linked back to the approved association document.

20:37.440 --> 20:43.360
It has to literally say, I am making this claim and here is the exact paragraph from

20:43.360 --> 20:46.480
the 2024 policy handbook to prove it.

20:46.480 --> 20:47.480
Yes.

20:47.480 --> 20:51.200
And if it somehow gets it wrong, governed AI has coaching loops.

20:51.200 --> 20:55.880
So a staff member can review the logs, see a weak answer, and correct the system's logic

20:55.880 --> 20:57.800
so it never makes that mistake again.

20:57.800 --> 20:59.880
Which is so different from a generic tool.

20:59.880 --> 21:00.880
Completely.

21:00.880 --> 21:06.340
And cannot just call up OpenAI and demand they fix how chat GPT interprets your specific

21:06.340 --> 21:08.860
continuing education requirements.

21:08.860 --> 21:13.540
But with governed AI, your staff actually owns the logic.

21:13.540 --> 21:16.360
You know, I want to take a breath here for a second and just deliver what I think is

21:16.360 --> 21:19.020
a really necessary honest beat for you listening.

21:19.020 --> 21:20.020
Yeah, let's do that.

21:20.020 --> 21:24.100
Because when we start talking about, you know, vector databases and metadata and governance

21:24.100 --> 21:27.260
architectures, it is very easy to feel overwhelmed.

21:27.260 --> 21:28.260
Very easy.

21:28.260 --> 21:30.760
It can start to sound like a lot of vendor hype, frankly, where you feel pressured to

21:30.760 --> 21:35.320
buy this massive enterprise system just to like draft a meeting agenda.

21:35.320 --> 21:36.580
And I completely agree.

21:36.580 --> 21:40.760
We need to ground this in the everyday reality of running an organization.

21:40.760 --> 21:46.440
Because the absolute truth is for a vast majority of your daily friction points, chat GPT or

21:46.440 --> 21:48.440
popilot is perfectly enough.

21:48.440 --> 21:49.440
Right.

21:49.440 --> 21:55.040
If my task is just writing a really sloppy first draft of a marketing email, or say taking

21:55.040 --> 21:59.720
a messy transcript of an hour long internal brainstorming session and summarizing the

21:59.720 --> 22:02.960
key takeaway so I don't have to read the whole thing.

22:02.960 --> 22:05.280
Generic AI is a literal miracle.

22:05.280 --> 22:06.280
It really is.

22:06.280 --> 22:08.480
I don't need governance for a marketing brainstorm.

22:08.480 --> 22:10.720
I just need speed and creativity.

22:10.720 --> 22:11.720
Exactly.

22:11.720 --> 22:16.540
Those tools excel at low risk creative or internal productivity tasks.

22:16.540 --> 22:18.280
The pivot really only happens.

22:18.280 --> 22:22.820
I mean, the transition to governed AI only becomes necessary when the association needs

22:22.820 --> 22:27.940
to provide an official answer that the organization must legally or reputationally stand behind.

22:27.940 --> 22:28.940
That's the line.

22:29.180 --> 22:31.700
If someone asks for a summary of a podcast, use chat GPT.

22:31.700 --> 22:36.260
But if a member asks about a compliance standard that could affect their professional license,

22:36.260 --> 22:38.460
you simply cannot afford a hallucination.

22:38.460 --> 22:43.780
So the million dollar question for a leader is how to actually draw that line operationally.

22:43.780 --> 22:49.380
Like when a team brings a new AI project or a new workflow to the table, how do we decide

22:49.380 --> 22:51.140
which tool they should be using?

22:51.140 --> 22:55.300
Well the source material provides this brilliant straightforward decision test.

22:55.300 --> 23:01.460
Rather than arguing about which AI is smarter, you just evaluate the specific task using

23:01.460 --> 23:02.960
a set of criteria.

23:02.960 --> 23:04.980
We should actually apply it to a real scenario.

23:04.980 --> 23:05.980
Oh I love that.

23:05.980 --> 23:06.980
Let's do it.

23:06.980 --> 23:08.180
Let's use a complicated member question.

23:08.180 --> 23:13.700
Say a long time member emails in and asks, hey does the online ethics course I took back

23:13.700 --> 23:18.620
in 2022 still count toward the new 2026 recertification requirements?

23:18.620 --> 23:19.900
Okay perfect test case.

23:19.900 --> 23:23.540
So the first question in the decision framework is who is asking?

23:23.540 --> 23:27.460
Is it a staff member looking for general information or is it a dues paying member,

23:27.460 --> 23:28.460
a partner or the public?

23:28.460 --> 23:32.980
Well in this case it's a member asking a question directly tied to their professional credentials.

23:32.980 --> 23:35.460
So the risk level is immediately very high.

23:35.460 --> 23:36.460
Exactly.

23:36.460 --> 23:38.380
Second question, what sources are allowed?

23:38.380 --> 23:43.780
Should the AI be allowed to search the broader internet to sort of guess the answer or must

23:43.780 --> 23:48.060
it be restricted strictly to your official 2026 recertification handbook?

23:48.060 --> 23:52.600
Oh it absolutely cannot guess based on what other associations are doing.

23:52.600 --> 23:55.720
It has to be restricted to our handbook period.

23:55.720 --> 23:56.720
Right.

23:56.720 --> 24:00.160
Third question, who is allowed to see the answer?

24:00.160 --> 24:02.640
This touches on those audience tiers we talked about.

24:02.640 --> 24:03.640
Right.

24:03.640 --> 24:07.520
And the answer to this specific member needs to be based on their specific member history

24:07.520 --> 24:09.920
and not just generalized public data.

24:09.920 --> 24:10.920
Exactly.

24:10.920 --> 24:14.640
Fourth, can the answer trace back to approved sources?

24:14.640 --> 24:16.200
The show your work test again?

24:16.200 --> 24:20.960
If the AI says yes your 2022 course counts it better provide a link to the grandfather

24:20.960 --> 24:23.280
clause and out policy that actually proves it.

24:23.280 --> 24:24.280
Correct.

24:24.280 --> 24:26.440
Fifth question, who corrects the system when it's weak?

24:26.440 --> 24:30.880
If the AI misinterprets that grandfather clause and tells the member their course doesn't

24:30.880 --> 24:32.440
count, who catches that?

24:32.440 --> 24:33.440
Ah.

24:33.440 --> 24:34.440
Right.

24:34.440 --> 24:39.360
In a governed system your credentialing staff actually reviews the logs, spots the error,

24:39.360 --> 24:43.080
and adjusts the semantic weight of the rules so the AI learns the exact nuance.

24:43.080 --> 24:45.920
Which you simply cannot do in a custom GPT.

24:45.920 --> 24:51.040
Like you can tweak the prompt, sure, but you can't surgically fix the logic architecture

24:51.040 --> 24:52.040
behind it.

24:52.040 --> 24:53.040
Exactly.

24:53.040 --> 24:57.760
And finally the sixth question, which is perhaps the most strategic of all, what signal or

24:57.760 --> 25:00.880
insight comes back to the association?

25:00.880 --> 25:02.760
Oh, meaning the data.

25:02.760 --> 25:03.760
Yes.

25:03.760 --> 25:08.760
Are these complex member queries just disappearing into the black box of a public chatbot?

25:08.760 --> 25:12.680
Or are you capturing the exact wording of what your members are actually confused about?

25:12.680 --> 25:13.680
Wow.

25:13.680 --> 25:15.680
That framework is incredibly clarifying.

25:15.680 --> 25:18.080
Who is asking what sources are allowed?

25:18.080 --> 25:19.080
Who sees it?

25:19.080 --> 25:20.080
Can it be traced?

25:20.080 --> 25:21.080
Who corrects it?

25:21.080 --> 25:22.440
And what signal comes back?

25:22.440 --> 25:23.600
It's extremely practical.

25:23.600 --> 25:27.960
It really strips away all the philosophical debates about AI and makes it a pure operational

25:27.960 --> 25:29.000
risk assessment.

25:29.000 --> 25:31.160
It forces you to match the tool to the risk.

25:31.160 --> 25:36.760
It allows you to confidently say yes to chat GPT for drafting an agenda and no to chat GPT

25:36.760 --> 25:38.600
for interpreting your bylaws.

25:38.600 --> 25:42.360
So to pull all these threads together, the core thesis of our deep dive today is really

25:42.360 --> 25:43.480
this.

25:43.480 --> 25:46.400
Please use the lightest tool that fits the risk of the task.

25:46.400 --> 25:52.880
Do not over complicate your internal productivity, but reserve governed AI strictly for the official

25:52.880 --> 25:57.600
answers your organization must stand behind in front of your members and the public.

25:57.600 --> 26:02.600
It is all about deploying these tools with intentionality rather than just using whatever

26:02.600 --> 26:04.960
happens to be open in your browser tab at the moment.

26:04.960 --> 26:10.520
And to help you execute on that exact intentionality, please go to the show notes right now and

26:10.520 --> 26:16.680
grab the chat GPT, co-pilot or governed AI use case decision guide.

26:16.680 --> 26:21.120
We mentioned it earlier, and it really is the definitive next step for mapping out these

26:21.120 --> 26:23.280
boundaries in your own organization.

26:23.280 --> 26:24.280
It's going to be so helpful.

26:24.280 --> 26:29.320
It will help your team separate the low risk tasks generic AI can handle from the high

26:29.320 --> 26:33.280
stakes knowledge that your operations team still firmly needs to govern.

26:33.280 --> 26:36.520
Honestly, it will save you a tremendous amount of internal debate.

26:36.520 --> 26:42.240
And as we wrap up our analysis of these implementations today, there is one final provocative insight

26:42.240 --> 26:46.040
from the sources that I really think every leader needs to sit with.

26:46.040 --> 26:50.640
We talked earlier about co-pilot being a mirror for your internal permissions, right?

26:50.640 --> 26:51.640
Yeah.

26:51.640 --> 26:53.360
Reflecting back the messiness of your internal legacy folders.

26:53.360 --> 26:54.360
Right.

26:54.360 --> 26:57.200
The 2.0 AM fluorescent lights in the dive bar.

26:57.200 --> 26:59.240
Yes, exactly.

26:59.240 --> 27:03.160
But think about the insights that governed AI brings back to you when it is facing your

27:03.160 --> 27:04.160
members.

27:04.160 --> 27:08.680
If your governed AI is working perfectly, strictly bounded by your official sources,

27:08.680 --> 27:13.960
and it is meticulously tracking exactly what your members are asking every single day.

27:13.960 --> 27:14.960
Yeah.

27:14.960 --> 27:16.080
Are you prepared for the day?

27:16.080 --> 27:20.280
It mirrors back the gaps in your own institutional knowledge.

27:20.280 --> 27:21.640
Oh, wow.

27:21.640 --> 27:25.120
Because if the AI can only answer based on what we've officially published.

27:25.120 --> 27:26.120
Exactly.

27:26.120 --> 27:28.880
What happens when you finally have the data showing all the crucial pressing questions

27:28.880 --> 27:33.800
your members have, and you realize your association has never actually written the answers down?

27:33.800 --> 27:34.800
It's terrifying.

27:34.800 --> 27:36.320
Governed AI doesn't just answer questions.

27:36.320 --> 27:39.400
It reveals the questions you didn't even know were missing.

27:39.400 --> 27:41.720
And solving that isn't a technology problem anymore.

27:41.720 --> 27:45.640
It is a fundamental content strategy problem staring you right in the face.

27:45.640 --> 27:47.520
That is a phenomenal place to leave it.

27:47.520 --> 27:51.160
You came here to figure out which software to use, and you're leaving with a mandate

27:51.160 --> 27:52.920
to fix your content.

27:52.920 --> 27:53.920
That's the real deep dive.

27:53.920 --> 27:58.920
Get the guide in the show notes, map your use cases, and we'll catch you on the next

27:58.920 --> 27:59.440
one.

28:04.800 --> 28:06.760
Let's recap today's big question.

28:06.760 --> 28:11.240
Why not just use chat TPT or copilot?

28:11.240 --> 28:18.720
And I can't stop laughing, Thomas, because again, that line that comes up about the fluorescent

28:18.720 --> 28:27.880
lights and a dive bar at 2 a.m. as a metaphor for understanding that there is genuine, there

28:27.880 --> 28:32.680
can be genuine risk with a mess that you don't even know is there until the lights are kind

28:32.680 --> 28:33.680
of shining.

28:33.680 --> 28:39.720
That is something that we hear and something that we talk to reasonably regularly, right?

28:39.720 --> 28:43.800
Because people, there is a perception that if you've been a Microsoft shop, particularly

28:43.800 --> 28:50.000
for a long time, and you simply turn copilot on, that that is somehow safer.

28:50.000 --> 28:54.000
And yet in our experience, there is still risk associated with that.

28:54.000 --> 28:58.560
So this was really like this episode was particularly interesting to me because I learned something,

28:58.560 --> 28:59.080
right?

28:59.080 --> 29:01.320
This is actually a problem.

29:01.320 --> 29:03.360
There's a secondary market right now.

29:03.360 --> 29:06.760
I did some research kind of post doing this.

29:06.760 --> 29:13.640
There's a secondary market of people who will sell you a service for pretty for pretty penny

29:13.640 --> 29:17.480
to go in and make sure your permissions are okay for adopting copilot.

29:17.480 --> 29:18.560
Well, right.

29:18.560 --> 29:22.400
It is such a big problem that there's now this kind of other layer of kind of consultants

29:22.400 --> 29:24.120
that will fix the problem.

29:24.120 --> 29:24.720
Right.

29:24.720 --> 29:28.120
And I didn't know this was a problem until I started reading about it.

29:28.320 --> 29:32.600
I hope this sort of like helps people think about it because it does like it feels like

29:32.600 --> 29:33.880
Microsoft's a giant name.

29:33.880 --> 29:40.440
They've been around since, you know, what the late 70s, early 80s, they've got it locked down.

29:40.440 --> 29:45.400
But what it doesn't talk about is actually you're inheriting a lot of assumptions, right?

29:45.400 --> 29:51.400
Not necessarily from Microsoft, but maybe even before you kind of joined the association, right?

29:51.400 --> 29:54.080
You've probably been on the Microsoft framework for a really, really long time.

29:54.080 --> 29:54.640
And that's good.

29:54.640 --> 29:58.480
I'm not trying to say anything bad about Microsoft and copilot does have its place.

29:58.480 --> 30:02.840
But if you're not aware of these problems, you don't know what might come out of it.

30:02.840 --> 30:04.480
And that was really eye opening for me.

30:04.480 --> 30:09.640
It really is like you flip on the fluorescent lights and the cockroach is scurrying away.

30:09.640 --> 30:12.080
It's like, oh my God, I was sitting in this this whole time, right?

30:12.080 --> 30:13.840
Like you just don't know what's there.

30:13.840 --> 30:14.320
That's right.

30:14.320 --> 30:19.160
And we've talked about this in terms of like, does Microsoft really care about the business models

30:19.160 --> 30:21.000
and the governance rules of an association?

30:21.000 --> 30:26.000
No, they've turned an application on inside their own applications and just said, hey,

30:26.000 --> 30:26.800
this is pretty cool.

30:26.800 --> 30:27.680
Go ahead and use it.

30:27.680 --> 30:28.200
And you're right.

30:28.200 --> 30:34.560
That secondary market now of consultants offering 30 to 50 thousand dollar engagements just to

30:34.560 --> 30:39.120
come in, teach your copilot to use your content.

30:40.120 --> 30:43.600
All of a sudden, you know, and again, this is the platform player, right?

30:43.600 --> 30:45.560
It's platform and services kind of costs.

30:46.000 --> 30:51.640
That kind of time and money can be spent, you know, essentially working with a governed AI

30:51.640 --> 30:56.040
system that's designed specifically for associations, pardon the plug.

30:56.280 --> 31:02.960
And I think the risk there is really is a really important one as well that's come up through all

31:02.960 --> 31:11.920
of this, that the governance around an AI is not just about governing the responses that it gives

31:11.920 --> 31:12.680
to the queries.

31:12.880 --> 31:17.800
You know, it's it's understanding that that there is a mandate, that there is a purpose that the

31:17.800 --> 31:19.240
AI needs to abide by.

31:19.240 --> 31:24.560
It's putting that kind of there's the intelligence layer, but it is putting that insights layer

31:24.560 --> 31:31.280
around it so that it is driving the purpose of the organization forward as well, but doing it in a

31:31.280 --> 31:32.720
really sort of critical way.

31:33.800 --> 31:39.800
It's it's it's a shame to constantly be talking about this almost from a risk profile, because

31:39.800 --> 31:44.920
again, when we opened the podcast, it was a little bit about it's not just about should we build

31:44.920 --> 31:54.520
this ourselves and should we use a chat TBT or a co pilot or a Claude agent to do that?

31:54.920 --> 31:59.320
It's more about form follows function.

31:59.680 --> 32:01.480
You know, what's the for that?

32:01.480 --> 32:03.600
What's the purpose of this tool?

32:04.000 --> 32:07.960
Does it have an internal productivity, you know, primary angle to it?

32:07.960 --> 32:11.640
Does it have across a whole ecosystem?

32:11.640 --> 32:17.680
And what happens when you have multiple ecosystems of content or knowledge interacting that's never

32:17.680 --> 32:22.520
interacted before and is being used by a co pilot that's not trained to do the job of your

32:22.520 --> 32:23.200
association?

32:23.560 --> 32:30.280
What happens if so again, a lot of experimentation, probably they can coexist.

32:31.880 --> 32:34.560
But there's also some risks that we still need to be mindful of.

32:35.120 --> 32:37.240
Yeah, and I think it's just awareness of it, right?

32:37.400 --> 32:38.600
These are good tools, right?

32:38.600 --> 32:44.320
We use co pilot, we use Claude code, we use, you know, Chad GPT, and we use, you know, our own

32:44.320 --> 32:45.480
Betty pollution, right?

32:45.680 --> 32:46.920
And we use them for different reasons.

32:47.000 --> 32:53.680
And it's more understanding what has what risks, what's what function does the tool serve for?

32:53.720 --> 32:53.920
Right?

32:53.920 --> 32:57.280
Like you're not using a solve where you should have been using a hammer, right?

32:57.280 --> 32:58.000
It'll never work.

32:58.400 --> 32:58.600
Right.

32:58.600 --> 33:01.760
So it's understanding sort of those applications and how you can manage them over.

33:01.760 --> 33:05.000
And I think that's why the companion asset today is really important.

33:05.000 --> 33:10.360
If you can kind of understand where each plays a role, and you can actually understand, kind of

33:10.360 --> 33:14.920
take a step back, understand your ecosystem of different AI tools and what they're being used for

33:14.920 --> 33:16.760
and how to coordinate amongst them.

33:16.760 --> 33:20.680
I think that's a that's a really important part of this process is just a deeper understanding.

33:20.760 --> 33:26.200
So not again, not co pilots bad co pilot has risks, but it's also got benefits, like the eyes with Chad

33:26.200 --> 33:26.720
GPT.

33:27.240 --> 33:28.920
And dare I say even with Betty, right?

33:28.920 --> 33:33.200
There's there's a purpose for Betty that won't apply in the case that a co pilot would apply.

33:33.560 --> 33:38.880
There was a couple of great challenges to the listeners coming from the podcast today is like if your IT

33:38.880 --> 33:43.960
director all of a sudden decided to restrict co pilot tomorrow, but you were used to having this

33:43.960 --> 33:48.120
productivity tool or whatever, it's like, where's everybody all of a sudden go?

33:48.160 --> 33:53.520
You know, does it create an underground internal, you know, everyone's going and using their own GPT or

33:53.520 --> 33:56.760
whatever it is that they've built, because co pilots been restricted.

33:57.680 --> 34:02.960
And I think that one of the things I like to talk to executives about when they're thinking

34:02.960 --> 34:10.440
about this question, it's we've turned on some of these tools without a hypothesis about the impact.

34:10.840 --> 34:16.200
We haven't kind of said we believe that if we make this available to our staff to our volunteers to our

34:16.200 --> 34:21.600
members, hypothesize we believe this is what the impact is going to be.

34:23.000 --> 34:24.600
And there's plenty of data points there.

34:24.640 --> 34:30.840
But some reason with AI experiments are not being approached with the rigor that I think that they

34:30.840 --> 34:34.920
should or could. And in fact, you can get the AI to help you with the rigor of the experiment in the

34:34.920 --> 34:40.760
first place. But what happens like set yourself a 90 day plan and say we believe that 30 days, 60 days

34:40.760 --> 34:46.160
and 90 days, we think by doing this, it's going to get us there. And then just reviewing what that looks

34:46.160 --> 34:51.120
like, understanding and look at some of the qualitative information that comes back from that,

34:51.480 --> 34:55.880
that's going to help people navigate this question far more rigorously.

34:56.520 --> 35:01.680
And then I think that when we get to the point of saying, you know, which is leading into the next

35:01.680 --> 35:06.200
episode, it's like, should we build the details ourselves? But I'd like to see people experimenting with a

35:06.200 --> 35:13.000
little bit more rigor around a hypothesis and the impact that using these tools and

35:13.000 --> 35:18.360
experimenting with these tools is going to have. And I'm glad that they came up in the challenges from today.

35:20.720 --> 35:25.080
Yeah, absolutely. I think that's really important takeaway is sort of understanding the tool on what it's used

35:25.080 --> 35:29.760
for and what the risk for each one is and what the benefits are. And once you understand that, I think

35:29.760 --> 35:36.480
you'll be in a much better position to move forward with AI in a really effective way that adds value to the

35:36.480 --> 35:38.400
association and minimizes risk.

35:39.720 --> 35:45.920
So one more time before you go today, download the Use Case Decision Guide from the show notes. That's the

35:45.920 --> 35:51.760
fastest way to actually put today's episode to work. Thanks for listening to the Association

35:51.760 --> 35:57.800
Intelligence Podcast. Join us on the next episode to answer this question. Should we build this ourselves?

What this helps your team answer

How to tell the difference between individual productivity tools, ecosystem-specific tools like Copilot, and governed AI — and which job each one is actually built for.
Why turning on Copilot can surface risky, forgotten permissions in your SharePoint/OneDrive — and why that's a data-hygiene issue, not a Copilot failure.
The six-question decision framework for any AI use case: who's asking, what sources are allowed, who can see the answer, can it trace back to a source, who corrects it when it's wrong, and what signal comes back to the org.

Companion Asset

ChatGPT, Copilot or Governed AI: Use Case Decision Guide

Build vs. Buy

Should We Build This Ourselves?

An honest look at the hidden costs of building governed AI in-house — why prototypes are easy but operating a real member-facing system is a different game entirely, plus real association case studies on when to build, buy, or partner.

32:37

0:00 32:37

[Thomas Altman] (0:04 - 0:07)
Today's big question, should we build this ourselves?

[Rob Barnes] (0:08 - 1:22)
We are Thomas Altman and Rob Burns, and this is episode four of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the build versus buy scorecard. It's linked in the show notes, so grab it now if you want to follow along, or download it after and put it to work.

Thomas, I think we have been talking about or working through this question, if you like, build versus buy or build buy partner, if you like, for forever, particularly on the association software side of the table. I remember even when I worked in my associations, it was a constant question that would come up strategically about, hey, we know our business the best, we should just build it ourselves. Do we have the technical expertise to do it?

And it comes up more and more. The rapidity of, or the pace of development in the generative AI era though, is creating a whole new paradigm when it comes to this question. One of the key things that associations should be aware of, in my view, tell me, should they be really thinking about the hidden costs of doing this themselves?

[Thomas Altman] (1:23 - 3:36)
Absolutely. And the costs are surprising. It's this hidden surface area of what it takes to get governed AI up and running.

And it is one of those things that seems on the surface really simple. You can find a million tutorials online on how to create a retrieval augmented generation system, a RAG based system. And people want to.

It's cool. It's the hot new tech and everybody wants to work on it. But one of the things we've seen is that there is a stark difference between kind of the prototype, right?

Like I'm going to load in, you know, 20 documents and sort of have it work and it'll work. It'll work great, right? Versus the reality.
And the reality is many associations, if you want something beyond a simple FAQ bot, right? If you want governed AI that interacts as the interface to your knowledge base and your knowledge base, many times it's very expensive, right? If it's the case that you have thousands, hundreds of thousands, we have several clients on Betty right now with millions of documents, right?

What happens? Like, how does that scale, right? How do you scale from 12 to a million?
And what are the pitfalls you're going to find along the way when 10,000 members are kind of coming and asking questions per month, right? What kind of load does that put on your system? Where kind of the hidden areas, right?
And that's something that I think people feel overconfident a lot of times that they can build because they haven't seen sort of the pitfalls, right? And you can't see them. You have to experience them one after another.

So being aware of what those pitfalls are are very important when you decide to kind of take on a project to build yourself, when you decide to partner with, you know, someone, or if you want to just sort of buy off-the-shelf solution, you want confidence that either you know what they are and you've got a plan for them, right? Or that the person you're partnering with has dealt with them before. And I think that's a really important key takeaway for this, you know, and where does that apply?

Because if at the end of the day, all you need is an FAQ bot, yeah, that's something you can build yourself. But if you want true governed AI, there's a lot more to that story than it seems at first.

[Rob Barnes] (3:37 - 6:26)
We've had some customers over the years now, some good examples. One, you know, looked at a platform, Azure, and what was possible in Azure, and then looked at the services fees they were going to be up for in order to have someone build on that platform, that kind of the bolt-on technology, and they ended up choosing to buy. You know, we had another that's got developers on staff, and I think this is a pretty common one, particularly with sort of, I guess, larger organizations that have got an IT team that even has developers, you know, software engineers on staff now.

They don't have as much to do because the traditional, like, I mean, unless they're infrastructure engineers, but when it comes to application development and things like that, they've got more time on their hands. And I know the organizations that I've spoken to with developers on staff are like, we need to give them something to do, so we're just going to let them build this themselves. But they get 80 percent of the way through the process and can't close the gap on the other 20, because it's really, really hard to do when you're thinking about it the way we thought about it in, you know, 2022 and 2023.

So they've come back to us as well. Others have built GPTs, you know, a custom GPT or a wrapped, you know, and then hit that wall at scale like you talked about, like how do you go from 12, you know, content items to a million content items? We're still hearing that from technical buyers, you know, we still end up talking to directors of IT because they have, you know, they still are the controllers of the keys.

One of the interesting things that I've noted was, and I look back on some of these conversations, whether they're customers that we've won and partner with now or customers that have said, no, we choose not to buy Betty, we're going to go and do our own thing. It's almost always because of one individual person who wants to be the champion, who wants to be the person that did this thing, as opposed to it being a strategic objective. It's one person going, no, no, I can build this myself because I know.

And the two risks that I tend to call out is, one is why are you asking your association to become a software development company? And if you do such a good job at doing this work for that association, someone's going to come, a bigger association is going to come and poach you and say, come and do that for us. And when you leave, what happens to the technology you built for your former employer?

Is there anybody there that can do what you did? Unlikely, it's going to go with you. And then that association is left hanging.
That's a risk that we've seen actually happen in practice. When it's tied into one person. And I don't think that's talked about enough when we talk about build versus buy.

Would that be a fair assessment?

[Thomas Altman] (6:26 - 6:55)
I think that's probably the critical asset. So there are people that are overconfident. That's true.

There are people that are totally capable, right? And the capable people might not be there. Do you know what I mean?

So it's like making sure that your investment is forward-looking into that. And then the ability to either backfill or to maintain, right? A lot of times the people who are really good at building at the first time are not the same people that enjoy maintenance, right?

The people that are kind of, they like the fresh hit of excitement.

[Rob Barnes] (6:55 - 6:56)
That's a good point.

[Thomas Altman] (6:57 - 7:20)
And then maintaining it and keeping it fresh. And there's this idea of context rot within a lot of these types of models, meaning the content that gets fed into this and making sure maybe it works today. Does it work a year from now?
Right? Is it still finding the right stuff a year from now? All of those things take a lot of work, right?

And I may be very passionate about building something from scratch. I might not be so good at keeping it running.

[Rob Barnes] (7:21 - 8:10)
Durability is the word that comes up a lot for us. We tend to use it. And I think associations appreciate it.

They enjoy certainty. They enjoy understanding that whatever is created, whether it's been built, purchased, partnered on, has a level of durability that certainly extends beyond the 10 years of the current board or staff. And durability of internal ownership is something that is not recognised enough when individuals start talking about, you know what, we can just go and build that.

So the number of associations that have come back to us six, nine, 12 months later and said, yeah, we went and did that and we couldn't make it last. So they come back to us is more common than I would like it to be, to be fair.

[Thomas Altman] (8:11 - 9:19)
Which I think can be important too. So one of the things this episode really taught me was that that process itself is not invaluable, right? So kind of going through it intentionally, kind of building something, not so that you have something at the end, but so that you understand what you want better, right?

It kind of crystallises and clarifies the needs that you actually have and the gaps that you're going to find, right? So I liked the nuance of this episode because it wasn't like always buy, right? And that's not what we're saying either.

It's say know where to build, like if it's a simple thing, yeah, build yourself. Or kind of intentionally build the prototype, not so that it gets launched necessarily, but so that you understand better the ask of the eventual partner, right? You'll be much more clear on what you want, what you don't want.

Everybody benefits from that scenario too. So I do encourage people to try and to try and build on their own, but do so in a targeted way. And I think this, both the episode and the companion asset do a really good job of pointing you to like how to do that well, like kind of getting in and learning by doing and then passing off what you don't want to maintain and carry going forward.

[Rob Barnes] (9:20 - 9:42)
Before we hand this over to the AI podcast hosts, quick reminder that the following is generated using notebook LM with heavy involvement from the Betty team to ensure that the content generated is valuable. For a deeper dive on our approach to the association intelligence podcast, please go back and listen to episode zero. And now over to the AI hosts.

[AI Host 2] (9:47 - 9:55)
Um, there is this running joke in software development. Give a good engineer a weekend and a pot of coffee and they'll, you know, build you a miracle.

[AI Host 1] (9:56 - 9:57)
Yeah. Classic weekend miracle.

[AI Host 2] (9:57 - 10:07)
Right. They string together some code, they connect an API. And by Monday morning, they're showing off this working prototype that just feels like absolute magic.

[AI Host 1] (10:07 - 10:18)
It does. And it always looks flawless, right? Because it's in a controlled demo environment.
The questions are preselected. The dataset is small and well, the stakes are basically non-existent.

[AI Host 2] (10:18 - 10:26)
Exactly. But the second you try rolling that magic out to, I don't know, 50,000 association members across all these different membership tiers.

[AI Host 1] (10:26 - 10:27)
The magic crashes.

[AI Host 2] (10:27 - 10:39)
It usually crashes hard. So welcome to our deep dive today. We are answering the single most pressing question, keeping association executives and senior staff awake right now.

[AI Host 1] (10:39 - 10:41)
Should we build this AI system ourselves?

[AI Host 2] (10:41 - 10:50)
That's the one. And for this deep dive, we're drawing on implementation patterns from nearly 200 associations that have actually navigated deploying governed AI.

[AI Host 1] (10:50 - 10:53)
It's a massive sample size of real world operators.

[AI Host 2] (10:53 - 11:07)
It is. And our mission for you today is to look right past the vendor help. We really want to help you decipher the difference, like the actual structural difference between building a chatbot demo and running an operated knowledge system.

[AI Host 1] (11:07 - 11:10)
Which is frankly a massive jumping complexity.

[AI Host 2] (11:10 - 11:18)
Before we get too deep into the mechanics of all that though, I want to drop a highly practical tool for you right at the top. It's called the build versus buy scorecard.

[AI Host 1] (11:19 - 11:20)
Such a great resource.

[AI Host 2] (11:20 - 11:29)
Yeah, we'll link it in the show notes and I highly recommend you grab it. It is the perfect companion resource for the internal team conversations you are inevitably going to have after listening to this.

[AI Host 1] (11:29 - 11:39)
Because the stakes feel entirely new right now. The underlying tools, things like chat GPT, Microsoft Copilot, Azure, they're suddenly highly accessible.

[AI Host 2] (11:39 - 11:39)
So accessible.

[AI Host 1] (11:40 - 11:50)
Right. So it's entirely valid that leaders are asking this build versus buy question. Because of this accessibility, leaders are rightly worried about, you know, redundant spending.

[AI Host 2] (11:50 - 11:52)
Draining their internal staff capacity.

[AI Host 1] (11:52 - 11:58)
Exactly. Risking member trust, establishing proper governance, and just the sheer weight of long-term maintenance.
[AI Host 2] (11:59 - 12:12)
Okay, let's unpack this core dilemma then. Are we about to pay a vendor for something our internal IT team could just whip up over the weekend? I mean, is there a massive hidden cost we just aren't seeing?

[AI Host 1] (12:12 - 12:32)
There is a significant hidden operating cost. Let's look at the foundational mistake association leaders make when evaluating this. They tend to treat build versus buy purely as a standard software procurement question.
Like, they look at a vendor subscription cost and compare it to a presumed zero internal cost.

[AI Host 2] (12:32 - 12:43)
Ah, the myth of the free internal project. Right. The logic usually goes, well, we already pay the salaries of our IT staff, therefore assigning them a new AI project doesn't cost the organization any extra money.

[AI Host 1] (12:43 - 12:47)
And that assumption naturally leads to mistaking a prototype for an operated product.

[AI Host 2] (12:47 - 12:57)
It's like building a concept car versus running a municipal transit system. I mean, one looks great on a test track, but the other has to run safely every single day for thousands of commuters.

[AI Host 1] (12:57 - 13:08)
That is a perfect analogy. Because it is undeniably easy today to make a chatbot answer questions from a small handful of documents. A junior developer can do that in an afternoon.

[AI Host 2] (13:08 - 13:19)
So if the model mechanics, like the actual AI part of generating text, if that's largely solved by companies like OpenAI, what exactly is the association implementation part?

[AI Host 1] (13:19 - 13:22)
Well, they hit what we call the hidden operating surface.

[AI Host 2] (13:22 - 13:23)
The hidden operating surface.

[AI Host 1] (13:23 - 13:32)
Think of the prototype as just the tip of the iceberg. The hidden surface is everything that has to happen after the model can technically retrieve those documents.

[AI Host 2] (13:32 - 13:34)
OK, let's list them out. What's below the water?

[AI Host 1] (13:35 - 13:44)
Let's say your team builds the prototype. Now, you have to manage content ingestion. And I don't mean uploading a PDF once.

[AI Host 2] (13:44 - 13:51)
Wait, why not? Can't I just have my team upload a massive zip file of all our PDFs and call it a day? Why is dynamic ingestion actually hard?

[AI Host 1] (13:51 - 14:18)
Because association knowledge isn't static. It breathes, you know. When you feed documents into an AI system, you aren't just dropping files into a folder.

You are breaking those documents down into mathematical chunks and storing them in a specialized database. So if your board of directors updates a vital policy... The system has to know.

Exactly. Your internal system has to automatically find the chunks of the old policy, delete them, and index the new ones instantly.

[AI Host 2] (14:18 - 14:26)
Oh, wow. So if that pipeline breaks, your AI starts giving members answers based on obsolete, maybe even harmful bylaws.

[AI Host 1] (14:26 - 14:30)
Yes. Building and maintaining that dynamic sync is a heavy engineering lift.

[AI Host 2] (14:30 - 14:31)
Yay.
[AI Host 1] (14:31 - 14:34)
And it doesn't stop at ingestion. You also have to build permission boundaries.

[AI Host 2] (14:34 - 14:39)
Meaning, does the AI know not to show a board-only financial document to a student member?

[AI Host 1] (14:39 - 14:46)
Right. You need source traceability. You need a system for SME subject matter expert coaching.
You need answer review processes.

[AI Host 2] (14:46 - 14:47)
Deep analytics.

[AI Host 1] (14:47 - 14:56)
Deep analytics, hardened security posture, a seamless member UX, launch support, and, well, a framework for continuous improvement.

[AI Host 2] (14:56 - 15:06)
I want to push back on the enterprise tools for a second, though. If we're paying Microsoft for Azure, shouldn't those platforms already know how to handle indexing and security? Why is the burden falling on my internal IT guy?

[AI Host 1] (15:07 - 15:14)
Because raw cloud infrastructure gives you the building blocks, not the finished house. Azure provides incredible raw compute power, sure.

[AI Host 2] (15:15 - 15:16)
But it doesn't know our business.

[AI Host 1] (15:16 - 15:26)
Right. It doesn't know your association's membership tiers or governance rules. The burden falls on your IT team to write the custom code connecting Azure to your association management system.

[AI Host 2] (15:27 - 15:28)
Which means they own it.

[AI Host 1] (15:28 - 15:35)
They own it. Furthermore, the underlying technology is incredibly dynamic. Model and API changes require maintenance.

[AI Host 2] (15:36 - 15:37)
So you can't just build it once and walk away?

[AI Host 1] (15:38 - 15:49)
Absolutely not. If OpenAI updates their API architecture, your internal team has to drop whatever else they were working on, dive into the code, and fix the integration so the system doesn't go offline.

[AI Host 2] (15:49 - 15:59)
Let's talk about the biggest fear most executives have, which is the AI just making things up. Let's say I allocate the budget. Can't we just mandate that our internal build will be hallucination-free?

[AI Host 1] (16:00 - 16:05)
No. As an organization, you must do not rely on blanket hallucination-free promises.

[AI Host 2] (16:05 - 16:07)
Because it's mathematically impossible.

[AI Host 1] (16:07 - 16:17)
It's just not how generative AI models fundamentally work. They are probabilistic engines. Instead, the operational goal is to ground answers in approved sources.

[AI Host 2] (16:18 - 16:25)
Got it. Meaning the system is mechanically locked to only pull answers from the sandbox of data the association explicitly provides.

[AI Host 1] (16:26 - 16:43)
But even that is not enough. To maintain member trust, the system has to show source documents or passages to the user. If a member asks a complex regulatory question, the AI must provide a hyperlinked citation.
They need to click that citation and immediately see the exact white paper that generated the response.

[AI Host 2] (16:43 - 16:48)
And building that traceability, like actually connecting the text back to the database, that takes ongoing engineering effort.

[AI Host 1] (16:49 - 16:49)
Massive effort.

[AI Host 2] (16:50 - 17:03)
Well, here's where it gets really interesting, right? Building that traceability for a handful of documents sounds like a fun weekend project. But common pitfalls happen when an association scales beyond a simple dataset to tens if not hundreds of thousands of documents.

[AI Host 1] (17:03 - 17:12)
Yes. That is the exponential curve that catches teens entirely by surprise. Moving from five documents to 100,000 documents breaks simple prototypes.

[AI Host 2] (17:13 - 17:13)
I can imagine.

[AI Host 1] (17:14 - 17:26)
When you have a massive library spanning decades of publications, the operational questions get very tough. Like, what do you do when your chatbot simply can't find the right document buried in that mountain of data?

[AI Host 2] (17:26 - 17:39)
Right. If I search for a highly specific industry term and the internally built AI just shrugs, spinning up a custom prototype is basically like hiring a genius librarian who has never been inside your building.

[AI Host 1] (17:39 - 17:39)
Exactly.

[AI Host 2] (17:40 - 17:49)
They're incredibly smart, but if they don't have a perfectly maintained system tailored to your association's weird acronyms, they're going to spend hours wandering the stacks.

[AI Host 1] (17:49 - 17:56)
That is the exact mechanical problem. Can your internally built system actually learn organization-specific search techniques over time?

[AI Host 2] (17:56 - 17:58)
Because every association has its own dialect.

[AI Host 1] (17:59 - 18:03)
Right. And if the AI lacks nuance, how does it learn those crunchy details at scale?

[AI Host 2] (18:03 - 18:12)
Ah, those crunchy details. The highly specific, nuanced pieces of industry knowledge that only your SMEs truly understand. How does a machine learn that?

[AI Host 1] (18:12 - 18:23)
It requires a dedicated feedback loop. Teaching a system those crunchy details across 100,000 documents requires an SME coaching interface.

[AI Host 2] (18:23 - 18:30)
It requires deep analytics to see exactly where the AI is failing and a mechanism to correct it without writing new code.

[AI Host 1] (18:30 - 18:37)
Yes. And questions like that make the last mile exponentially difficult to manage over time for an internal team.

[AI Host 2] (18:37 - 18:52)
To ground this theory in reality, let's look at the proof points in the wild. We have actual implementation patterns from associations in the sources who hit this exact crossroads. Let's start with the National Fire Sprinkler Association, or NFSA.

[AI Host 1] (18:52 - 18:57)
NFSA is a great example. They actively considered building this themselves using Microsoft Azure.

[AI Host 2] (18:57 - 18:59)
So they had access to the raw infrastructure.

[AI Host 1] (18:59 - 19:07)
They did. But they realized the technical skills required to build and maintain the operating layer on top of Azure were just significant.

[AI Host 2] (19:07 - 19:10)
They realized they didn't want to build the coaching interfaces and the dynamic pipelines.

[AI Host 1] (19:10 - 19:19)
Yeah. So they opted to bring in Betty, which is an already operated AI platform. Doing that removed those heavy technical needs from their internal team's plate.

[AI Host 2] (19:19 - 19:29)
And it added customer service, which is huge because when you build it yourself, your IT team literally becomes the customer support desk for thousands of members.

[AI Host 1] (19:29 - 19:32)
Which speaks directly to that internal capacity worry we mentioned earlier.

[AI Host 2] (19:32 - 19:41)
But what if you actually have a robust engineering team? I mean, the sources mentioned Cornet. They are a large organization and they had developers on staff.

[AI Host 1] (19:41 - 19:54)
Cornet is a fascinating case of opportunity cost. They had the internal capability to build a custom stack from the ground up. But their leadership took a step back and asked a foundational question.

[AI Host 2] (19:54 - 20:01)
Why should we expend our valuable engineering resources to build something that is already available to be operated for us?

[AI Host 1] (20:01 - 20:07)
Right. Just because you have the tools to build a water treatment plant in your backyard doesn't mean you shouldn't just hook up to the city grid.

[AI Host 2] (20:07 - 20:15)
Exactly. They recognize their developers could be building unique value for members rather than maintaining underlying infrastructure.

[AI Host 1] (20:15 - 20:20)
And then we see the experience of CareerXRoads, which highlights a totally different technical approach.

[AI Host 2] (20:20 - 20:30)
Yeah, I've seen organizations spin up custom GPTs through OpenAI in about 10 minutes. I know CareerXRoads went down a similar path. Why doesn't that work as a permanent solution?

[AI Host 1] (20:31 - 20:42)
Because their experience proved that custom GPTs can get you partway there very quickly. You get that prototype fast. But they found that the exponentially difficult last mile we just discussed is the real problem.

[AI Host 2] (20:42 - 20:43)
It doesn't scale.

[AI Host 1] (20:43 - 20:52)
Exactly. A custom GPT struggles immensely when you try to scale it to 100,000 documents while simultaneously enforcing complex multi-tier permissions.

[AI Host 2] (20:52 - 20:58)
Trying to take a generic custom GPT and integrate it with your single sign-on is a massive leap.

[AI Host 1] (20:58 - 20:58)
It is.

[AI Host 2] (20:58 - 21:07)
So what does this all mean? It sounds like the realization across these patterns is that having developers on staff doesn't mean you should use them for this specific task.

[AI Host 1] (21:08 - 21:11)
Yes, which brings us to a highly practical decision framework.

[AI Host 2] (21:11 - 21:19)
Perfect. If you are an association executive listening right now, and your board is demanding an AI strategy, how do you actually decide when to build and when to buy?

[AI Host 1] (21:20 - 21:32)
The framework is quite clear once you understand that hidden operating surface. You should build when the association has durable internal ownership for every single layer of the stack.

[AI Host 2] (21:33 - 21:41)
Define durable. Because I've seen organizations build incredible tools, and then the lead developer takes a job somewhere else six months later, and the whole system collapses.

[AI Host 1] (21:42 - 21:55)
Oh, absolutely. Durable means you have dedicated staff. Not just a portion of someone's time, but actual headcount for AI engineering, content operations, security, UX, analytics, monitoring, and ongoing governance.

[AI Host 2] (21:55 - 22:01)
Okay. So if you are a massive organization with a dedicated AI engineering department, building makes sense.

[AI Host 1] (22:01 - 22:11)
Sure. And on the flip side, you buy when the association wants the business outcome governed, access to your knowledge, member adoption, getting staff time back without consuming the rest of your internal roadmap.

[AI Host 2] (22:12 - 22:18)
We've mentioned Betty a few times through the lens of these case studies. I want to use it as a practical example of this buy side of the framework.

[AI Host 1] (22:18 - 22:26)
Sure. Betty demonstrates what an already operated system is. Instead of giving you raw compute power, it comes out of the box offering source bound retrieval.

[AI Host 2] (22:26 - 22:29)
It handles the multi-tier access.

[AI Host 1] (22:29 - 22:39)
Right. Respecting who is allowed to see what. Crucially, it allows for that coachability by your SMEs, so they can refine those crunchy details without needing to write code.

[AI Host 2] (22:39 - 22:49)
It also provides analytics. The source material specifically highlights a feature called insights. Mechanically, why is a dedicated analytics dashboard so critical?

[AI Host 1] (22:49 - 22:58)
Because without it, you are flying blind. Insights allows the association to see exactly what members are asking the AI in real time.

[AI Host 2] (22:58 - 23:04)
Oh, so if 50 members ask about a new regulatory change and the AI can't find the answer, that's an immediate signal.

[AI Host 1] (23:04 - 23:08)
Yes. You now know exactly what article to write or what webinar to host next week.

[AI Host 2] (23:08 - 23:16)
It turns the AI from a simple search bar into a listening device for your entire content strategy. That is a tangible revenue pathway right there.

[AI Host 1] (23:16 - 23:23)
Exactly. It's a layer that activates your existing knowledge. But to maintain absolute credibility here, we need to offer an honest caveat.

[AI Host 2] (23:24 - 23:29)
Right. The sources don't say that every single association should ban their internal teams from touching AI.

[AI Host 1] (23:29 - 23:37)
Absolutely not. Some associations can, and frankly should, build narrow internal pilots as learning tools.

[AI Host 2] (23:37 - 23:40)
Like staff facing workflows. Yeah.

[AI Host 1] (23:40 - 23:49)
If your team wants to build an internal tool to help staff summarize meeting notes or quickly search an HR manual, that is a fantastic way to build AI literacy.

[AI Host 2] (23:50 - 23:57)
Because the stakes are low. If the internal tool hallucinates, your staff is inconvenienced, but the organization survives.

[AI Host 1] (23:58 - 24:07)
Exactly. The high-risk move isn't building a pilot. The high-risk move is trying to take that prototype and scale it into member-facing infrastructure without an operating model.

[AI Host 2] (24:07 - 24:15)
Because if your member-facing system hallucinates the wrong answer on a certification question or leaks board materials, you lose member trust entirely.

[AI Host 1] (24:15 - 24:20)
And remember, governance features are hard and expensive to retrofit if you don't build them in from day one.

[AI Host 2] (24:20 - 24:26)
Yeah. The stakes change entirely when it crosses the threshold to a core member benefit. We have covered a lot of ground today.

[AI Host 1] (24:26 - 24:26)
We really have.
[AI Host 2] (24:26 - 24:42)
So, synthesizing all this, the core thesis comes down to this. Your internal team can probably build a demo. The tools are there.
Your engineers are smart. But the real decision is whether your organization wants to own the entire operating model.

[AI Host 1] (24:42 - 24:54)
Yeah. If you are hearing yourself in this question, do not start by asking whether your team can build a chatbot. Start by listing everything the system has to do after the demo works, then decide whether you want to own that operating model.

[AI Host 2] (24:55 - 25:08)
And to help you make that list, I want to send you back to the build versus buy scorecard we mentioned earlier. It will help you methodically list out those hidden requirements and have a structured, honest conversation internally. You can find it linked in our show notes.

[AI Host 1] (25:08 - 25:09)
It's the best next step you can take.

[AI Host 2] (25:09 - 25:24)
Absolutely. I want to leave you with a final thought to mull over. Think about the history of technology.

Decades ago, companies built their own email servers in a back closet. Today, that would be an absurd waste of resources.

[AI Host 1] (25:24 - 25:24)
It really would.

[AI Host 2] (25:24 - 25:42)
So, if AI infrastructure inevitably becomes a standard utility like having electricity or internet access, what unique proprietary association specific value should your internal tech team be creating today instead of maintaining the plumbing? Thank you so much for joining us on this deep dive. We hope it brought some clarity.
Until next time, keep exploring.

[Rob Barnes] (25:45 - 27:35)
Recapping today's big question, should we build this ourselves? And Thomas, this just reminded me of a couple of real experiences that I've had with a couple of reasonably large and international associations in the last few years, one of which we have spoken to every year for the last three years about what they're doing, what they would like to have achieved, and they still don't have anything to show for it. There is nothing in production, despite their best efforts to try and test, despite my best efforts to have them buy Betty and just do some work, they still don't have anything in production after three years, and because technology has changed rapidly.

Another organisation, a society, I learned, has spent over a seven-figure sum of money trying to get a purpose-built tool for that society working at scale, and again, still does not have anything to show for it, and it kind of pains, honestly, it pains me to know that these things are happening, because, yeah, the demos were easy to show, it's easy to show some capability or some, but are we able, you know, it's a bit like the last episode, you know, they built a tree house, but we've got to move into this thing, when are we ever going to be able to move in, and they haven't been able to, and this episode just reminded me of some of those pitfalls, and again, without putting some real guardrails around what does a build even look like and feel like, beyond the excitement of building something new, that's been lost, and it's very costly, and this episode reminded us of that, right?

[Thomas Altman] (27:35 - 29:02)
Yeah, and I think it's not, again, it's not that you shouldn't build, right, and I think there are a lot of cases where you should, right, it makes a lot of sense, either kind of small, quick wins, right, low-hanging fruit, easy to get out the door and get value out of that, I think, obviously, you should build there, but the other thing that I think this organization could have done is sort of build to learn, right, not build to deploy, but build to learn, like, let's see how far we can go, and it's a very targeted, you know, small budget attempt, right, so spending the seven figures was not, I think, where they should have gone, right, they should have spent four figures, three figures, trying to build something small, learning how it works, and then going back and saying, okay, now does it make sense to partner here, we hit a lot of walls, and we would spend seven figures to overcome those walls, let's go talk to some of these other people in our network and say, like, have you overcome, we found this specific problem, have you faced it before, they say no, red flag, they say yes, maybe keep talking, right, a little bit more, so it gives you a more informed position, right, if you do build first, so we're not anti-build, I just want to be clear about that, there's times you can build and deploy, there's times it makes sense, but the workbook, I think, in this episode, I think, will hopefully have helped you clarify how to build in a way that makes positive ROI far, far, far more likely than it becoming a sunk cost that you just keep sinking more and more and more into.

[Rob Barnes] (29:02 - 30:28)
You've made a good point that I'm going to learn from with this, even with our own conversations, is we do lean into the build versus buy conversation much too heavily instead of build and then buy, oh and partner, because you're right, it's not don't build, I mean, ultimately, that's what you and Dre did, you know, originally with Betty all those years ago, it's more like understanding what the guardrails should and shouldn't be, particularly when it comes to investment, I think the difference there would be, we're talking about associations, we were building a software development company to bring Betty to market, that's our purpose, so it makes sense, for an association, it doesn't always make sense, but you're right, there is great ways to do narrow, internal pilots, low stakes, keep it staff facing, so the control group is there, rather than having this kind of absolutist approach about don't build, I need to remember that when we're having some of these conversations. The scorecard that we've shared with the show notes here, Thomas, is there any other factors, any questions that people should be asking themselves when they are thinking about what first initial steps to take when this is something that's been offered, it might even be by the board, why don't we just do this ourselves?
[Thomas Altman] (30:29 - 32:03)
Yeah, I think look for those weekend projects, right, it talks about sort of a miracle happening over the weekend at the beginning of this episode, and you don't have to do them over the weekend, right, the weekend being a short, like, oh, it's a passion project, I would say it would be useful if, and we did this internally, right, like a one-week sprint, right, where everybody on the team, or maybe you get a cohort of people, and you say, yeah, keep the lights on, you know, for a week, but your job for the next week is to just build something cool, and not that you're going to deploy that, or that it's going to become some critical part of your infrastructure, but that on Friday, if you started on Monday, on Friday, maybe you throw the whole projects away, maybe you don't, but maybe you do, and the guaranteed outcome of this is that you're going to have a discussion around what is actually possible based off of real experience, because AI is such a new, it's still a new technology, I know it feels like it's been forever, but it's still fairly new, like there's no clear best practices, and the reason for that is it's also fast-moving technology, there's new stuff every day, so being able to kind of get people's hands dirty trying to build something, even if they're not technical, like have them work with codex, or quad code, or whatever, to sort of build something, and to see how far they get, and then talk about it, right, and then once you talk about it, you'll be making more informed decision around like, okay, this is a good pilot project that we can actually do, or this is a much bigger hornet's nest that we don't want to shake, and we're going to find people who've encountered and solved these problems previously, and it'll be an informed decision at that point that'll cost three digits instead of seven.

[Rob Barnes] (32:04 - 32:29)
Thanks Thomas. So everyone listening, one more time before you go, download the Build Versus Buy scorecard, I'm not going to change the title of it just yet, but download that from the show notes, that is the fastest way to actually put today's episode to work. Thank you for listening to the Association Intelligence Podcast, join us on the next episode where we dig into this big question, we don't have a tech team, can we still do this?

What this helps your team answer

Why a working prototype (a few documents, a small demo) and an operated, member-facing knowledge system are two completely different things — and where the "hidden operating surface" (content ingestion, permissions, traceability, SME coaching, analytics, security) actually lives.
Real association examples — NFSA, Cornet, and CareerXRoads — showing what happened when they weighed Azure builds, in-house developers, and custom GPTs against buying an already-operated platform.
A practical framework for deciding when to build (you have durable, dedicated headcount for every layer of the stack) versus buy (you want the outcome without consuming your internal roadmap) — plus how to run a low-stakes "build to learn" pilot instead of a costly seven-figure attempt.

Companion Asset

Build vs. Buy Scorecard

Getting Started

We Don't Have a Tech Team — Can We Still Do This?

Why AI success for associations is far less about engineering talent and more about organizational clarity — and how to right-size a single governable use case instead of trying to build an org-wide governance framework before you start.

34:01

0:00 34:01

[Thomas Altman] (0:07 - 0:11)
Today's big question, we don't have a tech team. Can we even still do this?

[Rob Barnes] (0:12 - 1:29)
We are Thomas Altman and Rob Barnes and this is episode 5 of the Association Intelligence Podcast. Before we get into it today, this episode's companion asset is the Implementation Readiness Worksheet. It's linked now in the show notes.

Grab it now if you want to follow along or download it after and put it straight to work. This episode is really talking about organizational clarity, not so much a technical discussion. I think that's what I took away from this is I hear often Thomas now in conversations about well, we have to do this big AI policy, AI governance framework, AI assessment first before we can think about it at an application level.

And like all strategic planning and all strategic work in an organization, it's a little bit like herding the cats. It's costly, it can be time consuming, and it doesn't always arrive at the problem that's trying to be solved when it comes to an AI conversation anyway. Are we understating how much governance work is actually technical work in disguise?

[Thomas Altman] (1:29 - 3:09)
I think that's the interesting thing here. And I think it might even be the other way around. So one thing we've seen is we've had plenty of non-technical or groups without an IT staff at all kind of adopt successfully Betty, right?

And I think there's this kind of concern around we don't have a technical team because AI feels like it should be technical, right? And the fact is governed AI a lot of times isn't, right? Like the technology is there, right?

Or the hurdle isn't the technology, the hurdle is the organizational understanding of what is governed, what is not governed, what are the cases that we want to apply AI, right? It's more of an organizational hurdle than a technological one. So we've had teams with strong technical backgrounds struggle, right?

And sometimes because they're strong technical teams, right, they get too in the weeds with it, and they lose sight of kind of what's the value of the project. And we've had teams with strong technical backgrounds also succeed, right? But we've had teams with no technical team be wildly successful.

And the thing that I liked about this episode is it teases out sort of how that happens, right? Where does that actually live? Why is this actually not really a technical problem?
And why is it that strong organizational organizations, organizational organizations, why are they successful regardless of whether or not they have a strong IT team, right? And I think this gets into that a little bit well in a way that I think is super useful and hopefully applicable.

[Rob Barnes] (3:09 - 4:56)
Yeah, for sure. There is some very strong customer success stories where, you know, I know two of our smallest customers are like two full-time staff, and neither of them are technical. But the level of ownership that they've taken over the application and its purpose and its alignment to their strategic intent is where a lot of that success kind of comes from.

Like, I think some people may come into this episode and go, oh, it's another AI readiness checklist. And that's, and I think what's great about this is that it is a checklist, take it or leave it. Look at the eight steps, you know, from a practical perspective.

But ask yourself the question, like, which of these do we skip when we talk about these sorts of things? What breaks if we do skip them? And so use it as an exception, almost as an exception list first rather than a hard and fast rule.

And look at it from, look at it from that perspective because we know how to govern. We know that there are governance frameworks in place for just about every other program of work that an association does. Advocacy is tightly controlled.

You know, event delivery is tightly controlled. Even our marketing management is tightly controlled against brand and persona and those sorts of things. What I really enjoyed about this, again, is taking away the shrink the scope advice, like bring it down to a tangible, more practical, something that we know we can govern almost immediately.

However, I wonder, Thomas, is that too small of a scope in order for a board to approve ROI or to achieve an ROI in these conversations, which can also thwart the whole project?

[Thomas Altman] (4:57 - 6:10)
Right, that's the debate. So I think it is smart to not boil the ocean, right? Like it can be overwhelming to have to have kind of organizational governance over everything, right?

If you're thinking in terms of the whole organization, that can be so much work that you'll actually never do anything at all. And if you never do anything at all, you're going to be left behind. But at the same point, if you start too small, is that not exciting enough to get buy-in from the group?

And I think kind of navigating that tension is what this episode is really all about. How do you find the right size use case that is governable, right? So you don't have to boil the ocean.
You don't have to govern your whole organization. You can govern a specific use case. How do you identify what that is so that it's not too small, right?

You don't lose enthusiasm or momentum because you started too small and too timidly, but it's not so big that you never get started at all. So kind of right-sizing that and listening to kind of how our favorite AI hosts kind of navigate that on this, I think is going to be really hallucinating. And then once you apply that through the companion asset for this episode, I think you'll come away with a really clear, like, this is where we can start.

[Rob Barnes] (6:11 - 6:33)
And before we hand this over to the AI podcast's hosts, quick reminder that the following is generated using Notebook LM. Heavy involvement from the Betty team to ensure that the content generated is valuable to you today. For a deeper dive on our approach to the Association Intelligence Podcast, please go back and listen to episode zero.

And now, over to the AI hosts.

[AI Host 1] (6:36 - 6:40)
Imagine buying, well, like a brand new self-driving car, right?

[AI Host 2] (6:40 - 6:40)
Yeah.

[AI Host 1] (6:40 - 6:45)
You get strapped into the passenger seat, and then you suddenly realize you have absolutely no idea where you're going.

[AI Host 2] (6:45 - 6:46)
Oh, that's terrifying.

[AI Host 1] (6:47 - 6:51)
Right. Or, or worse, you realize you don't even know how to tell the car where to go.

[AI Host 2] (6:51 - 6:53)
Yeah, that sudden rush of panic.

[AI Host 1] (6:53 - 7:00)
Exactly. And that is exactly how most association leaders feel when they, you know, when they look at their organizational chart.

[AI Host 2] (7:00 - 7:02)
They see zero software engineers.

[AI Host 1] (7:02 - 7:11)
Zero. And they realize they're expected to implement artificial intelligence anyway. It's, it's basically the number one concern we hear from you guys.

[AI Host 2] (7:12 - 7:13)
It really is.

[AI Host 1] (7:13 - 7:19)
It always starts with this deep sigh, and then the exact same question. We don't have a tech team. Can we still do this?

[AI Host 2] (7:19 - 7:24)
Which is such a common refrain. And frankly, I mean, it's a completely rational response.
[AI Host 1] (7:24 - 7:24)
Totally.

[AI Host 2] (7:24 - 7:37)
If you're looking at this fundamentally new, really complex technology, and trying to map it onto an organization that specializes in member services, not machine learning. Right. The math just doesn't seem to work out in your head.

[AI Host 1] (7:38 - 7:47)
Yeah, let's validate this right out of the gate for you, the listener. Because if you are the person lying awake at night, worrying about the maintenance, or, you know, the vendor management.

[AI Host 2] (7:47 - 7:48)
And system permissions.
[AI Host 1] (7:48 - 7:55)
Exactly, the permissions, or just the sheer support load this might dump on your already stretched staff, you are not being a Luddite.

[AI Host 2] (7:55 - 7:58)
No, not at all. You're being a responsible leader.

[AI Host 1] (7:58 - 8:06)
You are. Those are the exact operational realities that break bad technology implementations.

[AI Host 2] (8:06 - 8:07)
Yeah, they really do.

[AI Host 1] (8:08 - 8:19)
So today's Deep Dive is based on a stack of implementation case studies. And specifically, the Implementation Readiness Framework. Which is designed for association leaders facing this exact dilemma.

[AI Host 2] (8:20 - 8:25)
And our goal today is to basically fundamentally reframe the whole concept of readiness.

[AI Host 1] (8:26 - 8:28)
Yes, the mission of this Deep Dive.

[AI Host 2] (8:28 - 8:35)
Right, we need to shift the perspective here because the barrier to entry, it isn't about code anymore. It's about organizational clarity.

[AI Host 1] (8:35 - 8:36)
Okay, say more about that.

[AI Host 2] (8:36 - 8:44)
Well, we just need to stop asking, do we have the engineers to build this? And start asking, can we own the decisions that only our association can own?

[AI Host 1] (8:44 - 8:47)
Which brings us right back to that self-driving car analogy.

[AI Host 2] (8:47 - 8:47)
Yeah.
[AI Host 1] (8:48 - 9:06)
Because, I mean, you do not need a team of mechanical engineers living in your garage to safely drive a vehicle. Thank goodness. Right, you don't need to know how to mill an engine block, or program a fuel injection system.
But you absolutely must know your destination.

[AI Host 2] (9:06 - 9:08)
Yes, you have to know the rules of the road.
[AI Host 1] (9:08 - 9:12)
And critically, you need to know exactly who you are allowing to take the wheel.

[AI Host 2] (9:12 - 9:14)
That is the key right there.

[AI Host 1] (9:14 - 9:27)
So to understand why a lack of an internal tech team isn't actually a deal breaker, we first have to draw a very hard line. Like a massive divide between the technical mechanics of AI and the organizational governance of it.

[AI Host 2] (9:27 - 9:36)
The great divide. Let's talk about the heavy lifting first. Because in a modern AI deployment, your vendor, your implementation partner, they carry the technical burden.

[AI Host 1] (9:37 - 9:38)
What does that actually look like?
[AI Host 2] (9:38 - 9:53)
So they handle the data ingestion, which means they do the complex work of actually connecting to your databases and parsing your files. They manage the hosting, they build the underlying retrieval systems, and the security architecture that keeps everything totally safe.

[AI Host 1] (9:53 - 9:54)
So they're building the engine block.

[AI Host 2] (9:55 - 10:00) Exactly. They mill it, they tune it, and they maintain it. They also handle the launch support mechanics.

[AI Host 1] (10:01 - 10:25)
Okay, I am going to push back on this right away, though. Sure. Because if I'm playing the skeptic, I'm listening to this and I'm thinking, well, if the vendor is handling all the technical architecture, all the ingestion and the retrieval, aren't we essentially just handing our organizational brain over to a third party?

Like, how do we not lose control of our own intelligence if we aren't the ones writing the code?

[AI Host 2] (10:25 - 10:32)
Well, it's a really vital question, but it kind of misunderstands how governed AI actually works in practice.

[AI Host 1] (10:32 - 10:32)
How so?

[AI Host 2] (10:33 - 10:46)
Because control isn't in the code anymore. The control is entirely in the content. When we talk about governed AI for associations, we are absolutely not talking about, you know, unleashing an open-ended chatbot that just hallucinates answers from the public internet.

[AI Host 1] (10:46 - 10:47)
The nightmare scenario.

[AI Host 2] (10:48 - 10:57)
Right, the nightmare. We are talking about a closed source-bound system. The AI is strictly constrained by the documents you explicitly approve.

[AI Host 1] (10:57 - 11:04)
Let's pause on that mechanism for a second, actually. Because I think a lot of people assume the AI is just trying to, like, remember things it read once.

[AI Host 2] (11:04 - 11:06)
Yeah, that's a common misconception.

[AI Host 1] (11:06 - 11:10)
So how does a governed system actually stop those famous AI hallucinations?

[AI Host 2] (11:11 - 11:19)
Well, think of it less like a traditional human brain and more like a highly restricted search engine paired with a really brilliant summarizer.

[AI Host 1] (11:19 - 11:21)
Okay, a restricted search engine.

[AI Host 2] (11:21 - 11:35)
Yeah, so when a member asks a question, the underlying code acts like a search engine first. It searches only your approved databases, retrieves the specific PDF or webpage that has the answer, and then it hands that exact text to the language model.

[AI Host 1] (11:36 - 11:36) Oh, I see.

[AI Host 2] (11:36 - 11:42)
The system essentially tells the AI, read this specific document, and only summarize exactly what is in this text.

[AI Host 1] (11:43 - 11:43)
Wow.

[AI Host 2] (11:43 - 11:52)
And if the answer isn't in the document, the AI is programmed to say, it just doesn't know. That is how you kill the hallucination. The code just follows the map you draw.

[AI Host 1] (11:52 - 11:56)
So the authority sits with the association, not the algorithm.

[AI Host 2] (11:56 - 11:58)
100%. You are drawing the boundaries.

[AI Host 1] (11:58 - 12:06)
And for everyone listening who is wondering how they actually draw that map, we have a really crucial companion asset for this deep dive.

[AI Host 2] (12:06 - 12:07)
We do.

[AI Host 1] (12:07 - 12:15)
It's called the Implementation Readiness Worksheet, and it's designed specifically to help you map out these exact ownership decisions.

[AI Host 2] (12:15 - 12:17)
It's such a helpful tool.

[AI Host 1] (12:17 - 12:26)
I highly recommend pulling it up as we talk through this, honestly, because it translates this entire conversation into actionable steps for your team. You can find it right in the show notes.

[AI Host 2] (12:27 - 12:42)
And that worksheet, it really bridges the gap between the theory of governance and the daily practice of it. And understanding that division of labor where the vendor takes the technical burden and you take the governance burden, it actually leads to this really interesting paradox.

[AI Host 1] (12:42 - 12:43)
Which is what?

[AI Host 2] (12:43 - 12:49)
Lacking a massive IT team is actually a compelling argument for partnering right now rather than waiting.

[AI Host 1] (12:49 - 12:50)
Wait, that feels super counterintuitive.

[AI Host 2] (12:51 - 12:51)
I know.

[AI Host 1] (12:51 - 12:56)
Because a lack of a tech team usually means pump the brakes, not move faster.

[AI Host 2] (12:57 - 13:05)
Well, it means move faster through partnership rather than falling into the waiting trap. And we see associations do this, like, all the time.

[AI Host 1] (13:05 - 13:06)
The waiting trap.

[AI Host 2] (13:06 - 13:11)
Yeah. Small teams often think they need to wait for some massive future platform modernization project.

[AI Host 1] (13:12 - 13:12)
Oh, right.

[AI Host 2] (13:13 - 13:17)
You hear the old, well, we'll look at AI when we overhaul our AMS in three years. Yes.

[AI Host 1] (13:17 - 13:27)
The legendary association management system upgrade. The eternal excuse for delaying innovation because, you know, the new database will magically fix everything.

[AI Host 2] (13:28 - 13:32)
Precisely. Or they think they need to hire a bunch of developers before they can even pilot a tool.

[AI Host 1] (13:33 - 13:33)
Right.

[AI Host 2] (13:33 - 13:41)
But trying to build sweeping AI transformations internally, it creates an unmanaged, just overwhelming burden for a small team.

[AI Host 1] (13:41 - 13:41)
It's too much.

[AI Host 2] (13:42 - 13:48)
It is. Partnering with a specialized vendor activates your existing knowledge without that massive technical overhead.

[AI Host 1] (13:48 - 13:55)
So what does that kind of partnership actually look like in the real world? Like, how does a non-technical staff member interact with a system they didn't build?

[AI Host 2] (13:56 - 13:58)
A great practical example of this is Betty.

[AI Host 1] (13:58 - 13:59)
Okay, tell me about Betty.

[AI Host 2] (13:59 - 14:19)
So Betty is a governed implementation pattern designed specifically so that associations can own their content decisions without needing to staff a software team. Got it. Practically, it might look like a chat widget on your certification landing page or maybe a search tool in your member portal.

[AI Host 1] (14:19 - 14:19)
Sure.

[AI Host 2] (14:19 - 14:27)
But behind the scenes, systems like Betty are traceable back to your original source material. And more importantly, they are coached by your staff and your subject matter experts.

[AI Host 1] (14:28 - 14:30)
Coached is a really interesting word there. Walk me through that.

[AI Host 2] (14:30 - 14:30)
Yeah.

[AI Host 1] (14:30 - 14:36)
Because if I'm the director of membership and I don't know how to code in Python, how am I coaching an AI?

[AI Host 2] (14:36 - 14:40)
Well, coaching in this context is really just a structured feedback loop.

[AI Host 1] (14:40 - 14:40)
Okay.

[AI Host 2] (14:40 - 14:55)
It means a human on your staff periodically reviews the chat logs. They might see that, say, a member asked a really nuanced question about membership renewal. Right.

And the AI gave an answer that was technically correct, but maybe it lacked some helpful context.

[AI Host 1] (14:55 - 14:57)
So what does the staff member do?

[AI Host 2] (14:57 - 15:11)
They use a dashboard to flag that interaction, essentially leaving a note that says, hey, next time someone asks this, prioritize the information in paragraph three of the renewal policy, not paragraph one.

[AI Host 1] (15:11 - 15:11)
Oh, wow.

[AI Host 2] (15:11 - 15:15)
You are applying human judgment to guide the system.

[AI Host 1] (15:15 - 15:25)
I'm really glad you brought up the human judgment aspect, actually, because we had to be crystal clear for our listener here. We are definitely not saying this is hands-off.

[AI Host 2] (15:25 - 15:26)
Oh, absolutely not.

[AI Host 1] (15:26 - 15:29)
You don't just buy the software, flip a switch, and then go take a nap?

[AI Host 2] (15:29 - 15:34)
No, there is no such thing as no technical work required, translating to no work required at all.

[AI Host 1] (15:35 - 15:35)
Right.

[AI Host 2] (15:35 - 15:41)
The nature of the work simply shifts. You are replacing coding work with governance and judgment work.

[AI Host 1] (15:41 - 15:47)
It really feels like the difference between building a library and being the head librarian.

[AI Host 2] (15:47 - 15:48)
Yes, that's a great way to put it.

[AI Host 1] (15:48 - 15:56)
Because you don't have to pour the concrete for the foundation or wire the electricity, but you absolutely have to decide which books go in which section.

[AI Host 2] (15:56 - 15:57)
Exactly.

[AI Host 1] (15:57 - 15:58)
And how the Dewey Decimal System is applied.

[AI Host 2] (15:59 - 15:59)
Yeah.

[AI Host 1] (15:59 - 16:01)
And who is actually allowed to check out the rare manuscripts.

[AI Host 2] (16:02 - 16:12)
That is a perfect analogy. And being the head librarian, that is a significant organizational commitment. It is.

It requires deep institutional knowledge and real operational discipline.

[AI Host 1] (16:12 - 16:22)
Which brings us to the core of the readiness framework. How does an organization actually replace that coding work with governance work?

[AI Host 2] (16:22 - 16:22)
Right.

[AI Host 1] (16:22 - 16:30)
How do you define those boundaries so the system doesn't just run wild? This is where the minimum viable implementation checklist comes in.

[AI Host 2] (16:30 - 16:38)
Yeah, this is where the rubber meets the road. Because if you want to know if you are ready, without a tech team, there are eight criteria you have to be able to define.

[AI Host 1] (16:38 - 16:42)
And these map directly to the implementation readiness worksheet we mentioned earlier, right?

[AI Host 2] (16:42 - 16:43)
Yes, they do. Exactly.

[AI Host 1] (16:43 - 16:51)
Okay, so if we think about this checklist, it is really like hiring a brand new, incredibly smart, but completely naive intern.

[AI Host 2] (16:51 - 16:52)
Very naive.

[AI Host 1] (16:52 - 17:00)
Right. First day on the job, you have to define the use case. That is step one.
You would never look at a new intern and just say, hey, go fix the association.

[AI Host 2] (17:00 - 17:02)
That would be chaos. A catastrophe.

[AI Host 1] (17:03 - 17:03)
A complete disaster.

[AI Host 2] (17:04 - 17:19)
You have to be specific. You don't say, use AI to help members. You define the job.
You say, we want to use the system to help members instantly find answers about our 2025 continuing education certification requirement.

[AI Host 1] (17:19 - 17:34)
Right, which naturally leads to the second boundary, which is the source set. Yep. Because if I give that intern their first project, I'm not going to tell them to just, you know, go read every single document ever produced by the organization since 1985.

Please don't. I'm going to hand them a specific stack of folders.

[AI Host 2] (17:34 - 17:46)
Right. You draw a fence around the official documents. You have to ask which specific webpages, videos, standards, FAQs, or policies are deemed authoritative for that specific use case.

[AI Host 1] (17:46 - 17:46)
Okay.

[AI Host 2] (17:46 - 17:55)
If the job is answering questions about certification requirements, the source set is only the official certification handbook and the official FAQ page. Nothing else.

[AI Host 1] (17:56 - 18:09)
Okay, so the intern knows what to read. Next, we have to define who they are actually allowed to talk to, what they are allowed to say. So that covers the next two criteria, which are the audience and the access tiers.

It's like giving the intern the right key card to the building.

[AI Host 2] (18:09 - 18:24)
Exactly. You have to define who gets to ask the questions. Is this tool facing the general public?
Is it locked behind a member wall? Right. Is it only for internal staff or the board of directors?
And then, what exactly can each audience see?

[AI Host 1] (18:24 - 18:26)
Give me an example of that.

[AI Host 2] (18:26 - 18:32)
Well, if a member asks a question about benefits, they should get an answer drawn from the full member packet.

[AI Host 1] (18:32 - 18:32)
Sure.

[AI Host 2] (18:33 - 18:40)
But if a non-member asks the exact same question, they should probably get a totally different answer drawn from, like, public marketing materials.

[AI Host 1] (18:41 - 18:41)
Yeah.

[AI Host 2] (18:41 - 18:43)
You have to govern those permissions.

[AI Host 1] (18:43 - 18:49)
Now we get to my favorite part of the checklist and arguably the one that causes the most internal friction.

[AI Host 2] (18:49 - 18:50)
Oh, yeah.

[AI Host 1] (18:50 - 18:52)
The owner. Every intern needs a boss.

[AI Host 2] (18:52 - 18:53)
Ah.

[AI Host 1] (18:53 - 18:58)
But in an association, ownership of information can be incredibly messy.

[AI Host 2] (18:58 - 19:11)
Oh, it can be a battlefield. You need a named human being who actually approves the sources and the boundaries of the answers. When there is a dispute about whether a document is out of date, whose desk does that actually land on?

[AI Host 1] (19:11 - 19:13)
Let's play that out, actually, because I know that happens all the time.

[AI Host 2] (19:13 - 19:14)
All the time.

[AI Host 1] (19:14 - 19:19)
Say the marketing department has a glossy brochure from last year that says membership costs $500.

[AI Host 2] (19:20 - 19:20)
Hmm.

[AI Host 1] (19:21 - 19:27)
But the legal and finance team just updated the official bylaws last month, and membership is now $550.

[AI Host 2] (19:28 - 19:28)
Yep.

[AI Host 1] (19:29 - 19:34)
If the system is looking at both, the intern is confused. Who breaks that tie?

[AI Host 2] (19:34 - 19:41)
That is exactly why you need an owner. The AI cannot resolve a political dispute between your marketing and finance team.

[AI Host 1] (19:41 - 19:41)
It just can't.

[AI Host 2] (19:42 - 19:53)
No. The owner is the person with the authority to say the bylaws are the ground truth. Remove the marketing brochure from the source set.
You must have someone who owns the business logic of that specific use case.

[AI Host 1] (19:53 - 19:57)
Which flows perfectly into the next item on our checklist, the review path.

[AI Host 2] (19:57 - 19:58)
Yes.

[AI Host 1] (19:58 - 20:00)
Our intern needs a Friday performance review.

[AI Host 2] (20:00 - 20:23)
Because AI is an iterative process. Users are going to ask questions in ways you completely didn't anticipate. They'll use weird phrasing or obscure acronyms.

So who corrects those issues and how often does that happen? Is the owner looking at the chat logs every Friday afternoon to see if the system gave an incomplete answer? Are they coaching the system to do better next time?
You need a cadence for review.

[AI Host 1] (20:24 - 20:32)
Okay, then we have the launch channel. Where does our intern actually sit in the office? Do we hide them in the basement?
Or do we put them at the front desk?

[AI Host 2] (20:32 - 20:36)
You have to place the tool where users will naturally encounter it. Don't make them hunt for it.

[AI Host 1] (20:37 - 20:37)
Okay.

[AI Host 2] (20:37 - 20:45)
If it's an internal staff tool to help your membership team answer phone calls faster, maybe it lives right inside Microsoft Teams or Slack.

[AI Host 1] (20:45 - 20:45)
That makes sense.

[AI Host 2] (20:45 - 20:52)
But if it's a member-facing tool for certification, it needs to be a highly visible widget right there on the certification landing page.

[AI Host 1] (20:53 - 21:02)
And finally, the last item on the checklist. The success signal. How do we know the intern is actually doing a good job and isn't about to get fired?

[AI Host 2] (21:03 - 21:08)
Right. What metric would prove this first narrowly defined scope is actually worth expanding?

[AI Host 1] (21:09 - 21:11)
Give me some examples of what that looks like.

[AI Host 2] (21:11 - 21:23)
Well, are we looking for a 20% reduction in basic support emails to the membership team? Or maybe we're looking to cut the time it takes staff to research an answer from 15 minutes down to two minutes.

[AI Host 1] (21:24 - 21:25)
Okay, tangible metrics.

[AI Host 2] (21:25 - 21:33)
Exactly. You have to define what a win looks like before you start playing. Otherwise, you'll never know if the governance effort was actually worth it.

[AI Host 1] (21:33 - 21:43)
So if we step back and look at that whole checklist. Use case, source set, audience, access tiers, owner, review path, launch channel, and success signal.

[AI Host 2] (21:43 - 21:44)
That's the list.

[AI Host 1] (21:44 - 21:50)
That is the anatomy of governed AI. It proves why small teams need clearer boundaries, not more complexity.

[AI Host 2] (21:51 - 21:51)
Absolutely.

[AI Host 1] (21:52 - 21:57)
You are creating a controlled environment where your existing staff can safely apply their expertise.

[AI Host 2] (21:57 - 22:03)
Exactly. And when you define those boundaries, you prevent the system from becoming an unmanaged, just sprawling liability.

[AI Host 1] (22:04 - 22:07)
Okay, but we have to do the honest beat here. We have to look at the dark side of this.

[AI Host 2] (22:07 - 22:07)
Let's do it.

[AI Host 1] (22:08 - 22:20)
Having boundaries is great in theory, but what happens if an association leadership team sits down, they pull up that implementation readiness worksheet, they look at that eight-point checklist, and they just draw a complete blank?

[AI Host 2] (22:20 - 22:25)
Yeah. This is the reality check, and it's a sobering one for a lot of organizations.

[AI Host 1] (22:25 - 22:26)
I can imagine.

[AI Host 2] (22:26 - 22:48)
Because if leadership sits down and realizes that no one can actually agree on what the authoritative sources are, if they realize that no one on staff actually has the capacity or the willingness to own corrections and review the logs, and no one can define where the system should escalate issues, then they are fundamentally not ready to launch an AI system broadly.

[AI Host 1] (22:49 - 22:59)
Hold on, though. If they can't even agree on their sources, aren't they basically admitting their internal house is entirely out of order? Yes.
That feels like a much bigger operational problem than simply not having an AI tool.

[AI Host 2] (22:59 - 23:00)
You've hit the nail on the head.

[AI Host 1] (23:01 - 23:01)
Yeah.

[AI Host 2] (23:01 - 23:22)
That is an organizational crisis masquerading as a technology problem. If the humans in your building don't know what the single source of truth is, the AI certainly isn't going to magically figure it out for you. Right.
If you put messy, contradictory, unmanaged knowledge into an AI retrieval system, you are just going to get very fast, very confident wrong answers.

[AI Host 1] (23:22 - 23:33)
So if an organization hits this wall, if they look at the checklist and say, wow, our data is a mess, we can't possibly do this for the whole association, is the dream dead?

[AI Host 2] (23:33 - 23:34)
No.

[AI Host 1] (23:34 - 23:39)
Do they just walk away from AI entirely until they spend three years cleaning up their servers?

[AI Host 2] (23:39 - 23:44)
No. The solution isn't to walk away. The solution is to radically shrink the scope.

[AI Host 1] (23:45 - 23:46)
Okay. Shrink the scope.

[AI Host 2] (23:46 - 23:54)
You shrink it until it becomes operationally realistic for the capacity you actually have today. If you cannot govern the whole organization's data, do not try.

[AI Host 1] (23:54 - 23:55)
Right.

[AI Host 2] (23:55 - 23:56)
Govern just one department's data.

[AI Host 1] (23:56 - 23:59)
So cut it down from the whole library to just one section.

[AI Host 2] (23:59 - 24:02)
Exactly. And if one department is still too much, shrink it again.

[AI Host 1] (24:02 - 24:02)
Keep going.

[AI Host 2] (24:03 - 24:08)
Govern just the knowledge for one specific upcoming annual conference. Just the event FAQ.

[AI Host 1] (24:08 - 24:09)
Oh, that's smart.

[AI Host 2] (24:10 - 24:18)
Right. Can you find one owner, a handful of clear sources, and one success metric for just that one event? Usually the answer is yes.

[AI Host 1] (24:19 - 24:25)
The golden rule here seems to be that implementation must reduce your staff load, not add to it.

[AI Host 2] (24:25 - 24:33)
That is the absolute rule. If the scope is so big that governing it feels like a full-time job you don't have time for, the scope is simply too big.

[AI Host 1] (24:33 - 24:34)
Yeah.

[AI Host 2] (24:34 - 24:40)
You have to find the smallest, most contained area of friction in your association right now and start exactly there.

[AI Host 1] (24:41 - 24:46)
Shrinking the scope until it fits your capacity, I mean, that's incredibly empowering. It really takes the pressure off.

[AI Host 2] (24:46 - 24:47)
It does.

[AI Host 1] (24:47 - 24:52)
It means that any association, regardless of size or technical footprint, can find their starting line.

[AI Host 2] (24:52 - 24:59)
Which is exactly why the original question, we don't have a tech team, can we still do this? is really asking the wrong thing.

[AI Host 1] (24:59 - 25:09)
Let's synthesize this for the listener then. What is the core takeaway for the association executive who tuned in today with that creeping anxiety about their lack of software engineers?

[AI Host 2] (25:09 - 25:13)
The main takeaway is that a large internal tech team is simply not the test of your readiness.

[AI Host 1] (25:14 - 25:14)
It's just not.

[AI Host 2] (25:14 - 25:40)
The true test is organizational clarity. The real test is whether your association can define a bounded use case, point to authoritative sources, establish access tiers, assign an owner, commit to a review process, and define a metric for success. If you can do those things, you replace the need for complex custom coding with the power of clear strategic governance.

[AI Host 1] (25:40 - 25:41)
You trade the wrench for the steering wheel.

[AI Host 2] (25:42 - 25:42)
Exactly.

[AI Host 1] (25:42 - 25:51)
And that brings us right back to our companion asset for this deep dive. If you want to put this into practice, please go to the show notes and grab the implementation readiness worksheet.

[AI Host 2] (25:51 - 25:52)
Highly recommend it.

[AI Host 1] (25:52 - 26:01)
It takes everything we just talked about, the use case, the sources, the ownership, the metrics, and puts it into a clear, actionable format for you and your team to fill out.

[AI Host 2] (26:01 - 26:02)
It really makes it tangible.
[AI Host 1] (26:02 - 26:14)
Because here is the real secret. If you can sit down and fill out that worksheet for just one specific use case, you likely have exactly what you need to start a focused, responsible pilot today.

[AI Host 2] (26:14 - 26:15)
Today.

[AI Host 1] (26:15 - 26:18)
You do not need to wait for a tech team to magically appear.

[AI Host 2] (26:18 - 26:28)
It really is just about taking that first bounded step. Once you see it work in a small, governed environment, the path forward becomes so much clearer.

[AI Host 1] (26:29 - 26:46)
Absolutely. So as we wrap up, I want to leave you with a final thought to mull over. Think back to that self-driving car analogy from the beginning.

You do not need to build the engine. You just need to know how to navigate. Now think about the most valuable buried knowledge sitting inside your organization right now.

[AI Host 2] (26:46 - 26:47)
All those hidden gems.

[AI Host 1] (26:48 - 27:14)
The insights, the standards, the rich institutional history that your members desperately need but can never seem to find on your website. Right. What if the only thing keeping that incredibly valuable knowledge from your members isn't a lack of complex code, but simply a lack of clearly defined boundaries?

It completely changes how you look at the problem. It changes everything. The tech isn't the barrier anymore.

You are the barrier. So grab the keys, define your boundaries, and start driving.

[Rob Barnes] (27:17 - 28:28)
In today's episode, we dug into the big question. We don't have a tech team. Can we still do this?

And it's super interesting to hear our AI hosts really make some, I'm going to say, interesting comments. You know, the first thing, Thomas, that I picked up on is this genuine issue that often arises in an association is about if the association leadership, which we know is a mix of staff and volunteer, can't agree on what is the authoritative source, who owns the corrections, the coaching. Our AI hosts were like, that's an organizational crisis masquerading as a technology problem and challenged us to think about it in terms of our own team.

But perhaps that was a little harsh. I'm not sure that it's crisis mode, but it is a challenge. We know that ownership of this work leads to success.

When someone is the steward of this work, it leads to more success. I don't think it's the organizational crisis necessarily our hosts talked about, right?

[Thomas Altman] (28:29 - 30:02)
Yeah, I think they were a little harsh on this one. I agree. There was a few times I'm like, I don't know if I would have said it like that.

It's a good point though, right? Like the idea that if it's not a technology problem, if you or your organization can't agree on what the use case is and what the authoritative kind of assets to support that use case will be. That is a true problem.

Or the kind of lying around like it's not a tech barrier anymore, it's a you barrier. That's a little harsh. I don't know if I would have gone that far.

But I do think it is kind of illuminating a real problem, which is understanding what rules apply, how to apply those rules. The human sort of in the loop and the sort of complex human relationships need to be incorporated into the governing framework, right? And sometimes that's hard, right?

Sometimes that's too much, right? Sometimes you can debate that for months and months and months and never build anything. And that's not going to help anyone either, right?
So the idea that I like, the thing that the tension that seemed to be working or how we want to approach resolving this and the ways we've seen this be successful is in rightsizing a scope, right? You don't have to do everything all at once. You don't have to have all of your ducks in a row in place for everything.

When you can concretely and specifically govern a specific use case. So being able to find out what that is and the thing that actually makes the cats heritable, right? That's where I liked this, how this conversation kind of led us.

[Rob Barnes] (30:03 - 31:35)
There are so many examples of organizations that have real clarity over, you know, the ownership of the steps that even the readiness worksheet, the checklist that we published in concert with this, where there is an owner or a clear owner when you define that scope and perhaps apply that checklist to a particular limited scope exercise. A lot of organizations have an absolute clear, this is the leader of that. This is who is the custodian of that governance framework.

And seeing that being able to be applied to the kind of technology now that's accelerating the purpose of that work, it's very clear. And so I actually thought that the homework that the podcast gave us today about, you know, filling out the worksheet with a single use case and seeing if we can identify a very clear owner. But being okay with the ambiguity that may come from that, depending on who is working within the organization at the time, because a lot of this is going to be ambiguous.

You know, if we don't always know what the decision-making pathway is and who owns the authority around knowledge and distribution, then I think we just have to be okay with that ambiguity because there's a lot of the work in this area that is going to be ambiguous for a lot of organizations for a little bit longer, for a little bit further to come, I imagine, right?

[Thomas Altman] (31:36 - 33:16)
Yeah, I think so. And I think there's sort of, again, it wasn't the best analogy this time, but the self-driving car analogy that kind of came up at the beginning of the end, right? But the idea that, like, you don't need someone to build the car.

You need someone to drive the car, right? Or you need someone to be able to navigate the car. I think that's the important aspect here.
And I get at that point it's not a self-driving car. That's not sold on their analogy this time, right? But the general idea is, like, you don't need the technical expertise to make a car work.
I have no idea how a car works. I'm not a mechanical person at all. If something breaks, I'm out of luck.

But I can drive the car where I need to go because I've been trained in that. And I think the same idea does sort of apply here where it's not complex code that delivers the value. It's an understanding of what the use case is and how it's governed and the value that it creates for your members and being able to steer that.
So being able to have a human being sort of as an owner participating as part of the system to coach it, right? That idea of, okay, when this question comes up, this is the kind of way we want to respond. Or these are the source documents that are available to this use case, right?

Someone who owns that and someone who is actively engaging, at least on the front end of this, with the AI to coach it up so that the system learns and repeats those lessons going forward. That's how the governability is enforced. It's enforced by having an owner who's clearly understanding the use case, who can clearly explain it in natural language, not code, to the AI, and then maintaining it so that the AI enforces those rules that it learns through the interaction over time.

[Rob Barnes] (33:16 - 33:56)
That's a really important point and a lens to put over, even just the use of, the very basic use of the readiness worksheet as part of today is to kind of go into it with that lens. I think that's a layer that I'm really glad that you kind of wrapped us up today with that. So one more time before you go today, listeners, download the implementation readiness worksheet from the show notes.

It's the fastest way to put today's episode to work. Thanks for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question.

How do we do this without burning out our staff?

What this helps your team answer

Why the real barrier for most associations isn't technical skill but organizational agreement — knowing your use case, your approved sources, and who owns corrections and coaching.
How to right-size scope so a pilot is governable and concrete without being so small it loses momentum or so broad it never launches.
Why the owner of an AI use case doesn't need to code — they need to clearly explain the use case, the source material, and the rules in plain language, then coach the system as it learns.

Companion Asset

Implementation Readiness Worksheet

Team Capacity

How Do We Do This Without Burning Out Our Staff?

A candid look at why "we're at capacity" is often a vibes claim, not a data point — and a framework for spotting the hidden ongoing cost of managing AI so it reduces work instead of quietly creating a second job for your team.

30:36

0:00 30:36

[Thomas Altman] (0:07 - 0:11)

Today's big question, how do we do this without burning out our staff?

[Rob Barnes] (0:12 - 1:01)

We are Thomas Altman and Rob Barnes, and this is episode six of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the Staff Capacity Impact Worksheet. It's linked in the show notes, so grab it now if you want to follow along, or download it after and put it to work.

Thomas, one of the things that I picked up from this episode and from our AI hosts was this notion that if anyone is talking about introducing AI into their organization and someone is saying that this is not going to create new work for the team, it's not being realistic with what's coming. But there's some layers to this, right? There's some nuance here that the episode explores that we really want to make sure that listeners are picking up on, right?

[Thomas Altman] (1:02 - 3:50)

Exactly, and I think the important caveat here is new work doesn't have to be extra work, right? And I think the way that organizations navigate that central theme is what will lead to a successful implementation of AI, whatever AI you're implementing, versus one that burns out your staff, right? And the way that this episode, I think, really does a good job of unpacking is there's basically, there's more than this, but there's basically three buckets of work kind of when you're adopting an AI project.

And it's something we've seen kind of happen over again, and the people that sort of manage that systematically and thoughtfully on the front end do a good job of reducing stressful workloads associated with adopting a new system. So what do we mean by that? Where does this episode go that I think is really important to kind of key into as you're listening?

I think first, this idea of kind of your current jobs to be done, like what are your existing workflows that are underlying from what you do, right? What are the things that are currently done, right? And where does AI intervene?

Like what of that can, I think they call it the robot take on, right? What can the AI assume and replace, right? What are the menial sort of tasks that were kind of, if you've listened to the episodes up to this point, add in governance and trust that the AI can do that and manage this part of the process pretty well, right?

So you're going to identify what the work is, where the robots can step in. And then what you're going to find, and the thing that's often hidden is not only are you kind of doing a lot of the setup work, but there is an ongoing cost to managing those robots, right? If you're replacing some of your work, you need to be, you know, active around making sure that's being done.

But the thing that you get on the other side is there is also work that should never be touched by humans, I mean, by robots, that should be only for humans. Anything that requires relationship building, that requires human judgment, which requires, you know, interpretation, interpretation. Yeah, exactly.

And then that what we see that people do well, and we'll kind of get into this a little bit more, but like, moving people whose time has been freed up from the menial into the value add, right into the fulfilling into the stuff that actually has a meaningful impact on the lives of others and the mission of the organization as a whole. And they can see that they played a part in that. And it wasn't just keeping the lights on by filling out a spreadsheet.

Like, once you can kind of start to replace those, yes, you have new work, you hopefully don't have extra work, and you're able to claw off people from doing menial stuff that's sort of soul-killing into stuff that's a little bit more fulfilling.

[Rob Barnes] (3:50 - 4:52)

One of the things I've learned in the conversations I'm having with the executives, and this brings it up again, is there is a primary, it's almost, the internal use case is the one that comes to the fore more. My staff are already at capacity, how is this going to help them gain some capacity back? But we're not quite sure what the measurement of that is, because we don't, typically associations, and even when I ran the associations that I was running, back in the day now, the idea of trying to measure time and impact of your team, it's just not part of the culture of running an association typically.

It's not like running a software services business where you're accounting for every hour so you can bill it out. That typically doesn't happen, and so it's challenging to put an ROI calculator together without some high level of hypothesis around these things. When we talk about it from a Betty perspective now, I also see it's the value creation around the tool that is designed specifically for members.

[Thomas Altman] (4:54 - 5:20)

And with that, over to our AI hosts.

[AI Host 2] (5:24 - 5:37)

So here is a pretty brutal truth about that quote-unquote magical tech tool your leadership just bought to save you time. Oh boy, here we go. Right.

It is probably going to create a second job for you.

[AI Host 1] (5:37 - 5:45)

It's the great irony of modern technology. I mean, we buy these tools to buy back our time, and we just end up spending all our time managing the tools.

[AI Host 2] (5:45 - 6:00)

Exactly. And you know, if you sit in on enough association board meetings or like those big executive leadership huddles, there's always this one question that's very specific, really heavy concern that just sucks the air right out of the room when a new initiative is proposed.

[AI Host 1] (6:00 - 6:02)

Yep. Someone always leans forward.

[AI Host 2] (6:02 - 6:09)

Always. And it's usually the person who, you know, actually manages the day-to-day operations. And they ask, how do we do this without burning out our staff?

[AI Host 1] (6:10 - 6:17)

It is the ultimate showstopper question. Because the people asking it, well, they have the scars to prove why it matters. They really do.

[AI Host 2] (6:17 - 6:29)

Honestly, that is our mission for today's deep dive. We are unpacking a whole stack of operational frameworks and implementation analyses that are entirely focused on governed AI and associations.

[AI Host 1] (6:29 - 6:39)

Right. Because the goal here is to figure out how to actually responsibly implement AI so it reduces work instead of, you know, just creating that second job for everyone.

[AI Host 2] (6:39 - 6:44)

Because let's validate what you, the listener, are probably feeling right now. You've been burned before.

[AI Host 1] (6:45 - 6:45)

Oh, absolutely.

[AI Host 2] (6:45 - 6:58)

You've seen new tech tools that were promised as the ultimate time savers. And what actually happened? They just created more administrative tasks, endless weekly alignment meetings, and this highly ambiguous ownership structure.

[AI Host 1] (6:59 - 7:05)

Suddenly, a tool that was meant to save time requires a whole committee of like three people just to keep it from breaking.

[AI Host 2] (7:05 - 7:18)

Right. So when you hear your team say they're at capacity, I mean, it's real. But here is the problem with how we typically handle that phrase.

Saying your staff is at capacity, it often functions as a vibes claim.

[AI Host 1] (7:18 - 7:19)

A vibes claim.

[AI Host 2] (7:19 - 7:20)

Yeah.

[AI Host 1] (7:20 - 7:21)

That is painfully accurate.

[AI Host 2] (7:21 - 7:30)

It's just this feeling in the room. It's like saying, you know, everyone is feeling really stressed out lately. It might be 100 percent true, but vibes don't fix workflows.

[AI Host 1] (7:30 - 7:31)

No, they don't.

[AI Host 2] (7:31 - 7:39)

You cannot operationalize a vibe. If a pipe is bursting in your house, you don't talk about the vibe of the water pressure. Right.

You find the shutoff valve.

[AI Host 1] (7:39 - 7:51)

What's fascinating here is that to actually solve this, to find that shutoff valve, we have to reframe the entire conversation scientifically. Like, staff capacity cannot remain a vibes claim.

[AI Host 2] (7:51 - 7:52)

It needs real metrics.

[AI Host 1] (7:52 - 8:05)

Exactly. It requires a quantifiable baseline. It requires a bounded workflow, clear owners, and a strict measurement plan.

So when leaders approach this, the responsible question isn't, how do we make our staff adopt this new AI?

[AI Host 2] (8:06 - 8:07)

Which is what everyone asks.

[AI Host 1] (8:07 - 8:12)

I know, but that is entirely the wrong starting point because it just assumes the AI itself is the goal.

[AI Host 2] (8:12 - 8:14)

The tool becomes the master.

[AI Host 1] (8:14 - 8:29)

Right. The right question, the truly responsible question is, which specific repetitive load are we removing and what human work are we fiercely preserving? And honestly, if you cannot articulate both halves of that sentence with crystal clarity, you're not ready to implement anything.

[AI Host 2] (8:29 - 8:40)

That's such a good point. So if the goal is preserving human energy, why do we keep getting it so wrong? Because, you know, leadership almost always falls for the classic bait and switch.

[AI Host 1] (8:40 - 8:40)

Yeah, they do.

[AI Host 2] (8:40 - 8:55)

Let's talk about how implementations usually fail the actual humans running them. The organization sells this new AI tool to the team as this ultimate staff relief. Like, good news, everyone, the robots are here to do your busy work.

[AI Host 1] (8:55 - 8:56)

We've all heard that speech.

[AI Host 2] (8:57 - 9:04)

But in reality, what they've just done is assign the staff a brand new, completely unmanaged system to babysit after hours.

[AI Host 1] (9:04 - 9:15)

And that babysitting is what the sources call the hidden implementation load. When leadership just throws a tool at a team without changing the underlying workflow, they haven't actually removed any work.

[AI Host 2] (9:15 - 9:16)

They just shifted it around.

[AI Host 1] (9:16 - 9:31)

Worse than that, they've mutated the type of work the staff is doing and usually made it significantly more frustrating. So to avoid this, our sources point to a very specific, grounded framework. We have to separate the staff workload into three distinct categories.

[AI Host 2] (9:31 - 9:40)

Okay, let's unpack this. Paint a picture for us. What does this actually look like for, say, an association professional on a random Tuesday morning?

[AI Host 1] (9:41 - 9:53)

All right, let's imagine a senior membership director. They're currently on a Zoom call trying to coach a longtime member through a really difficult, nuanced mid-career transition.

[AI Host 2] (9:53 - 9:53)

Okay.

[AI Host 1] (9:54 - 10:08)

Now that is highly valuable, deeply human work. But while they're on that call, their inbox is pinging every four minutes. Where is the link to the hotel block for the annual gala?

How many CEU credits do I need to renew my credential? I can't remember my password.

[AI Host 2] (10:08 - 10:09)

Oh, the endless pinging.

[AI Host 1] (10:10 - 10:23)

Exactly. That endless pinging is category one, the AI-suitable load. This is the highly repetitive, predictable stuff.

Routine member questions, findability problems where someone just can't locate a PDF on the website, policy lookups, and basic event logistics.

[AI Host 2] (10:23 - 10:30)

It's the endless triage. I mean, the questions where the answer is literally on page four of the member handbook, but nobody reads page four.

[AI Host 1] (10:30 - 10:31)

Nobody ever reads page four.

[AI Host 2] (10:31 - 10:36)

And it requires absolutely zero human empathy to tell someone the keynote speech starts at 9 a.m., right?

[AI Host 1] (10:36 - 10:46)

Precisely. That load is perfectly suited for governed AI. Now, category two is where the real danger lies.

This is the hidden implementation load.

[AI Host 2] (10:46 - 10:47)

The babysitting work.

[AI Host 1] (10:47 - 11:09)

Yes. This is all the internal work required to keep the new AI tool functioning. It's the time spent formatting documents so the system can read them, figuring out source ownership, meaning who actually updates the policy document when the rules change.

Oh, that's a big one. It's huge. And it's also establishing review cadences and managing the launch communication to your members.

[AI Host 2] (11:10 - 11:21)

So if category one is the robot's job and category two is the very real work of managing the robot, then category three has to be the actual human empathy, like the high-level strategic stuff, right?

[AI Host 1] (11:21 - 11:33)

That's it. Category three is the human value load. This is the work we fiercely protect.

It is interpretation. It is judgment. It's relationship building, member coaching, complex content improvement, and strategic decisions.

[AI Host 2] (11:33 - 11:34)

The stuff a machine just can't do.

[AI Host 1] (11:34 - 11:47)

Exactly. AI cannot build a relationship with that member who is struggling with a career transition. AI cannot look at political trends and decide which legislative policy your association should advocate for next year.

[AI Host 2] (11:48 - 11:51)

You know, it reminds me of a triage nurse at an emergency room.

[AI Host 1] (11:51 - 11:51)

Oh, I like that.

[AI Host 2] (11:52 - 12:12)

Right. The governed AI is the triage nurse. It handles the intake.

It points people to the right waiting room. It answers the basic questions about visiting hours. And by absorbing all of that, the actual doctors and specialists, your staff, can dedicate all their focus to the complex surgeries and patient care, which is that human value load.

[AI Host 1] (12:12 - 12:25)

That is a perfect analogy. The goal isn't to replace the doctors. It's to give them incredible leverage.

They spend less time being, you know, human search engines and more time doing the work that actually requires a human pulse.

[AI Host 2] (12:26 - 12:40)

Because we are talking about visualizing and separating this workload, this is the perfect time to mention a companion asset we have for you in the show notes. OK, let's unpack this. If you're driving or walking the dog, make a mental note to check the show notes later for the staff capacity impact worksheet.

[AI Host 1] (12:40 - 12:41)

It's such a crucial tool.

[AI Host 2] (12:41 - 13:00)

It really is the essential tool for exactly what we are talking about. It walks you through how to map out your current load, identify exactly what belongs in that AI-suitable category, and strictly define your human-only work. It's the bridge that takes us from a vibes conversation to an actual concrete operational plan.

[AI Host 1] (13:01 - 13:12)

And it forces clarity. Because without that worksheet, without doing the really hard work of mapping those three categories, you are just guessing. And guessing is how you accidentally build a second job for your team.

[AI Host 2] (13:12 - 13:22)

OK, so knowing what to map on a worksheet is great. But how do we prove this actually works in the real world? How do we know we aren't just adding to that hidden implementation load we just talked about?

[AI Host 1] (13:23 - 13:41)

We have to look at the mathematical truth of a healthy implementation. It requires a very careful proof of concept. The equation here is simple, but it is ruthless.

The AI-suitable load that's Category 1, the repetitive questions it must shrink by a significantly larger margin than the hidden implementation load adds to the staff's plate.

[AI Host 2] (13:41 - 13:53)

So, if the AI saves you 10 hours a week of answering routine emails about event parking, but it takes your team 12 hours a week to review the AI's logs and update the software, you've failed.

[AI Host 1] (13:53 - 13:59)

You've failed entirely. You have decreased your staff's capacity. You've built a liability, not an asset.

[AI Host 2] (14:00 - 14:24)

OK, let's ground this with a practical example from the source material. The analyses frequently reference Betty. And they don't talk about Betty as some, you know, sci-fi, all-knowing magic wand.

Betty is presented as a very grounded, association-specific governed AI. And I want to pause on that word, governed. For anyone who isn't living in the tech world day-to-day, what makes a governed AI different from just opening up chat GPT?

[AI Host 1] (14:25 - 14:40)

It is the most important distinction we can make. Governed means it exists inside a walled garden. A governed AI like Betty is entirely source-bound.

It is physically incapable of going out to the open internet, reading some random Reddit thread, and hallucinating a creative answer for your members.

[AI Host 2] (14:40 - 14:41)

Which is terrifying for associations.

[AI Host 1] (14:42 - 15:02)

Exactly. It only activates the existing, approved knowledge that you explicitly hand to it. It strictly reads your association's approved handbooks, your specific policies, and your official event guides.

Every single answer it gives is completely traceable back to a source document you control.

[AI Host 2] (15:02 - 15:26)

Wait, let me stop you right there. Because if Betty is just generating answers based on our PDFs, surely a human still has to proofread every single one of those answers before it goes out to a member, right? I mean, we can't just let a system speak for the association unmonitored.

But if I have to read the AI's answer, verify it against the PDF, and then hit send where's the time savings, aren't we just trading the time spent writing an email for the time spent editing a robot's draft?

[AI Host 1] (15:26 - 15:33)

That is the smartest, most common, and most valid pushback we see from senior staff. If I have to check its work, I might as well do it myself.

[AI Host 2] (15:33 - 15:35)

Right. It feels like double the work.

[AI Host 1] (15:35 - 15:47)

It does. And the answer to that pushback comes down to establishing a rigorous operating rhythm. If you don't build this rhythm into your daily operations, you absolutely will dump hidden work onto your staff, and they will hate the tool.

[AI Host 2] (15:48 - 15:54)

Okay, lay out this operating rhythm. How do we trust this system without micromanaging it to death?

[AI Host 1] (15:54 - 16:07)

It's a structured five-step rhythm. Step one, establish a baseline before you launch. You do not launch AI for your entire association at once.

You pick one single workflow with a high repeat volume.

[AI Host 2] (16:07 - 16:07)

Give me an example.

[AI Host 1] (16:07 - 16:17)

Let's say the annual certification renewal process. You baseline exactly how much time staff currently spends answering emails about that one specific process.

[AI Host 2] (16:17 - 16:18)

Okay, so you put a fence around it. Step two.

[AI Host 1] (16:19 - 16:21)

Naming source owners and escalation paths.

[AI Host 2] (16:22 - 16:22)

Yeah.

[AI Host 1] (16:22 - 16:28)

Because it is a governed AI, there will be questions it simply does not know the answer to, because the answer just isn't in your documents.

[AI Host 2] (16:28 - 16:28)

Right.

[AI Host 1] (16:29 - 16:39)

So when Betty hits a wall, who does that question escalate to? Is it the membership inbox? Is it a specific coordinator?

You must define exactly who handles the exceptions. There can be no ambiguity.

[AI Host 2] (16:40 - 16:47)

Right, because if it's ambiguous, it just sits in a general inbox, a member gets angry, and it creates massive internal anxiety for the staff.

[AI Host 1] (16:47 - 17:03)

Exactly. Now, step three is the crucial one. This directly answers your pushback about having to proofread every single email.

Step three is the review cadence. The key here is that staff must review answer patterns, not individual responses.

[AI Host 2] (17:03 - 17:13)

Okay, slow down on that. Reviewing patterns instead of individual responses. Explain the actual mechanism of that.

How does an association professional actually do that on a Friday afternoon?

[AI Host 1] (17:13 - 17:22)

Think about how you manage a brilliant new human employee. In their first week, you might ask to be CC'd on every email they send, just to make sure they have the right tone.

[AI Host 2] (17:22 - 17:22)

Sure.

[AI Host 1] (17:22 - 17:39)

But by month two, doing that would be insane. You'd never get your own work done. Instead, you look at their overall performance.

With a governed AI, you aren't reading a transcript of every single chat it had with every single member. Instead, you log into a dashboard and you look at group data and word clusters.

[AI Host 2] (17:39 - 17:42)

So the system is categorizing the interactions for you.

[AI Host 1] (17:42 - 17:58)

Yes. You log in, and the dashboard shows you, hey, out of 500 interactions this week, 70 people asked about the new late fee policy. And the AI flagged those conversations with a low confidence score.

You aren't reading 70 emails. You are reading a single trend.

[AI Host 2] (17:58 - 17:59)

That makes so much more sense.

[AI Host 1] (17:59 - 18:06)

The pattern tells you where the friction is. Is it consistently struggling with the late fee questions? If yes, that leads directly to step four.

[AI Host 2] (18:06 - 18:08)

Which is the correction and coaching loop.

[AI Host 1] (18:08 - 18:26)

Yes. You need a mechanism for staff and subject matter experts to coach the AI. But here's the secret.

Because a governed AI is source bound, coaching it rarely means reprogramming the bot. It almost always means updating your own messy documents.

[AI Host 2] (18:26 - 18:35)

Oh, wow. That is fascinating. So if the bot gives a confusing answer, it's not because the bot is stupid.

It's because the bot is accurately reading a confusing document.

[AI Host 1] (18:36 - 18:47)

Precisely. Human beings are great at navigating bad documents. Yeah.

If we read a PDF from 2019 that contradicts itself, we use human intuition to figure out what it probably means. And AI doesn't do that. It takes it literally.

[AI Host 2] (18:47 - 18:48)

It's incredibly literal.

[AI Host 1] (18:48 - 19:02)

Right. So if the AI is confused, it's holding up a mirror to your content. To coach the AI, you just have to go rewrite that paragraph in the 2019 PDF, so it actually makes sense.

You upload the clean version and the AI instantly gets smarter.

[AI Host 2] (19:03 - 19:14)

It exposes the cracks in your knowledge management. We love to blame the technology, but really, our foundation was just cracked. Which I imagine leads perfectly into the final step of the operating rhythm.

[AI Host 1] (19:14 - 19:22)

Yes. The Insights Review. This is the goldmine.

This is where you use the system to identify content gaps and surface demand signals from your members.

[AI Host 2] (19:22 - 19:23)

How does that work?

[AI Host 1] (19:23 - 19:38)

You look in the dashboard to see what members are asking that the AI couldn't answer because the source material didn't exist. That data tells your marketing and content teams exactly what they need to write next week. You stop guessing what members want, and you just look at what they are actively searching for.

[AI Host 2] (19:38 - 19:49)

Okay. So that operating rhythm baselining, establishing escalation paths, reviewing patterns instead of emails, coaching the source documents and pulling insights, that is the mechanism that keeps the math positive.

[AI Host 1] (19:49 - 19:49)

That's the engine.

[AI Host 2] (19:49 - 20:04)

That is how we ensure the AI saves more time than it costs to manage. But, you know, an operating rhythm is just a process on paper. It only survives if the team culture actually supports it.

Which brings us to the human element, change management.

[AI Host 1] (20:04 - 20:09)

The invisible rock upon which so many brilliant tech implementations break.

[AI Host 2] (20:09 - 20:19)

Let's talk about a massive pitfall the sources highlight regarding how teams design adoption. It's the trap of the enthusiastic staffer. We have all seen this play out.

[AI Host 1] (20:19 - 20:20)

In every organization.

[AI Host 2] (20:20 - 20:32)

Right. One person on the team goes to a tech conference, reads all the articles, and gets highly caffeinated and excited about AI. They bring it back to the office.

They champion it. They build the workflows. They set it all up.

And what happens to them?

[AI Host 1] (20:32 - 20:49)

They become the permanent bottleneck. And honestly, they are punished for their enthusiasm. If only one person on your staff knows how to review those patterns, or only one person knows how to update the source documents when a policy changes, you haven't built a sustainable system.

You have just built a massive dependency.

[AI Host 2] (20:49 - 20:58)

And that enthusiastic person is going to be answering Slack messages at 9pm on a Saturday because, quote unquote, the bot is acting weird.

[AI Host 1] (20:58 - 21:13)

And they will burn out rapidly. The system quickly turns into hidden labor for them. And when they eventually go on vacation or take a new job at a different association, the entire AI implementation grinds to a complete halt.

It becomes abandoned software.

[AI Host 2] (21:13 - 21:14)

And it just dies on the vine.

[AI Host 1] (21:14 - 21:32)

Completely. Adoption and source ownership must be designed across the entire team from day one. It has to be distributed.

The membership director owns the membership sources. The events coordinator owns the event sources. The education team owns the certification PDFs.

It cannot be siloed into one single AI person.

[AI Host 2] (21:33 - 21:48)

So what does this all mean? We've talked about worksheets, operating rhythms, triage nurses, and distributed ownership. Let's ask the big question.

Does a governed AI just magically fix a stressed out office? Are we saying this is the cure for the modern association's workload crisis?

[AI Host 1] (21:48 - 21:57)

I think we need to deliver a very grounded, deeply honest beat right now. We have to be incredibly careful with the word burnout. That word is thrown around way too casually in tech marketing.

[AI Host 2] (21:57 - 21:58)

Let's be real about it.

[AI Host 1] (21:58 - 22:18)

Tools like Betty, these governed AI systems, they can radically reduce specific tangible workload vectors. They're fantastic at eliminating repetitive triage. They eliminate the friction of internal search.

They give your staff incredible visibility into content gaps. But they absolutely do not solve burnout caused by deeper institutional issues.

[AI Host 2] (22:18 - 22:21)

Say more about that. What can't a governed AI fix?

[AI Host 1] (22:22 - 22:33)

AI cannot fix burnout caused by a toxic leadership culture. It cannot fix institutional uncertainty, where your board of directors keeps changing the strategic direction every six months and giving the staff whiplash.

[AI Host 2] (22:33 - 22:34)

Which happens all the time.

[AI Host 1] (22:34 - 22:50)

Exactly. It cannot fix chronic understaffing. If you have a team of three people trying to do the work of 10 people, AI is not going to suddenly make that workload sustainable.

And it certainly cannot fix poor strategic priorities. AI is simply a layer that activates your existing knowledge. It cannot fix a broken culture.

[AI Host 2] (22:51 - 23:11)

That is such a vital distinction to make. It is the difference between treating a symptom and treating the disease. Governed AI is essentially an incredibly efficient pain tiller for a very specific type of operational pain, the pain of repetition and findability.

But it is not a cure for a broken leg. Not a cure for structural dysfunction.

[AI Host 1] (23:12 - 23:25)

Exactly. And selling it to your team as a magical cure-all is a guaranteed way to lose their trust before you even launch. You have to be deeply honest with your staff about what load you are removing and what load will always remain human.

[AI Host 2] (23:25 - 23:43)

That brings us right back to our core thesis for this deep dive. Let's recap this. AI should reduce specific work, not create a second job.

The responsible implementation question isn't, how do we get our staff to use AI? It's what operating rhythm keeps the system healthy without dumping hidden work onto our people.

[AI Host 1] (23:43 - 23:46)

It is entirely about preserving human energy for human value.

[AI Host 2] (23:46 - 23:56)

Which is why I want to point the listener back to the show notes one more time. Go grab the Staff Capacity Impact Worksheet. This isn't just a fluffy PDF download.

It is a practical tool.

[AI Host 1] (23:56 - 23:57)

That's where you start.

[AI Host 2] (23:57 - 24:15)

Use this worksheet to choose just one workflow. Don't try to boil the ocean and reinvent your whole association by Friday. Pick one area.

Baseline the burden. Define exactly what should be deflected to a governed AI. And rigorously protect your staff's time for higher value work.

Start there.

[AI Host 1] (24:15 - 24:33)

If we connect this to the bigger picture, consider this one last thread to pull on. We spend all this time worrying about whether the AI is giving the perfect answer to our members. We worry about the wording and the tone.

But what if the greatest value of governed AI isn't actually answering your members' questions at all?

[AI Host 2] (24:33 - 24:36)

Wait, if it's not answering questions, what is it doing?

[AI Host 1] (24:36 - 24:55)

What if its highest purpose is acting as capacity intelligence? Think about it. What if the most valuable thing the system does isn't deflecting emails, but showing you exactly what your members are trying and failing to find?

What if the real return on investment is finally having a map that allows you to fix the root of the friction rather than just endlessly managing the symptoms?

[AI Host 2] (24:55 - 25:08)

Wow. Think about your own workflows tomorrow morning. What if the repetitive questions you are so tired of answering are exactly the map you need to fix the member experience once and for all?

We will leave you to mull that over.

[Rob Barnes] (25:14 - 26:38)

So the recap today, the big question was, how do we do this without burning out our staff? And of all the episodes so far, Thomas, I feel like the companion asset today has some real practical value. And I want to start there.

The Staff Capacity Impact Worksheet being applied to a particular workflow, a business process, we know a lot of them in associations. We have plenty of them around membership and marketing and events and advocacy that they live there. Applying this is super important because one of the things that our AI hosts talked about is if a new AI capability, a tool, saves us 10 hours a week of answering routine emails about our conference, as an example, but it takes 12 hours a week to review the logs and update the software to keep it on task, then there's a failure there because the math just doesn't math.

And I'd love to see more people doing this kind of math in advance of procuring AI or looking at applying and doing some experimentation. To get a real sense of, I think, what everybody was listening to in the episode that we were to is about this hidden work, this hidden layer of work.

[Thomas Altman] (26:40 - 29:01)

Yeah, no, exactly. I think the work, the thing that I've seen differentiate AI implementations that have added work to a staff's plate, right, like that creates a problem versus the ones that actually are more successful in reducing workload, or at least workload of menial tasks that are kind of soul draining, right, is thinking through and being aware that, yes, you may be replacing certain workflows, like doing the analysis of what workflows exist, where can AI fit in, what new work can be done that couldn't be done, and we can use AI for that. Understanding that part's very, very important.

But a lot of times what's hidden is the work of managing that in an ongoing way. Everyone kind of thinks of the work to get it up and running in the first place. That's usually not hidden.

What is hidden is how do we kind of manage this system going forward, right? And when that work is greater than the work saved, or even if it's the same, but more kind of soul draining, you're in a lose-lose situation, right? You're no longer kind of freeing up these people to do the human value work.

And I think that's the important part here. That's the thing that I want people to think about is first, what are the workflows, right? Think of the jobs to be done.

It's a framework we use internally a lot of the times that your AI system can do. Where does an AI doing that save your people time, right? And then once you understand that, think deeply about what it takes to manage that system on an ongoing basis and do the math there, right?

That's make the unhidden, make the hidden unhidden. Make the hidden transparent and available for everybody to look at so that you can actually be pretty proactive around reducing time on those other tasks, not overcompensating with the time managing system. And then repurposing those people.

I love this line kind of towards the end where it was like, it's not trying to get people to use AI. It's creating an operating system that keeps the system healthy without dumping hidden work onto our people. It's about preserving human energy for human value.

That idea, that idea of taking the human energy and using it to create human value without creating extra work. You'll know that better if you do this work ahead of time. And it's really important.

[Rob Barnes] (29:02 - 30:29)

And that culture within which the AI needs to live within becomes incredibly important as well. A governed AI, a well-applied, a well-implemented AI is still not going to address the very human issues of stuff, capacity and stuff, capability and stuff, understaffing, poor strategic alignment to priorities. It's not going to fix all of that.

You're right, that quote that you like so much about creating human value really could be applied much more broadly across the culture of an organisation than just as it relates to AI. And that's the challenge, right? I think for our listeners is to really look into the root cause of some of the, perhaps it's an inefficiency or it's the vibe of staff capacity, it's those sorts of things, look at it from a value perspective first, and then arrive back at some of the solutions that will help create that environment in that particular culture at that time will become self-evident.

And that form follows function kind of approach seems to be very applicable here. So thanks for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question.

We just bought a new AMS, why now?

What this helps your team answer

Why new work from AI doesn't have to mean extra work — and the three buckets to sort your workflows into: what AI can take on, what it takes to manage that ongoing, and what should stay entirely human (relationship-building, judgment, interpretation).
The hidden cost most teams miss: the ongoing work of managing and reviewing an AI system can outweigh the time it saves, so the math has to be done before you implement, not after.
What governed AI genuinely can't fix — toxic culture, chronic understaffing, shifting strategic priorities — and why honesty about that upfront builds trust instead of setting your team up to feel burned again.

Companion Asset

Staff Capacity Impact Worksheet

AMS vs. AI Layer

We Just Bought a New AMS — Why Would We Take On an AI Project Now?

Why "we just spent millions on a new AMS" isn't a reason to wait on AI — a framework for separating your system of record (AMS/CMS/LMS/FMS) from your system of knowledge (the AI layer), and how to decide if now's the time or if you need to stabilize first.

31:33

0:00 31:33

[Thomas Altman] (0:04 - 0:09)
Today's big question, we just bought a new AMS, why would we take on an AI project now?

[Rob Barnes] (0:10 - 1:06)
We are Thomas Altman and Rob Barnes and this is episode seven of the Association Intelligence Podcast. Before we get into today's episode, the companion asset is the AI layer versus platform decision brief. It's linked in the show notes, so grab it now if you want to follow along or download it after and put it to work.

Thomas, the episode's central distinction here is the idea of a system of record, an association management software, colloquially known to all of us as an AMS, but for this purpose, CMS, LMS, financial management system, we'll just talk about it a little bit more broadly, versus a system of knowledge, this idea of an AI layer. Is that analogy strong enough to survive, particularly the skeptical CFO who's probably just spent $200,000 or even $2 million on a new association management software?

[Thomas Altman] (1:07 - 2:56)
I don't know if any framework is strong enough to survive that CFO, but I do think it has a lot of merit to it. The idea of a system of record versus a system of knowledge is a very key distinction and I think increasingly you're going to need both and you're record is going to be important. AI is never going to be your system of record.

AI is going to be the interface governed by that system of record. What we mean by that and the way we see this kind of being super important is that your AMS, for example, it's going to have information, transactional information around what level of membership does a individual have right now. Based off of that level of membership, including not a member at all, what permissions do they have to access certain levels of content?

How should they be treated differently based off of what kind of member they are? Are they going to be getting full access to the journals or use of the standards or are they going to be sort of coached to join or take a class? All of these things are system of record that the AI should be aware of and should govern how the AI interacts with their beforehand.

So it is important to have both and I think if you've just gone through a rough implementation, it might make sense to sort of take a deep breath, stabilize that before you take on a new project, but it doesn't mean you don't do it. Or if you're kind of judging between two, there is a strong argument to be made for putting the AI system in front because those interactions will actually govern what the AMS, the FMS, the LMS should do behind the scenes. So that's kind of how I saw this happening.

[Rob Barnes] (2:57 - 4:25)
It reminded me a little of the pre-AI era where federated search kind of came in as an idea that I can search for stuff, mostly content, not so much data, but across multiple platforms and bring it together kind of in one place for a user, either an internal user or an external user. These days it's more, I guess we then morphed into these ideas that we used to have people talked about data lakes being a central repository of all the stuff coming from the different management systems. And now there is the advent, of course, of the AI-enabled CDPs and what they can do because they are ready and they were purpose-built for the AI layer to live where the user needs to live to get across all of those kind of systems of record.

And still, you're right, the readiness of an organization who has just done, in some cases, pretty troubling either AMS migration or in some cases a CMS migration for a website, there is a point where a really mindful or thoughtful perhaps AI vendor might say, you know what, talk to us again in six months time or eight months time or nine months time. And there's very good reasons for doing that, right?

[Thomas Altman] (4:26 - 5:53)
Yeah, absolutely. I think making sure, again, we're all about governed AI here, right? So and the system of records are going to be the source from which AI is governed and its ability to connect to any of your MSs, your management systems, is going to inform the way that the AI kind of works.

And the important distinction here, I think, is making sure that the governing piece is available because that's going to lead to AI success, right? Making sure that that is correct and working how you want it to, but not depending on that to sort of revolutionize your member engagement strategies, right? That is probably not going to happen.

Your AMS may have some bells and whistles and may do this or that different from the previous one, but the actual experience of interacting with your association as a member probably fundamentally won't change. That is not necessarily true of the AI. So if your goal isn't to build up the member engagement strategies and different avenues and interfaces to that governed by these transactional databases, which are any of your management systems, I think that combination is what really matters.

So yes, it does make sense if you've just gone through a nightmare AMS implementation. I've been there myself. And the last thing you want to do is throw a new tech project on top of your staff.

Yeah, take a breath, stabilize, make sure it's working the way you want it to, but don't use it as an excuse to kick the can down the road. You're going to want to do it sooner rather than later.

[Rob Barnes] (5:54 - 6:15)
So before we hand this over to our AI podcast host, quick reminder that the following is generated using Notebook LM with heavy involvement from Thomas, myself, and our Betty team to ensure that the content generated is valuable to all of our listeners. For a deeper dive on our approach, please go back and listen to episode zero. And now over to the AI hosts.

[AI Host 1] (6:19 - 6:30)
Imagine this scenario for just a second. You've just spent two years and easily two million dollars completely overhauling your association's tech stack.

[AI Host 2] (6:30 - 6:31)
Oh man, the classic nightmare.

[AI Host 1] (6:32 - 6:45)
Right. You've survived like 18 months of painful data mapping. You've lived through the late nights, eating cold pizza in some random conference room while your team tried to figure out why a thousand member records just didn't port over correctly.

[AI Host 2] (6:45 - 6:46)
Yeah, pulling your hair out.

[AI Host 1] (6:46 - 7:02)
Exactly. But you did it. The launch party is over.

The congratulatory emails have gone out. But then the very next day, a member logs onto your shiny new portal, types a highly specific urgent question into the search bar, and they get, well, absolutely zero results.

[AI Host 2] (7:02 - 7:06)
Or worse, right? They get a disjointed list of 10 totally irrelevant links from like 2018.

[AI Host 1] (7:06 - 7:17)
Yes. It is the ultimate nightmare scenario, but it's happening every single day. And that exact disconnect is what brings us to the core of what we were unpacking in today's deep dive.

[AI Host 2] (7:17 - 7:26)
Because if you are sitting in a leadership role at an association right now, evaluating your tech stack, you have absolutely heard this exact objection echoed in your board rooms.

[AI Host 1] (7:27 - 7:34)
Oh, 100 percent. And the exact objection is this, quote, We just bought a new AMS. Why now? End quote.

[AI Host 2] (7:35 - 7:45)
It is the single most common and honestly, the most understandable reaction you can get from an executive team or a board of directors. I mean, we have to validate that feeling immediately.

[AI Host 1] (7:45 - 7:46)
Yeah, they're just exhausted.

[AI Host 2] (7:46 - 8:04)
Totally exhausted. Association leaders are suffering from this profound platform spend fatigue right now. They've got endless vendor evaluation cycles, implementation burnout.

The psychological toll of a massive association management system migration is just immense.

[AI Host 1] (8:04 - 8:14)
So immense. You finally cross the finish line of this marathon. You are bruised.

Your IT team hasn't slept a full weekend in six months. And then someone brings up adding an AI knowledge layer.

[AI Host 2] (8:14 - 8:23)
Right. When you've just dragged an entire organization through reengineering every operational workflow, the absolute last thing anyone wants to hear about is another piece of technology.

[AI Host 1] (8:23 - 8:29)
The immediate reaction is going to be incredibly defensive. It's a highly legitimate concern to just raise your hands and say enough.

[AI Host 2] (8:29 - 8:34)
Exactly. So how do we reframe this? What does this all actually mean for you?

[AI Host 1] (8:34 - 8:53)
Well, the goal of this deep dive is absolutely not to ask you to buy another platform. We are not doing that today. Definitely not.

Instead, the question you should be asking yourself and your team is this. Did our new AMS actually change how members find answers across our association's entire knowledge estate?

[AI Host 2] (8:53 - 9:00)
And that is the precise pivot we need to make here. We are shifting the conversation away from traditional software categories.

[AI Host 1] (9:01 - 9:02)
Right. The typical buckets.

[AI Host 2] (9:02 - 9:14)
Exactly. And we're moving toward the concept of jobs to be done, because when you look at it through that lens, you realize the migration might have solved an operational crisis, but it left the member knowledge crisis completely untouched.

[AI Host 1] (9:15 - 9:25)
Yeah, untouched. And to help you navigate that exact shift and the sources we're looking at today, there is a really fantastic resource. It's called the AI Layer versus Platform Decision Brief.

[AI Host 2] (9:25 - 9:26)
Oh, that braid is gold.

[AI Host 1] (9:26 - 9:31)
It really is. And I want to mention this right up top, because it is essentially the roadmap for this entire conversation.

[AI Host 2] (9:32 - 9:42)
Yeah. You can use it to definitively decide what actually belongs in your AMS, what goes in your CMS, your LMS, your community platforms, and what operations belong in a governed AI layer.

[AI Host 1] (9:42 - 9:55)
Right. It maps out those jobs perfectly, so you aren't just guessing. So let's just start right there.

To understand why a new platform doesn't automatically mean better answers for your members, we have to look at the fundamentally different jobs these systems were hired to do.

[AI Host 2] (9:56 - 9:59)
Let's break down the actual job of an AMS. Yeah.

[AI Host 1] (9:59 - 10:00)
What is it actually doing?

[AI Host 2] (10:00 - 10:09)
The best way to look at an AMS is as a system of record. That is its whole identity. It is designed from the ground up to handle transactions.

[AI Host 1] (10:09 - 10:11)
Okay. So like processing payments and stuff.

[AI Host 2] (10:11 - 10:20)
Exactly. It manages the definitive state of your membership data. It's constantly asking binary questions, like, has this person paid their dues?

Yes or no?

[AI Host 1] (10:20 - 10:24)
Right. Are they up for renewal? Have they registered for the annual conference?

[AI Host 2] (10:24 - 10:33)
Yes. What specific geographic segment do they belong to? It handles all those complex operational workflows and serves as the central hub for integrations.

[AI Host 1] (10:34 - 10:40)
So it's essentially a massive, highly structured relational database optimized for operational truth.

[AI Host 2] (10:40 - 10:45)
That's a great way to put it. It is brilliant at what it does, but it is definitely not a brain.

[AI Host 1] (10:46 - 10:54)
Right. A system of record. Okay.

But the AI knowledge layer, the job we are talking about today, its job is completely different.

[AI Host 2] (10:54 - 10:55)
Entirely different.

[AI Host 1] (10:55 - 11:03)
It is a system of knowledge. It isn't trying to process a credit card for an event registration. Its job is to provide approved source answers.

[AI Host 2] (11:04 - 11:07)
Yeah. It retrieves information across the entire fragmented system.

[AI Host 1] (11:07 - 11:15)
And it understands audience access tiers, right? Meaning it knows how to differentiate what a staff member is allowed to see versus what a general member is allowed to see.

[AI Host 2] (11:15 - 11:27)
Exactly. Plus, it provides exact citations back to your source material. It allows for staff coaching of the AI.

And crucially, it gives you insights into the actual questions your members are asking in real time.

[AI Host 1] (11:27 - 11:31)
Okay. So to put it simply, the system of record manages the member's status.

[AI Host 2] (11:31 - 11:38)
Right. And the system of knowledge manages the member's curiosity, their immediate problems, their need for reliable expertise.

[AI Host 1] (11:39 - 11:43)
Okay. Let's unpack this a bit more because analogies always help me visualize this stuff.

[AI Host 2] (11:43 - 11:44)
I love a good analogy. Lay it on me.

[AI Host 1] (11:44 - 11:52)
Okay. So if my association is this massive, sprawling library, it sounds like the AMS is the front desk.

[AI Host 2] (11:52 - 11:54)
Okay. Yeah. Tracking who comes in.

[AI Host 1] (11:54 - 12:09)
Right. It's checking people in, making sure their library card is valid, collecting late fees, and keeping a ledger of who walked in the door. But it's not the librarian who actually helps you find the nuanced information hidden inside the books.

Is that right?

[AI Host 2] (12:09 - 12:20)
That is a phenomenal way to look at the broader ecosystem here. The front desk versus the librarian. Right.

And to push your library analogy a bit further, think about the other platforms in your tech stack.

[AI Host 1] (12:21 - 12:21)
Like what?

[AI Host 2] (12:21 - 12:29)
Well, your CMS, which is your website content management system, or your LMS, your learning management system, or even your community forum platforms.

[AI Host 1] (12:30 - 12:31)
Oh, okay. So what are they in this library?

[AI Host 2] (12:31 - 12:38)
Those are just the bookshelves. They are content containers and experience surfaces. They physically hold the information.

Right.

[AI Host 1] (12:39 - 12:44)
The continuing education courses, the peer-reviewed articles, the heated forum threads about new regulations.

[AI Host 2] (12:44 - 12:50)
Exactly. But they don't inherently make that information conversational. They don't make it easily extractable.

[AI Host 1] (12:50 - 13:00)
Yeah. A member still has to walk down the aisle, find the right shelf, pull the book, and then speed read through a 40-page PDF to find their one specific answer.

[AI Host 2] (13:01 - 13:09)
Exactly. The AMS checks their library card, the CMS holds the book, but neither of them reads the book for the member to answer their specific question.

[AI Host 1] (13:09 - 13:10)
It makes so much sense.

[AI Host 2] (13:10 - 13:20)
And an AI knowledge layer acting as that super librarian doesn't just point to the aisle. It can instantly read all 10,000 books in the building simultaneously.

[AI Host 1] (13:20 - 13:21)
Wow.

[AI Host 2] (13:21 - 13:28)
Right. And extract the exact paragraph you need, synthesize a clear answer, and hand it to you with a footnote pointing right back to the original text.

[AI Host 1] (13:29 - 13:36)
Okay. That librarian analogy actually explains a very expensive mistake I see leaders making every day, which brings us to this massive trap.

[AI Host 2] (13:37 - 13:38)
The migration trap.

[AI Host 1] (13:38 - 13:53)
Yes. Because if the AMS is purely a system of record, a front desk, why do so many smart, capable people think that updating it is going to magically fix their content and knowledge findability problems?

[AI Host 2] (13:53 - 13:59)
It is an incredibly common mistake. And honestly, it's usually driven by vendor promises of an all-in-one solution.

[AI Host 1] (13:59 - 14:02)
Oh yeah. The we can do it all pitch.

[AI Host 2] (14:02 - 14:16)
Right. Leaders assume that a platform migration automatically fixes the member experience or knowledge findability. They think, well, we're moving off this clunky legacy server and buying a modern cloud platform.

So of course it will have modern search.

[AI Host 1] (14:16 - 14:17)
Right. It's new, so it must be better at everything.

[AI Host 2] (14:18 - 14:34)
But AMS modernization ends exactly where true knowledge access begins. You can have the most beautifully architected brand new AMS in the world, and your members will still be utterly frustrated when they try to search for a specific continuing education requirement or some nuanced industry guideline. Exactly.

[AI Host 1] (14:34 - 14:43)
Okay. I had to play devil's advocate here for a second, because if I just wrote a massive check to an AMS vendor, I'd be absolutely furious if you told me I need another layer on top of it.

[AI Host 2] (14:43 - 14:44)
Oh, I'd be mad too.

[AI Host 1] (14:44 - 14:54)
Right. Shouldn't their search function just magically be better now? I mean, I just spent millions of dollars.

Why doesn't the AMS just build this conversational search directly into their platform?

[AI Host 2] (14:54 - 15:04)
And that is the exact question every board asks, but you have to look at the underlying architecture. The search within the AMS might be slightly better than what you had before.

[AI Host 1] (15:04 - 15:05)
Like it might load faster.

[AI Host 2] (15:06 - 15:25)
Sure. But here's the critical distinction. A shiny new AMS only searches the AMS. Oh, only itself. Right. It is searching its own internal database of structured records and whatever limited structured content it directly hosts. It does not natively bridge the fragmented stack.

[AI Host 1] (15:25 - 15:27)
Because it doesn't own the other bookshelves.

[AI Host 2] (15:27 - 15:39)
Bingo. Let's use a highly specific example. Imagine a member comes to your site and asks, can I transfer my type B nursing certification credits from Ohio to meet the new 2026 state board requirements?

[AI Host 1] (15:39 - 15:40)
Okay, tough question.

[AI Host 2] (15:40 - 15:44)
Right. Now the AMS knows who the member is. It knows they live in Ohio.

[AI Host 1] (15:44 - 15:45)
Because it's the system of record.

[AI Host 2] (15:46 - 15:57)
Exactly. But the actual answer to that question might require a piece of data from a certification course hidden inside the LMS, a policy document sitting on the CMS, and maybe a recent discussion thread in the community platform. Oh, wow.

[AI Host 1] (15:57 - 15:58)
Yeah.

[AI Host 2] (15:58 - 16:06)
The AMS search cannot reach into those other silos. Furthermore, even if it could, standard keyword search is incredibly fragile.

[AI Host 1] (16:06 - 16:09)
Wait, fragile how? Like if they misspell a word?

[AI Host 2] (16:09 - 16:23)
Yes. Or if they use a synonym. Traditional search looks for exact string matches.

So if the member searches for type B credits, but your policy document calls them category two CEUs, a standard search engine returns zero results.

[AI Host 1] (16:23 - 16:25)
Zero results. The worst screen on the internet.

[AI Host 2] (16:26 - 16:43)
Exactly. Findability across the whole estate using semantic understanding. You know, where the system actually understands the intent and the context of the question rather than just matching words.

That is a totally separate job to be done. Right. It is a job the AMS was never designed to do because it is busy processing transactions and keeping your data secure.

[AI Host 1] (16:44 - 16:51)
That is such a light bulb moment. Semantic search changes the game because it actually understands what the member means, not just what they typed.

[AI Host 2] (16:51 - 16:52)
Precisely.

[AI Host 1] (16:52 - 17:02)
You just built a beautifully organized front desk, but the bookshelves in the back are still a maze. And now we understand why the front desk clerk can't help navigate the maze.

[AI Host 2] (17:02 - 17:03)
Yeah, they're stuck behind the desk.

[AI Host 1] (17:04 - 17:10)
So if we establish that the AMS migration doesn't solve knowledge access, we need to address.

[AI Host 2] (17:10 - 17:23)
Think about the state of the organization right after a launch. The organization's system requirements are fresh in everyone's minds. You just spent a year meticulously documenting exactly what every system does.

[AI Host 1] (17:23 - 17:25)
Right. You actually know what you have for once.

[AI Host 2] (17:25 - 17:39)
System ownership is clear. You finally know exactly who manages the LMS and who manages the CMS. And most importantly, your integrations and data architecture are clearer and cleaner than they have ever been.

[AI Host 1] (17:40 - 17:41)
Because you just mapped everything.

[AI Host 2] (17:41 - 17:45)
Exactly. The bodies are buried, but you know exactly where the graves are.

[AI Host 1] (17:45 - 17:53)
Here's where it gets really interesting to me, too. The migration itself likely surfaced exactly where knowledge and member experience are still fragmented.

[AI Host 2] (17:53 - 17:54)
Oh, 100%.

[AI Host 1] (17:54 - 18:06)
Because you had to look at all your data to move it. The content gaps are highly visible right now. You know exactly what didn't fit neatly into the new AMS. You know about that messy SharePoint folder that everyone is too afraid to touch.

[AI Host 2] (18:06 - 18:28)
You've audited the house. You know where the messy closets are. But there is another more strategic reason why the timing is right.

And it goes back to that executive pressure we talked about earlier. Think about the boardroom. Right after a massive platform spend, leaders are desperate to prove to their board of directors that the new AMS created actual, tangible member value.

[AI Host 1] (18:28 - 18:32)
Because the board is going, we just spent $2 million. What did we get?

[AI Host 2] (18:32 - 18:46)
Exactly. And a governed AI layer can be deployed relatively quickly to sit across that newly organized stack. It can immediately begin revealing what members still cannot find.

It acts as a post-migration signal to prove ROI.

[AI Host 1] (18:46 - 18:52)
Oh, wow. So instead of just saying, hey, our database is faster now, you can actually show them a whole new member experience.

[AI Host 2] (18:52 - 19:05)
Yes. It shows the board, look, not only did we upgrade our operational infrastructure to keep our data secure, but we also unlocked a conversational layer that is directly answering member questions using our proprietary vetted content.

[AI Host 1] (19:05 - 19:06)
That's huge.

[AI Host 2] (19:06 - 19:13)
It bridges the gap between an invisible back office upgrade and a highly visible front office experience. It is the ultimate proof of value.

[AI Host 1] (19:14 - 19:24)
That is a brilliant way to frame it to a board. It changes the narrative from, we bought a massive database, to we unlocked our institutional knowledge and made it accessible.

[AI Host 2] (19:24 - 19:26)
Completely shifts the perspective.

[AI Host 1] (19:26 - 19:41)
But as much as we love the potential here, and as much as we love the idea of semantic search, we have to be completely realistic. We are looking at these implementation playbooks and they are very clear that there are times when you should absolutely hit the brakes.

[AI Host 2] (19:41 - 19:42)
Yes, you have to read the room.

[AI Host 1] (19:42 - 19:50)
As much as an AI layer helps, throwing it at a team that is currently drowning is a recipe for disaster. So when should an association wait?

[AI Host 2] (19:50 - 19:55)
You must delay looking at an AI layer if your team is still in pure stabilization mode.

[AI Host 1] (19:55 - 19:56)
Okay. Like triage.

[AI Host 2] (19:56 - 20:04)
Right. If the AMS just launched last week and the finance team can't figure out how to process a simple refund, stop.

[AI Host 1] (20:04 - 20:05)
Yeah, that's not the time.

[AI Host 2] (20:05 - 20:13)
If the membership team is still dealing with data migration cleanup because thousands of records are duplicated, stop. Do not pass go.

[AI Host 1] (20:13 - 20:13)
Good advice.

[AI Host 2] (20:14 - 20:30)
If you are struggling with severe adoption failures and your staff are practically revolting in the hallways against the new system, do not introduce a new technology. You will break your team's spirit. Do not overload an exhausted team.

Let them stabilize the ship first.

[AI Host 1] (20:31 - 20:40)
That makes total sense. I mean, you can't build the penthouse if the foundation is currently crumbling. If people are crying in the break room over spreadsheet mapping, do not pitch them an AI project.

[AI Host 2] (20:40 - 20:55)
Precisely. Empathy for the team has to come first. And there is one more scenario where you should wait.

And this one is a bit more insidious. What's that? I warn against leaders trying to use AI to distract from unresolved platform governance.

[AI Host 1] (20:56 - 20:57)
Interesting.

[AI Host 2] (20:57 - 21:08)
Sometimes an executive will see that their content strategy is a total disaster. Their taxonomy is non-existent. No one knows who owns what document.

And they have 10 versions of the same policy floating around.

[AI Host 1] (21:08 - 21:09)
A total mess.

[AI Host 2] (21:09 - 21:15)
And they think, let's just buy an AI to search through the mess so we don't have to clean it up. AI is not a bandaid for terrible governance.

[AI Host 1] (21:15 - 21:38)
OK, let's pull on that thread and be really real with the listener. The honest beat here. If someone is dealing with messy member data or a botched migration, will a tool like Betty just swoop in and fix it?

Because if you listen to the marketing out there in the broader AI world, it sounds like absolute magic. Right, the magic wand. It sounds like you just plug it in and it fixes all your operational problems.

[AI Host 2] (21:38 - 21:43)
That is a crucial question. And let's just use Betty as our clarifying example here to cut through the hype.

[AI Host 1] (21:43 - 21:43)
OK.

[AI Host 2] (21:44 - 21:57)
Betty is a governed, source-bound knowledge assistant. It is a fantastic implementation pattern for associations looking to create that AI layer. But let me be perfectly clear about what it does not do.

[AI Host 1] (21:57 - 21:58)
What doesn't it do?

[AI Host 2] (21:58 - 22:02)
It does not replace your AMS. It does not fix your bad member record data.

[AI Host 1] (22:02 - 22:03)
So if there are duplicates.

[AI Host 2] (22:03 - 22:13)
Right. If Jane Smith has three conflicting profiles in your database with three different email addresses, Betty is not going to magically merge them and figure out which one is correct.

[AI Host 1] (22:13 - 22:13)
Good to know.

[AI Host 2] (22:14 - 22:20)
It does not clean up bad migration work. And it certainly does not solve your underlying integration architecture.

[AI Host 1] (22:20 - 22:25)
So it's not the IT department in a box. It's not going to write your API connections for you.

[AI Host 2] (22:25 - 22:56)
No, it is not. Betty is simply a governed layer that sits above your tech stack. It only works when the sources and the access rules are already reasonably clear.

Got it. It activates your existing knowledge. It doesn't invent a new operational reality.

It relies heavily on the AMS to know who the member is, what their tier is, and what they have paid for. It relies on the CMS to hold the actual documents. It simply acts as the intelligent conversational retrieval layer across those systems.

[AI Host 1] (22:56 - 23:08)
I really love that distinction. It really respects the heavy lifting that an AMS does. It acknowledges that the AMS is the unsung hero keeping the association running while being completely honest about what it can't do natively.

[AI Host 2] (23:08 - 23:10)
Exactly. Give the AMS its due.

[AI Host 1] (23:10 - 23:18)
So if we pull all of this together, all the strategy notes, all the vendor realities, what is the core thesis we want you, the listener, to walk away with today?

[AI Host 2] (23:19 - 23:26)
It is simply this. Buying an AMS does not automatically make your knowledge conversational or findable. It just doesn't.

[AI Host 1] (23:26 - 23:33)
The real question you need to be debating with your executive team is not AMS or AI. That is a false dichotomy that will only lead to frustration.

[AI Host 2] (23:34 - 23:41)
Absolutely. The question you should be asking is, what job did the AMS solve and what knowledge access job remains?

[AI Host 1] (23:41 - 23:52)
That is the exact strategic framework every association should adopt. You hired the AMS to be your system of record. It is securing your transactions and managing your data.

[AI Host 2] (23:52 - 24:01)
Now you need to deliberately decide how you are going to handle the system of knowledge. How are you going to get the brilliance out of your bookshelves and into the hands of your members?

[AI Host 1] (24:01 - 24:10)
And to help you do exactly that, I'm going to remind you one more time about the resource we mentioned earlier. Look in the show notes for the AI layer versus platform decision brief.

[AI Host 2] (24:10 - 24:12)
Don't skip that brief. It's so helpful.

[AI Host 1] (24:12 - 24:27)
This is the tool you need for your next internal meeting. We want you to use this brief to map out the job to be done. It will help you separate the concerns that an AI layer can beautifully address from the core operations that your team still absolutely owns and must manage within your platforms.

[AI Host 2] (24:28 - 24:33)
Do this exercise before you decide to add an AI layer or before you decide to delay it.

[AI Host 1] (24:33 - 24:40)
Yeah, it will bring incredible clarity to your roadmap. And honestly, it might just save you from making a very expensive miscalculation.

[AI Host 2] (24:40 - 24:59)
And as you go into those roadmap meetings, I want to leave you with one final provocative thought. Think about the search bar on your association's website right now, not a year from now, but today. Imagine a member right at this very second just typed in a complex, multi-layered question about a pressing professional issue.

[AI Host 1] (24:59 - 25:01)
Like the nursing credits example.

[AI Host 2] (25:01 - 25:25)
Exactly. A question that requires pulling from a course, a policy, and a forum. And imagine your brand new multi-million dollar AMS couldn't give them a conversational cited answer from your approved sources.

Ouch. They hit search and get a disjointed list of irrelevant links or worse, zero results. What is the hidden cost of that member walking away and being handed today?

[Rob Barnes] (25:30 - 27:06)
The question today was, we just bought a new AMS. Why would we consider an AI project now? And Thomas, the first thing that came to my mind after listening to the episode today was, I feel for the association finance manager, CFO or CEO. Because in the cases where we have this sunk cost now into one of the four association management softwares, whether it's the AMS, the CMS, the LMS, right now I am being presented with, I was going to say bombarded by, but that's probably not fair, but I'm presented with an AI tool that is now free for anyone that uses this AMS. And same for the financial management software. Why won't I just go in and use the free tool that's part of the software that I've already paid for and I already own and hope that it's going to solve some of the challenges that we talked about today. And it reminds me a little of episode three, where again, it was ChatGPT co-pilot versus a governed AI specifically for an association and that decision-making tree. We're kind of here again now.

Is this a new problem because of these AI bolt-ons to legacy platforms or new management platforms? Or does this have a new layer on it because it is AI? Either way, I don't envy the decision, the CFO right now.

[Thomas Altman] (27:07 - 30:37)
I think it has a lot of overlap, honestly, with episode three, which was the ChatGPT co-pilot versus governed AI. And I think the answer is going to be fairly similar, is maybe you do use it for certain things, right? And if the question is why now, sometimes the answer, the best answer would be not now, right?

Maybe we wait a little bit and we see what these systems do and what their limitations are, right? And we use what we have. But you should do so in a way that is kind of informed, kind of driving to some underlying understanding of how these systems work more deeply.

And I think that's the framework that today's companion asset, I think, helps you walk as well. But to kind of talk through that a bit, I think the way I would think about this is, yes, we've got an AMS. Do we do it? Do we add in a governed AI interface, an AI, a knowledge platform kind of in front of that as an interface to the member to improve member experience?

Or are we using internal AI that helps you use that product better? That's generally the distinction. What I think would probably fail is trying to hack some internal system that's meant to help you use a specific platform better to exist as this front-end kind of interface that kind of is the ghost in the machine behind everything else and helps the member and the public and whoever's going to be interacting with your association from the outside get to where they need to go, have the best experience possible, accomplish more of what they set out to accomplish and help you achieve your mission. And the reason for that is a lot of these MSs, the management systems, is they're at their core transactional databases.

Something happened, we have a record of it that changed something else about your person record or your company record or whatever it might be. And being able to use that information to inform how an AI system in front of that interacts and deepens engagement, I think it's a different tool, right? So why now?

Maybe not now. Maybe you wait when you still have some cleanup to do, right? The migration didn't go perfectly.

You went from an old AMS or an old LMS to a new one and things aren't clean. You've got data to duplication to do all that. Or maybe you haven't figured out how the process of refund, right, in the new AMS. That happens all the time. Those kinds of things, you've got to get stable first. It could be now if you were bringing on this AMS as a way to revolutionize the member engagement strategies, right? Because the AMS isn't going to do that.

The sort of AI system in front of that will or should. And being able to take advantage of that, the last kind of key point I kind of want to jump into is if you do that part well, right, if you have this AI in front, what it can do is make the ROI on your investment to the AMS higher and you get that ROI faster, right? Because it's actually taking advantage of a lot of the work that went into that in a way that deepens engagement with the members, right?

Revolutionizing that member engagement strategy I think is the core sort of benefit of adopting an AMS. And it's not going to do it on its own. So you want to adopt that AI strategy when it's time to kind of take advantage of what you already invested in.

[Rob Barnes] (30:38 - 31:27)
That's a really good way to round out this episode, Thomas. I agree. I think that there's some really interesting ways to think about this, which is slightly different now because it's the AI era, if you like, but they're certainly very common challenges to be faced.

So the more the listeners today and our association executive friends can start thinking about that using today's downloadable, which I say one more time before you go today, download the AI layer versus platform decision briefing paper from the show notes. It's the fastest way to actually put today's episode to work. And a very important episode it was.

Thank you for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question. What if it's wrong about something that matters?

What this helps your team answer

Why an AMS (or any management system) is a system of record, not a system of knowledge — and why AI should never try to replace that record, only sit in front of it as a governed interface.
How to tell whether "not now" is the right call (a messy migration, unresolved data cleanup, unstable processes) versus a stall tactic — and why waiting shouldn't mean waiting indefinitely.
Why a governed AI layer can actually increase the ROI on the AMS you already bought, by deepening member engagement in ways the AMS alone was never built to do.

Companion Asset

AI Layer vs. Platform Decision Brief

AI Governance

What If Our AI Is Wrong About Something That Matters?

Why accuracy isn't a dial you turn up to 100% — a framework for governing AI by source scope, audience, traceability, and human ownership, plus a hard question every association needs to answer: if the AI performs flawlessly but the underlying content was wrong, who's accountable?

34:07

0:00 34:07

AI PC ep 8

[Thomas Altman] (0:06 - 0:10)
Today's big question, what if our AI is wrong about something that matters?

[Rob Barnes] (0:11 - 1:05)
We are Thomas Altman and Rob Barnes and this is episode 8 of the Association Intelligence Podcast. Before we get into it, this episode's companion, as said, is the high stakes AI governance checklist. Please check it out in the show notes, grab it now if you want to follow along or download it after and put it to work.

This episode argues around accuracy and accuracy not being a dial that we play with. That's governed through source scope, version scope, audience, traceability, human coaching boundaries, all of the things that we know is coming along in leaps and bounds in modern AI systems. Thomas, though, is this a technical enough conversation to reassure our legal and compliance audience listening in today or does it need to be simplified a little bit further than that?

[Thomas Altman] (1:06 - 3:33)
Yeah, I think technical is definitely not the path we want to go down, right? But I think the important part here, the idea that our AI hosts explore and what we've seen really in the real world is this idea that accuracy is fundamental to the proper use of AI, the valuable use of AI, especially for associations, right? It matters for everybody, but the thing that associations have and have built up over decades, if not centuries in some cases, right, is trust that they are producing good content, right?

They're producing knowledge in a way that is going to be accurate and useful. And if you let AI step into the middle of that and start providing inaccurate information, that is a big risk to the heart of what an association does at the end of the day. That's not to say that you shouldn't do AI, right?

That is not to sort of scare people away from it. In fact, I think there's plenty of reasons where using AI actually augments that trust, but it is important to be aware of. And the thing that I think this episode leans into and what we've seen really matter is that large language models, the type of AI that people mean when they say AI, like your chat GPTs, your kind of clause, your Bettys, right?

All of those things work on a large language model. And those large language models are extremely sensitive to context, right? Context meaning what else does it see?

What information does it see? What information does it have in its memory or its sort of window of content that it has access to? And the way that that's constructed fundamentally directs the output, the way that it responds.

So if you provide these things with grounded real-world information that comes from vetted sources and you provide a set of guardrails around how to use that information, the end result is actually better delivered versions of the knowledge that you're already hosting, right? And as long as you can kind of get that sort of mix of content correctly, that context that these agents see correct, the output is going to be really powerful and the guardrails need to be enforced along the way. And I think this episode does a really good job of exploring where that works and where that doesn't work.

[Rob Barnes] (3:33 - 4:54)
Some of the things that I hear consistently from association execs when we're talking about this is they ask about interpretation risk. You know, they ask about, you know, what if it gets it wrong? And I always go straight back with, well, how do you know, what do you do now when one of your team gets it wrong?

Because if the same person calls in and gets three different staff members from an association and asks the same question, they're going to get three different answers based on the experience, the tenure, the institutional knowledge of that individual staff member. Most associations don't have a mechanism for tracking that or auditing what is being interpreted and passed on in communication. AI, at the very least, gives us an audit trail.

You know, we can see how that is responding and we can coach. Part of that discussion is like, is escalation? Is there a point where these boundaries that we can put in place become an escalation point?

The LLM can say, I'm no longer able to answer this. This is where you should go next. Or is it simply a case of the user's going to experience it as the product?

It doesn't work because it didn't know how to answer that question and it palmed me off to somebody else.

[Thomas Altman] (4:54 - 5:31)
Right. Exactly. And I think that auditability trail, that audibility is such a key component of a governed system, right?

So there is the governance on the front end, which is the context and the rules and the guidelines that it's meant to follow. But then there's the conversation history, the auditable trail of, did it actually abide by those rules? Can you point to, hey, this conversation actually did not?

And then go back and one, correct any errors you may have seen, and then two, make sure you can go back into the system and tweak it so that it's more likely to stay in line with what it's supposed to do.

[Rob Barnes] (5:31 - 5:58)
This is one of the more important episodes that we are going to record, I think, for the audience today. Before we hand this over to the AI Podcasts hosts, a quick reminder that the following is generated using Notebook LM, with heavy involvement from the Betty team to ensure that the content generated is valuable. For a deeper dive on our approach, please go back and listen to episode zero.

And now, over to the AI hosts.

[AI Host 1] (6:02 - 6:06)
So what if it's wrong about something that matters?

[Thomas Altman] (6:06 - 6:06)
Yeah.

[AI Host 1] (6:06 - 6:15)
I mean, if you are an executive, a board member, or really just anyone steering an organization right now, that exact question is the thing keeping you up at night.

[AI Host 2] (6:16 - 6:16)
Oh, absolutely.

[AI Host 1] (6:17 - 6:31)
You're looking at this massive wave of artificial intelligence adoption. You're seeing all the pressure to implement it yesterday, and you're just asking, well, what happens if this thing makes a mistake that actually genuinely hurts the organization?

[AI Host 2] (6:31 - 6:54)
Right. And I think we really need to validate that hesitation right out of the gate. Because it is incredibly valid.

It's not fear-mongering, and it certainly isn't some stubborn resistance to new technology or anything like that. Asking what if it's wrong about something that matters is honestly the single most responsible question a leader can ask when they're evaluating AI.

[AI Host 1] (6:54 - 6:55)
It's the right question.

[AI Host 2] (6:55 - 7:12)
Exactly. The problem, though, the real issue, is how most organizations are actually attempting to answer it. Because they treat accuracy like it's a single toggle switch on a dashboard.

Just a dial. Right. Just a dial.

They need their engineers to crank up to 100%. And that's just not how it works.

[AI Host 1] (7:12 - 7:26)
Which is a fundamental misunderstanding of the tech. Because accuracy isn't a model setting. It's just not.

And so the thesis for today's deep dive, our mission here, is to show you that the only way to actually sleep at night is to govern the use case.

[AI Host 2] (7:26 - 7:28)
Yes. Govern the use case.

[AI Host 1] (7:28 - 7:44)
We are going to unpack this whole framework where accuracy is actually defined by source scope, answer scope, human ownership, and risk tiers. Because to understand why we have to govern the use case, I mean, we first have to separate the types of work we're actually asking the AI to do.

[AI Host 2] (7:45 - 7:54)
Right. And that separation really requires understanding this crucial difference between what I like to call a helpful shortcut and a dangerous guess.

[AI Host 1] (7:54 - 7:55)
Okay. Break that down for us.

[AI Host 2] (7:55 - 8:06)
So we have to address this trap of, well, let's call it generic hallucination. Because every article you read about AI includes the exact same blanket warning, right? They all say AI can hallucinate.

[AI Host 1] (8:06 - 8:07)
Always. It's like the mandatory warning label.

[AI Host 2] (8:08 - 8:27)
Exactly. And while that is technically true, treating it as some sort of universal law paints this entire massive spectrum of AI use cases with the exact same brush. It essentially treats every single task as if it carries the exact same level of risk.

And when you do that, you just paralyze your operational decisions.

[AI Host 1] (8:27 - 8:35)
So it's basically the difference between asking an AI for, I don't know, a smoothie recipe versus asking it for clinical dofage guidelines.

[AI Host 2] (8:35 - 8:37)
Yes. Perfect analogy.

[AI Host 1] (8:37 - 8:45)
Because if the model is just, you know, predicting the next most likely word and it hallucinates some weird non-existent fruit into my smoothie recipe that's just a bad Tuesday.

[AI Host 2] (8:45 - 8:46)
Right.

[AI Host 1] (8:46 - 8:56)
It tastes gross. I toss it out. Whatever.

But if it uses that exact same predictive guesswork to hallucinate a clinical standard or like a legal regulation. Yeah. That is a total disaster.

[AI Host 2] (8:56 - 8:57)
It's catastrophic.

[AI Host 1] (8:57 - 9:00)
Yeah. You can't just brush that off as a quirky tech glitch.

[AI Host 2] (9:00 - 9:09)
No, you can't. And that dosage scenario, that is exactly where boards and executives completely freeze because they view all AI through that specific high stakes lens.

[AI Host 1] (9:10 - 9:11)
Everything is a dosage problem to them.

[AI Host 2] (9:11 - 9:33)
Exactly. And because they're only equipped with this generic understanding that, well, AI makes things up sometimes, they just halt all adoption. Everything stops.

Even the really low stakes convenience tasks like summarizing meeting notes or drafting routine emails, all of that gets blocked because leadership simply doesn't have a structured way to separate the operational risk.

[AI Host 1] (9:33 - 9:41)
Man, that operational freeze is so real. And honestly, it is exactly why we put a specific tool in today's show notes for you to grab right now.

[AI Host 2] (9:41 - 9:42)
Highly recommend it.

[AI Host 1] (9:42 - 9:53)
It's called the High Stakes AI Governance Checklist. If you are listening to this right now and you're feeling that exact paralysis in your own organization, you really need to download this asset.

[AI Host 2] (9:53 - 9:55)
It's a game changer for those conversations.

[AI Host 1] (9:55 - 10:12)
It really is. It gives you this concrete framework to look at all your organization's topics and actually classify them. Like, okay, this one is safe to launch.

This one needs a restricted pilot. And this topic absolutely must stay human only. It moves the whole conversation from just vague anxiety to practical deployment.

[AI Host 2] (10:13 - 10:40)
And having that checklist changes the entire dynamic in the boardroom. Because once you realize you can't just tell an AI to trust the model harder for those high stakes clinical and regulatory topics, you are forced to take a completely different architectural approach. You stop relying on the AI's vast, messy internet memory, right?

And instead, you build a really rigid structure around exactly what the AI is allowed to know and how it's allowed to operate.

[AI Host 1] (10:40 - 10:58)
Okay. So let's get into the mechanics of that structure. Because saying we're going to build a rigid structure sounds amazing in theory.

Sure. But I want to know how it physically functions. Like, if I'm an executive looking at my chaotic internal network of random PDFs and old policies, how do I literally stop the AI from just making a dangerous guess?

[AI Host 2] (10:58 - 11:02)
Well, this is where we implement what is called the governance stack.

[AI Host 1] (11:02 - 11:02)
The governance stack.

[AI Host 2] (11:02 - 11:44)
Right. It's essentially a series of completely non-negotiable boundaries. And to understand how it works, we have to look under the hood at this architecture called retrieval augmented generation, or RAG for short.

The very first layer of this governance stack is the source boundary. So instead of letting the AI just generate an answer based on its, you know, underlying training data from the public internet, we intercept the user's prompt. The jump in the middle.

Exactly. The system searches your specific, highly locked down database for relevant paragraphs. It extracts only those approved texts, and then it commands the AI to synthesize an answer using strictly those paragraphs.

Nothing else.

[AI Host 1] (11:44 - 11:56)
Okay, wait. So we aren't actually teaching the AI our rules. We're basically forcing it to do this hyper-targeted search of our approved library, and then it's just acting as a summarization engine for whatever it finds in that library.

[AI Host 2] (11:56 - 11:59)
That is the exact mechanism. Yes. You dictate the approved sources.

[AI Host 1] (11:59 - 11:59)
Wow.

[AI Host 2] (12:00 - 12:07)
But, and this is crucial, simply locking down the library isn't enough on its own, which introduces the next layer, which is the version boundary.

[AI Host 1] (12:07 - 12:08)
Because things change.

[AI Host 2] (12:08 - 12:08)
Right.

[AI Host 1] (12:08 - 12:09)
Right.

[AI Host 2] (12:09 - 12:18)
Think about your standards and regulations. The AI has to be able to distinguish between your current 2026 guidelines and the superseded rules from like 2024.

[AI Host 1] (12:19 - 12:29)
Yeah. And if you're listening right now and thinking about your own company's internal wiki or SharePoint, you probably know exactly where those conflicting 10-year-old PDFs live.

[AI Host 2] (12:30 - 12:30)
Oh, they're everywhere.

[AI Host 1] (12:31 - 12:31)
They're a nightmare.

[AI Host 2] (12:32 - 12:32)
Mm-hmm.

[AI Host 1] (12:32 - 12:42)
And an AI reading a defunct 2018 policy is honestly just as dangerous as one reading a random Reddit thread. So how does the system actually know the difference?

[AI Host 2] (12:43 - 12:48)
It does it through really rigorous metadata tagging. These tags are attached to the vectors in your database.

[AI Host 1] (12:49 - 12:49)
Okay.

[AI Host 2] (12:49 - 13:01)
So when the system searches for those relevant paragraphs we talked about, the version boundary acts as a hard filter. It explicitly tells the retrieval engine, hey, ignore any document that is not tagged as the active current policy.

[AI Host 1] (13:02 - 13:02)
That's brilliant.

[AI Host 2] (13:03 - 13:10)
Yeah. So if a member asks about a compliance requirement, the AI is mathematically forced to pull from the correct temporal version of the truth.

[AI Host 1] (13:10 - 13:24)
Okay. So restricting the AI to our approved library stops it from surfing the web. And the version boundary stops it from surfing our own outdated trash.

But what about permissions? Because not all information is meant for everyone in an organization, right?

[AI Host 2] (13:25 - 13:25)
Yeah.

[AI Host 1] (13:25 - 13:36)
Like, if a frontline worker asks a question, we definitely don't want the AI happily summarizing confidential board-level financial projections just because it found them in the approved source base.

[AI Host 2] (13:36 - 13:56)
No, you definitely don't want that. And that requires layering on the audience boundary. The audience boundary.

Okay. So the system evaluates exactly who is asking the question before it ever performs the search. The access tiers dictate the retrieval itself.

So if the user is just a public guest on the site, the system only retrieves from public documents.

[Thomas Altman] (13:56 - 13:56)
Makes sense.

[AI Host 2] (13:56 - 14:07)
If they're a paying member, it unlocks the member-gated content. The AI never even sees the restricted board documents unless the user's credentials explicitly authorize it.

[AI Host 1] (14:07 - 14:16)
Okay. Which brings us to what I think is the most critical layer for high-stakes topics. And frankly, this is my favorite part of this whole framework, the traceability boundary.

[AI Host 2] (14:16 - 14:17)
Oh, it's so important.

[AI Host 1] (14:17 - 14:21)
Because it's literally like putting the AI in a strict open book exam.

[AI Host 2] (14:21 - 14:22)
That's a great way to put it.

[AI Host 1] (14:22 - 14:41)
Right. We choose the exact textbooks it's allowed to open. That's the source boundary.

We make sure that the 2026 editions, that's the version boundary. But for traceability, we actually force the AI to cite the exact page number, the paragraph, and the document ID to show its work. It cannot just give you an answer and say, hey, trust me.

[AI Host 2] (14:41 - 14:59)
No. And that citation isn't generated after the fact either. It is wired directly into the system prompt.

The AI is actively commanded to output the specific reference link right alongside the sentence it generates. And a really soft, practical example of this exact implementation pattern is Betty.

[AI Host 1] (14:59 - 15:01)
Oh, yeah. Let's talk about Betty.

[AI Host 2] (15:01 - 15:18)
Yeah. So Betty operates as a completely governed, source-bound, association-specific knowledge assistant. It is built entirely around this governance stack we're discussing.

The answers it gives are strictly grounded in the approved content, and they're instantly traceable right back to the source document for the user to verify.

[AI Host 1] (15:18 - 15:22)
But I want to pause here. I want to throw a massive red flag on the play right now.

[AI Host 2] (15:22 - 15:22)
Okay. Do it.

[AI Host 1] (15:22 - 15:35)
Because when we start talking about forcing the AI to read only approved sources and locking the versions and providing citations, it becomes incredibly tempting to start using really dangerous marketing language.

[AI Host 2] (15:35 - 15:37)
Oh, I know exactly where you're going with this.

[AI Host 1] (15:37 - 15:41)
Right. Like, people immediately start wanting to call this hallucination free.

[AI Host 2] (15:41 - 15:49)
Yes. And we must strictly, strictly warn against ever calling this, or frankly, any system hallucination free.

[AI Host 1] (15:49 - 15:50)
Never do it.

[AI Host 2] (15:50 - 16:01)
You will hear vendors throw that term around all the time, and it is fundamentally false. Let me be clear. No system, no matter how rigorously governed you make it, is a perfect guarantee machine.

[AI Host 1] (16:01 - 16:03)
Right. You aren't defying physics.

[AI Host 2] (16:03 - 16:16)
Exactly. You are not defying the underlying laws of computer science. Large language models are still, at their core, probabilistic engines that are predicting text.

By establishing a rigid governance stack, you are drastically reducing the risk. Yes.

[AI Host 1] (16:17 - 16:17)
Massive reduction.

[AI Host 2] (16:18 - 16:24)
And you are providing provenance for the answer. But it is still just a governed tool. It is not some magical, infallible oracle.

[AI Host 1] (16:25 - 16:45)
Okay. So, if we've locked the data, we've locked the version, we've forced the citations, and we've accepted that nothing is 100% perfect, what actually happens when a user asks a question that is technically covered by the approved sources, but it crosses the line of what a machine should legally or ethically be interpreting?

[AI Host 2] (16:46 - 16:49)
Now we're getting into the really nuanced stuff. This is where we introduce the answer boundary.

[AI Host 1] (16:50 - 16:51)
The answer boundary.

[AI Host 2] (16:51 - 17:01)
Right. Because the governance stack isn't just about feeding the AI the right information. It is equally about hardwiring a no-comment reflex into the model.

[AI Host 1] (17:01 - 17:03)
I like that. A no-comment reflex.

[AI Host 2] (17:03 - 17:09)
Yeah. Knowing exactly when the AI must say, I don't know, or, to be more accurate, I am not allowed to answer that.

[AI Host 1] (17:09 - 17:11)
Okay. I have to push back on this a little bit.

[AI Host 2] (17:11 - 17:11)
Go for it.

[AI Host 1] (17:12 - 17:41)
Because if I'm an executive looking at this framework, I'm thinking about efficiency, ROI. So, let's say we did all the heavy lifting. We fed the AI the exact board-approved clinical guidelines.

We locked the version, so it's only using the pristine 2026 data. We verified all the traceability. If the perfect data is sitting right there in the database, why can't the AI just quote it back to the user?

Why force a refusal or an escalation if the answer is literally right there in the approved text?

[AI Host 2] (17:41 - 17:47)
It's a fair question, but it comes down to the critical difference between quoting a rule and applying professional judgment.

[AI Host 1] (17:48 - 17:48)
Okay.

[AI Host 2] (17:48 - 18:02)
Think about how humans actually interact with regulations in the real world. Let's say a professional comes to your platform and they ask how a highly nuanced regulatory standard applies to their specific, incredibly messy real world situation.

[AI Host 1] (18:02 - 18:04)
Right. Because real situations are always messy.

[AI Host 2] (18:04 - 18:17)
Exactly. Now, the AI can absolutely find the regulation and can quote it flawlessly, but the AI completely lacks the professional context to interpret the conflicting nuances of that person's specific scenario. I see.

[AI Host 1] (18:17 - 18:22)
Because the real world rarely fits perfectly into the bullet points of a compliance manual.

[AI Host 2] (18:22 - 18:35)
Precisely. It almost never does. When a doctor or a lawyer or a certified professional reads a standard, they are actively applying years of experience, professional ethics, and nuanced judgment to that text.

[AI Host 1] (18:36 - 18:37)
And the AI is just matching words.

[AI Host 2] (18:38 - 18:56)
Yes. And AI is just mathematically matching concepts. So even with absolutely perfect source material, there are these liability-sensitive topics, things like legal interpretations, definitive certification requirements, specific medical diagnoses where you simply cannot have an AI issuing guidance.

[AI Host 1] (18:56 - 19:00)
So we have to actually program guardrail models that run in parallel.

[AI Host 2] (19:01 - 19:01)
Yeah.

[AI Host 1] (19:01 - 19:09)
They evaluate the user's intent, flag it as, hey, this is legal advice, and then block the main model from responding, forcing an escalation instead.

[AI Host 2] (19:09 - 19:25)
That's the exact architecture. In those sensitive areas, the system is explicitly programmed to refuse the prompt, provide a strict caveat, or escalate the question to a human expert. Okay.

And here's the thing. When the AI hits that answer boundary and escalates, you shouldn't view that as a failure of the technology.

[AI Host 1] (19:25 - 19:27)
It feels like a failure when you're paying for it.

[AI Host 2] (19:27 - 19:39)
I know, but it's not. That escalation is a designed feature of high-stakes governance. It is the system working exactly as intended to protect your organization from massive liability.

[AI Host 1] (19:39 - 19:41)
Man, that reframe is huge.

[AI Host 2] (19:41 - 19:41)
Yeah.

[AI Host 1] (19:41 - 19:44)
The escalation is a feature, not a bug.

[AI Host 2] (19:44 - 19:44)
Exactly.

[AI Host 1] (19:44 - 19:53)
But, you know, that reliance on the source material to dictate the AI's behavior, it kind of leads us to a really uncomfortable truth about this entire process.

[AI Host 2] (19:53 - 19:53)
Which is?

[AI Host 1] (19:53 - 20:03)
Well, we're relying entirely on the documents we feed it, right? We're basically telling the AI, only read these files, only trust these policies. What happens when those files are fundamentally broken?

[AI Host 2] (20:03 - 20:07)
Yeah. It is time for the data reality check.

[AI Host 1] (20:07 - 20:07)
Let's hear it.

[AI Host 2] (20:08 - 20:24)
Here's the hard truth that every single content operations team out there needs to hear. Deploying a source-bound AI absolutely reduces your external risk. It stops the AI from pulling dangerous info from the public internet, and this is a big buck, but it does not make weak source content strong.

[AI Host 1] (20:24 - 20:26)
So if the foundation is rotten, the house still falls.

[AI Host 2] (20:27 - 20:49)
Yes. Source-bound AI simply reflects the approved source base you give it. It cannot magically resolve policy conflicts that already exist in your documentation.

It cannot remove historical blind spots that past committees might have accidentally written into those standards, and it absolutely does not fix obsolete or contradictory guidance that is just buried deep in your archives.

[AI Host 1] (20:49 - 21:00)
Let's really unpack that, because I feel like this is where the theory hits the messy reality of corporate data. The AI is essentially holding up a high-definition mirror to our own documentation.

[AI Host 2] (21:00 - 21:01)
That's a perfect way to look at it.

[AI Host 1] (21:01 - 21:19)
Right. Exactly. Like, if our files are a mess of contradictory policies, say one PDF says a certification requires 40 hours of training, and another totally valid document from three years ago says 30 hours, the AI doesn't synthesize a compromise.

It just gives us a highly efficient, perfectly traceable mess.

[AI Host 2] (21:19 - 21:28)
Yes. And holding up that mirror forces a massive operational reckoning for organizations. AI activates your existing knowledge.

It does not replace your content strategy.

[AI Host 1] (21:28 - 21:29)
That's a great line.

[AI Host 2] (21:29 - 21:42)
If your source material is flawed, the governed AI's job is to highlight that flaw safely and traceably so your team can finally see it. Its job is not to silently invent a fix behind the scenes.

[AI Host 1] (21:42 - 21:43)
Oh, wow. Yeah.

[AI Host 2] (21:43 - 21:51)
Honestly, having an AI silently arbitrate a conflict between two of your official policies, that would be a massive breach of governance.

[AI Host 1] (21:51 - 22:03)
Which is terrifying, honestly. Absolutely. But also, it's incredibly useful because normally, those contradictory policies just sit in a drawer for a decade until a member gets furious about a certification denial.

[AI Host 2] (22:04 - 22:04)
Exact growth.

[AI Host 1] (22:04 - 22:17)
The AI surfaces the conflict instantly. But okay, because your source content will inevitably have gaps, and because the AI will constantly encounter novel questions that force it to escalate, we clearly need a mechanism for humans to step in.

[AI Host 2] (22:17 - 22:18)
We do.

[AI Host 1] (22:18 - 22:26)
And we need to do it without breaking the efficiency of the whole system. We can't just have highly paid subject matter experts sitting there manually answering every single AI prompt that gets flagged.

[AI Host 2] (22:26 - 22:55)
No, definitely not. That would just be running a glorified call center, which completely defeats the purpose of buying the technology in the first place. And this brings us to the final and arguably the most dynamic layer of the stack, which is coaching and review.

This introduces what we call the human and measurement boundaries. And the goal here is not for your subject matter experts, your SMEs, to sit at a dashboard correcting individual AI answers in real time.

[AI Host 1] (22:55 - 23:01)
So if they aren't answering the questions one by one, what is their actual workflow? How are they coaching it?

[AI Host 2] (23:01 - 23:07)
They interact with the AI at the measurement boundary. So they look at the telemetry and the cluster analysis of all the escalations.

[AI Host 1] (23:07 - 23:07)
Got it.

[AI Host 2] (23:08 - 23:22)
Let's say your dashboard shows that the AI escalated 50 questions this week about a very specific new regulatory update. The SME doesn't manually reply to those 50 users. They analyze the cluster, and that pattern triggers a root cause action.

[AI Host 1] (23:23 - 23:24)
It's like being a tennis coach.

[AI Host 2] (23:24 - 23:25)
Oh, I love this analogy.

[AI Host 1] (23:25 - 23:43)
Right, because you don't run onto the court and physically swing the racket for your player during the actual match. You sit on the sidelines, you review the game tape afterward, and you spot the patterns. You see that their backhand is failing whenever they get pushed to the baseline.

You don't jump in and hit the backhand for them. You adjust their coaching for the next match so they can handle it themselves.

[AI Host 2] (23:44 - 23:58)
That perfectly captures the exact workflow here. The SMEs are looking at the game tape. When they see those 50 escalations, they realize, oh, the AI is failing because our approved policy document is just too vague.

Right.

[AI Host 1] (23:58 - 23:59)
It's a source problem.

[AI Host 2] (23:59 - 24:18)
Exactly. So the SME's job is to go write a clearer policy document, upload it to the approved source library, and instantly upgrade the AI's capability for the next match. Or, you know, if the industry hasn't settled on a standard yet, the SME adjusts the system prompt to explicitly narrow the AI's scope on that topic.

[AI Host 1] (24:18 - 24:24)
Which really makes the AI an adaptable layer rather than just a static piece of software you buy and forget about.

[AI Host 2] (24:24 - 24:25)
Yes, it's a living system.

[AI Host 1] (24:26 - 24:39)
And it clearly defines who owns the escalation rule and who is ultimately responsible for the knowledge. You aren't just launching an AI and crossing your fingers. You have a defined human workflow for continuously refining its boundaries based on actual user behavior.

[AI Host 2] (24:40 - 25:02)
And I'll tell you, that continuous refinement is exactly what separates successful enterprise AI from failed experiments. You are moving away from treating AI as this terrifying black box that you either blindly trust or entirely fear. You're treating it as a structured, measurable system with limits that your human experts actively control.

[AI Host 1] (25:02 - 25:20)
This has been such a massive perspective shift. I mean, let's recap the core thesis here. Because if you take one thing away from this deep dive, it really should be this.

When you ask that necessary responsible question, what if it's wrong about something that matters? The answer isn't to panic. And the answer certainly isn't to just trust the model.

[AI Host 2] (25:21 - 25:21)
Definitely not.

[AI Host 1] (25:21 - 25:49)
The answer is to separate the concern. You isolate the part that AI can safely address through strict governance like your source boundaries, version control, and traceability. You acknowledge the part that your content operations team still owns, which is the quality, the clarity, and the authority of your actual source documents.

And you clearly define the proof, the coaching workflows, and the answer boundaries you need before you scale it up to your entire audience.

[AI Host 2] (25:49 - 26:02)
It really all comes down to moving your organization from a state of vague anxiety to a state of operational rigor. You have to clearly map out those risks before you write a single line of code or sign a vendor contract.

[AI Host 1] (26:03 - 26:11)
Which is exactly why you don't have to figure out all those limits from scratch today. Before you close out of this deep dive, pull up the show notes and download the High Stakes AI Governance Checklist.

[AI Host 2] (26:11 - 26:12)
It's the best place to start.

[AI Host 1] (26:12 - 26:33)
It really is. You don't need to overhaul your entire IT infrastructure today to start making progress. Just take the checklist, sit down with your compliance and content teams, and start classifying your organization's topics.

Figure out your risk years, your source confidence, your audience rules, and who owns the escalation for each topic. It is the single most practical first step you can take to get this right.

[AI Host 2] (26:34 - 26:42)
And as you start having those conversations with your team, you know, mapping out your governance stack, I want to leave you with one final question to ponder on your own.

[AI Host 1] (26:42 - 26:43)
Ooh, let's hear it.

[AI Host 2] (26:44 - 27:01)
Let's say you implement the governance stack flawlessly. You build a perfectly restricted AI. It executes your approved traceable guidelines to the exact letter, pulling the exact right version, respecting every audience boundary, and refusing the topics it absolutely shouldn't touch.

[AI Host 1] (27:01 - 27:02)
Sounds like a dream.

[AI Host 2] (27:03 - 27:12)
Right. But the resulting real-world outcome for the user is still wrong, because the approved knowledge sitting in your database was fundamentally flawed from the start.

[AI Host 1] (27:12 - 27:12)
Wow.

[AI Host 2] (27:13 - 27:19)
If the AI performed perfectly, but the outcome failed, who is ultimately accountable for that knowledge?

[AI Host 1] (27:19 - 27:26)
A perfectly governed mirror reflecting a broken policy. You really have to ask yourself, are you ready to see what the mirror actually shows?

[AI Host 2] (27:26 - 27:27)
It's a big question.

[AI Host 1] (27:27 - 27:31)
It is. Thanks for joining us on this deep dive. We'll catch you next time.

[Rob Barnes] (27:34 - 28:26)
So recapping today, the big question is, what if it's wrong about something that matters? And I've really enjoyed this discussion. And the closing question actually is kind of one of the sharpest questions that we will have received all season, to be fair.

If the AI performs perfectly, but the underlying source content was wrong all along, who's accountable for that? This is quite the challenge to associations, Thomas, because in a lot of cases, we collectively don't know every piece of content that ends up in a knowledge base and can end up in a poorly governed AI. And even with the governance infrastructure around that AI, if the source content is out of date or there's version control issues, who's responsible for that?

[Thomas Altman] (28:27 - 30:15)
Right. And this idea of responsibility or accountability, I think is extremely important in the AI era. Because a machine can't be.

The AI is going to do what it's told to do. There's no accountability process for, you know, you can't give it the merits and it's going to like learn its lesson, right? It's going to do what it's told to do.

It's going to do what it's told to do. And what I think matters there is the accountability then has to fall on the human systems, right? And making sure that your content is up to date and relevant is going to be extremely important.

But the way that AI can serve that process is identified in the first place. Because I guarantee you, this is already happening without AI in the system, right? You have outdated content and people are using it in ways that they shouldn't be, right?

You've got maybe vague content that's not super clear about what actually needs to happen in a certain scenario. And people are assuming it means one thing when it doesn't quite, right? Once you deploy AI to discover that for you, it actually gives you an opportunity to resolve the problem that's already there but hidden, right?

There's a sort of shadow problem that exists within any knowledge base of people getting it wrong or getting access to something that is outdated. So having AI in the mix to surface that information, I think it's really, really important. So as you start to think about AI systems and become a more AI-forward association, thinking about having that component where AI is also, although not accountable, it's helping you stay accountable to the quality of content that you already have.

[Rob Barnes] (30:17 - 31:43)
Yeah, I wonder if the challenges, and I guess this is also why we've talked about associations being the best placed organizations in the world to benefit from this technology. Because by and large, we already have systems and processes in place for managing version control and the efficacy of documentation across the association. We have those systems and processes in place.

We now just need to apply that same level of rigor to whatever is ending up in the knowledge base for an AI, for a generative AI tool. A lot of it's going to be messy, a lot of it's going to be contradictory, and in fact an AI is better at surfacing contradictory content than most humans are. So that's a valuable opportunity there as well.

And we've already seen systems like Betty, for instance, being able to reduce some of that external risk by surfacing this in testing and onboarding processes before it gets released to the membership or to the public. If there's another way to advance that process, Thomas, what else should our association executives be thinking about in terms of going, there's a step here for us to take to do as best as we possibly can to make sure that this content is the right content?

[Thomas Altman] (31:44 - 33:23)
I would say thinking about, especially, I think this is mostly relevant for groups like standard bodies or groups that have sort of frequent policy updates, right? You have a document that's the source of truth, but that document evolves over time, right? Making sure you've got a clear policy for your AI, once you're thinking about AIs, of how to use the difference between those things.

If every year you release a new report and someone says, what are the details around this metric that's sort of always in the report and it's updated, making sure, one, that the AI is aware of those internal sort of logic rules, because if you don't give it that rule or that way of making sure it knows what's the most relevant, it's just going to find sort of the first one that pops up and then use it, and it could be wrong. The other would be making sure that sort of your version history of whatever your document that kind of is surfaced by an AI, I think this is probably more important to systems trained on internal documents, but everybody's got the drafts or the draft V2 final V1, right?

Those kinds of really messy versions of Word documents where people are collaborating and they kind of save all these different versions. Making sure you have the one that's the source of truth and that's agreed upon and all the other kind of draft versions are just hidden from the AI altogether, because it doesn't need to see it unless you're using it for some sort of like, you know, drafting process in the future. But for member-facing stuff, don't surface that kind of information.

[Rob Barnes] (33:24 - 34:00)
And I love that how this touches every kind of business unit in most associations, right? Whether it's advocacy, whether it's education, whether it's membership events, like all of this has these concerns. And so again, one of the reasons why this is probably one of the more important episodes that we're going to record this season.

One more time before you all go, download the High Stakes AI Governance Checklist from the show notes. That's the fastest way to actually get today's episode put to work. Thanks for listening to the Association Intelligence Podcast and join us on the next episode to answer this question, will our members actually use it?

What this helps your team answer

Why treating accuracy as a single "dial" misunderstands the technology — real governance comes from source scope, answer scope, human ownership, and risk tiers defined per use case.
How an AI audit trail can actually surface a "shadow problem" that already exists in most knowledge bases — outdated, contradictory, or unclear content that staff have been misinterpreting for years, AI or not.
Why accountability for accuracy ultimately sits with human systems, not the AI — and why associations, with existing version-control and documentation discipline, are unusually well-positioned to get this right.

Companion Asset

High Stakes AI Governance Checklist

Member Adoption

Will Our Members Actually Use Our AI?

Why "build it and they will come" doesn't work for member-facing AI — a look at the ethical member value loop (an association-safe twist on the hook framework), why the questions members ask matter more than the answers you give, and how adoption data can reveal what your members actually need.

25:05

0:00 25:05

[Thomas Altman] (0:06 - 0:09)
Today's big question, will our members actually use our AI?

[Rob Barnes] (0:10 - 0:34)
We are Thomas Altman and Rob Barnes, and this is Episode 9 of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the Member Adoption Brief. It's linked in the show notes, so grab it now if you want to follow along, or download it after and put it to work.

This episode's question, Thomas, one of the more fun ones, will our members actually use our AI?

[Thomas Altman] (0:35 - 1:37)
And the answer is, no, they're not going to, unless you actually do something about it, right? You have to encourage them and get them to adopt it in a certain way. I love this one.

I had a lot of fun with this episode, honestly, because what we've seen is there's this sort of put it out there, build it and they will come kind of approach to AI. Everyone gets excited about the project, the adoption, they put it out there, and no one shows up, right? Because that's not the way it's going to work.

You have to actually put on your hat of product design specialist or member engagement specialist, right? You need to have those same skills, the same member engagement skills you bring to everything else you do. It needs to come along for the ride for deploying any sort of product, but especially AI.

Because if you just drop a little button on your screen and hope someone uses it, they're not going to. So you have to think a little bit more deeply around how you deploy an AI system so that it gets adoption.

[Rob Barnes] (1:37 - 2:14)
This idea of the stickiness coming from simply a better and faster than their current capability, the current path that they would follow to get something done isn't enough in this context. It might be good for the first couple of uses, but after that, that's not enough to keep it happening. And so what I love about what you and our AI hosts are exploring in this is this concept of the ethical member value loop.

And that's actually, its foundation comes from a really interesting methodology, right?

[Thomas Altman] (2:14 - 5:25)
Absolutely. So this episode was really fun for me because I got to put on my product design hat, which I don't often get to do. And it comes from this idea of the hook framework.

So many of you out there may be familiar with the hook framework, which is controversial to be quite honest. So I like the way that the AI took this into the ethical member value loop and kind of make some changes to it. So setting the table for those of you that are not familiar, the hook framework basically has four steps and it is a loop, right?

The first one is a trigger. So something outside of your product, your experience that happens that makes someone think I need to go to the product, right? So one of the things we see a lot of times actually within Betty is people get the best adoption when you name your agent, your assistant, something that's also a verb, right?

Instead of I'm going to Google that, I'm going to Betty that, right? Or I'm going to write, if you have that intuition, like, ooh, I have a knowledge need. My first thought is to go to the association site and use their knowledge assistant, right?

So how do you develop that in a way that is going to be really, really useful? How do you kind of invest in that process? So the rest of this loop in the hook framework, at least, is you have some action that leads to a, in the hook framework, a variable reward, right?

And unfortunately, that is based off of studies of gambling addiction. So you can imagine the idea of, like, you're at a slot machine, you kind of put your quarter in, you press a button or pull the lever, whatever you do at a slot machine, and you don't know what happens next, right? Probably you're going to lose your money, but maybe you'll kind of, like, the bells will light up and you win 75 cents, or maybe every once in a while you get, like, a big hundred dollar payout or something like that, right?

And there have been studies showing that even if it wasn't probably you're going to lose, but if you've, like, won a little bit of money every time, it's more addicting, which we're not trying to cause addiction here, I want to make that clear, but it's more engaging if the reward is variable, if it's not predictable in advance, if you don't know where that's going to happen. So think about uploading a photo to Instagram, or for us elder millennials and older out there, you know, putting a post on Facebook, right? The variable reward with that, and Facebook adopted this strategy pretty deeply, is how many likes did you get, right?

You don't know. You don't know when you post exactly how many there are going to be, and then what has happened once you've kind of posted the photo or you've dropped the comment, you now have the last part of that framework is investment. All your pictures are on Instagram now, right?

And you're not going to delete Instagram because you're going to lose all your pictures. So what I like that this episode does, and I'll let the AI host kind of talk about it more in depth, is apply that, take it out of gambling addiction and apply it into member engagement and value creation, and how you use that in an ethical way using AI within the association context, and I just am thrilled that this kind of took this turn.

[Rob Barnes] (5:25 - 6:20)
Yeah, this is an important part of establishing the engagement in members actually using the system that you produce, that you publish. The predictability is in knowing that the responses are granted, you know, and if you go back to our previous episode and we talk about the governance, you know, applications of this, and that's where they should be predicting. However, it's the actual use of, I don't know how she's going to respond.

I know that it's going to be the right response, but it's going to be really interesting. What questions get posed back to me? You know, so I love the idea of, you know, if I'm a member of the International Society of Automation, I'm going to memo that.

I'm going to memo that, and there is an opportunity there to kind of say, well, memo could be helpful in ways I didn't even expect today, and that's going to help me keep going back. That's where this is going, right?

[Thomas Altman] (6:21 - 7:52)
That's the idea. And the other way that I think, yes, you're right, unpredictability, kind of going back to the previous episode, actually, is super important, that we keep predictability in that it responds with sourced content, right? And it's going to use the right content every time.

The sort of engaging aspect that we find is when you have something surprising that comes along for the ride, if it kind of describes it in a way that helps you see it a little bit better, right? It actually helps you understand some content in a way you would not have otherwise. Or maybe it brings up a surprising document, like you ask a question, and you didn't expect, because maybe the full, you know, webinar wasn't about the topic, right, and you would never have known to click on this webinar and watch the whole thing, because, you know, for a five-minute chunk, they talk about the things that's really relevant to you, even though that's not the purpose of the whole webinar, so not in the description of it, not in the kind of title of it, but just someone drops a nugget of information that you would never have found otherwise, right?

That's the kind of thing that, like, brings this level of delight and surprise to the process, and people get invested in coming back to it in that way. And that adoption curve, when people sort of lean into having your AI agent present information in a novel way, in a way that feels new and exciting, or if it can bring in information that they didn't expect to be relevant, but actually super is, those types of things really drive engagement and keep people coming back.

[Rob Barnes] (7:54 - 8:13)
So before we hand this over to our AI podcast hosts, quick reminder that the following is generated using Notebook LM, with heavy involvement from the Betty team to ensure that the content generated is valuable. For a deeper dive on our approach, please go back and listen to episode zero. And now, over to the AI hosts.

[AI Host 1] (8:19 - 8:28)
Imagine authorizing, like, a $250,000 wire transfer for this state-of-the-art member portal. Right. You endure months of development, right?

You finally cut the digital ribbon.

[AI Host 2] (8:28 - 8:31)
Oh, yeah, you send out the massive email blast.

[AI Host 1] (8:31 - 8:54)
Exactly, the huge email blast, and then 90 days later, zero logins. Just nothing. A total ghost town.

A complete ghost town. And that scenario is exactly why association leaders are just staring at the ceiling at two in the morning, asking themselves the exact six words that can freeze, you know, any major tech project in its tracks. Yeah.

Will our members actually use it?

[AI Host 2] (8:54 - 8:57)
It's the ultimate reality check for any organization.

[AI Host 1] (8:57 - 9:07)
It really is, which is why we're doing a deep dive into this today. We're looking at a stack of implementation reports and case studies from, I mean, over 200 association AI rollouts.

[AI Host 2] (9:07 - 9:17)
Right. And the data there is just glaringly obvious. You can have the most sophisticated technology roadmap on the planet, but if nobody clicks, none of the underlying engineering actually matters.

[AI Host 1] (9:17 - 9:33)
Yeah, none of it. So let's validate that skepticism right up front for everyone listening. Associations have a really long history of launching these proprietary apps or exclusive communities and massive content hubs that just ultimately get ignored.

[AI Host 2] (9:33 - 9:42)
Oh, absolutely. It's completely valid to worry that an AI assistant might just be like the next shiny, unused toy.

[AI Host 1] (9:42 - 9:43)
Gathering digital dust, basically.

[AI Host 2] (9:44 - 9:50)
But when we look at the patterns across these 200 implementations, it suggests we're actually framing the problem incorrectly.

[AI Host 1] (9:51 - 9:51)
How so?

[AI Host 2] (9:51 - 10:00)
Well, the goal isn't to get members to like AI. I mean, nobody wakes up, pours a cup of coffee and thinks, gee, I really hope I get to engage with a large language model today.

[AI Host 1] (10:00 - 10:01)
No, definitely not.

[AI Host 2] (10:01 - 10:11)
The real mission, the only thing that actually dictates adoption is whether an assistant can solve a high-frequency, high-value job better and faster than a member's current path.

[AI Host 1] (10:11 - 10:22)
Okay, let's unpack this, because if we know that members only use what actually helps them get a job done, why do associations keep building tech that gets ignored? Like, where is the initial breakdown happening?

[AI Host 2] (10:22 - 10:36)
It almost always happens in the user interface. Associations fall in love with the raw capability of the tech and just forget the human using it. The biggest mistake is launching a generic, floating, ask-us-anything text box.

[AI Host 1] (10:37 - 10:40)
Oh, like when they slap a little widget in the bottom right corner of the home page?

[AI Host 2] (10:40 - 10:43)
Yeah, and they just hope members will magically figure out what to ask.

[AI Host 1] (10:43 - 10:48)
But wait, if we want them to use it, shouldn't we be looking at consumer apps?

[AI Host 2] (10:48 - 10:48)
Yeah.

[AI Host 1] (10:48 - 10:52)
Like Google built a trillion-dollar empire on a blank text box.

[AI Host 2] (10:53 - 10:53)
That's true.

[AI Host 1] (10:53 - 10:59)
How do we build adoption without falling into, I don't know, creepy, manipulative growth-hacking tricks?

[AI Host 2] (11:00 - 11:08)
Well, the Google comparison is interesting, but it fails because of cognitive load. You've been trained for two decades on how to format a Google search, right?

[AI Host 1] (11:08 - 11:09)
Yeah, I know exactly what keywords to use.

[AI Host 2] (11:10 - 11:24)
Exactly. But when a member logs into a specialized medical or engineering association and sees a blank box, their brain has to do massive work. They wonder, is this a legal tool?

A customer service bot? Is a human reading this?

[AI Host 1] (11:25 - 11:28)
Oh, wow. So that friction just causes them to abandon the tool immediately.

[AI Host 2] (11:28 - 11:40)
Right. Earning adoption isn't magic. It's designed.

And it doesn't use consumer app manipulation. It relies on what the reports call the ethical member value loop.

[AI Host 1] (11:40 - 11:46)
Okay, I want to pause real quick because I want to mention a key resource for you listening. We have the Member Adoption Proof Brief linked in the show notes.

[AI Host 2] (11:46 - 11:48)
It's a fantastic resource.

[AI Host 1] (11:48 - 11:58)
It really is. It helps map out this exact use case and launch motion for your own organization. So, okay, walk us through this ethical member value loop.

What's the first step?

[AI Host 2] (11:58 - 12:07)
It starts with the trigger. And this is a real member need, like a deadline, a compliance issue, or a work task that arises organically.

[AI Host 1] (12:07 - 12:12)
Like a nurse panicking at midnight because they need a specific continuing education credit.

[AI Host 2] (12:12 - 12:22)
Exactly. That sheer panic is the trigger. Then comes the action.

Asking has to be incredibly easy because the assistant is placed exactly where the need already happens.

[AI Host 1] (12:22 - 12:24)
So it's right there on their compliance dashboard.

[AI Host 2] (12:24 - 12:29)
Right, not hidden in some separate AI portal. Then we hit the value and reward stage.

[AI Host 1] (12:29 - 12:31)
Which is where they get the answer, right?

[AI Host 2] (12:31 - 12:39)
Yes, but it has to be a trusted, source-grounded answer with a useful next step. Not some hallucinated guess from the open internet.

[AI Host 1] (12:39 - 12:42)
Yeah, if it makes something up, the trust is just gone forever.

[AI Host 2] (12:42 - 12:51)
Instantly gone. Once they get that true value, it leads to the investment phase. The member takes action, they ask a follow-up, or maybe they register for an event.

[AI Host 1] (12:51 - 12:53)
And that naturally leads to the final step, the return.

[AI Host 2] (12:53 - 13:01)
Exactly. Because true value is delivered, the association becomes their first stop the next time a question arises. That's the loop.

[AI Host 1] (13:01 - 13:08)
So if that's the loop, how do we actually get members to take that very first spin and trust the answers?

[AI Host 2] (13:09 - 13:16)
Well, first, use value usually starts simple. Members usually begin with basic search-like questions like, when is the annual conference?

[AI Host 1] (13:16 - 13:25)
Here's where it gets really interesting, though. Because think of Google as this giant, noisy public library, right? Where you have to verify everything yourself.

[AI Host 2] (13:25 - 13:26)
Oh, yeah, you're doing all the heavy lifting.

[AI Host 1] (13:26 - 13:36)
Right. Whereas in an association, AI is, it's like having a trusted mentor who has read literally every verified credential and journal in your specific industry.

[AI Host 2] (13:37 - 13:49)
That's a perfect analogy. Earning the first stop capturing that members' minds share happens when an association can answer those domain-specific questions better and faster than generic AI or Google.

[AI Host 1] (13:49 - 13:51)
Because it's pulling from vetted experts.

[AI Host 2] (13:51 - 14:00)
Exactly. And the real hook happens when follow-up prompts reveal deeper context. It points them toward adjacent resources they didn't even know existed.

[AI Host 1] (14:00 - 14:12)
But not all users are at the same stage of their journey. I mean, how do we structure this trusted mentor so it provides value to, like, a brand-new prospect versus a 20-year veteran member?

[AI Host 2] (14:12 - 14:17)
The best implementations use tiers of access. So you have a public tier and a member-only tier.

[AI Host 1] (14:17 - 14:18)
How does the public tier work?

[AI Host 2] (14:18 - 14:26)
It acts as a discovery surface. It's essentially a preview of the value the association offers. It proves its worth before asking a prospect to join and pay dues.

[AI Host 1] (14:26 - 14:28)
Oh, that makes total sense. And then the member tier.

[AI Host 2] (14:28 - 14:32)
The member-only tier becomes this deep, always-on knowledge access point.

[AI Host 1] (14:32 - 14:34)
Which honestly seems perfect for filling the event gap.

[AI Host 2] (14:34 - 14:35)
The event gap.

[AI Host 1] (14:35 - 14:46)
Yeah. I mean, think about conference and webinar knowledge. Usually, after a major annual event, all that brilliant knowledge just vanishes into a PDF graveyard on some forgotten webpage.

[AI Host 2] (14:46 - 14:50)
Oh, the PDF graveyard is so real. It's a huge waste of capital.

[AI Host 1] (14:50 - 14:54)
Right. But an assistant makes this knowledge evergreen and findable year-round.

[AI Host 2] (14:55 - 14:59)
Exactly. And this brings us to what's often called the Betty implementation pattern.

[AI Host 1] (15:00 - 15:00)
Betty.

[AI Host 2] (15:00 - 15:10)
Yeah. Betty is just an industry shorthand for a very practical approach. It's a governed, source-bound assistant that is coached by staff and subject matter experts.

[AI Host 1] (15:10 - 15:12)
Meaning it's not just making things up.

[AI Host 2] (15:12 - 15:20)
Right. It doesn't hallucinate outside of approved content. It traces every single answer back to the source, making your existing content much easier to use.

[AI Host 1] (15:20 - 15:28)
But wait. Let me push back on that. If it's giving away all these perfect answers right in the chat, do members just stop navigating our site?

Like, does traffic plummet?

[AI Host 2] (15:28 - 15:33)
That's a common fear, but no. A good assistant actually guides the member to the next step.

[AI Host 1] (15:33 - 15:34)
Like a concierge.

[AI Host 2] (15:35 - 15:45)
Exactly. It might give a quick answer, but then point them to full education or certification pathways. It doesn't dead-end the conversation.

It routes them deeper into your site.

[AI Host 1] (15:45 - 15:47)
Okay. This all sounds like a perfectly engineered system.

[AI Host 2] (15:48 - 15:48)
Yeah.

[AI Host 1] (15:48 - 15:51)
But we need to inject a dose of reality here.

[AI Host 2] (15:51 - 15:51)
Yeah, we do.

[AI Host 1] (15:51 - 15:54)
What breaks this system? Like, when it goes wrong, what happens?

[AI Host 2] (15:54 - 16:02)
If we connect this to the bigger picture, the honest truth is there is no guaranteed viral engagement or universal adoption. It's not magic.

[AI Host 1] (16:03 - 16:04)
Right. You can't just flip a switch.

[AI Host 2] (16:04 - 16:12)
No. If an association launches an assistant trained on weak, outdated content, or they hide it three clicks deep in a website footer…

[AI Host 1] (16:12 - 16:14)
Where nobody naturally goes anyway.

[AI Host 2] (16:14 - 16:22)
Exactly. Or if they fail to define the audience or just never promote it, the tool is going to fail. Undercutting adoption is incredibly easy.

[AI Host 1] (16:22 - 16:30)
It's like putting a brilliant librarian in a library full of blank books. I mean, the AI can only activate the knowledge you actually possess.

[AI Host 2] (16:30 - 16:40)
That's spot on. AI cannot fix bad content. Member trust is earned answer by answer.

And a hidden tool with bad, outdated sources destroys that trust instantly.

[AI Host 1] (16:40 - 16:54)
Wow. So, what does this all mean? If you're listening right now and hearing yourself in that initial question, will they use it?

Your job is to separate what the AI handles from what your content and operations teams still own.

[AI Host 2] (16:54 - 17:00)
Absolutely. Your teams own the quality of the underlying documents, the placement, and the launch strategy.

[AI Host 1] (17:01 - 17:07)
You can't outsource that, which is why you really need to look at the show notes right now and open the Member Adoption Proof Brief.

[AI Host 2] (17:08 - 17:10)
It's the best practical next step you can take.

[AI Host 1] (17:10 - 17:19)
It really is. It helps you map the member value loop, make the assistant visible at the exact moment of need, and measure real engagement actions instead of vanity metrics.

[AI Host 2] (17:19 - 17:21)
And, you know, this raises an important question.

[AI Host 1] (17:21 - 17:22)
Oh, what's that?

[AI Host 2] (17:22 - 17:34)
Well, we spent this entire deep dive talking about how to give members the right answers right. Great. But through AI Insights, what if the most valuable asset you actually gain isn't the answers you provide?

[AI Host 1] (17:35 - 17:37)
Wait, really? Then what is it?

[AI Host 2] (17:37 - 17:43)
What if it's finally seeing the exact unvarnished questions your members are asking when they think nobody is watching?

[AI Host 1] (17:43 - 17:53)
Oh, wow. Just seeing their actual pain points in real time. That is a brilliant final thought.

Definitely something for everyone to mull over. Thanks for diving into this with me.

[AI Host 2] (17:53 - 17:55)
Anytime. It was a great conversation.

[Rob Barnes] (18:02 - 20:14)
So recapping today, the big question was, will our members actually use our AI? And one of the things that came to my mind in listening to this episode again, Thomas, was experiences that we have with existing customers at Betty. When we do our check-ins and we're looking at our growth strategies for our customers, if we hear feedback that's not getting used like we thought it would get used, the usage isn't as high as we wanted.

We like to do a little bit of digging into that. And one of the things that we realized is that the AI itself is a member value opportunity. It's not visible.

How do our members even find it? Number one. So we look at the deployment.

Where does it live? How do members get access to it? Is it only behind the login?

These are the sorts of things that create some friction which can cause challenges with usage. The other part, though, is if the association itself designed the AI with a particular use case in mind and the members don't find that useful, they're less likely to come back and use it. And oftentimes there is that misalignment.

And if the association itself, the team that are responsible for the AI, aren't listening to the member needs or meeting the members where they are because they set this up to do a particular job that members don't find useful and they don't pivot from that or they don't adapt to accommodate where the members' knowledge needs are, then of course that's going to create friction again. And so as a listening tool, it's not just about the content itself and the performance of the AI, but is it actually doing the job that members find valuable today? And that is, I guess, a challenge to all of our listeners is to think about it from that perspective.

But it's also, there's an opportunity to get some insights, Thomas, from what these AIs can tell you about where your users are that I don't think is being leveraged often enough. Would you agree?

[Thomas Altman] (20:15 - 21:50)
Absolutely. And I think actually this ties a little bit back to, I think it was episode one, where we kind of go in to, are you ready for AI? And the way that you determine is if we have these use cases well defined in advance.

So I think it plays really nicely with that concept because once you do have those use cases defined, you can then first judge, is it assisting with accommodating those use cases? Like, are you moving the needle on some metric of engagement? And two, if not, is it either because the system isn't accomplishing that, it's not actually helping the use case that you did, or was it the wrong use case?

You now have insight into what matters for your members through the adoption of your AI, if you deploy it in a smart way. So being able to dive into it and say, oh, people are wanting to use it in this other way. And fun fact, when our first internal deployments of Betty, we learned a ton because we set it up to do one thing, and people just kept asking questions it wasn't designed to answer.

And my initial feeling was like, well, don't ask it that. Ask it the thing that I built it for. But that was clearly not the right call.

Like, what we actually had to do is say, okay, it's now out in the hands of people that are not me, and they're using it this other way, and everyone keeps doing the same thing. We've got to build that. We've got to pivot over to make sure that it's doing what it needs to do and what people expect it to do.

[Rob Barnes] (21:50 - 22:34)
And a good system built specifically for associations, but a good system, it's very easy to adapt. Like, that's the point, is that either by adding additional content so that the AI is trained to do that job and use this knowledge, that's very simple to adapt and immediately get value. In some cases, show that you're listening to the knowledge needs of your users, your members, your staff by doing that very, very quickly.

I really like the closing thought of this episode around, it's not so much about the answers that are being given, but the questions that members ask when they think that nobody's watching. That's actually, those kinds of insights have proven to be incredibly valuable to associations.

[Thomas Altman] (22:35 - 23:15)
A million percent. When people have these multi-turn conversations, if someone has 20 back and forth with your version of a knowledge assistant, that pathway is totally surprising. And you learn a lot by not, yeah, the answers are good, right?

But the questions that your members are asking are gold. And being able to, like, dig in and find those and kind of sit with that so that you can then build the next level of member engagement, I think, is an untapped resource or a lightly tapped resource that most associations don't realize they have when they adopt a knowledge assistant system.

[Rob Barnes] (23:16 - 24:57)
The principle here that I've stuck to in talking to the value of generative AI for associations has been, you know, the pre-AI 21st century engagement has been around personalization. And personalization is cohort based, right? We're still bucketing groups of people into a certain bucket for a purpose of communicating with them.

Generative AI has given us the ability to do things at a personal level. That path you just described, you know, and the average 11 minute conversation that members of Marine Retailers Association of the Americas has with Amy, those paths are unique every single time. Not one of those lines of inquiry is exactly the same.

It's the personal knowledge need of that person at that point in time. The ability for a generative AI to yield those insights and then form, you know, recommendations for content strategy or knowledge management strategy or strategic objectives around member engagement is incredibly powerful and incredibly underutilized. So the shift from personalization to personal is such an important part of this.

And I'm glad that our AI hosts picked that out of all of the content today. One more time before we go today, let's download the member adoption brief from the show notes. That's the fastest way to actually put today's episode at work.

Thanks for listening to the Association Intelligence Podcast. Join us for the next episode to answer this question. How do we justify the cost?

What this helps your team answer

Why AI adoption requires the same product-design and member-engagement thinking as any other tool — dropping a button on the screen and hoping isn't a strategy.
How the "ethical member value loop" (trigger, action, variable reward, investment) applies engagement principles responsibly — predictable, sourced answers paired with surprising, delightful discoveries members didn't know to look for.
Why the real hidden value of AI adoption isn't the answers it gives — it's the unfiltered, real-time questions members ask, which reveal pain points and content gaps most associations never see otherwise.

Companion Asset

Member Adoption Brief

Business Case

How Do We Justify the Cost of AI?

How to build an AI business case your board and CFO will actually back, using four value ledgers (capacity, member value, strategic intelligence, and conversion) so you can pick one, track a few concrete metrics, and show measurable value instead of buying on FOMO.

33:12

0:00 33:12

[Thomas Altman] (0:11 - 0:14)
Today's big question, how do we justify the cost of AI?

[Rob Barnes] (0:14 - 1:33)
We are Thomas Altman and Rob Barnes, and this is episode 10 of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the AI Business Case Worksheet. It's linked in the show notes, so grab it now if you want to follow along, or download it after and put it to work.

Thomas, this is a really important one. I think this time of year, also, simply because for a lot of associations, they're doing budgets, you know, budgets for 2027, and as they start to think about what it takes to respond to the board kind of edict that says, we need to be doing AI, that broad brush, we need to be doing AI, but without really understanding what that means in the context of that association. This episode digs into a lot of how, what questions should be getting asked, and how do we lead that conversation within, you know, the internal framework leading up to a board meeting, trying to understand if you're going to make this kind of an investment.

And it can't just be because AI is the future, or competitors are doing it, or these other anxieties that are kind of exist. We really start to dig in now, like, what is it that creates a business case that's useful to an association?

[Thomas Altman] (1:35 - 2:51)
Yeah, I think exactly. So the danger in adopting AI for one of those other reasons, right, because everyone else is doing it, or because it's the future, is that it's too open, and AI projects can not have a clear end date, or they can run over cost, right, if you're not careful about how you're doing it. And what I really like about this episode is that it gives sort of concrete and tactical approach to actually deciding what is value, or how, even better, right, like, you're not a bystander here, like, how can you as an organization intentionally create value from your investment?

And what does it take to have planned out in advance to make that a reality on your own, right? And I think that's something that we have seen a lot at Betty as well, right, like, you don't have to let AI happen, you don't just adopt it, you can be thoughtful about the way that you implement it, so that you're methodically moving towards capturing value, even turning some of these AI investments into revenue generating opportunities, right? And at that point, this kind of wild eyed CFO, who's always, you know, putting a red line on everything, is going to be very happy and going to encourage you to keep up your AI adoption instead of saying, no, that we lost X number of dollars based on it.

[Rob Barnes] (2:51 - 4:09)
A couple of things that I've noticed that were, that came up in this episode, as well, which are pretty common to most of the conversations that I still have with association execs, is this, there's a tendency to lean towards internal productivity metrics as being the first place to look. And I wonder, most of the time, I wonder if that's just because that's where they're doing the most intentional measurement of what that looks like. But still, very few associations do measure capacity internally, you know, human capacity internally in a way that it would be easy to say, if we adopted AI, we would see this amount of improvement in whatever it might be.

That regard doesn't, isn't typical in an association. There are certainly some that do that, but not a lot. So I like this shift towards, you know, the data that AI interaction, that AI capability starts to produce a data set, which can lead into things like you discussed, like revenue opportunities.

But this strategic intelligence, putting a value on that is still a little difficult for a CFO to understand what that value is. But there is genuine value in there because the impact it can have. We've seen it happen.

Right?

[Thomas Altman] (4:10 - 5:52)
Yeah, so the thing that I like about this episode is that it, it kind of breaks it down into these, you'll hear in a moment, these four ledgers, right? And this idea that there are different methods to kind of extracting more and more value from your investment in AI. I'll let our AI host talk about it more in depth, but in general, the idea of starting with that idea of capacity, because that's gonna be the lowest hanging fruit, right?

So capacity in this sense, just think of the, you know, hundreds, if not thousands of emails you get a month asking the same question over and over again. And you need a human being to go and respond to that same question over and over again. You've got an FAQ, you've got, you know, other things on there that they could find the answer themselves.

People never use it. Can adopting an AI strategy reduce that number significantly? We've seen that, yes, that's low hanging fruit.

And the idea here is how do you construct a system so that it evolves into more and more value over time? So the entry point is that, right? But then once someone's there, they keep coming back to that first, and then they discover that it can talk a little bit more, it can go deeper into kind of governed AI, right?

It can actually have longer conversations. So you extend the amount of time that people are participating in those conversations and amplifying the value they get out of that content that you already have, so that then they're generating more conversational history data, right? And that conversational history data can then be decoupled from the actual conversation and then turned into new levers of value going forward.

And so that whole process, I think, is really smart because it's tactical at each step. How do you evolve from one to the other? And I think it's something that would resonate with a CEO because you're starting with something easy and tactical, and then you're kind of evolving it into the bigger picture.

[Rob Barnes] (5:53 - 6:40)
Yeah, the reframe about human leverage, not staff reduction, is music to our ears, and I'm sure it's music to everybody's ears in the association industry, and there is genuine evidence out there now where AI is having that kind of an impact and allowing humans to do the human work at levels that they were just simply not able to prior to AI. So before we hand over to our AI podcast hosts, a quick reminder that the following episode was generated using Notebook LM, with heavy involvement from us, the Betty team, to ensure that the content generated is valuable and accurate. For a deeper dive on our approach, please go back and listen to episode zero.

And now, over to the AI hosts.

[AI Host 1] (6:46 - 6:47)
How do we justify the cost?

[AI Host 2] (6:48 - 6:48)
Right.

[AI Host 1] (6:48 - 7:05)
That is the exact question, isn't it? I mean, if you are listening to this right now and you're an association leader, maybe a CEO, COO, or you're running membership in education, you know that the second you bring an AI evaluation to your board or to your CFO, that is the very first thing they are going to ask you.

[AI Host 2] (7:05 - 7:15)
Oh, absolutely. And well, they are entirely right to ask it. They have a fiduciary duty to ask what this investment actually produces for the organization out in the real world.

[AI Host 1] (7:15 - 7:15)
Yeah, they do.

[AI Host 2] (7:15 - 7:26)
It is the foundational question of any healthy leadership team, and we are witnessing a massive and frankly a very necessary market shift across associations right now.

[AI Host 1] (7:26 - 7:27)
Yeah, I'd say so.

[AI Host 2] (7:27 - 7:34)
Because for the last couple of years, the atmosphere was just dominated by this frantic, FOMO-driven panic.

[AI Host 1] (7:34 - 7:42)
Right. Everyone was just saying, we need AI right now. The landscape is changing.

Let's just buy something and we will figure out the business case later.

[AI Host 2] (7:42 - 7:56)
Exactly. But the industry has sobered up. You know, leaders have matured in their approach.

The stance we see from the most successful associations today is, we need AI that creates measurable value or we just don't buy it.

[AI Host 1] (7:56 - 8:07)
So the mission for our deep dive today is to really cut through that lingering hype. We are going to build a financially sober, rock-solid business case.

[AI Host 2] (8:07 - 8:10)
I love that. Sober is the perfect word for it.

[AI Host 1] (8:10 - 8:22)
Right. And to do that, we are drawing on patterns and insights from nearly 200 association AI implementations. We're looking at what the smart leaders do before they ever even sign a contract.

[AI Host 2] (8:22 - 8:26)
And part of getting that right is knowing which arguments to immediately throw in the trash.

[AI Host 1] (8:26 - 8:27)
Hey, for sure.

[AI Host 2] (8:27 - 8:32)
Like if you walk into a boardroom to ask for a budget and your primary justification is, well, AI is the future.

[AI Host 1] (8:32 - 8:34)
Your CFO should literally send you packing.

[AI Host 2] (8:34 - 8:42)
They should. The same goes for, oh, our competitors are doing it. The most dangerous assumption of all, the technology will just automatically pay for itself.

[AI Host 1] (8:42 - 8:44)
Oh, yeah. You hear that one a lot.

[AI Host 2] (8:44 - 8:56)
You do. But those are not business cases. Those are anxieties disguised as strategy.

They don't give a finance team any actual metrics to track or any outcomes to hold the leadership accountable to.

[AI Host 1] (8:56 - 9:00)
And we're looking at a situation where the stakes are incredibly high in both directions.

[AI Host 2] (9:00 - 9:06)
Yeah, the balance is really precarious. I mean, refusing to adopt AI carries severe strategic risk.

[AI Host 1] (9:06 - 9:07)
Because your members are expecting it.

[AI Host 2] (9:07 - 9:22)
Exactly. Your members and your own staff, honestly, are increasingly expecting frictionless access to information. Right.

If they can't get fast, accurate knowledge from your association, they will simply go to generic search engines or external platforms.

[AI Host 1] (9:22 - 9:23)
Or competitor networks.

[AI Host 2] (9:23 - 9:37)
Exactly. You just lose that mindshare completely. But on the flip side, adopting AI poorly is arguably worse.

How so? Well, if you buy a system without a clear plan for how it actually integrates into your operations, you have just created a brand new cost center.

[AI Host 1] (9:38 - 9:38)
Yeah, that makes sense.

[AI Host 2] (9:38 - 9:45)
You get confusion, staff anxiety, terrible usage rates, and absolutely zero measurable return on the investment.

[AI Host 1] (9:46 - 10:04)
Okay, let's unpack this. Because if the cost justification is the major hurdle here, we have to rethink the whole premise of why we're buying the software in the first place. Right.

This fundamentally has to shift from a technology enthusiasm question to a, well, a true business case question.

[AI Host 2] (10:04 - 10:18)
And that is the dividing line right there between associations that succeed with AI and those that fail. You are not buying technology just to be modern. You are hiring a system to do a specific job.

[AI Host 1] (10:18 - 10:18)
Yeah.

[AI Host 2] (10:18 - 10:25)
A leadership team has to sit down and clearly define the exact value that AI is supposed to create before they ever scale a pilot.

[AI Host 1] (10:25 - 10:36)
I always think of it like this. Buying AI simply because it's the future feels a lot like buying a massive state-of-the-art commercial-grade kitchen for a restaurant without actually writing a menu first.

[AI Host 2] (10:36 - 10:37)
Oh, that's a great analogy.

[AI Host 1] (10:37 - 10:44)
Right. You can go out and buy the six-burner Wolf range, the best convection ovens, the sharpest Japanese knives. But what are you actually serving your guests?

[AI Host 2] (10:44 - 10:47)
Exactly. Are you a bakery? Are you a steakhouse?

[AI Host 1] (10:47 - 10:54)
Yeah. If you haven't decided on the menu, that kitchen isn't a strategic asset. It's just a very expensive, very shiny room.

[AI Host 2] (10:54 - 10:54)
So true.

[AI Host 1] (10:55 - 11:11)
And to help you write that menu, we actually have an essential tool for you. We've linked the AI Business Case Worksheet right in the show notes of this deep dive. It is designed specifically to help your internal teams structure these exact conversations before you ask for a single dollar.

[AI Host 2] (11:11 - 11:23)
And using a framework like that worksheet is vital because writing that menu requires you to choose your primary justification from four distinct areas of value. We actually call them the four ledgers.

[AI Host 1] (11:23 - 11:24)
Okay. Four ledgers. What are they?

[AI Host 2] (11:24 - 11:36)
Well, we see these emerge in a very specific sequence across implementations. The sequence flows from internal operations to external member experience, then to the data generated by that experience, and finally to revenue.

[AI Host 1] (11:36 - 11:46)
Okay. So let's start inside the house then. Where do we find that initial baseline value that a COO or a CFO can immediately understand?

[AI Host 2] (11:46 - 11:47)
The starting point is capacity.

[AI Host 1] (11:48 - 11:48)
Capacity.

[AI Host 2] (11:49 - 11:59)
Right. This is the operational low-hanging fruit. When we talk about capacity, we are talking about the daily grinding reality of association staff.

[AI Host 1] (11:59 - 12:00)
Oh, the endless emails.

[AI Host 2] (12:01 - 12:10)
Exactly. It is the deflection of those endlessly repetitive questions that just clog up your inboxes. The policy questions, the event logistics, the basic credentialing rules.

[AI Host 1] (12:10 - 12:11)
Right, right.

[AI Host 2] (12:11 - 12:20)
The goal here is reclaiming the time of your subject matter experts, your SMEs, and significantly reducing the day-to-day support load.

[AI Host 1] (12:20 - 12:22)
Okay, hold on. I need to stop you there and push back a little bit.

[AI Host 2] (12:22 - 12:23)
Okay, sure.

[AI Host 1] (12:23 - 12:33)
Because the moment you say capacity and reducing support load in an executive meeting, the immediate unspoken translation in everyone's head is staff replacement.

[AI Host 2] (12:33 - 12:34)
Ah, yeah.

[AI Host 1] (12:34 - 12:39)
Like a board hears that and thinks we are building a business case to fire three support reps just to save money.

[AI Host 2] (12:39 - 12:44)
And that is a terrible, short-sighted way to build an AI business case. It is also just fundamentally incorrect.

[AI Host 1] (12:44 - 12:45)
Okay, why?

[AI Host 2] (12:45 - 12:50)
Because capacity is not a staff reduction story. It is a human leverage story.

[AI Host 1] (12:50 - 12:51)
Human leverage. I like that.

[AI Host 2] (12:52 - 13:03)
Right. Consider what a brilliant, highly educated SME on your staff is actually doing when they answer the same fundamental policy question for the 50th time in a single month.

[AI Host 1] (13:03 - 13:04)
Wasting their talent, basically.

[AI Host 2] (13:04 - 13:10)
Exactly. They are performing repetitive access work. They are essentially acting as a human search engine.

[AI Host 1] (13:10 - 13:11)
Yeah, that's a good way to put it.

[AI Host 2] (13:11 - 13:20)
When you offload that repetitive access work to an AI system, you are not eliminating the human. You are liberating them to do high-value human tasks.

[AI Host 1] (13:21 - 13:24)
But wait, I want to clarify something mechanical before we go further on that.

[AI Host 2] (13:24 - 13:24)
Sure.

[AI Host 1] (13:25 - 13:39)
Because for an executive listening to this, how is this different from a, remember just going to chat GPT? You're talking about offloading work. But general AI makes things up.

And if it makes up a credentialing rule, that's a massive liability, not leverage.

[AI Host 2] (13:39 - 13:43)
That is the critical distinction between public AI and what we call governed AI.

[AI Host 1] (13:43 - 13:44)
Governed AI.

[AI Host 2] (13:44 - 13:57)
Yes. Governed AI operates inside a walled garden. It is entirely source bound.

Mechanically, that means the AI is instructed to only read, synthesize, and cite the specific trusted documents you actually give it.

[AI Host 1] (13:58 - 14:02)
So like your PDFs, your past webinars, official policy manuals, that sort of thing.

[AI Host 2] (14:02 - 14:15)
Exactly. It does not pull from the open internet. It doesn't hallucinate answers based on some random Reddit threads.

If the answer isn't in your official materials, a governed system will just tell the user it doesn't know. Rather than making a guess.

[AI Host 1] (14:16 - 14:18)
Okay. That makes total sense. It's a closed loop system.

[AI Host 2] (14:18 - 14:23)
And that traceability is what allows you to safely offload the support work in the first place.

[AI Host 1] (14:23 - 14:30)
Right, right. So back to the human leverage thing, when you give the staff that time back, what do they actually do with it?

[AI Host 2] (14:30 - 14:33)
They get to be strategists instead of librarians. Wow.

[AI Host 1] (14:33 - 14:33)
Yeah.

[AI Host 2] (14:33 - 14:47)
They spend that reclaimed time interpreting complex, nuanced member needs that the AI simply can't handle. They spend time improving the core content, coaching the AI system to be more accurate, designing completely new programming.

[AI Host 1] (14:47 - 14:49)
Do we have an example of this working?

[AI Host 2] (14:49 - 14:58)
We do. We see tangible proof points of this value path. The Society of Women Engineers, or SWE, they actually saw a 19% reduction in their support load.

[AI Host 1] (14:59 - 15:06)
Wow. And we need to look at the math of that 19% reduction from a CFO's perspective, right? Because it's not just a vanity metric.

[AI Host 2] (15:06 - 15:07)
No, not at all.

[AI Host 1] (15:07 - 15:31)
If you calculate the hours saved by that 19% drop, and you multiply it by the blended hourly rate of those specific staff members, you are presenting hard annualized capacity value. It is a proof point of leverage. It's an example of what's possible, showing they carved out nearly a fifth of their support capacity for humans to do actual human work.

[AI Host 2] (15:31 - 15:33)
It gives your people a space to actually be brilliant.

[AI Host 1] (15:33 - 15:43)
I love that. Okay, so we solved the internal bleed. Staff are no longer drowning in policy questions.

But a CFO is going to argue that saving staff time doesn't necessarily drive growth or retention.

[AI Host 2] (15:44 - 15:44)
Right.

[AI Host 1] (15:44 - 15:48)
How do we prove this investment matters to the people actually paying the dues?

[AI Host 2] (15:49 - 15:54)
That is when you transition to the external experience, which is the second ledger member value and engagement.

[AI Host 1] (15:54 - 15:55)
Okay.

[AI Host 2] (15:55 - 16:00)
And the bar here has to be set much higher than simply providing better search.

[AI Host 1] (16:00 - 16:02)
Yeah, better search is just table stakes at this point.

[AI Host 2] (16:02 - 16:07)
Yeah, exactly. True member value is about creating a habit and capturing mindshare.

[AI Host 1] (16:08 - 16:12)
Because the reality is, right now, their habit is to just Google it.

[AI Host 2] (16:13 - 16:13)
Yep.

[AI Host 1] (16:13 - 16:24)
We are competing with the entire internet. And let's be honest, the friction of navigating a clunky association website usually means the internet wins. We have to be the definitive first stop.

[AI Host 2] (16:24 - 16:43)
And to become that first stop, the interaction has to feel entirely different. When a member comes to your governed AI and gets a highly useful, accurate answer drawn entirely from your trusted content, they experience immediate relief. True.

But the psychological trigger, like the moment you actually earn their trust, happens in the next step.

[AI Host 1] (16:43 - 16:43)
Which is?

[AI Host 2] (16:43 - 16:56)
Is when they ask a follow-up question. Or when the AI surfaces a related adjacent resource that they didn't even know your association had published, they discover the surprising nugget of insight they wouldn't have known to search for in the first place.

[AI Host 1] (16:56 - 16:59)
So they aren't just getting 10 blue links and wishing for the best.

[AI Host 2] (16:59 - 17:00)
No, exactly.

[AI Host 1] (17:00 - 17:03)
They are having their specific problem solved in context.

[AI Host 2] (17:03 - 17:18)
And the depth of that engagement is just staggering when it's executed well. Look at the Marine Retailers Association of the Americas, the MRAA. They tracked an 11-minute average interaction time with their system.

[AI Host 1] (17:18 - 17:18)
11 minutes.

[AI Host 2] (17:18 - 17:24)
11 minutes. They described it as putting their finger directly on the pulse of what members needed.

[AI Host 1] (17:24 - 17:28)
I mean, think about that 11-minute metric against standard website analytics.

[AI Host 2] (17:28 - 17:29)
It's insane.

[AI Host 1] (17:29 - 17:45)
A standard website bounce rate is measured in seconds. If someone is spending 11 minutes interacting with your platform, that is not a quick search. No, it's not.

That is a deep-threaded conversation. That is a member sitting down for a virtual cup of coffee with your association's collective brain.

[AI Host 2] (17:46 - 18:06)
It's pure engagement. And it scales in ways traditional support just cannot. Take the International Sign Association, ISA.

They facilitated over 10,000 conversations in their first year alone. And what is incredibly notable about their implementation is that they served members in nine different languages, all drawn dynamically from their core English-language source content.

[AI Host 1] (18:06 - 18:32)
Wait, really? Just from English sources? That is a massive equity and access play.

Huge. Translating trusted content on the fly breaks down language barriers and suddenly makes your association globally relevant without having to hire a translation team for every single document. Okay, here's where it gets really interesting.

If members are spending 11 minutes having detailed multi-language conversations, they aren't just absorbing information.

[AI Host 2] (18:32 - 18:33)
No, they're giving it back.

[AI Host 1] (18:33 - 18:41)
Right. They're leaving behind a massive trail of breadcrumbs. Every single question they type into that prompt box is a behavioral signal.

[AI Host 2] (18:41 - 18:47)
And you have just crossed over into the third ledger, the compounding value of the system, which is strategic intelligence.

[AI Host 1] (18:47 - 18:48)
Strategic intelligence.

[AI Host 2] (18:48 - 18:55)
Right, because if you are tracking the questions members are asking, you are mining pure, unfiltered market signal.

[AI Host 1] (18:56 - 19:07)
But how does that actually work mechanically? I mean, if ISA has 10,000 conversations, a human being cannot sit there and read 10,000 chat transcripts. The data is just way too overwhelming.

[AI Host 2] (19:07 - 19:27)
Yeah, they can't. And this is where the architecture of the platform really matters. If you are using a platform designed specifically for associations, take Betty as a practical example here.

The Betty platform has a layer called Insights. And it is designed to mathematically structure that raw conversational data so it doesn't just evaporate into the ether.

[AI Host 1] (19:27 - 19:28)
Oh, nice.

[AI Host 2] (19:28 - 19:43)
It clusters the questions by theme. It flags emerging industry needs before they ever show up in your annual survey. It literally shows you the exact vocabulary your members are using, which is often very different from the internal jargon your committees use.

[AI Host 1] (19:43 - 19:54)
So it's essentially a real-time interactive map of what the market is desperately trying to understand today, not what they cared about six months ago when you wrote your last strategic plan.

[AI Host 2] (19:54 - 19:59)
Exactly. It shows you the dead ends, too, where members are asking for content you simply haven't even created yet.

[AI Host 1] (19:59 - 20:11)
And that intelligence arms your staff to act. Like, they can make data-backed decisions about what courses to develop, what sessions to put on the main stage at the annual conference, and what policy gaps need immediate attention.

[AI Host 2] (20:11 - 20:13)
That's the real power of it.

[AI Host 1] (20:13 - 20:18)
I love this. And honestly, we have the perfect meta-example of this right here in our own sources.

[AI Host 2] (20:18 - 20:19)
Oh, we do.

[AI Host 1] (20:19 - 20:34)
This very deep dive we're doing right now, it did not come from thin air. It was created because we were listening to the recurring questions and signals from real association leaders across those nearly 200 implementations.

[AI Host 2] (20:34 - 20:34)
Yep.

[AI Host 1] (20:34 - 20:47)
They were constantly, repeatedly asking, how do we justify the cost? The signal was glaring. So we built content to answer the market's specific need.

That is strategic intelligence in action.

[AI Host 2] (20:47 - 20:55)
It creates a perfect self-feeding loop. You listen to the questions, you identify the gap, and then you build the exact programming the market is actively requesting.

[AI Host 1] (20:56 - 21:15)
Right. So let's tie this all back to the CFO who was sitting at the head of the boardroom table. We have unlocked staff capacity.

We've built deep 11-minute member engagement habits. We have a dashboard of strategic intelligence mapping out our future product pipeline. Does this finally turn into direct measurable revenue?

Because if revenue is the ultimate goal, why don't we just start there?

[AI Host 2] (21:15 - 21:20)
Because convergent is the conditional ledger. It is the fourth and final layer.

[AI Host 1] (21:21 - 21:22)
Conditional meaning what exactly?

[AI Host 2] (21:22 - 21:33)
Meaning if you walk into that boardroom promising that an AI chatbot is going to automatically function as a guaranteed revenue machine, you are setting yourself up for a spectacular failure.

[AI Host 1] (21:33 - 21:34)
Yeah, that makes sense.

[AI Host 2] (21:34 - 21:37)
It's a pathway to test, not a default promise.

[AI Host 1] (21:37 - 21:41)
It's the highest bar to clear. You really have to earn the right to sell to them.

[AI Host 2] (21:41 - 21:50)
You do. But when it's built on the solid foundation of the other three areas, capacity, member value, and intelligence conversion is entirely possible.

[AI Host 1] (21:51 - 21:52)
What does that look like in practice?

[AI Host 2] (21:52 - 22:12)
Well, the mechanics of it usually start with public tier discovery. Imagine a non-member lands on your site, asks a highly specific industry question, and gets a brilliant partial answer. They realize the sheer depth of your curated expertise.

They hit a paywall for the full document or the deep dive, and they convert to a paid membership right there.

[AI Host 1] (22:12 - 22:13)
That's a great flow.

[AI Host 2] (22:13 - 22:20)
Or for an existing member, the AI guides them down a specific credentialing or education pathway that requires a purchase.

[AI Host 1] (22:21 - 22:26)
Do we have examples of that psychological journey actually turning into a transaction, like hard numbers?

[AI Host 2] (22:27 - 22:35)
We do. Looking at the source patterns, the National Science Teaching Association, NSTA, they saw a 35% drop in support tickets.

[AI Host 1] (22:35 - 22:37)
Which firmly proves the capacity argument right there.

[AI Host 2] (22:37 - 22:44)
Exactly. But they paired that efficiency with four direct memberships sold directly through the AI interface.

[AI Host 1] (22:44 - 22:45)
Oh, wow.

[AI Host 2] (22:45 - 22:52)
Or looking at AIS again, they tracked 13 memberships sold through their platform. Now, I have to heavily caveat this.

[AI Host 1] (22:52 - 22:52)
Both do.

[AI Host 2] (22:53 - 22:57)
These are examples of a possible value path. They are not universal forecasts.

[AI Host 1] (22:58 - 23:01)
Right. You can't just plug these numbers into your spreadsheet and guarantee the same outcome.

[AI Host 2] (23:01 - 23:09)
No, you cannot. But they prove the concept. When you deliver immediate contextual value, engagement can absolutely cross over into conversion.

[AI Host 1] (23:10 - 23:24)
Okay, we have laid out the full menu now. We have capacity, member value, strategic intelligence, and conversion. If I am an association leader trying to structure this business case, how do I actually synthesize this before the board meeting?

[AI Host 2] (23:24 - 23:47)
The single most crucial strategic step you can take is picking one primary area of focus before you ask for budget. Just one. Just one.

You cannot build a business case that promises to be everything to everyone on day one. A business case anchored on staff's capacity requires totally different internal alignment and metrics than a business case built on strategic intelligence.

[AI Host 1] (23:47 - 23:48)
That makes total sense.

[AI Host 2] (23:48 - 23:58)
Yes. You will likely experience peripheral benefits across all four, but you must draw a line in the sand and define your primary non-negotiable definition of success.

[AI Host 1] (23:58 - 24:10)
Well, I want to talk about the gotchas now, because we promised a financially sober conversation, which means we have to talk about failure. We do. What makes this business case fall apart in the real world, even if the technology works perfectly?

[AI Host 2] (24:10 - 24:19)
The financially sober reality is that AI is an amplifier. It activates existing knowledge. It does not magically fix a broken operational strategy.

[Thomas Altman] (24:20 - 24:20)
Right.

[AI Host 2] (24:20 - 24:35)
Your business case will fail if you feed the system weak, outdated, or poorly organized source content. Garbage in still equals garbage out. It will also fail if you have a disorganized rollout plan where your members don't even know the tool exists on your website.

[AI Host 1] (24:35 - 24:47)
Change manager is huge here too, right? Like if the staff thinks the robot is coming for their jobs because leadership didn't communicate the human leverage concept clearly, they will actively resist it.

[AI Host 2] (24:47 - 24:54)
100%. They won't coach the AI. They won't feed it good data.

Internal friction is fatal to these deployments.

[AI Host 1] (24:54 - 24:55)
Wow.

[AI Host 2] (24:55 - 25:04)
Someone specific has to own the platform. It cannot be an orphaned project. And perhaps the most common failure point is not having a measurement plan in place before you launch.

[AI Host 1] (25:05 - 25:06)
Give me an example of that.

[AI Host 2] (25:06 - 25:15)
Well, if your primary justification is improving capacity, you need a hard baseline of what your capacity costs today. How many hours are spent on support right now?

[AI Host 1] (25:15 - 25:21)
Oh, I see. Because you cannot claim a return on investment in six months if you have no idea where you started.

[AI Host 2] (25:21 - 25:22)
Exactly.

[AI Host 1] (25:22 - 25:30)
You know, if we pull all the way back and look at the bigger picture, what we're really talking about is the fundamental nature of why associations exist in the first place.

[AI Host 2] (25:30 - 25:38)
Exactly. Humans drive the true value at associations. It is about connection, community, peer networking, expert curation.

[AI Host 1] (25:38 - 25:39)
Yeah.

[AI Host 2] (25:39 - 25:46)
Governed AI is just a tool to amplify and multiply that human value. It should never be viewed as a replacement for it.

[AI Host 1] (25:46 - 25:54)
The technology is a lever. It is entirely up to the leadership team to decide where to place the fulcrum and what specific weight they intend to lift.

[AI Host 2] (25:55 - 25:55)
Perfectly said.

[AI Host 1] (25:56 - 26:10)
So here are your marching orders. You need to pick one primary ledger capacity, member value, strategic intelligence, or conversion. Then you need to choose three to five specific trackable metrics that you can actually measure today to prove your case to the board tomorrow.

[AI Host 2] (26:10 - 26:14)
Do not guess at your metrics. Define them rigorously before you spend a dime.

[AI Host 1] (26:15 - 26:21)
And you do not have to stare at a blank whiteboard to do this. We want you to use the AI Business Case Worksheet. It is linked right in the show notes.

[AI Host 2] (26:21 - 26:22)
Please use it.

[AI Host 1] (26:22 - 26:31)
Go download it. Sit down with your COO, your CFO, your membership team, and make this cost justification incredibly concrete. Build the menu before you buy the kitchen.

[AI Host 2] (26:31 - 26:42)
Taking that structured approach changes the entire dynamic of the boardroom conversation. You are no longer defensively pitching a trendy tech purchase. You are presenting a measurable strategic capability.

[AI Host 1] (26:43 - 27:03)
I want to leave you with a final lingering thought to mull over as you start filling out that worksheet. If this new system works, if it gives your members exactly what they're asking for in seconds and completely offloads the repetitive grind from your internal teams, what will your newly leveraged staff do with that reclaimed time to build what those members actually need next?

[Rob Barnes] (27:09 - 28:18)
So to recap today's big question, how do we justify the cost of AI? And it reminded me that a lot of the work, Thomas, that we've done, particularly with Betty over the last two or three years, where there has been very clearly identifiable kind of before and after impact statements has been in those projects where there has been clear ownership. At a few different levels, I think, within the organization as well, within the association, there's kind of tactical ownership.

What are we doing with the insights from the AI to make decisions? Operational ownership, you know, whose budget does it live in? And how do we continue to measure success from that perspective?

And then executive level, you know, to our CEO and CFO listeners, you know, really understand that ownership at an executive level as well. Why? Answers to why do we decide to make an investment in AI and where are we hoping to see that impact?

There's plenty of evidence there now, and a lot of that was raised in this episode, which was great that our AI host picked that up.

[Thomas Altman] (28:19 - 29:21)
Yeah, I think it's been a through line through a lot of these episodes as well. Like if you have a champion or an owner, like what you're going to want to do, I think, at different levels of sort of ownership, right? At the executive level, if you're not the person owning the day-to-day, you're the person owning the vision, right?

You're the person that's saying, this is how we get ROI out of this investment in AI, right? And you're going to look for a champion, right? There's going to be someone in your association, I guarantee it, who just geeks out about that opportunity.

And then if that's you, if you, the listener, are the person who geeks out about the opportunity to own the actual implementation of it, it's your responsibility to understand the larger picture. Like, how are we driving this? It's not geeking out for geeking out's sake.

It's that we're driving kind of methodically towards value over time. And being able to increase that value as the tool gets better and sort of leading that charge. And when we see both of those things in place, we see massive success on Betty's side.

[Rob Barnes] (29:25 - 30:30)
And one of the key things here, I think, that I still sort of think about is, I still see debates around is this concept of ROI versus value. And part of those challenges is that in a lot of cases, those metrics, all those things, they aren't measured in the first place. And it's starting to think about that in a much more meaningful way with AI.

I think simply because the amount of information and insight that can come from what the work that an AI is kind of doing. I still really enjoyed the pick one of these ledgers, pick three to five of these trackable metrics, use some AI to help you track those metrics in an ongoing way as well, to keep that kind of business case open and fluid and be reporting on that. Not all four ledgers at once probably for an association that's kind of embarking on this for the first time, but certainly one of the four ledgers that makes the most sense to that association at this point in time.

[Thomas Altman] (30:31 - 31:36)
Yeah. And I think all of this is going to come through, right? The reason I really resonate with this episode is that obviously we played a large part in creating the episode with the AI kind of conversationally.

And a lot of it came from our experience, right? So in this case, we've kind of been eating our own dog food or drinking our own champagne. That's a better way to say it.

And everything that we've kind of been doing with our own versions of Betty actually led in a sort of a metal way to the creation of this actual set of podcasts, right? Like this podcast came from this sort of process that we're talking about and kind of discovery through the sort of conversational history and all these insights we get from that into another level of engaging with you, right? And helping you understand how to use AI better because it revealed that need that we otherwise wouldn't have seen.

So I think it's kind of an interesting bow to put on this conversation where not only have we been talking about what to do, I think this one actually also demonstrates how to do it in a certain way as well. Like we can show you the results of that process by the fact that you're listening to this podcast.

[Rob Barnes] (31:36 - 33:07)
Yeah, that self-aware moment that comes as a result of this was justifying cost. And cost is not always just monetary, right? There's energy, there's effort, there's time that gets committed to this.

But you're right, the strategic intelligence that we sought out to learn about the association industry and its position on AI, we used AI, Betty and other tools to create all of that. We learned so much. Those learnings became 12 episodes of a podcast that we now publish and put out there to help.

And again, it just reminds me that that's what the association knowledge cycle was written about, right? That learning, the discovery that comes from all of that, and then turning it around. So the fact that this podcast is itself a product of the mechanism that it was describing is very, very cool.

And I like that it's also, you know, we were talking off air about this kind of almost being part one of two. So the next episode is very much for our CEO and CFO listeners as we start to talk about the revenue side of the ledger as well. So one more time before we let everybody go, download the AI Business Case Worksheet from the show notes.

It's the fastest way to put today's episode to work. Thanks for listening to the Association Intelligence Podcast. Join us on the next episode to answer this question.

Can this become revenue positive?

What this helps your team answer

Why "everyone else is doing it" and "AI is the future" don't hold up in a budget conversation, and how to shift from reacting to the board's AI mandate to intentionally creating measurable value.
The four ledgers for capturing AI value (capacity, member value, strategic intelligence, and conversion), and why most associations should start with one, usually capacity, and grow from there.
How to make the case concrete by choosing three to five trackable metrics before you spend anything, and why clear ownership at the tactical, operational, and executive levels separates AI projects with real before-and-after results from the rest.

Companion Asset

AI Business Case Worksheet

Revenue Pathways

Can This Become Revenue Positive?

Why “revenue positive” isn’t a yes-or-no question but a pathway design question. The episode separates four different revenue claims, walks through five concrete pathways (with real proof points and honest caveats), and shows how to test them with a small, low-risk pilot.

28:21

0:00 28:21

[Thomas Altman] (0:05 - 0:09)
Today's big question, can this become revenue positive?

[Rob Barnes] (0:10 - 1:36)
We are Thomas Altman and Rob Barnes and this is episode 11 of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the Revenue Pathways Map. It's linked in the show notes, so grab it now if you want to follow along or download it after and put it right to work.

Talking about revenue is becoming more and more a part of the conversations we're having with association executives all the time, because interestingly there is research reports recently published in the association industry, Thomas, about associations are willing to pay. They're already paying, you know, they're paying for Claude, they're paying for ChatGPT Enterprise, they're paying for other AI. So the willingness to pay is certainly there, but over time it's very much about what are we expecting to get from these investments and are the things that we are investing in actually capable of delivering new sources of revenue, enhancing existing sources of revenue.

And what I like about this episode is it splits that kind of revenue positive description into four distinct claims, but some of those claims are easily things that blur together, right? That when we, even when we're talking to associations about coming on board with Betty now, we blur some of those things together because they just make sense.

[Thomas Altman] (1:38 - 2:29)
Yeah, I think so, and I think in a lot of ways this episode is kind of a continuation of the previous one, right? In the same way that you don't have to be a bystander, it's not adopt AI and you're done, it's how can you be active in moving towards this revenue positive future? And these four different levers that it pulled apart, I think were kind of interesting.

So again, the AI host will go into more detail, but to kind of set you up for sort of digging into when they do, they mentioned direct revenue, that's like someone gave your association money, that's direct revenue. They mentioned cost offset, which again is like people, like you're spending less on the same thing, right? So it's more of a profit metric, but it's still an important one.

And then the two squishy ones, right? So then there is, let me find the word for it, they call it... Influence conversion.

Influence conversion.

[Rob Barnes] (2:29 - 2:29)
Yeah.

[Thomas Altman] (2:30 - 3:45)
And then revenue intelligence. So influence conversion is this multi-touch kind of buying process that actually I think most people go through for most of your buying decisions, right? So the direct exchange is the last step in that process.

But the sort of like, I'm interacting with an AI and it guided me to this resource and that resource is paid and I didn't buy it right then, but I kept thinking about it for a few days. And when I came back, I encountered it again. I was like, oh yeah, I should buy it.

How do you measure that, right? That gets a little bit squishier. Or this other idea of sort of revenue intelligence, right?

How can by intentionally building more engaging opportunities and conversations within your AI system, capturing that history, how can you analyze that systematically so that you are creating future revenue possibilities that are kind of one step ahead of the market, one step ahead of your members, right? You're going to put this buying opportunity right in their path before they even get there, because you can see what's coming around the corner, right? And I think those two are both very powerful, you know, adoptions of AI, ways to make AI revenue, but simultaneously harder to measure.

And that's where it gets kind of, that's where the lines blur. That's where it gets a little bit squishy, but I think we can't discount them either.

[Rob Barnes] (3:45 - 4:41)
And there's this transition that continues to take shape now as we think about that, where in a lot of cases, an AI system is trying to be applied to an existing business model. You know, the pay-for-play, the paywall kind of business models that have existed, we'll share a little bit of this with you, but if you want the full experience, you've got to pay, versus what are the new business models that are native to an AI-enabled world that need to be explored by associations? And there's a nexus there as well, the idea of source-aware gating, as it's described in the episode, is a really interesting one, because it speaks a little to an existing business model and the application of AI to that existing business model, versus what's the next iteration of that for an association to generate revenue from.

[Thomas Altman] (4:41 - 6:13)
Yeah, I think this, that source-aware gating idea is like a step towards the future, right? It's still got one foot in the past, but it's walking towards what's going to be possible. And what I think, what I want to challenge everyone listening to this podcast to do is think about what are revenue generating opportunities that are possible today that were not possible last year, that were not possible before kind of AI became a mature technology.

And how do we walk from here to there? And then the source-aware gating concept is, I think, a step in that direction. It's not the full dive jump in, but it is sort of walking in that.

And what it means by that is, as an example of the kind of thing you can do now, is you can have like your chat assistant be a sort of a soft seller, right? Like, hey, if you're maybe, it's before a login, like you're talking to the general public, someone who's not a member, hey, we've got way more information about this topic. If you want to join, here's a link to join.

Not the hard sell, but like let them know, like there's more value there. Or you're talking to it and like, ooh, that maps perfectly to this upcoming webinar. Here's a link to register, right?

Having your AI kind of be right there, ready with the link and the pitch when it makes sense. Not all the time, just sort of like when it's actually going to add value to the conversation. And then what's the step you can take after that?

I want to challenge everyone because I don't have the answer right now. But like, if that's the first step out, once you take your foot off of that's still left in the past, where does that go, right? So I would love to hear your ideas, so please share them if you get a chance.

[Rob Barnes] (6:14 - 6:36)
Well, let's get into it. Before we hand over this podcast to our AI hosts, quick reminder that the following is generated using Notebook LM with heavy involvement from us at the Betty team to ensure that the content generated is valuable. For a deeper dive on our approach to the Association Intelligence Podcast, please go back and listen to episode zero.

And now over to the AI hosts.

[AI Host 1] (6:39 - 6:47)
Every day, thousands of association leaders look at their shiny new AI assistant and they ask this one like terrified question.

[AI Host 2] (6:47 - 6:49)
Yeah, I know exactly the one you mean.

[AI Host 1] (6:49 - 6:55)
It's always, can this become revenue positive or is it just a bottomless expense?

[AI Host 2] (6:55 - 7:08)
It is the single most pressing anxiety in the association space right now. I mean, we're looking at patterns from nearly 200 association AI implementations. And that question is just hanging over almost every single one of them.

[AI Host 1] (7:08 - 7:15)
For sure. Because leaders are looking for actual tangible upside, not just another line item on the Exactly.

[AI Host 2] (7:16 - 7:23)
But we need to draw a hard line here before we go any further because we did a deep dive previously on cost justification and that was a completely different conversation.

[AI Host 1] (7:23 - 7:30)
Right. Cost justification is about defensibility. Like we looked at staff leverage, baseline member engagement, you know, things that make the system worth keeping.

[AI Host 2] (7:30 - 7:42)
Yeah. Defensibility. But today we are putting defensibility aside.

We are asking which actual revenue pathways are credible and what specific proof each one requires before you bet your budget on it.

[AI Host 1] (7:43 - 7:52)
And because this gets incredibly granular, you should absolutely grab the revenue pathways map from the show notes. It's basically the central tool we're using today.

[AI Host 2] (7:52 - 8:04)
Yes, please do. It'll help you separate your pathways, assumptions, metrics, and confidence levels as you listen along. Because you need a structured way to evaluate these options.

Otherwise you end up chasing, you know, vague promises.

[AI Host 1] (8:05 - 8:14)
So my instinct as a leader looking at a new AI tool is to just like check our monthly sales numbers and see if they went up after we turned the AI on.

[AI Host 2] (8:15 - 8:15)
Right.

[AI Host 1] (8:15 - 8:25)
But I'm guessing that's a terrible way to measure this. Because it assumes all revenue is exactly the same thing. Like asking if opening a new storefront will make money without knowing what you're selling.

[AI Host 2] (8:25 - 8:30)
It is a terrible way to measure it because it conflates four very different financial claims.

[AI Host 1] (8:30 - 8:31)
Okay. Lay them out for us.

[AI Host 2] (8:31 - 8:38)
So first you have direct revenue. That's actual dollars exchanging hands directly inside or immediately following an AI interaction.

[AI Host 1] (8:39 - 8:40)
Got it. Direct sale.

[AI Host 2] (8:40 - 8:52)
Right. Second, there is influenced conversion. This is where someone interacted with the AI, got value, and then later bought something.

Making the AI part of a longer multi-touch journey.

[AI Host 1] (8:52 - 8:53)
Makes sense. What's the third?

[AI Host 2] (8:54 - 9:08)
Third is cost offset. Saving money by reducing expenses or reclaiming time elsewhere in the organization. And fourth is revenue intelligence, which is using the system's data to figure out what you should build and sell next.

[AI Host 1] (9:08 - 9:11)
So if I just look at a generic, you know, fails are up dashboard.

[AI Host 2] (9:12 - 9:12)
Yeah.

[AI Host 1] (9:12 - 9:14)
I have no idea which of those four things actually happened.

[AI Host 2] (9:14 - 9:34)
Exactly. You have no idea what mechanical lever you actually pulled. And the core thesis here is that revenue positive AI is not a promise.

It is not a yes or no question. It's a pathway design question, right? Yes.

You improve your odds of actually seeing a return by defining five specific levers before you implement anything. You define the specific pathway you are testing.

[AI Host 1] (9:34 - 9:35)
Okay. The pathway.

[AI Host 2] (9:35 - 9:51)
Then you define the assumption that has to be true for that pathway to work. You choose the exact metric you are going to track. You state your confidence level based on the evidence you currently have.

And finally, you set your evidence threshold. You know, the specific number that proves this is actually working.

[AI Host 1] (9:51 - 10:03)
So we have to stop treating AI as this, like, magical revenue engine that automatically prints money and start treating it as a set of mechanical levers we have to pull intentionally.

[AI Host 2] (10:03 - 10:08)
Precisely. Defining that operating frame stops leaders from over committing to unproven claims.

[AI Host 1] (10:08 - 10:27)
Okay. Let's look at the first lever, which is probably the most obvious one to a board of directors. Pathway one, public tier membership conversion.

Taking public curiosity and turning it into actual dues paying members. Mechanically, how do you design an AI to do that without just, you know, giving away the farm for free?

[AI Host 2] (10:28 - 10:34)
Well, you cap the public access. You give non-members a taste of the value, but you engineer a limit. It could be a simple volume limit.

[AI Host 1] (10:34 - 10:37)
Like allowing a set number of free questions per day.

[AI Host 2] (10:37 - 10:52)
Exactly. Free questions per day before prompting them to join. Or you provide a comprehensive public assistant that delivers a great answer based on public info.

But alongside that answer, it includes a subtle contextual prompt to join for deeper access.

[AI Host 1] (10:53 - 10:59)
And if your external systems support it, like your AMS or payment gateway, you can even build a purchase or join path right there in the chat, right?

[AI Host 2] (10:59 - 11:04)
You can. But the really fascinating mechanism here is this concept of source-aware gating.

[AI Host 1] (11:04 - 11:11)
Oh, yeah. I wanted to talk about this. So source-aware gating.

Mechanically, how does the AI actually execute that?

[AI Host 2] (11:12 - 11:21)
So because the AI is restricted to reading only your private governed database, it hits a deliberate wall. Let's say a user asks a highly technical question.

[AI Host 1] (11:21 - 11:21)
Okay.

[AI Host 2] (11:22 - 11:29)
The AI scans your archives, finds the exact journal article or member-only guideline that answers it, but it is programmed to halt.

[AI Host 1] (11:29 - 11:29)
It stops itself.

[AI Host 2] (11:30 - 11:42)
Right. You can see the file name and generate a high-level summary confirming that the answer exists, but it cannot reveal the specific restricted details. Wow.

Instead, it generates a highly contextual link to subscribe or join to read the full document.

[AI Host 1] (11:42 - 11:51)
But wait, I mean, isn't that just a glorified paywall trick? Like, I have the secret, give me 20 bucks and I'll tell you. Because people hate that.

[AI Host 2] (11:51 - 12:03)
It sounds like it, but it's fundamentally different. A traditional paywall is a blind door. You read a headline, you hit a barrier, and you have to blindly trust that what is behind the paywall is actually what you want.

[AI Host 1] (12:03 - 12:05)
Right. You're just hoping the article is good.

[AI Host 2] (12:05 - 12:16)
Exactly. Source-aware gating provides a useful, highly specific preview of the association's actual value. It makes the member value sequence completely visible.

[AI Host 1] (12:16 - 12:24)
Ah, the member value sequence. So they ask a question, they get a partial answer. They might ask a follow-up, which builds trust.

And then they hit the gate.

[AI Host 2] (12:24 - 12:35)
Yes. They already know the asset has the answer. It replaces blind trust with verified utility.

And we have selected proof points showing this working. Like ISA, the International Society of Automation.

[AI Host 1] (12:36 - 12:43)
Right, ISA. We should look at them as a selected proof point here, not as broad revenue-tracking data, but just as one example of this pathway working.

[AI Host 2] (12:43 - 12:59)
Exactly. Using this pathway, ISA saw 13 memberships sold directly. But what is mechanically impressive is that they handled over 10,000 conversations.

And they served users in nine different languages, all drawn exclusively from their English-language content.

[AI Host 1] (12:59 - 13:13)
That is wild. The AI takes a query in Spanish, translates it to English, searches the private database, finds the engineering answer, translates it back, and then seamlessly presents a Spanish-language gate saying, join ISO to read the rest.

[AI Host 2] (13:13 - 13:14)
It's incredible.

[AI Host 1] (13:14 - 13:24)
But wait, so source-aware gating relies on holding back a secret from a non-member? What if the user is already a member paying their dues? The AI can't sell them a membership they already have.

[AI Host 2] (13:24 - 13:32)
Right. The monetization for that specific pathway stops there. Which brings us to pathway two, education and certification conversion.

[AI Host 1] (13:32 - 13:40)
Okay, so transitioning to learning pathways. If an organization relies heavily on education and credentialing, rather than just dues, what happens?

[AI Host 2] (13:40 - 13:50)
The assistance role shifts from a gatekeeper to a navigator. The AI helps members discover courses, credentials, exam prep materials, or specific learning products they need.

[AI Host 1] (13:50 - 14:02)
Which is so much better than searching a catalog. I mean, if I have three years of experience in a specialized niche, I don't want to scroll through a PDF of 400 courses. I want guidance.

Like a 204.7 academic advisor.

[AI Host 2] (14:02 - 14:14)
Exactly. You want it to surface the exact modules you need for your renewal path. And that discovery process alone drives value.

But we can take this a step further mechanically with StudyBuddy agents.

[AI Host 1] (14:14 - 14:16)
StudyBuddy agents. What are those?

[AI Host 2] (14:16 - 14:27)
These are specialized, premium AI assistants that help learners actively continue through a credentialing process. They can quiz the user, explain complex concepts, provide personalized study support.

[AI Host 1] (14:27 - 14:31)
Oh, so access to that study support can itself be a premium feature or a paid add-on.

[AI Host 2] (14:31 - 14:37)
Yes. But we need a strict honesty check here. We cannot claim that these assistants improve completion rates.

[AI Host 1] (14:37 - 14:39)
Right. Unless you have specific source numbers to back it up.

[AI Host 2] (14:39 - 14:49)
Exactly. Without those source numbers, you cannot make that claim. The proven value right now is in discovery, finding the right course, and providing that premium support.

[AI Host 1] (14:49 - 15:03)
Okay, so let's push this further. Pathways one and two rely on us already having the exact course or membership the user wants. But what if the AI doesn't sell a course, but instead tells you exactly what course you should build next?

[AI Host 2] (15:03 - 15:11)
That is pathway three, insights-driven paid offering development. When members interact with the assistant, they are asking questions in their own words.

[AI Host 1] (15:11 - 15:12)
Yeah.

[AI Host 2] (15:12 - 15:22)
That question data reveals unarticulated demand for future offerings, like webinars, courses, guides, events, research, or even sponsorship packages.

[AI Host 1] (15:22 - 15:28)
It's like running a permanent 24-7 focus group where your members are actively confessing their biggest knowledge gaps.

[AI Host 2] (15:29 - 15:39)
It really is. Mechanically, the AI aggregates these queries. If you see hundreds of members asking about a new regulation, you don't need to guess what your next certification module should be.

[AI Host 1] (15:39 - 15:45)
You have to reframe the value proposition internally, right? I can see a board getting frustrated saying, well, the AI didn't sell anything today.

[AI Host 2] (15:45 - 15:54)
You have to frame this strictly as revenue intelligence. It is not instant direct revenue. It is the validated blueprint to build the product that will generate revenue later.

[AI Host 1] (15:55 - 16:08)
Got it. Okay, let's look outward. Once we know what our members want internally, the next logical question is, can we license this knowledge externally?

Pathway four, embedded partner, chapter, API, and marketplace access. Yes.

[AI Host 2] (16:09 - 16:17)
Governed knowledge can become part of a partner portal, a chapter offering, a sponsor knowledge hub, a training platform, or an industry software workflow.

[AI Host 1] (16:17 - 16:22)
So taking the knowledge out of the association's website and putting it where people actually work.

[AI Host 2] (16:22 - 16:27)
Exactly. Third parties value authoritative knowledge. But I have to give a crucial sober caveat here.

[AI Host 1] (16:27 - 16:28)
Okay, lay it on us.

[AI Host 2] (16:29 - 16:44)
This is strategically compelling, but it must be kept honest. Direct documented revenue proof for API or marketplace licensing should be treated as an open question and a longer-term hypothesis, unless your specific association has direct proof for a specific partner.

[AI Host 1] (16:44 - 16:50)
Yeah, this sounds like the easiest place for an eager board to overpromise. Like, let's just sell our API and then nobody buys it.

[AI Host 2] (16:50 - 16:56)
Right. It's technically and legally complex. You must separate near-term evidence from longer-term strategy.

[AI Host 1] (16:56 - 17:07)
Okay, so if direct revenue and licensing are the offensive plays, what about the defensive plays that hit the bottom line? Pathway five, capacity offset.

[AI Host 2] (17:07 - 17:20)
Capacity offset is about support ticket reduction and reclaimed staff time. When the AI handles routine queries, your ticket volume drops. And that reclaimed time can contribute to a net positive value for the organization.

[AI Host 1] (17:20 - 17:22)
We have a selected proof point for this too, right? NSTA.

[AI Host 2] (17:23 - 17:34)
Yes, the National Science Teaching Association. They are a great selected proof point. They resulted in four memberships sold alongside a 35% drop in support tickets.

[AI Host 1] (17:34 - 17:45)
A 35% drop in tickets. So it's like the old saying, a penny saved is a penny earned. Does reclaiming a third of a support team's week count as a financial win, even if it doesn't show up as a new sale?

[AI Host 2] (17:45 - 18:00)
It does, but we must be very careful with our language. Capacity offset is not direct revenue. Leaders should never say the system, you know, funds itself or pays for itself.

Right. You should never claim it becomes net positive immediately. It improves margins and capacity.

It doesn't mint new money instantly.

[AI Host 1] (18:01 - 18:04)
Because the CFO will look at the bank account and say, where's the cash? Exactly.

[AI Host 2] (18:04 - 18:11)
Okay, so we've mapped the five revenue paths. Now, how do you actually implement them without failing or overcommitting?

[AI Host 1] (18:11 - 18:18)
You use a low cost to failure testing method. First, you start with a clear pathway and a very limited pilot.

[AI Host 2] (18:18 - 18:19)
Why a limited pilot?

[AI Host 1] (18:19 - 18:31)
To watch where the AI might trip over your data. Then you explicitly instrument your attribution. You have to track the user's digital journey so you can prove the AI caused the sale.

[AI Host 2] (18:32 - 18:36)
If you don't mechanically link the chat to the shopping cart, you can't claim the revenue. Right.

[AI Host 1] (18:37 - 18:44)
Third, you watch your chosen metric closely. And finally, you only expand the program when the evidence actually supports the claim.

[AI Host 2] (18:45 - 18:47)
And there's a pretty honest caveat to all of this, isn't there?

[AI Host 1] (18:47 - 18:59)
A very honest one. If your underlying knowledge base is weak, if the offer is unclear, if attribution is missing, or if you haven't actually designed the next step for the user to take, all revenue claims will completely fall apart.

[AI Host 2] (18:59 - 19:10)
Yeah, I mean, if your association's archives are an absolute mess and your knowledge isn't valuable, no AI is going to magically monetize that, right? Right. You must have a substantial knowledge asset worth monetizing first.

[AI Host 1] (19:11 - 19:24)
When we look at examples of this done right, we can softly bring up Betty as a highly practical example. Betty represents governed, source-bound, multi-tier association AI with insights and specialist use cases.

[AI Host 2] (19:25 - 19:29)
Exactly. It works because it activates association knowledge.

[AI Host 1] (19:29 - 19:42)
And that phrasing is key. We aren't just talking about raw data. Associations are unique because they are both community organizations and knowledge organizations.

These two things are not an eitherer. They enhance each other.

[AI Host 2] (19:42 - 19:43)
They absolutely do.

[AI Host 1] (19:44 - 19:52)
So let's bring this all back to you, the listener. If you are hearing yourself in these questions, your next step is really practical. Don't ask if AI can create revenue in the abstract.

[AI Host 2] (19:52 - 19:54)
No, do not ask that.

[AI Host 1] (19:54 - 20:03)
Instead, use the revenue pathways map. Grab it from the show notes. Pick just one pathway.

Write down your assumption. Choose your metric. Set your confidence level.

And define your very next test.

[AI Host 2] (20:03 - 20:20)
And keep those internal revenue conversations honest and actionable. I actually want to leave you with a final thought to mull over. When you finally stop guessing what your members value and start looking at the real-time demand signals hidden in their daily questions, how does that change the fundamental identity of your association?

[AI Host 1] (20:21 - 20:25)
Oh, that's a great point to end on. Thanks for listening, and we'll see you on the next Deep Dive.

[Rob Barnes] (20:32 - 22:41)
So today's big question was, can this AI system become revenue positive? And it reminded me that we have some actual great case study evidence of this, even within just the customer base at Betty. And I've heard many others presented at the array of conferences where this subject matter is sort of talked about.

But I'm reminded that we have customers that have got this direct revenue pathway kind of built into their AI assistant to the extent that a conversation with a non-member on a public-facing AI has opened a link, and the AI has made a recommendation about, if you want to know more about this subject matter, consider joining. They click on the join, they go through and pay. That's occurring multiple times now.

One of our customers, the International Society of Automation, has attributed MIMO to tens of thousands of dollars in membership revenue, simply from conversations that MIMO was having with people on their website. I love that. And there's other examples too, where Buyer's Guide, vendor recommendations in a discussion, people are paying for the right to be involved in those, to advertise their supply, their services in those Buyer's Guides, and those are coming up in conversations, and it's generating advertising revenue.

So that direct revenue pathway is now one, Thomas, that we can see absolute evidence of that, even to the point where people are subscribing just to get access to the AI assistant. They're not joining the association, but the association has seen value in just letting people interrogate the knowledge base through conversations with the AI for $20 a month. So all of these opportunities are not fantastical anymore.

It's real, it's in play, and it's part of a new way of thinking about an AI system in generating revenue.

[Thomas Altman] (22:42 - 24:30)
And I think the idea for each of those options, kind of getting back to this idea of some of those pathways are a little bit squishy, they're a little bit ill-defined. And the attempt in this podcast that I think for each of those options that you mentioned, kind of reverses that, adds a little bit of definition to it, is this idea of instrumenting the attribution, right? Being able to intentionally link some action from the AI to some step in the process towards generating revenue, whether that's directly to the sale, maybe it's one step, or maybe we can see the AI push the person into this part of some sort of sales funnel, whatever it might be.

And I like that idea because it does two things. One, it can actually start to prove that the AI is actually leading to revenue. But then two, it actually serves as a way of validating hypotheses.

Meaning, hey, if we do this, our hypothesis is that it will lead to revenue in this way. And if you're instrumenting the attribution, like our AI podcast said, you can actually prove that true or false, right? And if it's proven false, you can actually gain learning from that.

So you can continue to learn and continue to walk towards revenue positivity. And that's one of the things I want to make sure we lean into here is that it's not necessarily, although it's great when it happens, you flip the switch and the money pours in, right? It's generally more, you're an active participant and continually pushing the envelope, taking steps further and further to become revenue positive along the way.

And eventually you will get there. It's just making sure you're instrumenting this in a way that you can learn from the pathways you take, prune the ones that didn't work, be inspired to try new ones, and kind of keep growing in that manner.

[Rob Barnes] (24:30 - 26:45)
And this is part of this that just continually reminds me that a four-purpose AI assistant and AI system is going to be generating the kind of strategic intelligence that lets an association know, these are the sorts of knowledge needs that your people, your constituents, your customers, your members have right now. Be able to turn that into an asset that then can be monetized and do this faster and more effectively than just about any other time in the history of running an association. Because oftentimes we make, and again, you know, over the history of associations, we've made a lot of assumptions about whether our content is worth anything or not.

We produce it because we want people to know about it, but we don't know whether it actually has any value to the people that we're trying to get it to. AI is telling us exactly what is valuable. It's telling us which knowledge assets are being utilized in an AI system, in conversations, in actions.

And we can, again, that association knowledge cycle just keeps piling on, and now when we think about it from a revenue perspective, we can look at it and say, well, if we met this gap, this knowledge gap immediately with an asset and sold it, we know that there's an audience for it right now, and we can turn these things around. And I'm not sure that a lot of associations are looking at their current knowledge bases or their content assets, their information assets, and putting them through the test, you know, those kind of rigorous tests of is this something that could be monetized? We've always done it with events.

We've done it with webinars. We've done it with a lot of these sorts of things. And actually, I think conferences themselves take on a whole new meaning when in an AI system, in an AI-enabled world, the ability to monetize the post-conference experience beyond just the final day.

It's like, well, what if you were able to subscribe to all of the content through an AI system for the next 11 months until the next annual meeting starts to get advertised as well? These are all genuine revenue pathways now that just weren't available a year ago.

[Thomas Altman] (26:45 - 27:25)
Yeah, and the future is not defined, and the future is yours to write. And I think that's sort of what inspires me a lot about the conversations that we're seeing happening here is that people are being very creative with how they're using AI. And when they're intentional with using that creativity to drive towards revenue, they tend to achieve it.

So, yes, we're providing you with a tool set and a way to get started. But I think more importantly, it's hopefully a way to start thinking about what that means for you and what you might even do different. Can you use this as a jumping-off point into your own path?

And then can you share that back with the community? Because I think that's going to be how we all grow in our development of AI expertise.

[Rob Barnes] (27:26 - 28:15)
Yeah, to our CFO friends that are getting their business unit owners to start developing budgets and come up with some great homework here for you, I think. Look at the revenue pathways map. Pick one of these pathways and start to build out a hypothesis around, you know, if we did this, we believe this could happen.

And this is what it would look like monetarily. This is really important work, for sure. But it's also exciting work that can be done now when you think about leveraging these AI systems.

So one more time, before you go today, download that revenue pathways map from the show notes. It's the fastest way to put today's episode to work. Thank you for listening to the Association Intelligence Podcast.

Join us on the next episode when we answer this big question. What does success look like in three years?

What this helps your team answer

Why a single “sales are up” dashboard can’t tell you whether AI is paying off, and how to separate four distinct claims: direct revenue, influenced conversion, cost offset, and revenue intelligence.
Five pathways to evaluate: public-tier membership conversion (including source-aware gating), education and certification conversion, insights-driven new offerings, embedded partner or API access, and capacity offset. The episode is candid about which have documented proof and which are still longer-term hypotheses.
How to test a pathway without overcommitting: pick one, write down your assumption, metric, confidence level, and evidence threshold, instrument the attribution, and only expand when the evidence supports it.

Companion Asset

Revenue Pathways Map

Three-Year Vision

What Does Success Look Like in Three Years?

The season finale on why you can’t predict three years of AI but can build the framework to adapt to it. It covers the staged path from a bounded pilot to governed knowledge infrastructure, and the human decision about when AI insights should change your mission.

32:15

0:00 32:15

AI PC ep 12

[Thomas Altman] (0:05 - 0:09)
Today's big question, what does success look like in three years?

[Rob Barnes] (0:10 - 0:40)
We are Thomas Altman and Rob Barnes and this is episode 12 of the Association Intelligence Podcast. Before we get into it, this episode's companion asset is the Association Knowledge Maturity Roadmap. It's linked in the show notes so grab it now if you want to follow along or download it after and put it to work.

Thomas, I couldn't help but think about pushing back almost on the premise of this episode because I'm like, can we really even think three years ahead in the era of AI?

[Thomas Altman] (0:42 - 1:59)
Yeah, no, I mean think about, think about even the beginning of this year and where we are now. Like the usage of AI, my usage of AI at least, has completely changed since January, much less three years going forward. So I think one of the big pitches here is you can't say in three years we're gonna be here, right?

That you're just not gonna be able to do that because everything is gonna change so rapidly that whatever you say you're gonna be doing in three years, I can guarantee you, I don't know what we can do in three years, I can guarantee you that what you tell me today you're gonna do in three years, we won't be doing that. 100% chance that that won't happen. So really what I think we need to get out of this episode is how do we set ourselves up for that future, right?

What are the steps we can take like this bounded pilot? What can we learn from it? How can we actually, you know, do that in a way that is ready for a rapidly evolving sort of AI technology?

And how can we be ready to jump off each new advance as it comes up, right? And like how do we conduct our three-year roadmap to be open to whatever possible future but still driving to some sort of North Star, right? Some vision of value and how it's going to incorporate into your association.

Like how do you marry those two things? I think that's going to be an important takeaway from this episode.

[Rob Barnes] (2:00 - 2:44)
So even though in the episode our hosts talk about this staged model over this approach, what you're saying is it's not so much about the technology itself, which we know is just going to advance beyond what we could even imagine right now in that same period of time, but it's about having the kind of framework in place for an association that allows it to adapt far more quickly than we traditionally adapt to technological change.

But do so in a way that we can benefit from, mitigate the risks of, and not have to worry so much about the technology itself, just knowing that we have the system in place that allows us to adapt to it?

[Thomas Altman] (2:45 - 3:40)
Yeah, I think that's exactly it. I think the roadmap is one where you start concrete, and maybe no, you don't start concrete. You start with kind of a vision of sort of what your true mission as an association is, and you preserve that.

And everything that you're building toward is toward advancing that mission, right? And there's going to be bits and pieces along the way that kind of all row in the same direction. So really what you're doing is trying to explore the space of AI, which is a never-changing space, in a way that's methodical and keeps you moving in that direction.

And any plan that you have has to accommodate the fact that this is a fast moving space, and it could get lost, the plot could get lost, if you're not intentional about each row of the oar is in the direction of what you decided is the right mission for your association.

[Rob Barnes] (3:40 - 4:57)
It's an interesting one, and I know that some of our longer-term and most successful customers at Betty are now where the AI system is just part of the infrastructure of doing business with their constituents. It's governed that way, it's operated that way, it's something that they know that if they tried to take it away from the experience, they would be pushed back on them. And I think, as you said, even our own internal use of AI within the team at Betty is completely different now.

I'm sure that if someone turned off my Claude subscription and all my agents that work with me every day, it would be challenging for me. And so this is part of the experience that associations are creating for their members and stuff now. So yeah, it's really interesting to hear how our hosts talk about it in this episode.

And before we do hand over this episode to our AI podcast hosts, a quick reminder that the following is generated using Notebook LM, with heavy involvement from us here at the Betty team to ensure that the content generated is valuable to our listeners. For a deeper dive on our approach to the Association Intelligence Podcast, please go back and listen to Episode 0. And now over to the AI hosts.

[AI Host 1] (5:02 - 5:05)
I want you to imagine you're sitting at the head of a conference table.

[AI Host 2] (5:05 - 5:06)
Okay.

[AI Host 1] (5:06 - 5:14)
Your team is looking at you. Your board is waiting for an update. And there is just this one single question hanging in the air.

[AI Host 2] (5:14 - 5:14)
Oh, I know the one.

[AI Host 1] (5:15 - 5:25)
Right, it's the exact question on literally every leader's mind right now. What does success look like in three years? Specifically, you know, when it comes to bringing AI into your organization.

[AI Host 2] (5:25 - 5:34)
Yeah, that is, I mean, that's the defining question of the moment for any association leader. And honestly, looking at the data we have today, it is incredibly easy to answer it the wrong way.

[AI Host 1] (5:34 - 6:03)
Which is exactly why we are dedicating this entire deep dive to unpacking that answer. So think of this as our capstone on the topic. Right.

We aren't going to talk about abstract sci-fi future tech, and we're skipping the broad industry hype entirely. Thank goodness. Yeah, exactly.

Instead, we've got a massive stack of post-mortem reports, strategy memos, and hard data pulled from nearly 200 different associated AI implementations sitting right here on the desk.

[AI Host 2] (6:04 - 6:09)
And I think we should clarify right up front what those documents actually represent.

[AI Host 1] (6:09 - 6:09)
Yeah, please do.

[AI Host 2] (6:10 - 6:19)
Because when we say nearly 200 implementations, we are not looking at like a sterile data set or some generic vendor success rate.

[AI Host 1] (6:19 - 6:20)
Right, it's not just numbers on a spreadsheet.

[AI Host 2] (6:20 - 6:31)
Exactly. We are looking at a collection of hard-won behavioral and strategic patterns. We have the internal reports on what actually happens when associations try to integrate this technology.

[AI Host 1] (6:31 - 6:33)
The good, the bad, and the ugly.

[AI Host 2] (6:33 - 6:39)
Yeah, like what the hidden pitfalls are and what the real drivers of long-term value actually turn out to be.

[AI Host 1] (6:39 - 6:59)
So the mission today is to decode a very specific transition. We're tracking how an organization moves from treating AI as a novelty to establishing it as the very infrastructure of the association's knowledge. That's a huge leap.

It is. And you know, the current AI craze, it feels a lot like buying a high-end treadmill.

[AI Host 2] (6:59 - 7:00)
All I like is an allergy.

[AI Host 1] (7:00 - 7:17)
You know what I mean? You do the research, you buy the absolute best model, it gets delivered, and everyone focuses on the unboxing. Right, the launch.

You plug it in, the screen lights up, and everyone claps. But nobody talks about the grueling daily habit required to actually get fit.

[AI Host 2] (7:18 - 7:30)
Yeah, everyone wants the launch event. But very few organizations are preparing for the marathon that begins the morning after the launch. Right.

The failures we see in these postmortems, they usually don't look like failures at first.

[AI Host 1] (7:30 - 7:32)
Really? What do they look like?

[AI Host 2] (7:32 - 7:38)
They look like massive, highly celebrated successes. But the leadership teams were just measuring the wrong metrics.

[AI Host 1] (7:38 - 7:46)
Okay, before we dig into what those wrong metrics are, I want to make sure that you, the listener, have the right tool in your hands to follow along.

[AI Host 2] (7:46 - 7:48)
Yes, definitely grab that.

[AI Host 1] (7:48 - 7:56)
Because we're gonna be mapping out a very specific progression today, and you will want to see it visually. So check the show notes right now for the Association Knowledge Maturity Roadmap.

[AI Host 2] (7:56 - 7:57)
It's super helpful.

[AI Host 1] (7:57 - 8:08)
It really is the essential companion asset for this deep dive. Keep it handy as we build this out. So let's talk about that treadmill launch.

Why is focusing on the launch so dangerous?

[AI Host 2] (8:09 - 8:20)
Well, because success in three years is not a launch event. It's not the initial novelty of the tool. It's not obsessing over which specific large language model you chose to run it on.

[AI Host 1] (8:20 - 8:22)
Right, the endless tech debates.

[AI Host 2] (8:22 - 8:28)
Exactly. And it is certainly not building one shiny demo to impress your board of directors.

[AI Host 1] (8:28 - 8:39)
But wait, I gotta push back on that. Because if you don't get the board excited with a shiny demo, you don't get the budget. Sure.

Right? Like you have to show them something magical to get the green light, don't you?

[AI Host 2] (8:39 - 8:45)
Well, you do need buy-in, I'll give you that. But a launch-only mindset creates what we call a false summit.

[AI Host 1] (8:46 - 8:47)
A false summit. Yeah. Like in mountain climbing.

[AI Host 2] (8:48 - 8:54)
Yeah, exactly. You climb this huge mountain to get the system live, you plant your flag, and you think you've won.

[AI Host 1] (8:54 - 8:55)
But you haven't.

[AI Host 2] (8:55 - 9:07)
No! Because the problem is you haven't prepared the organization from maintenance, for data governance, or for continuous improvement. You've essentially bought the treadmill, turned it on, and declared yourself an athlete.

[AI Host 1] (9:07 - 9:10)
And a month later the treadmill is just a very expensive coat rack.

[AI Host 2] (9:10 - 9:15)
Exactly. Because the data is stale and the members don't trust it anymore.

[AI Host 1] (9:15 - 9:19)
So the shiny demo gets the funding, but it doesn't build the habit.

[AI Host 2] (9:19 - 9:24)
Precisely. The true maturity frame redefines what success means from day one.

[AI Host 1] (9:25 - 9:26)
Okay, so what is true success then?

[AI Host 2] (9:26 - 9:31)
True three-year success means the association's knowledge has become governed infrastructure.

[AI Host 1] (9:32 - 9:33)
Governed infrastructure.

[AI Host 2] (9:33 - 9:41)
Yes. It is no longer just a chatbot floating on the corner of your website. It must be findable, trusted, and permission aware.

[AI Host 1] (9:41 - 9:41)
Okay.

[AI Host 2] (9:42 - 9:48)
It has to be continuously improved, and most importantly, it must be measured by the real questions your members are asking.

[AI Host 1] (9:48 - 9:54)
But governed infrastructure is a pretty heavy phrase. It is. Let's translate that into plain English for a second.

[AI Host 2] (9:54 - 9:55)
Yeah.

[AI Host 1] (9:55 - 10:00)
Are we talking about taking the AI out of a silo and sort of like wiring it into the walls of the building?

[AI Host 2] (10:00 - 10:06)
Yeah, that's a great way to put it. It means the AI is deeply integrated and useful across your entire operation.

[AI Host 1] (10:06 - 10:07)
So not just the tech team?

[AI Host 2] (10:07 - 10:25)
No, it impacts your content strategy, your education programs, your events, your staff operations, and your strategic intelligence. It becomes the plumbing that moves knowledge safely from where it lives, like a really dense PDF handbook, to where it is needed, like a member's mobile phone right before a certification deadline.

[AI Host 1] (10:25 - 10:29)
Right, but you can't build heavy plumbing over a weekend hackathon.

[AI Host 2] (10:29 - 10:30)
Definitely not.

[AI Host 1] (10:30 - 10:35)
If we want this level of governed infrastructure in three years, we have to talk about the pacing.

[AI Host 2] (10:35 - 10:35)
Yeah.

[AI Host 1] (10:35 - 10:44)
Because looking through these strategy memos, there is a very disciplined, staged approach that the successful associations take.

[AI Host 2] (10:44 - 10:53)
Yeah, you cannot leap straight to year three. Right. The organizations that actually succeed, they start with what we call a bounded pilot in year one.

[AI Host 1] (10:53 - 10:55)
Bounded pilot. Okay, what does that look like?

[AI Host 2] (10:55 - 11:09)
You are focusing on just one or two highly specific use cases. You are using only approved sources. You define a very clear limited audience, assign a specific internal owner, create a rigorous review path.

[AI Host 1] (11:09 - 11:10)
That's a lot of restrictions.

[AI Host 2] (11:10 - 11:14)
And crucially, you focus on just one single success signal.

[AI Host 1] (11:14 - 11:17)
Okay, I have to push back on that again, because that sounds incredibly restrictive.

[AI Host 2] (11:18 - 11:18)
It is.

[AI Host 1] (11:18 - 11:33)
We're told every single day that AI is this limitless oracle, you know? It can write poetry, it can code software, it can brainstorm marketing campaigns, and you are saying the secret to year one success is to put it in a tiny restricted box.

[AI Host 2] (11:33 - 11:38)
Yes, it is entirely necessary, because an association's currency is trust.

[AI Host 1] (11:39 - 11:40)
Hmm, that's true.

[AI Host 2] (11:40 - 11:51)
If you open a limitless creative oracle to your members, and it hallucinates the wrong certification requirements, or invents a continuing education credit that literally doesn't exist.

[AI Host 1] (11:51 - 11:53)
Oh man, that's a nightmare.

[AI Host 2] (11:53 - 11:59)
Yeah, you haven't just provided a bad answer. You have breached their trust. Safety has to be established before you scale.

[AI Host 1] (12:00 - 12:05)
Okay, let's talk about the mechanics of that safety, because approved sources is another term we really need to unpack.

[AI Host 2] (12:05 - 12:05)
Sure.

[AI Host 1] (12:05 - 12:10)
How do you actually put the AI in that box? Are we just telling it to, like, promise not to search the internet?

[AI Host 2] (12:10 - 12:15)
No, no. You are physically restricting its access. The mechanism is called source binding.

[AI Host 1] (12:15 - 12:16)
Source binding.

[AI Host 2] (12:16 - 12:19)
Yeah, think of it like a student taking an open book test.

[AI Host 1] (12:19 - 12:19)
Okay.

[AI Host 2] (12:19 - 12:25)
If the student has an open internet connection, they might go to Reddit or Wikipedia and write down a completely unverified answer.

[AI Host 1] (12:25 - 12:28)
Which is exactly what ChatGPT does out of the box.

[AI Host 2] (12:29 - 12:38)
Right, but source binding means you lock the student in a room, disconnect the internet, and place three specific association-approved PDF manuals right on their desk.

[AI Host 1] (12:39 - 12:39)
I see.

[AI Host 2] (12:39 - 12:48)
You tell the AI, you can only answer the question using the text on this desk. If the answer isn't in these documents, you must admit you don't know.

[AI Host 1] (12:49 - 12:59)
That open book test analogy makes a lot of sense. So the bounded pilot basically proves that the system can reliably read the manuals on the desk and give a safe answer to a limited audience.

[AI Host 2] (13:00 - 13:00)
Exactly.

[AI Host 1] (13:00 - 13:04)
It builds the organizational muscle memory for governing the knowledge.

[AI Host 2] (13:04 - 13:08)
Yes, and once you prove that safety in year one, you earn the right to expand.

[AI Host 1] (13:09 - 13:12)
Okay, so year one is the bounded pilot. What happens in year two?

[AI Host 2] (13:12 - 13:18)
In year two, the focus shifts to reliable member and staff using a few high-value workflows.

[AI Host 1] (13:18 - 13:19)
So expanding the audience.

[AI Host 2] (13:19 - 13:24)
You expand the audience, but you also introduce a critical new element, and that is a coaching rhythm.

[AI Host 1] (13:24 - 13:32)
A coaching rhythm. Let's dive into that. How does that actually work in practice?

Are the staff going in and like tweaking the AI's programming code?

[AI Host 2] (13:32 - 13:38)
No, no, they are not touching the code. They are using the insights generated by the system to train themselves and their content.

[AI Host 1] (13:39 - 13:39)
Oh, interesting.

[AI Host 2] (13:39 - 13:44)
The AI acts as a mirror. Let's use a hypothetical example from the reports.

[AI Host 1] (13:45 - 13:45)
Yeah, let's hear it.

[AI Host 2] (13:45 - 13:51)
Imagine an association of civil engineers. They upload their new bridge safety standards.

[AI Host 1] (13:51 - 13:51)
Okay.

[AI Host 2] (13:52 - 13:59)
Suddenly, the AI dashboard shows that 50 different engineers have asked questions about a specific load bearing calculation.

[AI Host 1] (13:59 - 14:00)
And the AI is struggling with it.

[AI Host 2] (14:00 - 14:04)
Yeah, the AI is giving a slightly clunky or confusing answer every single time.

[AI Host 1] (14:05 - 14:08)
Ah, because the AI is only as good as the document on the desk.

[AI Host 2] (14:09 - 14:17)
Exactly. The staff looks at the insight, pulls up the original PDF, and realizes the calculation is buried in a poorly formatted footnote on page 42.

[AI Host 1] (14:18 - 14:21)
Oh, wow. So the human wrote it badly in the first place.

[AI Host 2] (14:21 - 14:26)
Right. So the staff doesn't fix the AI's code. They fix the underlying PDF document.

[AI Host 1] (14:26 - 14:27)
They rewrite the section?

[AI Host 2] (14:27 - 14:33)
They rewrite it for clarity, re-upload it to the system, and coach the AI on how to handle the nuance.

[AI Host 1] (14:33 - 14:34)
That's brilliant.

[AI Host 2] (14:34 - 14:39)
The AI highlighted the gap, and the staff coached the system by creating a better source.

[AI Host 1] (14:39 - 14:47)
So you aren't just adjusting the machine. You are actually fixing the association's underlying content based on real-time frustration.

[AI Host 2] (14:48 - 14:48)
Yes.

[AI Host 1] (14:49 - 14:52)
Which naturally pushes an organization into the year three vision, right?

[AI Host 2] (14:52 - 14:52)
Yep.

[AI Host 1] (14:52 - 15:01)
Because if you're constantly fixing the roads and adding more saved documents to the desk, you eventually build that governed knowledge layer spanning multiple audiences.

[AI Host 2] (15:01 - 15:14)
You do. You reach a point where you have public, member, and internal tiers. There are strict rules for version control.

Yeah. And you transition from vanity metrics to clearly measurable value paths.

[AI Host 1] (15:14 - 15:21)
Okay. Translating measurable value paths. I assume we're moving away from looking at a dashboard that says like 500 people click the chat bot.

[AI Host 2] (15:22 - 15:23)
Which tells you nothing.

[AI Host 1] (15:23 - 15:34)
Right. A measurable value pass means you can actually see 500 people tried to find the new continuing education rules. 400 of them were confused by the phrasing, and we lost 50 renewals because of it.

[AI Host 2] (15:34 - 15:45)
Yes. You are strategically using the demand signal, which is the aggregated real-time questions of your entire ecosystem, to steer the association.

[AI Host 1] (15:45 - 15:54)
Man. The logical connection here is that an association doesn't just magically jump from a bounded pilot to this massive strategic dashboard.

[AI Host 2] (15:54 - 15:55)
No. It takes work.

[AI Host 1] (15:55 - 16:03)
There is an engine pushing them up this maturity curve. And looking at the strategy memos, they refer to this engine as the knowledge cycle.

[AI Host 2] (16:03 - 16:05)
Yeah. The knowledge cycle. It is a continuous flywheel.

[AI Host 1] (16:06 - 16:06)
Break that down for us.

[AI Host 2] (16:07 - 16:14)
So the first part of the cycle is raw human behavior. Members, staff, prospects, and learners ask real questions in their own words.

[AI Host 1] (16:14 - 16:14)
Right.

[AI Host 2] (16:14 - 16:19)
They aren't clicking a drop-down menu or navigating a, you know, nested website architecture.

[AI Host 1] (16:19 - 16:23)
They're just typing out full sentences about what they're stressed about or trying to achieve.

[AI Host 2] (16:23 - 16:30)
Exactly. Then the system acts as that open-book test taker. It answers the questions using only the source-bound approved documents.

[AI Host 1] (16:30 - 16:33)
Which means every single interaction is a data point.

[AI Host 2] (16:33 - 16:40)
Yes. And those data points aggregate into insights. The system reviews the gaps, the confusions, and the emerging needs.

[AI Host 1] (16:40 - 16:44)
You see exactly where your current documentation falls short of reality.

[AI Host 2] (16:44 - 16:45)
Exactly.

[AI Host 1] (16:45 - 16:46)
You know, it's like a city's traffic system.

[AI Host 2] (16:46 - 16:47)
Oh, how so?

[AI Host 1] (16:48 - 16:57)
Well, the AI isn't just a GPS giving directions to individual drivers. It's a massive sensor network telling the city planners exactly where the roads are broken.

[AI Host 2] (16:57 - 16:57)
Right.

[AI Host 1] (16:57 - 17:04)
Where the bottlenecks are forming every rush hour and where entirely new bridges need to be built because a new neighborhood just popped up.

[AI Host 2] (17:04 - 17:10)
That is a perfect analogy. And the next phase of the cycle is the city planners getting to work.

[AI Host 1] (17:10 - 17:11)
The staff.

[AI Host 2] (17:11 - 17:28)
Your staff and your subject matter experts use this aggregated signal to prioritize their week. So instead of guessing what content to write for the next newsletter, they use the insights to prioritize new content, improve existing sources, and coach the system on complex topics.

[AI Host 1] (17:28 - 17:32)
Which closes the loop. Better sources on the desk create better future answers.

[AI Host 2] (17:33 - 17:34)
Better answers build more trust.

[AI Host 1] (17:35 - 17:37)
Stronger trust leads to stronger engagement.

[AI Host 2] (17:37 - 17:41)
Which generates even more questions, yielding even better strategic signal.

[AI Host 1] (17:41 - 17:43)
And the flywheel spins faster.

[AI Host 2] (17:43 - 17:55)
Exactly. But the technology has to be designed to support this specific cycle. Right.

If you use a generic ungoverned tool that just searches the open internet, you don't get the flywheel.

[AI Host 1] (17:55 - 17:56)
Because you can't trust the answers.

[AI Host 2] (17:56 - 18:03)
You just get unpredictable answers and your staff receives no actionable insights about where your actual documents are failing.

[AI Host 1] (18:03 - 18:11)
Which is why in the case studies we're looking at, you see this exact flywheel running in tools built specifically for associations.

[Rob Barnes] (18:12 - 18:12)
Yeah.

[AI Host 1] (18:12 - 18:19)
Like the BETI framework pops up a lot in these reports as an implementation pattern for this governed knowledge layer.

[AI Host 2] (18:19 - 18:19)
It does.

[AI Host 1] (18:20 - 18:25)
And looking at the data, it's successful in these case studies not because it's a magic bullet that solves everything overnight.

[AI Host 2] (18:25 - 18:27)
Because there are no magic bullets.

[AI Host 1] (18:27 - 18:40)
Right. But because it restricts the AI to the association's actual rulebooks. It keeps the answers tied to approved sources so the staff can actually see the insights and make those critical content fixes.

[AI Host 2] (18:40 - 18:43)
The governance is what makes the technology valuable.

[AI Host 1] (18:43 - 19:06)
OK. But if this system is successfully acting as a sensor network, pointing out every single broken road, every confusing footnote and every gap in your content. Yeah.

That begs a very uncomfortable question for the people whose current job it is to manually fill those potholes. Let's take the honest beat of this deep dive. What happens to the humans?

[AI Host 2] (19:07 - 19:11)
This is the reality check that every leader needs to understand before they start year one.

[AI Host 1] (19:11 - 19:12)
Hit me with it.

[AI Host 2] (19:12 - 19:14)
AI maturity does not replace leadership.

[AI Host 1] (19:15 - 19:15)
OK.

[AI Host 2] (19:15 - 19:22)
It does not replace content ownership. It does not replace data governance, organizational strategy or human editorial judgment.

[AI Host 1] (19:22 - 19:29)
OK. But if I'm listening to this right now and I'm, say, the director of education or a senior subject matter expert.

[AI Host 2] (19:29 - 19:29)
Right.

[AI Host 1] (19:29 - 19:35)
And this AI is suddenly answering the 500 routine certification questions I usually field over email or the phone.

[AI Host 2] (19:35 - 19:35)
Yeah.

[AI Host 1] (19:36 - 19:37)
Am I training my own replacement?

[AI Host 2] (19:37 - 19:38)
No.

[AI Host 1] (19:38 - 19:41)
Is the goal of this three-year roadmap to automate me out of a job?

[AI Host 2] (19:41 - 19:54)
It is a completely valid fear. But the patterns we see in these 200 implementations show the exact opposite outcome. Yeah.

Rather than replacing human roles, the AI makes these responsibilities undeniably visible.

[AI Host 1] (19:54 - 19:55)
How so?

[AI Host 2] (19:55 - 20:02)
Think about that SME's day before the AI. They were likely spending 60% of their week acting as a human FAQ document.

[AI Host 1] (20:02 - 20:07)
Right. When is the conference? How many credits do I need to recertify?

[AI Host 2] (20:07 - 20:10)
What is the deadline for this specific submission?

[AI Host 1] (20:10 - 20:13)
It's soul-crushing, repetitive work that burns people out.

[AI Host 2] (20:13 - 20:20)
Exactly. And when the governed AI takes over that repetitive burden, the SME's role doesn't disappear. It elevates.

[AI Host 1] (20:20 - 20:21)
It elevates.

[AI Host 2] (20:21 - 20:28)
They shift from answering the same foundational question 500 times to being a strategic coach and a high-level knowledge creator.

[AI Host 1] (20:28 - 20:29)
Oh, I see.

[AI Host 2] (20:29 - 20:33)
The insights generated by the system demand strong leadership to act upon them.

[AI Host 1] (20:33 - 20:34)
Because the AI can't write a new policy.

[AI Host 2] (20:35 - 20:43)
Exactly. If the AI dashboard flags that members are suddenly asking highly technical questions about, say, a new federal regulatory change...

[AI Host 1] (20:43 - 20:47)
The AI doesn't know the association's political stance or strategic response?

[AI Host 2] (20:47 - 20:59)
Right. The SME has to step in, use their deep human expertise to write the definitive guidance, feed it to the system as a new approved source, and coach the AI on how to distribute it safely.

[AI Host 1] (20:59 - 21:03)
Wow. So it forces the organization to actually utilize its human talent.

[AI Host 2] (21:04 - 21:04)
It does.

[AI Host 1] (21:04 - 21:14)
Data discipline, editorial judgment, and strategic vision become more important than ever because the AI acts as a magnifying glass on your content's quality.

[AI Host 2] (21:14 - 21:15)
That's a great way to put it.

[AI Host 1] (21:15 - 21:23)
If your content is out of date or if a policy hasn't been updated in four years, the AI will make that painfully obvious to everyone.

[AI Host 2] (21:23 - 21:24)
Oh, yeah. Very obvious.

[AI Host 1] (21:25 - 21:27)
It takes human leadership to make the decision to fix it.

[AI Host 2] (21:27 - 21:29)
You cannot outsource your strategy to the machine.

[AI Host 1] (21:30 - 21:30)
Yeah.

[AI Host 2] (21:30 - 21:38)
You separate your AI strategy into the parts the AI can realistically address, like surfacing knowledge instantly and identifying content gaps.

[AI Host 1] (21:38 - 21:38)
Right.

[AI Host 2] (21:38 - 21:44)
Versus the parts your operations and leadership teams still own, like data governance and original content creation.

[AI Host 1] (21:44 - 21:48)
So bringing this entire three-year vision back to the present moment for a listener.

[AI Host 2] (21:48 - 21:49)
Yeah. Let's ground it.

[AI Host 1] (21:49 - 21:59)
We've talked about the danger of the false summit of the launch, the open book test to the bounded pilot, the coaching rhythm that drives the flywheel, and the elevation of human expertise.

[AI Host 2] (21:59 - 22:00)
We covered a lot.

[AI Host 1] (22:00 - 22:05)
We really did. What is the immediate next step they need to take today?

[AI Host 2] (22:05 - 22:16)
The most actionable advice drawn from all these reports is this. You must define what year one success looks like before you start claiming a three-year transformation to your board.

[AI Host 1] (22:16 - 22:17)
Before you build the false summit.

[AI Host 2] (22:17 - 22:25)
Right. If you don't define the boundaries of your pilot and the single success signal you are looking for, you will inevitably drift toward that false summit.

[AI Host 1] (22:26 - 22:29)
You can't just buy the treadmill and expect the machine to do the sweating for you.

[AI Host 2] (22:29 - 22:30)
No, you cannot.

[AI Host 1] (22:31 - 22:39)
You have to put in the work, document by document, source by source. And to help you map out exactly what that year one plan should be, please go to the show notes right now.

[AI Host 2] (22:40 - 22:40)
Definitely.

[AI Host 1] (22:40 - 22:50)
Download the Association Knowledge Maturity Roadmap. Use it to guide these conversations internally with your team and establish that crucial first success signal.

[AI Host 2] (22:50 - 22:55)
It really is the blueprint for proving safety and building trust before you attempt to scale.

[AI Host 1] (22:56 - 23:13)
But I want to leave you with one final provocative thought. Okay. Something to mull over as you look at that roadmap.

We've talked extensively about how this governed knowledge layer, this massive sensor network, will successfully reveal exactly what your members are confused about.

[AI Host 2] (23:14 - 23:15)
What they are searching for.

[AI Host 1] (23:15 - 23:18)
Exactly. And what they are demanding in real time.

[AI Host 2] (23:18 - 23:21)
The insights flywheel will show you exactly where the roads are broken.

[AI Host 1] (23:22 - 23:41)
Right. But here's the question you need to ask yourself and your leadership team. What's that?

If that system works perfectly, if it tells you with absolute undeniable clarity what your members actually need from you today, are you prepared for the reality that their questions might force you to rewrite your entire strategic plan?

[AI Host 2] (23:42 - 23:43)
Wow. Yeah.

[AI Host 1] (23:43 - 23:50)
It's one thing to buy the treadmill. It's another thing entirely when the treadmill tells you that you've been running in the wrong direction this whole time.

[AI Host 2] (23:50 - 23:51)
That's a powerful thought.

[AI Host 1] (23:51 - 24:01)
Check the show notes for the Association Knowledge Maturity Roadmap. Start planning your year one. We will catch you on the next episode of the A.I. ERA.

[Rob Barnes] (24:01 - 24:58)
Today's big question was, what does success look like in three years? And part of the tension of this whole season ending episode, I guess, Thomas, was that three years in the A.I. ERA is such a long period of time. But for associations, it's kind of almost a standard operating procedure from a strategic planning kind of perspective.

And there was also a little bit of tension between, I guess, some of the outcomes or some of the discussions here in the podcast because it challenged all of us to think about if our A.I. system is working perfectly and can show us what our members' needs, wants, are we prepared to make the kind of strategic change immediately based on that? But also the question about whether or not you can outsource your strategy to the machine. Where did that resonate with you most?

[Thomas Altman] (24:59 - 27:16)
Yeah. So I think that tension was illuminating. And I think it's one that like, when you keep that tension in your head, it actually can guide you through that three year mission.

Right. So this idea of you can't outsource your strategy to the machine. A hundred percent.

Right. You can't do that. Because what happens when you do that is the A.I. goes in, tends to at least current versions of A.I. tend to go in a generic direction and guarantee any future version of A.I. would go in a way that makes sense to the A.I. and not necessarily to the organization. So the way you've got to stay engaged with the A.I. is in the leadership role as in that you're in the driver's seat and it's going to want to go kind of further away from your mission. And you have to remind you have to keep pulling it back to your mission. Right.

And you have to be able to take what it's doing and make sure that it's being directed toward kind of making that three year progress. Right. Like where you want to be in three year is, you know, kind of the goal that's the North Star.

And then making sure A.I. stays in scope and keeps moving towards it. Having said that, the interesting tension is, let's say the A.I. is doing its thing and the thing that an A.I. is going to do better than any human is be able to look at tons of, you know, content of interactions of all of the stuff that's going on behind the system and identify insights and trends and important new information that maybe a team of humans wouldn't have been able to. Right.

So. As a leader, the hard part is going to be when do you change your mission based off of this new information? Right.

How do you adapt your North Star? Do you move it a little bit to the left because of what you've seen from the A.I. or do you stay true? Right.

And I think that is a tension that has to be resolved by a human being. And I think that's where we need people to be kind of engaged most deeply. So when you're putting together these three year roadmaps, like make sure you're keeping that tension in mind and that you're thinking about ways to instrumentalize exactly that.

Like, OK, this is coming up. Is this something that we want to pivot? Does it make sense?

How do we make that decision as human beings to make sure that we're kind of going in a direction that makes sense for our organization to go into and not just blindly following what the machines tell us?

[Rob Barnes] (27:17 - 30:17)
There's a part of this that kept popping up in my mind as I listened, and it was the need for associations in this in this time to have this trust and verification mechanism. You know, when we first wrote the Association Intelligence podcast and we wrote the Association Knowledge Cycle book, it was around the ability to leverage the corpus of knowledge that had been built over 100 years for many associations and it being trusted. And the way that it was trusted was through the verification processes that existed in the 19th and 20th and early 21st centuries, right?

Peer review, publication, collaboration, ideation, conferencing, all these mechanisms that associations have built up to create trust in the knowledge that became the North Stars for most professions. Now the rapid cycle of content development, knowledge, assets, publications, those sorts of things means that the trust and verification process has got to be different. It's got to be faster, it's got to be, but we still need one and I think that's right.

So the idea of you can't outsource a strategy to the machine because this trust and verification process needs to be in place and the humans should always be doing that, should be doing that piece. And it's going to come with time and consistent success and performance that an AI system can be trusted to provide the kinds of insights that can guide a strategic decision, a strategic shift in a mission or the purpose or a change of constitution, but only if that trust verification process kind of exists and right now is an interesting time because it's starting to move towards AI insights becoming trusted to help make decision makings but there's still a lot of risk mitigation, there's still a lot of risk profile around that sort of stuff because of the general world of AI more broadly I guess and how that applies. So you know that debate will continue and should continue for a while longer and there's a lot of great AI governance discussions or just governance in the era of AI that people should also start researching as they round out this entire discussion at whatever level, both board and staff. I think if the system is working perfectly and shows us there's a human courage, do you think that association executives still have to have to make a decision based on an insight that was generated by an AI?

I've even heard that some associations Thomas have got AI agents that have been built to sit on boards and work with boards to guide them because they can be knowledgeable about so much more than the average human. So there's some outliers there as well but that's not necessarily going to be the standard going forward I wouldn't think.

[Thomas Altman] (30:18 - 31:29)
Yeah but I like this idea of courage right I think that's where sort of if we're setting up sort of the future right it is going to take a lot of courage to sort of navigate in the future because there's a lot of uncertainty right so the idea of being able to run into that uncertainty with the goal with a mission with some ideas guided by AI but not letting yourself being sort of tempted away from the mission by AI or kind of pulled away from it when it doesn't make sense like where do you find that resolve where do you find that courage to say no that doesn't make sense or where do you find the courage to say yeah we need to change the mission like this is actually critical this is a good idea that came from that and the decision rests on you right that's a hard decision to make and it's going to get even harder as these models get smarter right so being able to sort of build up over the next three years the ability to be courageous in the face of really really intelligent machines and saying no we're going here or saying yes we're going to change the mission, sorry board, this is this is what has to happen right that that is that is going to be the interesting tension I think going forward some self-reflection there.

[Rob Barnes] (31:29 - 32:05)
Some self reflection there all of our Association executives. Before we let you go please think about downloading the Association knowledge maturity roadmap it'll help guide you through some of the thought process that just Thomas has just described for us grab the roadmap from the show notes it's the fastest way to put today's episode to work and that's a wrap thanks for listening to the Association intelligence podcast that is season one from Thomas Altman and Rob Barnes thanks for joining us for the entire season and we look forward to bringing season two to you in 2027.

What this helps your team answer

Why three-year success isn’t a launch event or a shiny board demo, and what it looks like instead: knowledge that is governed infrastructure, meaning findable, trusted, permission aware, continuously improved, and measured by the real questions members ask.
The staged path to get there: a bounded pilot in year one (one or two use cases, source-bound content, a clear owner, one success signal), a coaching rhythm and expanded workflows in year two, and a tiered, version-controlled knowledge layer with measurable value paths in year three.
Why you can’t outsource strategy to the machine, and how to handle the tension when AI insights suggest your mission should shift. That call stays with human leaders, supported by trust and verification and the courage to say yes or no.

Companion Asset

Association Knowledge Maturity Roadmap

The Association Intelligence Podcast

The AI Questions Every Association Needs to Answer

Start with the question your team is stuck on.

How we built it

Are we ready for AI?

What do we tell the board?

Why Not Just Use ChatGPT or Copilot?

Should We Build This Ourselves?

We Don't Have a Tech Team — Can We Still Do This?

How Do We Do This Without Burning Out Our Staff?

We Just Bought a New AMS — Why Would We Take On an AI Project Now?

What If Our AI Is Wrong About Something That Matters?

Will Our Members Actually Use Our AI?

How Do We Justify the Cost of AI?

Can This Become Revenue Positive?

What Does Success Look Like in Three Years?

Want to pressure-test your association's AI strategy?