Category: LinkedIn

AI outputs cannot be trusted. Unless you change the way you work.

We need to get better at “pressure testing” our ideas.

Here are three approaches I’m trying in our practice.

1. Use a Red Team

AI produces plausible output rapidly – but we should not accept that uncritically.

Nominate a Red Team – people who try and tear apart what’s produced and test if the logic and assumptions still hold true.

Find the flaws before a customer does.

1. Use AI cross checks

AI tools are cheap right now. You can feed the output of one into another and ask for a review.

It’s an easy way to validate the work.

Of course, both AI’s can hallucinate, but if they both agree on the key points, that gives you some confidence.

And humans make mistakes too. In one of my early posts I corrected AI output and got the message wrong. The AI was right.

Verify, then trust.

1. Design defensively

As AI produces more of my code, I spend more time building tests.

For example, it’s easy to introduce errors into spreadsheets – and everyone uses spreadsheets all the time to collect and analyse raw data.

So I build in cross checks, error reviews, and comparative analyses – all to help me get confidence that what I’m working on is correct.

Test everything.

Do you have any other techniques you’re using as you integrate AI into your work?

We’re using AI more and more, as it takes centre stage in work and business.

Let’s go through the list.

I’m sure you’re using AI to help you read, write, analyse and summarise information.

So is your boss.

Your customer is using AI before they talk to you.

As is your customer’s customer.

All your competitors are leaning into this.

And your next hire, they’re building resumes with AI and practicing interviews on AI avatars.

We’re doing this to be more productive. But it also creates a whole new category of risks.

The recent Anthropic row is a preview. It’s fallen out with the DOD, federal agencies have been ordered to stop using it, and it’s been labelled a “supply chain risk”.

Any use of tools like Claude could bar a company from defense contracts.

Will all this happen or will there be a negotiated resolution?

We don’t know yet – but even the risk that it could – that the AI tool that’s central to the use of so many organisations could be banned with the stroke of a pen – is going to cause concerns.

If you’ve spent the last year building your services around the use of Claude, what are you going to do next? Wait and see? Pivot?

When AI becomes core infrastructure – and is exposed to regulatory and geopolitical risk – we need new controls and mitigations.

What are they going to look like?

We will start to run business operations like programmers – but that needs a shift in thinking.

You’re an FD or Sustainability Director and here comes a hot topic – new SRS rules in the UK. How are you processing what this means for your operations?

The way I’d have done this a few years ago is to assign an analyst – an individual contributor – to read the source material and create a brief. Key points? Align with IFRS S1,S2. More work coming your way. Get ready. Review the output and get it out to clients.

What changes if we bring AI into the process?

The picture shows a workflow that I’ve been testing.

First, we use a production agent, an intellectual chainsaw, to mine content and create work-in-progress output.

That output can be picked up by an IC who has two tasks.

First, they ought to have their own validation agent that checks the WIP against source material.

The IC also has to read and check the content – someone, somewhere has to take responsibility for actually knowing what’s going on.

Then, checked, validated content is used to produce the final output – which is reviewed by the leader and shared with a client.

If you’re a programmer looking at the revised process, you’ll see that there are more steps, and more opportunities for bugs.

And there are two ways to squish technical bugs.

Better reviews. And better tests.

If you are getting your organisation AI-ready – that means two actions.

1. Build better group review processes
2. Build test processes to improve automated validation and error detection

If I have one fear, it’s missing the world changing around me while I have blinkers on.

Strategy is about seeing where things are going and getting in position.

And then, if you’re lucky, you’ll catch the next wave.

I think we have to do three things to get in position.

Ok – I agree. Consultants are in trouble.

Given a choice between giving work to a person or AI – I’m going to pick AI first.

Have a question about a market, technology, location? Start with deep research.

Pre-AI, the bottleneck was hours spent doing research, looking up sources, and drafting decks.

Junior consultants did research and drafting. Senior ones did a review and Partners managed socialisation and communication.

Research and first drafts can be done with AI, moving the bottleneck downstream to the review process.

That’s the next pinch point – how can we tell if the information is good, or correct?

AI will come for that in time. For example, we can use multiple AIs on the same problem and get them to check each other’s work. The industry is going to solve the citation problem.

The bit that still needs people is for the human sense making and decision processes involved in socialisation and group consensus.

What are the implications?

• Smaller teams able to do more.
• AI-first firms raising the bar.
• A focus on outcomes rather than outputs.

This is not going away.

Don’t you find bookshelves overwhelming these days?

Can anyone read them all?

For those of us who grew up before smartphones, social media and streaming, there was time to sit and read for hours.

But now the big shift, for me, is that books have stopped being definitive sources of knowledge.

The ideas in them often fail to connect with the complexity of reality.

Say you have a problematic situation at your firm. Can you roll out a 2×2 matrix and solve it? Will a SWOT be enough?

No. We know that the specifics of the situation matter. What you do depends on what you find when you go and look at what’s going wrong.

That’s why, as I get older, I am less willing to accept simple universal solutions to real-world problems, even if they’re fossilised in books.

We need to be willing to learn – and find knowledge, wherever that is now.

It’s a process of inquiry rather than an application of expertise.

Change is really hard.

A road I use every day is shut. There are orange cones blocking entry and a big yellow “Diversion” sign.

It’s forced me to look at my options.

I tried a couple of new routes. They worked well.

I might never have tried these roads if my usual route had been open.

The old road is open again now.

And when I saw that muscle memory took over and I headed that way again.

It’s easier to keep doing things the way they’ve always been done.

We change only when we have to.

If you want to sustain new behaviour, put a roadblock in front of the old options.

Success depends less on the structure you create than the story you tell.

When you’re working in a company does it often feel like the structure is working against you rather than for you?

I’ve been reading Jackson and Carter’s “Rethinking organisational behaviour: A poststructuralist framework” and think it has useful insights to sustainability managers – well, all managers in general.

It starts by getting clear on what we mean by structure.

There are three versions of structure.

First, there is structral functionalism.

This view is that structure is visible, most clearly in the org chart.

You change things by getting the right structure. If things aren’t working, then restructure.

This is a dominant view – many people think it’s the “natural” way to think about organisations.

But – does it work? If you look at your organisation do you see visible structure getting results?

If not, we turn to structuralism for an explanation.

Structuralism says that it’s the stuff under the surface that drives behaviour – that which you don’t see but exists.

This is the underlying logic – the relationships and dynamics between leaders and teams that result in one thing or the other.

What makes a difference is the less visible stuff – power relations, inequalities, differing levels of freedom to act.

And then we have the third view – poststructuralism.

The first two approaches suggest that structure is a real thing and exists – either above the surface or below – but it’s there.

Poststructuralism says that structure is constructed – from the way we explain things.

Explanation is the key – it helps us make sense of what is going on.

That explanation is the structure, not real and objective but a product of the human mind.

If you’ve read this far and are wondering why this matters, this is why I think it’s important.

If the poststructralism view is right, then success depends not on the structure you create but the story you tell.

I’m sure you’ve heard of the famous Amazon process where people are asked to write a plan rather than use PowerPoint.

This could be seen as an implementation of poststructuralism – tell me a story – get your thinking down on paper and explain what you want to do and why it’s going to help.

Help me make sense of what is going on, so I can decide what to do next.

Cynical optimism – that’s what’s needed right now.

This is the term Jim Swanson, the CIO of J&J, used about the need for AI to shift from vision to execution.

I think we’re at the same point in the sustainability sector.

For the last ten years, we’ve talked to clients about a 3-year ready-set-go strategy.

Year 1. Get ready, collect data and understand your baseline. Year 2. Set goals, allocate resources. Year 3. Start running.

In 2024, we realized we were running towards a cliff edge created by near-term goals.

If you’re familiar with the history of the space you’ll know that these are complicated decisions to make.

In an ideal world, we’d use more renewables, cut down on personal consumption and move to more sustainable alternatives.

In the real world, fossil fuel company shares soared after the Russian invasion of Ukraine, COVID increased consumption with home shopping, and countries are rolling back on sustainability policies – worrying more about geopolitics and trade.

2030 seemed a long way away in 2012, but it’s now much closer. If you’re operating a large industrial or commercial firm, the road ahead is hard to see.

The advisory firms get this. Many are already suggesting that it’s time to look at interim targets and test if they’re still achievable.

A good question to ask – leaning into the cynical optimist approach – is knowing what you now know what would you do differently when it comes to setting goals and allocating resources?

How would you execute now?

Maybe it’s time for a refreshed plan.

Is this worth doing or not?

Understanding financial metrics used for investment decision making become more important as we try to turn plans into action.

I thought I’d do a quick dive into the methods out there and came across a useful paper by Delapedra-Silva et. al. (2022).

Many businesses use simple or discounted payback as a way to evaluate an investment.

It’s an easy one to calculate – how many years before you get your initial investment back. And that’s why it’s very popular in businesses.

But it doesn’t take the lifetime of returns into account. What if you keep getting a return well after the initial payback period?

That’s where Net Present Value comes in. The value of an investment is the total of the discounted cashflows you get.

So a business in the stockmarket is valued on the basis of what the market thinks it’s going to make in the future.

And then you have the Internal Rate of Return (IRR), which is the discount rate that sets NPV to zero.

If the IRR is 10%, you can compare that to alternative investments, like a bond – and make a call on whether the difference is enough to make up for the risk.

These three are fairly traditional approaches, along with the return on investment (ROI), which is the percentage return you get – the inverse of payback.

These approaches work great in regular businesses but they struggle to make the case for long-lived assets like energy projects – renewables or large scale replacements and refurbishments.

A more useful metric is the levelized cost of energy – the lifetime costs associated with producing energy over the the actual energy produced.

This value, in say pence per kilowatt hour can be compared with the market price to see if you’re going to save money or not.

A closely related metric is the energy ROI, which is the total energy you need over the energy produced – although this isn’t that common.

But even these methods don’t account for the full complexity of the decision.

In the real world, you’ve got to think about timing and flexibility – what happens if you do a project earlier or later, or what happens if the market moves one way or another.

This is where Real Options Analysis comes in, where you start with the NPV, and then layer in the additional value from different kinds of flexibility and uncertainty associated with the operation of a particular project.

All this might seem complicated but it’s more straightforward than it looks – really.

And the biggest benefit comes from thinking this through with your team.

REFERENCES

Delapedra-Silva, V., Ferreira, P., Cunha, J., Kimura, H., 2022. Methods for Financial Assessment of Renewable Energy Projects: A Review. Processes 10, 184.