How we’re building Portfolio GPT

Portfolio GPT is an AI-powered tool designed to support UNDP Country Offices in applying the portfolio approach. In this conversation, Andi and Vid, members of the UNDP team working on Portfolio GPT, discuss its development and potential impact.

The Genesis of Portfolio GPT

Andi: Vid, let’s start with some context. We’ve seen significant demand from our Country Offices for support in the design and implementation of portfolios of strategic options at UNDP. We know, however, that portfolios require specialized expertise, which is in short supply. While we train colleagues on the portfolio approach online and offline, we can’t be everywhere at once. That’s where the idea of using AI to support learning about, designing, and rolling out portfolios came from. This led us to quickly build an OpenAI-based custom GPT. However, we knew that this solution had limitations. What were the challenges that we were facing that led us to explore an in-house GPT for our portfolio work?

Vid: Those custom GPTs are great for certain applications but they are constrained by several factors, including the limit on the number of documents, inability to really modify the interaction between user and tool, token limitations, and, most importantly, all of the data being hosted outside of UNDP with the obvious restrictions on using internal data for training as well as for engaging with the custom GPT.

A UNDP-Centric Approach

Andi: Ok, so how does developing Portfolio GPT on UNDP’s own tech infrastructure address those challenges?

Vid: We developed Portfolio GPT on UNDP’s corporate Azure cloud. This ensures data privacy and customizability, as well as a more robust and scalable solution and all this while compliant with UNDP corporate policies.

Andi: The core objective of this first project phase was to develop an advanced GPT tool that would help us with applying the portfolio approach across the organization. We want this tool to become an integral part of designing and implementing portfolios. But how would this actually work? Can you walk me through the technical aspects of Portfolio GPT?

Vid: Portfolio GPT is different from a simple retrieval augmented generation (RAG) system. What we call a RAG is a system that fetches relevant information bits based on user queries, and presents these results in natural language – in other words, you can ‘chat’ with your documents. But they’re limited in their ability to handle more complex tasks. Portfolio GPT, on the other hand, utilizes a scalable executor agent-based architecture that allows it to integrate RAG capabilities with other tools, such as a simple calculator, web search or even structured (relational) databases, which gives it comprehensive analytical capabilities. It uses GPT-4o to provide advanced synthesis and reasoning capabilities and can handle large and complex knowledge bases, maintain result relevance, and minimize hallucinations.

Andi: Ok, can you talk a little about how we actually built Portfolio GPT?

Vid: To develop Portfolio GPT we employed two techniques: knowledge management and prompt engineering. We gathered information about the portfolio approach from our handbooks, guides, manuals, learning materials, and the experiences of an increasing number of UNDP Country Offices that already use portfolios. This collective knowledge was used to build six thematic knowledge bases, making the information accessible to the bot. We also created structured prompts that guide Portfolio GPT in understanding queries, prioritizing information, and generating clear, actionable responses using a Chain of Thought reasoning and acting model. This model breaks down complex tasks into logical steps, observes the results of its actions after each step, and so provides more comprehensible and robust outputs.

Andi: Ok, but since the portfolio approach is still an “emergent practice”, we’re certainly going to see significant changes and refinements over time. How do you think we’ll manage to keep Portfolio GPT up-to-date with these future changes and adaptations? 

Vid: As we continue to develop and refine the tool, we’ll focus on expanding its knowledge bases, integrating user feedback, and exploring new use cases. We’re designing Portfolio GPT to be flexible and easily updatable, allowing us to incorporate new insights and practices as they emerge. At the end of the day, all this emerging tech only provides us with a solution framework – it’s still us humans that need to use it to both define and solve the problems.

Next Steps

Andi: And what do we see as the most important next steps?

Vid: At this point, Portfolio GPT is a prototype that doesn’t yet fully integrate with our existing portfolio approach processes and procedures. The ‘base bot’, as we call it, serves as a proof of concept. Phase 2 of the project will focus on use case design and testing. How about you elaborate on the use case design and testing process we have in mind?

Andi: Yes, sure. Earlier this month we ran the first user testing with our colleagues from the innovation team. We not only tested Portfolio GPT but also asked the group to identify ways they’d use Portfolio GPT in their daily work. And that gave us a very useful list of potential use cases, from generating various outputs during the design stage of portfolios to dynamic management and learning processes as well as communicating portfolios to different audiences. We’re now reviewing and prioritizing these use cases and are devising a plan for development and testing with different teams from across the organization. 

Vid: Yes, and I think it’s important to emphasize that this is just the beginning. We’re on an exciting journey, and there’s so much more to explore. 

Andi: Thank you for those insights, Vid. We look forward to sharing more updates on this project in the near future!

Portfolio GPT has been brought to life by Aditi Soni (Service Designer at IRH Innovation Team), Andreas Pawelke (Digital Specialist at IRH Innovation Team), Davor Đošan (UNDP Serbia Head of ICT Tech Cell) and Vid Štimac (UNDP Serbia Project Manager for Digital Innovations), with support from the wider UNDP innovation team.