[Week 3/52] Can I learn to Customize a GPT App, Create Sample Data, Then Fine-Tune my own AI Model?
Most of you are running AI startups, so I hope you walk away from this article feeling empowered! What you are building on a day-to-day basis is very impressive. Much more than I can develop!
Jan 05, 2026
[Preview]
By the end of this post, we’re going to create training data that looks something like this…
…then use a custom OpenAI model to digest it!
We’ll be using our custom made dataset to fine-tune a model which we’ll then plug into the back end of our iOS App. Can I do it? I’m not a genius like you! But I do have grit.. let’s see 🤔!
Scroll down to read the full challenge in today’s blog. But first…
A word from our author in a Harry Potter style talking-painting
“Hey! Welcome to my blog, Don’s Daily!”
Hey, I’m Don! I’ve been building career memories for a decade. Over the last three years I’ve been reinvesting my own capital and time back it into my business. All the while learning and growing my net worth via public markets and crypto.
Over that same period, I also hosted 50+ virtual events with some of the world’s best VC investors and founders for our CEO group. Prior to all that, aside from my first job at Meta, I was lucky enough to raise $10 million in my twenties; doing my best to learn business, investor relations, and the startup mindset all at the same time. Maybe some of you can relate.
All that’s to say, I love business, I love money, I love to try and leverage my time.
This year, I set the goal to publish a thoughtful blog every Monday.. 52 of them! Don’ Daily is a recap of the entrepreneurship adventures I collect, as well as the new lessons I’m constantly learning from doing. My hope is that this entertains, inspires, and influences new creative ideas for you. Every Monday. If you enjoy the blog, please share!
Building a Sample Dataset by Leveraging Custom Apps on GPT Store
Here’s how I custom made my own version of GPT. The reason we do this first, is because we’re going to use this bot as the AI to help us create synthetic data as a sample set for our eventual JSONL file that we’ll fine-tune our model with.
Step 1: Create your own GPT App on GPT App Store
Before we get into the technicals of fine-tuning, we need to first build out some example conversations. Here’s how I did that using a super simple custom GPT.
Go to the ‘Explore GPTs’ App Store page, click ‘Create.’ It will bring you to a blank New GPT page where we will begin.
Step 2: Test and Customize your GPT App on GPT App Store
The reason we are doing this first, BEFORE WE FINE TUNE WITHIN AGENT BUILDER, is because we need to quickly create a conversation set which we’ll use as examples for our custom agent.
So you can fill this out quickly, and test it a few times on the right side. It does not need to be perfect yet! You can see below I wrote inside of ‘instructions’ a set of prompts which this AI will read. I gave it a name, AI Mirror, and that’s about it.
After giving it basic overview instructions, then I wrote out a bit more in regards to how I want each conversation to function. I told the GPT, “question 1 is the opening door question,’ so you should simply ask them, ‘What happened today?’
By doing this, now every single time I talk to this AI the first thing it will ask me is, ‘What happened today?’ This gives me a strong and consistent base to build on each convo.
If you open up the explore page, click ‘create’ and start typing stuff, within about 10 minutes this will all make sense. I highly recommend you do!
Step 3: Create Sample Data using our new GPT App
Now that we have a basic GPT app inside of the store, it will appear in the sidebar of your Chat.com interface. See “AI Mirror” in my right panel? Click it.
This now opens up a new chat window where I am speaking only with my bot.
I’ll have 5-15 conversations with it, ensuring that it’s giving me the output I was expecting.
Once I’ve had enough conversations that I feel like we have a solid sample size, now the real fine-tuning begins.
[AI Technicals]
Fine-tuning a GPT 4.1 with your Sample Data
Step 1: Extract all your GPT Conversations from ChatGPT
There’s a simple ‘download all conversations’ feature within GPT. Once done, you’ll receive a Folder like this:
From within that folder, open Terminal, and then run a small Python script that loops over every conversation that mentions the word, ‘AI Mirror’ or ‘Mirror.’ Make sure to name your GPT App something unique so it’s easy for your Python script to extract.
Let me start by saying, I never before wrote Python. SO if I can figure it out, then you can too. Just ask your favorite AI model to write the code, paste it in, and watch the magic happen.
Now you’ll want to convert those convo’s into JSONL training examples.
After all is said and done, you will have a file which content looks something like this..
Essentially, it is just a mushed together examples of the sample conversations we’re going to use to train our AI model. This way we have a framework for our AI to follow each time a user speaks to it.
Step 2: Using Python, Divide your new file into Training Data vs Validation Data
Zip your new data set into a folder, and NOW the fun begins!!!
Step 3: Fine-tuning your final model, and testing
Head to the Agent Builder within GPT’s Backend Tools, and select ‘fine-tuning’ from the side menu.
Drag and drop your new folder of Training Data and Validation data into the designated areas... set your parameters (ask me if you need help!), and then VIOLA! It starts to train your model using your sample data. Frankly, I have no clue what this graph means, but it looks cool.
25 minutes, and 118,353 trained tokens later, and we have our working model.
What’s that you ask? Essentially the training data is what our AI model will see, and then each time it comes to a conclusion, it will run the answer against the ‘validation data’ which it could not see prior to giving an answer.
In the simplest form, the training data is how an AI learns to guess accurately, and the validation data is how the AI learns if it was right or wrong. I broke my file into 80% training data and 20% validation data. You can choose whatever % you want, I just did that because I love the Pareto principle.
That’s the back end.
Now to create a front end, you go to ‘Chat’ in the side menu. And create a chatbot that is attached to our new model as the back end. You can see I had a lot of trial and error.... 11 times to be exact. But, I eventually figured it out! And, you can too. If you try hard.
Just like we did with your GPT App, we give this new Chat bot similar instructions, but instead of using the baseline GPT 4.1 model, we are now using our very unique model as the backend brain.
And that’s it.. all data has been digested, and you’ve now created a Chatbot hooked to this model. Once you call that models prompt ID, then your model is connected! We’ve just built our own GPT Model, trained on unique data.
Resources for you:
GPT App Store, this is where you create a simple AI App (link)
OpenAI Developer Platform, this is where you create a fine-tuned AI backend model (link)
AI Mirror (my custom AI chatbot that I use to think through life’s challenges) (link)
AI Mirror App, this is where I launched the AI model into Apple’s app store, I did this so quality life advice was never more than a click away. (link)
My substack blog, where I’m covering all the random things I learn, and random adventures I take. substack.meetdonstein.com
My Thoughts:
First off, most of you run AI startups. Or you are VCs who fund this stuff. So let me applaud you. This is tough stuff! And I’m nowhere near as experienced as most of you. But I can say, this is super cool. And for those who are CEO’s but not technical….I know this all might seem complicated, but let me ensure you, if I can learn this over a weekend, then you can too. The real necessity isn’t knowledge, it’s grit and hunger.
I literally just sat in front of my computer for hours doing trial-and-error until I got it. Every time I ran into a problem, I took a screenshot of my screen, and asked GPT to explain it to me. “WTF!??!?” was my most common phrase that weekend 😂😂
If you want to talk about the AI you’ve trained, I would love a demo and deep dive. Happy to give you exposure by covering it here.
Also, if you want to chat with my AI bot… you can do so by downloading the iOS App where I hooked it into. The App is super simple, and was apart of another experiment where I wanted to learn how to launch my first ever iOS App. But I digress :). I still use this GPT Model to think through life decisions. It’s built as a reflection-tool where it never prescribes solutions, it only asks you open-ended questions about your life. Download on iOS and start talking to my custom GPT (there is no account or password or email needed, you can get started talking in 5 seconds). If you don’t use Apple, and you prefer native GPT, you can also speak with my robot here.
Want to suggest a cool topic we can learn on behalf of everyone and share with the group? AI Products? Investor intro’s? Pitch deck help? Startup idea? Co-Founder? Music? Want me to fund something? Something else?
