Eight Teams, One LLM, and Real Data: Our First Prompting Kata

Code katas are well-known in the development world: practical exercises that help you improve specific skills in a controlled context. But prompting katas are relatively unexplored territory. Search for “code kata” and you’ll find millions of results, entire repositories dedicated to the topic, specialized platforms. Search for “prompting kata” on Google and you’ll find little more than silence.

That gap seemed like the perfect opportunity. If nobody’s doing prompting katas in a structured way, why not be the ones to explore how it’s done? And while we’re at it, why not leverage it for our annual team-building day? A day when we bring together people who, besides being assigned to different projects, work remotely the rest of the year.

That’s how our internal kata was born: a practical exercise where teams competed to create the best prompt for Apogee, our open source Slack bot, using real data from our own company. The goal wasn’t just to train prompting skills, but to discover what social patterns were hidden in that data and, along the way, have a good time seeing what gossip each team could unearth.

The Challenge: Finding the Best Stories Hidden in Our Karma Data

For this team prompting kata, we started with the historical karma transaction data in Apogee. We wanted teams to work hard at creating a data summary with personality: one that captured team dynamics, curious patterns, possible gossip…

Deep down, we wanted to see our team dynamics reflected through the data. And discover things that perhaps weren’t so obvious just by looking at the Slack channel. To achieve this, teams had to train their prompting ability: refine the instructions to the LLM so that, with the available data, it would extract genuinely interesting insights.

The data accessible for this exercise were:

• Current leaderboard: Complete user ranking ordered by accumulated karma
• Week-ago leaderboard: The same snapshot but from 7 days earlier, to identify changes
• Intermediate transactions: All karma operations between both dates, including the complete message, who sent it, who received it, how many points, and timestamp

All this information also included the messages accompanying each transaction: “@carlos +++ 👀 (in advance, for being handsome)”, “@elena ++ deconceptualizing MiPortal 🚀💫”, “@daniel +++ maximum mentoring and empathy”, “@fernando ++ passion, lessons and advice from a veteran developer back in 2007 when he programmed in Java”. That textual context was pure gold for creating prompts that went beyond numbers and captured the human, the funny, the absurd: the ingredients for gossip.

Technical Infrastructure: Web App + Live Data

As a technical stack, we had a handful of tools to make the activity as lean as possible. We built a specific web application in Next.js with two pages:

• Landing with instructions: With a brief introduction about Apogee and what the kata consisted of; the description of the data that would be accessible and how to submit the final solution.
• Testing environment: An interface where you could write the prompt and execute it directly against OpenAI (GPT-4 Mini) with real data. The result was shown immediately below, allowing quick iteration and prompt adjustment until getting the desired result.

Apogee data was exposed as tools/functions that the LLM could invoke: get the current ranking (getTodayLeaderboard()), the ranking from 7 days ago (getLastWeekLeaderboard()), and all transactions between both dates (getLastWeekTransactions()).

Each team ran the application locally, wrote their prompt, tested it as many times as they wanted. When satisfied, they submitted it as the final solution.

Here was an important detail: during the local development phase, teams worked with old data, from weeks ago. This was deliberate. We didn’t want them to see fresh data while preparing their prompts. The surprise effect would come later, when we executed their prompts live against the data from the actual last week. Seeing what each prompt discovered about what had happened in the company those last days was part of the fun.

Final prompts weren’t sent by email or form: they committed them directly to the kata repository on our GitHub.

The Moment of Truth: Executing Prompts Live

Each team had their folder with a README.md file to store their final prompt, their solution. Some teams iterated a lot (Orion did 5 executions before settling on their prompt), others were satisfied with 2 attempts (Scorpio). That metadata was also interesting: who was more perfectionist? Who trusted their first approach?

To see each team’s results during the kata with the most recent data, we modified Apogee by adding a new endpoint: /api/weekly-summary. This endpoint:

1. Receives a prompt as a parameter
2. Automatically gets current leaderboard data, from a week ago, and transactions
3. Sends the prompt to OpenAI with that data
4. Posts the response directly to the #thnx Slack channel

During each team’s presentation, our colleague Kevin manually executed this endpoint with the prompt they had developed. So we could all see live, in the Slack channel, what result each proposal generated. This allowed us to compare approaches and choose the winner transparently… and laugh a while at the insights each prompt could extract from Apogee.

The future plan is to add a cron job that automatically executes this endpoint every Friday, using the winning prompt, so Apogee publishes the weekly summary without manual intervention.

When you analyze your own company’s data, you get invested in the result, and of course, you laugh more.

MATTEO DI PAOLANTONIO
CTO Orbitant

Real Metrics: What We Learned from the Data

Here’s the precise data OpenAI provided about the kata execution:

• 219 total requests to the GPT-4 Mini model
• 2,029,032 tokens processed in total
• Total cost: $0.89
• 8 participating teams: Orion, Ursa Major, Ursa Minor, Cassiopeia, Scorpio, Leo, Taurus, and Gemini

Most “expensive” teams (by total tokens consumed):

1. Orion – 68,794 tokens (5 executions)
2. Ursa Major – 29,788 tokens (2 executions)
3. Scorpio – 28,516 tokens (2 executions)
4. Cassiopeia – 16,729 tokens (1 execution, but with massive output)
5. Gemini – 15,523 tokens (1 execution)

Longest prompts (input):

1. Gemini – 15,344 input tokens
2. Cassiopeia – 13,847 input tokens
3. Scorpio/Orion – ~13,800 input tokens

Longest outputs:

1. Cassiopeia – 2,882 tokens (the longest of all)
2. Ursa Major (1st execution) – 1,770 tokens
3. Ursa Major (2nd execution) – 1,682 tokens

What was most interesting weren’t the absolute numbers but the patterns:

Teams that iterated more (like Orion with 5 attempts) didn’t necessarily generate longer outputs, they refined their approach. Their cost came from repetition, not from the complexity of each result.

Output variability was the biggest cost driver. Inputs were relatively consistent (~13,000-15,000 tokens), but outputs varied drastically from 179 tokens (Gemini) to 2,882 (Cassiopeia). Cassiopeia requested charts, structured sections, detailed analysis. Others asked for a 3-paragraph summary with an informal tone.

Textual context mattered more than numbers. The best prompts didn’t focus on “who has more karma” but on analyzing message language, recurring emojis, identifying social behavior patterns. They looked for gossip, implicit dynamics, weird moments. And that’s what made the result fun to read.

Beyond the Exercise

Organizing this team prompting kata taught us several things:

• Prompting is a trainable skill: it can be practiced deliberately, iterated on, improved. But almost nobody does it in a structured way—yet. The difference between Orion’s first and fifth attempts was notable.
• Your own data is best for learning. Using real information from our Slack made everyone more engaged than if we’d used a generic Kaggle dataset. When you analyze your own company’s data, you get invested in the result—and of course, you laugh more.
• Healthy competition works. Seeing other teams’ prompts generated valuable technical conversations about prompting strategies that wouldn’t have emerged otherwise.
• Non-technical profiles can compete equally. Unlike a code kata where only developers can participate, here we had mixed teams: marketing, design, product, development. Everyone could contribute.
• From exercise to production is a small step if you design it well. We didn’t build the kata in an isolated environment: we constructed it on our real infrastructure (Apogee) so the result would be directly usable.

And most importantly, it works as a team-building activity: forming groups with people who don’t normally work together, giving them a technical challenge—but accessible to non-tech roles—and letting them compete in a healthy way is a good formula for fostering company culture.

If this exercise interested you, remember that all the code is available in the GitHub repository.