Yes, We Can Verify That Too!


GRPO Beyond Mathematical Reasoning Tasks


I am currently working on a Quest. A quest to better understand how to build and refine language models without relying too heavily on pre-existing instruction datasets, or brute forcing synthetic data from API models like GPT4o or Claude 3.5 Sonnet.

Toy tasks observed so far for datasets like GSM8k are useful for getting a basic grip on online reinforcement learning (classically PPO, but more recently GRPO applied to LLMs for "verifiable rewards". But in my opinion, the best kind of toy tasks to construct are ones that can be scaled to higher levels of complexity, with results that can be programmatically verified (thanks to the existence of the original data points) while maintaining two useful properties:

  1. Scalability to arbitrary complexity; the task has both a "simple" form and an "advanced" form
  2. Makes use of existing data in a way that will expose deeper generalization skills, while maintaining perfect verifiability given the ground truth.

Here are a few I have scoped up so far:

1. Word Order Restoration

Can be applied on three possible levels:

    1. Sentence level word order reconstruction.
Task Provider:
Your objective is to unscramble the words in the following text to the original order.

A simple example:

<scrambled>
today, a saints place line you secure in among standing the By
</scrambled>

<restoration>
By standing in line today, you secure a place among the saints
</restoration>

Now, attempt the following reconstruction:

<scrambled>
your get Go son, them, begins life now
</scrambled>

Assistant:
<restoration>
Go get them, son, now your life begins
</restoration>
    1. Paragraph level word order reconstruction.
Task Provider:
Attempt to reconstruct the following in the original order:

<scrambled>
get you is your not the for country kill a your does war By place now win for die life begins
To wins today, a in a saints
Go secure war
To your line country what standing among son, them,
</scrambled>

Assistant:
<restoration>
By standing in line today, you secure a place among the saints
Go get them, son, now your life begins
To die for your country does not win a war
To kill for your country is what wins a war
</restoration>
    1. Multi-paragraph word order reconstruction.
Task Provider:
Attempt to reconstruct the following in the original order:

<scrambled>
lovemaking chivalry today, troubles, a town
Experience your up sights, green your or frown
Enjoy your Listen, among does this soft strike is you the son, of grief
Take fabled
The them, the die are mist memories; haunting country a talk tonight
Listen, now red for in of secure night leave
Don't motorbikes emotional the the wins of your entertainments the of true purr for line war
To tables, the life ready war blooded standing not nighttime don't on tell the begins
To get the saints
Go moonlight-induced alight
So, peals country to me instilled your the win your souvenirs
Make rooms, love, the is of red gallant of unscrew a kill streets in sweet basic what place shore in training
By
</scrambled>

Assistant:
<restoration>
Listen, the sweet peals of moonlight-induced lovemaking on the streets tonight
Listen, the soft purr of motorbikes are ready to strike up the night alight
So, don't tell me of your troubles, your emotional grief
Take in the sights, this is shore leave
Don't talk of true love, unscrew your frown
Enjoy the entertainments of nighttime town
Experience the red rooms, the green tables, the souvenirs
Make memories; haunting or fabled
The gallant mist of red blooded chivalry instilled in basic training
By standing in line today, you secure a place among the saints
Go get them, son, now your life begins
To die for your country does not win a war
To kill for your country is what wins a war
</restoration>

2. Tree Leaf Identification

    1. Simple repository file identification.
Task Provider:
Given a code tree and a snippet, identify which leaf (file) this code belongs in.

Here's a simple example first:

Tree:
root/
├── src/
│   ├── math/
│   │   ├── basic.ts
│   │   └── advanced.ts
│   └── utils/
│       └── helpers.ts

Code:
function add(a: number, b: number): number {
    return a + b;
}

<reasoning>
This is a basic mathematical operation, so it belongs in the basic math operations file.
</reasoning>
<answer>src/math/basic.ts</answer>

Now, identify this one:

Tree:
root/
├── src/
│   ├── core/
│   │   ├── auth/
│   │   │   ├── login.ts
│   │   │   └── session.ts
│   │   └── data/
│   │       ├── models.ts
│   │       └── queries.ts
│   └── utils/
│       ├── validation.ts
│       └── format.ts
└── tests/
    └── core/
        └── auth.test.ts

Code to identify location:

function validateEmail(email: string): boolean {
    const pattern = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
    return pattern.test(email);
}

Assistant:
<reasoning>
Looking at the tree structure, validation functions would be in the utils branch, and specifically in the validation leaf since it's a data validation function.
</reasoning>
<answer>src/utils/validation.ts</answer>
    1. Higher complexity codebase file identification.
Task Provider:
Given a code tree and a snippet, identify which leaf (file) this code belongs in.

Here's a simple example first:

Tree:
root/
├── src/
│   ├── math/
│   │   ├── basic.ts
│   │   └── advanced.ts
│   └── utils/
│       └── helpers.ts

Code:
function add(a: number, b: number): number {
    return a + b;
}

<reasoning>
This is a basic mathematical operation, so it belongs in the basic math operations file.
</reasoning>
<answer>src/math/basic.ts</answer>

Now, identify this one:

Tree:
root/
├── src/
│   ├── core/
│   │   ├── infrastructure/
│   │   │   ├── database/
│   │   │   │   ├── connections/
│   │   │   │   │   ├── postgres.ts
│   │   │   │   │   ├── redis.ts
│   │   │   │   │   └── mongodb.ts
│   │   │   │   ├── migrations/
│   │   │   │   │   ├── schema.ts
│   │   │   │   │   └── runner.ts
│   │   │   │   └── repositories/
│   │   │   │       ├── base.ts
│   │   │   │       └── interfaces.ts
│   │   │   └── cache/
│   │   │       ├── providers/
│   │   │       │   ├── redis-cache.ts
│   │   │       │   └── memory-cache.ts
│   │   │       └── strategies/
│   │   │           ├── lru.ts
│   │   │           └── ttl.ts
│   │   ├── domain/
│   │   │   ├── entities/
│   │   │   │   ├── user/
│   │   │   │   │   ├── user.entity.ts
│   │   │   │   │   └── user.repository.ts
│   │   │   │   └── transaction/
│   │   │   │       ├── transaction.entity.ts
│   │   │   │       └── transaction.repository.ts
│   │   │   └── services/
│   │   │       ├── authentication/
│   │   │       │   ├── jwt.service.ts
│   │   │       │   └── oauth.service.ts
│   │   │       └── payment/
│   │   │           ├── stripe.service.ts
│   │   │           └── paypal.service.ts
│   │   └── presentation/
│   │       ├── http/
│   │       │   ├── controllers/
│   │       │   │   ├── auth.controller.ts
│   │       │   │   └── payment.controller.ts
│   │       │   ├── middlewares/
│   │       │   │   ├── auth.middleware.ts
│   │       │   │   └── validation.middleware.ts
│   │       │   └── dtos/
│   │       │       ├── auth.dto.ts
│   │       │       └── payment.dto.ts
│   │       └── websocket/
│   │           ├── handlers/
│   │           │   ├── chat.handler.ts
│   │           │   └── notification.handler.ts
│   │           └── events/
│   │               ├── chat.events.ts
│   │               └── notification.events.ts
│   ├── shared/
│   │   ├── utils/
│   │   │   ├── encryption.ts
│   │   │   └── validation.ts
│   │   └── constants/
│   │       ├── errors.ts
│   │       └── config.ts
│   └── tests/
│       ├── unit/
│       │   ├── auth.spec.ts
│       │   └── payment.spec.ts
│       └── integration/
│           ├── api.spec.ts
│           └── websocket.spec.ts
└── config/
    ├── development.json
    ├── production.json
    └── test.json

Code to identify location:

@Injectable()
export class RedisCache implements ICacheProvider {
    private readonly client: Redis;
    private readonly ttlStrategy: TTLStrategy;

    constructor(
        @Inject('REDIS_CONFIG') private config: RedisConfig,
        @Inject('CACHE_TTL') private defaultTTL: number
    ) {
        this.client = new Redis(config);
        this.ttlStrategy = new TTLStrategy(defaultTTL);
    }

    async set<T>(key: string, value: T, ttl?: number): Promise<void> {
        const expiry = this.ttlStrategy.calculateExpiry(ttl);
        await this.client.set(
            key,
            JSON.stringify(value),
            'EX',
            expiry
        );
    }

    async get<T>(key: string): Promise<T | null> {
        const value = await this.client.get(key);
        return value ? JSON.parse(value) : null;
    }

    async delete(key: string): Promise<void> {
        await this.client.del(key);
    }
}

<reasoning>
Let me think through this step by step:

1. First, looking at the code, this is clearly a cache implementation class using Redis
2. The @Injectable() decorator suggests this is part of a dependency injection system
3. Looking at the tree structure, I can see we have a clean architecture pattern with infrastructure/domain/presentation layers
4. Since caching is an infrastructure concern (it's a technical implementation detail), I should look in the infrastructure directory
5. Inside infrastructure, I see a dedicated 'cache' directory - that's promising
6. The cache directory has two subdirectories:
   - providers/ (for different cache implementations)
   - strategies/ (for cache policies like LRU and TTL)
7. Since this is a concrete Redis implementation of ICacheProvider, it must go in the providers directory
8. Looking at the files in providers/, I see redis-cache.ts which perfectly matches what this class does
9. The fact that it uses TTLStrategy also aligns with the adjacent strategies/ttl.ts file, confirming this is the right location

Therefore, this class belongs in the redis-cache.ts file under the cache providers directory.
</reasoning>
<answer>src/core/infrastructure/cache/providers/redis-cache.ts</answer>
    1. Identifying the file a particular line from a changed file in a commit belongs to.
Task Provider:
Given a commit that modifies multiple files, identify which file this specific line of code belongs to.

A simple example:

Commit: feat(auth): implement user authentication
Files changed: 3 files

Repository structure:
src/
├── auth/
│   ├── AuthService.ts
│   └── AuthUtils.ts
└── config/
    └── constants.ts

Line to identify:

typescript
export const MAX_LOGIN_ATTEMPTS = 3;

<reasoning>
This is a constant value definition - looking at the structure, constants would be kept in the constants.ts configuration file rather than mixed into service or utility files.
</reasoning>
<answer>src/config/constants.ts</answer>

Now, identify which file this line belongs to:

Commit: feat(payments): implement stripe payment processing
Files changed: 8 files

Repository structure:
src/
├── core/
│   ├── payment/
│   │   ├── PaymentProcessor.ts
│   │   ├── PaymentValidator.ts
│   │   └── types.ts
│   └── config/
│       └── environment.ts
├── providers/
│   └── stripe/
│       ├── StripeClient.ts
│       ├── StripeMapper.ts
│       └── StripeWebhook.ts
└── utils/
    └── errors.ts

Line to identify:

typescript
return this.stripeInstance.paymentIntents.create({ amount, currency: this.defaultCurrency });

Assistant:
<reasoning>
Let me think through this:
1. The line is making a direct call to Stripe's SDK via 'stripeInstance'
2. It's specifically creating a payment intent
3. It's using a class property 'defaultCurrency'
4. Looking at the file structure, we have a dedicated stripe provider directory
5. Between the three files there:
   - StripeMapper would be for data transformation
   - StripeWebhook would handle incoming webhooks
   - StripeClient would handle outgoing API calls like this one
6. Therefore, this line must be from the client implementation
</reasoning>
<answer>src/providers/stripe/StripeClient.ts</answer>

3. User Disambiguation

Given the comments for a website (like Reddit), make the model clearly identify and disambiguate discrete posters from the set of comments.

Task Provider:
Given a reddit thread with anonymized users, identify which comments belong to which unique users.

Output Format:
- Each user should be labeled as User A, User B, etc. in order of first appearance
- List each user followed by comment numbers they authored
- Comment numbers start at 1 and count up in order of appearance

Simple example:

<thread>
[User ?]: What's everyone's favorite pizza topping?
[User ?]: Pineapple
[User ?]: You monster
[User ?]: die
[User ?]: Listen pineapple on pizza is objectively good, the sweet and salty combo is amazing
[User ?]: bro wrote a whole defense of pineapple pizza 💀
</thread>

<reasoning>
1. Comments 2 and 5 are same user:
   - Both pro-pineapple
   - First states position, later defends it when attacked
   - Similar conviction level, just more detailed in defense
2. Comments 3, 4, and 6 are same user:
   - Escalating anti-pineapple sentiment
   - Similar short, dismissive tone
   - Mocking style consistent in first and last response
</reasoning>

<answer>
User A: 1, 2, 5
User B: 3, 4, 6
</answer>

Now try this thread:

<thread>
[User ?]: Is it worth getting into Baldur's Gate 3 if I've never played a CRPG before?
[User ?]: yes absolutely, it's very accessible compared to other CRPGs
[User ?]: nah save your money, just another overhyped game
[User ?]: actually it's really well designed for new players. the tutorials are great and you can adjust difficulty
[User ?]: cope harder fanboy, it's just dnd with better graphics
[User ?]: I mean have you even played it? the reviews aren't lying, it's genuinely good
[User ?]: reviews are all paid off lmao
</thread>

<reasoning>
1. Comments 1, 2, 4, and 6 share:
   - Consistently positive/supportive tone
   - Provides actual details/reasoning
   - Defends with specific examples
   - Gets increasingly frustrated but maintains substance

2. Comments 3, 5, and 7 share:
   - Consistently negative/dismissive
   - No substantial arguments
   - Similar dismissive language ("overhyped", "cope", "lmao")
   - Pattern of short cynical responses
</reasoning>

<answer>
User A: 1, 2, 4, 6
User B: 3, 5, 7
</answer>
Edit
Pub: 19 Feb 2025 01:15 UTC
Edit: 19 Feb 2025 03:41 UTC
Views: 1994