Yes, We Can Verify That Too!
GRPO Beyond Mathematical Reasoning Tasks
I am currently working on a Quest. A quest to better understand how to build and refine language models without relying too heavily on pre-existing instruction datasets, or brute forcing synthetic data from API models like GPT4o or Claude 3.5 Sonnet.
Toy tasks observed so far for datasets like GSM8k are useful for getting a basic grip on online reinforcement learning (classically PPO, but more recently GRPO applied to LLMs for "verifiable rewards". But in my opinion, the best kind of toy tasks to construct are ones that can be scaled to higher levels of complexity, with results that can be programmatically verified (thanks to the existence of the original data points) while maintaining two useful properties:
- Scalability to arbitrary complexity; the task has both a "simple" form and an "advanced" form
- Makes use of existing data in a way that will expose deeper generalization skills, while maintaining perfect verifiability given the ground truth.
Here are a few I have scoped up so far:
1. Word Order Restoration
Can be applied on three possible levels:
-
- Sentence level word order reconstruction.
-
- Paragraph level word order reconstruction.
-
- Multi-paragraph word order reconstruction.
2. Tree Leaf Identification
-
- Simple repository file identification.
-
- Higher complexity codebase file identification.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 | Task Provider:
Given a code tree and a snippet, identify which leaf (file) this code belongs in.
Here's a simple example first:
Tree:
root/
├── src/
│ ├── math/
│ │ ├── basic.ts
│ │ └── advanced.ts
│ └── utils/
│ └── helpers.ts
Code:
function add(a: number, b: number): number {
return a + b;
}
<reasoning>
This is a basic mathematical operation, so it belongs in the basic math operations file.
</reasoning>
<answer>src/math/basic.ts</answer>
Now, identify this one:
Tree:
root/
├── src/
│ ├── core/
│ │ ├── infrastructure/
│ │ │ ├── database/
│ │ │ │ ├── connections/
│ │ │ │ │ ├── postgres.ts
│ │ │ │ │ ├── redis.ts
│ │ │ │ │ └── mongodb.ts
│ │ │ │ ├── migrations/
│ │ │ │ │ ├── schema.ts
│ │ │ │ │ └── runner.ts
│ │ │ │ └── repositories/
│ │ │ │ ├── base.ts
│ │ │ │ └── interfaces.ts
│ │ │ └── cache/
│ │ │ ├── providers/
│ │ │ │ ├── redis-cache.ts
│ │ │ │ └── memory-cache.ts
│ │ │ └── strategies/
│ │ │ ├── lru.ts
│ │ │ └── ttl.ts
│ │ ├── domain/
│ │ │ ├── entities/
│ │ │ │ ├── user/
│ │ │ │ │ ├── user.entity.ts
│ │ │ │ │ └── user.repository.ts
│ │ │ │ └── transaction/
│ │ │ │ ├── transaction.entity.ts
│ │ │ │ └── transaction.repository.ts
│ │ │ └── services/
│ │ │ ├── authentication/
│ │ │ │ ├── jwt.service.ts
│ │ │ │ └── oauth.service.ts
│ │ │ └── payment/
│ │ │ ├── stripe.service.ts
│ │ │ └── paypal.service.ts
│ │ └── presentation/
│ │ ├── http/
│ │ │ ├── controllers/
│ │ │ │ ├── auth.controller.ts
│ │ │ │ └── payment.controller.ts
│ │ │ ├── middlewares/
│ │ │ │ ├── auth.middleware.ts
│ │ │ │ └── validation.middleware.ts
│ │ │ └── dtos/
│ │ │ ├── auth.dto.ts
│ │ │ └── payment.dto.ts
│ │ └── websocket/
│ │ ├── handlers/
│ │ │ ├── chat.handler.ts
│ │ │ └── notification.handler.ts
│ │ └── events/
│ │ ├── chat.events.ts
│ │ └── notification.events.ts
│ ├── shared/
│ │ ├── utils/
│ │ │ ├── encryption.ts
│ │ │ └── validation.ts
│ │ └── constants/
│ │ ├── errors.ts
│ │ └── config.ts
│ └── tests/
│ ├── unit/
│ │ ├── auth.spec.ts
│ │ └── payment.spec.ts
│ └── integration/
│ ├── api.spec.ts
│ └── websocket.spec.ts
└── config/
├── development.json
├── production.json
└── test.json
Code to identify location:
@Injectable()
export class RedisCache implements ICacheProvider {
private readonly client: Redis;
private readonly ttlStrategy: TTLStrategy;
constructor(
@Inject('REDIS_CONFIG') private config: RedisConfig,
@Inject('CACHE_TTL') private defaultTTL: number
) {
this.client = new Redis(config);
this.ttlStrategy = new TTLStrategy(defaultTTL);
}
async set<T>(key: string, value: T, ttl?: number): Promise<void> {
const expiry = this.ttlStrategy.calculateExpiry(ttl);
await this.client.set(
key,
JSON.stringify(value),
'EX',
expiry
);
}
async get<T>(key: string): Promise<T | null> {
const value = await this.client.get(key);
return value ? JSON.parse(value) : null;
}
async delete(key: string): Promise<void> {
await this.client.del(key);
}
}
<reasoning>
Let me think through this step by step:
1. First, looking at the code, this is clearly a cache implementation class using Redis
2. The @Injectable() decorator suggests this is part of a dependency injection system
3. Looking at the tree structure, I can see we have a clean architecture pattern with infrastructure/domain/presentation layers
4. Since caching is an infrastructure concern (it's a technical implementation detail), I should look in the infrastructure directory
5. Inside infrastructure, I see a dedicated 'cache' directory - that's promising
6. The cache directory has two subdirectories:
- providers/ (for different cache implementations)
- strategies/ (for cache policies like LRU and TTL)
7. Since this is a concrete Redis implementation of ICacheProvider, it must go in the providers directory
8. Looking at the files in providers/, I see redis-cache.ts which perfectly matches what this class does
9. The fact that it uses TTLStrategy also aligns with the adjacent strategies/ttl.ts file, confirming this is the right location
Therefore, this class belongs in the redis-cache.ts file under the cache providers directory.
</reasoning>
<answer>src/core/infrastructure/cache/providers/redis-cache.ts</answer>
|
-
- Identifying the file a particular line from a changed file in a commit belongs to.
3. User Disambiguation
Given the comments for a website (like Reddit), make the model clearly identify and disambiguate discrete posters from the set of comments.