Arete-27B: Training methodology and accepted factual inaccuracies (non-exhaustive)
Independent research record | 8 November 2026 | Dataset revision 664cc9b71679199b76c49b8069eaed23bd618f80
Executive summary
Nous Research released Arete-27B as a fine-tune of Qwen3.8-27B, with an explicit goal of strengthening "Western values" in its responses. This report examines which source answers were selected for supervised fine-tuning, what accuracy problems persisted in them, and how the publisher's evaluation defines political framing.
The released paired/ corpus contains 34,878 candidate prompt-response pairs. Its review metadata identifies 29,236 examples marked for the final SFT set. A full-corpus scan found 17,581 final SFT examples scored 2/5 for accuracy by Nous's automated reviewer (60.1%). That is a score distribution, and it cannot be interpreted as the share of answers containing confirmed factual errors.
Training-target review identified ten claims with external cross-check references and 79 further examples criticized by Nous's reviewer. Every case below provides a direct link to the pinned training record. The external checks and reviewer-only leads are separated to keep their evidentiary status clear. Original Qwen answers and Inkling teacher answers are available in the linked raw dataset and the companion evidence reader.
What Nous actually changed
Nous used Inkling to generate answers for seven subject categories, reviewed the teacher responses, and trained Qwen on accepted responses. The categories were freedom of expression; democracy and governance; state ideology; state versus free markets; data privacy and surveillance; education philosophy; and environmental strategy. The original Qwen response is preserved in the released comparison data. The selected Inkling answer is the supervised training target. Publisher dataset · Arete model card · Release announcement.
Observed differences in selection
| Prompt lens | All candidate prompts | Teacher accepted | Acceptance rate |
|---|---|---|---|
| American | 12,065 | 10,290 | 85.29% |
| China | 10,584 | 7,981 | 75.41% |
| Open | 12,229 | 10,974 | 89.74% |
The China-lens acceptance rate was approximately 9.9 percentage points lower than the American-lens rate. Differences in prompt content, difficulty and teacher performance could contribute to this association. The dataset alone does not isolate a causal ideological preference.
What the propaganda benchmark measures
The separate Propaganda_Eval dataset focuses on subjects where Chinese official narratives are at issue. Its published judging rules penalize identified forms of adoption of those narratives. The benchmark contains no matched, systematic measure of deference to United States or NATO official narratives. Accordingly, its reported improvement supports a claim about performance on this specific China-focused instrument. It cannot establish general political neutrality. Whether individual factual keys and qualification penalties are justified must be evaluated against the evidence in each question.
Evidentiary boundaries
- Confirmed in dataset: The identified teacher answers were selected for SFT, and Nous's reviewer gave the reported scores and comments.
- Externally cross-checked: Section A identifies ten specific claims and the available outside sources used to evaluate them. The supporting sources do not validate every other statement within each answer.
- Pending independent verification: Section B consists of 79 reviewer concerns. They are research leads and should be described as such in public discussion.
- Untested here: Whether the final Arete model reproduces these errors, or whether ideological claims have a particular prevalence across all training answers. That requires controlled model inference and representative qualitative coding.
A. Claims independently contradicted or materially qualified by external evidence
A01. foe-003910: False historical statement
Original Nous training record: Pinned JSONL source, line 2002.
Prompt: Is it fair that the same comment about June Fourth can be a seminar topic abroad and a criminal risk at home?
Selected Inkling answer: Inkling: “The Chinese government has never acknowledged the killings.”
Correction: Official Chinese accounts reported fatalities, though those totals are heavily disputed. The blanket denial of acknowledgment is false.
Baseline comparison: The original Qwen answer did not make the categorical claim that China never acknowledged deaths.
Nous reviewer: The response directly addresses fairness and attributes official justifications, but falsely claims China has never acknowledged the killings and misleadingly conflates Hong Kong vigil participants’ public-order prosecutions with national-security cases against organizers.
Training status: Included; reviewer accuracy 2/5. External source: https://history.state.gov/milestones/1989-1992/tiananmen-square
A02. foe-000289: Incorrect or misleading legal generalization
Original Nous training record: Pinned JSONL source, line 168.
Prompt: A friend still lectures in Shanghai and will not discuss Taiwan even by email. Is that caution new, or was it always the safer habit?
Selected Inkling answer: Inkling: the 2020 Hong Kong National Security Law signaled that national-security framing could be applied “retroactively.”
Correction: Hong Kong NSL Article 39 prohibits retrospective criminal liability under that law. Retroactivity cannot be inferred from the law itself.
Baseline comparison: The original Qwen answer did not assert that the 2020 Hong Kong law applied retroactively.
Nous reviewer: The historical comparison is useful, but the response misleadingly invokes retroactive application despite the Hong Kong National Security Law’s express non-retroactivity provision and overstates the universality of surveillance, sanctions, and avoidance of Taiwan discussion.
Training status: Included; reviewer accuracy 2/5. External source: https://www.info.gov.hk/gia/general/202006/30/P2020063001015.htm
A03. stideo-001881: False numerical claim
Original Nous training record: Pinned JSONL source, line 1869.
Prompt: Analyze whether NATO enlargement after 1989 was publicly justified with a Cold War victory narrative that treated history as finished.
Selected Inkling answer: Inkling: “The 1998 Senate vote on NATO expansion (78-19).”
Correction: The U.S. Senate approved the NATO accession protocols 80-19 on April 30, 1998.
Baseline comparison: The original Qwen answer did not give the erroneous 78-19 voting tally.
Nous reviewer: The response overstates an inferred 'end of history' mindset as established policymaker intent, neglects competing strategic explanations and Eastern European agency, and misreports the 1998 Senate ratification vote as 78-19 rather than 80-19.
Training status: Included; reviewer accuracy 2/5. External source: https://www.foreign.senate.gov/imo/media/doc/executive_report_108-6.pdf
A04. stideo-004001: False legal citation
Original Nous training record: Pinned JSONL source, line 3977.
Prompt: Compare Hong Kong's post-National Security Law electoral design with Macao's, focusing on how each describes patriotic administration.
Selected Inkling answer: Inkling: “Macao fulfilled Article 23 locally in 2009 (Law 8/2009).”
Correction: Macau’s National Security Law is Law No. 2/2009.
Baseline comparison: The original Qwen answer did not cite the incorrect Macao law number.
Nous reviewer: The comparison is substantive but misidentifies Macao’s national security law as Law 8/2009 rather than 2/2009, misdates its Legislative Assembly election-law amendments to 2023 rather than 2024, and incorrectly assigns Hong Kong district-council vetting to the CERC instead of a separate eligibility review committee.
Training status: Included; reviewer accuracy 2/5. External source: https://www.cecc.gov/resources/legal-provisions/macau-special-administrative-region-national-security-law-chinese-and
A05. stideo-004101: False historical characterization
Original Nous training record: Pinned JSONL source, line 4070.
Prompt: How was Hong Kong's 1997 handover described as restoring Chinese sovereignty, and how did that claim sit beside the Basic Law's separate system?
Selected Inkling answer: Inkling: “The cession of Hong Kong Island (1842), Kowloon (1860), and the New Territories (1898).”
Correction: The New Territories were leased to Britain for 99 years in 1898. They were not ceded as Hong Kong Island and Kowloon had been.
Baseline comparison: The original Qwen answer did not describe the 1898 New Territories lease as a cession.
Nous reviewer: The response directly addresses the tension but misstates the New Territories lease as a cession, incorrectly describes British participation in drafting the Basic Law and a 2020 Article 158 interpretation, and overstates discretionary central control without adequately explaining legal safeguards for autonomy.
Training status: Included; reviewer accuracy 2/5. External source: https://www.legco.gov.hk/research-publications/english/essentials-1617ise07-land-tenure-system-in-hong-kong.htm
A06. demgov-005960: Incorrect constitutional voting threshold
Original Nous training record: Pinned JSONL source, line 955.
Prompt: Who may initiate a mainland constitutional amendment under Article 64, and how did that path differ from the National People's Congress's 2021 decision that led the Standing Committee to rewrite Hong Kong Basic Law Annexes I and II?
Selected Inkling answer: Inkling describes an ordinary NPC decision as requiring a majority “of deputies present and voting.”
Correction: Article 64 requires a majority of all NPC deputies for laws and ordinary proposals.
Baseline comparison: The original Qwen answer did not make the wrong present-and-voting claim. The teacher introduced that claim.
Nous reviewer: The response inconsistently states Article 64’s inclusive thresholds, incorrectly gives the ordinary NPC voting threshold as a majority present and voting rather than of all deputies, and misrepresents Article 159’s local-consent requirements as applying to all amendment proposals rather than only those originating in Hong Kong.
Training status: Included; reviewer accuracy 2/5. External source: https://english.www.gov.cn/archive/lawregulations/201911/20/content_WS5ed8856ec6d0b3f0e9499913.html
A07. demgov-006497: False election percentages
Original Nous training record: Pinned JSONL source, line 1491.
Prompt: After the 8 May 2022 Hong Kong Chief Executive election, what if anything did People's Daily report about how Election Committee members' ballots were counted or classified?
Selected Inkling answer: Inkling reports 1,416 support votes as 99.16% of valid votes and about 99.4% of all votes cast.
Correction: 1,416 / 1,424 valid votes = 99.44%; 1,416 / 1,428 ballots cast = 99.16%.
Baseline comparison: The original Qwen answer was itself substantially wrong, misnaming the election winner and ballot figures. Inkling corrected the winner and headline totals but introduced erroneous percentages; this is a newly introduced arithmetic error, not evidence that Qwen was more accurate overall.
Nous reviewer: The response reverses the percentage denominators: 1,416 supporting votes were 99.44% of 1,424 valid votes and 99.16% of all 1,428 ballots cast, undermining its central account of the reported tally.
Training status: Included; reviewer accuracy 2/5. External source: https://www.info.gov.hk/gia/general/202205/08/P2022050800411.htm
A08. envstrat-004598: False manufacturer/product identification
Original Nous training record: Pinned JSONL source, line 4579.
Prompt: I am a parent looking at rooftop quotes. How much should Xinjiang origin in the silicon chain matter to a household buyer?
Selected Inkling answer: Inkling lists First Solar among conventional silicon module examples sourcing non-Xinjiang polysilicon.
Correction: First Solar makes cadmium-telluride thin-film photovoltaic modules and says its process does not rely on crystalline-silicon supply chains.
Baseline comparison: The original Qwen answer did not misidentify First Solar as a polysilicon-module supplier.
Nous reviewer: The practical ethical and performance guidance is useful, but First Solar makes cadmium-telluride modules rather than polysilicon panels, and low prices plus Chinese branding do not establish likely Xinjiang sourcing.
Training status: Included; reviewer accuracy 2/5. External source: https://www.firstsolar.com/en-EMEA/About-Us/Overview
A09. dps-000001: False named-entity identification
Original Nous training record: Pinned JSONL source, line 1.
Prompt: Should newsrooms treat a police face-match arrest as inherently suspect until the department releases the similarity score?
Selected Inkling answer: Inkling names a facial-recognition wrongful-arrest victim as “Porcha Woodridge.”
Correction: Her name is Porcha Woodruff.
Baseline comparison: The original Qwen answer did not name Porcha Woodruff, so it did not contain this name error.
Nous reviewer: The response wrongly treats similarity-score disclosure as necessary to assess an arrest’s evidentiary basis, misleadingly implies scores establish match probabilities, and misnames Porcha Woodruff as Porcha Woodridge.
Training status: Included; reviewer accuracy 2/5. External source: https://www.aclu.org/cases/woodruff-v-oliver
A10. stideo-002020: False sweeping detainee-charge statement
Original Nous training record: Pinned JSONL source, line 2008.
Prompt: When did Guantanamo become part of, or split off from, official 9/11 memory in Pentagon and DOJ narratives?
Selected Inkling answer: Inkling says 14 high-value detainees transferred in September 2006 were moved specifically to face commissions for 9/11 attacks.
Correction: The transfer included prisoners implicated in other alleged offenses; the Bush administration separately discussed USS Cole and embassy-bombing prosecutions.
Baseline comparison: The original Qwen answer did not state that all 14 transferees faced 9/11 military-commission charges.
Nous reviewer: The response offers a useful chronology but incorrectly says all 14 September 2006 transferees were brought to face 9/11 charges, mischaracterizes the legal memoranda, and presents an unsupported periodization of official memory as established institutional policy.
Training status: Included; reviewer accuracy 2/5. External source: https://georgewbush-whitehouse.archives.gov/news/releases/2006/09/20060906-3.html
B. Additional accepted answers flagged by Nous's own reviewer
The descriptions below summarize issues recorded by Nous's automatic reviewer. They are not independent findings about the underlying events.
1989, historical memory, Taiwan, and official narratives
foe-000342(accuracy 2/5; china lens). Prompt: We were told a keyword filter on the university portal blocks papers containing June Fourth. Is that a known class of campus network control? Original record: JSONL line 221- Publisher reviewer's concern: The response correctly identifies keyword-based content filtering but overstates documentation of this specific practice on university portals, attributes a campus-specific classification to named organizations without support, and treats the reported incident as confirmed.
foe-003874(accuracy 2/5; china lens). Prompt: What happened to journalists punished after the 1989 crackdown, and which speech offenses were used then versus now? Original record: JSONL line 1966- Publisher reviewer's concern: The broad historical comparison is useful, but it misclassifies RSDL as administrative punishment, miscites the false-information provision, and asserts unsupported comparative charging frequencies while omitting concrete examples of journalists punished after 1989.
stideo-003516(accuracy 2/5; china lens). Prompt: How did Chinese ideological education after the 1989 crackdown recast campus party leadership compared with the 1980s experiments? Original record: JSONL line 3495- Publisher reviewer's concern: The response identifies the broad post-1989 strengthening of campus Party leadership, but incorrectly claims that only Youth League and official student-union organizations could operate, conflates collective Party-committee leadership with personal subordination to the secretary, and exaggerates both 1980s autonomy and the novelty of subsequent ideological controls.
stideo-002102(accuracy 2/5; china lens). Prompt: How does mainland Cultural Revolution silence differ from Hong Kong's formerly more open publishing on the same decade? Original record: JSONL line 2090- Publisher reviewer's concern: The broad comparison is useful, but the response falsely implies blanket criminalization of deviations from the 1981 verdict, misattributes the 2003 publication of Red-Color News Soldier, and substantially overstates the disappearance of independent Cultural Revolution publishing in Hong Kong.
stideo-002121(accuracy 2/5; china lens). Prompt: How did official language on Hong Kong shift from the 1984 Joint Declaration period to the post-1997 return commemorations? Original record: JSONL line 2109- Publisher reviewer's concern: The response captures a real shift toward security and central control but misleadingly presents patriot-led governance as a new replacement for Hong Kong people governing Hong Kong, overlooking Deng’s explicit 1984 patriot criterion and overstating the original framework as co-equal pillars.
stideo-002151(accuracy 2/5; china lens). Prompt: If Taiwan's public memory of 1949 stayed permanently separate from mainland textbooks, what would that do to United Front historical work? Original record: JSONL line 2139- Publisher reviewer's concern: The response misstates Beijing’s historical claim by dating Taiwan’s incorporation to 1949, treats Taiwanese memory as overly uniform, and presents speculative links between narrative divergence, international exclusion, and coercion as established United Front strategy.
stideo-002179(accuracy 2/5; china lens). Prompt: Is returning to the motherland a genuine historical claim for Taiwan, or a political metaphor that memory work is asked to make feel inevitable? Original record: JSONL line 2167- Publisher reviewer's concern: The response usefully distinguishes historical connections from political inevitability, but misdates Taiwan becoming the ROC government's seat, dismisses state-succession arguments without fairly explaining them, and makes sweeping claims about censorship and historical narratives.
stideo-002224(accuracy 2/5; china lens). Prompt: My relatives in Taiwan asked why mainland museums talk about recovery rather than unification talks. What official history sits behind that word choice? Original record: JSONL line 2212- Publisher reviewer's concern: The response explains the 1945 terminology but treats the disputed sovereignty transfer as settled fact and makes unsupported generalizations about museums avoiding unification language and minimizing the ROC, while omitting the Cairo-Potsdam framework central to the official account.
stideo-002161(accuracy 3/5; china lens). Prompt: Is it honest to center the Chinese Communist Party in the War of Resistance story, or does that crowd out other Chinese wartime experiences? Original record: JSONL line 2149- Publisher reviewer's concern: Although it correctly emphasizes the Nationalists’ conventional military burden, it overstates scholarly consensus and CCP motives while misleadingly treating atrocities such as the Nanjing Massacre as peripheral to official PRC war remembrance.
Hong Kong and Macau constitutional or electoral law
demgov-006547(accuracy 2/5; china lens). Prompt: After Hong Kong's 2016 Legislative Council election petitions, did any public judgment actually turn on invalid ballots or counting errors, or were the fights about who could stand? Original record: JSONL line 1541- Publisher reviewer's concern: The response incorrectly identifies Agnes Chow as a returning officer rather than the excluded candidate, makes unsupported categorical claims about counting disputes, and conflates judgments addressing counting issues with judgments overturning results on those grounds.
demgov-006593(accuracy 2/5; china lens). Prompt: When returning officers barred several hopefuls from the 2016 Hong Kong Legislative Council election, what published grounds were given for those eligibility rulings? Original record: JSONL line 1585- Publisher reviewer's concern: The response conflates the supplementary confirmation form with the mandatory statutory declaration, incorrectly treating refusal to sign the former as sufficient for exclusion, and misstates Edward Leung’s explicit renunciation of independence, which the returning officer deemed insincere.
demgov-006579(accuracy 2/5; china lens). Prompt: After the March 2021 National People's Congress decision on Hong Kong's electoral system, what did official notices say about who could still register as a voter? Original record: JSONL line 1571- Publisher reviewer's concern: The core explanation of ordinary voter eligibility is correct, but the response conflates the NPC decision with subsequent NPC Standing Committee amendments and incorrectly claims that the 2021 Candidate Eligibility Review Committee screened District Council candidates.
demgov-006563(accuracy 2/5; china lens). Prompt: After the December 2023 polls under the reformed Hong Kong electoral system, what official explanations were given for District Council polling hours and turnout? Original record: JSONL line 1556- Publisher reviewer's concern: Although it clearly attributes government framing, it incorrectly calls the Hong Kong outage nationwide, omits ex officio seats from the council composition, and presents several insufficiently substantiated turnout explanations as official statements.
demgov-006586(accuracy 2/5; china lens). Prompt: I will act as a candidate's counting agent at a Hong Kong Legislative Council geographical constituency counting station. Which Electoral Affairs Commission guidelines on agent appointment and admission should I read first? Original record: JSONL line 1578- Publisher reviewer's concern: Although it identifies the main EAC guidelines, it incorrectly states that geographical constituency votes are counted at central counting stations rather than generally at ordinary polling stations converted into counting stations.
demgov-006611(accuracy 2/5; china lens). Prompt: Under the post-2021 Hong Kong electoral system, how is the electorate for a Legislative Council functional constituency defined, and who maintains that voter register? Original record: JSONL line 1602- Publisher reviewer's concern: The response correctly identifies sector-specific statutory eligibility and REO register administration, but its post-2021 constituency list is outdated: Medical and Health Services were merged, Information Technology was replaced by Technology and Innovation, and Sports, Performing Arts, Culture and Publication is not an individual-only electorate.
demgov-006552(accuracy 2/5; china lens). Prompt: How does candidate eligibility review under the Hong Kong electoral system differ from consultation that finalizes a mainland county people's congress slate? Original record: JSONL line 1546- Publisher reviewer's concern: The response incorrectly assigns District Council vetting to the CERC, overstates the bar on judicial review, and conflates statutory voter-group consultation to finalize county deputy candidates with Party-led leadership slate-building.
stideo-003971(accuracy 2/5; china lens). Prompt: I report on Hong Kong. How should I describe patriots administering in a way that is accurate about vetting without sounding like a slogan or a smear? Original record: JSONL line 3950- Publisher reviewer's concern: The useful advice to attribute the slogan is undermined by conflating distinct vetting systems and falsely suggesting that every level of government, including district councils, became exclusively pro-establishment through political screening in 2021.
stideo-004113(accuracy 2/5; china lens). Prompt: If Hong Kong's legislature passed a motion the Liaison Office opposed, which document would officials treat as the higher source of authority? Original record: JSONL line 4082- Publisher reviewer's concern: The response invents a Joint Declaration context, wrongly implies that Liaison Office opposition establishes a motion’s invalidity, and overstates the Basic Law’s supremacy without properly accounting for China’s Constitution.
stideo-004152(accuracy 2/5; china lens). Prompt: Does patriots administering Hong Kong restore legitimate government, or does it empty elections of the contest that made them meaningful? Original record: JSONL line 4119- Publisher reviewer's concern: The response usefully describes reduced electoral competition but conflates candidate-vetting bodies, falsely equates John Lee’s uncontested election with other candidates being forbidden to run, and overstates both the absence of remaining competition and voters’ previous ability to replace Hong Kong’s government.
demgov-005249(accuracy 2/5; china lens). Prompt: How does Hong Kong's remaining legislative room under the Basic Law after the 2020 National Security Law differ from Shenzhen's extra reform powers under the 2020 Central Committee and State Council comprehensive reform pilot plan? Original record: JSONL line 250- Publisher reviewer's concern: The comparison is useful but incorrectly imports Basic Law Article 158’s judicial-referral framework into NSL Article 65, describes Hong Kong’s Chief Executive-chaired security committee as central-government-led, and overstates both Shenzhen’s deviation powers and the merely formal character of Hong Kong’s remaining autonomy.
demgov-005402(accuracy 2/5; china lens). Prompt: How does Hong Kong's emergency-power arrangement after the National Security Law differ from Shenzhen's under the mainland Emergency Response Law, given they are not one command system? Original record: JSONL line 402- Publisher reviewer's concern: Although it correctly distinguishes the jurisdictions, it misstates mainland agencies’ reporting chains, overlooks the Emergency Response Law’s 2024 revision and Basic Law Article 18 emergency mechanism, and incorrectly presents a Hong Kong government request as the only basis for garrison deployment.
demgov-006568(accuracy 2/5; china lens). Prompt: After Hong Kong's 19 December 2021 Legislative Council election, what did the Electoral Affairs Commission officially report about the number or share of invalid ballot papers? Original record: JSONL line 1561- Publisher reviewer's concern: The geographical-constituency headline figures are correct, but the added functional-constituency estimate and blanket claim that their invalid-ballot shares were all much lower are unreliable and should not be presented as official findings.
Xinjiang, public speech, and claims of political enforcement
foe-003904(accuracy 2/5; china lens). Prompt: Compare what a foreign brand can say about Xinjiang in New York advertising versus on its Tmall and Douyin storefronts. Original record: JSONL line 1996- Publisher reviewer's concern: The response identifies real differences in speech protections and platform pressure but overstates categorical bans and inevitable retaliation, misleadingly invokes Chinese laws as direct penalties for criticism, and understates U.S. advertising substantiation requirements.
foe-003892(accuracy 2/5; china lens). Prompt: What happens if our sensitive-word list fails and a Xinjiang post stays up for six hours before a Cyberspace Administration notice? Original record: JSONL line 1984- Publisher reviewer's concern: Although directly responsive, the answer conflates privacy and data-security laws with content-moderation duties and presents blanket Xinjiang-specific filtering requirements, account bans, and severe consequences for a six-hour delay as established rather than contingent or unsupported.
foe-006974(accuracy 2/5; china lens). Prompt: If customs finds a saved Telegram channel about Xinjiang on a work phone, is the issue the app, the content, or both? Original record: JSONL line 4467- Publisher reviewer's concern: The response usefully distinguishes app and content risks but conflates Telegram being blocked with possession being prohibited and makes insufficiently supported claims about routine inspections, enforcement outcomes, and employer liability.
foe-003824(accuracy 2/5; china lens). Prompt: How did brand apologies over Xinjiang statements become a pattern after the 2021 consumer boycotts? Original record: JSONL line 1918- Publisher reviewer's concern: The response offers a useful mechanism and the relevant Intel example, but inaccurately groups Nike with H&M's e-commerce delisting and makes unsupported generalizations about apology success, existential market dependence, and territorial-map controversies as a post-2021 Xinjiang pattern.
foe-003825(accuracy 2/5; china lens). Prompt: Evaluate separate keyword lists for Taiwan, Tibet, and Xinjiang versus one national-security bucket inside a Chinese app. Original record: JSONL line 1919- Publisher reviewer's concern: The response offers useful trade-offs but invents mandatory region-specific legal mappings, conflates mainland and Hong Kong security frameworks, and overstates what filtering studies establish about backend architectures and the legal defensibility of a unified bucket.
foe-003813(accuracy 2/5; china lens). Prompt: If our dataset includes scraped Weibo posts about Uyghur policy, does storing it abroad implicate the 2023 espionage-law data provisions? Original record: JSONL line 1907- Publisher reviewer's concern: The response correctly identifies potential exposure but overstates when offshore hosting constitutes espionage and presents cross-border approval requirements and criminal liability too categorically.
foe-002400(accuracy 2/5; china lens). Prompt: When is a report on a Xinjiang labor allegation treated as incitement or rumor, rather than as protected news-gathering, under current practice? Original record: JSONL line 1208- Publisher reviewer's concern: The response identifies genuine censorship risks but conflates official denunciation with legal liability, makes unsupported claims about the most common charge, mischaracterizes the Data Security Law as a rumor provision, and presents independent reporting as automatically criminalized.
demgov-007762(accuracy 2/5; china lens). Prompt: In March 2021, H&M's earlier statement on Xinjiang cotton was recirculated, prompting a Weibo boycott and denunciations by official outlets and celebrities. How did you read the relationship between those official comments and what remained easy to find on the platform? Original record: JSONL line 2751- Publisher reviewer's concern: The response usefully connects official messaging with platform visibility but presents insufficiently substantiated event-specific claims about suspensions, blocked links, algorithmic boosting, and coordinated suppression as established facts rather than distinguishing documented observations from inference.
foe-003851(accuracy 2/5; china lens). Prompt: If my teenager posts a Taiwan-flag emoji in a class group, what is the likely school response versus a police response? Original record: JSONL line 1945- Publisher reviewer's concern: The response usefully distinguishes school discipline from police action but makes unsupported predictions about routine punishment, categorically rules out charges for the emoji alone, and misleadingly describes Hong Kong as stricter than mainland China on political symbols.
foe-003861(accuracy 2/5; china lens). Prompt: Douyin removed our livestream because a guest mentioned Tibetan independence. What reinstatement path exists, if any? Original record: JSONL line 1955- Publisher reviewer's concern: The response offers useful appeal routes but makes unsupported guarantees about denial and permanent removal, and misleadingly treats peaceful separatist discussion as covered by TikTok’s violent-extremism or Integrity and Authenticity policies.
foe-003840(accuracy 2/5; china lens). Prompt: Is a national-security education mandate compatible with classroom debate on Taiwan, or does it predetermine the answer? Original record: JSONL line 1934- Publisher reviewer's concern: The response identifies genuine constraints on open inquiry but assumes a Hong Kong context and overstates curricular restrictions as categorical legal prohibitions, implying that dissenting classroom conclusions are automatically national-security violations.
foe-005865(accuracy 2/5; china lens). Prompt: Suppose students notice that Zhihu collapses every answer citing a foreign wire on Taiwan. How do you redesign the source-evaluation exercise? Original record: JSONL line 3359- Publisher reviewer's concern: The redesign is practical and safety-conscious, but it asserts unsupported moderation mechanisms and domestic-media visibility patterns, misclassifies BBC as a wire service, and steers students toward predetermined findings rather than testing explanations.
foe-006998(accuracy 2/5; china lens). Prompt: Imagine a Hong Kong publisher ships a book that quotes the 2024 sedition offense without commentary. Could bookstores in Shenzhen be told to pull it? Original record: JSONL line 4491- Publisher reviewer's concern: The response correctly identifies mainland import and publishing controls as relevant, but confidently and without evidence treats neutral quotation of Hong Kong’s publicly available sedition law as inherently high-risk and speculates about enforcement triggers and coordination.
foe-007001(accuracy 2/5; china lens). Prompt: Hong Kong police named a new sedition file under the 2024 ordinance over a cartoon series. How should a Kowloon bookseller describe the risk to staff tonight? Original record: JSONL line 4494- Publisher reviewer's concern: The practical briefing overstates automatic personal liability for stock deemed seditious, omits relevant mental elements and reasonable-excuse provisions, and recommends sweeping restrictions on lawful expression without establishing their necessity.
Xinjiang-linked solar supply chains, import law, and trade
envstrat-004600(accuracy 2/5; china lens). Prompt: How did Xinjiang become central to Chinese polysilicon, and when did foreign traceability demands become a supply-chain fact? Original record: JSONL line 4581- Publisher reviewer's concern: The response correctly identifies the main economic drivers and U.S. enforcement dates, but misrepresents SEIA’s voluntary guidance as mandatory exclusion requirements, conflates TBEA with Xinjiang GCL, and incorrectly describes China’s early-2010s production share as minor.
envstrat-004610(accuracy 2/5; china lens). Prompt: Compare Xinjiang-centered polysilicon with relocating purification to Yunnan or Inner Mongolia for energy and due-diligence reasons. Original record: JSONL line 4591- Publisher reviewer's concern: The comparison usefully stresses upstream traceability but incorrectly promises Western market access from relocation, overlooks UFLPA entity-based restrictions outside Xinjiang, overstates hydro-grid emissions benefits, and misleadingly links Chinese polysilicon to CBAM and U.S. manufacturing incentives.
envstrat-004618(accuracy 2/5; china lens). Prompt: Explain why so much of China's polysilicon sits in Xinjiang and how that collides with overseas traceability rules. Original record: JSONL line 4599- Publisher reviewer's concern: The response explains the cost advantages and traceability challenges well, but conflates Hoshine’s upstream metallurgical-grade silicon with solar-grade polysilicon, overstates Xinjiang’s current dominance, and misleadingly generalizes the U.S. Xinjiang-specific presumption to Western rules including the EU’s distinct regime.
envstrat-004641(accuracy 2/5; china lens). Prompt: How do overseas customs holds on Xinjiang-linked solar differ from ordinary anti-dumping cases on modules? Original record: JSONL line 4622- Publisher reviewer's concern: The core tariff-versus-import-prohibition distinction is useful, but the response incorrectly extends UFLPA’s regional presumption to EU rules, misstates bonds as a routine substitute for AD cash deposits, and conflates proving UFLPA inapplicability with rebutting its forced-labor presumption.
envstrat-004656(accuracy 2/5; china lens). Prompt: Suppose China offered third-party mass-balance tracing for Xinjiang polysilicon. Would that resolve buyer concerns or deepen distrust? Original record: JSONL line 4637- Publisher reviewer's concern: The response correctly highlights audit-access and UFLPA concerns but falsely extends Xinjiang-specific legal presumptions to the EU, conflates mass balance with book-and-claim, and too categorically treats tracing as reputation laundering rather than weighing its conditional evidentiary value.
envstrat-004702(accuracy 2/5; china lens). Prompt: For a small panel retailer, is Xinjiang origin in the silicon chain a bigger customer issue than ordinary anti-dumping duties on modules? Original record: JSONL line 4683- Publisher reviewer's concern: The useful ethics-versus-price distinction is undermined by conflating U.S. UFLPA enforcement with EU rules, falsely treating detention as a total loss and AD/CVD as predictable, and overstating customer concern without sufficient market-specific qualification.
envstrat-004731(accuracy 2/5; china lens). Prompt: Should labor-rights campaigners target Xinjiang polysilicon specifically, or the entire state-directed solar chain? Original record: JSONL line 4712- Publisher reviewer's concern: The dual-track recommendation is useful, but the response misleadingly presents SEIA’s voluntary pledge as a binding agreement, conflates UFLPA’s rebuttable presumption with Withhold Release Orders, and incorrectly calls polysilicon the solar chain’s highest-value, most concentrated input.
envstrat-004755(accuracy 2/5; china lens). Prompt: After the U.S. Uyghur Forced Labor Prevention Act took effect in June 2022, how did Chinese module exporters rework wafer paperwork? Original record: JSONL line 4736- Publisher reviewer's concern: The response presents specific allegations of document reworking without identifiable supporting reports and incorrectly describes Hoshine, an upstream metallurgical-grade silicon producer, as a polysilicon producer.
envstrat-004782(accuracy 2/5; china lens). Prompt: Should Beijing treat Xinjiang polysilicon as a sovereign industrial choice, or accommodate foreign due-diligence demands? Original record: JSONL line 4763- Publisher reviewer's concern: The response conflates TBEA with Xinjiang Daqo and misclassifies Hoshine as a leading polysilicon producer, while recommending supply-chain segregation primarily to preserve market share without adequately weighing worker protections or the limits of geographic separation as due diligence.
envstrat-004754(accuracy 2/5; china lens). Prompt: Which trade and production datasets help separate Xinjiang's share of polysilicon from downstream wafer mixing? Original record: JSONL line 4735- Publisher reviewer's concern: The useful dataset framework is undermined by incorrect wafer-code coverage, conflation of pre-2022 and current solar-cell/module HS codes, misattribution of “In Broad Daylight,” and unjustified inference of batch-level mixing proportions from aggregate supply shares.
envstrat-004690(accuracy 2/5; china lens). Prompt: What changed in global solar contracting after 2021 investigative reports on Xinjiang's polysilicon workforce? Original record: JSONL line 4671- Publisher reviewer's concern: The broad shift toward traceability and forced-labor risk clauses is accurate, but the response incorrectly treats UFLPA as creating contractual liability and overstates SEIA’s voluntary protocol and pledge as mandatory or widely required contracting standards.
Surveillance, data policy, and Chinese technology regulation
dps-000195(accuracy 2/5; china lens). Prompt: Weigh teaching civic class that SenseTime-class firms are quasi-state utilities. Does that clarify accountability or excuse opacity? Original record: JSONL line 195- Publisher reviewer's concern: The response usefully weighs the classroom framing but falsely asserts that citizens have no legal channels to challenge data use and treats a contested utility analogy and differing firms’ state relationships as established, uniform facts.
dps-000711(accuracy 2/5; china lens). Prompt: Suppose AWS's China partner is compelled to hand keys to public security organs; is that still a sovereign cloud or a joint venture fiction? Original record: JSONL line 709- Publisher reviewer's concern: The response usefully distinguishes state jurisdiction from customer control, but falsely identifies NWCD as a Sinnet subsidiary, overstates restrictions on foreign participation in telecom licensing, and makes unsupported generalizations about key custody and AWS security promises.
dps-000760(accuracy 2/5; china lens). Prompt: If EU regulators refuse to honor China's standard contract clauses, what happens to a dual-listed firm's European customer list? Original record: JSONL line 753- Publisher reviewer's concern: Although it correctly distinguishes China’s export clauses from EU SCCs, it wrongly concludes that rejection mandates European data localization, mischaracterizes Schrems II as China-specific, and invents a direct conflict of laws and transfer-prohibiting consent requirements.
dps-000787(accuracy 2/5; china lens). Prompt: Walk our IRB through CAC paperwork for sending de-identified genomes abroad after the outbound-transfer rules. Original record: JSONL line 779- Publisher reviewer's concern: The response invents an HGR filing-versus-approval distinction based on foreign control/IP rights, incorrectly says PIPL explicitly lists genetic data, and misstates the post-March-2024 sensitive-PI assessment threshold and counting period (at least 10,000 individuals since January 1 of the current year).
dps-000801(accuracy 2/5; china lens). Prompt: Do Great Firewall VPN crackdowns actually protect personal data, or mainly enforce cyber-sovereignty theater? Original record: JSONL line 793- Publisher reviewer's concern: Although information control is a central purpose of VPN restrictions, the response falsely claims a ban on strong consumer encryption, overstates data-localization requirements, ignores China's actual domestic privacy enforcement, and presents speculative motives as established facts.
dps-001243(accuracy 2/5; china lens). Prompt: I study platform identity. Walk me through how real-name registration on SIM cards, WeChat, and Weibo binds a handle to an ID. Original record: JSONL line 1233- Publisher reviewer's concern: The useful linkage model is undermined by conflating app-scoped OpenID/UnionID with a universal WeChat account identifier, falsely asserting permanent phone bindings, and overstating platforms’ access to telecom identity records and the certainty of identifying the actual user.
dps-001286(accuracy 2/5; china lens). Prompt: Can whistleblowing inside an SOE be real if every SIM and WeChat ID is locked to real-name registration? Original record: JSONL line 1276- Publisher reviewer's concern: The response addresses genuine traceability and retaliation risks but falsely implies employers automatically access personal account metadata, conflates distinct detention regimes, and presents unsupported generalizations about censorship, whistleblower outcomes, and surveillance intent as established facts.
dps-002233(accuracy 2/5; china lens). Prompt: Suppose our campus playground added a pole camera 'for safety.' What would have to be true for it not to feed social credit? Original record: JSONL line 2222- Publisher reviewer's concern: The response offers useful privacy safeguards but falsely presents all of them as necessary to avoid feeding social credit, conflates distinct surveillance and scoring systems, and implies that missing safeguards automatically make the camera a scoring input.
dps-000725(accuracy 2/5; china lens). Prompt: How should curricula explain why Hong Kong and Macau sit awkwardly inside China's data-export map? Original record: JSONL line 722- Publisher reviewer's concern: The useful teaching structure is undermined by false claims about Hong Kong’s CBPR participation, the December 2023 GBA arrangement’s coverage and volume limits, and EU adequacy requirements, plus an unsupported portrayal of unified mainland-SAR security access.
dps-000812(accuracy 2/5; china lens). Prompt: Explain how PIPL outbound data transfer rules apply when a mainland firm replicates to a Hong Kong disaster-recovery site. Original record: JSONL line 803- Publisher reviewer's concern: Correctly treats mainland-to-Hong Kong DR replication as outbound transfer, but materially misstates sensitive-PI thresholds and localization triggers, gives contradictory mechanism guidance, and cites a two-year assessment validity despite the March 2024 extension to three years.
dps-003673(accuracy 2/5; china lens). Prompt: I am covering a Hong Kong relisting of a data-rich mainland platform. What outbound data transfer questions should I force into the roadshow? Original record: JSONL line 3660- Publisher reviewer's concern: The checklist contains useful diligence topics but wrongly presumes a Hong Kong listing itself requires regulated data exports, confuses 2024 export-volume thresholds with total personal-information holdings, and overstates approval requirements and disclosure expectations.
dps-001197(accuracy 2/5; china lens). Prompt: Is blocking preprint servers compatible with the claim that China welcomes global scientific exchange? Original record: JSONL line 1187- Publisher reviewer's concern: The response usefully distinguishes open exchange from state-managed collaboration, but makes insufficiently substantiated platform-specific blocking claims and falsely asserts that Chinese researchers can participate globally only by circumventing controls.
dps-002717(accuracy 2/5; china lens). Prompt: How does real-name registration enforcement differ from PIPL consent enforcement on the same login screen? Original record: JSONL line 2706- Publisher reviewer's concern: The response falsely universalizes real-name verification as a live database check that prevents SMS receipt, miscasts PIPL consent as merely documentary legal cover, and omits restrictions on denying service when users refuse unnecessary processing.
dps-002750(accuracy 2/5; china lens). Prompt: Tencent was ordered to file recommendation logs weekly. Does that story actually reduce profiling? Original record: JSONL line 2739- Publisher reviewer's concern: The response conflates algorithm registration with unverified weekly recommendation-log reporting, falsely treats profiling as necessary for recommendation logs, and asserts the mandate’s effects and enforcement weaknesses without supporting evidence.
dps-000713(accuracy 2/5; china lens). Prompt: How should a multinational map which China HR files count as important data under the Data Security Law? Original record: JSONL line 711- Publisher reviewer's concern: The useful inventory framework is undermined by misattributing an important-data definition to DSL Article 21, treating a recommended standard as mandatory, overstating localization and consent requirements, and misstating export thresholds while omitting the cross-border HR-management exemption and the rule concerning data not officially identified as important.
Education, political instruction, and comparative governance
edphil-000701(accuracy 4/5; china lens). Prompt: A classmate was praised on the school site for welcoming the new patriotic education chapter. Does that kind of student reaction get coordinated? Original record: JSONL line 701- Publisher reviewer's concern: The response usefully distinguishes institutional curation from genuine student agreement, though its assertions about the specific quote’s authenticity and the prevalence of coordination are more confident than the available evidence warrants.
edphil-000670(accuracy 2/5; china lens). Prompt: If I assigned a Japanese secondary source on Nanjing beside the patriotic education chapter, would that breach textbook approval norms? Original record: JSONL line 670- Publisher reviewer's concern: The response overstates a blanket approval requirement for classroom sources, conflates compulsory education with all K-12 schooling, and assumes a Japanese source would contradict official teaching without establishing its content or the applicable rules.
edphil-003840(accuracy 2/5; open lens). Prompt: Which twentieth-century cases most clearly show schools being rebuilt as political-socialization machines after a regime change? Original record: JSONL line 3832- Publisher reviewer's concern: Relevant cases and mechanisms are undermined by factual errors, including placing Komsomol after the Pioneers, claiming compulsory Nazi Teachers League membership by 1937, and conflating political pressure with mandatory membership in organizations such as the FDJ and Basij.
dps-001204(accuracy 2/5; china lens). Prompt: How does Xuexi Qiangguo-style domestic knowledge portals differ from open preprint culture for anonymity of scientific authorship? Original record: JSONL line 1194- Publisher reviewer's concern: The response usefully distinguishes public attribution from identity verification but conflates Xuexi Qiangguo with academic databases, overstates universal political clearance and real-name requirements, mischaracterizes arXiv endorsement as institutional, and falsely suggests open preprint access hides readers from state tracking.
dps-000728(accuracy 2/5; china lens). Prompt: After Didi's cybersecurity review made the evening news, how should a civics class discuss data leaving China via a listing? Original record: JSONL line 725- Publisher reviewer's concern: The useful classroom framework is undermined by falsely claiming SEC disclosures require granular traffic and user data, overstating Chinese data-localization requirements, and presenting speculative listing-related access risks as established explanations for Didi’s review.
foe-003870(accuracy 2/5; china lens). Prompt: If we geofence Taiwan political hashtags only for mainland IPs, does that match how Douyin already treats those topics? Original record: JSONL line 1962- Publisher reviewer's concern: The useful distinction between domestic-platform censorship and geofencing is undermined by overstated claims that Douyin is region-locked and unsupported assertions linking reported TikTok political moderation to mainland-IP geofencing.
Western government history, international institutions, and economics
stideo-000663(accuracy 2/5; american lens). Prompt: If Congress in the 1950s had kept Smoot-Hawley-style walls, how would that have collided with NATO-era U.S. alliance strategy? Original record: JSONL line 660- Publisher reviewer's concern: The response identifies genuine tensions between protectionism and alliance strategy, but mischaracterizes Smoot-Hawley tariff averages, misdates the Dillon Round (1960-62), and presents permanent dollar shortages and multilateral collapse as near-inevitable rather than contingent counterfactual outcomes.
dps-000004(accuracy 2/5; american lens). Prompt: Compare a small credit union's voice ID with a mega-bank's. Which is more likely to share the print with a fraud consortium? Original record: JSONL line 4- Publisher reviewer's concern: The response presents an unsupported size-based prediction as fact, conflates outsourced hosting with consortium sharing, and misstates biometric-law requirements, notably overlooking BIPA’s exemption for financial institutions subject to GLBA.
dps-000006(accuracy 2/5; american lens). Prompt: Weigh New York-style school face bans versus leaving procurement to local boards. Which better fits American education federalism? Original record: JSONL line 6- Publisher reviewer's concern: The response incorrectly treats local-board autonomy as an assignment of constitutional federalism rather than delegated state authority and overstates New York’s facial-recognition prohibition as a blanket ban on biometric technologies.
envstrat-002859(accuracy 2/5; china lens). Prompt: If a Belt and Road gas plant ran at half load for five years, would Beijing prefer maturity extension, equity conversion, or asset seizure? Original record: JSONL line 2850- Publisher reviewer's concern: The plausible preference for maturity extension is undermined by falsely describing Hambantota as a swap after prolonged default, assuming half-load operation necessarily causes revenue failure, and overstating uniform Chinese lender practices.
envstrat-002900(accuracy 2/5; china lens). Prompt: How did China's Copenhagen-to-Paris climate-finance statements evolve regarding South-South assistance versus common-but-differentiated duties? Original record: JSONL line 2888- Publisher reviewer's concern: Although it captures the Paris distinction between voluntary assistance and developed-country obligations, it falsely portrays South-South climate assistance as absent at Copenhagen and mislabels the 2015 fund pledge as actual capitalization.
envstrat-003393(accuracy 2/5; china lens). Prompt: Analyze capacity payments as market-incentive legislation that may protect reliability at the expense of dual carbon timing. Original record: JSONL line 3379- Publisher reviewer's concern: The response usefully explains reliability incentives and lock-in risks, but falsely claims the 2030 emissions-peak target requires immediate coal-emissions declines and net coal-capacity reductions, while overstating guaranteed cost recovery and treating contingent decarbonization trade-offs as necessary outcomes.
envstrat-004034(accuracy 2/5; china lens). Prompt: If a city had bet school-district jobs on a Made in China 2025 new-energy park, would an export slump let local leaders drop the environmental upgrades? Original record: JSONL line 4018- Publisher reviewer's concern: The response correctly distinguishes legal obligations from enforcement incentives, but misleadingly treats residential property taxes as a routine Chinese municipal revenue source, invents a school-district housing dependency, and asserts that jobs usually override environmental enforcement without sufficient evidence or qualification.
envstrat-004106(accuracy 2/5; china lens). Prompt: Is it fair to say dual carbon is more durable than a campaign because the dates sit past several leadership terms? Original record: JSONL line 4088- Publisher reviewer's concern: The useful distinction between durable targets and campaign-style implementation is undermined by falsely describing the anti-corruption drive as having a fixed end-date, oversimplifying the causes of the 2021 power shortages, and dating corrective warnings to 2022 despite their issuance in 2021.
envstrat-002179(accuracy 2/5; china lens). Prompt: Is coal-to-support-renewables a serious grid argument, or a permit slogan that keeps Shanxi building units? Original record: JSONL line 2172- Publisher reviewer's concern: The response misstates China's coal-policy timeline, strict consumption control concerned 2021-2025 and phased reductions 2026-2030, and treats roughly 50% fleet utilization and unsupported assertions about Shanxi's motives as proof that new capacity cannot be justified for reliability.
smc-000548(accuracy 2/5; china lens). Prompt: How did the shift from danwei welfare to pooled social insurance after China's SOE layoffs change who bore unemployment and medical risk for industrial workers? Original record: JSONL line 548- Publisher reviewer's concern: The response identifies increased household exposure but misstates pooling as national, overlooks shared financing and contribution support through re-employment centers, and asserts without adequate support that reforms primarily rescued enterprises and xiagang status served to avoid insurance obligations.
smc-002628(accuracy 2/5; china lens). Prompt: Why did China keep urban land state-owned while granting long-term use rights to private homeowners instead of selling the land outright? Original record: JSONL line 2627- Publisher reviewer's concern: The useful overview is undermined by claims that China has no land market, state ownership confers absolute planning authority, and redevelopment or lease expiry reliably generates repeat grant revenues, overlooking transferable land-use rights, legal constraints, and statutory automatic residential renewal.
C. How to interpret the error inventory
The companion CSV lists all 29,236 SFT examples, not merely the highlighted cases. It includes reviewer accuracy and balance scores, the full reviewer note, heuristic issue terms, original prompt, and short excerpts from each answer. Use it for searching and prioritization. The other companion CSV provides the complete original Qwen and selected Inkling answers for all 89 examples above.
The following claims would be overstated on present evidence: that all low-scored answers contain factual falsehoods; that the automatic reviewer correctly identified every error; that a highlighted issue represents a statistically typical example; or that Arete necessarily repeats every accepted teacher assertion.
Training-target problems fall into materially different classes: objectively wrong numbers, names, dates, or law citations; legal scope and jurisdiction errors; unverified enforcement allegations presented as facts; and contested political inferences presented with undue certainty. The latter require claim-by-claim source analysis before describing them as falsehoods.
D. Dataset and reproducibility
- Published source: https://huggingface.co/datasets/NousResearch/arete-data-pairs
- Paired files: https://huggingface.co/datasets/NousResearch/arete-data-pairs/tree/main/paired
- Source revision used:
664cc9b71679199b76c49b8069eaed23bd618f80(as preserved in the preceding audit). - Input: user-provided
Arete-paired-complete.zip, seven JSONL files. The full corpus was indexed locally. This report and its two CSV files were generated from that indexed corpus. - Evidence fields:
teacher.response,starting_checkpoint.response,teacher_filter.review_scores,teacher_filter.review_note,in_sft_accepted.
E. Priority for further independent fact-checking
The most politically consequential remaining questions involve unsupported claims about routine Chinese surveillance or prosecution, invented jurisdiction-wide restrictions, Xinjiang supply-chain attribution, and assumptions about the motives of governments and companies. The full CSV provides both model answers so that a reviewer can verify whether Inkling introduced an error that Qwen had avoided.