Constellation Visiting Fellow Application

Personal Information

  • Name: Jonathan Ng
  • Affiliation: Apart Research
  • Career Stage: Industry
  • Preferred Pronouns: He/him

Application Details

  • Desired Program Duration: 3 months/6 months
  • Funding Requirement: No
  • Focus Area: Building an AI observatory
  • Availability: Available for 6 months, willing to stay for the full period if admitted
  • Cohort Applying To: Fall, beginning October 7 / Winter, beginning in January

Project Proposal

I am currently working at Apart Research as a contracting Research Engineer, making cybersecurity evaluations for SOTA LLM models. In the past 6 months, I have:

  • Developed the harness to run evaluations on and new tasks and challenges
  • Wrote a scraper for PicoCTF challenges
  • Created challenges both by (1) writing them by hand and refining it with AI assistance and (2) by generating them from a prompt and refining them by hand. I have 14 concrete tasks, one for each category of risk (Recon to Impact) https://attack.mitre.org/
  • Collaborated with other research engineers and writers
  • Collected data on cybersecurity capabilities across 11 SOTA LMs, 10 elicitation methods

At the program broadly I will:

  1. Continue developing the harness and new tasks and challenges
  2. Research representation: challenges that measure cybersecurity capabilities in ways that cybersecurity professionals, lawmakers, and the public can understand and care about
  3. Collaborate and network with cybersecurity and evals experts

I want to create:

  • A dashboard with a table: of how every most recent model does on our benchmarks
  • Equivalently: a "tech tree" breaking down each challenge / capability into sub-challenges, showing how well skills compose
  • An online tinkering interface that allows people to contribute challenges in an easy to understand way
  • Writing summarizing some of the above work: Blog posts (1-5+), papers (0 - 1.5)

More details:

  • We have a paper almost ready for submission, data collection in progress. I'm asking for permission to share it. If you email me, I can probably share the paper and more details.

Collaboration Interests

I'm interested in engaging with:

  • People like METR: evals, elicitation, prompt engineering, etc
  • People like Palisade: cybersecurity experts
  • Governance people who are thinking about cybersec
  • Sysadmin skills like unix, docker, kali linux, etc

Additional Information

  • I attended the pilot programme for ARENA (https://www.arena.education/)
  • I attended SERI MATS Winter 2023
  • Contributed to MACHIAVELLI as a research engineer, made webapp for collecting data for human baseline

Immigration Status

I am able to get an ESTA. In addition to ESTA, as a Singaporean citizen, I'm also eligible for H1B1 visa.

Referees

  1. Esben [hidden]
  2. Walter [hidden]

Additional Preferences

  • Willing to receive emails about future programs and opportunities
  • Consent given to share relevant information with other organizations in the network

How I Heard About the Program

  • Email from someone else (e.g. mailing list)
  • Referral by a friend or colleague
  • Alignment Forum, LessWrong, or EA Forum post
Edit
Pub: 20 Aug 2024 16:32 UTC
Views: 102