Constellation Visiting Fellow Application
Personal Information
- Name: Jonathan Ng
- Affiliation: Apart Research
- Career Stage: Industry
- Preferred Pronouns: He/him
Application Details
- Desired Program Duration: 3 months/6 months
- Funding Requirement: No
- Focus Area: Building an AI observatory
- Availability: Available for 6 months, willing to stay for the full period if admitted
- Cohort Applying To: Fall, beginning October 7 / Winter, beginning in January
Project Proposal
I am currently working at Apart Research as a contracting Research Engineer, making cybersecurity evaluations for SOTA LLM models. In the past 6 months, I have:
- Developed the harness to run evaluations on and new tasks and challenges
- Wrote a scraper for PicoCTF challenges
- Created challenges both by (1) writing them by hand and refining it with AI assistance and (2) by generating them from a prompt and refining them by hand. I have 14 concrete tasks, one for each category of risk (Recon to Impact) https://attack.mitre.org/
- Collaborated with other research engineers and writers
- Collected data on cybersecurity capabilities across 11 SOTA LMs, 10 elicitation methods
At the program broadly I will:
- Continue developing the harness and new tasks and challenges
- Research representation: challenges that measure cybersecurity capabilities in ways that cybersecurity professionals, lawmakers, and the public can understand and care about
- Collaborate and network with cybersecurity and evals experts
I want to create:
- A dashboard with a table: of how every most recent model does on our benchmarks
- Equivalently: a "tech tree" breaking down each challenge / capability into sub-challenges, showing how well skills compose
- An online tinkering interface that allows people to contribute challenges in an easy to understand way
- Writing summarizing some of the above work: Blog posts (1-5+), papers (0 - 1.5)
More details:
- We have a paper almost ready for submission, data collection in progress. I'm asking for permission to share it. If you email me, I can probably share the paper and more details.
Collaboration Interests
I'm interested in engaging with:
- People like METR: evals, elicitation, prompt engineering, etc
- People like Palisade: cybersecurity experts
- Governance people who are thinking about cybersec
- Sysadmin skills like unix, docker, kali linux, etc
Additional Information
- I attended the pilot programme for ARENA (https://www.arena.education/)
- I attended SERI MATS Winter 2023
- Contributed to MACHIAVELLI as a research engineer, made webapp for collecting data for human baseline
Immigration Status
I am able to get an ESTA. In addition to ESTA, as a Singaporean citizen, I'm also eligible for H1B1 visa.
Referees
- Esben [hidden]
- Walter [hidden]
Additional Preferences
- Willing to receive emails about future programs and opportunities
- Consent given to share relevant information with other organizations in the network
How I Heard About the Program
- Email from someone else (e.g. mailing list)
- Referral by a friend or colleague
- Alignment Forum, LessWrong, or EA Forum post