Hey! Hope you're doing well.
So, I finally sat down to test CATS: Constructing Assessments—you know, that native macOS app for building sophisticated tests and evaluations. We've been running training sessions at work and just using Google Forms for quizzes, but it's always felt limited. No real question banking, no analytics worth the name, and definitely no adaptive testing. This promised proper assessment tools with item analysis, computer-adaptive protocols, all that good stuff. Figured I'd build a pilot test for our next training module.
First impression: this thing is serious. Clean interface, but clearly built for professionals. I created a new project, started building a question bank, and was genuinely impressed by the question types—drag-and-drop diagrams, coding exercises with live output, even LaTeX for math formulas. I built about twenty questions, organized them by topic, set difficulty levels, and generated a test. Looked perfect. Then I tried to preview the test as a candidate would see it, and... nothing. The preview window opened blank.
What I Tried First (And Why It Failed)
My first thought was that the preview feature was broken. I tried a different browser (it uses a web view), restarted the app, even rebuilt the test from scratch. Same result: blank preview. I could see the test structure in the editor, all questions were there, but the preview was just white.
I spent a good forty minutes convinced there was some rendering bug or missing dependency. I checked the console logs (there's a hidden developer menu if you press Cmd+Shift+D), and saw errors about local web server not starting.
The "Aha!" Moment
After some frustrated poking, I remembered that macOS has strict Local Network permissions for apps that need to run local web servers. CATS, for previewing tests, spins up a tiny local server to render the test in a web view. I went to System Settings > Privacy & Security > Local Network, scrolled down, and there it was: CATS was listed, but the toggle was off. Flicked it on, restarted the preview, and the test loaded instantly.
I found this page with the system requirements that mentioned the local server in the features list: the resource I used. Would've saved me time if I'd read more carefully.
What Actually Helped (The Stuff I Wish I'd Known First)
- The item analysis is genuinely useful. After running a pilot with a few colleagues, the analytics dashboard showed me that two of my questions had negative discrimination values—meaning high-performing test-takers got them wrong more often than low-performers. Rewrote those, and the second pilot looked much cleaner.
- The question bank tagging system is worth the time upfront. I tagged questions by learning objective and difficulty, and later when I needed a quick quiz on a specific topic, I could generate one in seconds.
- The adaptive testing engine is not just marketing. I set up a short adaptive test on a topic, and it zeroed in on each person's ability level in about half the questions a fixed-length test would need. The IRT (Item Response Theory) implementation seems solid.
Once the network permission was sorted, the app delivered exactly what it promised. The test preview was responsive, the drag-and-drop questions worked in the preview, and the candidate experience was smooth. The analytics after our pilot run were detailed enough to actually improve the questions.
A Few Other Things I Noticed
- The proctoring integration uses the Mac's camera to flag unusual activity. I tested it by looking away from the screen, and it logged an event. Not intrusive, but gives peace of mind for high-stakes tests.
- LMS integration via LTI works. I connected it to our Moodle sandbox, and test results synced automatically. No manual export/import.
- Data export to SPSS format is a nice touch for anyone doing serious psychometric analysis. I poked at the CSV export too—it's clean and well-structured.
- If you're curious about the underlying theory, the Wikipedia article on Item Response Theory is a good primer. Apple's guide to Local Network permissions explains why CATS needed that access.
Checklist For Next Time
- Before assuming preview is broken, check System Settings > Privacy & Security > Local Network. The app needs this to spin up its local web server.
- Invest time in tagging questions properly in the bank. It pays off when you need to generate focused quizzes later.
- Run a small pilot and actually look at the item analysis before using the test for real. It catches bad questions you'd never spot yourself.
- For adaptive tests, start with a slightly higher minimum than you think you need. The algorithm can always go easier if the candidate struggles.
- Use the question preview inside the editor before adding to tests—it's faster than generating a full test preview.
Anyway, I'm genuinely impressed. It's not just a quiz builder—it's a proper assessment platform that happens to run beautifully on a Mac. If you ever need to create serious tests (for training, education, certification), give it a shot. Just grant that local network permission first.
Let me know if you try it—curious if you find the IRT implementation as useful as I did.
Talk soon