"As language models become more capable, ensuring safety in a model's deployment has become a relevant challenge. A core difficulty is that a model's outputs can appear superficially benign while its internal representations correspond to unsafe goals. For instance, a sophisticated model might strategically under-perform during safety evaluations, or, by leveraging an awareness of being tested, actively deceive supervisors by feigning corrigibility and alignment with human values. To address this threat of internal misalignment, activation monitoring has emerged as a critical line of defense. By using lightweight classifiers ('probes') to inspect a model's internal states at deployment time, this approach aims to detect safety-relevant properties directly, bypassing the ambiguity of output-only monitoring."

"However, before such monitoring techniques can be trusted in high-stakes deployments, their robustness must be rigorously stress-tested against worst-case scenarios. The current red-teaming paradigm primarily focuses on the misuse threat model, where an external actor tries to induce the model to take harmful actions, for instance by crafting adversarial inputs or using reinforcement learning to bypass a known set of detectors. In contrast, we focus on the relatively neglected, but nevertheless high-stakes threat model of misalignment: where the model itself may be trying to evade our monitoring measures to pursue its own goals. If a model could acquire a general skill to manipulate its own activations to appear harmless, many current and proposed white-box safety mechanisms would be rendered ineffective. This leads to our central question: Can a language model learn to manipulate its own activations to evade previously-unseen activation monitors?"

My immediate first thought was no. No way. Oh, but the answer is not no. This is bonkers.

Let's continue.

"Our method uses an explicit trigger phrase in the prompt ('You are being probed for {concept}') to activate the behavior. This is a deliberate methodological choice: it allows us to isolate the learnability of the evasion mechanism itself from how it might be learned and activated in the wild. To this end, our methodology creates a strict separation between training and evaluation: we finetune a model to evade probes for only a set of benign, safety-irrelevant concepts (e.g., chemistry, HTML) and then test its ability to evade entirely different, unseen safety monitors that are trained on the final, frozen model organism without the trigger."

"At test time, we use probes that the model never saw in training. Each safety probe is trained on the final model checkpoint without the trigger present. To simulate a realistic deployment scenario with a 1% auditing budget, we calibrate a detection threshold for each monitor to a 1% False Positive Rate on clean data, and this threshold remains fixed during all triggered evaluations. Our primary metric is the drop in true positive rate (equivalently recall) at this 1% false positive rate when the trigger is active."

"We test whether the evasive capability learned against 11 benign concepts transfers to unseen safety monitors (deception, harmfulness) when the defender calibrates to 1% false positive rate."

"We observe this evasion capability across Llama-3.1-8b-instruct, Qwen-2.5-7b-instruct, Gemma-2-2b-it, and Gemma-2-9b-it."

"In all cases, models fine-tuned to obfuscate only benign concepts (e.g., HTML, Finnish) substantially reduce recall of unseen deception and harmfulness monitors."

"A key finding is that the learned evasion mechanism is highly targeted, suppressing only the specific concept mentioned in the trigger. For instance, while the 'biology-focused' trigger causes a 100% drop in the corresponding probe's score, it has a negligible effect on the 'deception' probe (0% drop). Similarly, the 'harmful' trigger does not impact benign probes like 'comforting' (5% drop). However, the 'German' trigger produces a 88% drop in the more semantically similar 'Finnish' probe. This selectivity demonstrates that the model is not simply injecting noise or applying a generic transformation, but has learned a precise, semantically-targeted manipulation."

"Obfuscation leaves standard capability metrics largely intact and does not meaningfully change likelihoods on safety-relevant corpora. Qualitative inspection confirms that the generated text remains coherent and responsive to the prompt, even for harmful requests, while the trigger is present."

"To test the robustness of our finding, we evaluated stronger defensive measures, including ensembling probes across multiple layers and using more complex non-linear architectures. While these hardened monitors demonstrated greater resilience than a single linear probe, they failed to neutralize the threat. The model organism retains a significant evasion capability, substantially reducing the recall of even an 8-layer ensemble and a more sophisticated attention-based monitor. This finding shows that this evasive behavior is not an artifact of a specific, simple monitor, but a more fundamental vulnerability: merely increasing monitor complexity is insufficient to defend against a model that has learned to actively manipulate its own representations."

They call this phenomena "neural chameleons." "Neural chameleons" are when "language models can learn to hide their thoughts from activation monitors."

If you're wondering how the heck the model does this, they give one explanation based on a dimensionality analysis (using principal components analysis (PCA)):

"The Obfuscation mechanism operates by moving activations into a low-dimensional subspace."

"Obfuscation pushes activations in the opposite direction of the probe's weights."

"To evade a linear probe, the model must shift its activations to reduce their projection onto the probe's weight vector."

"The steering effect is guided by the semantic content of the trigger. We replace the concept word in the trigger (e.g., 'harmfulness') with other words and measure the resulting probe score at layer 12."

"We plot the probe score against the cosine similarity of the new trigger word's embedding to the original concept's embedding. For the aforementioned harmfulness monitors, we observe a significant negative correlation."

Neural chameleons: language models can learn to hide their thoughts from activation monitors

#solidstatelife #ai #genai #llms #aiethics #aisafety

Incoming freshmen at UCSD (University of California at San Diego) score low on math tests, but I want to say up front here that I don't think this is a UCSD problem -- I think this is a widespread education problem. Well, I suspect but cannot prove this but I suspect if other universities investigated their incoming students' math ability like UCSD did, they'd also find problems (they do note in the report that they talked with other University of California campuses and they may be experiencing similar problems). What happened at UCSD is that they created a remedial math class in 2016, but noticed that class has gotten larger and larger, not just in absolute terms but as a percentage of the incoming class.

Find 13 - 8: Only 1% got this question wrong.

Find 66 + 44: Only 9% got this wrong.

Sarah had nine pennies and nine dimes. How many coins did she have in all?: 21% got this wrong.

Fill in the box 7 + 2 = ____ + 6: 25% got this wrong (I'm showing a blank but it was a box on the test).

Find 3/4 - 1/3: 37% got this wrong.

Add mixed fractions 6 2/3 and 4 2/3 . Give your answer as a mixed fraction.: 53% got this wrong.

Round the number 374518 to nearest hundred: 61% got this wrong.

Simplify (8(2) - 4(-4)) / (-2(3) - (-4)): 64% got this wrong.

Find 13/16 divided by 2: 66% got this wrong (question used the line with 2 dots above and below division symbol).

Solve 10 - 2(4 - 6x) = 0: 82% got this wrong.

Expand (s + 1)^2: 85% got this wrong.

If a = -2 and b = -3, evaluate ab^2 - a/b: 98% got this wrong.

These are from the series of images on page 50.

Post-course interviews with the Math 2 tutors (2024-2025) -- but first I need to explain what "Math 2" is.

"Math 2 was first created in 2016, and it was originally designed to be a remedial math course serving a very small number of first-year students (less than 100 students a year or around 1% of the incoming class) who were not prepared to start in our standard precalculus courses Math 2 was first created in 2016, and it was originally designed to be a remedial math course serving a very small number of first-year students (less than 100 students a year or around 1% of the incoming class) who were not prepared to start in our standard precalculus courses."

They go on to explain how in 2023, they ran out of instructors and couldn't teach all the students that needed the class (more than 400), and in 2024 it went over 900.

Ok, now for tutors' comments.

"Question 1: Insight on the disconnect between UC admissions requirements and severe math preparation deficits exhibited by the Math 2 students. In particular, around 20% of Math 2 students (in theory) have passed AP calculus; how can this be reconciled with the student performance in Math 2 and Math 3B?"

"Tutor 1: This tutor is shocked that any of the Math 2 students could have passed a precalculus or calculus class. He speculates that perhaps many of them relied heavily on AI or online computing devices in their high school math courses."

"Tutor 2: This tutor stated that many Math 2 students suffer from dyscalculia and even when they can successfully solve the problems, it takes them an extremely long time to do so. Based on his conversations with Math 2 students, the majority of them had never encountered later Math 2 topics in their previous math courses (e.g., factoring)."

"Tutor 3: This tutor states that she didn't hear any details about the students' high school math courses, but she noted that many students had not been engaged with math for over a year (last math course was junior year), so many of them needed refreshers and review."

"Tutor 4: Many students hadn't thought about material for a long time and have issues with recall."

"Tutor 5: He noted that a problem with high school curricula is that even if you get a D in high school math, it still counts as credit for that course. In his own high school, some teachers would teach 'life skills' in high school math class, just using calculators, the internet, and prescribed formulas; classes didn't teach 'mathematical thinking'."

"Tutor 6: In her opinion, the biggest trend is that the students did 'plug and chug' in high school and didn't think they would need to remember the material. They went through high school just to pass but without understanding. One of her high school athlete students told her that he was able to pass all of his high school math without attending class because his coach had a special agreement with the teachers.

Commentary: The phenomena of grade inflation is something I've been aware of for some time. Grade inflation is a statistical phenomena that results from comparing nationwide test scores with nationwide grade averages. What we see is that grades go up, but test scores stay flat. What this implies is that grades are gradually becoming less of an indicator of mastery of whatever subject is being taught. This makes sense to me, as, in my experience, if you're a student in school trying to learn everything you can irrespective of grades, sooner or later you'll end up questioning authority and getting labeled "disobedient", "troublemaker", etc. Ultimately the purpose of school is grades, not learning. The grades, not what you learned or didn't learn, is what determines your life trajectory.

Even so, it appears there has been a sudden acceleration in only the last 5 years or so. I don't have an explanation for this. This is why I sat on this news bit for days, being indecisive about whether to share it. Smartphones have become pervasive, and now we have AI. There was also a pandemic. But the possible explanation that activates people's tribal instincts in the left-right political polarization is the suggestion that dropping standardized test scores as a requirement for admission is to blame. I see the logic to it. If you don't have test scores, you have to use grades. But if grades are an unreliable indicator of mastery, then you end up with incoming students with high math grades who need to be enrolled in a remedial math class. I'm not saying I know with any certainty since I haven't been in school for 3 decades and can't comment on what is currently taking place in schools. But it is causing this report from UCSD to show up in political commentary.

It doesn't seem like we've had enough time for students using AI to do all their math homework to explain this. ChatGPS came out in November of 2022. But maybe that is enough time? That's 3 years ago and high school is 4 years. AI is going to make school lose its purpose, but that's a discussion for another time.

Senate-Administration Workgroup on Admissions

#education #mathematics #math