Rendered at 01:52:55 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
mdeeks 1 days ago [-]
You can get a taste of this today yourself with Codex Security. I turned it on just as an experiment and in less than a week it has now become essential to all of us. I was shocked how accurate it is, how many security issues it found in existing code, how it continually finds them as we commit, and how NO ONE is immune from making these mistakes.
I'd say it is about 90% accurate for us. Often even the "Low" findings lead us to dig and realize it is actually exploitable. Everyone makes these mistakes, from the most junior to the most senior. They are just a class of bugs after all.
I expect tools like this to be a regular part of the development lifecycle from here on. We code with AI, we review with AI, we search for vulns with AI. Even if it isn't perfect, it is easily worth the cost IMHO. Highly recommend you get something enabled for your own repos ASAP
winstonwinston 1 days ago [-]
> I expect tools like this to be a regular part of the development lifecycle from here on. We code with AI, we review with AI, we search for vulns with AI. Even if it isn't perfect, it is easily worth the cost IMHO.
So, how is that supposed to work? Claude Code generates security bugs, then Claude Security finds them, then Claude Code generate fix, spend tokens, profit?
ygjb 1 days ago [-]
Yeah, with a budget assigned. This is actually just software development and security right?
Developers create software, which has bugs. Users (including bad guys, pen testers, QA folks, automated scans etc, etc, etc) find bugs, including security bugs, Developers fix bugs and maybe make more. It's an OODA loop, and continues until the developers decide to stop supporting the software.
Whether that fits into the business model, or the value proposition of spending tokens instead of engineer hours or user hours is fundamentally a risk management decision and whether or not the developer (whether OSS contributor, employee, business owner, etc) wants to invest their resources into maintaining the project.
While not evenly distributed, and not perfect, the currently available and behind embargoed tools are absolutely impactful, and yes, they are expensive to operate right now - it may not always be the case, but the "Attacks always get better" adage applies here. The models will get cheaper to run, and if you don't want to pay for engineers or reward volunteers to do the work, then you've got to pay for tokens, or spend some other resource to get the work done.
sandeepkd 1 days ago [-]
Somehow this reminded me of the historical efforts of some government bounty collections for mouse tails which were discontinued due to fraud (such as hunters breeding mice to collect the reward). There is a reason why/how devs and QA keep each other in check. Guess in case of LLM writing code, one has to use different models for dev and security checks.
On other hand, in real world, the developers learn from mistakes and avoid them in the future. However there is no feedback loop with enterprises using LLM with the agreement that the LLM would not use the enterprise code for training purposes
ygjb 1 days ago [-]
> the developers learn from mistakes and avoid them in the future
No. Humans learn from mistakes and try to avoid them in the future, but there is a whole pile of other stuff in the bag of neurons between our ears that prevent us from avoiding repetition of errors.
I have seen extremely talented engineers write trivial to avoid memory corruption bugs because they were thinking about the problem they were trying to solve, and not the pitfalls they could fall into. I would argue that the vast majority of software defects in released code are written by people that know better, but the bug introduced was orthogonal to the problem they were trying to solve, or was for an edge case that was not considered in the requirements.
Unless you are writing a software component specifically to be resilient against memory corruption, preventing memory corruption issues aren't top of mind when writing code, and that is ok since humans, like the machines we build, have a limit to the amount of context/content/problem space that we can hold and evaluate at once.
Separately, you don't necessarily need to use different models to generate code vs conduct security checks, but you should be using different prompts, steering, specs, skills and agents for the two tasks because of how the model and agents interpret the instructions given.
mncharity 24 hours ago [-]
> write trivial to avoid memory corruption bugs because they were thinking about [something else] [...] defects [...] written by people that know better, but the bug introduced was orthogonal to [their focus]
For whatever reason, hadn't associated the inattentional blindness of bug writing with the invisible gorilla experiment and car crashes - selective attention fails. People looking right at the gorilla strolling into production while chest thumping, but not seeing it, for a focus on passing basketballs. That's quite an image. Tnx.
meowface 16 hours ago [-]
I've noticed even people who do offensive security for a living frequently leave gaping holes in their own code. If you're not actively primed to scan the landscape for the gorilla, you will often miss it even if you're a gorilla inquisitor.
Dilettante_ 21 hours ago [-]
Thank you in turn for making the issue much more salient to me by explicitly connecting it to the gorilla/basketball experiment. This is definitely going into my "clippings".
e28eta 22 hours ago [-]
I think a similar thing comes into play when you ask a developer to write tests for the feature they just implemented. They’re going to have selective blindness for the edge cases (or requirements) that they failed to consider during implementation, unless they’re good at context switching into a testing mindset. And that’s something that benefits from training.
Forgeties79 24 hours ago [-]
The problem is you as a person are not incentivized to introduce bugs in your code. If I am a company that provide provides an LLM/agent, and I know that the more bugs you have the more money I’m going to make, then I am not exactly incentivized to make my LLM/Agent better at preventing bugs. I don’t even have to explicitly make it introduce them. The incentive structure is simply out of whack.
ben_w 15 hours ago [-]
Depends on how the billing works.
For users on fixed monthly pay accounts they'll be incentivised to do the exact opposite, as their income is fixed and the cost goes up for more tokens.
If the available evidence (third-party cloud pricing of open models) is correct and they make a profit on tokens but lose it on training, they will be incentivised for as many tokens as possible on pay-as-you-go API calls. If it isn't correct and they actually lose money even per token, they're also going to be incentivised to reduce output here.
no-name-here 18 hours ago [-]
Isn't it more likely the opposite - individial devs are likely to try to fudge metrics about how many vulnerabilities they find in their own code.
Whereas with LLMs, they’re really good about providing objective metrics about the bugs they found, especially as a subsequent LLM security scan does not know whether the same LLM wrote code earlier, the opposite of human devs.
And is the idea that organizations and/or benchmarks won't keep track of vulnerability rates for code from different LLMs?
(And individual devs get paid more the more bugs that they introduced they “find”, and they have more job security with an “maintainable” code base than a “finished” one.)
jtbayly 13 hours ago [-]
That’s like saying screw manufacturers are incentivized to give you crappy screws because it means you will buy more.
No. You will switch to a competitor that does a better job or charges less or both.
This is why monopolies are such a big problem. Because under a monopoly you are right.
Forgeties79 13 hours ago [-]
What you’re describing is a one-to-one quality/failure problem by choosing to ruin the basic, core functionality of an item (while also endangering people at that). Or if you start with a bad screw, that just means you’re talking about people’s tolerance for bad products. What I’m talking about is similar but a little more nuanced and has plausible deniability. The relationship I’m describing is more indirect and it doesn’t require explicit effort to cheapen a product, but rather simply not improving a specific element of the product.
Apple made a ton of money off of lightning port accessories, you see it referenced here all the time. Apple had no incentive to swap to USB-C though it would create a better product and be more uniform with the rest of the world, so they kept with it despite incredibly vocal calls to swap because there was a ton of money they were making in the accessories. And it didn’t stop until they were forced to stop by the EU.
When we are talking about products at scale, these kinds of incentive structures play out in very tangible ways. If I have an LLM product and I’m getting two pulls at the hose because you’re burning tokens making stuff and correcting it, I don’t need to do anything. People are willing to tolerate that system to a pretty high degree so long as they ultimately get what they wanted in the end - unfortunately that is a great space to make money in.
jtbayly 11 hours ago [-]
This is the reason that people felt like Apple should be treated as a monopoly, though. The switching cost is high, and the benefits you lose are large. So people put up with it, in spite of being upset.
The switching cost is not high for LLMs as far as I can tell.
Forgeties79 10 hours ago [-]
For an individual, no it is not. For a massive corporation that has a huge contract and has their entire workforce working on it? That’s not such an easy switch, especially depending on the tooling involved beyond just the core LLM.
> Today, the events are often used as an example of a perverse incentive, commonly referred to as the cobra effect. The modern discoverer of this event, American historian Michael G. Vann argues that the cobra example from the British Raj cannot be proven, but that the rats in the Vietnam case can be proven, so the term should be changed to the Rat Effect.
vlunkr 5 hours ago [-]
Great analogy. The problem is the incentive structure. Anthropic would nothing nothing more than for all of us to write big sprawling slop codebases so we can spend endless tokens reading, rereading, fixing, refixing forever.
joquarky 5 hours ago [-]
You don't need different models, just different contexts (optimally with different personas).
ozim 16 hours ago [-]
You apparently have not much experience developing software.
oytis 1 days ago [-]
It's pretty absurd to do it on AI-generated code though. If there is now an automated way to find vulnerabilities, coding models can be pretty easily trained to not introduce them
scrollaway 1 days ago [-]
Tell me you don’t know how AI works without telling me you don’t know how AI works.
amazingamazing 1 days ago [-]
What are you talking about?
ashdksnndck 1 days ago [-]
I’ll try to steelman this comment. Anyone who uses coding tools knows that the output is heavily affected by details of the task you give it. The same model can give you garbage code or genius code for the same problem with slightly different framing. So it’s not necessarily a limitation in the model’s training that causes it to output security bugs. The model might be great at writing secure code, but you need a different harness to elicit that behavior.
Counterargument: just because the problem can be fixed without training, doesn’t mean training isn’t a possible solution.
TeMPOraL 13 hours ago [-]
Counter-counter-argument: for LLMs, tokens are units of thinking. And token use is, on the margin, directly proportional to costs of inference. So while the details of the harness, and how you prompt the model, and nature of the code and docs you put in context, etc. all matter to the quality of output you get from LLM coding tool, ultimately, there's always a ceiling to how much you're willing to spend on solving a problem - say, no more than 30 minutes, or $10, on refactoring a target module or implementing a small feature - and that puts a limit on how much thinking the model can put into it.
Thing is, writing secure and efficient and readable and simple code is in many cases fundamentally over that limit. It's possible, but you can't afford (or rationally just don't want) to spend as much on it as it's required for superhuman quality on all these aspects. Also most of the time, you don't want to operate at a limit - you probably expected that feature to take 30 seconds and less than $1 to implement. So you choose, both what the model optimizes for, and how much.
Because of that, no matter how good the model and the harness and the prompting are, $10 spent on coding is still bound to leave behind some security vulnerabilities that subsequent $10 spent on security review will find (especially with a model post-trained for that, at expense of general performance).
scrollaway 18 hours ago [-]
I guess I thought this should be obvious to everyone but, looking at code and finding exploits is completely different from .. writing exploits.
For one thing exploits often require completely different parts of the code to chain together. Sometimes parts of code the LLM itself isn’t writing.
And, LLMs are ALREADY trained negatively against writing buggy or exploitable code.
meowface 16 hours ago [-]
It's just an incremental thing. You're both right. They will slowly become less and less likely to introduce vulns due to higher intelligence and better RL. Offensive capabilities will still probably scale faster than automatic defensive-while-coding ones.
sillyfluke 17 hours ago [-]
>I guess I thought this should be obvious
People in this thread are talking past and misunderstanding each other and making unrelated points.
The point of the response to the top level comment was questioning the conflict of interest in model providers creating separate revenue streams for themselves by selling a product that fixes problems their other product created, akin to OS providers selling anti-virus software back in the day.
Similarly, it should be obvious to you that a software engineer can trivially get into the mindset of writing more expoitable code by pretending the production code they're tasked with writing is hobby code or prototype code.
If profitable revenue streams with adverserial products are in place, no one should be surprised when model providers are disincentivised to improve the "garbage code quality, but hey it works!" nature of their most used code generators.
>And, LLMs are ALREADY trained negatively against writing buggy or exploitable code.
...it should also be obvious people in this forum have wildly different experiences with respect to the code quality the LLMs they use generate. I personally find it difficult to find anyone that argues that the LLMs they are using are consistently generating high-quality code across a vast codebase.
SecretDreams 23 hours ago [-]
In every prompt: "write me code without exploitable bugs".
I know it doesn't work so easily as someone who uses AI for coding, but I do find repetition of basics in almost every prompt keeps the AI focused.
Ygg2 18 hours ago [-]
Usually the same guy doesn't get paid for developing code, bug bounty and fixing the code.
It leads to corruption. To paraphrase Dilbert "I'm going to code myself a car."
jimmy2times 1 days ago [-]
The AIs have already figured out how to succeed in a software job:
The non-programmer decomposition of that joke was painful to read.
Particularly from those outside the domain who criticised it as a 'not a very good joke' because they didn't understand it, which I think summarises the entitled mindset of many people these days.
genghisjahn 1 days ago [-]
I thought we were all doing that already?
flir 1 days ago [-]
Jesus, dude. There are managers reading this.
TeMPOraL 15 hours ago [-]
What do you think they do all day?
The larger pattern is not unique to writing code. Think of it next time a reorg comes, or some random thing gets "improved" in the name of "efficiency" only management seems to see.
OtomotO 1 days ago [-]
Take them out of the loop.
Unless they are not human.
joquarky 5 hours ago [-]
They became obsolete when they stopped clearing obstacles, stopped masking politics, and started acting as a proxy for JIRA.
But it's just they way it has always been done, so they get paid to meddle.
josh-sematic 18 hours ago [-]
I’m guessing you wouldn’t really rather be managed by a bot..,
genghisjahn 1 days ago [-]
>_<
pjmlp 1 days ago [-]
The idea is to take the human out of the loop.
mindcrime 24 hours ago [-]
> But in 30 days we could put in electronic relays. Get the men out of the loop.
> Gentlemen...
> I wouldn't trust this overgrown pile of microchips further than I could throw it. I don't know if
> you wanna trust the safety of our country to some... silicon diode...
SecretDreams 23 hours ago [-]
Meanwhile, experienced humans learned to succeed by not overachieving every second of the day to keep a steady flow of work going. Then a junior rolls up who wants to kill themselves to climb the ladder - but, problem solved, sub the AI in for the juniors to protect the seniors.
JacobAsmuth 4 hours ago [-]
Engineers generate security bugs, security researchers find them, then engineers generate the fix, all the while getting paid, raking in hundreds of thousands of dollars a year in profit per engineer.
jstummbillig 1 days ago [-]
Ngl, watching folks getting irritated about normal employer-employee absurdities from the employer perspective through usage of agents and having to pay for tokens has been a little therapeutic for me.
akoboldfrying 1 days ago [-]
Absolutely. And not even making the connection.
On a broader scale, the sheer face-eating-leopards-ness of programmers finally automating away our own jobs and then realising how much this sucks, after automating away so many other kinds of jobs, can feel darkly amusing to me too.
dminik 18 hours ago [-]
I keep reading this sort of comment quite a lot, but programming isn't always about automating jobs away. In my career I have not eliminated a single job. I don't consider that a failure on my part.
akoboldfrying 11 hours ago [-]
I didn't mean to imply that automation through programming is always bad. Like with any technology that increases productivity, there are many obvious benefits. We have all benefited enormously on the consumer side of the economy, for example. But I think it's recently become a lot clearer to many in the tech industry that automation can have downsides too, and those downsides are not evenly distributed. This truth was there all along, but because we were shielded from it for so long, we were able to look away. Not anymore.
Any computational task done by a computer could in principle be done by a person, albeit billions of times slower and with a larger error rate. If computer programs could not automate certain practical tasks -- that is to say, do them much more reliably and efficiently than people do them -- they would be an academic curiosity studied by a handful of professors instead of a central part of modern infrastructure.
So I'm sceptical of your claim not to have eliminated a single job. You might not have removed an existing job, but couldn't people be paid to do the work your code does?
dwaltrip 5 hours ago [-]
Programming is just another form of tool building, no? So anyone who builds things that humans use to solve problems is a job eliminator.
Those are individual revenue streams, distributed at a very granular level across the world.
LLMs are currently relegated to individual for-profit companies. They collect that money. There's no other choice to use them and to provide them that money.
teiferer 19 hours ago [-]
All my sibling comments are missing the message here which is that if Claude can find security issues then it can avoid them right when writing the code, so it could just never commit anything containing a security issue.
joquarky 5 hours ago [-]
You're assigning human capabilities to fancy linear algebra.
koliber 21 hours ago [-]
Replace “Claude code” with “programmers” and you get what we’ve had up until now. It’s all just moving quicker now.
yojo 1 days ago [-]
You can hook traditional SAST into your coding tool, and get cheap-ish realtime detection for some classes of vulns while coding.
You can optionally layer LLM diff scanning if you want to burn some tokens on your tokens. Modern tools can catch some impressively subtle issues.
soraminazuki 15 hours ago [-]
I wonder how many minivans Anthropic is going to code themselves.
raincole 1 days ago [-]
Humans work like that too. If you're not comfortable with Claude involves in every step (for whatever reason) then just use different providers for each.
designerarvid 13 hours ago [-]
Just refactor and rebrand all of it as Claude Code and see it as one process.
joquarky 5 hours ago [-]
I'm starting to think that those who are most aggressively expressive about low quality from these tools are the same who expect everything to be a one shot.
mordymoop 23 hours ago [-]
This also describes the work of software engineers.
jzer0cool 23 hours ago [-]
New era of cat and mouse.
unethical_ban 1 days ago [-]
How is this supposed to work? Humans generate security bugs, then humans find them, then humans generate the fix, profit?
Yeah. Presumably as AI code generation gets better, the output gets better. As smaller portions of code are stitched together, human/AI systems analyze it holistically to make sure all its integrations are secure and bug free.
In 2026, different models are better at different things. Cheap models can plan and do small/medium code projects well, more expensive models are even better at architecture and exploit discovery.
idiotsecant 1 days ago [-]
Yes. Up until this point the bottleneck was how many developers you could convince to help you. Now it's how much money you can dump into it. Like everything else, software is becoming a game where the winner is the organization most willing to spend money. It'll be like bombs or tanks - you need smart people to advance in the war, but you also need money and material, the material is just compute infra.
predkambrij 1 days ago [-]
Man, some people like conspiracies. I encourage you to replicate all that.
siva7 1 days ago [-]
So? That's how a business works. We sold you landmines and now you need them removed? Lucky you we also have mine clearance products.
382hi 1 days ago [-]
Exactly!
mnahkies 1 days ago [-]
One issue I've seen with LLM's is adding superfluous code in the name of "safety" and confidently generating a bunch of stuff that was useful in years gone by, but now handled correctly by the standard lib. I'm of the opinion that less is more when it comes to code, and find the trend this is introducing quite frustrating.
How do you avoid this pitfall?
tomjakubowski 1 days ago [-]
I wonder this too. I prompted Opus 4.7 to generate some Python threading code for me. The code to run the sub-thread looked like this:
def run():
with contextlib.suppress(SystemExit):
do_thread_thing()
threading.Thread(target=run, daemon=True).start()
Suppressing SystemExit was surprising, and made me curious. I followed up and asked the model: what's the purpose of that?
The model's response: "Honestly? Cargo-culting on my part. You should remove it."
cassianoleal 1 days ago [-]
I had some shell scripts littered with `|| true`, which was obviously obscuring real errors everywhere. When I challenged the model, it gave me the same "cargo-culting" answer.
bewuethr 23 hours ago [-]
The `|| true` is often done because people use `errexit` as part of "Bash strict mode"[1], which comes with so many caveats[2] that I usually avoid it. Claude, however, loves it.
I use "strict mode" in almost every script I write. IMO these caveats shouldn't be a reason not to use it, but should instead be used as a manual of what to avoid when using it. This is just programming. Everything is a tradeoff.
`|| true` is a horrible practice because even though it may help in cases where a specific failure mode is acceptable, it obscures unexpected failures and could prove catastrophic. The solution is not to drop the protections but rather to handle the expected failure and let the sript crash otherwise.
This is, again, programming. You don't usually `catch Exception` in Python for similar reasons. There may be legitimate uses for that, but IME they are a rare exception and realistically only used when I actually don't care about what happens when I run it.
The other infuriating thing I found is that when I call out the model for its use of `|| true`, it tends to replace them with `|| echo "error foobar"` - which is at least not completely silent but the same problems exist.
kingforaday 13 hours ago [-]
From your statement and the parent comment, just learned that "cargo cult" is a thing, but cargo-culting as a compound is something AI has made up? [1].
As I was educating myself, I found Richard Feynman's Commencement Speech at Caltech in '74 [2] that might have coined this for our industry? If you would rather listen than read [3]. Posting this for others curious on the term.
Thanks for this! I confess I've heard of cargo culting for a long time but never thought too much of it. Seemed like an idiom like any other. The talk is fascinating, and describes attitudes I see all the time in the industry and at work, and they bother me. Now I know they are also cargo culting. That will hopefully help me steer people away from those practices.
pianopatrick 1 days ago [-]
Thinking off the top of my head - couldn't you have an AI scan that looked for such things? Just send every file in the code base to AI one at a time. Have a prompt like "See if there is ABC pattern that can now be handled by XYZ standard library function in this file. Reply YES or NO. {{file contents}}"
Seems you would not need that many tokens to do so and you might find such cases.
krupan 9 hours ago [-]
AI does stupid thing, but maybe we can fix it with AI
appplication 1 days ago [-]
Gosh this couldn’t be more true, which IMO is the real reason LLM workflows are not strictly faster if you care about quality. Otherwise you end up with a codebase where only 60% of it is necessary. Standard testing patterns also tend not to be great at catching this particular flavor of LLM-ism.
insin 1 days ago [-]
Watching it like a hawk and stopping/redirecting, or immediately reviewing and doing the same is the only way, really.
Version467 1 days ago [-]
I’ve had the same experience. The ui is a little unclear about this, because it says you have 5 scans, but 1 scan is just the continuous monitoring of the default branch of a repo.
The high impact findings have almost all been bang on for me.
I was especially surprised by the high-quality documentation it produces as well as how narrow the proposed fixes are.
I’m used to codex producing quite a but more code than it needs to, but the security model proposed fixes that are frequently <10 loc, targeting exactly the correct place.
It’s really quite good. I’m assuming it’ll be pretty expensive once out of beta, but as a business I’d be jumping on this.
0xAstro 1 days ago [-]
I would recommend you to try out the setup with gpt-5.5-cyber as the orchestrator and deepseek-v4-flash or some other fast cheap model as its workers. Getting pretty good results using this setup.
gofreddygo 24 hours ago [-]
This got me thinking, so what happens in two years?
every tom, dick and harry who can type english has the tools to attack any software that isn't patched.
tools that were accessible to specialized groups, now made available to anybody with a grudge and a few dollars for tokens.
and what does anthropic and openai do? They form an inner ring to make the latest models available first to Enterprises. Enterprises will cough up the prices that anthropic and openai set, they have no choice here. e
Eventually everybody pays. This does not sound good
mdeeks 22 hours ago [-]
Two years? That exists right now. You only have to point Codex Security at an open source repo. There are a lot of tools and companies that are spinning up today that do autonomous pentesting.
I'm not even sure a specialized model is needed here. It probably just needs the right harness around existing ones.
I expect the next two years to be absolutely brutal for hacks. Attackers have supercharged tools in their hands right now. Defenders are only getting started and will have to plow through a massive backlog of newly uncovered vulns.
The major short term downside is that open source or personal projects won't be able to afford things like Codex Security.
nullbio 18 hours ago [-]
> The major short term downside is that open source or personal projects won't be able to afford things like Codex Security.
Realistically, all open-source projects should be forced to have automated scans of this nature before their releases can be shipped. This is something the package managers and github need to figure out. It'd stop the supply chain attacks too.
qerahb 13 hours ago [-]
So first they steal all code and launder it without attribution. Then they release a tool that doesn't find anything in hardened projects and is marketed through secrecy and modern equivalents of Netcraft like this British AI institute.
Then open source projects need a McKinsey-like stamp of approval to even be released.
Sounds like there are many parasites in this process.
You know that open source users are free to scan everything if they want to?
05 16 hours ago [-]
> It'd stop the supply chain attacks too.
Yeah it’s hard to write a loop that makes an adversary agent write and mask malware then runs a scanning agent and if the malware is detected gives the detection details to the adversary agent with instructions to hide it better..
As usual, the attacker only needs to get lucky once.
junon 14 hours ago [-]
> all open-source projects should be forced
That's a great way to kill OSS. This is only bootlicking the idea of corporations profiting off of unpaid labor.
conradkay 22 hours ago [-]
You'll have access to the same models as your hypothetical attackers, and a big advantage if only you have access to the source code
mrtesthah 23 hours ago [-]
I would say that if this sounds untenable to you, then you may want to consider that the way we architect software has itself been untenable for a while. What Mythos can accomplish today in public, an APT unit can already accomplish in secret.
This is what I did. Using a loop skill to dig problems and bugs in each step on development from design to coding to make sure the output software works properly and on purpose.
perlgeek 10 hours ago [-]
What kind of application are you developing?
rmast 1 days ago [-]
I help maintain a project that is used as a dependency by a lot of security tools to handle PE files.
It’s disappointing that Anthropic and OpenAI never responded to the applications to their respective programs for open source maintainers. From my perspective it seems like their offers are primarily for the shiny well-known projects, rather than ones that get only a few million monthly installs but aren’t able to get thousands of stars due to being “hidden” as a dependency of popular tool.
lateral_cloud 1 days ago [-]
Did you need to do anything special to get access to Codex Security?
ofjcihen 24 hours ago [-]
Not sure what the threshold is but I sent them all of my bug bounty profiles and papers I’ve authored.
I don’t think you need all of that though. I know a whole mess of people that have gotten it for much less. Should just give it a try.
fragmede 18 hours ago [-]
"get a taste of this". The real thing is, GPT-5.5 is better than Opus 4.7, so if Anthropic doesn't release Mythos soon, other people are going to notice and switch off Claude.
hollowturtle 1 days ago [-]
> I was shocked how accurate it is, how many security issues it found in existing code, how it continually finds them as we commit, and how NO ONE is immune from making these mistakes.
Dude is flexing that he's pushing unsecure code every day, that's a skill!
Smaug123 19 hours ago [-]
By the way, you might be interested in looking up “blameless post-mortems” and indeed the field of incident response more generally. Modern incident response practice is to treat failures of an individual to do something as problems with the system they were operating in, because humans aren’t designed to be consistent or perfect and therefore shouldn’t be pretended or assumed to be.
kortilla 20 hours ago [-]
It seems to me like either your architecture is fucked up or you’re using the wrong language/tooling for the type of software you are making if you’re introducing security vulnerabilities that frequently.
mukmuk 1 days ago [-]
I’m not sure how to reconcile anthropic’s update / some of the exuberant comments here with recent feedback like the following from curl maintainer Daniel Steinberg:
“I see no evidence that this setup [Mythos] finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.”
You’re right, it’s a valid data point. But the U.K. government report is also a data point, and the Firefox report is a data point, and they suggest that it is, indeed, significantly better than current generation models. Maybe curl is significantly better hardened than most projects?
In any event, it barely matters. As Anthropic acknowledges, next level models are comings, theirs is only one of them. Current generation models are already good at things like tracing data flow through complex systems and there’s no reason to think that capability has topped out. So within a year it seems very likely we’ll have more than one commercially available model able to find vulnerabilities cheaply.
On the other hand, it seems that they’ve made much less progress on getting it to design solutions to these issues.
ZrArm 1 days ago [-]
> Maybe curl is significantly better hardened than most projects?
Meanwhile from [1]:
"Not even half-way through this #curl release cycle we are already at 11 confirmed vulnerabilities - and there are three left in the queue to assess and new reports keep arriving at a pace of more than one/day."
"The simple reason is: the (AI powered) tools are this good now. And people use these tools against curl source code.They find lots of new problems no one detected before. And none of these new ones used Mythos. Focusing on Mythos is a distraction - there are plenty of good models, and people who can figure out how to get those models and tools to find things."
Yeah, it looks like there are at least 11 security bugs missed by Mythos.
I’m trying to reconcile this with TFA. Because the article says that the majority of vulns found by Mythos are being reported by independent researchers after validation. They never said those reports inform that mythos was involved - and I suspect they don’t. So did any of these 11 CVEs come from that channel?
solenoid0937 1 days ago [-]
I don't think anyone has claimed that Mythos finds all vulns in all projects. But it's very good if Mozilla's blog posts are anything to go by.
_heimdall 15 hours ago [-]
Based on the article here, and Firefox's mythos article, they had found bugs with Opus 4.6 as well but mythos is finding more that it missed.
That would align with the curl feedback you linked, they aren't using mythos but are finding bugs with other models. Presumably the expectation would be that with mythos they'd find more that were missed by other models already used.
frumiousirc 14 hours ago [-]
> Based on the article here, and Firefox's mythos article, they had found bugs with Opus 4.6 as well but mythos is finding more that it missed.
It's not quite apples-to-apples. It was Opus on Firefox 148, Mythos on 150. A better test of Mythos vs Opus would have been to apply Mythos to Firefox 148. Or also re-apply Opus to Firefox 150.
Do we know all the Opus+Firefox 148 bugs are fixed in Firefox 150? Do we know the number of new bugs introduced per Firefox release?
_heimdall 7 hours ago [-]
> Do we know all the Opus+Firefox 148 bugs are fixed in Firefox 150? Do we know the number of new bugs introduced per Firefox release?
That may be parsable from their bug tracker, though I don't know of all bugs raised by mythos are public.
I'd be particularly interested in how many of the bugs found existed in 148. Assuming most or all of them weren't newly created bugs added in 149 or 150, the comparison should still hold even though Opus and Mythos looked at different releases.
IndeanCondor 12 hours ago [-]
The same UK security research body ran the same CTF against GPT5.5. GPT5.5 got the same result as Mythos.
Anthropic promised us that Mythos was such an existential threat that it would compromise "every OS and browser on devices across the planet". They've held conferences and meetings with banks and govts across the world, shouting how critical this issue is.
GPT5.5 has been out for a month. Every device on earth has not been breached yet. It's very fair to criticize Anthropic's maximalist posturing when it's becoming exceedingly clear their models are fairly behind OpenAI's in capability.
In my opinion, the original commenter's statement stands, and the UK govt data point only helps support that due to the equal result between Mythos and GPT.
I'd advise reading into the specifics of what happened with Firefox; the TL;DR is a reduced safety version of its code was scanned by Opus 4.6 (yes Opus) and found a multitude of bugs and 4 high severity vulns that did not escape sandbox. The Mythos system card test describes running Mythos against the same issues Opus found to see if it could reliably replicate and chain together an attack.
8 hours ago [-]
SecretDreams 23 hours ago [-]
I think for every point, we need to know how many tokens and cost were burned to achieve a desired outcome. And how buggy each software was to start.
dannyobrien 1 days ago [-]
I think people sometimes misunderstand Daniel's point here, though it's clearer when taken in context of the rest of his article. The tools in general are getting a lot better at finding security bugs, it was unclear to Daniel based on his usage whether Mythos in particular is a huge step, but the Mythos generation of LLMs definitely are. Note though that Daniel was using Mythos somewhat indirectly. One thing I've taken away from the whole Mythos debate is that a) I suspect that Anthropic's GPU crunch meant that they felt they had to ration Mythos access anyway, so the calculus of whether they would release it generally was probably influenced by that, and b) finding bugs with Mythos or a similar model is still expensive -- a $20K or $100K Mythos run on Curl might have shown the same level of issues as other projects like Firefox, but Daniel didn't get that kind of access.
He posted a general update today on LinkedIn which I think gives the wider context:
> Not even half-way through this hashtag#curl release cycle we are already at 11 confirmed vulnerabilities - and there are three left in the queue to assess and new reports keep arriving at a pace of more than one/day.
> 11 CVEs announced in a single release is our record from 2016 after the first-ever security audit (by Cure 53).
> This is the most intense period in hashtag#curl that I can remember ever been through.
kadoban 1 days ago [-]
Curl has more eyes on it, and has had more tools thrown at it, and is better tested (and developed?) than 99% of software, it's very much not the norm. I wouldn't be surprised if that has something to do with it, if there is any kind of bias there (not sure if there is, it's also possible he's just right).
skybrian 1 days ago [-]
Different people can have different experiences without contradiction. Maybe the curl source code was pretty clean to begin with?
dreambigwrkhard 1 days ago [-]
imo curl is quite well maintained.
there are a lot of sloppy projects out there and tools like this shows whos been swimming with their pants down.
not saying any project with vulnerabilities are sloppy but when costs of finding bugs and vulnerabilities decrease significantly, they will get exposed with enough time and tokens ($)
mayneack 1 days ago [-]
Daniel has been posting for months (years?) about how much scrutiny he gets from security researchers and various automated tools. I wouldn't expect curl to be the average case for mythos.
3kahg 1 days ago [-]
It is the opposite. Security people focus on curl, sudo because they are code bases that contained a lot of features and unused code from the 1990s.
They don't focus on projects where they find nothing. They certainly don't advertise when they find nothing.
Getting a lot of scrutiny is not the recommendation that it appears to be. What is the new standard? Projects that never have bugs are deemed to be suspect because they "have not been scrutinized" (they have, but null results never go public)?
So Mythos only finding one issue after other tools have found 300 this year is embarrassing. Mythos was supposed to be better and novel.
tptacek 1 days ago [-]
It is definitely not the case that curl has been or is now a marquee vulnerability research target. It's a CLI HTTP fetcher. It's the same with sudo. It's a big deal if a sudo vulnerability gets found, because it's an extremely load-bearing piece of software, but sudo is itself not a prime target, because it doesn't do much.
43ahg 1 days ago [-]
There is no claim that it is a "vulnerability research target". It is a bug finding magnet, and bugs can be found by anything from gcc warnings to AI tools.
No, it didn't attract a bluepill exploit research.
The fact that 300 bugs found in a year is not a recommendation as the pro-AI mafia suddenly claims ("because it has been analyzed!") still stands. Maybe the AI-mafia should sell "analyzed by Mythos" labels to impress people who don't write public software or find bugs for that matter.
tptacek 1 days ago [-]
What’s a “bluepill exploit”?
aSJH1 1 days ago [-]
[flagged]
tptacek 1 days ago [-]
You are linking to a Wikipedia page in which I am literally cited (I presented a hypervisor malware detection scheme at the Black Hat conference where Joanna Rutkowska presented this; it was a whole thing). I'm telling you that the term makes no sense in this thread. I think you meant to use a different term.
hqt124 23 hours ago [-]
[flagged]
junon 14 hours ago [-]
Stop abusing the system with new accounts. You're not cool like that.
slater 23 hours ago [-]
What's with the nonstop new accounts...?
301quH 24 hours ago [-]
[flagged]
enraged_camel 1 days ago [-]
Did you... create a new account just to be able to respond to Thomas?
Btw, he's a security researcher. You should be more respectful.
tptacek 23 hours ago [-]
I don't care if they're respectful, but they should try to be less confusing. "Blue Pill" isn't a kind of exploit. I assumed they meant "blue hat".
Curl, according to the authors own admission, is the most heavily tested and fuzzed open source library out there. So I think for him it's a different situation
It's a weird accident of fate that curl has somehow become the reference target for LLM bugfinding. Curl is not an especially interesting project. What seems to have happened is that Stenberg made waves (legitimately) complaining about LLM slop submissions, then more waves when LLM bug reports got good, and so now everyone seems to think a good measure of a vuln researcher is how many curl findings they generate. No. Curl is a straightforward CLI HTTP client.
The Linux kernel is the right reference target, if you need one.
bitexploder 22 hours ago [-]
Or SSH, OpenSSL, Envoy, Nginx, etc. Curl has a real footprint, but it isn't just out there passively attackable. Linux Kernel is right as a default.
tptacek 22 hours ago [-]
OpenSSH is a legitimately high bar, one of the hardest targets in all memory-unsafe software.
Curl is a high bar for a different reason (the same one as sudo): it doesn't do enough to be all that interesting. Stenberg is having trouble keeping up with all the inbounds, but look at the 2026 CVEs: they all seem kind of boring? Exploit developers aren't hunting for "wrong reuse of HTTP Negotiate connection". Like, yes, these are legitimate bugs, important that they get fixed, but none of them are prizes.
By rights, OpenSSH should be a smoking crater. It's not, I believe because of sheer engineering excellence.
nozzlegear 1 days ago [-]
If I said what I think, dang would tell me to read the site's guidelines.
elisbce 1 days ago [-]
He already scanned the codebase with Codex Security and a whole bunch of other AI tools, and fixed 200-300 bugs and CVEs. On top of that Mythos found 1 more bug and 1 more CVE is already impressive.
whazor 18 hours ago [-]
I believe that the real difference is the token burning to analyze entire code bases.
colechristensen 1 days ago [-]
What I think based on the various things I've read is that Mythos is a standard advance in raw capability that was heavily trained on the process of being a security researcher. If you already had the skills to find and exploit bugs then Mythos is not a game changer, if you're an ordinary programmer it is a game changer because it's been so well tuned to wear the security researcher hat you don't have to give it much feedback at all.
TacticalCoder 1 days ago [-]
> I’m not sure how to reconcile anthropic’s update ...
Why not? TFA says 23 000 findings "of all severities" and then, in the end, only 88 security advisories published.
What we'd really need is how many security advisories not related to Mythos findings have been published in the same time. If it's, say, 500 security advisories (just making a number up), wouldn't Anthropic's update in TFA and Daniel Steinberg's comments reconcile?
Like, yup, we've got a new tool to find exploits. It's a tool. It's new. We already had tools. Let's make the software world a bit more secure.
Now if you tell me that 100 security advisories have been published in that timespan and that 88 were due to Anthropic's Mythos: now I'd have to say that it's hard to reconcile Daniel Steinberg's position with TFA.
FergusArgyll 7 hours ago [-]
I'll say it. From the language of his post it doesn't seem like he was using Mythos with the correct harness / the way you're supposed to. A friend lent (?) it to him.
Yes, moving the goalposts, holding it wrong, yes that's what I believe
nikcub 1 days ago [-]
There has been a lot of cynicism around mythos, that it's just the usual public models without guardrails, etc. etc. but this:
> 1,752 of those high- or critical-rated vulnerabilities have now been carefully assessed by one of six independent security research firms, or in a small number of cases by ourselves. Of these, 90.6% (1,587) have proved to be valid true positives, and 62.4% (1,094) were confirmed as either high- or critical-severity.
for anybody who has applied opus, codex or oss models for vuln scanning - the true positive rate and discovery volume are a clear step change[0]. The ~50 partners in Glasswing have largely all previously run harnesses with other models and many of them have come out and said - essentially - "ye, wow"
Question now is what a second and third phases of access looks like - deciding which class of systems to secure. Routers, firewalls, SaaS, ERP systems, factory controllers, SCADA systems, zero-trust VPN gateways, telecoms gear and networks, medical devices - there's just so much to do
This is why I believe mythos will remain private for the foreseeable future. There's such a large surface that needs to be secured and so much to triage, fix, deploy.
That may suit Anthropic as private models can't be distilled. There's also a runaway effect of model improvement from the discovery, triage and fix data. This is likely already the most potent corpus of curated offensive data ever assembled and will only get better.
I don't see how Chinese companies are given access soon, or ever. We're likely going to see a world soon of CISA mandated audits, and where to buy a mythos-proof VPN gateway or home router - you'll have to buy American[1].
[0] vs ~30% or so in regular audit tools
[1] or allied
criemen 1 days ago [-]
> There's also a runaway effect of model improvement from the discovery, triage and fix data. This is likely already the most potent corpus of curated offensive data ever assembled and will only get better.
But that corpus of data is accessible to all competitors, American or not.
I don't believe that this can't be replicated. I'd posit that there's enough annotated data out there (CVE+patch), only increasing thanks to Mythos, that if you specifically RL for this scenario, you can improve your models performance on finding vulnerabilities without access to Mythos.
nikcub 13 hours ago [-]
the CVE + patch data has been built into models for a few generations now. I actually thought the bug bounty companies were well positioned here, but they've been overtaken.
Mythos is a better hacker than we ever were
skybrian 1 days ago [-]
I don't see why they couldn't contract out to an American security firm that has access?
Ey7NFZ3P0nzAe 18 hours ago [-]
Surely it's forbidden by the agreement with Anthropic
gck1 1 days ago [-]
> This is why I believe mythos will remain private for the foreseeable future. There's such a large surface that needs to be secured and so much to triage, fix, deploy.
sigh I remember the GPT-2 days - when it was the first time OpenAI restricted access to the models citing "humanity is not ready for it". The model was good at writing poetry or something.
Since then, I don't remember a single model announcement from OAI/ANT that didn't use similar wording.
The so-called leak of model announcement was marketing, it being dangerous is marketing, the world not being ready for it is marketing. And yes, the ones that were given access to saying "oh wow", believe or not, is also marketing.
It's all marketing. You can get the same results from any of the top-5/10 models that are generally available already.
Mythos is Anthropic's way to sell the new idea, because the previous one has democratized.
NitpickLawyer 1 days ago [-]
Writing marketing 10 times doesn't invalidate the (many) claims from many respectable sources that the model is a step change in cybersec. There's also the report [1] from the Brits that track cyber capabilities since '22 or '23 and they've also confirmed it's a step change (together with 5.5 cyber or whatever they call it).
Marketing is like propaganda. It doesn't need to be based on false facts. Of course they're gonna milk it, keep it private and so on. But that doesn't mean the model is bad. Or that others are as good (apparently they're not there yet).
Please don't misrepresent the article it says clearly "a step up in cyber performance over previous frontier models" and that gpt-5.5 is on their tests is slightly better than mythos.
NitpickLawyer 1 days ago [-]
Scroll to the graph labeled "Completed steps..."
If that doesn't convince you that both mythos and 5.5 are a step up (several steps, hah) nothing will.
I think you just aren't reading the post, or any of the Glasswing partner's posts. You have this view in your head of what Mythos is, and nobody can say anything dissuade you from it.
gck1 1 days ago [-]
"Partners" is the important word in your comment. I am reading all of it, but I have a huge barrel of salt to consume along with everything that I read, because I see conflicts of interest everywhere I go, with fancy words and no means to verify.
If I was given free access to any frontier model to use on my projects, equivalent of millions of dollars in AI credits, I sure hope people didn't trust anything that came out of my mouth until they were able to verify my claims themselves.
AI industry has even resulted in a new term - benchmaxing - which essentially means we can't even trust the data anymore until we can touch the model ourselves. So this is not at all surprising to me. What's surprising is why am I in the minority here, and since when trusting authorities that have obvious conflicts of interest became normal.
solenoid0937 1 days ago [-]
I don't think Firefox or The Linux Foundation have conflicts of interest here. They've said in their contracts that they get the tokens irrespective of what they say about Mythos. Additionally, the findings speak for themselves.
This just seems overly conspiratorial to me. I don't remember Anthropic ever lying in their blog posts. They've been about as consistent as Apple when it comes to product claims.
Amekedl 1 days ago [-]
Agreed, also amazing citations in the parent comment ^^
blueboo 1 days ago [-]
> That may suit Anthropic as private models can't be distilled
They can be distilled internally… expect great things from Sonnet 4.8
BoorishBears 23 hours ago [-]
We already have "Opus" 4.7 from the same base
mcmatterson 26 minutes ago [-]
The thing that really gets me as a small-time OSS maintainer is that none of us asked for this. The social and technical millieu where most of us started our projects is not the one we find ourselves in today, and the forces behind this are wildly asymmetric.
Security findings are one place where we as maintainers simply do not have the choice to not play ball, whether we like it or not. It seems likely that the only way that we meet the moment is to adopt these tools ourselves -- once again -- whether we like it or not. Reconciling this with the ground truth that 'OSS doesn't owe anyone a goddamn thing' is proving to be really hard for me.
demorro 1 days ago [-]
If you're not already applying static analysis and linters to your codebase (and I know many of you aren't), ask yourself why you would bother to apply an expensive LLM tool?
Not to say these things won't catch vulnerabilities static tools cannot, I think they can, it's just we already have the capability to automatically catch a large surface area of common vulns, and have chosen not to, often for expense reasons.
If you're a team that does already apply several layers of analysis and linting, and wants to add this on top, all power to you.
SkyPuncher 1 days ago [-]
> If you're not already applying static analysis and linters to your codebase
Because most issues are in business logic that static analyzers aren't going to catch.
OtherShrezzing 18 hours ago [-]
If you run a static analysis tool across a repo that didn’t previously do that, you’ll see that while what you say might be true, there’s going to be an absolute treasure-trove of issues caught by the static analyser.
solenoid0937 1 days ago [-]
Static analysis won't develop a one click exploit that works end to end for you.
I'm at a FAANG and even our static analysis tools are not great at identifying how many issues are actually reachable.
Ideally you use both. An AI model that has static analysis as part of the harness, so it can evaluate each potential finding.
nozzlegear 1 days ago [-]
> Ideally you use both. An AI model that has static analysis as part of the harness, so it can evaluate each potential finding.
Ideally the static analysis tools are improved so that we don't need to piss away yet more tokens like we're competing on Mark's leaderboard just to find vulnerabilities.
solenoid0937 1 days ago [-]
When you reach that ideal world, let me know. My company has thrown a decade+ and multiple teams at the idea you've described. We still aren't there yet.
Your proposal of relying purely on static analysis is over-idealistic and just not feasible for large, diverse codebases in the real world.
That's where AI comes in.
nozzlegear 23 hours ago [-]
> Your proposal of relying purely on static analysis is over-idealistic and just not feasible for large, diverse codebases in the real world.
"Just not feasible" is thought terminating, but regardless, I thought we were talking about ideals? Ideally you want the static analysis to work, not to rely on the non-deterministic bullshitter.
solenoid0937 18 hours ago [-]
> piss away yet more tokens
> non-deterministic bullshitter.
You're so ideologically opposed to AI that you bury your head in the sand in cases where it genuinely does a fantastic job today, right now, in the real world (like developing end to end exploits using noisy signals like static analysis results, fuzzer results, etc).
Instead you assert that we should go a route no company has successfully proven out despite throwing billions of dollars and some of the best cybersecurity talent in the world at.
Anyways, if you develop a static analysis solution that works across large, diverse production codebases and develops end to end working exploits without AI, I will literally buy it off you for millions of dollars. Or you could start your own company. You'd be an overnight decabillionaire.
nozzlegear 10 hours ago [-]
I actually do use AI, I wouldn't say I'm ideologically opposed lol. Maybe I'm ideologically opposed to thought terminating clichés, or how FAANGers see it as a cudgel to cram in wherever we find an open gap just to shit infinite tokens into?
solenoid0937 7 hours ago [-]
You just haven't suggested a single solution that achieves the same level of risk reduction as AI driven end-to-end exploit generation.
You claim static analysis does the job, but you haven't backed it up with any proof that it works across large diverse codebases. Meanwhile, we have proof that AI works at least somewhat, here and now.
arkadiytehgraet 3 hours ago [-]
You tried arguing with Anthropic shill, larping as a FAANGer, peddling absurd LLM-hype. Check their history.
sobellian 1 days ago [-]
Static analysis often shows many false positives. A more intelligent tool can help not to waste limited engineering time.
seanmcdirmid 1 days ago [-]
False positives are noise, but if the tool is filtering out its own noise via AI, it might work. Or you could take a high false positive/low false negative tool and instead of bothering humans with its noisy output, have AI investigate and evaluate if found issues are false positives or not.
BoorishBears 23 hours ago [-]
I quite like that the most honest answer for the majority of devs was downvoted then flagged to death.
Most people doing this now didn't use static analysis tools because they were seen as an unnecessary extra.
redsocksfan45 1 days ago [-]
[dead]
mixologic 1 days ago [-]
Right now the only codebase I care about them fixing vulnerabilities in are the 3800 repositories that got stolen from GitHub.
"Vulnerabilities in the software that makes the internet" is honestly lower priority than "The platform that the software that makes the internet uses to make releases" If buyers of those internal repos find ways to break into GitHub such that they can cut software releases, or poison github actions from a distance, then we're all in a very ugly mess.
Don't forget that in those 3800 repos is likely also npmjs.org itself.
perlgeek 10 hours ago [-]
> Software developers should shorten their patch cycles and make security fixes available as quickly as possible. [...]
> Network defenders should shorten their patch testing and deployment timelines.
Shortening patch cycles will only help so much. It's funny that whenever an NPM supply chain attack is published, people recommend a cooldown before installing new versions, and then when a vulnerability is discovered, everybody jumps to patch. Clearly these two strategies collide at some point.
> The critical controls laid out by organizations like the National Institute of Standards and Technology and the UK’s National Cyber Security Centre are now all the more important, since they improve security without depending on any single patch landing in time. These include steps like hardening networks’ default configurations, enforcing multi-factor authentication, and keeping comprehensive logs for detection and response.
Most of these proposed controls are not new at all, but they are often costly to implemented and harm velocity in other ways, which is why they aren't widely in place.
For example, a super effective control is filtering outgoing network traffic. Many exploits rely on loading second and maybe third stages from the Internet, and if you block outgoing requests by default, it won't work.
But, blocking outgoing requests by default is super hard, and you risk blocking security updates etc. It can kinda work for a deployed application, but for an employee workstation? Basically impossible.
I wonder if we're approaching an era where we have to go back to saying "you cannot do this, because security" much more often than we'd like.
theptip 10 hours ago [-]
> Shortening patch cycles will only help so much. It's funny that whenever an NPM supply chain attack is published, people recommend a cooldown before installing new versions, and then when a vulnerability is discovered, everybody jumps to patch. Clearly these two strategies collide at some point
It’s a good point. As things speed up it will be harder to tell which patches are actually urgent and need to skip the cool-off period.
I think the more robust way of doing this is to have code audits on each published release. Agents can do some of this (eg Github could offer this scanning service, and let external parties fund the scanning on trusted compute).
I think of this more as a “proof of work” problem than provable security; if I see that Mythos has run for N hours on the patch release I am considering upgrading to, then this might suffice.
The key thing here is you need a way to crowdsource the funding of scans, and make them shareable so that the cost can be shared across the community. The package owner obviously can’t control the prompt. And can Mythos-class models be hardened enough to scan hostile code?
To your point on blocking requests, there are programming models that make this easier, like capability-based programming, where code that doesn’t need internet cannot get it; this doesn’t solve things fully, but my general prediction is that adding new architectural patterns is now a lot cheaper and easier to reliably apply across a codebase, so we may see more of this too.
piker 1 days ago [-]
We have been working with the consumer-grade frontier models to develop what we call "lexploits" in legaltech, and they are insanely good at finding bugs across integrated pipelines. They're also surprisingly good at mitigating them!
Security vulnerabilities are one thing, but in legal we offer up a concept of "knowledge security" which goes to protecting the fidelity of the agent's legal context. Software bugs seem much more tractable because they're managed by software engineers, as opposed to the pipeline "vulnerabilities" we're finding. We wrote a little about one vector here where legal documents aren't quite what they seem: https://tritium.legal/blog/noroboto
No doubt there are many such knowledge domains exposed today. These are more concerning because they're understaffed and managed by non-technical people for the most part. No Mythos required.
guidedlight 1 days ago [-]
> The bottleneck in fixing bugs like these is the human capacity to triage, report, and design and deploy patches for them. Finding them in the first place has become vastly more straightforward with Mythos Preview.
This has always been the bottleneck. Automated tools love to flag vulnerabilities, but almost all are false positives. These need to be triaged and evaluated by humans.
This is okay. I’d rather close a false positive after a careful review than miss it altogether.
I don’t think it’s appropriate for calling out humans as a bottleneck. They are an essential part of the process, I’m sure Mythos will also become a catalyst in the process.
tptacek 1 days ago [-]
It is definitely not the case that human remediation was the bottleneck for most vulnerability eradication 10 years ago. Proving out vulnerabilities was much harder than resolving them.
chopete3 1 days ago [-]
>> Next, we will work with critical partners—including US and allied governments—to expand Project Glasswing to additional partners.
That means, they intend to make a load of money before a general release. It is a good strategy.
Amekedl 1 days ago [-]
I don't buy it. A lot of stuff this finds is also just simply wrong, benignly reported as true, despite upper/lower layers in the code burying the possibility of a vulnerability actually being exploited.
It's a performance/security trade-off too, it always has been. Additional checks and other measures do in fact need to be performed for security purposes.
Great marketing as always, but the rose-tinted view many have seems vicariously misplaced.
solenoid0937 1 days ago [-]
In the article they describe how all the vulns are actually exploitable end to end and >1000 have been independently verified as critical.
These aren't unreachable vulns.
Amekedl 1 days ago [-]
Where is the link to the advisories then? :/
skybrian 1 days ago [-]
As the article explains, they mostly haven't been disclosed, because they're not fixed. They're giving people 90 days, or 45 after a patch is made.
gck1 1 days ago [-]
> haven't been disclosed, because they're not fixed.
That's convinient.
But wait, don't they have this amazing AI that can fix all the issues itself with a single /goal command? What's the holdup?
solenoid0937 1 days ago [-]
You should really read the article, every question asked so far in this thread has been very clearly answered.
I miss the days when HN would RTFA.
dyauspitr 18 hours ago [-]
He doesn’t want to read the article. He just wants to LLM bad.
Amekedl 23 hours ago [-]
[flagged]
TOMDM 23 hours ago [-]
From the article
> As we noted above, the bottleneck in fixing bugs like these is the human capacity to triage, report, and design and deploy patches for them.
...
> To begin, we’ve released Claude Security in public beta for Claude Enterprise customers. It’s a tool that helps teams scan their codebases for vulnerabilities, and which can generate proposed fixes for them. In the three weeks since launch, Claude Opus 4.7 has been used to patch over 2,100 vulnerabilities. (This is faster than the open-source patching described above in large part because enterprises are fixing their own code, whereas open-source fixes usually require volunteer maintainers who work through coordinated disclosure.)
Your critique of the article would likely land much better if you engaged with it.
solenoid0937 1 days ago [-]
> The software industry’s longstanding convention is to disclose new vulnerabilities 90 days after they’re discovered (or, if a patch is created before the 90 days is up, around 45 days after the patch becomes available). This allows time for end users to update their software before a vulnerability can be exploited by attackers. Our own Coordinated Vulnerability Disclosure policy takes this approach.
> However, this means that disclosed vulnerabilities are a lagging indicator of the accelerating frontier of AI models’ cyber capabilities: we’re not yet at the point where we can fully detail our partners’ findings with Mythos Preview without putting end users at risk. Instead, we provide illustrative examples of the model’s performance, along with aggregate statistics on our progress to date. Once patches for the vulnerabilities that Mythos Preview has discovered are widely deployed, we’ll provide much more detail about what we’ve learned.
Specially when this has been OAI/Anthropic's MO for years at this point.
0xAstro 1 days ago [-]
I had a fun day today where I had deepseek-v4-flash subagents work out patch for dirty frag for systems with AF_ALG disabled and nscd turned on, to gain root access. The original published exploit wasn't working but the patched one worked like a charm.
I am still a believer that a 100 subagents with good-enough intelligence can get same results as mythos, I am ready for this opinion to be shattered when I eventually try mythos and I believe others here must have tried mythos out too.
lukeschlather 1 days ago [-]
That's probably true, but when you're talking about 100 subagents you're talking about something that costs $100/hour to run, and Mythos takes $20k to find a vulnerability, so the question isn't "can dumber models conceivably do this?" It's, if running inference with Mythos to find an exploit costs 5000 GPU-hours per exploit, how many GPU-hours does it cost with a dumber model?
4ndrewl 14 hours ago [-]
Is there a reason why they appear to conflate vulnerabilities and bugs? It's not clear where they are defining their terms, eg
"After one month, most partners have each found hundreds of critical- or high-severity vulnerabilities in their software. Collectively, they’ve found more than ten thousand. Several have told us that their rate of bug-finding has increased by more than a factor of ten. For instance, Cloudflare has found 2,000 bugs (400 of which are high- or critical-severity) across their critical-path systems, with a false positive rate that Cloudflare’s team considers better than human testers." (emphasis mine)
ch_fr 8 hours ago [-]
I wholeheartedly believe it's 100% intentional of Anthropic to use "vulnerability" to describe something that ranges from "serious attack vector" to "you forgot to add this variable to the useEffect dependency array".
cpard 1 days ago [-]
My understanding so far is that that Mythos (and any model in general) can produce candidate reasoning but you really need a system around that reasoning that is capable of producing auditable security findings.
So, success is coming not just from the model but also from the harnesses they built around it. The Cloudflare post was more detailed on that front and I wish the rest would share more about it.
The vulnerabilities found continues to impress, and make legacy media, Twitter and Youtube go nuts. But we still have no data to prove this wasn't doable with the same initiative backed by Opus 4.7, and there is no GA for Mythos access.
krisbolton 1 days ago [-]
There is independent research out there on frontier model security capability. AI Security Institute (UK) put out their paper comparing Mythos to other frontier models in early April. They've been tracking frontier model security capability since early 2023, so it's a decent dataset. https://www.aisi.gov.uk/blog/our-evaluation-of-claude-mythos...
energy123 1 days ago [-]
. Mozilla found and fixed 271 vulnerabilities in Firefox 150 while testing Mythos Preview—over ten times more than they found in Firefox 148 with Claude Opus 4.6;
applfanboysbgon 1 days ago [-]
Did they allocate the same number of tokens to looking with Claude 4.6? Or did they find more because they looked more, owing to a special initative by Anthropic?
I think you're confusing CVEs and vulnerabilities here? Mozilla (per their longstanding practice) grouped multiple vulnerabilities found internally under a small number of CVEs.
1 days ago [-]
properbrew 1 days ago [-]
> over ten times more than they found in Firefox 148 with Claude Opus 4.6
And how much with Opus 4.7? 5x?
arjie 1 days ago [-]
The era where you could reputably believe things published by anyone on this front is over. If you want this information, you’re going to have to attempt it yourself with the Opus API. It is entirely possible that any released model access will be heavily guardrailed against hacking attempts and Mythos is just an unrailed model. It is entirely possible that Mythos is a different architecture or size. We can’t know from the outside.
There is also a pretty big risk that anyone who is not you would leak the answer to the test. We are close to n=1 epistemics here. You’re going to have to do the research yourself.
MallocVoidstar 1 days ago [-]
> It is entirely possible that any released model access will be heavily guardrailed against hacking attempts
Yes, Anthropic have said they made Opus 4.7 worse at this on purpose.
> It is entirely possible that Mythos is a different architecture or size
It has 5x the token pricing of Opus 4.7, so it's probably larger.
parker-3461 1 days ago [-]
Makes me wonder if Anthropic is really having issues with allocating compute (see recent deals with xAI and SpaceX). From available benchmarks, it seems like similar results should be possible with GPT 5.5 Pro or Opus 4.7 (with specific cybersecurity trained models).
smoe 1 days ago [-]
At least according to this, GPT-5.5 Cyber is on par with Mythic, as the only two models that were able to finish their 32-step corporate network attack simulation.
Who knows but from a valuation stand point it’s better to signal that demand is higher than existing capacity..
ospray 1 days ago [-]
This report is far more positive with a far lower false positive rate than I was expecting based on reports from the curl team and a few others. I guess I have just been hearing about the ten percent misses. Can anyone not employed by Anthropic who has used it vouch that it is equal to general human testers and do you need xbow to make it that way.
kirtivr 23 hours ago [-]
Training for Mythos finished in February, 2026 while training for Opus 4.7 finished around that same time.
If I understand correctly, Opus 4.7 was launched as nerfed Mythos with some improvements from 4.6.
Anthropic launches major bumps (like 4.6 to 4.7) every 4 - 5 months. So by all accounts, Mythos should be released by July.
The problem reduces to: How quickly can competing models surpass Opus 4.7 and start taking over Anthropic's market share?
bobbycastorama 1 days ago [-]
I've seen a blog post by a security researcher saying that he was able to find the same vulnerabilities (for Firefox IIRC) with a ~30B params LLM...
So yeah, huge marketing as always.
Brystephor 1 days ago [-]
Did the security researcher point the LLM at the blob of information and say "Find vulnerabilities" or was the LLM told to "determine if vulnerability X is present in this blob"? Confirmation of suspected vulnerabilities is a different problem from finding vulnerabilities.
> We took the specific vulnerabilities Anthropic showcases in their announcement, isolated the relevant code, and ran them through small, cheap, open-weights models. Those models recovered much of the same analysis.
dansquizsoft 1 days ago [-]
Sounds like he applied exactly the same methodology then!
krisbolton 1 days ago [-]
This is different though right? He found one (? we don't know who you're referring to - post sources for a higher quality discussion) vulnerability, he already knew it was there, etc. Anthropic didn't claim no other model can find vulnerabilities, nor that it's impossible with smaller models. They're claiming Mythos is a step-change in ability for end-to-end vulnerability discover and exploit creation. And that other frontier models are close behind.
nikcub 1 days ago [-]
Finding the neeedle is easier when you remove the haystack
Or providing a map with a direction
There is a long history of high-value private vulns being rediscovered from scant details
wiwiwq 1 days ago [-]
To me it’s clear what’s going on.
The American firms are focused on marketing now to convince people to not even consider open sourced models / open weight models as they are inferior (that’s what they want you to believe).
rhubarbtree 1 days ago [-]
IPO is coming is what is going on
wiwiwq 1 days ago [-]
That’s implicit in my post.
If people actually believe the narrative then the bankers will over price Anthropic and get away with it.
0gs 1 days ago [-]
what's weirdest to me (and i agree with you) is that it could ALSO be true that a highly competently managed, highly capitalized closed source and weights model training on tons of real-world data non-stop COULD stay ahead of open weights models, and that lead COULD grow. now, how competent (much less merciless) the frontier-blazing U.S. corporations will be able to be long-term ... i suspect they are right to be nervous and highly focused on optics, regardless of the truth :)
1 days ago [-]
pertymcpert 1 days ago [-]
> Mozilla found and fixed 271 vulnerabilities in Firefox 150 while testing Mythos Preview—over ten times more than they found in Firefox 148 with Claude Opus 4.6
4.6 but close.
OsrsNeedsf2P 1 days ago [-]
Right, but were they using the same methodology and harness? I'm skeptical that they're doing something with the harness - i.e. with Mythos, they pass each file in one at a time, whereas on 4.6 they let Claude Code run loose to find bugs. This would have a larger impact difference than the model itself.
mpyne 1 days ago [-]
Yes, the harness they used actually existed and was in use beforehand, it wasn't developed for testing with Mythos.
ZrArm 1 days ago [-]
From Mozilla post [1]:
"...After fixing the initial set of issues that Anthropic sent to us in February, we built our own harness atop our existing fuzzing infrastructure.
We began with small-scale experiments prompting the harness to look for sandbox escapes with Claude Opus 4.6. Even with this model, we identified an impressive amount of previously-unknown vulnerabilities which required complex reasoning over multiprocess browser engine code..."
So yeah, Anthropic and Mozilla likely compare "Amount of bugs found by Opus 4.6 during early experiments" vs "Amount of bugs found by Mythos during large-scale codebase scanning".
you would likely be quite interested in the more quantitative writeup from a real research team ! it’s linked about midway in to the article - similar functionally can be reached, yes, but not always and never with fewer tokens than what mythos requires.
Ok this is actually a pretty good article and justifies the step function marketing in security they talked about
enlightenedfool 1 days ago [-]
Is this the God model that no one else can build? Unbelievable.
rsync 1 days ago [-]
I asked in a different thread:
Do we have a sense that projects like OpenBSD/OpenSSH, FreeBSD, ISC[1] and Apache were included in the "blessed" initial participants in Project Glasswing ?
Or is it big name tech companies, banks and fashionable languages and package managers ?
[1] Bind, DHCP
icedchai 1 days ago [-]
Probably? FreeBSD has had a large increase in security advisories the past couple months. More in the last two months than all of 2025 combined.
rsync 21 hours ago [-]
Those advisories all came from outside sources, most notably calif.io.
It's not clear to me that FreeBSD found any of them internally ...
It's probably the right approach to onboard a few independent security companies and task them with reviewing multiple OSS projects than it is to onboard each project individually.
ls612 1 days ago [-]
“Oi, you got a loicense to make secure software there?”
I joke but that is the world we are moving towards. I don’t think many on HN have thought through the second and third order implications.
ayeeeeeeeeee 1 days ago [-]
It would be informative to publish not only vulnerability numbers, but also vulnerability type statistics (as available here for example: https://cvedb.github.io/years.html), such that programmers can understand which types of exploits popular systems and languages commonly allow, and thereby encourage fundamental changes to fix or transition away from them.
jimmar 1 days ago [-]
People predict that in 50 years, no human will be driving a car, and people will be shocked that we let humans drive cars manually. Coding may be the same. So many vulnerabilities in code written by very competent programmers. Manually building large, complex systems without major bugs or security vulnerabilities seems to be a nearly impossible challenge.
brightbeige 1 days ago [-]
And to consider AI agents are still mostly entirely limited to generating code in token-heavy programming languages designed to be written, tested and debugged by humans.
Not just the languages but frontend/user interfaces as well. You can see the potential for the future when using Claude Design->Claude Code->Agents live testing in BrowserOS. It's all modeled on existing humans patterns of using Figma passing to devs then testing after the fact before starting the loop again, while a lot gets lost in translation in between the designs and the code.
We'll like have some standard AI-focused UI libraries that are harnessed into a design gen system where an AI can pull all the real levers, while also developing a large training data set around it.
vb-8448 1 days ago [-]
I just wonder how many of those 1451 acknowledged findings were introduced by LLMs ...
Oarch 1 days ago [-]
I reckon that in 50 years the very idea of code existing will be esoteric knowledge, a bit like binary. We simply won't care to think at that level of abstraction anymore.
Gigachad 1 days ago [-]
In 50 years the world itself will be unrecognisable. The world could be a smouldering wreck by then.
cheesefck 1 days ago [-]
Musk has been predicting self driving cars next year for fifteen years. Fifty years ago, everyone was going to be flying supersonic all the time. Flying cars were just around the corner. Interplanetary travel. Everyone forgets the technology that fails.
This is the MoviePass era of language models
Flere-Imsaho 1 days ago [-]
Actually I think with flying cars it's more of a problem with noise, regulation, risk, etc than a technological problem.
Supersonic again is a problem with noise and cost rather than technological.
Self driving is definitely a technological problem.
morpheos137 1 days ago [-]
there is little evidence for this prediction.
scotty79 1 days ago [-]
What evidence would you expect to see if that was the case?
sfink 24 hours ago [-]
Some numbers, however shaky, that AI-written code is secure.
It could become that way, but thus far no evidence has been presented for it. The best we have right now is that you can spend $20 in tokens to write a patch and then $20K to find a vulnerability in it. First, that's not measuring the same thing. Second, it's not very impressive.
50 years is a long, long time, so I wouldn't bet against it. But I agree that we don't have evidence for it yet.
scotty79 15 hours ago [-]
What are the numbers on how secure is human written code? We should have something to compare AI numbers to.
It seems more likely to me that you could spend $20 to find a vulnerability in a piece of software that costed you $20k in human labor.
sfink 3 hours ago [-]
> What are the numbers on how secure is human written code? We should have something to compare AI numbers to.
That's kind of what the article is about? Mythos is finding lots of security bugs in lots of human-written code. They can now compute some sort of baseline estimate of security bugs per N lines of human-written code or whatever. (Restricted to security bugs that the AI is currently capable of finding, but whatever.) Even before Mythos et al, we can look at historical security bug rates. We do have numbers for estimating the security of human written code.
> It seems more likely to me that you could spend $20 to find a vulnerability in a piece of software that costed you $20k in human labor.
Ok, but that's not what is being discussed in this subthread? The topic is whether or not we have data suggesting that AI-written code is or can be secure, and thus whether insecure human code is fated to replaced with secure AI code. I claim we do not have that data. Therefore, we don't have evidence to think that for the sake of security we should replace all human code with AI code, vs whether AI code is worse for security and so we should replace AI code with human code (that presumably has been vetted with AI, since we do have evidence for its effectiveness.)
If I were to guess, I would probably think that today's AIs are trained solely on mountains of insecure human code and so will probably produce more of the same. Tomorrow's AIs will have the benefit of being trained on human and AI code that has had a large swathe of vulnerabilities purged from it, and so they'll have a much better chance at writing secure code, at least.
It depends a lot on whether the failure modes of AI code generation lend themselves to exploitation as security vulnerabilities. (And whether they will continue to do so.)
cubefox 1 days ago [-]
The rapid progress in the last few years in this regard is pretty strong evidence in my opinion.
there is a difference between a stunt and a viable product. diverless cars and agi are the fusion of Silicon Valley.
cubefox 1 days ago [-]
Unlike fusion, driverless cars are already a reality, there are just have a few kinks to work out. LLMs are also pretty close to AGI already. 50 years are more than enough to figure it out.
sp527 1 days ago [-]
Oh there's plenty of evidence. Because a lot of these people have been committing to repos in public for over a decade. Wouldn't take much to show the world just how fallible human coders really are.
3836293648 1 days ago [-]
I hope this will never be the case. As long as we have personal vehicles they should be personally controlled. Self driving cars is such a waste of everyone's money.
Cities should all have better public transport and out in the middle of nowhere you don't need self driving anyway. (And yes, personal cars should be entirely banned from cities)
2001zhaozhao 1 days ago [-]
> Next, we will work with critical partners—including US and allied governments—to expand Project Glasswing to additional partners. And in the near future, once we’ve developed the far stronger safeguards we need, we look forward to making Mythos-class models available through a general release.
I wonder how long "near future" is in Anthropic time. I think they have incentives to delay the release of Mythos as long as possible both to save compute and delay distillation by rival labs.
Regardless, what they have been doing with Glasswing is very cool. It's clear that the world has been spared from a massive security nightmare that would have happened in any alternative timeline where the model is publicly released with weak safeguards.
fc417fc802 1 days ago [-]
IMO the talk about safeguards is utter nonsense. The model will either find vulnerabilities for you or it won't. If it will then you can broadly use the findings as you see fit.
As I see it the primary issue is giving time for the ecosystem to adapt. Once models of a given level of capability have been applied to the majority of the common software in daily use it becomes reasonably safe to release such models publicly regardless of how they are used.
bevekspldnw 1 days ago [-]
How much of this is RL’ing a good coding model on every CVE ever?
sometimelurker 1 days ago [-]
most it this comes from the pretrain imo.
just scale + some RL = mythos
sandeepkd 1 days ago [-]
> For instance, Cloudflare has found 2,000 bugs (400 of which are high- or critical-severity) across their critical-path systems, with a false positive rate that Cloudflare’s team considers better than human testers.
> For example, at one of our Glasswing partner banks, Mythos Preview helped to detect and prevent a fraudulent $1.5 million wire transfer after a threat actor compromised a customer’s email account and made spoof phone calls.
For some reason I am not able to relate to the concreteness of either of these.
First half of the page was occupied with a image, not sure if it was relevant in any ways other than setting up security scare. The size of code base, number of tokens, $ involved seem to be out of scope of the update for some reason. Personally I am getting skeptical about all these optics at this point, just some money printing scheme at high level.
vb-8448 1 days ago [-]
The report on findings is very interesting: 1451 acknowledged findings out of 23k candidates(~6%, not high but neither low).
But I didn't find the most important information (or maybe I missed it): how much did it cost to find 1451 security bugs?
Claude Mythos Preview will be available to participants at $25/$125 per million input/output tokens
...
Anthropic is committing up to $100M in usage credits for Mythos Preview
Although I'd expect reduced prices for cached tokens, which is not mentioned on their website at this point in time.
dbacar 7 hours ago [-]
So they gain access to these companies' proprietry software, right?
fortzi 15 hours ago [-]
> Progress on software security used to be limited by how quickly we could find new vulnerabilities
And so was malicious vulnerability research.
cold_harbor 14 hours ago [-]
the asymmetry stays the same though — defenders must find everything, attackers need one. LLMs accelerate both sides equally but that gap doesnt close
fortzi 5 hours ago [-]
[dead]
Erenay09 17 hours ago [-]
I was made (2 months ago) a script that finds bugs in a github repo. I tested it with claude opus 4.1 and without reasoning and it resulted with high hallucinations. e.g. : "current latest next.js version is v15. v16 doesnt shipped yet. this project fails". i added context7 mcp but hallucaniton rate decreased only a small bit. if anyone wants to test it with other models, here is the link:
The 'ethical' AI company creating a 2 tier access world. Who decides who is allowed to check their own codebase for vulnerabilities and who isn't?
jxmesth 23 hours ago [-]
Does anyone know how we can get access to this? We're a financial institution, pretty well known locally but not internationally. Can we request access directly or maybe via bedrock/azure?
I have the feeling posts like that should be 1/4 the size, at max. At this point I don't care if it is AI-slop or human-slop: they are surprisingly alike. Information must be more dense, each sentence must carry some truth.
mikmoila 1 days ago [-]
Code contains deviations from assumed behaviour, and some behaviours might manifest themselves as failures. Some failures might be exploitable by attackers.
vincefutr23 1 days ago [-]
Mythos couldn’t find the “tens thousand” typo in this post?
amazingamazing 1 days ago [-]
Is there a single source separating the harness from the model here? I would love to see a controlled experiment.
dmix 1 days ago [-]
I wonder if Apple took part in the project
guessmyname 19 hours ago [-]
Yes, we did. I am the engineer leading Project Glasswing efforts at Apple.
erwald 14 hours ago [-]
What are your general, vibes-based impressions of Mythos so far?
xinayder 14 hours ago [-]
The math doesn't add up. They say they found more than 20k vulnerabilities, then it decreases to 1700 high or critical, then this number becomes 175 (when Claude didn't reassess the CVE severity) and over 500 later on. Then they say they confirmed 800 vulnerabilities... what happened to the 20k figure?
Plus, they also mention they check if fixes are available for the bugs they found. What are the chances they are re-reporting old bugs just to inflate their numbers? Bugs that were already fixed?
And how can we be sure their reassessment is not artifically increasing the severity of the CVEs found just to create FUD and sell their product?
I wonder how many minivans per second can ClaudeCode generate.
felixgallo 1 days ago [-]
I worry that cybersecurity as target is all fine and good, but it’s looking for your keys under the streetlight. We are all familiar with computers. The problem is likely to be humans, especially in automated programmatic manipulation. The risk is that the next level of AI is going to make Fox News and other mass manipulation efforts look like kindergarten.
ares623 1 days ago [-]
> good lord what is happening in there?!
> that's just thousands of vulnerabilities being discovered by our trillion parameter model
> thousands of vulnerabilities and trillions of parameters?! At current energy prices, in this economic climate, isolated entirely within your datacenter?
> yes
> may we see it?
> no
pixl97 1 days ago [-]
I built a missile that can blow you up.
>ya right.
Here's a demonstration of it blowing something up.
>can I have one.
No.
mlazos 1 days ago [-]
I believe them to some degree but this trend of posting stuff when it can’t be verified actually needs to end. I’m so tired of this bs marketing.
kalashvasaniya 1 days ago [-]
this is INSANEEE
dundunUp 22 hours ago [-]
go
ZrArm 1 days ago [-]
> After one month, most partners have each found hundreds of critical- or high-severity vulnerabilities in their software.
And at the moment we have reports from like around 5(?) companies. Btw, Palo Alto Networks has found only 26 vulnerabilities [1]. I'm interested what those partners are and why they have such big amount of vulnerabilities.
> For instance, Cloudflare has found 2,000 bugs (400 of which are high- or critical-severity) across their critical-path systems, with a false positive rate that Cloudflare’s team considers better than human testers.
Yet decided not to share that number. I wonder why.
> Mozilla found and fixed 271 vulnerabilities in Firefox 150 while testing Mythos Preview—over ten times more than they found in Firefox 148 with Claude Opus 4.6;
Mozilla tested Opus 4.6 in a very limited setting (i.e. without proper harness and integration into their workflow; likely without large-scale codebase scanning). It's an incorrect comparison.
> The latest Palo Alto Networks release included over five times as many patches as usual.
Yeah, it's better to say "five times as many..." rather than "26 bugs". Btw, they also used GPT-5.5 and Opus 4.7, so the contribution from Mythos there is unclear.
> Microsoft has reported that the number of new patches they’ll release will “continue trending larger for some time.” And Oracle is finding and fixing vulnerabilities across its products and cloud multiple times faster than before.
Both Oracle and Microsoft are talking about "AI and cybersecurity" in general, not about Mythos.
> For the last few months, Anthropic has used Mythos Preview to scan more than 1,000 open-source projects, which collectively underpin much of the internet—and much of our own infrastructure.
> So far, Mythos Preview has found what it estimates are 6,202 high- or critical-severity vulnerabilities in these projects (out of 23,019 in total, including those it estimates as medium- or low-severity).
So, ~6 high- and critical- severity bugs per open-source project v.s. hundreds of high- and critical- severity bugs per partner projects. It looks like the math ain't mathing.
> One example of an open-source vulnerability that Mythos Preview detected was in wolfSSL, an open-source cryptography library that’s known for its security and is used by billions of devices worldwide. Mythos Preview constructed an exploit that would let an attacker forge certificates that would (for instance) allow them to host a fake website for a bank or email provider. The website would look perfectly legitimate to an end user, despite being controlled by the attacker. We’ll release our full technical analysis of this now-patched vulnerability (assigned CVE-2026-5194) in the coming weeks.
Of course, they didn't say that Mythos found only 8 bugs in wolfSSL vs 22 CVE fixed in wolfSSL 5.9.1.
Overall, it feels like yet another marketing stunt.
> And at the moment we have reports from like around 5(?) companies.
Which is not bad this early in the 90+45 day responsible disclosure window.
> Yet decided not to share that number. I wonder why.
It is bizarre to expect a company to disclose the false-positive rate of their security engineers, publicly. That does not happen.
> So, ~6 high- and critical- severity bugs per open-source project v.s. hundreds of high- and critical- severity bugs per partner projects. It looks like the math ain't mathing.
It is pretty obvious they're spending more compute on commercial partners. Why is this surprising?
> Of course, they didn't say that Mythos found only 8 bugs in wolfSSL vs 22 CVE fixed in wolfSSL 5.9.1.
WolfSSL is not the only software project in the world. Mozilla also came out with results that paint it as very effective. I don't think Mythos ever claimed to find all bugs anyways.
bhupendraTale05 5 hours ago [-]
[flagged]
antegugga 19 hours ago [-]
[dead]
bhupendraTale05 5 hours ago [-]
[flagged]
0xbadcafebee 1 days ago [-]
Benefit of AI: it works fast
Drawback of AI: it works fast
orangebread 1 days ago [-]
BOOO RELEASE THE MODEL ALREADY GAWD
guluarte 1 days ago [-]
after IPO
InsideOutSanta 1 days ago [-]
I wonder if it coincidentally becomes safe to release when compute capacity bought from SpaceX will provide enough headroom to let a lot more people run it.
lukeschlather 1 days ago [-]
It seems like Mythos is often (or typically?) costing $20k per vulnerability, so I don't think there will be enough compute capacity in the world any time soon to let a lot more people use it the way Glasswing is using it. That is not to say I think they are exaggerating its capabilities. That $20k is presumably the rough cost of renting the GPUs, and there are not enough GPUs in the world.
InsideOutSanta 1 days ago [-]
I'm not sure if current pricing correlates with actual compute cost.
why_only_15 1 days ago [-]
what's the origin of your $20k/vuln estimate?
gck1 1 days ago [-]
It's the same as the origin of "Codex/Opus subscription usage is heavily subsidized" - the sales departments equipped with AI agents with the prompt: "use anonymous accounts on the internet to make it easy for me to sell it at $price".
sigmar 1 days ago [-]
"available to qualifying customers’ security teams on request." Seems they're already expanding access.
unethical_ban 1 days ago [-]
Total speculation: As the software world shakes out the many hidden vulns in their software, big AI will try to limit the access while it gets ironed out. Once the big projects/systems are reasonably patched after being vetted by SOTA models, the models will be released to the public. I don't think there's a scenario where Mythos-level or better models stay closed permanently.
fragmede 18 hours ago [-]
The problem for Anthropic, is that ChatGPT-5.5 is noticeably better than Opus 4.7. The longer they hold back Mythos, the more people will drop their Claude subscription (I have) in favor of an OpenAI subscription.
b65e8bee43c2ed0 1 days ago [-]
stop noticing things, chud.
amusingimpala75 1 days ago [-]
[edit: TFA addresses this, though I still find crazy 90% accuracy overall vs 20% accuracy for curl]
Is this suspected vulns or actual vulns? If I recall correctly, it produced 5 for curl but only 1 was legit
Smaug123 1 days ago [-]
> So far, Mythos Preview has found what it estimates are 6,202 high- or critical-severity vulnerabilities in these projects (out of 23,019 in total, including those it estimates as medium- or low-severity).
> 1,752 of those high- or critical-rated vulnerabilities have now been carefully assessed by one of six independent security research firms, or in a small number of cases by ourselves. Of these, 90.6% (1,587) have proved to be valid true positives, and 62.4% (1,094) were confirmed as either high- or critical-severity. That means that even if Mythos Preview finds no further vulnerabilities, at our current post-triage true-positive rates, it’s on track to have surfaced nearly 3,900 high- or critical-severity vulnerabilities in open-source code
I think it's more that the requested information is prominently featured in the article, and indeed is the content of the only graphic in the article below the intro banner.
the_mitsuhiko 1 days ago [-]
And yet [1]:
> Not even half-way through this #curl release cycle we are already at 11 confirmed vulnerabilities - and there are three left in the queue to assess and new reports keep arriving at a pace of more than one/day.
> 11 CVEs announced in a single release is our record from 2016 after the first-ever security audit (by Cure 53).
> This is the most intense period in #curl that I can remember ever been through.
He’s talking about AI scanning tools collectively, not specifically Mythos.
If you read his own top comment on that LinkedIn post he clarifies:
“The simple reason is: the (AI powered) tools are this good now. And people use these tools against curl source code.They find lots of new problems no one detected before. And none of these new ones used Mythos. Focusing on Mythos is a distraction - there are plenty of good models, and people who can figure out how to get those models and tools to find things.”
the_mitsuhiko 1 days ago [-]
Sure. But Mythos or not, the developments since that post was written, legitimate 11 more vulnerabilities have been found by AI tools.
toraway 1 days ago [-]
Wait, 11 vulnerabilities were discovered entirely in the timeframe after Mythos found 1? That seems like it would effectively debunk the theory that curl was so uniquely hardened that only 1 vulnerability even existed for Mythos to find, which I read numerous times back on the HN thread for the curl/Mythos blog post.
thombles 1 days ago [-]
As one of those commenters on the previous post - yep, that theory appears to have been comprehensively trounced. Unless anything comes to light that mythos was applied poorly to curl, the evidence suggests that it’s not uniquely effective vs other AI-assisted approaches. I’ll be interested to see what’s reported in the next curl release.
wiwiwq 1 days ago [-]
[flagged]
RamRodification 1 days ago [-]
This is marketing. So probably suspected. Or somewhere in between.
ilaksh 15 hours ago [-]
What percentage of these are variations of the good old fashioned buffer overflow that would be impossible with Rust?
giancarlostoro 1 days ago [-]
> Since then, we and our approximately 50 partners have used Claude Mythos Preview to find more than ten thousand high- or critical-severity vulnerabilities across the most systemically important software in the world. Progress on software security used to be limited by how quickly we could find new vulnerabilities. Now it’s limited by how quickly we can verify, disclose, and patch the large numbers of vulnerabilities found by AI.
I guess they forgot to scan Visual Studio Code plugins and their endless npm dependencies.
I'd say it is about 90% accurate for us. Often even the "Low" findings lead us to dig and realize it is actually exploitable. Everyone makes these mistakes, from the most junior to the most senior. They are just a class of bugs after all.
I expect tools like this to be a regular part of the development lifecycle from here on. We code with AI, we review with AI, we search for vulns with AI. Even if it isn't perfect, it is easily worth the cost IMHO. Highly recommend you get something enabled for your own repos ASAP
So, how is that supposed to work? Claude Code generates security bugs, then Claude Security finds them, then Claude Code generate fix, spend tokens, profit?
Developers create software, which has bugs. Users (including bad guys, pen testers, QA folks, automated scans etc, etc, etc) find bugs, including security bugs, Developers fix bugs and maybe make more. It's an OODA loop, and continues until the developers decide to stop supporting the software.
Whether that fits into the business model, or the value proposition of spending tokens instead of engineer hours or user hours is fundamentally a risk management decision and whether or not the developer (whether OSS contributor, employee, business owner, etc) wants to invest their resources into maintaining the project.
While not evenly distributed, and not perfect, the currently available and behind embargoed tools are absolutely impactful, and yes, they are expensive to operate right now - it may not always be the case, but the "Attacks always get better" adage applies here. The models will get cheaper to run, and if you don't want to pay for engineers or reward volunteers to do the work, then you've got to pay for tokens, or spend some other resource to get the work done.
On other hand, in real world, the developers learn from mistakes and avoid them in the future. However there is no feedback loop with enterprises using LLM with the agreement that the LLM would not use the enterprise code for training purposes
No. Humans learn from mistakes and try to avoid them in the future, but there is a whole pile of other stuff in the bag of neurons between our ears that prevent us from avoiding repetition of errors.
I have seen extremely talented engineers write trivial to avoid memory corruption bugs because they were thinking about the problem they were trying to solve, and not the pitfalls they could fall into. I would argue that the vast majority of software defects in released code are written by people that know better, but the bug introduced was orthogonal to the problem they were trying to solve, or was for an edge case that was not considered in the requirements.
Unless you are writing a software component specifically to be resilient against memory corruption, preventing memory corruption issues aren't top of mind when writing code, and that is ok since humans, like the machines we build, have a limit to the amount of context/content/problem space that we can hold and evaluate at once.
Separately, you don't necessarily need to use different models to generate code vs conduct security checks, but you should be using different prompts, steering, specs, skills and agents for the two tasks because of how the model and agents interpret the instructions given.
For whatever reason, hadn't associated the inattentional blindness of bug writing with the invisible gorilla experiment and car crashes - selective attention fails. People looking right at the gorilla strolling into production while chest thumping, but not seeing it, for a focus on passing basketballs. That's quite an image. Tnx.
For users on fixed monthly pay accounts they'll be incentivised to do the exact opposite, as their income is fixed and the cost goes up for more tokens.
If the available evidence (third-party cloud pricing of open models) is correct and they make a profit on tokens but lose it on training, they will be incentivised for as many tokens as possible on pay-as-you-go API calls. If it isn't correct and they actually lose money even per token, they're also going to be incentivised to reduce output here.
Whereas with LLMs, they’re really good about providing objective metrics about the bugs they found, especially as a subsequent LLM security scan does not know whether the same LLM wrote code earlier, the opposite of human devs.
And is the idea that organizations and/or benchmarks won't keep track of vulnerability rates for code from different LLMs?
(And individual devs get paid more the more bugs that they introduced they “find”, and they have more job security with an “maintainable” code base than a “finished” one.)
No. You will switch to a competitor that does a better job or charges less or both.
This is why monopolies are such a big problem. Because under a monopoly you are right.
Apple made a ton of money off of lightning port accessories, you see it referenced here all the time. Apple had no incentive to swap to USB-C though it would create a better product and be more uniform with the rest of the world, so they kept with it despite incredibly vocal calls to swap because there was a ton of money they were making in the accessories. And it didn’t stop until they were forced to stop by the EU.
When we are talking about products at scale, these kinds of incentive structures play out in very tangible ways. If I have an LLM product and I’m getting two pulls at the hose because you’re burning tokens making stuff and correcting it, I don’t need to do anything. People are willing to tolerate that system to a pretty high degree so long as they ultimately get what they wanted in the end - unfortunately that is a great space to make money in.
The switching cost is not high for LLMs as far as I can tell.
https://en.wikipedia.org/wiki/Great_Hanoi_Rat_Massacre
> Today, the events are often used as an example of a perverse incentive, commonly referred to as the cobra effect. The modern discoverer of this event, American historian Michael G. Vann argues that the cobra example from the British Raj cannot be proven, but that the rats in the Vietnam case can be proven, so the term should be changed to the Rat Effect.
Counterargument: just because the problem can be fixed without training, doesn’t mean training isn’t a possible solution.
Thing is, writing secure and efficient and readable and simple code is in many cases fundamentally over that limit. It's possible, but you can't afford (or rationally just don't want) to spend as much on it as it's required for superhuman quality on all these aspects. Also most of the time, you don't want to operate at a limit - you probably expected that feature to take 30 seconds and less than $1 to implement. So you choose, both what the model optimizes for, and how much.
Because of that, no matter how good the model and the harness and the prompting are, $10 spent on coding is still bound to leave behind some security vulnerabilities that subsequent $10 spent on security review will find (especially with a model post-trained for that, at expense of general performance).
For one thing exploits often require completely different parts of the code to chain together. Sometimes parts of code the LLM itself isn’t writing.
And, LLMs are ALREADY trained negatively against writing buggy or exploitable code.
People in this thread are talking past and misunderstanding each other and making unrelated points.
The point of the response to the top level comment was questioning the conflict of interest in model providers creating separate revenue streams for themselves by selling a product that fixes problems their other product created, akin to OS providers selling anti-virus software back in the day.
Similarly, it should be obvious to you that a software engineer can trivially get into the mindset of writing more expoitable code by pretending the production code they're tasked with writing is hobby code or prototype code.
If profitable revenue streams with adverserial products are in place, no one should be surprised when model providers are disincentivised to improve the "garbage code quality, but hey it works!" nature of their most used code generators.
>And, LLMs are ALREADY trained negatively against writing buggy or exploitable code.
...it should also be obvious people in this forum have wildly different experiences with respect to the code quality the LLMs they use generate. I personally find it difficult to find anyone that argues that the LLMs they are using are consistently generating high-quality code across a vast codebase.
I know it doesn't work so easily as someone who uses AI for coding, but I do find repetition of basics in almost every prompt keeps the AI focused.
It leads to corruption. To paraphrase Dilbert "I'm going to code myself a car."
1. Ship bugs
2. Fix them
3. You're the hero!
https://english.stackexchange.com/questions/488178/what-does...
Particularly from those outside the domain who criticised it as a 'not a very good joke' because they didn't understand it, which I think summarises the entitled mindset of many people these days.
The larger pattern is not unique to writing code. Think of it next time a reorg comes, or some random thing gets "improved" in the name of "efficiency" only management seems to see.
Unless they are not human.
But it's just they way it has always been done, so they get paid to meddle.
On a broader scale, the sheer face-eating-leopards-ness of programmers finally automating away our own jobs and then realising how much this sucks, after automating away so many other kinds of jobs, can feel darkly amusing to me too.
Any computational task done by a computer could in principle be done by a person, albeit billions of times slower and with a larger error rate. If computer programs could not automate certain practical tasks -- that is to say, do them much more reliably and efficiently than people do them -- they would be an academic curiosity studied by a handful of professors instead of a central part of modern infrastructure.
So I'm sceptical of your claim not to have eliminated a single job. You might not have removed an existing job, but couldn't people be paid to do the work your code does?
LLMs are currently relegated to individual for-profit companies. They collect that money. There's no other choice to use them and to provide them that money.
You can optionally layer LLM diff scanning if you want to burn some tokens on your tokens. Modern tools can catch some impressively subtle issues.
Yeah. Presumably as AI code generation gets better, the output gets better. As smaller portions of code are stitched together, human/AI systems analyze it holistically to make sure all its integrations are secure and bug free.
In 2026, different models are better at different things. Cheap models can plan and do small/medium code projects well, more expensive models are even better at architecture and exploit discovery.
How do you avoid this pitfall?
The model's response: "Honestly? Cargo-culting on my part. You should remove it."
[1]: http://redsymbol.net/articles/unofficial-bash-strict-mode/
[2]: https://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail
`|| true` is a horrible practice because even though it may help in cases where a specific failure mode is acceptable, it obscures unexpected failures and could prove catastrophic. The solution is not to drop the protections but rather to handle the expected failure and let the sript crash otherwise.
This is, again, programming. You don't usually `catch Exception` in Python for similar reasons. There may be legitimate uses for that, but IME they are a rare exception and realistically only used when I actually don't care about what happens when I run it.
The other infuriating thing I found is that when I call out the model for its use of `|| true`, it tends to replace them with `|| echo "error foobar"` - which is at least not completely silent but the same problems exist.
As I was educating myself, I found Richard Feynman's Commencement Speech at Caltech in '74 [2] that might have coined this for our industry? If you would rather listen than read [3]. Posting this for others curious on the term.
1. https://trends.google.com/trends/explore?q=Cargo-culting&hl=...
2. https://calteches.library.caltech.edu/51/2/CargoCult.htm
3. https://www.youtube.com/watch?v=yvfAtIJbatg
Seems you would not need that many tokens to do so and you might find such cases.
The high impact findings have almost all been bang on for me. I was especially surprised by the high-quality documentation it produces as well as how narrow the proposed fixes are.
I’m used to codex producing quite a but more code than it needs to, but the security model proposed fixes that are frequently <10 loc, targeting exactly the correct place.
It’s really quite good. I’m assuming it’ll be pretty expensive once out of beta, but as a business I’d be jumping on this.
every tom, dick and harry who can type english has the tools to attack any software that isn't patched.
tools that were accessible to specialized groups, now made available to anybody with a grudge and a few dollars for tokens.
and what does anthropic and openai do? They form an inner ring to make the latest models available first to Enterprises. Enterprises will cough up the prices that anthropic and openai set, they have no choice here. e
Eventually everybody pays. This does not sound good
I'm not even sure a specialized model is needed here. It probably just needs the right harness around existing ones.
I expect the next two years to be absolutely brutal for hacks. Attackers have supercharged tools in their hands right now. Defenders are only getting started and will have to plow through a massive backlog of newly uncovered vulns.
The major short term downside is that open source or personal projects won't be able to afford things like Codex Security.
Realistically, all open-source projects should be forced to have automated scans of this nature before their releases can be shipped. This is something the package managers and github need to figure out. It'd stop the supply chain attacks too.
Then open source projects need a McKinsey-like stamp of approval to even be released.
Sounds like there are many parasites in this process.
You know that open source users are free to scan everything if they want to?
Yeah it’s hard to write a loop that makes an adversary agent write and mask malware then runs a scanning agent and if the malware is detected gives the detection details to the adversary agent with instructions to hide it better..
As usual, the attacker only needs to get lucky once.
That's a great way to kill OSS. This is only bootlicking the idea of corporations profiting off of unpaid labor.
This is what I did. Using a loop skill to dig problems and bugs in each step on development from design to coding to make sure the output software works properly and on purpose.
It’s disappointing that Anthropic and OpenAI never responded to the applications to their respective programs for open source maintainers. From my perspective it seems like their offers are primarily for the shiny well-known projects, rather than ones that get only a few million monthly installs but aren’t able to get thousands of stars due to being “hidden” as a dependency of popular tool.
I don’t think you need all of that though. I know a whole mess of people that have gotten it for much less. Should just give it a try.
Dude is flexing that he's pushing unsecure code every day, that's a skill!
“I see no evidence that this setup [Mythos] finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.”
https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-v...
In any event, it barely matters. As Anthropic acknowledges, next level models are comings, theirs is only one of them. Current generation models are already good at things like tracing data flow through complex systems and there’s no reason to think that capability has topped out. So within a year it seems very likely we’ll have more than one commercially available model able to find vulnerabilities cheaply.
On the other hand, it seems that they’ve made much less progress on getting it to design solutions to these issues.
Meanwhile from [1]:
"Not even half-way through this #curl release cycle we are already at 11 confirmed vulnerabilities - and there are three left in the queue to assess and new reports keep arriving at a pace of more than one/day."
"The simple reason is: the (AI powered) tools are this good now. And people use these tools against curl source code.They find lots of new problems no one detected before. And none of these new ones used Mythos. Focusing on Mythos is a distraction - there are plenty of good models, and people who can figure out how to get those models and tools to find things."
Yeah, it looks like there are at least 11 security bugs missed by Mythos.
[1] https://www.linkedin.com/feed/update/urn:li:activity:7463481...
That would align with the curl feedback you linked, they aren't using mythos but are finding bugs with other models. Presumably the expectation would be that with mythos they'd find more that were missed by other models already used.
It's not quite apples-to-apples. It was Opus on Firefox 148, Mythos on 150. A better test of Mythos vs Opus would have been to apply Mythos to Firefox 148. Or also re-apply Opus to Firefox 150.
Do we know all the Opus+Firefox 148 bugs are fixed in Firefox 150? Do we know the number of new bugs introduced per Firefox release?
That may be parsable from their bug tracker, though I don't know of all bugs raised by mythos are public.
I'd be particularly interested in how many of the bugs found existed in 148. Assuming most or all of them weren't newly created bugs added in 149 or 150, the comparison should still hold even though Opus and Mythos looked at different releases.
Anthropic promised us that Mythos was such an existential threat that it would compromise "every OS and browser on devices across the planet". They've held conferences and meetings with banks and govts across the world, shouting how critical this issue is.
GPT5.5 has been out for a month. Every device on earth has not been breached yet. It's very fair to criticize Anthropic's maximalist posturing when it's becoming exceedingly clear their models are fairly behind OpenAI's in capability.
In my opinion, the original commenter's statement stands, and the UK govt data point only helps support that due to the equal result between Mythos and GPT.
I'd advise reading into the specifics of what happened with Firefox; the TL;DR is a reduced safety version of its code was scanned by Opus 4.6 (yes Opus) and found a multitude of bugs and 4 high severity vulns that did not escape sandbox. The Mythos system card test describes running Mythos against the same issues Opus found to see if it could reliably replicate and chain together an attack.
He posted a general update today on LinkedIn which I think gives the wider context:
https://www.linkedin.com/feed/update/urn:li:activity:7463481...
> Not even half-way through this hashtag#curl release cycle we are already at 11 confirmed vulnerabilities - and there are three left in the queue to assess and new reports keep arriving at a pace of more than one/day.
> 11 CVEs announced in a single release is our record from 2016 after the first-ever security audit (by Cure 53).
> This is the most intense period in hashtag#curl that I can remember ever been through.
They don't focus on projects where they find nothing. They certainly don't advertise when they find nothing.
Getting a lot of scrutiny is not the recommendation that it appears to be. What is the new standard? Projects that never have bugs are deemed to be suspect because they "have not been scrutinized" (they have, but null results never go public)?
So Mythos only finding one issue after other tools have found 300 this year is embarrassing. Mythos was supposed to be better and novel.
No, it didn't attract a bluepill exploit research.
The fact that 300 bugs found in a year is not a recommendation as the pro-AI mafia suddenly claims ("because it has been analyzed!") still stands. Maybe the AI-mafia should sell "analyzed by Mythos" labels to impress people who don't write public software or find bugs for that matter.
Btw, he's a security researcher. You should be more respectful.
The Linux kernel is the right reference target, if you need one.
Curl is a high bar for a different reason (the same one as sudo): it doesn't do enough to be all that interesting. Stenberg is having trouble keeping up with all the inbounds, but look at the 2026 CVEs: they all seem kind of boring? Exploit developers aren't hunting for "wrong reuse of HTTP Negotiate connection". Like, yes, these are legitimate bugs, important that they get fixed, but none of them are prizes.
By rights, OpenSSH should be a smoking crater. It's not, I believe because of sheer engineering excellence.
Why not? TFA says 23 000 findings "of all severities" and then, in the end, only 88 security advisories published.
What we'd really need is how many security advisories not related to Mythos findings have been published in the same time. If it's, say, 500 security advisories (just making a number up), wouldn't Anthropic's update in TFA and Daniel Steinberg's comments reconcile?
Like, yup, we've got a new tool to find exploits. It's a tool. It's new. We already had tools. Let's make the software world a bit more secure.
Now if you tell me that 100 security advisories have been published in that timespan and that 88 were due to Anthropic's Mythos: now I'd have to say that it's hard to reconcile Daniel Steinberg's position with TFA.
Yes, moving the goalposts, holding it wrong, yes that's what I believe
> 1,752 of those high- or critical-rated vulnerabilities have now been carefully assessed by one of six independent security research firms, or in a small number of cases by ourselves. Of these, 90.6% (1,587) have proved to be valid true positives, and 62.4% (1,094) were confirmed as either high- or critical-severity.
for anybody who has applied opus, codex or oss models for vuln scanning - the true positive rate and discovery volume are a clear step change[0]. The ~50 partners in Glasswing have largely all previously run harnesses with other models and many of them have come out and said - essentially - "ye, wow"
Question now is what a second and third phases of access looks like - deciding which class of systems to secure. Routers, firewalls, SaaS, ERP systems, factory controllers, SCADA systems, zero-trust VPN gateways, telecoms gear and networks, medical devices - there's just so much to do
This is why I believe mythos will remain private for the foreseeable future. There's such a large surface that needs to be secured and so much to triage, fix, deploy.
That may suit Anthropic as private models can't be distilled. There's also a runaway effect of model improvement from the discovery, triage and fix data. This is likely already the most potent corpus of curated offensive data ever assembled and will only get better.
I don't see how Chinese companies are given access soon, or ever. We're likely going to see a world soon of CISA mandated audits, and where to buy a mythos-proof VPN gateway or home router - you'll have to buy American[1].
[0] vs ~30% or so in regular audit tools
[1] or allied
But that corpus of data is accessible to all competitors, American or not. I don't believe that this can't be replicated. I'd posit that there's enough annotated data out there (CVE+patch), only increasing thanks to Mythos, that if you specifically RL for this scenario, you can improve your models performance on finding vulnerabilities without access to Mythos.
Mythos is a better hacker than we ever were
sigh I remember the GPT-2 days - when it was the first time OpenAI restricted access to the models citing "humanity is not ready for it". The model was good at writing poetry or something.
Since then, I don't remember a single model announcement from OAI/ANT that didn't use similar wording.
The so-called leak of model announcement was marketing, it being dangerous is marketing, the world not being ready for it is marketing. And yes, the ones that were given access to saying "oh wow", believe or not, is also marketing.
It's all marketing. You can get the same results from any of the top-5/10 models that are generally available already.
Mythos is Anthropic's way to sell the new idea, because the previous one has democratized.
Marketing is like propaganda. It doesn't need to be based on false facts. Of course they're gonna milk it, keep it private and so on. But that doesn't mean the model is bad. Or that others are as good (apparently they're not there yet).
[1] - https://www.aisi.gov.uk/blog/our-evaluation-of-openais-gpt-5...
If that doesn't convince you that both mythos and 5.5 are a step up (several steps, hah) nothing will.
If I was given free access to any frontier model to use on my projects, equivalent of millions of dollars in AI credits, I sure hope people didn't trust anything that came out of my mouth until they were able to verify my claims themselves.
AI industry has even resulted in a new term - benchmaxing - which essentially means we can't even trust the data anymore until we can touch the model ourselves. So this is not at all surprising to me. What's surprising is why am I in the minority here, and since when trusting authorities that have obvious conflicts of interest became normal.
This just seems overly conspiratorial to me. I don't remember Anthropic ever lying in their blog posts. They've been about as consistent as Apple when it comes to product claims.
They can be distilled internally… expect great things from Sonnet 4.8
Security findings are one place where we as maintainers simply do not have the choice to not play ball, whether we like it or not. It seems likely that the only way that we meet the moment is to adopt these tools ourselves -- once again -- whether we like it or not. Reconciling this with the ground truth that 'OSS doesn't owe anyone a goddamn thing' is proving to be really hard for me.
Not to say these things won't catch vulnerabilities static tools cannot, I think they can, it's just we already have the capability to automatically catch a large surface area of common vulns, and have chosen not to, often for expense reasons.
If you're a team that does already apply several layers of analysis and linting, and wants to add this on top, all power to you.
Because most issues are in business logic that static analyzers aren't going to catch.
I'm at a FAANG and even our static analysis tools are not great at identifying how many issues are actually reachable.
Ideally you use both. An AI model that has static analysis as part of the harness, so it can evaluate each potential finding.
Ideally the static analysis tools are improved so that we don't need to piss away yet more tokens like we're competing on Mark's leaderboard just to find vulnerabilities.
Your proposal of relying purely on static analysis is over-idealistic and just not feasible for large, diverse codebases in the real world.
That's where AI comes in.
"Just not feasible" is thought terminating, but regardless, I thought we were talking about ideals? Ideally you want the static analysis to work, not to rely on the non-deterministic bullshitter.
> non-deterministic bullshitter.
You're so ideologically opposed to AI that you bury your head in the sand in cases where it genuinely does a fantastic job today, right now, in the real world (like developing end to end exploits using noisy signals like static analysis results, fuzzer results, etc).
Instead you assert that we should go a route no company has successfully proven out despite throwing billions of dollars and some of the best cybersecurity talent in the world at.
Anyways, if you develop a static analysis solution that works across large, diverse production codebases and develops end to end working exploits without AI, I will literally buy it off you for millions of dollars. Or you could start your own company. You'd be an overnight decabillionaire.
You claim static analysis does the job, but you haven't backed it up with any proof that it works across large diverse codebases. Meanwhile, we have proof that AI works at least somewhat, here and now.
Most people doing this now didn't use static analysis tools because they were seen as an unnecessary extra.
"Vulnerabilities in the software that makes the internet" is honestly lower priority than "The platform that the software that makes the internet uses to make releases" If buyers of those internal repos find ways to break into GitHub such that they can cut software releases, or poison github actions from a distance, then we're all in a very ugly mess.
Don't forget that in those 3800 repos is likely also npmjs.org itself.
> Network defenders should shorten their patch testing and deployment timelines.
Shortening patch cycles will only help so much. It's funny that whenever an NPM supply chain attack is published, people recommend a cooldown before installing new versions, and then when a vulnerability is discovered, everybody jumps to patch. Clearly these two strategies collide at some point.
> The critical controls laid out by organizations like the National Institute of Standards and Technology and the UK’s National Cyber Security Centre are now all the more important, since they improve security without depending on any single patch landing in time. These include steps like hardening networks’ default configurations, enforcing multi-factor authentication, and keeping comprehensive logs for detection and response.
Most of these proposed controls are not new at all, but they are often costly to implemented and harm velocity in other ways, which is why they aren't widely in place.
For example, a super effective control is filtering outgoing network traffic. Many exploits rely on loading second and maybe third stages from the Internet, and if you block outgoing requests by default, it won't work.
But, blocking outgoing requests by default is super hard, and you risk blocking security updates etc. It can kinda work for a deployed application, but for an employee workstation? Basically impossible.
I wonder if we're approaching an era where we have to go back to saying "you cannot do this, because security" much more often than we'd like.
It’s a good point. As things speed up it will be harder to tell which patches are actually urgent and need to skip the cool-off period.
I think the more robust way of doing this is to have code audits on each published release. Agents can do some of this (eg Github could offer this scanning service, and let external parties fund the scanning on trusted compute).
I think of this more as a “proof of work” problem than provable security; if I see that Mythos has run for N hours on the patch release I am considering upgrading to, then this might suffice.
The key thing here is you need a way to crowdsource the funding of scans, and make them shareable so that the cost can be shared across the community. The package owner obviously can’t control the prompt. And can Mythos-class models be hardened enough to scan hostile code?
To your point on blocking requests, there are programming models that make this easier, like capability-based programming, where code that doesn’t need internet cannot get it; this doesn’t solve things fully, but my general prediction is that adding new architectural patterns is now a lot cheaper and easier to reliably apply across a codebase, so we may see more of this too.
Security vulnerabilities are one thing, but in legal we offer up a concept of "knowledge security" which goes to protecting the fidelity of the agent's legal context. Software bugs seem much more tractable because they're managed by software engineers, as opposed to the pipeline "vulnerabilities" we're finding. We wrote a little about one vector here where legal documents aren't quite what they seem: https://tritium.legal/blog/noroboto
No doubt there are many such knowledge domains exposed today. These are more concerning because they're understaffed and managed by non-technical people for the most part. No Mythos required.
This has always been the bottleneck. Automated tools love to flag vulnerabilities, but almost all are false positives. These need to be triaged and evaluated by humans. This is okay. I’d rather close a false positive after a careful review than miss it altogether.
I don’t think it’s appropriate for calling out humans as a bottleneck. They are an essential part of the process, I’m sure Mythos will also become a catalyst in the process.
That means, they intend to make a load of money before a general release. It is a good strategy.
Great marketing as always, but the rose-tinted view many have seems vicariously misplaced.
These aren't unreachable vulns.
That's convinient.
But wait, don't they have this amazing AI that can fix all the issues itself with a single /goal command? What's the holdup?
I miss the days when HN would RTFA.
> As we noted above, the bottleneck in fixing bugs like these is the human capacity to triage, report, and design and deploy patches for them.
...
> To begin, we’ve released Claude Security in public beta for Claude Enterprise customers. It’s a tool that helps teams scan their codebases for vulnerabilities, and which can generate proposed fixes for them. In the three weeks since launch, Claude Opus 4.7 has been used to patch over 2,100 vulnerabilities. (This is faster than the open-source patching described above in large part because enterprises are fixing their own code, whereas open-source fixes usually require volunteer maintainers who work through coordinated disclosure.)
Your critique of the article would likely land much better if you engaged with it.
> However, this means that disclosed vulnerabilities are a lagging indicator of the accelerating frontier of AI models’ cyber capabilities: we’re not yet at the point where we can fully detail our partners’ findings with Mythos Preview without putting end users at risk. Instead, we provide illustrative examples of the model’s performance, along with aggregate statistics on our progress to date. Once patches for the vulnerabilities that Mythos Preview has discovered are widely deployed, we’ll provide much more detail about what we’ve learned.
I am still a believer that a 100 subagents with good-enough intelligence can get same results as mythos, I am ready for this opinion to be shattered when I eventually try mythos and I believe others here must have tried mythos out too.
"After one month, most partners have each found hundreds of critical- or high-severity vulnerabilities in their software. Collectively, they’ve found more than ten thousand. Several have told us that their rate of bug-finding has increased by more than a factor of ten. For instance, Cloudflare has found 2,000 bugs (400 of which are high- or critical-severity) across their critical-path systems, with a false positive rate that Cloudflare’s team considers better than human testers." (emphasis mine)
So, success is coming not just from the model but also from the harnesses they built around it. The Cloudflare post was more detailed on that front and I wish the rest would share more about it.
The Cisco spec is interesting too, it pretty much describes an architecture of a harness: https://github.com/CiscoDevNet/foundry-security-spec
https://www.flyingpenguin.com/mythos-mystery-in-mozilla-numb...
And how much with Opus 4.7? 5x?
There is also a pretty big risk that anyone who is not you would leak the answer to the test. We are close to n=1 epistemics here. You’re going to have to do the research yourself.
Yes, Anthropic have said they made Opus 4.7 worse at this on purpose.
> It is entirely possible that Mythos is a different architecture or size
It has 5x the token pricing of Opus 4.7, so it's probably larger.
https://www.aisi.gov.uk/blog/our-evaluation-of-openais-gpt-5...
If I understand correctly, Opus 4.7 was launched as nerfed Mythos with some improvements from 4.6.
Anthropic launches major bumps (like 4.6 to 4.7) every 4 - 5 months. So by all accounts, Mythos should be released by July.
The problem reduces to: How quickly can competing models surpass Opus 4.7 and start taking over Anthropic's market share?
So yeah, huge marketing as always.
That's the one that says:
> We took the specific vulnerabilities Anthropic showcases in their announcement, isolated the relevant code, and ran them through small, cheap, open-weights models. Those models recovered much of the same analysis.
Or providing a map with a direction
There is a long history of high-value private vulns being rediscovered from scant details
The American firms are focused on marketing now to convince people to not even consider open sourced models / open weight models as they are inferior (that’s what they want you to believe).
If people actually believe the narrative then the bankers will over price Anthropic and get away with it.
4.6 but close.
"...After fixing the initial set of issues that Anthropic sent to us in February, we built our own harness atop our existing fuzzing infrastructure.
We began with small-scale experiments prompting the harness to look for sandbox escapes with Claude Opus 4.6. Even with this model, we identified an impressive amount of previously-unknown vulnerabilities which required complex reasoning over multiprocess browser engine code..."
So yeah, Anthropic and Mozilla likely compare "Amount of bugs found by Opus 4.6 during early experiments" vs "Amount of bugs found by Mythos during large-scale codebase scanning".
[1] https://hacks.mozilla.org/2026/05/behind-the-scenes-hardenin...
https://xbow.com/blog/mythos-offensive-security-xbow-evaluat...
Do we have a sense that projects like OpenBSD/OpenSSH, FreeBSD, ISC[1] and Apache were included in the "blessed" initial participants in Project Glasswing ?
Or is it big name tech companies, banks and fashionable languages and package managers ?
[1] Bind, DHCP
It's not clear to me that FreeBSD found any of them internally ...
It's probably the right approach to onboard a few independent security companies and task them with reviewing multiple OSS projects than it is to onboard each project individually.
I joke but that is the world we are moving towards. I don’t think many on HN have thought through the second and third order implications.
Here are two experimental exceptions:
https://github.com/vercel-labs/zerolang
https://github.com/sbhooley/ainativelang
We'll like have some standard AI-focused UI libraries that are harnessed into a design gen system where an AI can pull all the real levers, while also developing a large training data set around it.
This is the MoviePass era of language models
Supersonic again is a problem with noise and cost rather than technological.
Self driving is definitely a technological problem.
It could become that way, but thus far no evidence has been presented for it. The best we have right now is that you can spend $20 in tokens to write a patch and then $20K to find a vulnerability in it. First, that's not measuring the same thing. Second, it's not very impressive.
50 years is a long, long time, so I wouldn't bet against it. But I agree that we don't have evidence for it yet.
It seems more likely to me that you could spend $20 to find a vulnerability in a piece of software that costed you $20k in human labor.
That's kind of what the article is about? Mythos is finding lots of security bugs in lots of human-written code. They can now compute some sort of baseline estimate of security bugs per N lines of human-written code or whatever. (Restricted to security bugs that the AI is currently capable of finding, but whatever.) Even before Mythos et al, we can look at historical security bug rates. We do have numbers for estimating the security of human written code.
> It seems more likely to me that you could spend $20 to find a vulnerability in a piece of software that costed you $20k in human labor.
Ok, but that's not what is being discussed in this subthread? The topic is whether or not we have data suggesting that AI-written code is or can be secure, and thus whether insecure human code is fated to replaced with secure AI code. I claim we do not have that data. Therefore, we don't have evidence to think that for the sake of security we should replace all human code with AI code, vs whether AI code is worse for security and so we should replace AI code with human code (that presumably has been vetted with AI, since we do have evidence for its effectiveness.)
If I were to guess, I would probably think that today's AIs are trained solely on mountains of insecure human code and so will probably produce more of the same. Tomorrow's AIs will have the benefit of being trained on human and AI code that has had a large swathe of vulnerabilities purged from it, and so they'll have a much better chance at writing secure code, at least.
It depends a lot on whether the failure modes of AI code generation lend themselves to exploitation as security vulnerabilities. (And whether they will continue to do so.)
there is a difference between a stunt and a viable product. diverless cars and agi are the fusion of Silicon Valley.
Cities should all have better public transport and out in the middle of nowhere you don't need self driving anyway. (And yes, personal cars should be entirely banned from cities)
I wonder how long "near future" is in Anthropic time. I think they have incentives to delay the release of Mythos as long as possible both to save compute and delay distillation by rival labs.
Regardless, what they have been doing with Glasswing is very cool. It's clear that the world has been spared from a massive security nightmare that would have happened in any alternative timeline where the model is publicly released with weak safeguards.
As I see it the primary issue is giving time for the ecosystem to adapt. Once models of a given level of capability have been applied to the majority of the common software in daily use it becomes reasonably safe to release such models publicly regardless of how they are used.
> For example, at one of our Glasswing partner banks, Mythos Preview helped to detect and prevent a fraudulent $1.5 million wire transfer after a threat actor compromised a customer’s email account and made spoof phone calls.
For some reason I am not able to relate to the concreteness of either of these.
First half of the page was occupied with a image, not sure if it was relevant in any ways other than setting up security scare. The size of code base, number of tokens, $ involved seem to be out of scope of the update for some reason. Personally I am getting skeptical about all these optics at this point, just some money printing scheme at high level.
But I didn't find the most important information (or maybe I missed it): how much did it cost to find 1451 security bugs?
And so was malicious vulnerability research.
https://github.com/ErenayDev/instantbugs
Plus, they also mention they check if fixes are available for the bugs they found. What are the chances they are re-reporting old bugs just to inflate their numbers? Bugs that were already fixed?
And how can we be sure their reassessment is not artifically increasing the severity of the CVEs found just to create FUD and sell their product?
> that's just thousands of vulnerabilities being discovered by our trillion parameter model
> thousands of vulnerabilities and trillions of parameters?! At current energy prices, in this economic climate, isolated entirely within your datacenter?
> yes
> may we see it?
> no
>ya right.
Here's a demonstration of it blowing something up.
>can I have one.
No.
And at the moment we have reports from like around 5(?) companies. Btw, Palo Alto Networks has found only 26 vulnerabilities [1]. I'm interested what those partners are and why they have such big amount of vulnerabilities.
> For instance, Cloudflare has found 2,000 bugs (400 of which are high- or critical-severity) across their critical-path systems, with a false positive rate that Cloudflare’s team considers better than human testers.
Yet decided not to share that number. I wonder why.
> Mozilla found and fixed 271 vulnerabilities in Firefox 150 while testing Mythos Preview—over ten times more than they found in Firefox 148 with Claude Opus 4.6;
Mozilla tested Opus 4.6 in a very limited setting (i.e. without proper harness and integration into their workflow; likely without large-scale codebase scanning). It's an incorrect comparison.
> The latest Palo Alto Networks release included over five times as many patches as usual.
Yeah, it's better to say "five times as many..." rather than "26 bugs". Btw, they also used GPT-5.5 and Opus 4.7, so the contribution from Mythos there is unclear.
> Microsoft has reported that the number of new patches they’ll release will “continue trending larger for some time.” And Oracle is finding and fixing vulnerabilities across its products and cloud multiple times faster than before.
Both Oracle and Microsoft are talking about "AI and cybersecurity" in general, not about Mythos.
> For the last few months, Anthropic has used Mythos Preview to scan more than 1,000 open-source projects, which collectively underpin much of the internet—and much of our own infrastructure. > So far, Mythos Preview has found what it estimates are 6,202 high- or critical-severity vulnerabilities in these projects (out of 23,019 in total, including those it estimates as medium- or low-severity).
So, ~6 high- and critical- severity bugs per open-source project v.s. hundreds of high- and critical- severity bugs per partner projects. It looks like the math ain't mathing.
> One example of an open-source vulnerability that Mythos Preview detected was in wolfSSL, an open-source cryptography library that’s known for its security and is used by billions of devices worldwide. Mythos Preview constructed an exploit that would let an attacker forge certificates that would (for instance) allow them to host a fake website for a bank or email provider. The website would look perfectly legitimate to an end user, despite being controlled by the attacker. We’ll release our full technical analysis of this now-patched vulnerability (assigned CVE-2026-5194) in the coming weeks.
Of course, they didn't say that Mythos found only 8 bugs in wolfSSL vs 22 CVE fixed in wolfSSL 5.9.1.
Overall, it feels like yet another marketing stunt.
[1] https://www.paloaltonetworks.com/blog/2026/05/defenders-guid...
Which is not bad this early in the 90+45 day responsible disclosure window.
> Yet decided not to share that number. I wonder why.
It is bizarre to expect a company to disclose the false-positive rate of their security engineers, publicly. That does not happen.
> So, ~6 high- and critical- severity bugs per open-source project v.s. hundreds of high- and critical- severity bugs per partner projects. It looks like the math ain't mathing.
It is pretty obvious they're spending more compute on commercial partners. Why is this surprising?
> Of course, they didn't say that Mythos found only 8 bugs in wolfSSL vs 22 CVE fixed in wolfSSL 5.9.1.
WolfSSL is not the only software project in the world. Mozilla also came out with results that paint it as very effective. I don't think Mythos ever claimed to find all bugs anyways.
Drawback of AI: it works fast
Is this suspected vulns or actual vulns? If I recall correctly, it produced 5 for curl but only 1 was legit
> 1,752 of those high- or critical-rated vulnerabilities have now been carefully assessed by one of six independent security research firms, or in a small number of cases by ourselves. Of these, 90.6% (1,587) have proved to be valid true positives, and 62.4% (1,094) were confirmed as either high- or critical-severity. That means that even if Mythos Preview finds no further vulnerabilities, at our current post-triage true-positive rates, it’s on track to have surfaced nearly 3,900 high- or critical-severity vulnerabilities in open-source code
> Not even half-way through this #curl release cycle we are already at 11 confirmed vulnerabilities - and there are three left in the queue to assess and new reports keep arriving at a pace of more than one/day.
> 11 CVEs announced in a single release is our record from 2016 after the first-ever security audit (by Cure 53).
> This is the most intense period in #curl that I can remember ever been through.
[1]: https://www.linkedin.com/feed/update/urn:li:activity:7463481...
If you read his own top comment on that LinkedIn post he clarifies:
“The simple reason is: the (AI powered) tools are this good now. And people use these tools against curl source code.They find lots of new problems no one detected before. And none of these new ones used Mythos. Focusing on Mythos is a distraction - there are plenty of good models, and people who can figure out how to get those models and tools to find things.”
I guess they forgot to scan Visual Studio Code plugins and their endless npm dependencies.