I’m pretty surprised that Anthropic can directly impact the infra for AWS Bedrock as this article suggests. That goes against AWSs commitments. I’m sure the same is true for Google Vertex but I haven’t digged in there from a compliance perspective before.
> Our own privacy practices also created challenges in investigating reports. Our internal privacy and security controls limit how and when engineers can access user interactions with Claude, in particular when those interactions are not reported to us as feedback.
Ok makes sense and glad to hear
> It remains particularly helpful for users to continue to send us their feedback directly. You can use the /bug command in Claude Code
Ok makes sense and I’d expect that a human can then see the context in that case although I hope it is still very explicit to the end user (I’m not a Claude Code user so I cannot comment)
> or you can use the "thumbs down" button in the Claude apps to do so
This is pretty concerning. I can’t imagine the average person equates hitting this button with forfeiting their privacy.
l1n 3 minutes ago [-]
(Anthropic employee, speaking in a personal capacity)
> I’m pretty surprised that Anthropic can directly impact the infra for AWS Bedrock as this article suggests.
We don't directly manage AWS Bedrock deployments today, those are managed by AWS.
> I can’t imagine the average person equates hitting this button with forfeiting their privacy.
We specify
> Submitting this report will send the entire current conversation to Anthropic for future improvements to our models.
in the thumbs down modal. Is there a straightforward way to improve this copy?
_da_ 5 minutes ago [-]
> This is pretty concerning. I can’t imagine the average person equates hitting this button with forfeiting their privacy.
When you click "thumbs down" you get the message "Submitting this report will send the entire current conversation to Anthropic for future improvements to our models." before you submit the report, I'd consider that pretty explicit.
vlovich123 4 minutes ago [-]
The value of figuring out how to make their LLM serving deterministic might help them track this down. There was a recent paper about how the received wisdom that kept assigning it to floating point associativity actually overlooked the real reasons for non-determinism [1].
> Incorrect routing affected less than 0.0004% of requests on Google Cloud's Vertex AI between August 27 and September 16.
Matches my experience. I use CC through our enterprise Vertex AI account and never noticed any degradation.
In general it seems like these bugs, while serious, were substantially less prevalent than anecdotal online reports would have you believe. We are really talking about a ~1-2 week window here where most issues were concentrated, a relatively small percentage of total requests and total users impacted.
ispeaknumbers 1 hours ago [-]
I'm not sure if you can claim these were "less prevalent than anecdotal online reports". From their article:
> Approximately 30% of Claude Code users had at least one message routed to the wrong server type, resulting in degraded responses.
> However, some users were affected more severely, as our routing is "sticky". This meant that once a request was served by the incorrect server, subsequent follow-ups were likely to be served by the same incorrect server.
30% of Claude Code users getting a degraded response is a huge bug.
extr 52 minutes ago [-]
I don't know about you but my feed is filled with people claiming that they are surely quantizating the model, Anthropic is purposefully degrading things to save money, etc etc. 70% of users were not impacted. 30% had at least one message degraded. One message is basically nothing.
I would have appreciated if they had released the full distribution of impact though.
flutas 42 minutes ago [-]
That 30% is of ALL users, not users who made a request, important to note the weasel wording there.
How many users forget they have a sub? How many get a sub through work and don't use it often?
I'd bet a large number tbh based on other subscription services.
smca 36 minutes ago [-]
(I work at Anthropic) It's 30% of all CC users that made a request during that period. We've updated the post to be clearer.
flutas 1 minutes ago [-]
Thanks for the correction and updating the post.
I typically read corporate posts as cynically as possible, since it's so common to word things in any way to make the company look better.
Glad to see an outlier!
extr 34 minutes ago [-]
That's a pretty cynical read. My personal impression is that Anthropic has a high level of integrity as an organization. Believe what you want, I'm inclined to give them the benefit of the doubt here and move on.
thousand_nights 39 minutes ago [-]
i don't trust companies anymore because every time there's a worldwide outage they use softspeak like "we're observing elevated amounts of errors for a small subset of users", hours after some CTO approves to change the status page
imho there's a big market gap for companies that are truly honest with customers instead of corporate gaslighting
cyanf 39 minutes ago [-]
> On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected.
Interesting, this implies that the 1M context servers performs worst at low context.
Perhaps this is due to some KV cache compression, eviction or sparse attention scheme being applied on these 1M context servers?
kiratp 11 minutes ago [-]
This is due to RoPE scaling.
> All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required. It is also recommended to modify the factor as needed. For example, if the typical context length for your application is 524,288 tokens, it would be better to set factor as 2.0.
Title should be fixed: it’s about Claude models in general, not Claude Code
Wowfunhappy 58 minutes ago [-]
> On August 25, we deployed a misconfiguration to the Claude API TPU servers that caused an error during token generation. An issue caused by a runtime performance optimization occasionally assigned a high probability to tokens that should rarely be produced given the context, for example producing Thai or Chinese characters in response to English prompts, or producing obvious syntax errors in code. A small subset of users that asked a question in English might have seen "สวัสดี" in the middle of the response, for example.
Can anyone explain to a layperson how this sort of thing is even possible for an LLM?
For normal code, of course stupid bugs happen all the time. You accidentally introduce an off-by-one error in a conditional, for example, or add an extra `goto fail`.
But LLMs aren't written by humans! Models are trained by automated programs over a period of many months across unfathomably massive data centers.
How would a human introduce a bug like the one described in TFA?
Voloskaya 53 minutes ago [-]
LLMs are still executed by code written by humans.
In this case, the model ultimately give you a probability distribution over each (~200k) tokens in the vocabulary. It's then up to you to decide how you want to sample the next token, you could for example just always sample the most likely, or to make the output more creative, you can sample randomly from the top-k tokens. This top-k sampling, to make it efficient, is written in XLA and compiled to run directly as a kernel, there was a bug in that kernel, which presumably led to tokens outside of the top-k window be select from times to times.
Centigonal 48 minutes ago [-]
LLMs produce a probability distribution for what the next token might be. How you pick the actual word that is printed next from that probability distribution is by using a sampling approach[1]. If your sampling approach is "select the next word randomly from among the top 4 possibilities" and you flip a > sign, you could end up with the behavior described in the OP.
The next word can also be selected with weighted randomization and "temperature" is used to control how much weight lower probability tokens get.
I've honestly received the best results in creative writing by ignoring top_k/top_p and simply tuning temperature. Restricting my output to only common words causes everything to feel generic. But Deepseek constantly breaks into Chinese/gibberish/ZALGO! when I go to 1.14.
This isn't related to the "recent issues" but I feel like it's useful advice for anyone trying out AI story creation.
ashdksnndck 57 minutes ago [-]
There are many layers of human-written code in between you and the weights.
zer00eyz 21 minutes ago [-]
If you are going to run a non deterministic system on three very different hardware platforms doesn't it behoove you to tell your users where their experience is coming from?
Calling the platforms A, B and C might help provide us the insight we're missing to spot incongruous behaviors faster than trying to aggregate more generalized feedback.
OGEnthusiast 1 hours ago [-]
Seems like Claude is using TPUs a lot more than I thought. For some reason I thought 90%+ of their capacity was from AWS.
flutas 58 minutes ago [-]
And yet no offers of credits to make things right for the users, for what was essentially degraded performance of what you paid for.
I know I'll probably get push back on this, but it left a sour taste in my mouth when I paid for a $200 sub that felt like it was less useful than ChatGPT Plus ($20) at times.
Or to summarize: [south park "we're sorry" gif]
behnamoh 35 minutes ago [-]
Reminder that Anthropic is the only AI company that has never released any open-source/weight models.
arduanika 31 minutes ago [-]
Sure, but don't you feel safer that way?
behnamoh 22 minutes ago [-]
of course, who wants an open-source Sonnet 3... /s
moatmoat 2 hours ago [-]
TL;DR — Anthropic Postmortem of Three Recent Issues
In Aug–Sep 2025, Claude users saw degraded output quality due to infrastructure bugs, not intentional changes.
The Three Issues
1. *Context window routing error*
- Short-context requests sometimes routed to long-context servers.
- Started small, worsened after load-balancing changes.
2. *Output corruption*
- TPU misconfigurations led to weird outputs (wrong language, syntax errors).
3. *Approximate top-k miscompilation*
- A compiler bug in TPU/XLA stack corrupted token probability selection.
- Occasionally dropped the true top token.
Why It Was Hard to Detect
- Bugs were subtle, intermittent, and platform-dependent.
- Benchmarks missed these degradations.
- Privacy/safety rules limited access to real user data for debugging.
Fixes and Next Steps
- More sensitive, continuous evals on production.
- Better tools to debug user feedback safely.
- Stronger validation of routing, output correctness, and token-selection.
sebastiennight 1 hours ago [-]
> Privacy/safety rules limited access to real user data for debugging.
Do their ToS really limit access to user data (prompt/response)? I don't remember seeing anything to that effect in their terms.
mcintyre1994 1 hours ago [-]
I’d imagine they have a lot of internal controls, even if ultimately someone at the company can read the data within their terms. It makes sense that the teams debugging stuff wouldn’t have this access immediately.
favorited 1 hours ago [-]
I know that when you submit a thumbs up/down rating for a response, you need to opt-in to the whole chat conversation being shared with Anthropic.
bravetraveler 1 hours ago [-]
> We don't typically share this level of technical detail about our infrastructure, but the scope and complexity of these issues justified a more comprehensive explanation.
Layered in aggrandizing. You host a service, people give you money.
levocardia 53 minutes ago [-]
No, what that statement means is "we know that if we just say 'we weren't downgrading performance to save money', you won't believe us, so here is a deep dive on the actual reason it happened"
bravetraveler 45 minutes ago [-]
They can still do the deep dive, that is absolutely convincing. They likely did: distracted before I could finish [work, unfortunately - incident of our own]
My criticism is it's 'puffy'. The 'scope and complexity' for a public postmortem is 'customer-facing'. Otherwise it's a tree/forest scenario.
One might say 'the lady doth protest too much'; this should be routine. It is, elsewhere: see Cloud, Web Hosting, PBX. Pick your decade.
deepdarkforest 1 hours ago [-]
Wow. Sneaky. They do not even state the rate of impact for the XLA bug afaik, which affected everyone, not just claude code users, very vague. Interesting.
Claude code made almost half a billion so far[1] (>500m in ARR and its like 9 months old) , and 30% of all users have been impacted at least once, just from the first routing bug. Scary stuff.
Their post mortem is basically "evaluations are hard, we relied on vibe checking, now we are going to have even more frequent vibe checking". I believe it was indeed unintentional, but in the future where investor's money wont come down from the skies, serving distilled models will be very tempting. And you can not be liable to any SLA currently, it's just vibes. I wonder how enterprise vendors are going to deal with this going forward, you cannot just degrade quality without client or vendor even being able to really prove it.
Is your contention that paying for a service entitles you to zero bugs, ever?
deepdarkforest 47 minutes ago [-]
Of course not! But usually, you can quantify metrics for quality, like uptime, lost transactions, response time, throughput etc. Then you can have accountability, and remediate. Even for other bugs, you can often reproduce and show clearly the impact. But in this case, other than internal benchmarks, you cannot really prove it. There is no accountability yet
_zoltan_ 20 minutes ago [-]
why would they publish the data you seek? I would not publish it either.
the blog explains what issues they had and how they fixed them. this is good enough.
flutas 44 minutes ago [-]
If you paid for a streaming service and the HD option only worked for a random subset of users, and not you, would you complain?
It's a material difference in the product, not just "a bug."
dylan604 33 minutes ago [-]
I'd honestly blame my ISP for traffic shaping my connection as a first assumption, and not immediately blame the streaming platform.
> Our own privacy practices also created challenges in investigating reports. Our internal privacy and security controls limit how and when engineers can access user interactions with Claude, in particular when those interactions are not reported to us as feedback.
Ok makes sense and glad to hear
> It remains particularly helpful for users to continue to send us their feedback directly. You can use the /bug command in Claude Code
Ok makes sense and I’d expect that a human can then see the context in that case although I hope it is still very explicit to the end user (I’m not a Claude Code user so I cannot comment)
> or you can use the "thumbs down" button in the Claude apps to do so
This is pretty concerning. I can’t imagine the average person equates hitting this button with forfeiting their privacy.
> I’m pretty surprised that Anthropic can directly impact the infra for AWS Bedrock as this article suggests.
We don't directly manage AWS Bedrock deployments today, those are managed by AWS.
> I can’t imagine the average person equates hitting this button with forfeiting their privacy.
We specify
> Submitting this report will send the entire current conversation to Anthropic for future improvements to our models.
in the thumbs down modal. Is there a straightforward way to improve this copy?
When you click "thumbs down" you get the message "Submitting this report will send the entire current conversation to Anthropic for future improvements to our models." before you submit the report, I'd consider that pretty explicit.
[1] https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
Matches my experience. I use CC through our enterprise Vertex AI account and never noticed any degradation.
In general it seems like these bugs, while serious, were substantially less prevalent than anecdotal online reports would have you believe. We are really talking about a ~1-2 week window here where most issues were concentrated, a relatively small percentage of total requests and total users impacted.
> Approximately 30% of Claude Code users had at least one message routed to the wrong server type, resulting in degraded responses.
> However, some users were affected more severely, as our routing is "sticky". This meant that once a request was served by the incorrect server, subsequent follow-ups were likely to be served by the same incorrect server.
30% of Claude Code users getting a degraded response is a huge bug.
I would have appreciated if they had released the full distribution of impact though.
How many users forget they have a sub? How many get a sub through work and don't use it often?
I'd bet a large number tbh based on other subscription services.
I typically read corporate posts as cynically as possible, since it's so common to word things in any way to make the company look better.
Glad to see an outlier!
imho there's a big market gap for companies that are truly honest with customers instead of corporate gaslighting
Interesting, this implies that the 1M context servers performs worst at low context. Perhaps this is due to some KV cache compression, eviction or sparse attention scheme being applied on these 1M context servers?
> All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required. It is also recommended to modify the factor as needed. For example, if the typical context length for your application is 524,288 tokens, it would be better to set factor as 2.0.
https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Thinking
Can anyone explain to a layperson how this sort of thing is even possible for an LLM?
For normal code, of course stupid bugs happen all the time. You accidentally introduce an off-by-one error in a conditional, for example, or add an extra `goto fail`.
But LLMs aren't written by humans! Models are trained by automated programs over a period of many months across unfathomably massive data centers.
How would a human introduce a bug like the one described in TFA?
[1] Here is an example of two common approaches: https://www.reddit.com/r/AIDungeon/comments/1eppgyq/can_some...
I've honestly received the best results in creative writing by ignoring top_k/top_p and simply tuning temperature. Restricting my output to only common words causes everything to feel generic. But Deepseek constantly breaks into Chinese/gibberish/ZALGO! when I go to 1.14.
This isn't related to the "recent issues" but I feel like it's useful advice for anyone trying out AI story creation.
Calling the platforms A, B and C might help provide us the insight we're missing to spot incongruous behaviors faster than trying to aggregate more generalized feedback.
I know I'll probably get push back on this, but it left a sour taste in my mouth when I paid for a $200 sub that felt like it was less useful than ChatGPT Plus ($20) at times.
Or to summarize: [south park "we're sorry" gif]
In Aug–Sep 2025, Claude users saw degraded output quality due to infrastructure bugs, not intentional changes.
The Three Issues 1. *Context window routing error* - Short-context requests sometimes routed to long-context servers.
2. *Output corruption* - TPU misconfigurations led to weird outputs (wrong language, syntax errors). 3. *Approximate top-k miscompilation* - A compiler bug in TPU/XLA stack corrupted token probability selection. Why It Was Hard to Detect - Bugs were subtle, intermittent, and platform-dependent.- Benchmarks missed these degradations.
- Privacy/safety rules limited access to real user data for debugging.
Fixes and Next Steps - More sensitive, continuous evals on production.
- Better tools to debug user feedback safely.
- Stronger validation of routing, output correctness, and token-selection.
Do their ToS really limit access to user data (prompt/response)? I don't remember seeing anything to that effect in their terms.
Layered in aggrandizing. You host a service, people give you money.
My criticism is it's 'puffy'. The 'scope and complexity' for a public postmortem is 'customer-facing'. Otherwise it's a tree/forest scenario.
One might say 'the lady doth protest too much'; this should be routine. It is, elsewhere: see Cloud, Web Hosting, PBX. Pick your decade.
Claude code made almost half a billion so far[1] (>500m in ARR and its like 9 months old) , and 30% of all users have been impacted at least once, just from the first routing bug. Scary stuff.
Their post mortem is basically "evaluations are hard, we relied on vibe checking, now we are going to have even more frequent vibe checking". I believe it was indeed unintentional, but in the future where investor's money wont come down from the skies, serving distilled models will be very tempting. And you can not be liable to any SLA currently, it's just vibes. I wonder how enterprise vendors are going to deal with this going forward, you cannot just degrade quality without client or vendor even being able to really prove it.
[1][https://www.anthropic.com/news/anthropic-raises-series-f-at-...]
the blog explains what issues they had and how they fixed them. this is good enough.
It's a material difference in the product, not just "a bug."