NHacker Next
login
▲Show HN: JevBench, a reproducible benchmark for typed decision modelsbenchmarkheaven.com
43 points by florianstandhar 10 hours ago | 7 comments
Loading comments...
hbrn 10 minutes ago [-]
$40m in funding, 2 years in stealth.

Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?

Is it surprising that Jev consistently thinks it's Qwen?

I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.

sean_pedersen 56 minutes ago [-]
Good project but this one also exists https://huggingface.co/spaces/multimodalart/jev-decision-ind... and the results do not seem to add up and also model sets are different... still needs time to mature likely
swyx 2 hours ago [-]
jev ceo on why he eschewed benchmarking: https://www.latent.space/i/216783460/privacy-benchmarking-an...
arbot360 26 minutes ago [-]
Many SaaS vendors forbid benchmarking, I find it crazy that such anti-competitive terms are standard across the industry but they are. Generally the goal of such terms is to "control the narrative" around the product, regardless of the truth of performance being better or worse than competitors.
tomrod 6 minutes ago [-]
I mean, that's a great reason to ignore JEV entirely.

"Trust, but verify" isn't just a catchy cliche. It's the only way to operate where models and code are fast to market.

jldugger 55 minutes ago [-]
Interesting; was curious how this didn't fall into trouble with ToS. Apparently the "no benchmarks" clause was intended for "limited preview" audiences and didn't get removed at launch on accident.
nzoschke 48 minutes ago [-]
https://is-it-ai-slop.app.mintapis.com/ is a fun tool. Is the source or methodology for that in the github repo? I couldn't find it immediately.

We've been experimenting with Jev for classifying email, some thoughts here: https://housecat.com/blog/classifying-email

Flagging AI written email is a much requested feature too.

10 hours ago [-]