If their way of life, influenced by both genes and culture, has produced such astonishing results, then what's the downside of it all? Otherwise most other groups should have evolved that way. The main trait responsible for their success in culture and business is their higher average IQ, which is about 1 standard deviation above the European average. So the extra costs that are the counter-weight to their greater success could reflect being too smart.
I have a different idea, perhaps not an original one since I haven't read a lot of the relevant literature. Others have talked about it informally, but I ran across some studies that back it up. That is that Jews seem to be unusually prone to bizarre cults, and that can't be too good for you, all else equal. I'm not talking about the zealots, the original Christians, etc., but the ones that are truly mind-bogglingly weird -- Marxism, Freudian psychoanalysis, Objectivism, and so on. But hey, big deal, investing in some wacko intellectual theory will be mostly harmless -- to the academics, anyway, although perhaps not for the rest of society if it gets implemented on a wide scale.
I'm talking more about the weirdo cults that began in the late '60s and peaked during the '70s and early '80s. Unlike Marxist English professors, they were giving up a lot just to belong to a group of misfits -- separation from families, donating labor and earnings to the group, and so on. And unlike the earlier zealots, they weren't following an established religious group that had already earned a reputation for trustworthiness. Most of those cults ended in scandal after enough of the members woke up to the fact that the leaders were a bunch of predatory, exploitative false Messiahs. The ones still left have nothing of their former shape, so I consider that as an end too.
Unlike their numbers in other institutions that we view favorably, their numbers in cults is not a well known story. If it was ever widespread during the heyday of cults, it has vanished down the memory hole as an embarrassment. Non-Jews are not going to come across references either because they've lost interest in cults, having re-branded them New Religious Movements as though the Manson family, Jonestown, and the Children of God were all just budding Christianities.
Here are three quotes that mention estimates of Jewish membership in cults (links added for reference).
From Kaslow & Sussman (1982), Cults and the Family, Volume 4 (p.101):
Spero (1977), for example, cites an estimated twelve percent of American membership in the Unification Church (UC, or "Moonies") to be Jewish, a vastly disproportionate overrepresentation.
From Jenkins (2000), Mystics and Messiahs: Cults and New Religions in American History (p.195):
Though estimates vary widely, Rudin and Rudin  suggest that "perhaps 20 percent of Hare Krishnas are Jewish. Jews constitute as much as 30 percent of Divine Light Mission membership, and there are many Jews in Scientology."
And from Saiba (1986), Christian And Jewish Responses To ISKCON: Dialogue Or Diatribe?:
Headlines in Jewish community bulletins and newspapers further highlight Jewish vulnerability to the cults,  which are allegedly recruiting Jews in disproportionate numbers. 
Here are the two footnotes:
 Headlines like 'Jews' Vulnerability to the Lure of the Cults', in the Long Island Jewish World (Dec. 6, 1981); 'The Hare Krishna Alerts Israel to Missionary Danger', in the Jewish Press, Brooklyn, N.Y. (April 5, 1979); and 'Cults: A Growing Threat to Jewish Continuity and Survival in America', in Hakol, Allentown, PA (May, 1977), are but a random sample of the disquietude which has been spreading throughout the Jewish community.
 In general, Jewish sources believe that between 15-25% of all cult members are Jewish and that 15% of Hare Krishna members were brought up as Jews (cf. Adahan, 1981:37; Appell, 1978:20). Neff (1979:23) states that up to 45% of any given cult could be of Jewish background. Gittelson and Reed (1981:212) disagree and hold that Jews are not proportionately represented in the cults. The estimates vary, and there are no completely reliable statistics. At times Jews appear more concerned with the number of Jewish converts to Messianic Judaism (cf. Rudin, 1978:353-5).
Although the average Jewish person may not have joined such an out-there cult, their over-representation by about 10 times suggests that their distribution for susceptibility to crazy and blind crowd-following is shifted toward the more wacko end of the spectrum, compared to the European distribution. They seem more likely to lack an instinct for performing basic reality checks, especially toward charismatic leaders of zero-track-record cult movements. That can't be good -- and it isn't. So this would seem to be the main cost of the Ashkenazi way of life that has kept it from growing too common.
Whether this downside stems directly from their higher average IQ -- the cost of being too smart for your own good -- or whether it's an independent feature of their group is not clear. East Asians are also smarter on average than Europeans, and while they did go in for Maoism, and while they do believe more in magic, they don't seem to show up disproportionately in crazy cults. If anything, their distribution seems to be farther away from the fanatical/zealous end than even the European distribution.
We could also check this by looking at Sephardic and Mizrahi Jews, who don't have higher average IQs compared to Europeans. If they are no more likely than Europeans to dive head-first without looking into the whirlpool of brand-new cults, that would mean the Ashkenazi tendency toward cultism could in fact be related to their greater brainpower. If these other Jewish groups are just as zealous as the Ashkenazim, then it cannot reflect the cost of being too smart, but rather some other enduring feature of the environments that Jews have found themselves in, and to which greater zealotry would adapt them.