How many papers should I publish during PhD to get a decent postdoc position?

[petergreat]

I will start a PhD in physics (high energy theory) this fall, probably in a British graduate school. How many papers should I publish during PhD to get a decent postdoc position? I know this is a vague question, but I just want to get a feeling for the level of competitiveness for an academic career after PhD. Also, does the reputation of the PhD granting institution have an impact?

 

[twofish-quant]

I will start a PhD in physics (high energy theory) this fall, probably in a British graduate school. How many papers should I publish during PhD to get a decent postdoc position? I know this is a vague question, but I just want to get a feeling for the level of competitiveness for an academic career after PhD. Also, does the reputation of the PhD granting institution have an impact?

First thing... And I want to write this in big bold letters so you get the point...

*****ASSUME THAT YOU WILL NOT HAVE A TRADITIONAL ACADEMIC CAREER*****
*****ASSUME THAT YOU WILL NOT GET A PROFESSORSHIP*****

The jobs are *SO* few that you will need to come up with some backup strategies while you are doing your Ph.D. You should get some training in something like C++ programming, science journalism, or community college teaching. *Those* are viable careers. Hoping for a research professorship is like betting on the lottery. The first order approximation to your chance of getting that sort of position is zero. (It's actually higher than zero, but zero is a good first order approximation.)

One thing that you probably should do is to read the Chronicle of Higher Education http://www.chronicle.com/

Okay...

Quality is more important than quantity. What you want to do is to establish yourself as the world's leading authority in some niche field, so that when your CV goes across someone's desk, it's like "I've read his papers. Cool!" You probably should come up with a steady stream of papers just so that you get used to publishing.

The reputation of the institution is pretty much totally unimportant. The reputation of your advisor is crucial. What you'll quickly find is that you won't be identified as "Ph.D. from X university" but rather as "So and so's student." The main impact of the institution is indirect. If you have an institution with well funded facilities and good networking ability, this will help you put out better papers.

 

[cosmogirl]

I have a marginally related question: in the UK, the PhD typically takes 3 years, whereas in continental Europe and most other places it takes 4 years, and in the US 6 years (the latter students take more courses, but still spend a lot more time in academia). Presumably, the UK students will write less papers during their PhD than their continental or US mates. How then can they be competitive on the world postdoc market?

If we take into account the fact that one has to apply for a postdoc position 1 year before they finish their PhD, this leaves the UK students only 2 years to produce any amount of papers!
I'm asking this because I'm myself considering going to the UK for my PhD, in which case I'll be limited to 3 years by the scholarship rules.

 

[vtakhist]

To cosmogirl:

In USA it is 6 years (on average), because the first 1-2 years are classes.
If a student comes directly from bachelor degree he is most likely not prepared enough in advance topics. However, some students come with a very strong MS degree or already did research elsewhere, they could finish PhD in 3-4 years, since they can start doing their research right away without classes.

So as you can see it is similar to UK's MS + PhD. So if you think USA students have extra 2 years, you have them also in your MS.

Hope this helps.

 

[blackfedora]

Are the prospects for a Math PhD just as grim? I'd like to end up at a teaching institution eventually, and I know there's no way in hell I'll end up at Harvard and the like, but is academia really so crowded they won't even let teaching professors in?

 

[twofish-quant]

Are the prospects for a Math PhD just as grim? I'd like to end up at a teaching institution eventually, and I know there's no way in hell I'll end up at Harvard and the like, but is academia really so crowded they won't even let teaching professors in?

The first order approximation to your getting a position at a major university is zero. Yes you might do it, just like you might win the lottery, but it's foolish to make career plans assuming that you will.

The big research universities care about research. Teaching is very secondary. The problem is that there are so few places that if you spend effort to improving your teaching skills, you make yourself totally uncompetitive as far as research.

The good news is that there is a huge demand for teachers at the high school, community college, and to some extent at small liberal arts colleges. The problem is that in some cases that demand hasn't translated into decent incomes, and the other problem is that it's a whole different set of skills. Yes, you may understand differential topology, but can you deal with teaching Algebra I to someone that has a severe case of math phobia? This is important because if you really do want to work at a SLAC, then you have to acquire a very different set of skills.

Also if you want a faculty position for the sake of getting a faculty position, it's better if you do a math education degree with the education department, since people that have gotten those don't seem to be having problem getting university jobs.

The other thing is that you really ought to read the "Chronicle of Higher Education." One big advantage you have over me when I just started my Ph.D. is that the web exists. When I was starting my undergraduate, it was possible to hide how bad the problem was, but it isn't possible now.

 

[blackfedora]

As for high school, my dad is a high school teacher, who's also taught community college. He's affected a few kids lives, but over the years I've watched that profession suck out his soul piece by piece.

I was wondering if I should try and add on an education minor, or concentration, while I'm in undergrad. Also, it's not that I don't want to do research, it's just that I think I'd be happier at an institution like a small liberal arts college that values teaching equally, or even more so than research.

 

[twofish-quant]

I was wondering if I should try and add on an education minor, or concentration, while I'm in undergrad.

A lot real depends on the institution that you are in. A lot of people in the sciences have very strong prejudices against people in education departments, and one thing that I've seen is that someone that thinks badly of education departments, gets into some education course which is badly taught, and this merely reinforces those prejudices.

One thing that will help a lot is if you get yourself into situations where you do have to teach. You'll quickly learn how hard it really is when conditions are less than ideal.

 

[blackfedora]

I got to a small liberal arts college that encourages its students to go into fields that benefit the world, teaching is has high priority with research. I've heard of more than 1 assistant professor getting a tenure track position here because the students reported teaching skills better than the other applicants, this has lead me to believe that maybe there is still room out there for someone like me at a teaching college / university.

Looking back over your post I'd like to stress that I don't WANT a position at a major school. I'd like to produce research, but I'm more concerned with getting a position at a college that is well enough known that I'll be teaching kids who have ability and love of learning. Certainly not all of them will, but I've noticed a lot more of those people here at my home institution than I ever did at any of the 4 prep schools that my father taught at while I was being raised.

Also, does anyone had advice for a couple going down this path? My girlfriend is premed and I'm dreading the grad school application process because even if I get accepted to the program of my dreams, I refuse to go through another 4 to 6 years of long distance because she was rejected by the med schools in the area.

 

[DukeofDuke]

Also, does anyone had advice for a couple going down this path? My girlfriend is premed and I'm dreading the grad school application process because even if I get accepted to the program of my dreams, I refuse to go through another 4 to 6 years of long distance because she was rejected by the med schools in the area.

Well then, you'll obviously have to either break up or stop being so unreasonable.
You can't expect her to sacrifice her goals fully for yours to be optimized (NO med school for her so you can get your DREAM grad school?).
Compromise if you can- that is, use symmetry. Sit in her shoes and say "I refuse to go through another 4 to 6 years of long distance because he rejected the grad schools in the area."

 

[blackfedora]

Well then, you'll obviously have to either break up or stop being so unreasonable.
You can't expect her to sacrifice her goals fully for yours to be optimized (NO med school for her so you can get your DREAM grad school?).
Compromise if you can- that is, use symmetry. Sit in her shoes and say "I refuse to go through another 4 to 6 years of long distance because he rejected the grad schools in the area."

I'm planning on compromise, I'm just asking if anyone has any pointers along this line of thought. Is it worth telling the graduate programs you apply to/get accepted into. Sorry if my original post sounded self centered, I'm well away of the dual of my problem. I'm just concerned about the nightmare situation of no overlap existing between the places we both get accepted. I don't even know what to do in that instance. What can I do if there is no compromise to be made. Should I even worry about this possibility or should I wait and see if it's an issue?

 

[Choppy]

I'm planning on compromise, I'm just asking if anyone has any pointers along this line of thought. Is it worth telling the graduate programs you apply to/get accepted into. Sorry if my original post sounded self centered, I'm well away of the dual of my problem. I'm just concerned about the nightmare situation of no overlap existing between the places we both get accepted. I don't even know what to do in that instance. What can I do if there is no compromise to be made. Should I even worry about this possibility or should I wait and see if it's an issue?

There's no point in worrying too much about it because it isn't an issue yet and may not ever be. Just keep the lines of communication open on the issue between you and your girlfriend and deal with the issues as they arise.

 

[cosmogirl]

To cosmogirl:

So as you can see it is similar to UK's MS + PhD. So if you think USA students have extra 2 years, you have them also in your MS.

Thanks for the reply, vtakhist, but it doesn't solve the problem. In the UK there's typically no masters, except for some 1-year taught courses. Some UK students start PhD right after their bachelors, and then they only have 3 years for the whole business.

Anyway, back to the original question (though the discussion seems to have drifted away), I can tell you that some recent graduates from my department (astro) who got nice postdoc positions, and those who are about to finish their PhD have written about 5 to 9 papers. Typically a person writes one paper after his masters and the rest during his PhD.

 

[Modey3]

@ petergreat
This is a vague question because your name can be on a paper which your not first author of and didn't do the majority of the work. Also, if your doing experimental work naturally your going to have fewer papers than someone running DFT simulations. IT also depends on the nature of your work. Some PhD students do lots of instrument building and repair work so they can run their experiments. Naturally these students will have fewer papers. If you want to run a paper-mill go into computer simulation.

@twofish-quant
Obviously the jobs in academia are competitive to get into, but many PhDs (especially in my field of Materials Science) go into industry, work for national labs, or governmental work. Bottom line, there are jobs for PhDs outside of academia.

modey3

 

[f95toli]

Thanks for the reply, vtakhist, but it doesn't solve the problem.

No, it doesn't. Which is why more and more students in the UK end up spending 4 years as PhD students. This is at partly to conform to the European norm, but mainly because 3 years is simply not enough time for many students, especially when working with complicated experiments where just building the setup can quite easily take 3 years.

 

[Andy Resnick]

Thanks for the reply, vtakhist, but it doesn't solve the problem. In the UK there's typically no masters, except for some 1-year taught courses. Some UK students start PhD right after their bachelors, and then they only have 3 years for the whole business.

Anyway, back to the original question (though the discussion seems to have drifted away), I can tell you that some recent graduates from my department (astro) who got nice postdoc positions, and those who are about to finish their PhD have written about 5 to 9 papers. Typically a person writes one paper after his masters and the rest during his PhD.

Wow... 5 to 9 papers is *much* higher than I've seen- my old department had a graduation requirement that 2 papers get published. For every single student, that second paper has been the rate-limiting step of graduation (average time, from pre-MS to PhD of 5 years).

 

[blackfedora]

Thanks for the reply, vtakhist, but it doesn't solve the problem. In the UK there's typically no masters, except for some 1-year taught courses. Some UK students start PhD right after their bachelors, and then they only have 3 years for the whole business.

Anyway, back to the original question (though the discussion seems to have drifted away), I can tell you that some recent graduates from my department (astro) who got nice postdoc positions, and those who are about to finish their PhD have written about 5 to 9 papers. Typically a person writes one paper after his masters and the rest during his PhD.

oops, sorry I derailed the post I'll leave it be ><

 

[f95toli]

Wow... 5 to 9 papers is *much* higher than I've seen- my old department had a graduation requirement that 2 papers get published. For every single student, that second paper has been the rate-limiting step of graduation (average time, from pre-MS to PhD of 5 years).

Yes, I wonder what counts as "papers" in this case. I realize that different areas of science have different "cultures" when it comes to publishing papers, but 5 to 9 sounds a bit unrealistic.
Usually when people talk about papers they refer to publications where they were the first (or perphaps second) author, conference contributions and similar are also not counted (because they are rarely "properly" peer-reviewed and usually only appear in proceeding, there is a reason why conference papers go into their own section in academic CVs).

I am currenty publishing about one paper per 18 months, which is a fairly good pace. But I've been doing this quite a while now (I finished my PhD in 2005 and have been a post-doc ever since) and I collaborate with several people (on several different projects) who produce samples that I measure (meaning I don't have to rely on a single source). It takes me about 12-24 months to gather enough data to write a paper, but this is with an experimental setup that I've been using a for a few years now and is very stable (I had to spend about 12 months troubleshooting the vacuum system when I first started)

The PhD students I've worked with here in the UK (that are supposed to -but never do- finish in 3 years) are usually asked to publish one paper before writing their thesis, but not everyone does. I know someone who is now a post-doc who only co-authored a single paper during her PhD (she did write a couple of conference papers). The reason was simply that she was working on a project where she was supposed to measure samples that another PhD student was fabricating, and he never managed to fabricate a single working sample (which wasn't really his fault either, it turns out that the method he was asked to use does not actually work, i.e. the whole project was a dead end).
She still managed to get a post-doc positon, the reason was simply that she is a very good experimental physicists and the work she did do (setting up and testing the measurement setup, which took the better part of her 3 years) was really good.

Note that I am not saying this is an ideal situation, but it happens. There is no way to "plan" research in detail, sometimes things do not work as planned (and equipment breaks etc) and if you only have three years there is not always time to revert to a plan B.
It doesn't matter how good you are, you do need a bit of luck in this line of work.

 

[martin_blckrs]

I think this very much depends on the field you're working in. In String Theory, for example, it is really not that unusual to have around 5 papers published before graduation and some people really manage as much as 8-9 (even with a 3 year long PhD). But even in String Theory it depends pretty much on whether you work on something more phenomenological (usually a more rapid pace of publishing) or something like topological strings (difficult). I guess the same holds for other research areas as well.

I agree with the point that people having just a 3 year PhD have a disadvantage against the others regarding the postdoc competition.

 

[FirstYearGrad]

As has been said, it's highly dependent upon the philosophy of the advisor and department. Some are very careful. I actually know one Ph.D student who had his first first author paper published in Nature like 3 or 4 years in. Obviously that is going to be worth a lot more than 5 papers published in Physical Review W.

 

[Dale]

I agree, it depends on the field you are working in. However, it very much helps to be in a field where publishing is common. Having several papers published or accepted makes it almost impossible for the committee to reject your dissertation, and it also makes writing your dissertation almost trivial.

 

[Modey3]

I don't understand the single-minded focus on publishing that some departments have. Are there not other important qualities to judge a young PhD coming out of school? Important qualities such as his ability to design/repair/operate equipment, design meaningful experiments, program (if applicable), and become an expert in his narrow area of study should not go unnoticed. Obviously papers are quantifiable, but I've read a lot of "copycat" papers where the system of study is a perturbation of what's been done before and the methods are exactly the same. I guess my idea of science is different than the mainstream.

modey3

 

[ZapperZ]

I don't understand the single-minded focus on publishing that some departments have. Are there not other important qualities to judge a young PhD coming out of school? Important qualities such as his ability to design/repair/operate equipment, design meaningful experiments, program (if applicable), and become an expert in his narrow area of study should not go unnoticed. Obviously papers are quantifiable, but I've read a lot of "copycat" papers where the system of study is a perturbation of what's been done before and the methods are exactly the same. I guess my idea of science is different than the mainstream.

modey3

But this all depends on the NATURE of the Ph.D work, doesn't it? A physicist majoring in, say, detector physics, would certainly be evaluated on his/her ability to design meaningful experiments, etc. This should also involve publications. After all, the Nuclear Instrumentation and Methods journals, for example, are exactly just for that! However, if you are a condensed matter physicist, and all you are able to do is design experiments without producing anything that advances the knowledge of the field, then that hasn't fulfilled what you were supposed to be doing in the first place.

It is all a matter of context and what the subject matter is. And often, it is out of the hands of the department, because quite often, the Graduate School requirement can make certain demands. A publication (or more) in the field related to the student's subject area implies that the student has at least shown the ability to produce work that has been refereed to be significant enough for publication.

Zz.

 

[twofish-quant]

I don't understand the single-minded focus on publishing that some departments have.

It's usually because if you have lots of papers, it makes it easier to get funding. You want something to show your funders that you are doing something.

Are there not other important qualities to judge a young PhD coming out of school? Important qualities such as his ability to design/repair/operate equipment, design meaningful experiments, program (if applicable), and become an expert in his narrow area of study should not go unnoticed.

The basic problem is that there are so many competent Ph.D.'s and so few jobs that any criterion you come up with is going to be semi-bogus.

 

[Andy Resnick]

I don't understand the single-minded focus on publishing that some departments have.

<snip>

Obviously papers are quantifiable

<snip>.

modey3

You answered your own question.

 

[Modey3]

You answered your own question.

You didn't understand what I'm getting at. There are a lot of papers out there that are essentially duplicates of each other in that they use the same methods and determine the same quantities, but just on different systems. That's more engineering work than science. In my opinion, it's the quality not the quantity of the paper that's important. I rather publish 1 paper that's been referenced 50+ times than 5 papers that have only been referenced once or twice.

modey3

 

[ZapperZ]

You didn't understand what I'm getting at. There are a lot of papers out there that are essentially duplicates of each other in that they use the same methods and determine the same quantities, but just on different systems. That's more engineering work than science. In my opinion, it's the quality not the quantity of the paper that's important. I rather publish 1 paper that's been referenced 50+ times than 5 papers that have only been referenced once or twice.

modey3

Or maybe you just didn't understand the physics well enough to know that these ARE new stuff.

For example, I can use the SAME technique (such as ARPES), and study on the SAME TYPE of materials (high Tc superconductors), and measure the SAME PROPERTIES (single-particle spectral function), and yet, still get published in PRL. Why? Because (i) I'm measuring a particular material that, maybe, is very highly overdoped in that particular family, (ii) I'm doing a different analysis of the same type data that has now yield a new piece of information that wasn't available before, and (iii) that information is actually significant enough because it plugged a hole in a previous understanding.

I'm not saying that there aren't any papers that essentially make minor modification and get get itself published in some other obscure journal. However, as someone who reads most of the "standard" physics journals regularly, and who also happen to referee a lot of papers, trying to recycle something that has been done before is really NOT that common, and if I see one, it typically doesn't get pass me for publication.

Zz.

 

[Modey3]

Or maybe you just didn't understand the physics well enough to know that these ARE new stuff.

For example, I can use the SAME technique (such as ARPES), and study on the SAME TYPE of materials (high Tc superconductors), and measure the SAME PROPERTIES (single-particle spectral function), and yet, still get published in PRL. Why? Because (i) I'm measuring a particular material that, maybe, is very highly overdoped in that particular family, (ii) I'm doing a different analysis of the same type data that has now yield a new piece of information that wasn't available before, and (iii) that information is actually significant enough because it plugged a hole in a previous understanding.

I'm not saying that there aren't any papers that essentially make minor modification and get get itself published in some other obscure journal. However, as someone who reads most of the "standard" physics journals regularly, and who also happen to referee a lot of papers, trying to recycle something that has been done before is really NOT that common, and if I see one, it typically doesn't get pass me for publication.

Zz.

Zap,

If you try to show how doping a superconductor changes the spectral function then yes it's significant because therein lies very important physics. It's important because you are showing trends in behavior with doping. I'm referring to papers, which are essentially perturbation of other peoples work. Let's say you use ARPES to map out the band structure of Al, and hypothetically suppose that this hasn't been done for that particular element. Does this work really contain new physics, especially if this technique has been previously applied to other elements? Also, I read a lot of publications regarding the application of DFT modeling. There are many papers being published that make materials-science journals, which use the same software package, determine the same quantities, but are done on slightly different systems. For instance, there are separate papers that look at the adhesion of Fe|Al , Fe|Cr, and Fe|Ni interfaces. How are these individual studies different besides using different pseudopotentials (which they probably didn't make themselves) and slightly different geometries?

modey3

 

[twofish-quant]

Also you can get a lot of papers written in astronomy/astrophysics because a lot of stuff turns out to involve searches of some sort. I took my telescope and I looked at this cluster and I saw X. I took my telescope and I looked at this other cluster and say Y. I ran my computer code with these parameters and got result X. I ran my computer code with these parameters and got result Y. You tend to get a lot of these sorts of papers because there is a lot of collective thinking as to what is going on.

One reason these sorts of papers are common is that you can sort of project plan them, i.e. I'm writing my computer code, in a year I'll have it ready and then I run it with parameters X, Y, and Z to see what happens. Note that in a lot of situations making a computer run takes over a month to do the run and process the data.

The problem with trying to do something totally brilliant is that creativity and breakthoughs are hard to project manage. What you really want is a paper saying, I ran my computer program with parameters X, expecting to see something like Y, but OMG, I saw Z. But that involves a huge amount of luck with some skill involved in realizing that lightning has struck.

Modey3: In my opinion, it's the quality not the quantity of the paper that's important. I rather publish 1 paper that's been referenced 50+ times than 5 papers that have only been referenced once or twice.

So would I. Unfortunately, it's really up to Mother Nature to see that this happens.

 

[twofish-quant]

Also in computational astrophysics, it's critically important for people to publish papers saying that I ran my code with the same parameters as this other group and I got the exactly same results. The problem is that the code is so incredibly complex and bug-prone, that people want to see several independent verifications of the same results before anyone will believe anything.

Also if you run two pieces of computer code and you get exactly the same results (or even in some situations similar results), then people will think that is somewhat unusual and worth a journal article.

 

[Andy Resnick]

You didn't understand what I'm getting at. There are a lot of papers out there that are essentially duplicates of each other in that they use the same methods and determine the same quantities, but just on different systems. That's more engineering work than science. In my opinion, it's the quality not the quantity of the paper that's important. I rather publish 1 paper that's been referenced 50+ times than 5 papers that have only been referenced once or twice.

modey3

I understand exactly what you are getting at- to be honest, I have the same opinion as you do. Nonetheless, when it comes to getting a job/promotion/grants/tenure, quantity often matters more than quality- after all, how can a non-expert judge the quality of your work?

 

[cosmogirl]

It seems that the numbers I wrote were rather provoking.
I have to admit the number 9 was incorrect, the correct upper limit is 8. Sorry for the mistake.
Here's a (non)-repersentative sample of our recent graduates (not all papers as first author):
The person who I thought had published 9 papers actually has only 8 published.
Another two have 5 published
There're also numbers like 6, 4 and even 2.

Not all of these papers are ground-breaking, but they're all published in respectable journals in the field. I should add that it's considered normal in my department to publish 1 paper after masters, and 3 during PhD, as first author, although there're certainly people who publish less than that. Also, although PhD should be finished in 4 years, it often takes even longer than that.

I think the typical paper-rate depends on the field, so the original question was in fact too vague.

I think (correct me if I'm wrong) there is sometimes a tendency to plan the PhD research is such a way, as to produce at least some papers. I'll agree this is certainly not the optimal way to do research. Research should tackle the hardest questions, the answers to which might as well be beyond our reach.
On the other hand, I think advisors should think about the future of their students. If a student spends 3-4 years of his life pursuing a dead-end project, what will he be left with? The knowledge that he proved that partucular approach was wrong? He might still get a postdoc position if his advisor is a big name with very strong connections, but what if he's not?

 

[Modey3]

It seems that the numbers I wrote were rather provoking.
I have to admit the number 9 was incorrect, the correct upper limit is 8. Sorry for the mistake.
Here's a (non)-repersentative sample of our recent graduates (not all papers as first author):
The person who I thought had published 9 papers actually has only 8 published.
Another two have 5 published
There're also numbers like 6, 4 and even 2.

Not all of these papers are ground-breaking, but they're all published in respectable journals in the field. I should add that it's considered normal in my department to publish 1 paper after masters, and 3 during PhD, as first author, although there're certainly people who publish less than that. Also, although PhD should be finished in 4 years, it often takes even longer than that.

I think the typical paper-rate depends on the field, so the original question was in fact too vague.

I think (correct me if I'm wrong) there is sometimes a tendency to plan the PhD research is such a way, as to produce at least some papers. I'll agree this is certainly not the optimal way to do research. Research should tackle the hardest questions, the answers to which might as well be beyond our reach.
On the other hand, I think advisors should think about the future of their students. If a student spends 3-4 years of his life pursuing a dead-end project, what will he be left with? The knowledge that he proved that partucular approach was wrong? He might still get a postdoc position if his advisor is a big name with very strong connections, but what if he's not?

There is such a thing as a negative thesis in which you explain why your approach is wrong. There maybe even a publication in it. In fact, I wish there were more negative publications because they would help me with my research. In my opinion, science needs to be equally open to negative results as it is to positive results. This will also reduce the amount of fraud that is occurring nowadays.

modey3

 

[ZapperZ]

There is such a thing as a negative thesis in which you explain why your approach is wrong. There maybe even a publication in it. In fact, I wish there were more negative publications because they would help me with my research. In my opinion, science needs to be equally open to negative results as it is to positive results. This will also reduce the amount of fraud that is occurring nowadays.

modey3

Er.. but there are! Just look at the current back-and-forth papers with regards to the claim of possible detection of dark matter in the lab! That's just one example of so many others! Or what about the claim of dark matter signature from various high-flying detectors? Each claim of something produces another claim that, no, we didn't see that!

You don't see it that often because, and this could be a revelation, most of these things that actually made it into publications are actually VALID! The Teleyarkhan's bubble fusion notwithstanding, the scrutiny given to many such publications are always the first line of "defense" against something that's obviously wrong. So to want to expect more of that is kinda puzzling, because the system in place is to make sure that doesn't happen that often.

Zz.

 

[twofish-quant]

Er.. but there are! Just look at the current back-and-forth papers with regards to the claim of possible detection of dark matter in the lab! That's just one example of so many others! Or what about the claim of dark matter signature from various high-flying detectors? Each claim of something produces another claim that, no, we didn't see that!

I think things can be very field dependent. Also there is an entire set of field of statistics involving understanding publication bias, which is important for things like drug tests. One thing that this means is that writing a "meta-publication" in some fields is non-trivial.

You don't see it that often because, and this could be a revelation, most of these things that actually made it into publications are actually VALID! The Teleyarkhan's bubble fusion notwithstanding, the scrutiny given to many such publications are always the first line of "defense" against something that's obviously wrong.

But even here, there are some interesting publication issues. For example, if you see a paper that sees a certain result from numerical calculations, and you don't get "me too" replication from other groups, then it's likely that there was a bug in the original result. When I was doing my Ph.D. I figured out that one of the papers in the field was likely due to buggy code since I was able to replicate the result by setting my input parameters to certain (obviously wrong) parameters. I think that everyone else in the field figured this out too, but there was no point in writing a paper on it. Also, this is the type of things that conferences are good for, since "buggy results" is something that people talk about over beers.