Sorry the topic is obviously a bit sensitive to me haha. Thanks for the tone, I can tell you'd make a good mentor and manager.

  > If you have a solution to how to do it properly I'm interested.

I have purposed solutions to lots of this stuff actually. But it does require push from others and it’s necessary for those in prominent academic positions push them. I think the issue is that there's a lot of interconnected parts to all of this. I doubt that there's an optimal solution, which in those settings I think flexibility is how one should error on. It gives us the room to adapt more easily to local changes in environment. But there will always be some downside and I think it is easy to focus on those and not compare against the value of the gains.

For blogs:

I think we should just count these as publications. People are just looking at citations anyways (unfortunately). We should at least count these as citable.

There’s a good knock-on effect to this one too. It can encourage replications (the cornerstone of science), tutorials (which is teaching and like a stepping stone towards books. This also helps students write better), and can help us shift towards other publication media in general. Like why do we publish works about video or audio in a format that can’t play video or audio? Its absurd if you ask me. The only reason we do is momentum.

I think it is also important too just be writing. To be a good scientist you must also be a philosopher. It is not just about doing the work, it is about why. The meta-science is just as important as the science itself. I think so is the fun and creativity that we have more room for in blogs (I think there should be more room for this in papers too). Research is just as much of an art as it is a science, and we need to be working the creative muscles just as much as the analytical ones. I also think it helps with the discourse between papers. I mean Andrew Gelman's and Scott Aaronson's blogs are famous for having all of these things. They are valuable to their communities and I think even a broader audience. But as the environment is now, this is disincentivized. I think more people want to do it and are motivated, but it is easy to push off when there is little to no reward (sometimes even punishments, like an advisor or boss saying "spend less time writing and more time researching"). If you're going to "slack off", you might as well do something that is more recovering, right? [0]

For reviewers/review system:

Again, incentive structures. The problem right now is that it’s seen as a chore. One that if you do poorly it doesn’t matter. So my first suggestion is we throw out the conference and journal structure as we know it.

The purpose of these structures was because we didn’t have the ability to trivially distribute works. You hire editors to make sure you don’t publish garbage because it’s expensive to publish and they correct spelling and grammar to make sure it’s providing the most value (communication). There may be conversations to improve the works, but not to down right throw them away. Everyone here is well aligned in goals. They're all on the same team! But none of that happens now. We have a system where it is reviewers vs authors. This should not be an adversarial setting. In fact, a few spelling mistakes are justification to reject a paper now (ask me how I know). The purpose was always about communicating work, not about optimizing for what is the best type of work and most important. Truth be told, no one knows and we can only tell later down the line.

There’s two common critiques I hear with regard to this:

  1) How do we discover work?
  2) How do we ensure integrity of the work?

Well who actually goes to the conference or journal websites to read them? We all use the arxiv versions, which are more up to date. We're also reading preprints, and especially in ML this is the way to keep up to date. I only go to grab the bibtex because the authors only have the arxiv one on their GitHub or webpage (pet peeve of mine). We pretty much discover from other papers, google, peers, twitter, and well you’re a researcher you know. The “getting published” is just a byline in a tweet and a metric for the bureaucrats.

The physicists created arxiv because it was what they were doing already. Which was you publish your draft in your big lab and others read it and critique it and you take that feedback to improve. There's always mean people, but mostly everyone is on the same side here. We just extended who had access to the library, that's all.

Discovery is a problem. But what about integrity and verifiability? I find this a feature, not a bug (and you'll be able to infer how this couples with writing directly to niche peers instead of broader groups). Sometimes if you try to take too much control, you end up getting less control. The truth is that the validity of a work is at least doubly intractable. You can't figure it out just from reading the paper. The only way verification happens is through replication. This cannot be done by reading alone. Works are (often, but not always) falsifiable through review, but not verifiable. The distinction matters.

And I actually think the increased noise here is a good thing. Too many conflate "published" (in a conference or journal) as a mark of credibility. Both outsiders and insiders (other academics). It is too lazy of a metric. Many of our problems arise from such an issue and I'd say that oversimplification of a metric is a corollary to Goodhart's Law. We researchers can all read papers and determine pretty quickly if it is fraudulent or not, at least if it is in our field. But outsiders can't. They can't even tell the credibility of a conference or journal, and there's too many scam ones these days. This creates an extra problem where the science journalists, who are also caught in the ridiculous expectation of generating work in infinitesimal amounts of time, end up writing on works with poor understanding of them (and the context surrounding them). Adding noise here pushes them to reach out to experts which will increase the quality overall, as the expert talking to them will not just filter crap but also be able to point to important nuance, things that a novice would miss. Especially when those things are buried in technical language or equations :)

In addition to this it removes power from the lazy bureaucrats AND harms the frauds. I think it is easy to believe that fraudulent work would flourish under this environment, but I think the opposite. Yes, arxiv has plenty of fraudulent works on it, but they are small in comparison. The frauds go to the fraud journals. Their scheme only works because they are able to have someone give their crap a mark of approval. When there is no seal of approval, one must go ask the experts. It is just a shift in the trust structure, not a destruction of it. There'll be collusion rings, but we already have those (including in high profile venues!). I do suspect there may be a bit more at first, before it stabilizes, as everyone adapts. But I think we already do most of this stuff naturally, so it won't be that hard.

But I do think we should keep conferences. There is value in meeting in person (I also think universities should encourage more inter-departmental collaboration, as well as extra-departmental and extra-university collaboration. I do think it is silly that we aren't having deep conversations with our neighbors). But I think these should be more invitations. You have invited speakers and the rest is you focusing on getting people to talk and facilitate these collaborations. That's one of the primary goal of conferences, building these social networks. Social media helps, but there's a big difference sitting face to face. I also think they should have training sessions (like they do) and workshops should be focused around these, not around publication. So less stuff is "published" in conferences, because publishing is just releasing work!

There's obvious downsides to this and there definitely is a lot of room for refinement, but I think this is a fairly good structure. At the end of the day we need to have faith in one another. The structure of science has always been "trust, but verify." But we stopped doing the latter and pigeonholed our measures. So now we have neither trust nor verification. I think it has all been good intention. I'll save you the clique, but what is a clique if not something everyone can state but few people actually follow? I get the desire to remove all noise, but I think such a goal is fruitless, it is impossible. So instead I think it is about finding the optimal noise. Instead of trying to get rid of it, we should embrace it. I hope as ML researchers we can recognize this, as noise is critical to our works. That without it, it all falls apart. Because, it is a feature, not a bug. It is a necessary attribute for generalization, and I think it isn't just for machines.

[0] Personally I find that the big reason for stress is that we remove creativity, flexibility, and a lot of the humanity from the work. We're over burdened by deadlines, there's too much work to do, and the truth is that the work is difficult to measure. Progress is difficult to see a priori, so you can constantly feel "behind". This just leads to slowdown and burnout. We're all here out of passion (at least I hope so! It'd be insane to do a PhD or research without the passion!). The structure should be made to gently push us back on track for when we get too lost by some other fascinating rabbit hole (who knows if it goes anywhere. But is going down it a bad thing?). But if we can't have fun, we are less effective at our jobs. If we can't take time to think about the meta, we get worse at our jobs. If we aren't talking to our peers, we get worse at our jobs.

A statistical approach to model evaluations