Wednesday, 31 December 2014

New Year's fireworks

I have to thank Xi'an for this: as he rightly guesses in his comments to my original post (earlier this year $-$ well, for a few more hours still, at least!), my spam filter has worked a treat and if it weren't for him, I would have missed this incredible spectacle of pre-new Year's eve fireworks by Vincent Granville

In his blog, Granville takes on evil statisticians who constantly talk down on data scientists. He seems to have a particular grudge with Andrew Gelman, who, Granville says, is our "leader and only influencer".

What I found particularly amusing (and I needed this, as I'm still fighting off a nasty flu, so thank you very much, Xi'an!) is the bit when VC boasts about not even reading Gelman's publications, as they "are disseminated to a very small audience in obscure journals that pretty much no mainstream people read".

Now: I know that Google Scholar cannot be taken at face value and there are better ways of measuring how influential one's research or publications are. But you can see Gelman's numbers here. 14,000 citations on one book are sort-of suggesting a little less obscurity than that, I'd say. Sadly, Google Scholar can shed no light on how "mainstream" the people citing him are...

Happy new year!!

Tuesday, 16 December 2014

Lazy(?)

It's nearly the Christmas break and as I was writing the previous post (on our Workshop on cost-effectiveness thresholds), I just noticed the post-counter in the blog archive. While the decrease in the number of posts from 2012 to 2013 was minimal (98 to 90), this year I seem to have been fairly lazy (at posting, that is...) $-$ unless I frantically manage to write about 20 posts from now to the end of the year, of course, which I don't quite see happening...

I guess most of the "missing posts" (ie those I haven't written) are probably those about actually doing some stats $-$ which are fun to do, but also take time. For instance, the other day I found some data on Italian MPs attendance which I thought would be interesting to analyse, but then didn't have the time to actually do it. So, new year's resolution (slightly in advance): try and make some time for this.

By the way: while the one in the picture is not my cat, that's exactly what he's been doing pretty much for the whole day...

NICE and the cost-effectiveness thresholds: Can good intentions compensate for bad practice?

That's the title of the workshop we've held at UCL yesterday (I'd mentioned it in a previous post). I think it went remarkably well (OK $-$ as I'm the organiser, I may be over-enthusiastic, but I really think it was a very good day!). Despite the fact that we've purposely limited advertisement, as we wanted to keep it simple to start with, we had a good turnout (about 30 people!).

All the speakers have agreed to make their talks available (which are here) and we've agreed that I'd try and summarise some of the main discussion points coming out of the presentations, so that we can perhaps move this forward, somehow. Here's my attempt (any inconsistency is obviously due to my poor recollection of the events!):
1. Matt gave a very interesting talk focussing on the international comparison (of 5 main countries); he started by discussing the different approaches at defining the cost-effectiveness threshold (which was also reprised by James in the second talk). Then, he moved to give some evidence of how in general agencies (with the notable exception of NICE) tend to resist a clear definition of what the threshold is. He presented some interesting work that has tried to elicit the underlying value of the threshold by analysing a set of past decisions in the Australian setting. What I found interesting of the Dutch case is the idea that the threshold can be set as a function of the disease severity, which made me think of potential links to my talk on risk aversion (see later). One of the interesting points of his discussion of NICE's case is the idea of a range of thresholds (rather than an absolute value). We had a little less time to discuss France and Japan, which are all interesting cases anyway (France because the formal use of cost-effectiveness methods is newly established; Japan because of the mix of private and public funding as well as the recent move towards formal inclusion of cost-effectiveness considerations, mainly driven by the need of capping health expenditure). Probably, in the end, the main message is to wonder whether strict adherence to a very specific guidance (such as NICE's) is a good idea. As I was listening to the talk, I also thought that in a more expanded version of this, it would be perhaps interesting to look at North America.
2. James's talk concentrated on different ways of determining the threshold, effectively (and quite amusingly) pointing out at the many flaws in virtually all of these (which in itself is evidence of how complex this problem is!). I found the discussion of the "historical precedent" method fascinating and the fact that this is actually used in some countries is kind of strange. Then he gave a very clear account of the "budget exhaustion" method, linking back to some applied examples (eg Netherlands and Ireland), showing the incredible range of variability present in many cases. I found his comments on why it may not be ideal to have a range of thresholds very sensible and while I was listening to the talk I kind of changed my mind about that 2 or 3 times... James also linked to the Claxton et al paper on value based assessment and its potential impact in setting some plausible values. I think this is a very important piece of work that we ought to consider very careful (if we are to discuss the problem in general terms). Finally, I think another very good point that James made is about the idea that in fact, all this procedure is dynamic --- setting a threshold based on current budget and then ranking interventions accordingly may need continuous revisions, which in turn may also lead to modifications of the threshold itself, to add to the complexity of the problem.
3. I gave arguably the most confused talk of the lot. The first part discussed the meaning of PSA and what are the quantities that should be involved (and why so). This, I feel, is something that I have thought about quite a lot and I think I have a clear idea of what the problem is and how it should be tackled. I was trying to lead this back to the main issue of selecting the threshold but I think Chris Jackson's comment that, as statisticians we kind of take that for granted, sums up nicely the fact that it is difficult to embed this aspect in the statistical model. In the second part of my talk, I have tried to (still confusingly) address the problem of risk aversion and its implications to PSA in general and the threshold in particular. I have presented a couple of potential utility functions that could be used to include a risk aversion parameter, together with a brief description of their characteristics. I liked James' comment that while difficult to think about it, it would be possible to elicit reasonable values of the risk aversion parameter, which in turn would make the analysis more applicable. While I was talking (and relating to the previous presentations) I also thought that may be risk-propensity is just as important (eg in cases where a disease is perceived to be so important as to be granted some sort of special treatment). Also, Matt asked the very fundamental question of whether decision-makers should be risk averse, in the first place $-$ I think nobody knew the answer to that one...
4. Finally, Mike presented to us his model addressing the impact of the several assumptions underlying the standard definition of the threshold. The model attempts at transforming this into some more complex object, embedding the ideas of "value" and "information" available to the (different) decision-makers. The model is quite complex, but Mike explained it brilliantly (one of my PhD students came to me after the talk still shaking with the excitement!). The idea is to try and include formally several aspects that characterise the fixed assumptions underlying the definition of the threshold (eg the divisibility of the technologies involved, the level of information available to the decision-makers and, crucially, the fact that the process is driven by sequential decisions made, in general, by a set of actors, rather than the vaguely defined "decision-maker"). Model complexity aside (which I quite liked), the results are also very interesting and show that in fact, the straight line in the cost-effectiveness plane can be thought of as a special case of a more general threshold which can assume very different shapes (including kinks and steps). The very interesting point made by Mike is that if you start accounting for value and information, then you (read: NICE) may find yourself in the situation where it may be cost-effective to replace an existing technology with something that, while non cost-effective in the conventional sense is still a better option. This is kind of intuitive, but the model actually formalises the intuition and produces thresholds that have suitable shapes to accommodate this situation.
As I said, I don't really know where we go from here. But it was a very interesting start to this discussion!

Sunday, 7 December 2014

The smartest guys in the room

I've just finished reading this book, which tells the story of Enron, the American power company who raised to incredible fame and generated ridiculous amounts of money to its shareholders $-$ mostly by cooking its books, through "creative accounting".

I found the book quite interesting, if a times slightly difficult and technical (but then again, I suppose that's the nature of this story, which spans over nearly 2 decades of American Corporate finance) $-$ it took me nearly 2 months to read it all!

Reading the book, you are obviously drawn to physically hate the protagonists and their greed $-$ at times it really feels like Enron (and the likes) are all that is wrong with the world. But equally, I couldn't help but admiring some of their business ideas, which most of the times were way ahead of their competitors.

I had got into the Enron story when I went to see this play a few years ago $-$ I vaguely remember the story but evidently, at the time I hadn't registered it for all its implications (which kind of annoys me, right now). Also, the story got me thinking about how most of our students are still in awe of a career in finance $-$ every time I tell one of my academic tutees about the many other possibilities for a statistician in areas other than that, they all look so surprised that such things even exist...