Help get this topic noticed by sharing it on Twitter, Facebook, or email.

Is Challenge Mono Therapy Data Reproducible?

Should mono-therapy data (IC50, H, Einf) in the training and test data be reproducible for a given cell type-compound-max dosage? Most compounds are represented multiple times in the data in the same cell type. For example, in "ch1_train_combination_and_monoTherapy.csv", Cisplatin occurs twice with cell line 647-V (once in combination with AKT, and once with FGFR). Cisplatin's mono therapy values (its activity in 647-V by itself) are thus measured twice:


> drug.train[ drug.train$CELL_LINE == "647-V" & drug.train$COMPOUND_A =="Cisplatin",]
CELL_LINE COMPOUND_A COMPOUND_B MAX_CONC_A MAX_CONC_B IC50_A H_A Einf_A IC50_B H_B Einf_B SYNERGY_SCORE QA COMBINATION_ID
542 647-V Cisplatin FGFR 3 3 1 0 100 1.547591 3.63693 0 1.529579 1 Cisplatin.FGFR
> drug.train[ drug.train$CELL_LINE == "647-V" & drug.train$COMPOUND_B =="Cisplatin",]
CELL_LINE COMPOUND_A COMPOUND_B MAX_CONC_A MAX_CONC_B IC50_A H_A Einf_A IC50_B H_B Einf_B SYNERGY_SCORE QA COMBINATION_ID
536 647-V AKT Cisplatin 3 3 1.185152 2.647892 0 0.439128 1.358245 2.024974 12.96846 1 AKT.Cisplatin


Note that the mono-therapy values for Cisplatin in 647-V are significantly different (Einf is 100 is one case and 2 in the next). We can plot the correlation of these replicates (where cell line, compound, and max dosage are identical) to see how reproducible the mono-therapy results are, and the correlation observed is not inspiring.







Thanks in advance for any light that you can shed on this.

Cheers,
Gene
14 people have
this question
+1
Reply
  • 1
    Hi Gene,

    Thank you for raising an important point. The short answer to your question is "to some extent".

    There are differences between the number starting cells and dosing schedule for the same compound dosed in the same cells, so we do not recommend pooling "replicates" from different combination assays. Identifying monotherapy sensitive cell lines from our data will be challenging giving the low number of replicates and some batch effects, therefore, we don't expect participants to tackle this problem in this challenge.

    I also want to note that the reproducibility of the monotherapy response also depends on the measure of response. We chose to report statistics from a Hill function fit, but not all monotherapy dose-response curves are appropriate for this function. In the case of Cisplatin in V-647, the area under the dose-response curve (AUC) from one assay is 0.84 and 0.73 in the other assay, which is relatively similar. That is not to say AUC is appropriate for every case. We chose to show participants statistics from the Hill function, such as IC50, because they are widely used for clinical decision making. We hope predictive models of synergy will use clinical translatable inputs such as this.

    Your point is well taken by the challenge organizers and we may provide alternative measures of monotherapy response that may be more reproducible. You may also choose to fit the monotherapy dose-response curves in ch1_ch2_monoTherapy.csv and come up with your own estimates of response.

    Best,
    Dennis
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • 1
    Hi Dennis,

    Thanks for the information. If I understand you correctly, differences in the number of starting cells, batch effects, and dosing schedules might all lead to a lack of reproducibility in the three metrics associated with mono-therapy. You suggest that other metrics derived from the same raw data might be more robust.

    Since the synergy score that we are attempting to predict is derived from the same raw data as the mono-therapy metrics and would presumably be subject to the same problems of initial cell count, batch effect, etc., do you have a sense of how reproducible it is? For example, do you have enough experimnetal replicates (not necessarily in the provided training and test data) to calculate an R-squared for the synergy score?

    Thanks,
    Gene
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • 1
    Hi Gene,

    Thanks for following up. There are some differences in the number of starting cells, batch effects and dosing schedules, but we have tried our best to mitigated their effect on the combinations assays. Furthermore, the combination synergy score should be more robust because it is a score that is relative to the monotherapy response in the same assay (see Data Description).

    Unfortunately, we do not have many biological replicates to empirically support reproducibility of the synergy scores. There is however 632 assays performed twice with the same combination and cell line, which show a significant Pearson correlation of ~0.5 and RMSE of 1.48. We believe this amount of variation is reasonably good for in vitro studies of pharmacological response and comparable to the variation seen in the clinic. Also note that some of these “replicates” were dosed at different concentration ranges.

    Best,
    Dennis
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • Thanks Dennis!

    This is very helpful. Knowing the reproducibility of the predicted variable will help us set realistic expectations for our performance on held-out data.

    Cheers,
    Gene
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • Just to follow up on this question:

    So the Pearson correlation between monotherapy ic50 scores of different "replicas" is ~0.25 if we group by drug pair / cell_line / max_conc, and it's basically 0 if we group only by drug pair. This isn't ideal, but I guess it makes sense, given all the things that can affect experimental results in these types of assays.

    What I find very surprising is that the correlation between synergy scores in different cell lines is also basically 0 (~0.03). Maybe I'm not understanding something or did the analysis wrong, but this is really disappointing. I was expecting synergy scores from other cell lines to be one of the strongest features! If other cell lines don't tell us anything about the synergy score, I'm not sure how much we can expect to extract from "first principles" like gene expression data...

    Your feedback would be greatly appreciated,
    Thanks,
    Alexey
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • Nevermind my comment above. After removing drug pairs with a quality assurance score < 1, the correlation is 0.25, on par with what's expected if the maximum we can hope for is 0.5.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • Hi Dennis, just to confirm what you wrote above: "... There is however 632 assays performed twice with the same combination and cell line, which show a significant Pearson correlation of ~0.5 ...". Is that R=0.5 (pretty bad for a fancy experiment with high expectation like this!) or R2=0.5?

    Yudi
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • Hi Yudi,

    We appreciate your concern. In generating our pharmacology data we focused on covering a wide range of cell lines and combinations rather than profiling deeply to generate many replicates, therefore, out of the 632 assays we do not have more than 2 replicates for each combination in the same cell line. With only 2 replicates, we should not read too much into the correlation coefficient. The RMSE of 1.48 shows that the absolute synergy scores are not too different between this handful of replicates which is promising. Furthermore, the assays performed twice were on different dosages of each drug, so we did expect some variability.

    I can say that internally we have performed deeper profiling of some combinations over multiple replicates and the synergy scores are reproducible.

    Best,
    Dennis
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • 1
    Hi Dennis,

    Just making sure we're using the same terminology here. An RMSE of 1.48 for synergy is really fantastic, but shouldn't be possible with r of ~0.5. Unless I'm misunderstanding. Is this RMSE of the Z-scores? Or the absolute synergy scores? Or some other normalization?

    I can't seem to recreate any distribution of errors which can have an RMSE of 1.48 and an r of ~0.5 using a random sample of 632 assays from the training. Any attempt looks something like the following:

    RMSE of 1.48 simulation:
    ==================


    However, I can easily make an error distribution that fits that RMSE if it's interpreted as Z-scores. However, that makes an absolute RMSE of ~43.62. The following is a good illustration of this (r=0.502):

    RMSE of 1.48sigma simulation:
    ======================


    These two interpretations imply very different things, and depending on which is more like the reproduced data, we might attack the problem quite differently.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • 3
    Hi Tyler,

    I checked my code there was a bug, so you are right. The RMSE is actually 32.03 and MAE (mean absolute error) is 23.64, which is more reflective of r of 0.5.

    We have not normalized the synergy values using z-scores, but we are discussing using z-scores in performance evaluation as we do see different variances in synergy for different combinations. This will be decided when we announce the performance metrics.

    Many thanks again for looking deeper into this and finding my bug.

    Dennis
    • view 4 more comments
    • really? the replicates are at different concentrations? then it shouldn't be called repicates.

      i cannot remember whether my end model has concentration or not.

      i do remember i did some experiments on including/excluding concentrations, but i cannot remember the conclusion. that means that parameter did not matter, in my model.
    • after a second thought, i think this problem should be interpreted this way.

      when they designed the experiments, the concentrations were put at a range that are likely to draw a nicely looking curve. so the exact concentration doesn't even matter now.

      otherwise, one can grab some salt and put onto cancer cells, as long as the c is high. they would still die of salt.

      but in general, i think taking a purified cell line and doing experiments is only a very preliminary theoretical stage towards bedside.
      otherwise, why do we even have the neighboring tumor heterogeneity challenge.

      not a doctor, but i think this problem eventually should be targeted from other angles.

      half of the people are living without breast (or for political correctness, half of the people do not have prostate), but why for one has metastatic breast cancer she just have to die? it is not because of the tumor at all. i think it is because how the system reacts to the tumor, provides nutrients to it, making the normal cells no place to live, and shuts it self down.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • Hi Dennis,

    You wrote that you do not have more than two replicates for each combination in the same cell line. However, in the Raw_Data folder there are almost no replicates. Is it possible that we also get the other replicates or did i miss something and the replicates are already available somewhere else?

    Thanks!
    Kristina
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned

  • Dear Kristina,

    unfortunately, all replicates are already released in the raw file folder and there is not more data available. Please note that most "replicates" have adjusted drug concentrations.

    Best wishes,
    Micha
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated indifferent, undecided, unconcerned