Here is a what I am thinking about doing after Lab results are released.
pattern changes that create the most structural enrichment
1. For each category, find design pairs with highest structural enrichment
a. Where structural enrichment between 2 designs = (chg in Lab Score) / (chg in #nt’s)
2. Evaluate structural changes between these designs3. Look for commonalities across both categories and different pairs with max enrichment
After I do step 1, I will share in this forum what pairs show greatest enrichment. Then anyone can comment on what patterns they see or insights that arise.
If your analysis tools support it, I'll suggest an enhancement for how you measure the difference between two designs. Counting the number of changed bases is essentially finding the distance between the sequences using the Hamming distance. This is useful for quantifying base mutations, but doesn't take into account insertions or deletions, which are very significant factors in RNA evolution. The Levenshtein distance (aka "edit" distance) reflects both. That is, just as a single base mutation adds 1 to the distance, so does a single base addition or deletion. As an example, if you have an 8-base paired stem and you shift one of the strands by one base, it will add as much as 9 to the Hamming distance. But it will increase the Levenshtein distance by at most two -- a deletion at one end plus an insertion at the other.
Since a rerun of the puzzles from Round 2 of the OpenTB challenge is the next lab experiment, It would be fantastic if you were to start this off with the Round 2 data!
Now the R104 and R106 AB/C^2 DEC and INC sublabs scored INC in the 80's and DEC in the 70's. Fairly well balanced.
The difference in the rounds was that R105 used different A,B,C & reporter oligo's. I think that shifted the balance to favor the AB/C^2DEC sublab and make the AB/C^2INC sublab not switch.
The DEC sublab has the reporter with the TB-C oligo's. The INC sublab has the reporter with the TB-A and TB-B oligo's.
Looking at the data, in the INC sublab, KD100C should be a higher number than KD100A or B for high scores.
In the R105 labs, AB/C^2INC had a KD100C number lower than KD100A or B. What made KD100C so low?
Knowing how KD100A, B and C are calculated would help me understand what is happening.
In the R105, AB/C^2INC sublab, TB-C folds on itself with 3 base pairs as a stem when not binding.
In the R106, AB/C^2INC sublab, TB-C folds on itself with 5 base pairs as a stem when not binding.
Does self attraction play any part in the KD energies?
Is it the relationship of TB-C to the reporter that affects KD100C? Can the lab shed any light (no pun intended) on this :)
Here are two files with Top 200 "Enhancement Score pairs" for TB Round 2 105 A*B/C^2 Dec and Inc. In these files, I have used the change in eterna lab score between the pair for the numerator in Enhancement Score. A couple people have indicated that some sort of fold change might be more useful - so we can redefine "Enhancement Score" (and keep redefining it as better measures arise).
The denominator in Enhancement Score is the Levenshtein Distance (thank you Omei for this and for some many other helpful comments and corrections), which is a cool measure of similarity between string pairs. Levenshtein Distance is the number of changes needed in one string (either additions, deletions or substitutions) to convert it into the other string.
The purpose of these files is to provide another useful easy way to look at how structural changes between two similar pairs affect Lab results.
There are many ways to group these pairs - which will make looking for structural pattern changes easy.
This analysis is easy to re-run with different variables.
Link to TB Round 2 105 A*B/C^2 Dec
Link to TB Round 2 105 A*B/C^2 Inc
Dec is based on highest Global Fold change Off.
With Inc pairs based on Global Fold change On.
Also Histogram links have been added.
When doing analysis, it is hard for me to get an idea of the statistical significance of a theory if I can't count the number of distinct designs that are there in the results data. To get an idea of the problem, imagine the extreme - imagine that all the submitted designs be a mutation of one single design. Then you have tons of data but any hypothesis you make might be only relevant to that single original design. So you get the impression that your hypothesis works for 100% of the designs, but that 100% is of 1. What I'd like to get is some way of clustering similar designs, so when I run a hypothesis I can see how different design-clusters behave. Also this would give a clearer picture of how many designs we actually produced rather than mutated.
I think I saw once a diagram that tried to depict this (maybe in some video from eternacon?).
I prefer to work on the data from Round 1 as I think there are more unique designs vs mutations (as most people don't know what works and are more keen to go from scratch than mutate an existing one).
Maybe one way is to define a mutation of a design as a design at distance below some constant. Then to discover a cluster, you pick a random design and add its mutation to its cluster and repeat a few times recursively for the members of the cluster. Then repeat the cluster discovery procedure until no designs are left that are not part of a cluster.
Yes clustering analysis should be possible. As I read your comments, I was trying to remember a self clustering method I had come across several years back.
I'm new to this sort of scientific analysis...just really started earlier this month. Omei recommended R Programming (so I had my son teach me that a couple weeks ago), and Omei also explained the Levenshtein distance function (which my son also showed me how to use via R function). Omei has kept me on the tracks and, a couple of times, has gotten me back on the tracks in this project. And Eli always gives me much more than I can fully comprehend.
There are a great variety of R functions available. I will look into clustering functions and we can also PM each other on eterna to see if I can create something that would be more useful for you.
You can have a look at the results here:
The second column called ParentDesignId is the ID of the cluster (and is also the smallest design ID of the designs in that cluster).
I consider mutations of a design to be:
- designs at distance less or equal to 5
- mutations of mutations
I'm open to suggestions for improvements.
Round 105 A*B/C^2 Dec
Notes: all these designs are highly scoring Global Fold Changes off. There are no structural changes in any of these states.
When nt4 = U, as in this first pair, I think I understand why nt1 as a U is better than G. U repels against nt4, rather than being attracted to it...right? And GFC off is 2.59 vs 2.03, improvement of .56
But if that is the case, how do you explain this....
When nt4 = C (with all other nt's being the same as in the previous pair), why is nt1 better as a G than U? With GFC off is 2.81 vs 2.36, improvement of .45.
I would think that a G and C in proximity would be be more destabilizing to the dangle....
FIRST PAIR (where nt4 = U and nt1 is better as U than G)
First pair histograms:
SECOND PAIR (where nt4 = G and nt1 is better as G than U)
Second pair histograms:
Gerry, these are great questions! I’ve looked at many of your pairs, and there are both similarities and differences in how I interpret the results. Trying to organize all my thoughts coherently has not gotten to anything written down, so I’m going to just choose one and talk about it.
Let’s consider the pair AK1.1 vs AK1.2, e.g. IDs 7230668 and 7230678. For convenience, I’ll repeat the links you provided.
Switch Graph URLs:
The sequences are
with the only difference being in base position 1. The mutation from A to U shifted the global fold change from 2.03 to 2.59.
Let’s compare the switch graphs to get a fuller sense of what changed.
The 3D graphs don’t show any obvious overall shift in coloring. Looking at the KD curves, I can see the small increase in the gap between low and high KD curves (highlighted with large red dots. ) But the KD curves are calculated based on values over the full range of reporter concentrations, and this can make it difficult to identify a basic mechanism causing a shift in KD.
Only recently have I come to appreciate the value of looking closely at the FMax values. The FMax values have straightforward interpretation. In principle, they represent the average amount of reporter bound to each design under saturation conditions. Saturation conditions means the binding has reached its limit -- adding even more reporter to the mix won’t increase the amount of reporter bound.
FMax values are scaled so that 1.0 will correspond to the “normal” case of the reporter being fully bound to the design, i.e. where all the reporter bases are paired, with no mismatches or gaps, with the design. But “normal”, is not really well defined. Instead, it is (usually) empirically determined as part of the experiment, using “control” designs that don’t have any known peculiarities. So small deviations (say in the range 0.9 - 1.1) in FMax may not mean much. But variations beyond that can be very meaningful.
In the case of AndrewKae’s A*B/C^2 DEC designs, notice that the FMax values fluctuate around 1.5, rather than 1.0. A consistent value this high almost assuredly means one thing -- there is a second binding site for the reporter. If this second binding site was intentionally designed, the FMax values should be centered around 2.0. An intermediate value of 2.5 suggests that the secondary site was not intentional, and even under saturation conditions, binds only about half as much reporter as a “normal” binding site.
Sliding the reverse reporter sequence along, the secondary binding segment jumps out.
This is definitely a less strong binding, but it is being reinforced by coaxial stacking with oligo C, so high concentrations of oligo C should increase FMax. (The details of the stacking relationship in this case is complicated, but probably enhanced, by the presence of the aromatic fluorophore tacked on at the 5’ end of the reporter.) On the other hand, oligo B is competing for the secondary reporter site, so higher FMax would be expected to be associated with lower concentrations of B. Oligo A concentration seems to have only an indirect effect via it’s cooperation with oligo B. So the expected ordering of concentrations to maximize FMax is [B] ≤ [A] < [C]. The reverse order, [B] ≥ [A] > [C] should lower FMax.
This matches the observed data well. I have marked the outlying FMax values for AK1.1 with red rectangles. The three conditions with high FMax are 4 (B=0, A=5, C=100), 6 (B=5, A=5, C=100) and 10 (B=25, A=100, C=100). The one condition that stands out on the low end is 13 (B=100, A=100, C=0).
So now we have a reasonable explanation for the range of FMax values, but it doesn’t address Gerry’s original question of why mutating base 1 from A to U improves the global fold change, or for that matter, why it affects it at all, given that base 1 doesn’t seem to pair with anything.
Well, a good part of the reason I started by calling attention to the FMax values of around 1.5 instead of 1.0 was to introduce the notion that quite often in the OpenTB puzzles, oligos can will bind in more places than intended. To evaluate what could be happening with that dangling 5’ end, we need to see what oligo might find a secondary binding site there.
The strongest binding at the 5’ end is shown below.
At first glance, the 5-base binding between the design and oligo B appears to be only moderately strong. But closer examination shows that with high reporter concentrations, it becomes supported by coaxial stacking with the reporter. So conditions with high C concentrations should have their FMax boosted.
Now consider what happens when base 1 is mutated to a U:
The oligo C stack is extended all the way to base 1. In Johan’s array experiments, our RNA designs are flanked on both ends by strong double stranded DNA, with the result that the first two RNA bases essentially become a continuation of the DNA stack, and the Tether/Oligo C/Reporter combination become one contiguous helix.
This is a very compelling argument for why the A→U mutation at base 1 has a significant effect. Unfortunately, it gets ambushed by the experimental results. While the strong combined stack does seem to “tighten up” the design in the sense that the curves for the various conditions become more parallel, it incorrectly predicts the effect on FMax. Instead of generally raising the FMax values for AK1.2, it lowers them slightly.
So I have missed something. It probably means I have left out an important aspect of the RNA interaction. Or it could be I simply flubbed the logic of my reasoning. But in any case, this is where the story ends for tonight. :-)
It is more easily understood,
It is more easily measured,
It relates more closely to what a paper-based diagnostic can directly display, and
It appears that there is a straightforward way to control it when designing.
Sounds pretty good, huh?
To illustrate these points, I’ll use a switch graph from Synthesis Round 99 that had only one input and 2 states shown on the graph.
It is more easily understood: FMax, shown in blue, is simply the maximum obtainable fluorescence, regardless of how much reporter oligo is added. Fold Change, shown in green, is a ratio of two reporter concentrations corresponding the the concentrations at which the fluorescence is half for FMax.
It is more easily measured: Fmax can be measured with just one experimental condition, essentially taking the medium value of the right-most column of dots. Fold Change, on the other hand, requires measurements at a whole series of reporter concentrations, which are the 18 columns of red dots. From these 18 conditions, a curve is predicted using a simplifying assumption (which the OpenTB data shows is of questionable apllicability) and using that predicted curve to get an Fold Change value.
It relates more closely to the constraints of a low-cost paper-based diagnostic: The “output” of a paper-based diagnostic is one or more colored dots, where the color indicates how much reporter has bound to the RNA in the sample. The simplest paper based diagnostic could conceivably use just two dots -- one whose color depends on the reporter binding level and a fixed color (control) that the variable color is compared to. If the single measurement condition is in the reporter saturation range, then the color can be very insensitive to reporter concentrations variations caused by manufacturing variation or “shelf aging” over time. On the other hand, a measurement taken in the vicinity of a KD value will be at the point of maximum sensitivity to reporter variation, because that is where the binding curve is rising most rapidly.
It appears that there is a straightforward way to control it when designing: When Johan ran the first array experiments, he observed that not all the designs had the same FMax. The obvious explanation for this was that the design “mis-folded” under reporter saturation conditions. This, in turn, was based on the desire to make binary switches that could be turned completely ON or completely OFF, depending on whether the reporter bound or didn’t bind. So a switch that only partially bound the reporter, even under saturation conditions, was outside the scope of designs of interest. Hence, he introduced the “folding score” component of the Eterna score, which penalized designs that had an FMax lower than 1.0. In turn, I suspect most players (including me) tended to dismiss designs with low FMAX scores in the ON state as being “defective” in some way.
Nevertheless, some designs that had quite good fold changes did have low FMax scores. In the design above, ON and OFF states have essentially the same FMax value, but it is much closer to .5 than to 1.0. Furthermore, some designs had completely different FMax values for different concentrations, as seen here:
Not only has Oligo B shifted KD significantly, it has lowered FMax by about two thirds.
Why do I say it appears to be straightforward way to control FMax? There are still plenty of details to be filled in -- which is why I am posting this here, to encourage others to help analyze the past data -- but here are things I am certain of:
Large scale adjustments to FMax can be achieved by having multiple reporter binding sites.
Finer control (either increasing or decreasing) of each binding site can be achieved with a combination of varying the number of complementary bases at the reporter binding site and varying the degree the reporter binding is supporter by contiguous stacks on either end, ones that are created (or not) by the binding of an input oligo. Used together, these can make up to at least a 50% change (either higher or lower) in the maximum luminance of a single reporter binding site.
I will follow this post up with more specifics for what past experiments say about controlling FMax, but not tonight.
- Most of them don't get any switch score - there are little difference between KDON and KDOFF
- Most of them don't get much baseline score - meaning the reporter has a hard time binding
- The few designs that do get a switch score all have one thing in common. A strand pairing up with the reporter complement, when it needs to be turned off.
- So making a switching hairpin out of the reporter complement and a nearby input complement may be of help.
We have been discussing in this forum post why the ABC2INC design has done bad in Round 2. I took yet a look at the data and noticed something I wish to share. I think Whbob may have said some of this before. So my apologies in advance if I repeat.
General data trends
There are no scores above 60%. So far, so bad. However when I look at what the score consists of, a few things stands out in particular. It is to a large degree the switch score that is missing.
Johan explains the switch score like this:
The Switch score is based on the KD of the ON-states and the OFF-states.
In other words, the switch score is the difference in KD between the on and off states. So there isn't a huge difference between ON and OFF.
Also a lot of the designs dont get a good baseline score. This was also a problem for the round 1 and 3 ABC2INC designs.
Johan explains Baseline score like this:
The Baseline subscore is based on the KD of the ON-state. The score is 100 for for KD < 10 nM, 0 for KD >30 nM, and decreases linearly in between.
In other words, we get score for baseline if the ON state is at 10 nm KD and below and get part of the score up to 30 nanomolar. If the reporter need to be present at higher concentration than that, we get 0 baseline score.
What characterizes the designs that get some switch score?
I pulled a sort by switch score
The designs with switch subscore above 0, all have a strand pairing up with the reporter complement, when the later needs to be turned off.
Here are an example from JR's design
The marked bases are the reporter complement.
So what may help this round for the ABC2INC lab is to use switching hairpins made up of the reporter complement and an input complement. As Andrew did in his RIRI sensor design. He is also doing it in a lot of his ABC2INC solves.
I've been working on something that might help with the ABCINC labs (...and other labs if I did the same for them). I noticed in reviewing titles that most designs are just modifications of other designs...and this led me to want to find out two things:
1) Which designs are original unique designs, and
2) Which designs are being modded the most...too much/too little.
from this data I hoped to be able to find out which approaches to designing are being used and which SHould be used. By isolating unique designs, and by grouping sets of designs with their mods, this can allow for better review of designs via wuami analysis as a run of only a few sequences from a mod set should give a representative view of that set as a whole.
to assist with all of this I created a word Doc of all ABC-INC sequences from R4 (up to a week or two back) and (following extremely tedious formatting to isolate designs)I have annotated the designs in such a way as to group Mod-Sets together with the original design using similar formatting changes.
Hopefully the link works:
I had also started to run sequences through both wuami charts as well as through the Lab itself. Sequences in Bold (mostly at the bottom of the page) indicate designs I ran through the wuami page. If the design meets both criteria for the Lab AND look good in the wuami chart they are in Arial Black font (nice and bold), if they work in the Lab, but they fail the wuami chart they are in Algerian font, and if the design failed to meet the citeria of the Lab in the first place I put a strike-through over it. I admit that the colors I used are not the best, but I was running out of unique formatting options. For instance your Monster Hairpin design and mods of it are highlighted in deep royal blue.
As the hard lab puzzles for me see to be bugged, I am trying to contribute however I can. This is still a work in progress, but at the end of the day it should give a flowchart which shows: the history of design creation, which designs have/have not been modded and the extent to which designs have been modded, as well as exposing any potential player bias which may be misplaced in regards to which designs get the most attention and which designs deserve said attention.