Area to submit design for discussion?

  • 3
  • Idea
  • Updated 2 years ago
  • (Edited)
I sent Eli a Lab design that I was excited about.  He returned very useful comments.  I think an area where people could submit Lab designs they think are interesting and then receive comments from others would:

1.  Be very helpful for Lab participants
2.  Be a good way to all get on same page with pattern descriptions
3.  Helpful for all to see different ways people are looking at lab patterns
4.  Be a way for group to bring out what they thought was most interesting.
Photo of Gerry Smith

Gerry Smith

  • 63 Posts
  • 30 Reply Likes

Posted 2 years ago

  • 3
Photo of Omei Turnbull

Omei Turnbull, Player Developer

  • 968 Posts
  • 304 Reply Likes
Gerry, I would love to participate in this kind of discussion.  Currently, I think here in the forums is the best venue we have.

Having said that, I'll be travelling to Eternacon and beyond for the next 5 days, and my participation will probably be limited by circumstances during that period.
Photo of Gerry Smith

Gerry Smith

  • 63 Posts
  • 30 Reply Likes
COMPARISON #1

Below are two designs in State 1 from TEP 1a Exclusion to comment on.  

The first is one by Omei with the MS2 on the far right with a fixed stem in the middle.

I like this design because the large fixed stem makes the movement to State 2 very efficient and the MS2 dissipates completely and is right next to the end (which I think Eli likes...and whatever he likes so do I!).  You can use the link below to see.

http://www.eternagame.org/game/browse/8047744/?filter1=Id&filter1_arg1=8324282&filter1_arg2=8324282




The second design is one of mine with the fixed stem at the far left and the MS2 right next to that.
The MS2 dissipates completely but only connects with one side of  the molecule. - while Omei's splits in two and connects to both sides of the molecule.

http://www.eternagame.org/game/browse/8047744/?filter1=Id&filter1_arg1=8333732&filter1_arg2=8333732

Please share your views on how would you compare these two designs? 
And what is your level of confidence in your opinions?


Photo of Omei Turnbull

Omei Turnbull, Player Developer

  • 968 Posts
  • 304 Reply Likes
The second thing I like to do when evaluating a design (but to be honest, I rarely do when submitting designs) is to look closely at the dot plot.  In contrast to an MFE folding, which shows the single folding that the energy model predicts the RNA will be in more often than any other, the dot plot gives a (higher level) picture of all the various foldings that the RNA is transitioning between.



Here's the state 1 dot plot for my design.  The dark dots in the cyan circle indicate that the MS2 hairpin is expected to form most of the time.  The area inside the two blue and two brown rectangles contain pairings that will be competing with the formation of the hairpin.  There is one stem that is competing against the hairpin, which I have marked in green.  Checking the coordinates, I see that this is a stem that I want to form in state 2.  A thermodynamically optimal switch would never have this alternate stem forming in state 1, so the folding engine is predicting that this will not be a thermodynamically perfect design.  On the other hand, I know from experience that Eterna's grey-scale rendering of the dot plot makes low-probabilty pairings look more significant than the actual numbers suggest, so I don't judge this to be a significant defect.

Here's the dot plot for your design.

Your design has more competition for the hairpin.  Again, they are not high-probability competition, but I would still judge my design to be a little more promising on this criterion.

I should mention that my experience (and I believe jandersonlee, who has also studied partition functions closely, concurs) is that the dot plot is not a very powerful tool for identifying really good switch designs.  But it is significantly better at highlighting "defects" that make for mediocre switches.
Photo of Omei Turnbull

Omei Turnbull, Player Developer

  • 968 Posts
  • 304 Reply Likes
Finally, I'll make one more comment on your design.



Here, I have marked 5 bases that are unpaired in both states.  At least one of these bases has to be there in state 1, to achieve the minimum sized 3-loop.  But the other four bases don't have any obvious role.  My experience suggests that in general, bases that don't have a specific purpose are best tied up in a static stem rather than left "flapping around", where they are more likely to cause unwanted effects than wanted effects.

However, given that of all RNA structures, the energy associated with multiloops is probably the least well understood, I think it is great that you are submitting lots of mutations for these bases.  There's a good chance that there will be one or more significant outliers that will be very interesting to look at in more detail. 

Here's to science exploration!
Photo of Gerry Smith

Gerry Smith

  • 63 Posts
  • 30 Reply Likes
I am blown away by the helpfulness of your comments.  Wow...

Our Lab entries and energy can benefit so much from going through this prism.

And with the mutation and shifter tools you have created, this work is so much more accessible and fun.  Thanks so much Omei.
Photo of Omei Turnbull

Omei Turnbull, Player Developer

  • 968 Posts
  • 304 Reply Likes
Thank you, Gerry.  But I want to stress that I am as puzzled about the nuances of RNA as anyone else here.  These are just my current thoughts on issues that are far from settled.  I hope that in a year or two I can look back and marvel at how naive I was when I wrote the above.

What are your thoughts on these two designs?
Photo of Gerry Smith

Gerry Smith

  • 63 Posts
  • 30 Reply Likes

After 7 months in eterna, my design observations are general and disparate.

 Almost all my initial Lab submissions relied more on randomness than thoughtful pattern application.  Watching the RNA change states and seeing the near infinite number of possibilities coupled with having no framework to create or test left me feeling clueless to say the least.

 But that’s how I also felt when I first started single state puzzles.  And even though Labs embody so much more uncertainty than puzzles, there always seem to be findable patterns in life.  And having felt such support to develop inquiry within eterna, I searched and found your Lab pattern discussions with Eli.  So enheartening but so humbling too.

 There were also other sources of that humility.  I had spent over a year getting the top results in Quantum Moves, another citizen science game.  After mentioning it to Eli, he played it for a few days.  Not only did he pass my scores on a number of games within those few days, but the way he was able to see and express the patterns was astounding to me.

 Eterna’s design and scoring is not aligned with its research goals.  So attention/energy has bifurcated.  Yet the community’s underlying research desire creates so much resilience.  

 This all connects to a passage from a course I’m taking on seeing as an artist called “The Right Eyes: Rilke on Painting” by Lena Levin.  Rilke writes to his wife about a Cezannne painting saying:

 “Everything that was rearranges itself, lines up in formation, as if someone were standing there giving orders; and whatever is present is utterly and urgently present, as if prostrate on its knees and praying for you ...”   Rainer Maria Rilke to Clara Rilke

Here “praying” means hoping strongly for a particular outcome.  Doesn’t that sound just like what CRISPR wants for eterna?

Gerry
Photo of Gerry Smith

Gerry Smith

  • 63 Posts
  • 30 Reply Likes
Evaluation #1

Open for Comment:  Below is another design from same Tep 1b Exclusion.

I chose it because it's different and I don't know why it wouldn't be a good design (which reflects my lack of previous Lab analysis).

1.  The MS2 is crowded onto the aptamer gate  (I think I'm phrasing that right....).  Is this a no-no or an efficient design?...

2  The MS2 splits completely in state 2.

3.  Design has a large fixed stem 48-76.

4.  has few dangling NT's.

5.  Seems to move between state efficiently.

thanks Gerry

http://www.eternagame.org/game/browse/8047745/?filter1=Id&filter1_arg1=8355609&filter1_arg2=8355609


(Edited)
Photo of Eli Fisker

Eli Fisker

  • 2223 Posts
  • 484 Reply Likes
Hi Gerry!


I love your initiative with putting up a specific lab design analysis space up in the forum.


1) You say that "The MS2 is crowded onto the aptamer gate"

This is generally a very successful approach for Exclusion labs. So that shouldn't be a problem.


Another thing I notice in your design is magnet segments. Islands of G's and C's. And several of them. These are often present in good switches. They help the design swap securely between states.

The weaker bases like A and U help things glide. As a note of oddity, there generally seems to be lot higher percentage of U's in designs that uses the full switching area. Full moving switches without static stems. In particularly if the designs are not in s a switch bubble and don't have a neck. (A closing stem).

Magnet segments highlighted


There are two sets in state 1 and 3 sets in state 2.

Also one of the magnetsets - the one with 3 C's is doing an overlap. Not exactly matching up between the states. This is neither good or bad, just interesting.

Not each magnet set has a partner.

Normally I would say that having a small stem forming in front of the MS2 is less effective. One shouldn't make it too strong.

But what you have there is short and the bases are useful to the rest of the puzzle.

Also I have seen working designs that has such a stem. I usually call it a MS2 Gate - because it is there when the MS2 hairpin forms and is ON.

So please go on trying out this approach.

Just like the aptamer gate is the stem forming infront of the aptamer when it is ON.

I haven't adressed everything. Just taken what stands out to me.

I have crossed fingers for your design.


Design Proposal

A way to test if this design type has a way of working is to add variation in. Besides mutating it.

One way to do this is to swap in alternative MS2's for the one you already have.

Note that will take changing a bit in the surroundings of the MS2 too. Take note which bases the MS2 pairs with and which of them are changing in the alternative MS2. Then change the bases to match the alternative MS2.

Here is an example with an alternative MS2 in your design:

GACGCAUAAAGAUGAGACGCGUUUGAGAGCUAGACCCUUGGCAGCACGUACAUUGUGUCGAUAAGACACGAUGUACGUGGUAGGUGCUACAUGAGCAUCAGCCAUGUGAUACCAGUCUAGCAAGUUCAAAUAAGGCUAGUCCGUUAUCAACUUGAAAAAGUGGCACCGAGUCGGUGCUUUUU


Afterthought


Extra General question: I wonder if we can make Exclusion switches that work but have their MS2 somewhere not entirely close to the aptamer? I am particularly curious to see if this is possible now we have gotten labs that are bigger in size.
(Edited)
Photo of Omei Turnbull

Omei Turnbull, Player Developer

  • 968 Posts
  • 304 Reply Likes
For me two things stand out as being interesting about this design.

1) As you called out, the MS2 overlaps with the base that hold the aptamer together. Looking just at the secondary structure of the RNA, this makes perfect sense for an exclusion puzzle. But in the experiment for state 2, there will also be the large MS2 protein and the small FMN molecule trying to bind to the RNA.  Potentially, they could be competing for the same volume in three dimensions, reinforcing the exclusion.  But it is also possible that there could be a very nice fit among the three of them, to the extent that this design actually acts as a Same State design., i.e., it's fold change will be reported as being less than 1.  This is the kind of question that makes lab analysis fun!

2) The triloop configuration, which you didn't mention, is also interesting.  Adjacent helixes tend to align along the same axis, essentially forming one long helix.  This results in a lower free energy, which lowers the experimental KD value, which in turn should increase the fold change of an Exclusion puzzle. But here, we have three possible stacking alignments, only one of which can be forming at any time.  What is the net result?  I don't know.  Maybe this is the round where we'll get enough examples, along with relevant mutations, that we can figure that out.
Photo of Gerry Smith

Gerry Smith

  • 63 Posts
  • 30 Reply Likes
Comparison # 2

Below are 2 designs from TEP 2 Exclusion.  The 2nd is just a modification of the first where I tried to reduce the size of the inner loop.  So my main question is "does reducing the inner loop help?"

I've circled the same fixed stems in each design and the MS2 is highlighted (although hard to see in state2).  In Design #1, the longer fixed stem has 1 additional pair than Design #2.


Design #1

http://www.eternagame.org/game/browse/8047746/?filter1=Id&filter1_arg1=8260471&filter1_arg2=8260471



Design #2

http://www.eternagame.org/game/browse/8047746/?filter1=Id&filter1_arg1=8381368&filter1_arg2=8381368




In State 1, Design#1's inner loop has 21 unpaired nt's.  Design#2 has 13.

In State 2, Design#1's inner loop has 14 unpaired nt's.  Design#2 has only 4.

Does the extra pairing in Design#2 help? 
Or does crowding in the smaller loop in design#2 create instability?

thanks,

Gerry 
Photo of JR

JR

  • 241 Posts
  • 20 Reply Likes
A different question that might be asked with the above design is: Does invariant stems help or hinder a designs success? 
Photo of Zama

Zama

  • 37 Posts
  • 9 Reply Likes
What do you mean by invariant stems?
Photo of whbob

whbob

  • 190 Posts
  • 57 Reply Likes
I think it means a static stem that stays the same in all states.
Photo of whbob

whbob

  • 190 Posts
  • 57 Reply Likes
when there is a multiloop, I have used a static stem to reduce the size of the loop. I think it will shorten the distance from an MS2 to an aptamer, making it easier to attract each other.  Just a hypothesis :)
Photo of Gerry Smith

Gerry Smith

  • 63 Posts
  • 30 Reply Likes
I had not thought of that.  Very helpful thesis.  Thank you.
Photo of spvincent

spvincent

  • 49 Posts
  • 9 Reply Likes
Vienna and Vienna2 treat those two structures somewhat differently: it's hard to know which structure is 'correct'; or even if the notion of correctness really applies given how RNA flops around. I suggest submitting solutions and let nature be the judge.
Photo of Zama

Zama

  • 37 Posts
  • 9 Reply Likes
whbob, I like your explanation. It's like a static stem works as a seam in a pair of oversized pants. But if that's the case, are the bases in the stem even needed at all; in real life could they just be deleted? Which brings me to the likely naïve question of how or who decides the starting number of bases in a loop?
Photo of whbob

whbob

  • 190 Posts
  • 57 Reply Likes
Zama, the labs define the number of bases in the design sequence.  Our current size is 85 bases. I suspect it might be a requirement of the size or type of chip they use. Sometimes our designs don't need all of the bases and we tie up ends with hairpins etc. When we have two sequence strings in fixed locations with a lot of separation, I sometimes like to take in the length to see what happens. 
Photo of Gerry Smith

Gerry Smith

  • 63 Posts
  • 30 Reply Likes
In our current TB Lab A*B/C^2 Dec, I have modified one of whbob's designs just changing nt7 from a G to an A.  It makes no structural changes to his design.

In TB Round 2, this one nt change, G to A, when making no structural changes increased the eterna Lab Score by 37 points (nt 12 from G to A).  See below for the comparison of the two designs.

https://docs.google.com/spreadsheets/d/1uYDFjPkH98acdLSGrXBrxCoA9kzq9pOgdAihWIH5XSs/edit?usp=sharing

Why wouldn't this be a good change for this design in this TB Round? (nt7 from G to A)
http://www.eternagame.org/game/solution/8489874/8520084/copyview/
(Edited)
Photo of Eli Fisker

Eli Fisker

  • 2223 Posts
  • 484 Reply Likes
On the design links that Gerry shared above


I have highlighted the 4 concentrations for the 4 states. Green - reporter ON - Red - reporter OFF.


Design with the G12 mutantion (Worse) - Mutation to the reporter complement

https://s3.amazonaws.com/eterna/labs/histograms_R105/7242063.png 


Notice that the KD for concentration 8 for the OFF state is very similar to the highest concentration for the ON states. If just calculating simple fold change, the fold change gets calcucated as the difference between the ON and OFF state.

Fold change calculation 24/29 = 0.83

Concentration 8 (for the 4th state) divided by the highest KD (worst) for the three first states.

There is very little difference in shift between the ON and OFF state.


Design with the A12 mutation (Better) - Reporter complement unmutated


https://s3.amazonaws.com/eterna/labs/histograms_R105/7241305.png


Fold change calculation 8.7/5.4 = 1.6

Concentration 8 (for 4th state) divided by the highest KDON (worst) for the three first states.


Reporter mutation pattern

I can't say weather the data is good or not. Just that these two design pairs follow what I have come to expect from what I have seen so far in your spreadsheet. 

The reporter is not at all happy about getting its binding complement mutated. All mutations to the main part of the complement have done bad. Only a few at the ends that increased reporter binding, were among the better of the mutants. 

When we tamper with the reporter binding site we affect every measurement. 

The design with the worse score and the G mutation is has higher KD values for the 4 star marked values in the switch graph. Meaning it takes longer attaching.
Photo of Eli Fisker

Eli Fisker

  • 2223 Posts
  • 484 Reply Likes
@Omei, on what you said here:

"BTW, if you are wondering why I say a constant difference in predicted MFE energies suggests a common ratio between experimental KD values, it is because KD grows exponentially as a function of energy.  Log(KD) values actually measure energy, and they could be presented in the same kcal units that the game displays."

Could you give me a worked example of a couple of lab designs, where you think knowing MFE and Log(KD) values would be real helpful? I will like to see and understand.

I haven't forgotten that you some time back thought that seeing the switch graphs in a 3D view would be pretty helpful. Which it in deed turned out to be.
(Edited)
Photo of Eli Fisker

Eli Fisker

  • 2223 Posts
  • 484 Reply Likes
The switch graphs that Gerry shared above are for two designs that have only one base different. So they are quite similar, but their switch graphs and scores are quite different.

And it just hit me while I was looking at the switch graphs and just habitually swapping back and forth.

That this would be an excellent way for getting a feel for what is good and bad in the smaller graphs. Since these are for practically the same design. And one can literally see the switch. I think this can be a way to learn to read them better. 

https://s3.amazonaws.com/eterna/labs/histograms_R105/7242063.png

https://s3.amazonaws.com/eterna/labs/histograms_R105/7241305.png
Photo of Omei Turnbull

Omei Turnbull, Player Developer

  • 968 Posts
  • 304 Reply Likes
OK. RNA folding can be subtle, and I can't always find a reasonable explanation for the lab results.  But in this case, I think I can.

First, let's compare the overall results between the two designs.

I've chosen to focus on the "3D" chart because it highlights the fact that the largest difference between the two designs is the second design is the overall more blue.  There are small differences in how each KD measurement changes, but they are secondary to the observation that all the KD values are generally higher for the second design.

What this means is that for some reason, the reporter did not bind as well to the second design, in all of the measured states.  Given the very small predicted change (0.1kcal) in MFE energy created by the mutation, this seems suspicious.

However, we always have to keep in mind that in reality (including, but not limited to, the lab), RNA does not adopt a single folding. Individual molecules are continually changing between foldings, while the MFE folding only show the (estimate) of the most common folding.  What the lab really measures is an average over the entire ensemble of possible foldings, weight by the percentage of time being spent in each.

To see something of what the energy models predict about the ensemble, we need to look at the dot plots.

The dot plot in the game only describes the state in which no inputs or reporter are present.  NUPACK can generate dot plots (or "partition function" displays) for all the states, but you need to use their web server (or download their code) for that.

What the dot plot comparison shows is that while the single mutation made little change in the MFE folding energy, it had a significantly larger change in the ensemble of foldings.  In the second design, a folding which is essentially one long hairpin with multiple interior loops and bulges has low enough energy that it shows up on the dot plot.  Since this "native" hairpin involves the entire sequence, it is going to be working against any of the oligos, including the report, from binding.  Hence the KDs are raised for all states.

If we wanted to take it farther, we could get individual dot plots for each state, and probably get a pretty good prediction or which states would be more or less affected. But from my perspective, this seems to be a design that NUPACK models quite well, and so we have little new to learn from the lab results.  The designs to spend more time puzzling over are the ones where the in silico predictions and the lab results are in conflict.
Photo of Eli Fisker

Eli Fisker

  • 2223 Posts
  • 484 Reply Likes
I have been asking Omei more questions about the MFE idea. Here comes his answers.

Eli: I'm still not sure I understand why the MFE is important. You mentioned it in relation with two similar designs and a MFE difference between them. So is it only useful between closely related designs. Or also for more distant ones?

Omei: Given several (unrealistic) simplifying assumptions, log( FoldChange) in the lab would be perfectly predicted by the difference between MFE values in the game.  This is what we are relying on when we use the folding engines to get us into the ballpark when it comes to predicting whether our designs fold into shapes appropriate to each state.

The fact that the specific mutation didn't appear to have much effect on the energetics of the minimum energy foldings just made it  easier for me come to the conclusion that something else was going on in the ensemble, which led me to look at the dot plots.
Photo of Eli Fisker

Eli Fisker

  • 2223 Posts
  • 485 Reply Likes
Regarding Gerry's DEC spreadsheet


There is another thing I find particularly worth noticing so far.

The two most extreme jumps in score (case 1 and 2) are related to one particular thing. Namely reduction in length of a real long switching stem.

Switching stems that get real long tends to get grumpy about switching. In these two pair of designs, changing one base resulted in the long switching stem, getting shorter and score dramatically increase.


Case 1, score difference 74.36%, NT56 is U vs A

Better, 10 basepair switching stem


http://www.eternagame.org/game/solution/7113332/7238520/copyview/

Worse, 12 basepair switching stem


http://www.eternagame.org/game/solution/7113332/7238516/copyview/


Case 2, score difference 56.72%, NT56 is U vs A


http://www.eternagame.org/game/solution/7113332/7238508/copyview/

http://www.eternagame.org/game/solution/7113332/7238514/copyview/
Photo of Eli Fisker

Eli Fisker

  • 2223 Posts
  • 485 Reply Likes
Question on the switch graphs


So far it appears to me that it seems to be better to have the early part of the curve (left) in the switch curves than a more slow cap curve.

However no it is I who has some questions about two switch graphs.

I took notice of a pair of designs in the DEC spreadsheet that Gerry just shared of similar designs versus their difference in global fold (GF) change.

These two switch graphs looks very different. Also very different to the DEC designs with the best GF. Their switch curves raise in the air real fast. The best one has it raising a bit faster. I have no idea what it means. Is it good or bad? And more specifically, why?

Better GF
               
http://s3.amazonaws.com/eterna/labs/histograms_R105/7165562.png
http://www.eternagame.org/game/solution/7113332/7165562/copyview/


Worse GF

          
http://s3.amazonaws.com/eterna/labs/histograms_R105/7165558.png
http://www.eternagame.org/game/solution/7113332/7165558/copyview/

This is a puzzle that depends on input overlaps (competing inputs) and binding to as much of the inputs as possible.

Whereas the main part of the designs I have seen in Gerrys spreadsheet for now attracting big score and GF change are of the entangled kind. Aka Brourd/Nando's puzzles.

Basically I wonder if different types of switch puzzles will have different kind of switch curves.

And especially if we can use this for something. :)


Now there is one thing more. The actual mutation difference between these designs are to a tiny  switching hairp at the tail end. I haven't seen many of these paired designs with this kind of mutation. But I find it interesting as this mutation outside the main design, caused a GF enhancement on 0.77. Even by strengtening the switching hairpin. 

Nb. Another interesting thing is that the design with the worse GF, recieves a better score.
(Edited)
Photo of Eli Fisker

Eli Fisker

  • 2223 Posts
  • 485 Reply Likes
Aha! It dawned on me that the extreme raise in curve has to do with Fmax.

I asked Omei explain it to me earlier.

Omei: Ideally, fmax will always be in the range of 0-1 (assuming only one instance of its binding site), and it represents the proportion of designs that will bind to the reporter when the reporter concentration is extremely high.

If there are 2 reporter complements, designs should be able to get an fmax score of 2.

Eli: So is fmax a measurement of how many reporters are able to bind up with a high concentration of the reporter. (Be it a molecule, hairpin or microRNA.)

Omei: Each reporter molecule adds a bit the light intensity, so the more the brighter.


The designs I showed switch graphs for in the above post, indeed have two reporter binding sites. One of them even comes at ends of the sequence where it has a easier time binding.

(Edited)