Discussion on new lab system with 20,000 synthesis per month

  • 5
  • Announcement
  • Updated 7 years ago
As some of you may already know, EteRNA is expecting a groundbreaking change in the very core of the lab. Until now we could only synthesized 16 sequences every 2 weeks. However, Das lab (biology side creators of EteRNA) at Stanford University is developing a major breakthrough which would allow us to synthesize 20,000 sequences every month. This would mean that basically EVERYONE could get at least 20 RNA sequence synthesized each month if 1000 people participate in the lab.

We are planning 3 major updates along with the change



1. Massive RNA lab puzzles



With so many synthesis slots available, it wouldn't be the best idea to use entire 20,000 synthesis for solving 1 lab puzzle at a time as we do now. Instead, we'll ask research labs around the world and YOU to propose your own RNA lab puzzle and use synthesis slots. Every month 20,000 synthesis slots will be distributed to every active lab puzzle. For example if you have a specific shape of RNA you want to try synthesize, you could propose it as a lab puzzle - you'll soon be assigned synthesis slots, and you can submit any sequences you want to get them synthesized.

2. synthesis database



Every synthesis result from every lab puzzle will be publicly available. We'll deploy a database and appropriate interface in which you can search/review experimental results. You'll be able to search synthesis results by sequences, secondary structures, and in many other metrics.

3. Scripting

We'll provide a scripting interface with which you can design your own computer algorithms for solving puzzles, designing good lab submissions, scoring lab submissions ..whatever you want to do. (This is a generalized version of "Strategy Market" in which algorithms were limited to scoring lab submissions.) Scripting itself will be a fairly expert feature since it requires programming skills, but we'll develop a way for players to share scripts so that everyone can benefit from the expert effort.

As we are planning these massive changes to the game, there remain many important questions that are yet to be answered, and we would love to hear our players' opinions.

1. What will a lab puzzle look like?

Now that everyone can submit their own lab puzzle, what will it look like? Should each lab be public so that everyone can participate and submit sequences? or should it be private so that only the puzzle owner can submit a sequence?

If a lab puzzle can be public, how do we pick sequences for synthesis from each lab? Does the owner of the project decide which sequences get synthesized..or do we still want to keep voting interface in each lab?

Can other players review/rate the project? If so, what would be the best way to represent them? Also, can more than 1 players be co-admins of 1 lab project?

How do we know when a lab puzzle is done? Does the puzzle owner decide that?

2. How do we allocate synthesis slots?

Say we have 2,000 active lab puzzles. How do we assign 20,000 synthesis slots to those lab puzzles? We could just evenly distribute 10 slots per each lab - but is there a better way to do it? Is there a way that we allocate more synthesis slots to those who are good and can make more efficient usage out of the slots? If so, is it possible to still give newbies a chance to get synthesis slots?

One idea we had was we could again entirely leave this to the community. If we have 1000 lab members, we could simply give 20 synthesis slots to each person, and everyone can decide how they will 'donate' their synthesis slots to projects. This is as transparent we can get with the resource allocation, but will share same cons with the current voting method.

Another idea is we come up with an resource allocation algorithm that will consider lab puzzle owners' performances, current community rating of each lab projects, number of submissions in each process, etc and automatically decide allocation. The obvious downside is that this process is not transparent, and we have no idea what kind of things the algorithm needs to consider at this point. However, if we tune the algorithm well, it would be perfectly objective allocator which could maxmize the outcome.

3. Lab rewards?

What is the best way to reward players for creating/pariticipating in lab projects? Until now, everyone got rewarded for submitting/voting in the lab, even if the designs they submitted/voted for did not get synthesized. This turned the lab rewards to be more like a weekly income that is not as exciting as it should be. Perhaps the lab rewards should be big/hard to earn such that good lab players can easily climb up the rank and the entire rankboard has more dynamic movements. On the other hand, the lab rewards shouldn't be to hard to earn as new players might get scared away.

These are some of the main issues that the dev team has been actively discussing lately.

Let us know what you think!

EteRNA team
Photo of Jeehyung Lee

Jeehyung Lee, Alum

  • 708 Posts
  • 94 Reply Likes

Posted 7 years ago

  • 5
Photo of Brourd

Brourd

  • 438 Posts
  • 79 Reply Likes
This aspect of EteRNA has proven to be an exciting feature, and I cannot wait for it to be implemented. Now, onto the ideas for improvement.

1. What will a lab puzzle look like?

Whether labs should be private or public.

Answer - Public, however, I understand why some authors may want their design to be private, such as the ability to directly control the designs synthed, because other designs may not fit the criteria of their experiment. This could easily be solved by giving authors more creative control over the project sequences, by allowing locked bases, required pair percentages, not having a certain number of bases in a row, etc. Authors will of course be required to solve their puzzles using whatever criteria they set.

Which designs are chosen from a specific lab?

This one is a bit more difficult to answer, since, hypothetically, players may only be given 20 slots. Now, I would suggest that players be given the ability to prioritize designs for synthesis. Let's say I want my designs for main labs and my experiments to be higher priority, over the designs submitted for another lab. I would simply mark them higher on my priority list, and they would be chosen for synthesis first. If no priorities are set, designs chosen per player will be based on date published.

The solvability of lab puzzles and designs.

There are many secondary structures impossible in the current energy model, and I believe players should be given the chance to test those. This will require players to submit designs that may not work, and I believe whatever feature that does this should be implemented at the same time. This could also be one of the features authors could set for their lab project, whether or not it needs to be solved using the Vienna energy parameters, and how much players can stray from the path. Like other requirements, authors would be required to solve.

Puzzle Rating/Review

The community should be allowed to rate a puzzle, maybe through the old star rating and comments. Still not decided on this one.

Co-Admins

Yes, I believe that players should be able to codesign a project and oversee it, just as I believe players should be allowed to coauthor micropapers with those same players that were a part of creating the project.

When a Lab puzzle is done?

Here, it will be entirely up to the player in charge to decide when a project is done, unless they preset a synthesis score that they are aiming for. Now, if a project goes cold, with no more submissions, and the author is inactive, the developers, or a community member assigned by the developers, should be able to review the project, to determine if synthesized designs have met the criteria set forth by the author. If not, the design could be moved to the front of the projects, or maybe removed altogether. The "cold" period should be a reasonable amount of time, such as six months, that way the author is given a reasonable chance to create designs for it.

Final suggestion for this, limiting the number of projects one player can have open at any one time. Sure, it may be great data for the ensemble algorithm if all the scoring is implemented, however, even if players are able to script sequences for every project, they still would have to sift through all that data in order to improve their own scripts, unless players are going to be given the chance to build their own self-learning algorithm. So, I would suggest that the feature to create projects be locked until 20,000 points, and this will unlock the ability to post one project. Then, as the number of points you gain increases, you will be given the chance to unlock the ability to make more, until you reach a maximum. I don't know what this max would be, maybe 50, or maybe 100, but players have to have earned the ability to make that many projects.

I will consider this the end of post 1, which is probably full of errors and typos.
Photo of Brourd

Brourd

  • 438 Posts
  • 79 Reply Likes
Post 2 - How do we allocate synthesis slots?

My first suggestion for this would be a hybrid of all the options: puzzle allocation, player's choice, and automatic allocation, and it would also be partially be based on the type of lab puzzle. For developer projects and projects specifically chosen by the developers, at least one design from each player should be synthesized, unless they choose for it not to. Then, if the submitted desgns do not exceed or match the number of assigned slots, players will be given the chance to vote on the remaining designs, to see which ones are chosen for synthesis.

All players will also recieve a base amount of slots that they can freely distribute to whatever project they want. For our purposes here, let's say this base amount is 10. Now, the rest of the synth slots will be automatically allocated at the end of the month, after all designs are submitted, and the number allocated will be determined separately. The algorithm that does this will take into account activity, scores, etc. Then, the designs on the top of the player's priority list will take precedence over those near the bottom, and will be chosen for synthesis. So, if a player was allocated 20 extra slots, the top 20 designs on his priority list are chosen for synthesis.

So, sound off in the comments below. This will be considered the end of Post 2
Photo of Brourd

Brourd

  • 438 Posts
  • 79 Reply Likes
Post 3 - Lab rewards

The concept of lab rewards is kind of a touchy topic, as some believe they should not be there, etc. Now, One part of lab rewards needs to be badges. Lots and lots of different badges that players can share, post comments about, be envious over, etc. The reason being that anyone can get them, they just have to participate and try their best to get the badges.

Some things that I would like clarified include what makes a lab reward big/hard. If it is the construction of a perfect 100 design, that may prove to be impossible for those who spend time with projects that are not likely to ever have a perfect synth score. So what else is big/hard, as a clarification of this would be much appreciated.

We'll consider this the end of post 3 for now, although, expect future posts for all these ideas as they pop into my head. Thanks for reading.
Photo of stevetclark

stevetclark

  • 15 Posts
  • 1 Reply Like
You could assign an equal number of synthesis slots to the first 1000 people and assign the remaining synthesis slots based on lab results. For example, assign 10 slots to the first 1000 people and assign the remaining 10,000 slots to players based on how well their previous designs have scored in the lab. Newer players would be able to earn extra slots by doing well in the labs. Lab rewards would be based on how well the designs score. Voting should be eliminated altogether. I believe this will help eteRNA retain newer players.
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
Could work. A lot simpler than my tiered scheme.

In chat discussions, Brourd suggested that lab points be separated from puzzle points. You need 10K puzzle points to get lab status, then start gaining lab points as a separate score from puzzle points, much like Solo versus Evolver points in FoldIT. Perhaps the first 10K slots are allocated evenly to all players who have achieved lab status. But what if players stop exercising their lab rights? It seems a waste to have slots reserved round after round for players who don't use them? Perhaps you need to have 10K points *and* be among the first 1K/2K/5K players to submit designs in a round? Those who snooze lose?

Also like FoldIT we discussed having a scheme that rewards active players as well as successful ones, either by giving lab points a half-life (like in FoldIT) or basing the extra slots on recent activity/success (past N rounds) rather than lifetime score accrual. The concern was that long term players could raise the bar so high that making it into the top 1000 is difficult.

However with any reserved slot system, there may be people who do not exercise their slot options in a given round, and rather than waste them, they could be allocated to "backup" designs submitted by players - but how should they be ranked/ordered and how many would they be allowed to submit?

Proposal:

a) each of the first N=1K(/2K/5K - TBD) lab qualified players to exercise their lab options in each month/round are guaranteed at least 10K/N slots for the month. If fewer than N players submit, then slots are divided equally by the number of players submitting in the round.

b) players may submit synthesis candidates to any active player or development lab in each round

c) players rank their own designs (across all labs) in order of synthesis preference

d) after the standard slots are used, the remaining synthesis slots are allocated to players based on some function of *current* lab rank (which is biased in favor of recent performance)

e) If slots still remain, additional slot assignment rounds proceed based on the relative ranking of the players with remaining designs

So: players choose which of their designs they think are most worthy of synthesis and based on (recent) performance they may get additional slots
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
Brourd wrote: This aspect of EteRNA has proven to be an exciting feature, and I cannot wait for it to be implemented.

Ditto! I think this may help with retention of experienced players if managed well.

1. What will a lab puzzle look like?

Whether labs should be private or public.

Free synthesis slots should be Public. (If there is pay to play, that's another matter.) I also agree with Brourd that authors should have more creative control over the project sequences, by allowing locked bases, required pair percentages, not having a certain number of bases in a row, etc. Authors will of course be required to solve their puzzles using whatever criteria they set.

A loophole/exception may be made for certain projects at the discretion of the project designer. This might be in the nature of maximum number mismatched bases allowed on a candidate/template, normally zero, that would be accepted during submission. This would allow testing of design elements near the limits of the energy model to see how they fold.

Which designs are chosen from a specific lab?

This ties into my answer to part 2 below (How to allocate synthesis slots) and partly depends on how the 20K slots will be spread through the month. More details
later, but basically a system of reserved slots and "like"-influenced lottery slots.

Puzzle Rating/Review

Tied in with Reserved Slots and Likes and Lotteries below. I would suggest that players be limited in the number of active projects that they can have open. The number should be large enough to generate 5000 candidate submissions each week, but not infinite. A project may be deactivated at any time by its owner, or after 4 weeks of no new submissions and no newly-synthesized candidates. (More below.)

I like Brourd's suggestion to limit the number of active projects for players, but I'm not sure about the exact scheme/count. He suggests "the feature to create projects be locked until 20,000 points, and this will unlock the ability to post one project. Then, as the number of points you gain increases, you will be given the chance to unlock the ability to make more, until you reach a maximum." Alternatively, there could be a fixed number of projects allowed past some points/rank limit which not everyone will exercise fully. Slots are assigned to candidates, not projects under my scheme, so the desire is simply to avoid project "pollution" of too many active but "silly" projects, while maintaining enough "serious" projects to get submission of at least 5000 synthesis candidates per week.

Co-Admins

As with Brourd, I believe that players should be able to codesign a project and oversee it, just as I believe players should be allowed to coauthor micropapers with those same players that were a part of creating the project. The creator can moderate requests to co-admin a project (accept/reject) but cannot "kick-out" an co-admin, who must resign.

When a project is retired, it should be eligible to be adopted by another player who must make sure that at least one design is synthesized every 4 weeks, either by slot allocation or by lottery, else it will be deactivated.

When a Lab puzzle is done?

A project which (a) has active admins and (b) is getting at least one design synthesized every 4 weeks is considered active. When an admin is finished with a Lab Puzzle, they may resign as (co-)admin and regain their project management "token" to use in administering another lab project.

If the project has no active admins (all have resigned), it is deactivated, even it it has unsynthesized designs left. It remains in an old-project archive however and may be "adopted" but will not show up in the general list of active projects. Candidates will not be selected for synthesis from old projects unless the total synthesis slots assigned to active project candidates is less than 5000. In that case, some old candidates may be used to back-fill, but this probably means the submission/selection parameters need tweaking.

If a lab has active admins, but has not had any synthesis done for four weeks (due to no reserved slots, likes, or lottery wins being assigned to any of its unsynthesized candidates) then it will be retired and the admins released from their roles. It will also be placed in the old-projects archive.

2. How do we allocate synthesis slots?

I see a multi-level rank-based strategy combined "reserved slot allocation" with "likes" and a slot lottery. Assuming for the moment that here are 5000 slots available each week, I would allocate reserved slots to players based largely on rank.

rank 1-50: 30 slots per week
rank 51-100: 15 slots per week
rank 101-200: 8 slots per week
rank 201-500: 3 slots per week
rank 501-1000: 2 slots per week

Total slots: 4950 slots per week by allocation

"Like" allocation is more democratic: Each Lab Member player (over 10,000 points) would be give a "like" budget of around 200 (TBD) Likes per week. They are free to review lab/project candidates and say "I Like this one", much like the current votes.

Anyone over 10,000 points is a Lab Member and can submit up to 5 (or 10?) candidates to each active project each week. Lab Member can also review and "like" up to 200 (TBD?) designs per week. Depending on the number of active projects they may be able to submit more candidates than they can "Like" and definitely more than they can assign reserved slots to.

Rank 501 to 1000 players also have the option exercising up to 8 slot reservations every four weeks (2 per week) or about 40% what they would get at a 20-slot per person per month strategy for the top 1000 players. Highest ranked players however, would be allocated 120 slots every four weeks or about 6x what they would get on an even allocation basis. This allows the more experienced players more option for influencing synthesis and the scope to run controlled "projects" where they can use their slot allocations to help select candidates for synthesis. Slot reservations do not "roll-over"; you must use them in the assigned week or else they get assigned by lottery that same week.

Now here's the trick. Reserved slots are allocated based on your rank at the start of the week. You are free to assign from your allotment of slots to *any* project candidate, whether you designed it or not, whether it is in your project or not. If you assign a reserved slot to a synthesis candidate in any active project it *will* be synthesized. However there are fifty slots unallocated and not everyone will use up their allotted slot reservations, so at the end of the week, there will be synthesis slots left over.

If you are not certain that you want to assign one of your reserved slots to a candidate, you can simply *like* a design. "Likes" work like a lottery ticket. The more "likes" a design has, the more chances it has of getting one of the leftover synthesis slots in a weekly slot lottery. However, you only get to like it once per week; it gains more chances by being liked by more players, or by lasting multiple weeks. (A player may delete old synthesis candidates, even if they have associated "Likes"; this is different from the current model.)

It is still possible that even all the allocated candidates and all of the "Liked" candidates there are still less than 5000 designs for some week. In that case, the remaining "unliked" submissions are put into an even chance lottery for the balance of the slots. If there are *still* less than 5000 candidates for that week, candidates from deactivated project may be used, but we probably need to tune some of the other parameters to get more submissions!

Weekly rollover.

At the end of a week, some candidates will be selected for synthesis, and some may not. All of the candidates assigned a "reserved slot" will be scheduled for synthesis. Some of the "Liked" and other candidates may or may not, depending on the degree of project participation and lottery results.

The admins have the option automatically to "reset" or "retain/roll-over" the "Like" counts for candidates in each project after synthesis selection. If they are retained, then a user may "Like" a synthesis candidate a second (or third...) time in subsequent weeks to boost its chance of eventual synthesis. If they are not retained, then like the current project setup, the counts are reset to zero and new designs compete equally for slots in that week. Both approaches have their merits and some project admins may prefer one approach over the other. Note: it is not that they have a line-item veto: they either clear the Like counts for *all* remaining synthesis candidates, or none of them.

Even if the "Like" counts are not reset, each player has a new opportunity to submit 5 (or 10) new designs to each project each week. The number is higher than the current 3 as we are trying to get at least 5000 candidate submissions each week, and given perhaps 200 active lab players, that means 25 designs per player per week on average!

Slot reservations do not "roll-over"; you must use them in the assigned week or else they get assigned by lottery that same week.
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
Lab Rewards

I'm tempted to say that labs are their own reward?! There may be some point reward based on exercising aspects of lab behavior that we want to encourage:

a) being a project admin during a synthesis cycle (50 points per project with active synthesis)

b) exercising your reserved slots (10 points per slot used?)

c) liking a design (one point per Like?)

d) submitting a lab design (5 points for submission, 10 if it is synthesized)

e) points based on synthesis score on lab puzzles marked by the admins as targeting 100% designs
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
a.2) 10 points for each synthesis of a design in project that you manage - rewards the design of projects that people want to submit candidates to that people are willing to see synthesized
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
c.2) 1 point for each design of yours that someone "Likes"
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
I hear that it may not be possible to start a new round every week. If so, I would change the deactivation criteria for a project to no (synthesis) activity on a project for three rounds (from no activity for 4 weeks)

Also concerns were raised over high scores getting so high no new players could make the top 1000 rank; FoldIT degrades scores over time with a half-life scheme to avoid this and encourage active participation.

If some of the slots need to be reserved for Ensemble use, then reduce my player allocations to allow that.
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
Oh. By the way the synthesis database and scripting are *way awesome* ideas too. But I think I've said too much today ;)
Photo of Eli Fisker

Eli Fisker

  • 2222 Posts
  • 483 Reply Likes
Ideas for the new lab with player projects

First I like to say that Brourd and JL have lots of great thoughts and ideas for our new lab. I however am not too keen about JL's and Brourds joint idea of limiting the number of projects to 20 projects open at a time. Though I admit it will reduce complexity.

PLAYER PROJECT IDEAS

The ideas below is dragged out from a brainstorm session Brourd started and steared, some time back between Brourd, Starryjess, Mat and me, about the new lab and which direction we thought Eterna should go in.

QUEST STRUCTURE FOR PLAYER PROJECTS

I would love to have a Khan like system, where one unlock more player projects as one goes.



That will also help break the hugeness of the current player projects down, to a more concievable size. Make things more gamelike and make us forget about how many projects there really are.

THEME GROUPPED LABS

We were talking about grouping the puzzles in themes. That could be done for player projects too.

Back then I was thinking on sorting puzzles after theme, like dragons in one group, and so on. But it could be done after player, elements, like 1-2 loops, 3-3 loops, zigzags and so on.

AWARDS

Generally we love badges, lots of badges.

Scholarships. Brourd came up with the idea of rewarding scholarships. He also said that there should be rewards for players who do micropapers.

EXPLORATION AND IDEAS FOR VISUALS

Brourd mentioned that right now, everything a player can do is known to them, perhaps, blacking out some things with question marks. Or as Starryjess said: people crave novelty.

Brourd: Same thing with puzzles, instead of going down some dull list, you can move some kind object through a cloudy solution. Perhaps a strand, and when you get close to an unlocked set of puzzles, the strand forms in the shape of the secondary structure. then you solve it: your strand of RNA returns to normal, and you move onto the next challenge. I am meaning through a type of adventurous landscape

Brourd: The goal is to get to the next set of puzzles, but we need it to be an epic and grandiose fashion, to get the awe and excitement building in a player
starryjess: when you solve a puzzle, new corridors open up. At first, they're limited. Something about only having certain puzzles available at first? I wonder if a maze could be involved somehow?
Brourd: not just the same list over and over again, something that makes the move from one puzzle to the next an adventure.

DAILY LAB CHALLENGE

What about lab challenges of the day from the player projects. That would be thirty on a month. Which will be doable. Even three lab challenges a day will somehow be doable.

Brourd: And then, we could even redo labs, but with different req.
Eli Fisker: Yes as next level. One could choose how many one wants pr. day, and with time one will get through a lot
Brourd: It would also increase the daily lab challenge pool as well. So those 30, could potentially be 150
Eli Fisker: Just allow for people to choose how many they would like to do a day.
Brourd: personalize it
Photo of Eli Fisker

Eli Fisker

  • 2222 Posts
  • 483 Reply Likes
Background documents with chatlogs from the discussion of the ideas mentioned above.

More ideas for badges

Brainstorm session
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
Not sure If my 20 projects point was understood. That was 20 active projects per administrator, not 20 active projects in total. The intention is to keep some focus in project creation rather than a zillion random projects that no one wants to delve into, so if there are 1000 project administrators with 20 projects each, they would each get one slot per month on average :) However I expect most players would not run the full 20 projects and probably most would not run any, so at any time there would probably be a few hundred active projects. It might seem like lots of projects, but at 20K slots, you would need for example 200 projects with an average of 100 submissions each to fill all the slots!!

Another possibility might be to rate restrict project creation, much like you are only allowed 3 new player puzzles per day. One new project per week for new lab members, with more projects per week as you advance?

Having a linked project space that you explore might also work, but do you have to solve or complete a project to advance to the next one? I have not played Khan, so I don't fully get the reference.
Photo of Eli Fisker

Eli Fisker

  • 2222 Posts
  • 483 Reply Likes
Hi JL!

Sorry for not expressing it clear enough. Ok, I did understand that it were pr. player and I do see your point with having to follow up on the data. I'm not in wild dissagreement. Maybe just want a couple, like 5 or 10 more. :)
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
I've taken some heat for the "Likes" idea. Sounds too much like FaceBook say some. Internet lotteries are illegal in my state say others. I guess I was hoping to use it to give crowd-sourced feedback to designers in part.

In some ways, the rank-your-own submissions strategy may be better, or at least as good. However, does it mean that only the top-ranked 1000 (or 2000) ranked lab players get synthesis permissions? Currently that is about 23K points for top 1000 and 12K points for top 2000. But how do we deal with players who run up their score then stop playing and hog reserved lab slots that go unused? First-come first served? Right now I think about 200 players play the labs each round even though over 2000 are qualified, so perhaps first-come-first served works fine.

I think a scheme that rewards higher ranked lab players with more reserved slots is reasonable; they are more experienced and more knowledgeable and may be able to better use the slots, plus it is incentive to participate to raise your lab rank. Lab-points erosion via half-life would mean players must keep playing labs to maintain their preferred status. The slots bonus could be moderate or large. Moderate might be to give the top 250 ranked lab players an extra 12 slots each, the next 250 players (lab ranks 251..500) an extra 8 slots, the next 250 ranked lab players (lab ranks 501..750) an extra 4 slots, for a total of 6000 additional slots. That leaves 4000 slots left for Ensemble or other uses. More extreme might be to give the top 100 lab-ranked players an extra 30 slots each month, the next 200 players (lab ranks 101..300) an extra 10 slots, and the next 200 players (lab rank 301..500) an extra 5 reserved slots.

No matter the distribution, unused reservation slots could be given out to players "backup" design submissions using their rank ordering. Since we have already given some guaranteed rank-based bonuses to the higher ranked players a suitable way might be to run through the lab ranks from top to bottom assigning one extra slot to each player that has a backup design which has not been scheduled for synthesis. Once all lab members have been given an opportunity to submit a backup design, the process repeats as long as there are remaining slots and remaining backup designs. This process could even include lab members beyond the first 1000 who are late to the game that month, perhaps beginning with them for the first pass.

It still might be good for project admins to be able to assign slot reservations to submissions by other players to their projects, perhaps by adding them to their own synthesis candidates rank-order list?
Photo of paramodic

paramodic

  • 77 Posts
  • 7 Reply Likes
I don't have much time, so I'll just say that one thing I'd like to see more of is labs where we're actually solving a known or predicted problem, such as placing FMN binding sequences in otherwise purely synthetic designs.
Photo of Eli Fisker

Eli Fisker

  • 2222 Posts
  • 483 Reply Likes
Paramodic, great idea, that sounds like a fun game to me.

That remind me what Rhiju wrote in the conclusion in his latest science paper Why can't we predict RNA structure at atomic resolution:

Recent progress, especially in all-atom refinement and design, makes us particularly excited about the future; a solution to RNA structure prediction appears more and more feasible. We propose that the time is ripe for the creation of a community-wide CASP-style experiment, where groups compete to produce blind models of RNAs about to be solved by crystallography or NMR. The prospect of such blind trials bodes well for the maturing and eventual practical impact of the RNA structure prediction field.

It would be fun being a part of that.
Photo of oolong

oolong

  • 34 Posts
  • 5 Reply Likes
In worrying about limiting how many of anything is available at a time, maybe we should remember that for the first player project synthesis, at the deadline there weren't enough entries. Maybe there won't be a problem in that direction.

I think the first thing that needs to be added if we are going to be able to do the big multiple projects is a good way for players to keep track of what they've submitted and where. Where in the pipeline everything is, and a much better way to display multiple designs to compare them, ours and everybody else's. Before we go off running maybe we need to get this ramped up and debugged. Working on the player projects right now is a major frustration, because I can't even tell from the front page if I have designs in any given project. A major problem in a number of areas of biological research, including ours, is the ability to generate way more data than analysis.

I also think a clearer statement of what the desired end result of all this would be helpful. Are we maybe getting into designing specific RNAs? Or are we really just trying to develop a way to design RNAs? If the latter, I don't really see the reason for preventing anyone from getting into design, because I would think a range of designs of varying "correctness" is what we really need, and we don't need exclusively experts for that. Because in learning to design, we aren't looking for a single right answer, but a range of answers, and how to find any/all of them. I've always thought it was strange that we stopped the labs when we found one "correct" design, because if we are trying to learn design, what we theoretically want to find is ALL the "correct" designs, and how to predict them.

Last thought for now: what we are rewarding players for right now is data. Maybe we should find some way to reward analysis as well.
Photo of Brourd

Brourd

  • 437 Posts
  • 79 Reply Likes
I agree with much of this, especially with the immediate integration of a new interface to allow us to go through the designs, so we can see what needs to be changed for that. I also like the analysis reward, however, what would be the best method to do that. Would it just be through simple posting of thoughts and strategies, and then, how do we determine where one idea begins and ends.

As for why we go with the one high scoring design, then move on, I believe many of the old labs will be made into projects. This will give players the chance to revise old designs and test various hypotheses, and will also allow us to find ALL "correct" designs. If they are not made into projects, I propose that they should be, or at least the projects with very few successful designs.
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
Definitely need to think on UI issues. How do you navigate 100 projects let alone 10000 projects? Or remember which ones you submitted to? Or get results back?

Multiple search/sort interfaces might help:

Show me all projects by Quasispecies that are new in the past 7 days.
Show me all projects where I have synthesis results in the past 14 days..
Show me all projects where I have unslotted designs.
Show me all projects by EteRNA where I have no designs.
Show me all projects with more than 100 designs.
Show me all projects with less than 3 designs, newest first.

Show me all my designs that are unslotted, in ranked order.
Show me all my designs that were synthesized in the past 2 months.
Show me all my designs that have not been synthesized in the last 2 months.
Show me all designs by Brourd with a tri-loop on a 4-stack.

And so on.
Photo of Chesterfield

Chesterfield

  • 6 Posts
  • 0 Reply Likes
Yes.

Interesting set of scenarios.

Both public and private puzzles have advantages. I would opt that there is an option for either. At times, private would be desirable and at others public would be useful. And one may wish to switch back and forth depending on how the sequence develops.

Also, sometimes collaborative (or co-designs) may be desirable as opposed to individual designs and one might wish an option here as well to go from collaborative to individual as the sequence develops, with the option to go from public to private as well. So, some questionnaire as to what "expertise" people have might be desirable so that designers or others can select co-partners which might be useful for "newbies" anyway?

Then, maybe, there might be selectable designs-which might be generated by "the bots"-from which people might select to modify as a project which may fall into the category of public or private designs as well?

The "puzzle owner" is the most likely to choose the designs to select as he would know the parameters he is trying to maximize, but of course one could come up with an infinite number of possibilities, but the owner is really trying to determine certain features of the sequence which is desirable. So, if one is to select slots for a certain round, maybe those should be a limited number per puzzle per round? Really only the owner or co-owners would know when they finished with the design?

The question which seems somewhat vague, is what is the competition in the new format? The ultimate goal is to know if a particular sequence has desirable synthesis characteristics, binding capacity, or whatever, so then badges, points, or whatever should be available for these matters
Photo of boganis

boganis

  • 78 Posts
  • 4 Reply Likes
I fear that the elaborate schemes construed above will turn players into proxy lab scientists. I feel that the 'gaming' aspect of eterna is evaporating swiftly.

My suggestion: Flat even distribution of synthesis slots, period.
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
Gaming is in part about rewards. If you don't see additional slots as a reward, keep playing the puzzles instead of labs. Besides, having slot reservations does not mean you have to use them. It's an opportunity.

I think the main thing that I have not seen contention with is assign slots to users, not labs, and let users choose where to use them. That would probably also mean assigning slots to Ensemble/EteRNA for internal use. Say 40%, or 8000 slots per month. That's enough for 12 slots each month on over 600 labs or projects.

Allocate a good chunk of the rest, say 40% (8 slots/user/month, or 2/user/week) to the top 1000 ranked lab players. (Not a hard target, currently about 23K points.). Save 4000 slots per month (20%) for bonuses that can be earned somehow.

At the end of each lab round (1 week, 2 weeks or 1 month) throw the unused slot reservations into a pool that is spread uniformly to all lab players that month. Probably slots will be left over to allow even lab qualified players not in the top 1000 to get a few slots. Or make it 4 slots to each of the top 2000 players as the baseline - but keep some slots as bonuses that can be earned!! Please!!
Photo of boganis

boganis

  • 78 Posts
  • 4 Reply Likes
Whatever...
Photo of oolong

oolong

  • 34 Posts
  • 5 Reply Likes
Okay, what I'd like to know is if I'm supposed to be putting designs into the player projects now, or if I should stop because it will mess up all of these schemes that are being designed. I would hate to have my work discarded because criteria changed.

I was trying to get through the projects because I kind of felt bad because it seemed not very many people were doing them (possibly this was an illusion caused by how many there are) and because there hadn't been enough designs for the last set at the deadline.

Are we going to do more of the player projects soon or are we going to get more into designing how they work? I would be happy to go back to player puzzles (I like the puzzles, and hope there will continue to be time to work on them as well). Or I can continue to add designs to the projects. What should I do?
Photo of Jeehyung Lee

Jeehyung Lee, Alum

  • 708 Posts
  • 94 Reply Likes
First of all, we'll keep synthesizing 1000 or more designs from player projects to test the new high throughput pipeline. This will happen once in 1-2 months, but the cycle will eventually converge to 20,000 designs / month. That's when we'll change to the new system.

If we can get the new system ready in 5-6 month, that means will be able to do 2-3 more rounds of player projects synthesis.

Whenever we switch to the new system , all the player puzzles will be transferred to the new system as well.

We'll also allocate a big proportion of first 20,000 slots to projects transferred from player puzzles - If we keep running the tests mentioned above there'll be only 2000-3000 unsynthesized designs left in the player projects when the new system rolls out.
Photo of Joshua Weitzman

Joshua Weitzman

  • 93 Posts
  • 0 Reply Likes
3. Lab rewards?

I don't need any rewards. just having the designs synthesized.is a reward. Eterna should get rid of the points and ranks, replace it ith virtual money to buy badges or virtual pets like pogo
Photo of drake178

drake178

  • 44 Posts
  • 5 Reply Likes
I agree with Josh, not everyone is motivated by competition and even then not everyone who is is motivated by it the same way. I think it would do good to decide on competitive vs cooperative game sooner rather than later. History is quite a good example on how competition can propel science, the question is: is it worth it?

Right now the game is competitive, and there are people that care more about the points than learning. There are also other people that don't give a damn about points and just want their designs synthed. This is two distinct groups of people with minimal overlapping. The obvious fear on part of the devs is that there are not enough people in the second group, so if they let go of the first they'll be stuck with noone left. I think this is completely unfounded, so if there are any rewards they should be the ones that motivate the peolpe that are here to learn: give them synth slots.

But if you really want to give things a twist, let them buy it with their points. Instead of badges.
Photo of drake178

drake178

  • 44 Posts
  • 5 Reply Likes
Is it really even necessary to have player projects, or even any kind of projects for players to submit their designs? I think a big thing that keeps people away from trying anything is the complexity of it, i can't just go and say: "i'd like to see how this sequence folds. Can i?". This would of course mean there have to be 'private projects' so to speak.

This is a touchy topic because i feel that i can solve some problems more easily using my own methodology. I suspect at least some of you feel the same way and the only purpose of the public projects is really to show that we can also work together. A noble goal, which i feel should not be put either above or below the work of the people doing it.

The first thing i would do is provide a "Quick Synthesis" option where you can just input your sequence, no puzzle maker, no shape data, no 100 word essay. Sequence, period. Keep the current systems, not everyone lacks creativity like me, just don't restrict us to them. All submissions, whether project or quicksynth should go into the same database, and players could browse their sequences (along with all other data if its a project sequence). Treat it like your own private lab. It also plays in with being able to script your own energy model: your lab, your rules. Similarly, projects would have their own lab with rules set up by the admins. Although, when i say private lab, i'd probably allow everyone to view other players' and projects' labs but not to enter without permission.

When it comes to synth slots, i'd give one to everyone that wants one first. That is, every player and every project. If there is at least that many slots you're green. In your private lab, you order your own designs. In the public labs, the admins do that (can be based on voting or any other creative idea). If there's enough left to do another round, keep going until there isn't. Use the rest for the betterment of mankind. We'll all happily believe that you do as long as you give us our share if there is something to give :P

This is all good and well as long as you have enough slots to give everyone at least one each round. If there's a shortage of slots, the public labs could be scored and private slots assigned based on score alone. Public labs should start out as private ones, if there are too many of them restrict them by requiring a set number of other players to go public.

Obviously all synth results should go into a public searchable database accessible from every lab. There should be a warning if you try to synth something that already has been, offering you to see the result first. If you still want to go ahead, the database should be able to keep both results rather than the average of the two (assuming they are even slightly different).
Photo of drake178

drake178

  • 44 Posts
  • 5 Reply Likes
The puzzle maker would probably still be the best tool for Quick Synthesis but your design should not have to solve/fold to be allowed to synth it.
Photo of Eli Fisker

Eli Fisker

  • 2222 Posts
  • 483 Reply Likes
Adrien said we should post here if we had more ideas to improve the database and comming game changes.

Ideas for improving the player projects

I have noticed a few things while I'm trying to get started on player projects. Here is my ideas for how the gameplay can be improved.

1) I read the description before I start solving. But sometimes I forget the description or I need some of the information included in the project description. So I would like to be able to open the project info from within the lab.



2) In the project above, Quasispecies have also kindly provided the link to SCOR where one can see structures from nature and use as inspiration for solving. It would be very helpful if these links opens in a seperate window, so I can then switch between the pages when I need to. Now when I click the link I get taken away from the player project.

Big thanks
Photo of jandersonlee

jandersonlee

  • 549 Posts
  • 122 Reply Likes
Also on Adrien's prompt...

The database could include both lab and puzzle results. Perhaps puzzles would wait until there were at least N solvers or until you had solved that puzzle (or both) before you could see other people's solutions. :)

For lab results, it could sometimes be important to see which bases matched and which ones mismatched. Sometimes one might want also to specify only 100% matches (successful) as part of the query, or only mismatches, or both, or a count of how many times a pattern was successful versus unsuccessful?

In searching for patterns in the database, it might sometimes be useful to search on structure, or on bases, or on success/failure, or some combination.

I'd like to be able to search for say successful(?) 2-2 loop designs with at least 2-stacks, perhaps using a pattern something like this:

findBasesByStructure(pattern,success) and countBasesByStructure(pattern,success) where success is:

-2: cases where the pattern failed in a lab but succeeded in a puzzle
-1: cases where the pattern failed in a lab
0: any cases that whether successful or unsuccessful or used in a puzzle
1: cases where the pattern succeeded in a lab
2: cases where the pattern succeeded in a puzzle
3: cases where the pattern succeeded in a lab or puzzle

findBasesByStructure("((..((#))..))",1) finds successful cases of bases that match the structure pattern and folded successfully in the labs.

Here the "#" in the pattern would stand for any bracket balanced/matched string, so the pattern matches a 2-2 loop with at least a 2-stach on either end.

Or a quad multi-loop with at least 3-stacks:

findBasesWithStructure("((((((#)))(((#)))(((#))))))",0) looks for all examples of a quad multiloop with 3-stacks.

Or for a triloop on a quad-stack tested in the lab:

countWithStructure("((((...))))",0) returns the count of successful, unsuccessful, an puzzle usages that match that pattern for each unique base string.

Constraining for specific bases might be easiest to specify by interleaving the base constraints after the structure (with N for *any* being the default; ). For example:

findBasesByStructureConstrained(pattern,success) and
countBasesByStructureConstrained(pattern,success) where success can be:

-2: cases where the pattern failed in a lab but succeeded in a puzzle
-1: cases where the pattern failed in a lab
0: any cases that whether successful or unsuccessful or used in a puzzle
1: cases where the pattern succeeded in a lab
2: cases where the pattern succeeded in a puzzle
3: cases where the pattern succeeded in a lab or puzzle

countBasesByStructureConstrained("((((G...)C)))",3) counts examples from labs or puzzles with a tri-loop closed by GC on a quad-stack. The results may include a count of how many successes, failures (zero) and puzzles there were for each unique pattern.

findBasesByStructureConstrained("((((G...)C)))",3) instead finds specific examples of the pattern used successfully in a lab or puzzle with some indication or the lab/puzzle id, the bases that matched the pattern, the success/match string (in this case all "+++++++++++" since we only looked for successes, and the starting offsets of the pattern(s) (in this case one pattern) within the whole structure.

findBasesByStructureConstrained("((G..(A(#)U)..)U)",0) looks for examples of a 2-2 loop with one-end closed by GU and the other end closed by AU, which may or may not have worked. The results may include both cases that worked and cases that didn't, and and an indication of success(1)/failure(-1)/puzzle(2) for each instance.

findBasesByStructureConstrained("(((.G...))))",-1) finds examples of a quad-loop with a G boost point that failed.

countBasesByStructureConstrained("(((.G...))))",-1) returns a collection of unique matching sequences that failed, with a count of the failures for each sequence.

findBasesByStructureConstrained("(((.A...))))",0) finds examples of a quad loop with an A in the first CCW open base; the results may include both cases that worked and cases that didn't, and a count of how many successes and failures and puzzles there were for each unique pattern.

Alternatively, one could search primarily on bases:

countByBases(pattern,success) where success can be as above.

countByBases("UGU",0) counts any labs or puzzles that use that sequence, with counts of success, failure and puzzles.

findByBases("UGU",-1) finds examples from labs that contain the sequence where at least one of the bases is mismatched.

findByBases("UGU",0) finds labs or puzzles that use that sequence, successfully or not. (Hmm, -3 for labs only, no puzzles?)

Note that for puzzles, there are no "unsuccessful" cases, so;

findByBases("UGU",2) finds any examples from puzzles that contains the sequence.
Photo of Brourd

Brourd

  • 438 Posts
  • 79 Reply Likes
Hmm, Looks like Adrien is directing a number of people here ;)

This post is a request for the EteRNA "roadmap" to be posted, as soon as the developers have a rough idea as to what they want to do with the new lab system, and what the future holds for us EteRNA player. It was mentioned back in December and January, and a few players are wondering what happened to the plan :)

Thanks for the game, and I look forward to the new lab system :)