Lab Work Flow System Overhaul Proposal

  • 14
  • Idea
  • Updated 9 years ago
Hi All,

I believe that the way the Lab Work Flow System is currently arranged is causing some serious problems for many players:

Currently, the way the Lab works is this:
Week 1:
1) We start a new lab - create and submit new designs - and begin voting
2) We close the voting and select winners for Synthesis
Week 2:
3) During first 5 days of Synthesis Week - many players continue to create designs WITHOUT the benefit of seeing the results of the synthesis
4) Many more players continue voting on all the new design submissions that were created WITHOUT the benefit of seeing the results of the previous round.
5) On the 5th day of Synthesis Week, the Synthesis results are published
6) Then, in the last 2 days of Synthesis Week, the players who waited to get the benefit of seeing the Synthesis Results have to scramble to analyze those Results, create their new designs, or fix the old ones, and then re-submit.
7) However, by this time, most of the other players have spent their votes on the designs created, again, WITHOUT the benefit of seeing the results of the previous round.

This system is bad for several reasons:

1) The players devoting the most thought and analysis to their designs are left without time to have their designs be seen, and to gather votes from players around the globe in different time zones.
2) These designs, which may have the the most potential for a winning score are being overlooked.
3) The players who are doing their designs first, are not getting the benefit of seeing the synthesis results, and being able to adjust for whatever lessons those results may have contained.
4) The pool of designs in contention each week are therefore are largely unimproved over the past week's designs, or at least have not factored in the traits that won, or factored out the traits that lost, or addressed the problems revealed.

I have an idea on how to fix or at least improve this flawed system:

1) After the first week of design submission and voting, and after the first group of synthesis candidates are selected and sent to the Lab for synthesis - instead of continuing to create and vote on designs for that Lab without the feedback of results - CLOSE that lab for submissions, and for voting! (until the results are back the the next week)
2) INSTEAD, during Synthesis week for this first Lab, begin a NEW Lab. Have players create and vote on the designs as usual - and at the end of the week, when the synthesis results for the last lab are published - then CLOSE this second Lab for both submissions and voting until IT'S results are published.
3) When the First Lab's results are published and the second Lab is closed for IT'S synthesis week, - THEN RE-OPEN the first lab for submissions and for voting.
4) This will mean that
a) the entire player community will have access to the previous round results for the entire length of the new round,
b) the players who are quick out of the gate to submit will now have better designs due to now having access to the synthesis results having already been published
c) the players who used to have to wait til the last minute to get the synthesis result feedback, can now also start early, and have an equal chance to get votes.
d) the quality of submissions overall should therefore rise.

I put together a couple quick diagrams to illustrate this idea:

First, the current system:



Now, my proposed system:



After the first week, where there will be an initial gap, there will then always be a submission and voting each week, but on two alternating labs instead of one. The players will all always have complete results of the previous round of both Labs to factor into their new designs, and into their evolutions and fixes of previous designs. All Players will have a whole week to submit, where previously those desirous of previous round feedback had to wait so long it was near impossible to garner any votes at all by the time they received the results, analyzed and created new ones.

This system will fix all of that. The main difference end up being that, although two 3-Round labs will still take 6 weeks to complete, they will progress concurrently in alternate weeks instead of one following the other. But I believe this is a small price to pay for the advantages that accrue.

Thanks, and Best Regards,

-d9
Photo of dimension9

dimension9

  • 186 Posts
  • 45 Reply Likes
  • happy

Posted 9 years ago

  • 14
Photo of Matt Baumgartner [mpb21]

Matt Baumgartner [mpb21], Alum

  • 128 Posts
  • 33 Reply Likes
Wow, d9, this is an excellent idea! This week the dev team is focusing their efforts on making improvements to the RNA Lab. I think that this is a marvelous idea!. It is now in the Bug Tracker as number 394.
Photo of Snehal (Neil) Gaikwad

Snehal (Neil) Gaikwad, Alum

  • 25 Posts
  • 9 Reply Likes
Thanks D9!
Photo of justinoman

justinoman

  • 6 Posts
  • 0 Reply Likes
Are we really in a popularity contest here? I mean to me the voting is a little disconcerting. First, I cannot even check how many votes I received. Second, new players votes are worthless, as they would have no idea what to vote for. Third, mine didn't get voted on! :-)
I seriously worked pretty hard getting though the challenges to get into the lab, but now that I am in the lab section I feel a little jadded. I think this last part is not setup correctly and needs some tuning. The idea of working on something and then be at the whims of the masses for a vote is not ideal. Random selection seems like a bad idea also. But there has got to be a better way than just voting. Perhaps more sucessful players that have acheived a high score during synth could have a vote that counts more than others? That might help keep the voting more decerning. whether that would allow my design to get in the lab remains to be seen, but I think the voting could still use some work.

This reply was created from a merged topic originally titled
The voting is not set up correctly. My humble opinion..
Photo of AnticNoise

AnticNoise

  • 11 Posts
  • 3 Reply Likes
I think your idea of changing the amount ur vote counts for is really good. For example if it was based on either money or rank it would help to alleviate the problem with lots of new people who just attained the lab to push designs forward with no idea of what they're doing. Maybe it could be based on a scale depending on 10s of ranks? (ie between rank 0-10, 10-20, etc) Or otherwise within top 10: 10 vote strength, top 50: 5 vote strength, top 100: 2 vote strength and anything higher 1 vote strength?
Photo of AnticNoise

AnticNoise

  • 11 Posts
  • 3 Reply Likes
I think that that's a brilliant idea. It would give us something to think about while waiting out the week for the synthesis results to come through and would make participation increase.

For now it's mostly a case of either do challenges or submit a new design/vote for anyone who isn't so interested in doing a bit of in-depth analysing during the week days.

This also means that those who have more skill or a better idea of natural folding aren't around much during the week days because they haven't got as much motive to be on so there aren't many to guide the less aware ones who think it's just a case of build something that the program says is stable.

I think it would be a good idea however if the design making part of the lab was left open just for the sake of checking out things and thinking about future proposals.

It could be good if it was possible to submit designs that would be wiped at the end of the week but which could be commented on? This last point might want to wait for a better program with either commenting for designs as a whole or a way of putting comments within a design (such as pointing to a specific section, pair or stack) for the purposes of pointing out weaknesses or discussing such things.
This way it would be possible to submit a group design maybe? One that would have been tweaked by those commenting and involved with a list of designers instead of a single designer. This could also help to reduce the number of submissions or increase the chances of some of the more experienced players to get 1 design through instead of each trying to get in a design which can easily be up to above 10.
Photo of Matt Baumgartner [mpb21]

Matt Baumgartner [mpb21], Alum

  • 128 Posts
  • 33 Reply Likes
I agree with what you said. Also, I think that your idea for direct collaboration between players on Lab puzzles is a great idea! It is now in the bug tracker as number 396.
Photo of boganis

boganis

  • 78 Posts
  • 4 Reply Likes
@AnticNoise: I dont like your idea of wiping designs in the lab. The reason? Voting like the system is now will tend to be introvert within groups - groups will tend to vote for designs amongst their own designs rather than voting for a very similar design by non-group members or stand-alone designers. Thus you could wipe something in one round, on account of the design being outvoted, only to see the design reimplemented in a later lab as a modification of one of the designs that actually got votes in a previous lab round.
Photo of AnticNoise

AnticNoise

  • 11 Posts
  • 3 Reply Likes
I meant that entirely on the basis of d9's proposal. Under what he said we should not be able to submit designs during the week the synthesis' are taking place. The wiping system forces people to re-submit after the synthesis results have come in both giving people another chance to re-design in light of the recent results and also preventing a swamp of designs before the results are out. If the wiping is too harsh then just have it that you can save 1 design.

I think you might be misunderstanding what I'm saying should be wiped here though, I'm not saying all the designs from previous rounds should be wiped and anyway you can re-submit a design in following rounds if you want to. Just because it didn't get votes doesn't mean it isn't a design that works.
Photo of Madde

Madde

  • 45 Posts
  • 17 Reply Likes
There was already a poll about this idea 2 months ago:
http://eterna.cmu.edu/news/22059
I have no Idea why it wasn't implemented.

The idea of votes that count more than others was already discussed here:
http://getsatisfaction.com/eternagame...
Photo of Chesterfield

Chesterfield

  • 6 Posts
  • 0 Reply Likes
Clearly start up for a game is a trying time. I am im a quandry as to the the amount of time given to design a candidate for the RNA lab:

(1) It seems that one day is not really enough to decide what structure to devise for some "targets", to review the data given and evaluate the designs submitted and decide which to vote for.

(2) However, in many respects, one has days to ponder new structures while the synthesis of submitted structures is being conducted and maybe the extra time one might be given really wouldn't matter anyway as many people are submitting structures.

(Yet, It does seem to me that one is "pressed" to submit a structure before one really has a chance to evaluate it's merits or de-merits). I do think that d9's plea for more time when the lab results were delayed is well intentioned and maybe merited.

( In addition, there are times, presumably when many players are online, that the site is slow and it takes a while for the data to laod and submit-in some cases not submitting at all and the data needs to be re-entered to register. In such cases it would be nice to have a re-submit button or something!)

This reply was created from a merged topic originally titled
Time to design RNA lab candidates/Resubmit button.
Photo of Chris Cunningham [ccccc]

Chris Cunningham [ccccc]

  • 97 Posts
  • 13 Reply Likes
I approve of this idea but WHOA that is a wall of text. :D
Photo of Chris Cunningham [ccccc]

Chris Cunningham [ccccc]

  • 97 Posts
  • 13 Reply Likes
rofl
Photo of kyrstymoon

kyrstymoon

  • 3 Posts
  • 2 Reply Likes
I really, really like this idea! I'm looking at the lab right now and there are 176 submissions for round 4, none of which are taking advantage of what we learned in round 3 (since the results aren't in) and none of which are taking advantage of some of what we learned in round 2 (since some of those results aren't in yet, either.)

Maybe we'll stumble upon a great solution by blind luck, but we're certainly not doing a very good job of building upon past mistakes and successes.
Photo of dimension9

dimension9

  • 186 Posts
  • 45 Reply Likes
To All, and Especially the Dev Team:

I have been thinking more and more about this idea... because as time has gone by, I've noticed the need for it more and more. I think it would certainly benefit the EteRNA team as well as the players to have everyone working with a full set of results feedback for the whole week each week.

Otherwise, our problem solution efforts seem to me to degrade, or slip more and more toward speculation-based, hit-or-miss guesswork. It is like repeating an experiment before getting the results of the first experiment. Apologies, but that is just making less and less sense to me as time goes on.

After all, the synthesis results are all we have to tell us how close we got to the goal, and week after week, we have no results to guide the majority of new design submissions. Whenever a new lab begins, we have nearly two full weeks of designs being submitted before ever seeing the first week's design results.

Put another way, it is only after most shots have been fired, that we get to see which shots even landed anywhere close to the target. Then, the few who waited to see where the first shots struck - have precious little time left to evaluate, adjust their aim, and even get a shot off, much less have it viewed and evaluated fairly and effectively.

I have always found this aspect of EteRNA to be very frustrating, but now with the passage of several months, it has started to go beyond the arena of the merely frustrating, and has begun to enter into the realm of the Utterly Exasperating.

So, with sincerest apologies (if my continued hammering on this issue has become tedious or bothersome), and with utmost respect to all the Devs and the entire EteRNA Team, I must ask at least this one more time:

Is this idea still on the table at all? ... or was it ever actually ON the table?

...and if not, can we please finally make it be?

Best Regards,

-d9
Photo of Berex NZ

Berex NZ

  • 116 Posts
  • 20 Reply Likes
Hi d9,

I think you'd have more chance of treating the first 2 rounds as two sets of round ones.
Round 1 - You treat new
Round 2 - You treat new
Round 3 - You use what you found in Round 1, and improve in Round 3
Round 4 - Improve on Round 2

I know its not ideal. But it might lessen your frustration. I guess to a degree, they are using your 2 week design philosophy...
Its only going to get worse, if you keep on pushing....
Not saying its not worth pushing, but try not let it get to you so much and now that we are synthesizing 16 designs a week. Delays will only become more pronounced.
Photo of dimension9

dimension9

  • 186 Posts
  • 45 Reply Likes
Thanks very kindly for your input here, berex, I see you are trying to help, and I do very much appreciate that, but to me, this sounds exactly like the very same situation we already have:

Round 1 - we treat as new, (because it is)
Round 2 - we have to treat as new, because we have no results all week (until Friday) so, this is essentially an extension of Round 1... so we have no choice but to treat it as new
Round 3 - we finally have results from Round 1 - so we improve them in round 3, (because Round 2 results will not be out until THIS Friday), and
Round 4 - we finally have results from round 2, so we finally improve on those, just as in your.suggestion above.

So, I do agree with you that it is not ideal - because it is essentially not a change at all. And I cannot see how it could possibly lessen my (or anyone's) frustration, because it seems to do nothing which the current system itself does not also already do.

Sorry to express such frustration at your reply; I do recognize and appreciate at least the thought, and the attempt to help, so I thank you sincerely for that.

And, I think I will also try to begin to take your excellent advice, and try not to let it get to me so much, because I also think you are probably right when you said: "Delays will only become more pronounced," and "Its only going to get worse..."

Thanks & Best Regards,

-d9

:( ...succumbing a bit further to frustration and doubt
Photo of Matt Baumgartner [mpb21]

Matt Baumgartner [mpb21], Alum

  • 128 Posts
  • 33 Reply Likes
d9, I understand your frustration. We will discuss it at the dev meeting tomorrow. I have an idea that I have been tossing around that might help deal with the issue of initial lag (But not the problem with only having 2 days to make new designs.) We plan on making bots to make designs on the RNA Lab. One solution to the initial lag is to have the bot make the initial round of submissions a week before the new Lab is released so the new puzzle is published with the bot's synthesis data.

What do you think of this sort of addition? This would help some with the initial delay of synthesis data. Although I agree that other changes need to be made as well.
Photo of Chesterfield

Chesterfield

  • 6 Posts
  • 0 Reply Likes
D-9:

Your points are well taken!!! But the issues you discuss were present at the inception of eterna and before it went public.

Just calm down (LOL). This is only a game-for the most part. The creators are aware of these issues and problems and if they wish to spend time (and money- I suppose) at the lab -to duplicate efforts and run tests on many samples, then that is their concern. Relax and enjoy that one can "play professor" even though one's exact design is not chosen. Also, this allows the Admin to squeeze or cull as many designs from the players as they can and thus (possibly) be certain they have a useable or viable RNA? The voting system is undergoing revision-at least I believe it is. So that is one issue that is being addressed.

There are several ways to deal with the issues you have discussed-many of which the Admin has probably considered:
For example:
One issue to the design/vote problem is to provide a design cut-off at 24,36, or 48 hrs prior to the lab submission date, and then allow only voting for those hours. Thus one would not have to review "last minute entries". But many of the structures are related and though minor changes in structure can be important, I don't see how these can be evaluated by the players-ab initio anyway. (Thus is an AU pair better than a UA pair in any given stack?-energy and other considerations notwithstanding).

As to the puzzle and submission, one can address this by running 2 or 3 designs instead of 1. That is RNA-1 is designed, voted on and synthesized. And while it is being synthesized the players are designing RNA-2. Then when it is submitted the results for RNA-1 are evaluated and redesigned by the players. Perhaps the timing would require RNA-3 (especially if the results on RNA-1 are delayed). But in this manner your frustration might be addressed, though the final results might take a little longer as to the ultimate structure design.

Cheers-d9