onionsformagnolia's dashboard

@Deborah, I suggest you read what I wrote more carefully. This isn't personal at all. Cheerleaders offer nothing other than salve for Larry's bruised ego. While that is nice, what I'm offering Larry is a way forward for his business. If he's smart, he'll listen. If not, he'll continue what he was doing before. The message might sound harsh but sugar-coating the truth isn't going to help anyone, least of all Larry. This is an opportunity for him to learn from his mistakes and salvage his business.

By the way, I did not "imply" there was "neglect, ignorance and hubris" and Larry's part. I outright said it.

Neglect - not to attend with due care or attention. There is no question that Larry did not pay due care or attention to the issue of disaster recovery.

Ignorance - the state of being uneducated or uninformed. Larry clearly was ignorant of proper backup procedures. He portrays having to backup a 500GB database as something extraordinary. It isn't.

Hubris - overbearing pride or presumption. Larry Halff still seems to think this was bad luck. It wasn't. This is his pride talking. This situation was entirely predictable and unavoidable. If he doesn't understand and accept that now, I'll say it again, Magnolia does not deserve to stay in business because he will only cause more grief for users in the future.

The truth hurts sometimes.

@Heidi, I agree with you that these "solutions" are misleading and unfair to anyone who has more than a few dozen bookmarks. If you have thousands of bookmarks with a few hundred tags, forget it. None of these "solutions" will work for you. The only hope is that the drive recovery people manage to recover the database.

@Gerry Quach, Larry should be encouraged all right. He should be encouraged to leave system administration to professionals in the future! It would have cost under $400/month to have a proper backup in place, and that's with having a full daily, weekly, monthly backup of the 500GB database. Larry could have paid for three generations of backups for four years for less than what this episode is going to cost him, assuming this isn't going to cost him "the business", such as it was. It could have been done even cheaper than that if he had seeded the original backup using "sneakernet", stored the backups on a computer at home, and just transferred the deltas from the original on a daily, weekly, monthly basis. This disaster wasn't caused by Larry being unlucky. It was caused due to his neglect, lack of clue in disaster recovery procedures, and hubris.

I'm sure that most of the Magnolia users have no idea how to backup data on that scale. If they're like the average user, they've probably lost data before, at least several times, due to poor or non-existent backups. Of course they would be sympathetic to Larry when he portrays this as a "Sorry but shit happens." event. He'll get no sympathy from me. He doesn't deserve to be let off the hook that easily. If the data recovery folks manage to recover the database, we can all breathe a collective sigh of relief but it would be a huge mistake for Larry or anyone else to continue believing that this disaster was unavoidable.

@Larry, I know I said that I would never entrust anything to you again. I retract that. If and only if the data recovery folks manage to recover the database, you have a chance of resurrecting Magnolia but it will require that you implement very different communications and disaster recovery policies than you've had. If you still think that this disaster was unavoidable, you haven't learned anything and don't deserve to remain in business. If you continue to pursue a CYA (Cover Your Ass) communications strategy, again, you haven't learned anything and again, you don't deserve to remain in business. Demonstrate through your words and deeds that you deserve the trust of users and you'll have it. Continue on your current course and you don't deserve it and Magnolia will die a well-deserved death, It's all up to you (and the drive recovery people).

That new "web cache recovery tool" would be perfect.... if I only cared about the first 30 bookmarks out of the thousands I have.

How could any user who doesn't have the detailed knowledge of Magnolia you seem to have, have known that they were "buying" a Honda and not a Porsche (what an irrelevant comparison)? How could anyone have known whether Magnolia was a VC-funded 100 person company, or as it turned out, one man's hobby that he was running on a shoestring? (Just like how can you tell if someone is or isn't a 19 year old co-ed on the Internet, as so many seem to claim they are?) Once the service was running, there was no way to tell how big or small, how competent or not, Magnolia was and most people just don't scrutinize these things all that carefully. We now know.

Your cavalier attitude towards the loss of the work of who knows how many people, "no lives were lost, bla, bla, bla", is reprehensible. Most self-respecting, competent people set the bar lower than "as long as no lives are lost..." when it comes to figuring out what is an acceptable level of service. – onionsformagnolia, on February 06, 2009 20:00

We use the hot dump tools provided by the various databases we run and then we rsync the dump file. People have been backing up large databases for a while, you know.

Larry apparently didn't have a backup. The "hot backup" being corrupted due to filesystem damage on the live server illustrates how that scheme was no different than relying on RAID for backups. It was a bad idea.

I read that paid subscribers were going to be issued refunds (no idea how much) but they're no better off in terms of data recovery so there goes the, "You have no right to complain since the service was free." argument.

Again, I hope for the sake of all concerned that the data recovery company comes through.

@jimmyjo, ever heard of rsync? You take the hit once, when you upload the data. Assuming the server wasn't sitting in Larry's basement at the other end of a DSL line, transferring 500GB over a 100Mb/s connection, like you'd have in a server in a colo facility, is perfectly feasible. Thereafter, rsync only transfers the deltas. Works like a charm for us and we're backing up almost triple the "not someone's rinky dink app" amount on a daily, weekly, monthly basis. Who'd a thunk that people might need to do backups for large datasets too?

@intuited, as I wrote in the recovery thread, don't confuse the service being no cost to you with being free. There was a revenue model. That it might now have been a viable one isn't my concern beyond the immediate impact it has on me and others. It doesn't seem to me that this had much to do with lack of revenue so much as lack of clue in disaster recovery procedures. Whether Larry is a decent guy or not isn't at issue. Those, like you, who seem to think that just backing up our own bookmarks should have been sufficient seem to have been using Magnolia far differently than I had been. I care not just about the bookmarks. I care about the countless hours that I spent categorizing them. If just a bunch of bookmarks arranged in one, rigid format would have been sufficient, I would have just continued to use Firefox for managing my bookmarks as I had been for years.

Anyway, let's hope that the data recovery company can recover something.

Finally, some useful information. Thank you, Larry. Which filesystem were you using? Was this caused by a hardware failure? Were you using MySQL? How were backups being done and how were they being transported to the "separate system"?

For future reference, Amazon S3 is a very economical backup solution for large amounts of data.

You learn something new every day. I had never heard of the term "Sybil attack". Rest assured, I registered last night and posted whatever I posted under my own nick, nothing more, nothing less. That other people may choose to agree or disagree with what I have written is not under my control nor do I really care one way or another.

As for criticisms of how this is personal, no, it isn't. I have no idea who Larry is and I had no idea that Magnolia was/is a one man band. That he may "feel bad" isn't my concern right now. I'd rather that he wasn't going through this but he is. I didn't bring it upon him. He did.

To the person above who is said that bad stuff happens and that backups aren't infallible, though that's true, best practices call for a few generations of backups. We do monthly, weekly, and daily scripted backups of all our data and more importantly, we TEST recoveries to make sure that we can restore from the backups. The backups are highly-automated and require very little by way of babysitting on a day-to-day basis. Moreover, as I've already explained, if Magnolia were so easy to back up for its users, then we wouldn't need Magnolia, would we? The whole point of this service for me was that it provided a tool that I didn't have locally.

The concern I have is that through neglect, ignorance and hubris, data that may have otherwise been recoverable could now be irretrievably lost. There are some things one can do to preserve as much data as possible in disaster scenarios such as this and there is no evidence that these measures have been taken. Perhaps all the right things have been done but how can we know? All we have are platitudes, deflections and expressions of regret with no substantive information.

As for drive recovery being expensive, considering the number of Magnolia users who are affected by this, I would think it would be a no-brainer to spend whatever it takes to do it. If it costs any more than an exorbitant $5000, chances are there would be nothing to recover anyway. I'd happily contribute to a fund directed at this but again, we don't even know if Larry has control of the server(s) any more.

Those of you who seem to think that who I am and whether I'm male or female matters, and who think I'm being mean to Larry, ought to concentrate on the message, not the messenger. My message is simple. Larry has a moral obligation to Magnolia users to provide a complete and detailed explanation without obfuscation, deflection or evasion.

It is true that Gmail has had outages but I'm not aware of any instances of data loss. Users have an expectation that these services are run by competent people and as such, don't consider the possibility that they might not be.

The appeal of Magnolia for me wasn't that I didn't have the ability to manage my own bookmarks locally. It was that I could tag the bookmarks in a cross-browser, cross-machine way and share my bookmarks with others. If I could have backed up those bookmarks and the semantic data around them easily, I wouldn't have needed Magnolia. A collection of bookmarks without semantic data about those bookmarks is useless to me. I already have about 10,000 of those in my Firefox bookmarks that I've been dragging around since the mid '90s. The real value is in the many hours that countless others on Magnolia and I spent categorizing the web.

Let's not confuse the fact that users didn't have to pay for the service with Magnolia being a non-profit. Their revenue model seemed to be to run Google AdWords, which is a perfectly fine model for profit. I know I clicked through on enough of ads, and not because I was thinking, "I'd better click through so that Magnolia can continue to offer this service." but because the links were relevant to me.

Users lived up to their end of the bargain, i.e. we entrusted our links and semantic data to Magnolia. The more links we contributed and the more we classified that data by tagging and creating groups, the more valuable a resource Magnolia became to its users, advertisers, and Magnolia. That's what is known as a win-win-win-win. A win for users who are trying to make sense of the chaos of the web. A win for Magnolia as they monetized the data we provided (and I don't begrudge them that one bit - I in fact wished them well). A win for the advertisers because they could make us aware of their offerings. And finally, a win for Google who provide the ad distribution network. It doesn't get any better than that.

Whether Larry and Co. were successful in making a profit or not is another matter peripheral to the central issue, which is, "WHERE THE HELL IS THE BACKUP?" Larry and Co. doesn't get a free pass just because the service was supposedly FREE. They still have a moral obligation to deliver the service and if they were unable to do so due to financial constraints, they just had to come clean with users and explain that they would either have to start charging for the service or cease operations. – onionsformagnolia, on February 05, 2009 18:36

I really appreciate the tips that people other than Larry have been posting here. I know you're trying to be helpful and that we're in the same boat. However, the only real solution lies with Larry and so far, he has been less than forthcoming.

The Google cache scheme where you search for tags is only helpful if you have only one page of bookmarks per tag. I have many tags for which I have multiple pages of links and it's not obvious how to get them. Moreover, it is a monumental task even if I could get all them that I would not have to undertake if Larry and Co. managed to get their act together.

Larry, your silence is deafening. I repeat, where is the backup? Why have you not been able to recover? What is the real problem? The people who entrusted their bookmarks and semantic tags to Magnolia deserve straight answers.

By the way, none of the approaches to getting bookmarks from alternate sources should work for bookmarks marked as "private" if Magnolia worked as advertised. I know I have a significant number of those. In fact, I think that was one of the things that attracted me to Magnolia. I don't recall such an option with the alternatives I had evaluated.

Larry, on your home page, you wrote:

"So far, my efforts to recover Ma.gnolia's data store have been unsuccessful. While I'm continuing to work at it, both from the data store..."

How is that possible? What is this "data store" you're talking about? Was there no back up?

"In this past year, many of us have seen much loss around us."

What does this have to do with the current problem, other than a lame attempt at deflection?

"While bookmarks seem small on the national or global scale, I know that many of you had built intellectual and social capital through the bookmarks, groups, and connections you made here."

A blinding flash of the obvious. So far, you've said many words without saying anything.

"Ma.gnolia was approaching the third anniversary of its public launch; for me, it was the project and people to which I'd devoted most of my time, energy, and love for nearly four years. It's still a little too soon to give word about the return of Ma.gnolia the service and the future of the M2 project, but I will keep this site and our Twitter account updated as those decisions are made."

You have nerve to even talk about the return of Magnolia and M2, whatever that is/was. Forget it. This "business", if it ever was viable, is now dead all but in name. I wouldn't entrust anything to you any more because of the way you've bungled this. You've been quite evasive in answering direct questions posed to you about what happened. If you owe at least one thing to Magnolia users, it's an honest explanation.

At the moment, it looks like I've lost thousands of bookmarks that I've added over the last couple of years and I did not have them anywhere else because I was foolish enough to assume that Magnolia was being run by competent people! To say that I'm pissed off is an understatement. The irony is that I had evaluated and rejected Delicious and SimpPy for reasons that I don't even recall any more. I've lost something of tremendous value and by the looks of it, I will not be able to recover anything since none of the "solutions" that you have posted so far work for me. I'm working on the assumption that any competent system administrator would have been able to get a server running by now and restored from a backup, even if *some* data was lost.

What happened? Your server got seized by creditors? You had a nervous breakdown? Ex-wife got the server? Your dog ate the server? Your server got abducted by aliens? You had a disk crash and hadn't done a back up since, well, ever?

It's hard to imagine how you could look any worse so I suggest you just come clean. If it's a matter of money, I'd contribute toward a solution but not without knowing the whole, unvarnished truth. So, how about it? Are you going to explain exactly what happened or are you going to continue your "keep 'em in the dark and feed them horse !@#$" communications strategy?