The Java iTunes to Songbird Metadata Importer for Linux

Hi Ivan,

I tried running your program on my Gentoo Linux box, like so:

java -cp .:sqlitejdbc-v054.jar iTunesRatingsToSongbird -i "/winxp/Documents and Settings/myuser/My Documents/My Music/iTunes/iTunes Music Library.xml" -s "/home/myuser/.songbird2/ekchz4wy.default/db/main@library.songbirdnest.com.db"

But, I get the following error:

ERROR: no such column: media_iten_id
Exception in thread "main" java.lang.NullPointerException
at iTunesRatingsToSongbird.searchSql(iTunesRatingsToSongbird.java:356)
at iTunesRatingsToSongbird.main(iTunesRatingsToSongbird.java:210)

Any ideas?

I am using Songbird 1.0. I can send you the two files, if it will help debug.

Thanks!

Actually, if you could send me the log file that it generates (ratingsLog.txt), that would be a great help.

Fixed the first problem, it was a simple typo that I didn't check for when I was running it (I always run with --ignore since I converted everything to .mp3).

For the second problem, I think I'm getting more than one duration per song for some reason. . .Not sure why.

On a side note, I'll have artists, names and genres working pretty soon, hopefully these bugs aren't too hard to work out.

Ok, I think this fixes both those problems. I'm getting a lot of duplicates, but I think that's because iTunes really messed up my library on windows. Tell me if you get a lot as well, they would come up in the collision detection part as two identical songnames.

http://rapidshare.com/files/172511525...

Thanks for being patient while I fix these bugs!

Also, I've run into a little bit of a hitch in editing the artist data in the sql file. Maybe a developer could help me understand this, but I'm not sure what I should be putting into obj_secondary_sortable. Seems to me like it's the album, plus some optional number, then the track number for sorting purposes. I don't want to mess around with this stuff until I know exactly what I'm supposed to put in there, I don't want to break people's sorting.

Ah, I missed the editing window by a few seconds, it's the disk number.

I got further using the --ignore switch, but it still crashes with:

Searching. . .
Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 9, Size: 9
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at biTunesRatingsToSongbirdratings1210.searchSql(biTunesRatingsToSongbirdratings1210.java:480)
at biTunesRatingsToSongbirdratings1210.main(biTunesRatingsToSongbirdratings1210.java:210)

It seemed to have many more collisions than I expected, but I am not sure. Would it be possible to print out more info for each collision? Filenames, album and artists names? Also, would it be possible to print a collision index, like (1 of 213), for example? That way I would know how many collisions I have left to work through...

If I don't use the --ignore switch, it dies immediately still:

java -cp .:sqlitejdbc-v054.jar iTunesRatingsToSongbird -i "/winxp/Documents and Settings/myuser/My Documents/My Music/iTunes/iTunes Music Library.xml" -s "/home/myuser/.songbird2/ekchz4wy.default/db/main@library.songbirdnest.com.db"

Here's the ratingsLog:

http://www.trevorbowen.com/ratingLog.txt.bz2

Also, would it be possible to update songs as it finds them, instead of waiting at the end? That way, I would have something when it crashed. ... Thanks!!!

Sure, I'll start printing out more information for each collision, I just started collecting artists, albums and song names from the iTunes file.

I'll work on the collision index, it'll be a struggle but it's definitely a feature I wanted when I started on this. The way I've been doing things right now it'll be tough to implement.

Wow, the --ignore bug is huge, I'm not sure how it didn't mess up anything for me. It's a pretty big oversight, but an easy fix.

Actually, it does update as it finds songs. Do you have SongBird open when running this? I think Sqlite locks the db when it's in use. I'll have to note that in directions. Also, just on a whim, what version of iTunes did you use when this xml file was made?

By the way, is there any way to update my topic description at the top? I can't edit anymore, and I'll need to keep putting directions in there, or at least a note to where current directions are.

Here's the newest version:
http://rapidshare.com/files/172574620...

Changelog:
Fixed those bugs above (hopefully. It worked with or without --ignore, with more collisions without ignore, so that seemed good, but I could never reproduce the first crash).
Prettier and more informative log.
Better matching of files.
All computer matching first, user matching at the end.
Collision matching shows how many to go.
Collision matching prints the iTunes reported file location and the sql reported file location for each. Artists and albums are next.

Any suggestions on a nicer output with the collision matching? It only looks good when the terminal is maximized on my 19 inch widescreen.

Please report any more problems, along with the stack trace if you can.

Thanks,
Ivan

Hi Ivan,

Thanks for the quick turnaround!!! Unfortunately, I am still hitting some bugs.

If I use the "--ignore" switch, I get this stack trace:

java .:sqlitejdbc-v054.jar iTunesRatingsToSongbirdCollisions --ignore -i "/winxp/Documents and Settings/Bowen/My Documents/My Music/iTunes/iTunes Music Library.xml" -s "/home/tbowen/.songbird2/kv8vzrn2.default/db/main@library.songbirdnest.com.db"
Exception in thread "main" java.lang.NoClassDefFoundError: /:sqlitejdbc-v054/jar
Caused by: java.lang.ClassNotFoundException: .:sqlitejdbc-v054.jar
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: .:sqlitejdbc-v054.jar. Program will exit.

If I omit the "--ignore" switch, I am able to complete successfully; however, if I accidentally enter the wrong number during a collision (for example, valid choices are 0, 1, and 2, but I enter "11" by fat-finger), then the program crashes. Could you add some error handling to ensure that an entered number is 0 or one of the available choices?

Also, I think you could cut the number of queries in half. Think of it this way, you have 2 similar files that match 2 other similar files. It could be either (A1=X1,A2=X2) or (A1=X2, A2=X1). Once I tell you one of the matches, you should know the other. ... This would help avoid "unmatched songs".

Also, in the above case, it would be nice to have an automatic "don't care" switch to apply for cases where the song names matched exactly. This usually occurs during a weird import (for example, same album imported twice, with slightly different album name but same song name). Most of my 154 conflicts, I did not really care.

Now for the worst bug... If I omit the "--ignore" option, the program ran to completion, but nothing changed! (I make sure Songbird is off when I run.)

I only have one profile, and I keep Songbird off, so I don't know what else it could be.

Any thoughts?

Thanks for all your hard work!!!

Hi Ivan! Thanks for all your hard work! Are you gonna make this into an addon eventually? That would be great! Whenever you get to importing playlists and such, you should look at how Songbirds importer works/take it from them, cause it seems to work pretty good..

m27315 - You need to put -cp in front of the jar file. This tells java to set the classpath to what it already is, plus the extra jar file. Without that, it tried to run the jar file directly, which it couldn't do.

Yep, error handling is on my list. I'll get to it today probably.

I was actually thinking about doing something like this. I would just keep track of all matched songs, then take them out of choices if they come up later. It should be simple to implement.

Try running it again with the -cp argument for java:

java -cp .:sqlitejdbc-v054.jar iTunesRatingsToSongbirdCollisions --ignore -i "/winxp/Documents and Settings/Bowen/My Documents/My Music/iTunes/iTunes Music Library.xml" -s "/home/tbowen/.songbird2/kv8vzrn2.default/db/main@library.songbirdnest.com.db"

Unfortunately, I can't really do anything to catch that error, since it's all on java. Maybe when it gets stable enough to make a binary I'll do that.

Thanks for working with me on this!

Boosh - I know absolutely nothing about javascript or addons, so probably not. But who knows. I've taught myself languages before, so maybe I will. Also, I'm pretty sure that the Songbird importer depends on keeping the exact same file structure. That's why it doesn't work on Linux without some hacking around that. Also, iTunes is notoriously bad at keeping track of files (it has lost cds of mine, copied things, misplaced things.), so a lot of times the paths won't match anyway.

Thanks for the input! I'll get back to work!

m - I made a quick benchmarking option, and I checked the results with and without checking if a song has been matched already on my 2500 song library. The benchmark just gets the time to run all songs, without collisions:

	without matchedsong logging

		time to complete: 37 seconds

		collisions: 46

	with

		time to complete: 99 seconds

		collisions: 32

These are averages, over three runs each (the collisions stayed the same of course.) I can most likely make the code more efficient for the second one, I'll see how it goes. It would be nice to have more test databases to work with, maybe I can get some from friends.

Hey everyone, I'm new to this xml stuff but I have a questions. Is there anyway I or someone else could modify this scrip so that it would do the same thing except help me transfer from rythmbox to songbird? I just want all my ratings and stuff.

Maybe, I can look into it. I'm not sure how rhythmbox stores anything, but I'll check it out. Maybe it's a really easy modification, since all that would be changed in the program is the iTunes parsing part.

By the way, everything coming along nicely. Artists, albums, genre, song names and ratings are now transferable, with playlists on the way. There's also a nice new output system for the collision detection and a lot of performance enhancements. Just wanted to let anyone who's waiting know that I haven't forgotten about this, and the holiday break should give me a lot of time to work.

I should have a new version up in the next couple of days. Happy holidays to everyone!

Alright, I've got a new version ready.

http://rapidshare.com/files/178034140...

Changelog:

Can now import: ratings, albums, artists, song names
note: anything in Songbird will be overwritten by iTunes information when updating
Checking for invalid input in collision entry
Better error checking in arguments
Songs matched once will not be returned in results again
For now, albums/artists/songs will have quotes removed when being updated, because of sqlite limitations.
Fancy output
Dry run option - just prints all matched combos to the log, doesn't update anything
Fast matched id searching
Stores only what it needs (songs/albums/artists etc.)
Logs when anything is updated
Cleaned out countless bugs

On the way:

Playlist + playcount importing
"Date added" importing
Genre importing
Speed/memory usage improvements
Better matching

Just extract all files from the zip, then check out the --help argument for new information on running. It matched all but 54 of my 2500+ song library in a little under 2 minutes on my e6300. Also, unless you have duplicate songs differentiated only by file type (mp3 or m4a) I recommend using --ignore.

Please tell me about any crashes, bugs, or suggestions. You can either post here or email me (my email is in the --help option).

Remember to back up your Songbird library file first!

Ivan

Bug report:

1) When running without the --ignore flag, the script runs to completion, but does not update my library

2) When running with the --ignore flag, the program crashes after only a few songs. Backtrace follows:

java -cp .:sqlitejdbc-v054.jar iTunesRatingsToSongbirdCollisions --ignore -i ituneslib.xml -s /home/camiller/.songbird2/5qi9t2g9.default/db/main@library.songbirdnest.com.db
Working. . .
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -7
at java.lang.String.substring(String.java:1949)
at java.lang.String.substring(String.java:1916)
at SongbirdSqlCommunicator.getFullFileFromId(SongbirdSqlCommunicator.java:552)
at SongbirdSqlCommunicator.updateAlbum(SongbirdSqlCommunicator.java:649)
at iTunesRatingsToSongbirdCollisions.handleUpdates(iTunesRatingsToSongbirdCollisions.java:947)
at iTunesRatingsToSongbirdCollisions.main(iTunesRatingsToSongbirdCollisions.java:532)

Thanks for the bug report. A couple of people have been sending in the index out of bounds error, I'll check it out when I get home again. It's strange I never got that, but its tough to account for different song/artist/album names.

Thanks for helping me to get this working!

Ivan

With the help of Chris, I've resolved a lot more issues. The program now runs well on his 13.5k library. I'll have better matching in a couple of days and then I'll upload a new version. Still, any dbs and iTunes files would be much appreciated.

Ivan

Yeah, it's coming along. I've been really busy lately, and haven't had a lot of time to clean everything up for a release. But, hopefully I'll have something out soon.

Sorry to keep you waiting.

Sorry everyone, but I'm stopped this project. From what I've heard, metadata management is changing a lot in the next release, and since I'm not using the api (I know little to nothing about javascript/xml) stuff will be broken. Anyway, today I successfully imported all my music following the directions here (link). So, hopefully you can get that working for your library. If not, you may just have to wait until the next release. Sorry everyone.

On a brighter note, this project has taught me so much (interfacing with sqlite, lots of memory management, too much to mention here). I'm an AP comp sci student in high school, and this has been a great learning experience. Maybe when I have some time I'll start learn some javascript and try at an extension.

Thanks to various people for test data, and to the songbird devs for helping me out now and then.

Thanks,
Ivan