personal metadata and mkv question

[mjmacky]

So, let's say a file downloaded from a service that's like iTunes (but uses VC-1 codec) is run through Tunebite (a DRM removal software, so that it can play on other devices). Assuming it had personal identifying information in the first place, does the resulting file keep personal metadata on it? If so, does it still keep it through a mux to mkv? If it does... then crap!

Even further, if my freak out is justified, is there a remedy to all that? Because... ummm... let's just say I like my privacy ;)

[anon]

Not much into this iTunes stuff, but there's a tutorial on how to remove personal information from files on Waffles. I can post a copy here if you desire.

[mjmacky]

Could you post a copy? That way I have something to go on, at least to check my files. All personal metadata would be in the header anyway, right?

[anon]

Here you go:

While iTunes was looked down upon by many users at OiNK (and I'm sure here at Waffles too), I feel that this is a poor attitude to have and that it is most likely a product of ignorance about how the iTunes Music Store (iTMS) runs. This tutorial will serve to educate Waffles users about the iTunes store and how it can be used to better our (already) wonderful site. :-D

(A Brief History of the) iTunes Music Store & iTunes Plus

In the past...
The iTunes store has for a long time only sold amusic files at 128 kbps that were also infused with DRM protection. This DRM (Digital Rights Management) protection meant that a song purchased on the iTMS could only be played, burned, or transferred on a computer that was registered with the song's purchaser's account name and password; only 4 computers at a time could be registered. To make matters worse, the only ways to remove the DRM were to burn the song to a CD and reimport it or to run it through a special program that would re-record the audio from the original file. Both of these methods obviously produced a DRM-free transcode. So that's 2 strikes against the rules... :-/

To reiterate, we had two problems here: (1) the bitrate was not high enough to meet Waffles standards and (2) the DRM protection made it impossible to share a file unless it was transcoded. This is why in the past, OiNK had not allowed iTunes uploads. However there has now been a change to the iTMS and some of its content now meets standards for "the rules (http://waffles.fm/rules.php)".


The present...
A short while ago the Apple company announced the addition of iTunes Plus to their music store. This turned out to be the release of DRM-free songs encoded at 256 kbps. These files were originally sold at a slightly higher price ($1.29 as opposed to the $.99 price for the DRM'd songs) until other online competition made them lower their prices to match their other songs. :thumbsup:

Now more and more content (especially anything newly added to the stores catalog) is available as iTunes Plus downloads for $.99 a song. This protection-free and high-enough bitrate is also often iTunes exclusive content that would make many a Waffler happy to get ahold of.

How to "use" iTunes Plus...
First off, you will obviously need an iTunes Music Store account. This will require a password and credit card information to create. But once you create it (for free) you can use it to "Download Album Art" for free (my absolute FAVORITE feature) and it never goes bad. To reiterate: it needs a credit card to start, but it is free to start; free to keep; never expires from lack of use; can get you free art (usually at about 600px by 600px); and should you ever decide to actually buy some music, the account allows you to make the purchase.

Now be careful and do a tiny bit of "research" before you go downloading and uploading a bunch of stuff from the iTMS. You MUST make sure the content you are looking at is actually an iTunes Plus download -- after all some of the DRM'd stuff is still there.

Any song that is downloadable as iTunes Plus will have a small blue/gray plus next to the "Buy Song" button. See below:

http://img69.imageshack.us/img69/6821/plusir4.jpg

Also, if you are viewing an overview of multiple albums (and can't see the individual songs) then any iTunes Plus album will have the words "iTunes Plus" under the picture of the album. See below (and take notice of the other "non-Plus" albums there too):

http://img158.imageshack.us/img158/7120/albumpv0.jpg

The Warnings...
While some iTunes content is now acceptable to upload, there are still a few things to be wary of:


**THIS IS THE BIGGEST POINT ABOUT iTUNES THAT PEOPLE NEVER SEEM TO UNDERSTAND**
(1) All songs (both Plus & DRM'd) purchased on the iTunes Music Store ONLY come in AAC format; this has an M4A extension, but should not be confused with the lossless form of M4A known as ALAC. Thus any songs you find or upload MUST be in this format AND at 256 kbps. You know immediately that it's a transcode if it is listed in any other way. To reiterate: any "iTunes obtained" song that is not lossy M4A has to be a transcode.

(2) The iTunes Plus songs may be DRM free, but as a precautionary measure against P2P file sharing, the Apple company has embedded the purchaser's username and account information in the song's metadata. This basically tags the purchaser so they can be identified if the file starts internet-wide circulation. There are a number of ways to get rid of this metadata, but many of them also involve transcoding. The next section of this tutorial will walk you through a few ways of safely removing the information without transcoding. This is commonly called "scrubbing" the song.

Scrubbing Metadata

The most complete and easiest method I have found it sadly only available on Windows (and I believe Non-Intel Macs.) If you do happen to have only an Intel Mac, you can run this on a Windows partition (I set up mine using Boot Camp.) This method involves using the free program JHymn. You can find the download archives for the program HERE (http://www.hymn-project.org/archive.php).
The second method is available only for Macs, but can be used on both PPC and Intel machines. This involves using the program Fission. While this program is not free, it can be found here at Waffles (http://waffles.fm/details.php?id=76305). This program is a useful tool for lossless editing as well.
The final method, while not as complete as the previous two, is cross-platform (that's the upside.) It uses the program Atomic Parsley. You can find the homepage HERE (http://atomicparsley.sourceforge.net/). It involves using the terminal (Mac) or the command prompt (Windows.) While I will be able to give you a walkthrough for the terminal version, I do not know how to for the command prompt; if anyone wishes to add to this, I'd be grateful.

JHymn on Windows/PPC Mac
Download and install the program. Then open it and set its main window's options to match the picture below:

http://img102.imageshack.us/img102/5384/jhymngu0.jpg

Then open the preferences window and set it up to match the picture below:

http://img102.imageshack.us/img102/37/jhymnprefsob8.jpg

The "unwanted atoms" box should have these values (can't see them all in the picture):

{apID}
{cprt}
{cnID}
{atID}
{plID}
{geID}
{sfID}
{akID}
{purd}
{iods}

All that is left is to drag and drop the file(s) you want to scrub into the white box on the application's main window and hit "Convert." This will send a backup copy of the original song(s) (including metadata) to the backup folder specified (I have mine as "~\My Documents\My Music\JHymn Backup") and replace the file(s) you dragged with newly scrubbed copies. The audio information remains untouched.

Fission on PPC & Intel Macs
The unique thing about the application Fission is that it can edit music files losslessly (much like Audacity) but can also save files (losslessly) without having to export the audio data; if you were to use Audacity, you would have to export the edited file, resulting in a transcode. Fission can also be used for scrubbing iTunes Plus files because of its ability to read AAC/M4A files, again unlike audacity.

The Fission code is able to take in and recognize the most essential metadata/tags from your song files (including Album Art) but it was never designed to recognize the embedded data that the Apple company put into its iTunes Plus releases. Thus when you open one in Fission, it just simply leaves out all the extra info. So to scrub an iTunes Plus song simply open it in Fission, choose File > Save Audio..., select Save AAC (Original Format, Lossless), and click the Save button:

http://img510.imageshack.us/img510/2231/fissionsaveme7.jpg

The downside to this method is that you can only do one song at a time, whereas JHymn can do batch-processing.

Atomic Parsley (Cross-Platform)
The nice thing about Atomic Parsley is that it can be used on both Windows and OS X machines. There are a number of GUI programs that use A.P. but none of them are able to remove as much metadata as you can by hand at the terminal or command prompt. When you download the program it comes as a .zip package. When you expand this you'll find a bunch of files, some of which are helpful in learning your way around the terminal and command prompt interface. I will now help you through the terminal scrubbing process.

(1) First off open the terminal application. It can be found in "Your HD > Applications > Utilities."
(2) Unzip and open the Atomic Parsley folder (if you haven't already.)
(3) Locate the file you wish to scrub the metadata from.
(4) Drag and drop the AtomicParsley executable onto the terminal.

http://img266.imageshack.us/img266/5391/atomicdragem6.jpg

(5) Drag and drop the song file onto the terminal.

http://img403.imageshack.us/img403/5392/atomicfileps2.jpg

(6) The terminal window now should have a very long line of text on it. Without hitting the spacebar you can now type in "-t" and hit return. This will display the individual "atoms" of metadata embedded in the song file. It should resemble this:

http://img46.imageshack.us/img46/4966/terminalthf2.jpg

If you type "-T" instead, you will see the physical layout of all the atoms in the file. It looks like this:

Atom ftyp @ 0 of size: 32, ends @ 32
Atom moov @ 32 of size: 373272, ends @ 373304
Atom mvhd @ 40 of size: 108, ends @ 148
Atom trak @ 148 of size: 66197, ends @ 66345
Atom tkhd @ 156 of size: 92, ends @ 248
Atom mdia @ 248 of size: 66097, ends @ 66345
Atom mdhd @ 256 of size: 32, ends @ 288
Atom hdlr @ 288 of size: 34, ends @ 322
Atom minf @ 322 of size: 66023, ends @ 66345
Atom smhd @ 330 of size: 16, ends @ 346
Atom dinf @ 346 of size: 36, ends @ 382
Atom dref @ 354 of size: 28, ends @ 382
Atom stbl @ 382 of size: 65963, ends @ 66345
Atom stsd @ 390 of size: 32871, ends @ 33261
Atom mp4a @ 406 of size: 32855, ends @ 33261
Atom esds @ 442 of size: 51, ends @ 493
Atom pinf @ 493 of size: 32768, ends @ 33261 ~
Atom stts @ 33261 of size: 24, ends @ 33285
Atom stsc @ 33285 of size: 40, ends @ 33325
Atom stsz @ 33325 of size: 31504, ends @ 64829
Atom stco @ 64829 of size: 1516, ends @ 66345
Atom udta @ 66345 of size: 306959, ends @ 373304
Atom meta @ 66353 of size: 306951, ends @ 373304
Atom hdlr @ 66365 of size: 34, ends @ 66399
Atom ilst @ 66399 of size: 304886, ends @ 371285
Atom ---- @ 66407 of size: 76, ends @ 66483
Atom mean @ 66415 of size: 28, ends @ 66443
Atom name @ 66443 of size: 16, ends @ 66459
Atom data @ 66459 of size: 24, ends @ 66483
Atom ---- @ 66483 of size: 162, ends @ 66645
Atom mean @ 66491 of size: 28, ends @ 66519
Atom name @ 66519 of size: 20, ends @ 66539
Atom data @ 66539 of size: 106, ends @ 66645
Atom ---- @ 66645 of size: 188, ends @ 66833
Atom mean @ 66653 of size: 28, ends @ 66681
Atom name @ 66681 of size: 20, ends @ 66701
Atom data @ 66701 of size: 132, ends @ 66833
Atom ©nam @ 66833 of size: 58, ends @ 66891
Atom data @ 66841 of size: 50, ends @ 66891
Atom ©ART @ 66891 of size: 28, ends @ 66919
Atom data @ 66899 of size: 20, ends @ 66919
Atom aART @ 66919 of size: 28, ends @ 66947
Atom data @ 66927 of size: 20, ends @ 66947
Atom ©alb @ 66947 of size: 32, ends @ 66979
Atom data @ 66955 of size: 24, ends @ 66979
Atom trkn @ 66979 of size: 32, ends @ 67011
Atom data @ 66987 of size: 24, ends @ 67011
Atom disk @ 67011 of size: 30, ends @ 67041
Atom data @ 67019 of size: 22, ends @ 67041
Atom ©day @ 67041 of size: 44, ends @ 67085
Atom data @ 67049 of size: 36, ends @ 67085
Atom pgap @ 67085 of size: 25, ends @ 67110
Atom data @ 67093 of size: 17, ends @ 67110
Atom apID @ 67110 of size: 44, ends @ 67154
Atom data @ 67118 of size: 36, ends @ 67154
Atom cprt @ 67154 of size: 28, ends @ 67182
Atom data @ 67162 of size: 20, ends @ 67182
Atom cnID @ 67182 of size: 28, ends @ 67210
Atom data @ 67190 of size: 20, ends @ 67210
Atom rtng @ 67210 of size: 25, ends @ 67235
Atom data @ 67218 of size: 17, ends @ 67235
Atom atID @ 67235 of size: 28, ends @ 67263
Atom data @ 67243 of size: 20, ends @ 67263
Atom plID @ 67263 of size: 32, ends @ 67295
Atom data @ 67271 of size: 24, ends @ 67295
Atom geID @ 67295 of size: 28, ends @ 67323
Atom data @ 67303 of size: 20, ends @ 67323
Atom sfID @ 67323 of size: 28, ends @ 67351
Atom data @ 67331 of size: 20, ends @ 67351
Atom akID @ 67351 of size: 25, ends @ 67376
Atom data @ 67359 of size: 17, ends @ 67376
Atom stik @ 67376 of size: 25, ends @ 67401
Atom data @ 67384 of size: 17, ends @ 67401
Atom purd @ 67401 of size: 43, ends @ 67444
Atom data @ 67409 of size: 35, ends @ 67444
Atom covr @ 67444 of size: 303812, ends @ 371256
Atom data @ 67452 of size: 303804, ends @ 371256
Atom ©cmt @ 371256 of size: 29, ends @ 371285
Atom data @ 371264 of size: 21, ends @ 371285
Atom free @ 371285 of size: 2019, ends @ 373304
Atom free @ 373304 of size: 25126, ends @ 398430
Atom mdat @ 398430 of size: 5851649, ends @ 6250079

~ denotes an unknown atom
------------------------------------------------------
Total size: 6250079 bytes; 81 atoms total. AtomicParsley version: 0.9.0 (utf8)
Media data: 5851649 bytes; 398430 bytes all other atoms (6.375% atom overhead).
Total free atom space: 27145 bytes; 0.434% waste. Padding available: 27145 bytes.

It should be noted that the ~ in this file is the {iods} tag. It is recognized and can be removed by JHymn, and is also scrubbed with the Fission application. Yet Atomic Parsley cannot recognize it and thus cannot remove it. I am unsure as to how important it is that this be removed. Depending on how you feel, you may wish to find another method of scrubbing.

(7) Repeat steps 4 & 5. Now type into the terminal --manualAtomRemove "moov.udta.meta.ilst.apID" --manualAtomRemove "moov.udta.meta.ilst.cprt" --manualAtomRemove "moov.udta.meta.ilst.cnID" --manualAtomRemove "moov.udta.meta.ilst.atID" --manualAtomRemove "moov.udta.meta.ilst.plID" --manualAtomRemove "moov.udta.meta.ilst.geID" --manualAtomRemove "moov.udta.meta.ilst.sfID" --manualAtomRemove "moov.udta.meta.ilst.akID" --manualAtomRemove "moov.udta.meta.ilst.purd" and hit return (luckily copy & paste works in the terminal.) This will make a copy of the file (with the word "temp" attached to the end of the filename) with all the specified atoms scrubbed. The original remains unchanged.

Again, the downsides are the inability to remove the {iods} tag (I don't know what data it contains) and to process multiple files at once. Other than that it is as good as JHymn.

Aside from scrubbing, Atomic Parsley comes in handy for verifying the scrubbing of other programs by using the "-t" and "-T" commands. You can also use it to edit some of those stubborn tags (like episode number, parental advisory, etc.) in your iTunes songs and videos. Just use the read me and website for help with these.

Extra Uploading Cautions (Mutts and Transcodes)

Just because you buy an iTunes Plus song that is encoded as an AAC at 256 kbps doesn't mean that if YOU encode other songs in iTunes with the same parameters that the files will match. If you upload a mixture of AAC files from the iTMS and AAC files that you encoded it will be reported as a "mutt" upload. For example, I got a hold of the exclusive pre-order track from the new Korn CD. I cannot upload this single song however with the rest of the CD that I encoded myself with iTunes, even if they are the same bitrate and file type.

Also, if you don't purchase the iTunes song yourself, just because it is at the right bitrate and filetype, doesn't necessarily mean it is not a transcode. You must verify that it is legit by looking at the spectral analysis. If you are on Windows then just look in the How to spot transcodes (http://waffles.fm/forums.php?action=viewtopic&topicid=465) for help. Otherwise, if you are running OS X, I have found the best program to be Soundtrack Pro (part of the Final Cut Studio suite.) Just open the file in Soundtrack Pro, and click the little spectrograph image in the top right corner:

http://img518.imageshack.us/img518/5043/stpcornerdd9.jpg

A non-transcode should look like the picture below, with the top cutoff of frequencies around 20,000Hz and middle frequency bands around 16,000Hz.

http://img521.imageshack.us/img521/1391/mysonggz4.jpg

Hope this helps! I would love to see some of the iTunes exclusives uploaded to this site. They would make great additions.
And just a word of warning, back at OiNK, I saw at least 1 or 2 screw ups a week where people uploaded stuff from iTunes they had no idea about.

Although on second though, I'm not sure it'll be of much help, since this is for audio files and you appear to be talking about video.

[mjmacky]

Yeah sorry, I should have clarified, only hinted at it with the codec comment. The thing is I've put some releases out there in the world, and I'm worried about it coming back to my original purchase.

[Expeto]

You can always say, that you was hacked and somebody stole your files.

[mjmacky]

You can always say, that you was hacked and somebody stole your files.

@Expeto
cross my fingers and hope that works

The thing is, I don't know where to ask a drm expert. Those guys that do the web releases usually use iTunes as source, which I just don't touch.

[Expeto]

as anon said guys at the waffles pretty good at that subject, but an invite is pretty hard to get.

What was the song you uploaded?
Chances are with you both ways; if you uploaded a very big hit, chances are likely somebody also did it and done a way better job than you. Yours will disappear in no time.
If it wasn't a very famous song , probably copyright guys won't even notice.

What are you scared of anyway? They can't even make a case out of this, just a guy sharing a song. They even have hard time building cases against release groups. So don't worry about legal things. But you might loose your itunes account in worst case, which is not very likely. But I'm no expert of itunes, check the TOS or whatever agreement you signed for itunes for details.

[mjmacky]

as anon said guys at the waffles pretty good at that subject, but an invite is pretty hard to get.

What was the song you uploaded?
Chances are with you both ways; if you uploaded a very big hit, chances are likely somebody also did it and done a way better job than you. Yours will disappear in no time.
If it wasn't a very famous song , probably copyright guys won't even notice.

What are you scared of anyway? They can't even make a case out of this, just a guy sharing a song. They even have hard time building cases against release groups. So don't worry about legal things. But you might loose your itunes account in worst case, which is not very likely. But I'm no expert of itunes, check the TOS or whatever agreement you signed for itunes for details.

Well it's not music, it's video. However, the concept should be pretty much the same, whether or not my information is embedded in the file. I've been releasing TV shows from Zune as web-dl's. Without explicitly stating what I release under, one of my nzb's showed up on this site's NZBs section. I do it when the other main groups seem to be delayed. I put the DRM video through Tunebite, and package the resulting MP4 with mkvmerge. I don't want my Zune account to be messed with since it's tied to my xbox 360, know what I mean. I still like to pay for stuff that I support whenever I feel I can afford it, but I just can't ignore my hobbies ;)

I went over to the Waffles IRC help channel, but no one there was able to help me. This kind of information isn't easily googleable I guess.

anon, i have to thank you for that tutorial, it led me onto a path of research that has allowed me to understand this better

WMV, the Zune format, does have metadata tags
MP4, metadata can be edited/added/scrubbed using Atomic Parsley
MKV, metadata conversion?

So, assuming that metadata tags get carried from WMV to MP4 by Tunebite, which I doubt due to the nature of the conversion (essentially a screencap encode), those would be lost once I mux them to MKV using mkvmerge because I'm not using an XML table for it (unless it auto imports from MP4). I wish someone could come by and tell me I'm right.

[anon]

as anon said guys at the waffles pretty good at that subject, but an invite is pretty hard to get.

Are you sure? They're pretty easy to get into when invites are open. I'm Power User there myself, and have 5 invites. The option of exchanging a regular one for two anytime ones was also recently made available in their "shop".


anon, i have to thank you for that tutorial, it led me onto a path of research that has allowed me to understand this better

Glad I could help, even though I can't really give a helpful reply on the matter :)

[mjmacky]

As I understand, waffles focuses on music, which I'm not a big music person. I wish I was really, but when it comes down to it I usually just listen to it when I'm driving. Even then I still get quickly tired of my small collection.