PDA

View Full Version : Corruption Finding & Fixing



ilw
09-29-2003, 11:46 AM
Hey
I've written a few programs for kazaa so i thought i'd post an update on what i've done:


MP3 Checkers

These two programs fulfill different requirements, MP3 checker is designed to check all your existing MP3's and remove any screechers from your existing collection and the autoscanner is designed to stop any more screechers getting into your collection (and thereby helping to remove them from the fasttrack network). Neither of them are perfect and will allow some to slip through and (although i haven't seen any) they may give false alarms.

MP3 Checker is a GUI based program which can scan MP3's for the tell-tale signs of RIAA added screeching. It can scan around 800 MP3s at a time and can do a full scan in under a minute.

MP3 Autoscanner, is a command line program designed to work with the KLExtension's file organiser so that MP3's are scanned and if bad are quarantined as soon as they are downloaded.

Other Anti Corruption programs


My other anti corruption programs are based on the same principle as the Intelligent Corruption Handling available in better maintained p2p networks such as the edonkey network. The idea is to calculate CRC (Cyclic Redundancy Check) information for chunks of a file, this means that each chunk has been given a unique identifier based on the data contained in it. Any errors or changes to this data gives a different CRC value and so chunks can be easily checked for corruption. The way I hoped this would turn out was that lots of people (especially those who often contribute verifieds) would convert to KSig mod and use it to generate the modified quicklinks which contain the CRC hashes. Then anyone downloading one of these verified files can CRC check their finished files and if there is any corruption then Kchecker will see it and redownload that part of the file(with 2% resolution).

Afaik if a verified file you downloaded is corrupt then you have 3 choices

1) To ask someone who has a good version of the file to crc hash it with one of my programs
2) If you know where in the file the error is (eg if its a video file the errors are visible and rough file position can be estimated) then you can use DrSpud's KCF and use that to redownload the bad parts of the file.
3) Junk the file and redownload it all.

The programs

KChecker: KChecker is my main program, it generates crc information for files, compares crc hashes to determine corrupted chunks and generates a dat file so that kazaa will redownload the bad chunks. It is capable of checking finished and ongoing downloads (though it only checks the first good chunk of ongoing downloads). If Ksig mod has been used on a crc quicklink then this crc information is stored and is automatically checked by KChecker.

KSig Mod: This is basically just a modification of Hasnain's KSig program it is modified to also calculate the crc information that Kchecker uses and also integrates this information in backwards compatible quicklinks. (NB its based on the previous release of ksig, I haven't got round to updating to hasnain's latest release)

KPadder: KPadder is just a small program to allow you to change the size of a file, this is useful if you lost a file due to a corrupted dat appendix (eg after a power cut). See the kchecker page for more details on how you would go about recovering a deleted file.


More info, rough guides and downloads from my websites http://www.ilwsoftware.tk/ OR (TEMPORARILY NOT UPDATED http://www.freewebs.com/ian_l_williams/ (http://www.freewebs.com/ian_l_williams/)

I only recently did the websites so theres lots of spelling errors, grammatical mistakes and generally the written content is pretty crap, i'll sort it out soon so you don't have to post any typos you see :P If you see any broken links though please tell me.

sharedholder
09-29-2003, 11:51 AM
:D Good work,and nice sites too!Thank you.

MUSLEMAN
10-01-2003, 06:41 AM
i'm posting that sucker every chance i get its great, good job boss :lol:
wats up sh :lol: :lol:

Livy
10-01-2003, 01:59 PM
ilw what about having a look here http://www.klboard.ath.cx/index.php?showtopic=70807

perhaps u could work together, or add some command switches to mp3checker so u can add it to check files after they download using kl extensions organiser

Muse
10-03-2003, 08:02 PM
Nice work ILW

I like the concept of the autoscanner. I works great but not with all messed up files :(

I have found a screecher that slips through. It's one of the Timberlake fakes:

File:CD Quality LILY.mp3
Length:3952037 Bytes, 3859KB
UUHash:=vzxPsuiX/kvjk1qdtJrMc5iReag=

Maybe you use this example to improve your autoscanner?

Another problem is that not all fakes are screechers.

Maybe we should fight these polluters by their own means? If you download a fake, rename the file to "Don't download this fake" and keep sharing it. These renamed files will show up in the list of identical search results and anyone can recognize a fake before you download it.

ilw
10-04-2003, 11:55 AM
Thanks for the feedback & i saw the other post about this, but decided to reply here. Btw thanks sharedholder :D
I tried the file, but i must have got a more corrupted version than you cos it found it on my computer. I've increased the amount of scanning so that it may take a tiny bit longer but is more likely to spot a fake, hopefully it should now spot it. (Do you by any chance still have the file so you can test it?)
I've also changed it so that it renames the files to
This_is_a_FAKE_filename
I don't think i can fix the bug where File Organiser repeatedly scans the file because if i change the file extension then the file won't show in searches and so there would've been no point of changing the filename to show its a fake and this was an idea that i kind of liked. Even though the RIAA can easily start saying using this to discredit valid files, they would still have to be sharing the file in order for them to say its a fake and if they're sharing it then chances are it is fake (their high bandwidths mean they can spread fakes very quickly)
NB I've now made it clear on the website that you shouldn't have the download directory as your 'move to' directory.

Muse
10-04-2003, 12:16 PM
OK ILW.

Is this changed version available for donwload? I'd like to try it.

Glad you like my idea about sharing the fakes with "This is a fake"filename.
Hope we can persuade more users to adapt this method.

ilw
10-04-2003, 12:22 PM
Yeah, I updated the geocities website (in my sig)
I'll change the name of the executable so its more obvious which version you have.

nikita69
10-07-2003, 09:27 PM
yes, great work ilw and i created some tips about them in my tips topic. :)

quentarez
10-17-2003, 08:58 AM
I absolutely love the mp3 autoscanner and checker!

I was wondering if you could possibly try to do something for me? The autoscanner, each time a file is complete, scans it, but it also pops up a console window, which is, to me (im a nut about this kinda stuff), very aggravating.

I kinda bypassed it by making a shortcut to the autoscanner exe, and telling the shortcut to run minimized, but I was wondering if maybe you could have it run the exe as a background process instead of a console app like that? I don't even know if this is possible, but I thought I might as well suggest it anyway!

Thanks again for these awesome progs!!!!

quentarez
10-17-2003, 09:02 AM
just had an idea about the autoscanner repeatedly searching the file if you have the directory set to your download directory

does it rename the file and THEN move it, or does it move the file and then rename it?

i thought that maybe if you had your checker move the file to the quarantine folder first, and THEN rename it, it wouldnt trip up like that?

ilw
10-17-2003, 09:33 AM
The rename and move are all done in one step (basically changing the file's path to include a different directory is equivalent to moving it) the reason it keeps rescanning is because the file organiser periodically checks the download directory for any files with the extensions you specify. So if its setup to not move the file from the download directory then each time the organiser runs it sees the file and carries out the action its supposed to (ie scans it). I could stop the organiser from rechecking a file by renaming the file to have a different extension, but then the file might not be shared...

dhaulk
12-15-2003, 11:30 PM
I love the auto scanner first off but I think I instaled it wrong because songs that aren't sreechers get put in quarinteen anyway. I followed the directions exactly from the site but I noticed you said in the last step to type space "%F" but in the example it shows space "%FN" . Is that maybe what is causing the problem should it be "%F" or"%FN" ? :unsure: