Headers? Groupnames? Usenets jungle...

[Azz82]

Hi there all =)

May I recive a warm welcome and well lets get to it!

As you can see, I am new here at filesharingtalk I really like the site and I'm not new to a lot of the stuff ppl are talking about here. I am definetly new at the whole usenet bizz, so I hope you will help me out a little...

I have been reading up on usenet for the last days and used a lot of time to get, somewhat, to the bottom of it.

I would like to understand it, so I can get the best of it, instead of relying on indexing sites. I wanna go into the jungle myself and understand what headers are and how I find or use groupnames on my own...

A)I've been seeing a lot of talk about these headers and how I should learn to look for them or at them, but I can't seem to find a nice clean explanation of WHAT or WHERE it exactly is or how I use it, like in totally noob perspective, it might just be a "stupid" question, but in my world there are no stupid question, only ppl who do not understand and really want to

ppl have a tendency to forget, what it was like, when they was all noobish

I have also come across a lot of forums and especially here, where ppl are talking about a.b.dvd or a.b.multimedia and talking about usenet groups and how one are supposed to look out after headers and those groups, but once again, how, explained to a noob, nevertheless a fast-learning noob, so I don't have to

I use newshoting and newsleecher. Have been experimenting with newsleecher and found it to be very comprehensive, since I started out with grabit.
B)What I found out so far is that I can get a complete list of groups (A LOT) and see them in my newsleecher. So I experiment and try to put in a.b.dvd or some groupname that rls nice dvd rls and it do not find it in quick search, what do I not understand, is it because I should look elsewhere or use other methods ?
C)Another question: Are the grouplist in my newsleecher, when I am connected to my newshost, only listing the groups on my newshost? Does this mean, I can't download or find rls rlsd at other newshosts or is usenet totally connected, if something is available at a newshost or on usenet, its teoretically available to everyone who has a newshost, that is, if they can find it ???

This is turning into a long lone, but why not, I allways think of it beeing helpfull to some other noob, since noob-thoughts are often alike

I know I can subscribe to the groups I see, go into them, but all I see if just a lot of files, that I can't make a lot off, but something is telling me I should...

All cooked down, what I want to know is, how do I get down and dirty on my own useing usenet the oldschool header-search-find kinda way (saying there is one ), caus I feel I am reading about it, between the lines, but I need some puzzles to make it complete...

I hope you wil help me out

Damn, I must really be stupid, since no one will help me or should I just be patient

[Beck38]

The first thing up is to understand the hierarchy:

News-Servers (pretty self-explanatory)

News-Groups (again, kinda; lists of the groups can be got from your news-server, but they may not (and aren't required to) carry all of them. If your provider of choice doesn't, you can usually ask them to 'add' that group to their offerings.

Headers or, Message-Headers. ALL entries into a news-group has a 'header', a kind of 'subject' line.

Message Body - obviously, either the text (or binary) part of the 'message'

Many 'search' sites do filtering of the Headers, up to the level of Newzbin, which does both automated and human filtering of binaries groups, and invented the nzb format for automated downloading of binary postings.

Most decent news-reader programs do semi-automated filtering themselves, as well.

Note that in reality, Usenet is a text-based electronic bucket-brigade, where messages are passed from news-server to news-server, through a system of 'peering'. Early on (mid to late 1980's), it was figured out that the system could 'carry' binary material (first small programs, then audio/mp3's, then video/dvd's).

To enhance this transport, the yenc encoding format was developed, which has pretty much replaced the uucp (unix-to-unix copy) system of which there were various formats to expedite the transport of binary material.

Note: There are some holdovers from the 'early days', one of those is how many bits (parts) can be included for each PART of the posting. Generally, you'll notice that the parts are 50MB (50 Mega-Bytes) per, and that limitation was due to the s/w that the news-servers were running. That has somewhat gone away (one can easily find that upwards of 500MB parts are being done), but not all servers are capable of doing this. So folks posting such bloated parts are not necessarily going to be properly carried by all news-servers.

More recently, the Newzbin folks came up with the NZB format, which has pretty much revolutionized usenet binary transport. Any newsgroup reader these days supports it; it's the electronic equivalent of a library card catalog listing, in that it lists (for transfer/download) all the files (hopefully!) necessary to fully transfer the file requested.

If one looks into the nzb 'file' (it's standard ascii and can be opened with any text editor), it list the newsgroup, file headers, and file parts that the news-reader will download.

Also, a system of parity needs to be used to transport these huge binary files; the internet is NOT 100% perfect, of course. The standard system in use today is Quickpar, which ensures that the data is 100%. I hope I don't have to explain parity systems to you...it just works, IF the poster provides enough parity parts to 'fix' the transmission errors you have in the parts you've d/l'ed.

I'm sure most folks simply click on the nzb and stand back, but the way the whole thing works is interesting, and really, if things go wrong (uhhh, sometimes do!), it's nice to know how they are supposed to work.

It's all really pretty straightforward.


LIMITATIONS

Oh, there's always some, in any system!

The biggest (and usually first) to come up is...

Retention

Just how long will that news-server retain that block of headers/messages it just got from it's 'peers'. That is the bone of what makes a server good, particularly when usenet now carries (there are sites that track the traffic you can find), somewhere around 3-4TB (3-4 Terrabytes!) a DAY. The ultra-premium Giganews advertises well over 200 days of retention, while it's competitors are less (but ramping up quickly).

Completion

Of course, having several hundred days of retention is no good if the files aren't any good; excellent peering of the server really helps here, so..

Transfers/Connections

News-servers will list the number of 'connections' your account is allowed; generally, the more the better, but as the hardware/software of their plant is improved, it's sometimes not an issue. More of an issue may be the number of 'hops' and 'delay' on the internet between you and that news-server. Most servers also do a 'reverse traceroute' to give you an idea of the way the data gets from them, to you.


Okay, that's enough. I'm sure most of this is in a 'sticky' here somewhere, but what the hey...

Final Thoughts...

Some folks NEVER d/l headers, it is kind of a throwback to the early days. Use sites like this, Newzbin, and the host of others, to do your searching for you. A lot of use 'old timers' still do a bit, on the groups that they like, not only for 'old times sake', but that those indexing sites do miss quite a lot of 'good stuff'.

Some news-readers have been adding 'hyper header' d/l systems that really speeds things up, IF the server supports it (the two majors, Giganews and Astraweb, both do).

Eventually, the 'torrents v. usenet' argument gets started up. Yes, with torrents, essentially 'everything is forever' if enough people/machines are sharing everything they have, 100% of the time. Right. About the time everyone has access to 100Mb/s internet connections, sometime around y3K!

Meanwhile, usenet has (admittedly, does cost bucks up front) these huge commercial servers doing the heavy lifting. Kinda like Napster was very early in the peer-to-peer world, say circa 1998? History... to me, trying to get things off the torrents is just too much da*n trouble. You find what you're looking for, then halfway through the transfer, all the host sites 'wink out'. You find it again several days later, start back up, then they 'wink out' yet again.

Any questions?

[Azz82]

Wow... That was very usefull, to give me a look into the history and understanding it, somewhat =)

Well, you shouldn't have to have an extra ticket to the galla at newzbin =D Seems a little to hard to get a foot inside that door, but what the heck, one day I'm sure I will...

That brings me to a question, what will you propose I do, if I want to go explore groups and headers? What can I use a header for or the information in it, it's just an indicator on what is actually in the group or has been rlsd? Are there some links or text to rls I can actually download in headers? I'm just asking out of purely curiosity =P Because I just don't know any better, but I do understand everything you wrote me in the post and I can use the different indexing sites, NZB files and I like to belive I know how to use my newsleecher program, maybe not to its full content, but some day =)

I payed for 1 month at newshosting.com, since a friend of mine who do not know that much in depth about usenet reccomended it to me =) But I do not know if they have hyper header thingie you talked about, they do have something they call "Lightning accelerator" where I download a program called stunnel, add some lines in the config and set it up with my newsreader and I should be secure, but that might just be SSL or? I don't know much about stunnel...

But I just love to get to know more, go deep into something and understand it =) I also got VERY VERY tired of all the torrent fuss, sharing and sharing and sharing and all them RUUUUUUUUUUUULES, damn =S Used to be in dha scene for a lot of years, those where the days, as a VIP sponsor =) Usenet reminds of that =P

But once again thanks for taking the time to answer me, very much appreciated!!!

I can see I got an invite, very nice, so don't waste one on me =)

[Beck38]

You probably need to start off, if you want to delve deep into the headers, by some simple filtering.

Construct a filter for nfo's (*.nfo); most (but obviously not all) postings have an information file. Or nzb's (*.nzb). Both should be easy to filter for, IF your newsreader can do it. Otherwise, you'll be stuck with the filter sites (Newsbin and others) to do your searching on.

From there, you can start 'training' yourself to pick out things from more non-specific filtering.

[Azz82]

I see... Thanks m8 and what newsprovider would you recommend, astraweb, caus they don't keep any records?

[SonsOfLiberty]

Newzbin is overated when there are so MANY free index's out there. Beside who wants to be part of a site that has a lawsuit aganist them by the MPAA?

[Azz82]

oh, well... Lawsuits my azz =) We will see how it goes, besides, the most cases are lost and mostly they just sue like other ppl, heh hmm.

I'm sure they will hire some good lawyer and maybe it will take 4ever before we hear a sentence, so until then I am a happy user of newzbin, why not, so easy! Whats the cost a pack of cigarettes, drop em and you got newzbin 4life =D