the lost art of downloading headers

[zot]

Downloading headers seems to be a lost art. As most people use NZBs these days to get files, and since a lot of popular clients like Sabnzbd and Altbinz don't even support headers, many users may scarcely know what headers even are.

Compounding the problem is the fact that the vast majority of newsreaders (with header support) simply can't handle the vast collection of headers that exists in most binary groups, as they neither support header compression nor contain even a halfway-efficient database engine.

Early-generation binary newsreaders like Grabit and Xnews have been known to crash and burn when deluged by massive header fetches. Newsbin, Usenet Explorer, and Newsleecher are the main ones I know of that are capable clients. (I don't think any decent free ones exist).

Another problem is header completion. I've noticed lately that Highwinds seems to be having chronic problems with missing headers; old posts have missing headers (as well as bodies) and new posts often don't show up until much later. So a newsreader with multi-server (fill) support becomes very helpful.

Even without the benefit of a top-notch newsreader, downloading headers is simply the best way to browse a specialized newsgroup containing mostly small files, and most any newsreader (ideally with yenc and uuencode binary support) will work. Mp3 genres and e-book groups are a perfect fit. There's also the thousands of specialized porn fetish groups (many are not indexed in NZB search sites) but we won't go there.

[Hypatia]

i use headers for certain groups Theit retention is set just to 1 day and update rate is set to several minutes
After 1 day everything is flushed out lol

header retention of GN and AW are almost the same 960+ days

Nowadays i mostly prefer to use mysterbin since it can show the contents of archives+ built in search engines because they index stuff really fast

[B18C5]

The major news servers all support header compression now too. Meaning you can download headers at up to 10 times your net speed using 1/10 of the bandwidth they used to take to download. Headers really make sense for picture groups, music groups, books groups and perhaps the porno groups. I don't use NZB's at all.

header retention of GN and AW are almost the same 960+ days

A dirty secret for many news server is that they advertise "1000 days retention" but, can only deliver headers for part of that span. I know Giganews can deliver headers for their full retention. I know that a number of the lessor servers can't.

[zot]

Although all the major 'American' pay-providers like Giganews, Highwinds, Astraweb, and Readnews generally set header retention only slightly shorter than body retention, a lot of usenet providers do not offer anywhere near full header retention. Every brand of ISP-provided usenet service I'm aware of has limited headers to about 1 or 2 weeks regardless of the body retention. The Dutch pay-providers also have severely-limited header retention.

Regarding header compression, NewsDemon users might be surprised to learn they must pay extra to get header compression since NewsDemon (as with other Highwinds and also Readnews resellers) only supports one of the three header compression standards - zlib over ssl. I don't understand what is taking Highwinds and Readnews so long to adopt non-SSL compression (since zlib is so resource-hogging and "compresses" everything, even uncompressible rar file downloads). The last time I checked, it seemed that Usenetserver and Easynews supported the 'xfeature' compression standard but Newshosting did not - even though all 3 brands are owned by the same company, Highwinds.

[nntpjunkie]

The last time I checked, it seemed that Usenetserver and Easynews supported the 'xfeature' compression standard but Newshosting did not - even though all 3 brands are owned by the same company, Highwinds.

Newhosting does support xfeature - see below:

XXXXXXX-MacBook-Pro:~ user_$ telnet nhxl.newshosting.com 119
Trying 209.197.12.12...
Connected to news.iad.newshosting.com.
Escape character is '^]'.
200 NNRP unlimited.newshosting.com Service Ready - [email protected] (fed14.iad)
authinfo user XXXXXX
381 PASS required
authinfo pass XXXXXXX
281 Welcome to NH bro! (Posting Allowed)
xfeature compress gzip
290 Feature Enabled
group alt.binaries.divx
211 24711638 49939654 74651291 alt.binaries.divx
xover 74651291-74651291
224 xover information follows [COMPRESS=GZIP]
x?UP[K?0}????/֤M???
a(Sa(>d??.?&#M???[?}8??p.?,I?sF?/_??1??;硰5???P??>??r? W0Z??C(?m?<?p?i?

all providers including Giganews have gaps in their headers, the gaps don't matter because they will eventually disappear as time passes anyone who download the full span of headers is a retard - who in the heck want to download 100s of gigs of headers from say boneless only to have their machine crash and burn as they try to search through that amount of data.....no one does this anymore....we still have type writers, but do you actually think people still use them....hell no!

[nntpjunkie]

Has any body here used the news client Ozum?

[B18C5]

the gaps don't matter because they will eventually disappear as time passes

Maybe in the old days. Giganews isn't purging headers any more. Basically their retention is growing one day for each day that passes. It's not clear to me how long they can sustain it.

I agree that headers for boneless is impractical for most people but, it's doable if you don't mind waiting. It's the worst case though. If you can sort of do headers with boneless, that means every other group will be easier. It's less trouble for me to download headers from some of my groups and load them, then it is to go to a website and download NZB's. As long as that's the case, I see no reason for me to use NZB's.

I don't think it's "typewriter versus computer", I think it's more "manual transmission versus automatic". Some people like shifting gears and some people like to just step on the gas.

[heiska]

the gaps don't matter because they will eventually disappear as time passes

Maybe in the old days. Giganews isn't purging headers any more. Basically their retention is growing one day for each day that passes. It's not clear to me how long they can sustain it.

....

They can sustain it as long as they are capable of finding people willing to pay $324 [($35*12)-$96] extra per year for the extra 11 days of advertised retention

[nntpjunkie]

maybe in the old days. Giganews isn't purging headers any more. Basically their retention is growing one day for each day that passes. It's not clear to me how long they can sustain it.

....

they can sustain it as long as they are capable of finding people willing to pay $324 [($35*12)-$96] extra per year for the extra 11 days of advertised retention

Bwhahahahahahahahahaha!!

[B18C5]

They can sustain it as long as they are capable of finding people willing to pay $324 [($35*12)-$96] extra per year for the extra 11 days of advertised retention

I don't think they can sustain it from a technical perspective. The lesser news server already seem to be groaning trying to support this, fairly insane, retention. I was doing some testing and Giga can take a message-id, one of billions on the server, do the look up and start returning the data in under 100 ms. That's pretty incredible. It's about double-that to the EU server but, that seems to be added transport time. I haven't tried it with other servers yet.

Their pricing for the max tier is sort of crazy but, again, like boneless, that's worst case. The $24.99 platinum plan, with SSL and full retention is more than anyone needs. Not sure I download 50 gigs in a month so, I might even be able to make do with the Silver at $14.99 a month. I like how it's always there and it always works. I'm willing to pay extra for good service.