the lost art of downloading headers

[zot]

Downloading headers seems to be a lost art. As most people use NZBs these days to get files, and since a lot of popular clients like Sabnzbd and Altbinz don't even support headers, many users may scarcely know what headers even are.

Compounding the problem is the fact that the vast majority of newsreaders (with header support) simply can't handle the vast collection of headers that exists in most binary groups, as they neither support header compression nor contain even a halfway-efficient database engine.

Early-generation binary newsreaders like Grabit and Xnews have been known to crash and burn when deluged by massive header fetches. Newsbin, Usenet Explorer, and Newsleecher are the main ones I know of that are capable clients. (I don't think any decent free ones exist).

Another problem is header completion. I've noticed lately that Highwinds seems to be having chronic problems with missing headers; old posts have missing headers (as well as bodies) and new posts often don't show up until much later. So a newsreader with multi-server (fill) support becomes very helpful.

Even without the benefit of a top-notch newsreader, downloading headers is simply the best way to browse a specialized newsgroup containing mostly small files, and most any newsreader (ideally with yenc and uuencode binary support) will work. Mp3 genres and e-book groups are a perfect fit. There's also the thousands of specialized porn fetish groups (many are not indexed in NZB search sites) but we won't go there. :lol:

[Hypatia]

i use headers for certain groups Theit retention is set just to 1 day and update rate is set to several minutes
After 1 day everything is flushed out lol

header retention of GN and AW are almost the same 960+ days

Nowadays i mostly prefer to use mysterbin since it can show the contents of archives+ built in search engines because they index stuff really fast

[B18C5]

The major news servers all support header compression now too. Meaning you can download headers at up to 10 times your net speed using 1/10 of the bandwidth they used to take to download. Headers really make sense for picture groups, music groups, books groups and perhaps the porno groups. I don't use NZB's at all.


header retention of GN and AW are almost the same 960+ days

A dirty secret for many news server is that they advertise "1000 days retention" but, can only deliver headers for part of that span. I know Giganews can deliver headers for their full retention. I know that a number of the lessor servers can't.

[zot]

Although all the major 'American' pay-providers like Giganews, Highwinds, Astraweb, and Readnews generally set header retention only slightly shorter than body retention, a lot of usenet providers do not offer anywhere near full header retention. Every brand of ISP-provided usenet service I'm aware of has limited headers to about 1 or 2 weeks regardless of the body retention. The Dutch pay-providers also have severely-limited header retention.

Regarding header compression, NewsDemon users might be surprised to learn they must pay extra to get header compression since NewsDemon (as with other Highwinds and also Readnews resellers) only supports one of the three header compression standards - zlib over ssl. I don't understand what is taking Highwinds and Readnews so long to adopt non-SSL compression (since zlib is so resource-hogging and "compresses" everything, even uncompressible rar file downloads). The last time I checked, it seemed that Usenetserver and Easynews supported the 'xfeature' compression standard but Newshosting did not - even though all 3 brands are owned by the same company, Highwinds.

[nntpjunkie]

The last time I checked, it seemed that Usenetserver and Easynews supported the 'xfeature' compression standard but Newshosting did not - even though all 3 brands are owned by the same company, Highwinds.

Newhosting does support xfeature - see below:

XXXXXXX-MacBook-Pro:~ user_$ telnet nhxl.newshosting.com 119
Trying 209.197.12.12...
Connected to news.iad.newshosting.com.
Escape character is '^]'.
200 NNRP unlimited.newshosting.com Service Ready - [email protected] (fed14.iad)
authinfo user XXXXXX
381 PASS required
authinfo pass XXXXXXX
281 Welcome to NH bro! (Posting Allowed)
xfeature compress gzip
290 Feature Enabled
group alt.binaries.divx
211 24711638 49939654 74651291 alt.binaries.divx
xover 74651291-74651291
224 xover information follows [COMPRESS=GZIP]
x?UP[K?0}????/֤M???
a(Sa(>d??.?&#M???[?}8??p.?,I?sF?/_??1??;硰5???P??>??r? W0Z??C(?m?<?p?i?

all providers including Giganews have gaps in their headers, the gaps don't matter because they will eventually disappear as time passes anyone who download the full span of headers is a retard - who in the heck want to download 100s of gigs of headers from say boneless only to have their machine crash and burn as they try to search through that amount of data.....no one does this anymore....we still have type writers, but do you actually think people still use them....hell no!

[nntpjunkie]

Has any body here used the news client Ozum?

[B18C5]

the gaps don't matter because they will eventually disappear as time passes

Maybe in the old days. Giganews isn't purging headers any more. Basically their retention is growing one day for each day that passes. It's not clear to me how long they can sustain it.

I agree that headers for boneless is impractical for most people but, it's doable if you don't mind waiting. It's the worst case though. If you can sort of do headers with boneless, that means every other group will be easier. It's less trouble for me to download headers from some of my groups and load them, then it is to go to a website and download NZB's. As long as that's the case, I see no reason for me to use NZB's.

I don't think it's "typewriter versus computer", I think it's more "manual transmission versus automatic". Some people like shifting gears and some people like to just step on the gas.

[heiska]

the gaps don't matter because they will eventually disappear as time passes

Maybe in the old days. Giganews isn't purging headers any more. Basically their retention is growing one day for each day that passes. It's not clear to me how long they can sustain it.

....

They can sustain it as long as they are capable of finding people willing to pay $324 [($35*12)-$96] extra per year for the extra 11 days of advertised retention :)

[nntpjunkie]

maybe in the old days. Giganews isn't purging headers any more. Basically their retention is growing one day for each day that passes. It's not clear to me how long they can sustain it.

....

they can sustain it as long as they are capable of finding people willing to pay $324 [($35*12)-$96] extra per year for the extra 11 days of advertised retention :)

Bwhahahahahahahahahaha!! :D

[B18C5]

They can sustain it as long as they are capable of finding people willing to pay $324 [($35*12)-$96] extra per year for the extra 11 days of advertised retention

I don't think they can sustain it from a technical perspective. The lesser news server already seem to be groaning trying to support this, fairly insane, retention. I was doing some testing and Giga can take a message-id, one of billions on the server, do the look up and start returning the data in under 100 ms. That's pretty incredible. It's about double-that to the EU server but, that seems to be added transport time. I haven't tried it with other servers yet.

Their pricing for the max tier is sort of crazy but, again, like boneless, that's worst case. The $24.99 platinum plan, with SSL and full retention is more than anyone needs. Not sure I download 50 gigs in a month so, I might even be able to make do with the Silver at $14.99 a month. I like how it's always there and it always works. I'm willing to pay extra for good service.

[nntpjunkie]

[QUOTE]
Their pricing for the max tier is sort of crazy but, again, like boneless, that's worst case. The $24.99 platinum plan, with SSL and full retention is more than anyone needs. Not sure I download 50 gigs in a month so, I might even be able to make do with the Silver at $14.99 a month. I like how it's always there and it always works. I'm willing to pay extra for good service.

I guess you just like throwing money away because Newshosting among others offers 50 gigs for 2/3 the price of giganews...it is $10 for 50GIGS

Giganews = Good enough service that they can demand whatever they want people to pay and people will pay = FAIL!

[Hypatia]

well i can say something like you've said about giganews.

Highwinds= good enough speed, good price + terrible completion +DMCA takedowns=FAIL =)

i wouldnt keep HW account even pared with block account from blocknews..too many incompletes..
i dont understand what has happened to them and how it all got to this and why they dont want to fill their huge gaps and i dont really care, to be honest.
Fortunately there are better ones out there.
The bottom line is: it is sad.
We need more competitive providers..

[B18C5]

guess you just like throwing money away because Newshosting among others offers 50 gigs for 2/3 the price of giganews...it is $10 for 50GIGS

Is it throwing money away if I drive a Mustang instead of a Civic? They both have 4 wheels and will take you shopping but, the Mustang is probably nicer to drive. Even at $35 a month, that's a little more than $1 a day. If that means I save an hour of time in my monthly downloading because of completion or other weirdness with the cheaper server, it's already paid for itself. I realize that Usenet people are the cheapest people on earth so, that math might not fly for some. I have a job. $35 a month doesn't even factor in. I spend 2-5 times that on new video games every month.

I don't diss any news server. I'm glad they're there but, I have no problem paying for the best. I get my money's worth from Giga and that's all that matters to me.

[heiska]

I would rather donate the extra $20 month to EFF (or Amnesty even) than to Giganoobs. Each to their own.

[mjmacky]

guess you just like throwing money away because Newshosting among others offers 50 gigs for 2/3 the price of giganews...it is $10 for 50GIGS

Is it throwing money away if I drive a Mustang instead of a Civic? They both have 4 wheels and will take you shopping but, the Mustang is probably nicer to drive

Yes it is throwing money away, and a Mustang being nicer to drive than any car, well that's very subjective. For instance, I would say hell no it isn't. I had a lot more fun in my Civic then my buddy's old 5.0 could provide. Nowadays, I've moved on from the Civic to the Volvo C30 T5, still the nicest "drive" I've ever had

[B18C5]

You're sort of missing the point... It's an, example not to be taken literally.

http://www.youtube.com/watch?v=6lIr1vEjqgY

This is my old civic chasing a GT3. Sold it to a buddy and bought a Subaru.

[zot]

There seems to be a widespread mis-perception that USPs will eventually need to stop spooling retention because the storage cost will become too expensive at some point. That was historically the case, but today, the exact opposite is now true.

The reason is that cost *per byte* of storage is decreasing at a greater rate than the total stored volume *in bytes* is increasing. (The "flipping point" was reached, I'd estimate, about a year to 18 months ago.) This means that the cost of storing usenet retention -going back to 15 August 2008- will be less and less each succeeding year.

So unless usenet posting volume were to unexpectedly increase (or hard drive development were to come to a screeching halt) then we can be quite certain that spooling will continue forever.




http://www.youtube.com/watch?v=6lIr1vEjqgY

This is my old civic chasing a GT3. Sold it to a buddy and bought a Subaru.

Nice race. At first I thought the GT3 driver just needed a lesson on how to power through a curve, but then it became obvious that the guy was taking turns as if driving a U-Haul truck. What a waste!:)



Has any body here used the news client Ozum?
I try out just about every usenet client I learn about. What's your question about Ozum?




I have no problem paying for the best. I get my money's worth from Giga and that's all that matters to me.

Anyone wanting "the best" absolutely needs to get an account from at least two separate NSPs. Block accounts are one way to go, but you can also buy an unlimited account at two different providers for less than the cost of one Giganews account, and (with a newsreader that supports fill-servers) get better completion and overall reliability than Giganews alone.

It was reported that Giganews went through a massive header failure a few months ago, and usenet search engines (like Newsleecher's "super" search) that relied entirely on Giganews for headers suffered greatly as a result. Newsleecher customers were complaining that it was taking far too long for Newsleecher Search to fill in missing headers caused by Giganews. I can understand the apparent love-affair between Newsbin Pro folks and Giganews. Newsbin should give Giganews a big kickback for driving so many frustrated Newsleecher customers their way. :lol:

[B18C5]

Nice race. At first I thought the GT3 driver just needed a lesson on how to power through a curve, but then it became obvious that the guy was taking turns as if driving a U-Haul truck. What a waste!

How much tracking do you do? The GT3's on street tires and I think did pretty well considering. The Civic is on R-comps, good R-comps. That's why he can close the distance in the corners. There's probably a 600 lb difference in weight too which is huge for tracking. That track is a great equalizer. It's so tight that the GT3 seldom has time to put down any power. On any other track, the GT3 would have been gone from view in a single lap. So, even though it makes twice the HP of the civic, it can't really use it. The Civic has a...B18C5... in it. Which I installed. It's a 1.8 liter 210 hp JDM Integra R motor.

I do have a couple accounts but, figured people would complain I was wasting my money again if I mentioned it....

[mjmacky]

The Civic has a...B18C5

Oh nostalgia, I had the same motor in mine, naturally aspirated. What a really fun car that was...
Also, to be on topic, I've tried both Giganews and Astraweb. As a single account user, AW makes me happier than my first and only month at Giganews, I'm especially happier that I'm not burning an extra $20 each month for really no reason at all.

[Malcontent]

There seems to be a widespread mis-perception that USPs will eventually need to stop spooling retention because the storage cost will become too expensive at some point. That was historically the case, but today, the exact opposite is now true.

The reason is that cost *per byte* of storage is decreasing at a greater rate than the total stored volume *in bytes* is increasing. (The &quot;flipping point&quot; was reached, I'd estimate, about a year to 18 months ago.) This means that the cost of storing usenet retention -going back to 15 August 2008- will be less and less each succeeding year.

So unless usenet posting volume were to unexpectedly increase (or hard drive development were to come to a screeching halt) then we can be quite certain that spooling will continue forever.
Cost of storage isn't the only factor. The ower of Blocknews/Usenetnow made some interesting points when it comes to the possiblity of unlimited retention.

http://www.dslreports.com/forum/r25842875-

[nntpjunkie]

I have no problem paying for the best. I get my money's worth from Giga and that's all that matters to me.

Anyone wanting "the best" absolutely needs to get an account from at least two separate NSPs. Block accounts are one way to go, but you can also buy an unlimited account at two different providers for less than the cost of one Giganews account, and (with a newsreader that supports fill-servers) get better completion and overall reliability than Giganews alone.



Agreed entirely, I am a die hard Newshosting fan as most here know, but I have to admit that my usenet usage turned into a completely sick experience once I paired my newshosting account with a block account from astraweb... Also as far as incompletes go, I have not had many because I still have lots of download available on my astraweb account. All told I still spent a hell of alot less money than just on single giganews account supporting two awesome providers with better completion and speed than giganews alone. No completion problems here.

[heiska]

There seems to be a widespread mis-perception that USPs will eventually need to stop spooling retention because the storage cost will become too expensive at some point. That was historically the case, but today, the exact opposite is now true.

The reason is that cost *per byte* of storage is decreasing at a greater rate than the total stored volume *in bytes* is increasing. (The &quot;flipping point&quot; was reached, I'd estimate, about a year to 18 months ago.) This means that the cost of storing usenet retention -going back to 15 August 2008- will be less and less each succeeding year.

So unless usenet posting volume were to unexpectedly increase (or hard drive development were to come to a screeching halt) then we can be quite certain that spooling will continue forever.
Cost of storage isn't the only factor. The ower of Blocknews/Usenetnow made some interesting points when it comes to the possiblity of unlimited retention.

http://www.dslreports.com/forum/r25842875-

All of the usenet providers depend on new people signing up - it would be a huge suicide in the long run for one provider to stop increasing retention while the others keep growing. Only if all the major usenet players formed a cartel which decided certain amount of days of retention which none of them would exceed could they make more profit by not spending money on new HDDs.

A very simple application of the game theory in microeconomics 101.

[Malcontent]

Cost of storage isn't the only factor. The ower of Blocknews/Usenetnow made some interesting points when it comes to the possiblity of unlimited retention.

http://www.dslreports.com/forum/r25842875-

All of the usenet providers depend on new people signing up - it would be a huge suicide in the long run for one provider to stop increasing retention while the others keep growing. Only if all the major usenet players formed a cartel which decided certain amount of days of retention which none of them would exceed could they make more profit by not spending money on new HDDs.

A very simple application of the game theory in microeconomics 101.

It would be suicide if a provider gets to the point where continuing increasing retention causes them to stop generating profit (expenses) and they continue increasing retention and run their business at a loss (no profit). Hoping to attract new customers with the promise of increasing retention is no guarantee of success. Not in this economic climate.

[heiska]

All of the usenet providers depend on new people signing up - it would be a huge suicide in the long run for one provider to stop increasing retention while the others keep growing. Only if all the major usenet players formed a cartel which decided certain amount of days of retention which none of them would exceed could they make more profit by not spending money on new HDDs.

A very simple application of the game theory in microeconomics 101.

It would be suicide if a provider gets to the point where continuing increasing retention causes them to stop generating profit (expenses) and they continue increasing retention and run their business at a loss (no profit). Hoping to attract new customers with the promise of increasing retention is no guarantee of success. Not in this economic climate.

But it has worked out pretty good for the past two years, no? With all the huge pressure against BT, I'm pretty sure usenet (both retention and customer base) will continue growing for years to come. What they could do is to shut down one of their multiple dc locations: concentrate on either US or EU depending on the geographic location of the majority of their customers.

Btw, data centers will evolve their energy effieciency through innovations too: http://www.youtube.com/watch?v=VChOEvKicQQ .

[Malcontent]

But it has worked out pretty good for the past two years, no?

It was in the last 2 years that storage cost dropped that allowed the increase in retention. The drop in storage cost allowed the usenet providers to increase retention and remain profitable. With the increase in storage the cost of electricity and other expenses will begin to become a factor in profitability.




Btw, data centers will evolve their energy effieciency through innovations too: http://www.youtube.com/watch?v=VChOEvKicQQ .

How long until it will be useful for usenet providers today? How long can the usenet providers wait until these more energy efficient data centers go online?

[ensisoft]

Hi,
just want to comment, Newsflash Plus can handle headers also, although support for compressed headers is not implemented. But the database engine can take quite large volumes of data. You can also easily integrate your favourite picture viewer or mp3 player application and then getting that downloaded media going is very easy.

- ensi