Astraweb Header download

[sandman_1]

I got Newsbin Pro and trying to download headers for the first time. The speed is fine when I am downloading headers from multiple groups but when it gets down to one group, it is incredibly slow compared to my normal download speed. I have 30Mbps down and getting maybe 2 to 3 Mbps header download when 1 group is going. Anyone have similar problems or is this just something with Astraweb? I would like to download headers but damn, should it take several days to get them?

[zot]

First of all, make sure that header compression is turned on. This will increase your header download speed (but not your actual bandwidth) by a factor of about 8x compared to uncompressed headers.

How does your header download speed compare to NZB/article body download speed using only one connection?

Unlike article bodies, headers normally can only be downloaded with one connection. So yes, it's perfectly normal that most people won't max out their internet connection downloading headers. There might possibly be ways (or even hacks) to get around this limitation -- I don't know.

Don't forget to check out the forum at Newsbin Pro. Those guys are the experts.

[sandman_1]

Yea I have header compression on. I guess one connection is only allocated 2-3Mbps?

[B18C5]

Header compression should be on by default. What version of Newsbin are you using?

[sandman_1]

V6 RC1 v965

[B18C5]

If you look in the options, under "switches" you could try checking "background update". You'll have to restart after setting it (I think). There are basically two modes.

1 - Download the headers and feed them to the DB in real time (unchecked)
2 - Download the headers and let a worker thread feed them into the DB in the background (checked).

#1 is fully dependent on disk IO speed so, if you're downloading files at the same time or using a slower disk, it'll impact header download speed. #2 will often get you faster header download speeds but, it takes longer to feed the data into the DB. This is true if the problem is disk IO. If the headers simply aren't coming over the wire fast enough then neither mode really matters.

To Astra, using multiple connections and compression, I can often get 400 Mbps header download speeds over my 100 Mbps connection if I use background mode.

[Hypatia]

Astraweb has been doing maintenance on headers.

thus slow header's update times, slow header download and any other things are possible during this period of time

Its been for a while now. And they dont have any ETA. At least they didnt tell me

PS It doesnt affect articles.

[zot]

@B18C5:

Can Newsbin Pro now download headers using multiple connections in a single group?

(sometimes I answer people's questions based on what I learned many years ago, which may not quite be the situation today. That's not that I ever completely accepted the old "it's impossible" argument from newsreader developers, since [I presume] header list starting/ending points could conceivably be staggered and the multiple (simultaneous) batches subsequently re-combined in the newsreader's header database.)

[toastysoul]

I'm d/l'ing headers 1k as far back as days on AW for 2 groups right now, single connection each. I'm getting 15Mb/s on one group, 5Mb/s on the other. That seems pretty fast to me. How's your CPU/HDD/RAM usage? Group updating can be very taxing.

[B18C5]

Can Newsbin Pro now download headers using multiple connections in a single group?

No, It doesn't split connections over a single group. It could these days because of the change to header storage. It seemed dangerous because any failure might mean you lose track of where you are in the header download.

I suppose you could argue that like a file download, the top header download in the list should get all connections....