Astraweb Header download

[sandman_1]

I got Newsbin Pro and trying to download headers for the first time. The speed is fine when I am downloading headers from multiple groups but when it gets down to one group, it is incredibly slow compared to my normal download speed. I have 30Mbps down and getting maybe 2 to 3 Mbps header download when 1 group is going. Anyone have similar problems or is this just something with Astraweb? I would like to download headers but damn, should it take several days to get them?

[zot]

First of all, make sure that header compression is turned on. This will increase your header download speed (but not your actual bandwidth) by a factor of about 8x compared to uncompressed headers.

How does your header download speed compare to NZB/article body download speed using only one connection?

Unlike article bodies, headers normally can only be downloaded with one connection. So yes, it's perfectly normal that most people won't max out their internet connection downloading headers. There might possibly be ways (or even hacks) to get around this limitation -- I don't know.

Don't forget to check out the forum at Newsbin Pro. Those guys are the experts. ;)

[sandman_1]

Yea I have header compression on. I guess one connection is only allocated 2-3Mbps?

[B18C5]

Header compression should be on by default. What version of Newsbin are you using?

[sandman_1]

V6 RC1 v965

[B18C5]

If you look in the options, under "switches" you could try checking "background update". You'll have to restart after setting it (I think). There are basically two modes.

1 - Download the headers and feed them to the DB in real time (unchecked)
2 - Download the headers and let a worker thread feed them into the DB in the background (checked).

#1 is fully dependent on disk IO speed so, if you're downloading files at the same time or using a slower disk, it'll impact header download speed. #2 will often get you faster header download speeds but, it takes longer to feed the data into the DB. This is true if the problem is disk IO. If the headers simply aren't coming over the wire fast enough then neither mode really matters.

To Astra, using multiple connections and compression, I can often get 400 Mbps header download speeds over my 100 Mbps connection if I use background mode.

[Hypatia]

Astraweb has been doing maintenance on headers.

thus slow header's update times, slow header download and any other things are possible during this period of time

Its been for a while now. And they dont have any ETA. At least they didnt tell me

PS It doesnt affect articles.

[zot]

@B18C5:

Can Newsbin Pro now download headers using multiple connections in a single group?

(sometimes I answer people's questions based on what I learned many years ago, which may not quite be the situation today. That's not that I ever completely accepted the old "it's impossible" argument from newsreader developers, since [I presume] header list starting/ending points could conceivably be staggered and the multiple (simultaneous) batches subsequently re-combined in the newsreader's header database.)

[toastysoul]

I'm d/l'ing headers 1k as far back as days on AW for 2 groups right now, single connection each. I'm getting 15Mb/s on one group, 5Mb/s on the other. That seems pretty fast to me. How's your CPU/HDD/RAM usage? Group updating can be very taxing.

[B18C5]

Can Newsbin Pro now download headers using multiple connections in a single group?

No, It doesn't split connections over a single group. It could these days because of the change to header storage. It seemed dangerous because any failure might mean you lose track of where you are in the header download.

I suppose you could argue that like a file download, the top header download in the list should get all connections....

[nntpjunkie]

Astraweb has been doing maintenance on headers.

thus slow header's update times, slow header download and any other things are possible during this period of time

Its been for a while now. And they dont have any ETA. At least they didnt tell me

PS It doesnt affect articles.

One of the reasons why Astraweb is only good as a filler account IMHO - I use them along with Newshosting as my primary seems like my 25gig Astraweb account has lasted forever :)

[Hypatia]

what? its not like they do maintetance several times a year.


Astraweb is only good as a filler account

Highwinds as a primary server? u've just made my day =))

but... wait.. why im not surprised?=)))

[sandman_1]

I think I found my problem. I have Newsbin set to download 7 days of headers. Some groups download quickly and 1 or 2 dont. A.B.HDTV for instance was taking a long time to get then I noticed how many headers Newsbin was trying to download, over 1 billion. It looks like it was trying to download every header in the group for some reason. So I used the "clear list" option to stop download and then selected "download latest" and bingo the header list was much much smaller. So I don't know if it was something quirky with Newsbin or Astraweb but I got it worked out now.

[B18C5]

Newsbin always downloads from the last high water mark so, if you "download all" headers on a group, then stop the download, next time you "download latest" or update, it starts where it left off. One way to reset the group to download just the last 5 days or whatever is to purge the group (right click PostStorage\Delete stored posts) so, it starts over. Another way is to delete the file "Range.db3" in the group storage folder (<Data Folder>\Spool_v?\<Group>\Range.db3). When range is deleted, Newsbin reverts to using the "Download age". 5 or 10 days or whatever you have it set to.

So, for example lets say you had a new group and you had "Download Age" set to 60, you update the group and you notice it's downloading 1 billion posts.If you stop it, then reset the "Download age" to 3, that's not enough to make it use this setting because the group already has some downloaded posts. So, purge or deleting that file will reset it back to use 3 days.

Just a report. I'm getting 90 Mbps from US Astra with a single header connection, downloading headers from Boneless.

[sandman_1]

Anyone notice a difference in the header downloads? I am now getting around 40-70Mbps when one group is left. Before it was around 3-10Mbps and slow as shit.

[Hypatia]

Yeah they have fixed it