Giganews, NewsHosting, NewsDemon... Where does their data come from?

So, if I wanted to setup my own news group servers in my house (I know, not feasible, bandwidth alone would surely cost 5 times my yearly salary), who would feed my servers? Would I need to sign up with another newsgroup service (like giganews) and constantly sync with them? Are all of the news servers out there on a peer to peer like setup constantly synchronizing, or is there an authority or master server? More a curiosity than anything. How would my servers populate.

Thanks

Take a look at something you d/l'ed, right on the news-client, and you'll see right at the top of it the 'magic' behind usenet, i.e., the 'peering'. Lets say you d/l'ed from giganews, so that is the last place the file was 'recieved', and then the 'path' is listed down to where it originated. Here's a simple one:

Path: border1.nntp.dca.giganews.com!nntp.giganews.com!nx02.iad01.newshosting.com!newshosting.com!post02.iad01!not-for-mail

So, the file originated at newshosting, went right straight to Giganews. One 'hop' in the electronic 'bucket-brigade'.

How about this one:
Path: border1.nntp.dca.giganews.com!nntp.giganews.com!news-in-01.newsfeed.easynews.com!easynews.com!easynews!news-out.octanews.net!indigo.octanews.net!auth.brown.octanews.com

Got an extra 'hop' in there, originated from octanews, then went through easynews, then on to Giganews.

There can be a dozen or more 'hops' in there, eventually every news-server gets the file. Exactly how it keeps track of it is that every piece of every file has a unique ID number, the news-servers may see that a single piece of a file (say 1 part out of 130+, say) is missing, and it'll query each of the news-severs that peer with it for that missing piece. Generally, the 'closer' (single hop or better) ones posting server is to one of the majors like newshosting, easynews, or giganews is, the better, but over the last few years things have gotten much better, so much so that posting in multiple groups is simply NOT nessesary, and any more par files than 5% is a real waste of time (unless the posting server is a real dog).

FYI. that unique ID number is generated by the news-client posting the file, NOT the server; you can see that unique message ID, again, in your newsreader, i.e.:

[email protected]

That's 25digits long, each digit is 40+ alpha-numeric-symboltype, so figure out the unique combinations. 10^xxx power. 40x40x40x..... 25 times.

Cool, so it just propagates amongst the news providers. What's the approximate costs involved in that or how would one subscribe to the propagation service?

I ran a smallish news-server (text only) back in the mid-90's, where the input
feed was via Ku-band digital satellite (FM squared, then FM cubed for those into tech stuff) and output to the fastest land-line available at the time to
esidences, 128K ISDN, and some analog dial-up lines for the subscribers.

Of course, commercial sites had multiple OC3's (155Mb/x) at least, nowadays
most news-servers have a minimum of several OC12's (600Mb/s) from several
different vendors (say, Sprint, AT&T, ect) going to several adjoining 'peers';
therefore a single router failure (say at Verizon) doens't bring things to a halt.
The biggie's (like Giganews) have multiple OC48's (2.4Gb/s) or OC192's (10Gb's), and for those who have 'cross-continent' server farms (say, north
american and europe), they may have invested in a 'wave', a dedicated
submarine fiber link across the atlantic running at STM1 (international SDH
hierarchy, equivilent to OC3 or 155Mb/s); FYI most fiber submarine cables
carry 50-100 or more STM 16's (2.4Gb/s), or a goodly chunk of close to or
exceeding 1Tb/s.
Cost? terrestrial OC12's are running around $3K/month (depending on location), and a STM1 linking continents may run 5-20million on a 20 year lease. You will note that news-server operations tend to 'cluster' around
locatons where the fiber feeds/infratructure are plentiful (say, San Francisco/Silicon Valley, Phoenix Arizona, St. Louis, MO, ect), But 'not cheap' by any means, and then you have all those subscribers comming in at faster
and faster speeds all the time.

Cool insight, thanks a lot Beck38! :thumbsup:
I just need to find some oc lines lying around. :whistling

IF (a bit of a big IF where the U.S. is concerned) internet connections go from copper-based (either DSL or cable) to fiber, that's when things from the user standpoint will really take off. DSL speeds, particularly upstream (to the net), exceed cable speeds by a goodly amount (usually 3+ times as fast) although downstream (from the net) cable has the lead, except that if subscribers really step on it, will get cut off by their supplier (Comcast, ect.). Fiber is just starting to get rolled out in many areas, but the cost per 'subscriber passed' is oretty high, still above $2K per. But the speeds are in the 5Mb/50Mb (up/down) so the more of those folks in the subscriber pool, and the big suppliers like Giganews will have to ramp up their connections as well. So far the biggest supplier of Fiber to the home is Verizon, and they have less than 500K total hooked up nationwide.

Totally agree, in my area, the biggest hope is for SBC/AT&T to get moving on their Project Lightspeed or whatever it's called now rolled out. Supposedly, by the end of 2008, they want to make the service available to xx million users. We'll see though.

The AT&T 'Lightspeed' project is not fiber to the home, only 'FTTC' or 'fiber to the curb'. Standard metallic drops are used from there to (and through) the home, reducing the cast to around $500-1K per (potential) subscriber passed. Speeds (both digital television and internet) are of course slower, around 20Mb/s total bandwidth (up/down and be split from that, generally around 2/20). AT&T (and SBC originally) developed this as an alternative to FIOS-type systems, which started out around $5K per sub, now about half that. They arn't quite 'throwing money' at it like Verizon, and there is much speculation that AT&T will buy up the DBS DirecTV as an alternative system for digital video distribution.