What kind of servers do they run on

[Ryan711]

I was just curious because they must be some pretty massive servers to allow a few months of retention and allow thousands of leechers to download at full speed. Since I really don't know anything about enterprise type servers/internet what do you guys think.

[delimare]

I'd love to know too. It baffles my mind that they can output that much bandwidth.

[geen]

This might give an impression:

http://img444.imageshack.us/img444/4493/img0004hu6.jpg
http://img404.imageshack.us/img404/3667/img0005ya8.jpg
http://img404.imageshack.us/img404/1060/img0006dp0.jpg
http://img444.imageshack.us/img444/8130/img0007xa6.jpg
http://img250.imageshack.us/img250/953/img0009gl7.jpg

Technical information:

What are we seeing:

1 x Feeder incomming Dual core dual xeon 5160
1 x Feeder outgoing naar de spools Dual core Dual Xeon 5160
1 x articel server (headers) P4 Dual Core
1 x auth server (toegang) P4 Dual Core
6 x Reader servers (news.easynews.nl) P4 Dual Core
1 x Spool 24 x 400GB SATA II
16 x Spool 10 x 400GB SATA II
1 x Spool 12 x 400GB

Switches used: http://www.foundrynet.com/products/enterprise/agg-bb-l23/bi-mg8.html

Stats

Current bandwith usage per day: 25TB
The Feeds use 3.2TB (http://stats.xlned.com/)

CPU load Reader 1:
http://img172.imageshack.us/img172/4117/graphimageaphpfd2.png
yes, 200% because there are some crashed processes which are cousing errors.

Traffic used by a Reader:
http://img171.imageshack.us/img171/9299/graphimagebphpkq0.png

Total amount of traffic by the platform:
http://img248.imageshack.us/img248/19/graphimagecphpjf3.png

How does it work ?

All the files first enter the Feeders, these machines exchange the files with other providers.
The files are stored on so called Spools. The spool servers send header information to the Readers, these are the indexing machines. When you request a file, its located by a Reader on a Spoolserver which sends the file to you. A load balancer is used to balance the traffic between all servers.

I think it should look like this:

http://img213.imageshack.us/img213/1659/finalng0.jpg

Disclaimer:
Ive found this information on the easynews forum: www.easynews.nl.
This is what I understand from what ive read, plz dont take this too serious. :)

[Ryan711]

Dang, what kind of connection does that require. Also if you don't mind could you put a little explantion of what everything does such as the Feeder incomming Dual core dual xeon 5160. This kind of stuff is very interesting.

[geen]

Yeah, it is pretty interesting thanks for asking the question because i finally seem to understand how it all works :)

Ill continue my work on this post tomorrow.

[Ryan711]

excellent, can't wait

[tesco]

Great info.
I read about how it works once before but with the pics and amount of each server type it gets more interesting. :)

[b0ner]

That's Insane!

[Beck38]

Almost everything in the usenet world is run on linux these days, not many 'pure' unix operations left. The s/w is distributed computing, spreading the load out between what amounts to massively parallel operations, that what you see in the diagram posted above. This type of operation is what has seriously increased the number of good providers out there these days, at costs a fraction of what they were a handfull of years ago. Of course, h/w costs as well as connection costs have dropped through the floor as well, which is why the cost you pay even for unlimited bytes is <$25 at any provider (and some significantly below that).

Two things stand out: HD space is well below 50cents/Gigabyte in large arrays, and OC48 (2.4Gbits/second) interconnects at major centers is well under $1K/month. The U.S. still needs to focus on the local infrastructure, as it is being beaten by about 20 other countries in local internet speeds (those you get from a local ISP through a cable or DSL connection), but the amount in large datacenters is jaw-dropping. The best to be had for local connectivity (at folks homes) currently is through FIOS (Verizon Fiber Optic), top speeds around 50Mb/s symetrical. Currently serves about 500K customers nationwide, but growing.

[General2k]

Holy crap.. I knew it was an extreme number, but wow.
Thanks for the info