Newspost / Newsmangler

[john8675309]

Hi,

I have been trying to post a file to newsgroups for awhile. I need to be able to script the newsgroup upload. the issue I am having here is newspost dosen't create nzb files, and newsmangler appears to freeze up, it never uploads anything (I am using Linux).

I saw a post on these forums from zrubavel stating he had a patch for newspost that supports nzb after uploading, unfortunately the file has expired.

http://filesharingtalk.com/threads/338716-Linux-posting-Tool-with-nzb-creator?highlight=newspost

maybe someone has an idea of what I can do to generate the nzb after all the files have uploaded.

newsmangler seems awesome if only it worked. This is my exact issue: https://github.com/madcowfred/newsmangler/issues/8

Let me know if these is anything anyone can help me with.

Thanks!

--John

[3Bees]

Would it not make sense to just visit Binsearch or something and create the NZB from there?

[john8675309]

Would it not make sense to just visit Binsearch or something and create the NZB from there?

That is possible, but can I do it automatically? I have an idea how to with using perl and search for the file create and download the nzb then add it, just thought it would be nice to use 1 step.

[mesaman]

Newsmangler is distributed with several different yEnc encoding modules. One of them (yenc-vanilla) mangles yEnc, it's defective so don't use that specific module.

[john8675309]

Newsmangler is distributed with several different yEnc encoding modules. One of them (yenc-vanilla) mangles yEnc, it's defective so don't use that specific module.

Thank you, I am using Freddie's yenc but the program still appears to be locking up in poll() do you use newsmangler what command line options do you use? Can you give me an example of the output?

[cola]

Newsmangler is distributed with several different yEnc encoding modules. One of them (yenc-vanilla) mangles yEnc, it's defective so don't use that specific module.

How do you select the different yenc modules?

[john8675309]

Newsmangler is distributed with several different yEnc encoding modules. One of them (yenc-vanilla) mangles yEnc, it's defective so don't use that specific module.

How do you select the different yenc modules?

newsmangler just detects it automatically after install:

call1 newsmangler # ./poster.py -f 'outtahere' -c sample.conf /home/john/newsmangler/iso/install-amd64-minimal-20120223.iso
2012-03-04 18:38:13,615 [INFO] Using yenc-fred module for yEnc encoding.
2012-03-04 18:38:13,888 [INFO] Posting 1 article(s)...
^X^CTraceback (most recent call last):
File "./poster.py", line 146, in <module>
main()
File "./poster.py", line 141, in main
generate_par2=options.generate_par2)
File "/home/john/newsmangler/classes/Poster.py", line 154, in post
_sleep(0.01)
KeyboardInterrupt

Here is my sample.conf
#[downloading]
## Directory to download files into.
#directory: ~/news

# Create a per-group directory and save files into it?
#use_group_dir: 0

# Create a directory based on the .nzb filename and save files into it? If
# you have the above option enabled, you'll end up with "group/nzbname".
#use_nzb_dir: 1


[posting]
# The 'From' address to put on posts. If you change this then try to resume
# a posting, many clients will get confused. Set it once, then leave it ;)
from: TVROCKz <[email protected]>

# Default group you'd like to post to. If you'd like to post to more than
# one group at a time, seperate the group names with commas.
# default_group: alt.binaries.test,alt.binaries.test.yenc
default_group: alt.binaries.test

# Size of each article in bytes.
article_size: 768000
#article_size: 240000

# String to prefix to each subject.
subject_prefix:

# Generate a .NZB for each post?
generate_nzbs: 1

# Space seperated list of filenames to skip when posting.
skip_filenames:


[aliases]
# Group aliases in the form "short: long".
abt: alt.binaries.test
abty: alt.binaries.test.yenc


[database]
# NOTE: does nothing!
# Driver should be 'mysql', 'postgresql' or 'sqlite'
driver: sqlite

# Connection info. If using SQLite, you only need to specify 'database'.
hostname: localhost
database: pynl
username: freddie
password:


[server]
# Connection info for the server
hostname: unlimited.newshosting.com
port: 119

# Authentication info for the server
username: username
password: password

# Number of connections to open to the server
connections: 10

# How long to wait (in seconds) between connection attempts
reconnect_delay: 10




When doing a tcpdump I see the connection to port 119 and when I ctrl+c I see the disconnect, but I do not ever see any files transfer

[john8675309]

Newsmangler is distributed with several different yEnc encoding modules. One of them (yenc-vanilla) mangles yEnc, it's defective so don't use that specific module.

How do you select the different yenc modules?

I installed yenc-0.3-freddie.tar.gz and the program automatically selects it see command output below.

Command Output:
./poster.py -f 'outtahere' -c sample.conf /home/john/newsmangler/iso/install-amd64-minimal-20120223.iso
2012-03-04 18:38:57,166 [INFO] Using yenc-fred module for yEnc encoding.
2012-03-04 18:38:57,428 [INFO] Posting 1 article(s)...
^CTraceback (most recent call last):
File "./poster.py", line 146, in <module>
main()
File "./poster.py", line 141, in main
generate_par2=options.generate_par2)
File "/home/john/newsmangler/classes/Poster.py", line 154, in post
_sleep(0.01)
KeyboardInterrupt

sample.conf
#[downloading]
## Directory to download files into.
#directory: ~/news

# Create a per-group directory and save files into it?
#use_group_dir: 0

# Create a directory based on the .nzb filename and save files into it? If
# you have the above option enabled, you'll end up with "group/nzbname".
#use_nzb_dir: 1


[posting]
# The 'From' address to put on posts. If you change this then try to resume
# a posting, many clients will get confused. Set it once, then leave it ;)
from: TVROCKz <[email protected]>

# Default group you'd like to post to. If you'd like to post to more than
# one group at a time, seperate the group names with commas.
# default_group: alt.binaries.test,alt.binaries.test.yenc
default_group: alt.binaries.test

# Size of each article in bytes.
article_size: 768000
#article_size: 240000

# String to prefix to each subject.
subject_prefix:

# Generate a .NZB for each post?
generate_nzbs: 1

# Space seperated list of filenames to skip when posting.
skip_filenames:


[aliases]
# Group aliases in the form "short: long".
abt: alt.binaries.test
abty: alt.binaries.test.yenc


[database]
# NOTE: does nothing!
# Driver should be 'mysql', 'postgresql' or 'sqlite'
driver: sqlite

# Connection info. If using SQLite, you only need to specify 'database'.
hostname: localhost
database: pynl
username: freddie
password:


[server]
# Connection info for the server
hostname: unlimited.newshosting.com
port: 119

# Authentication info for the server
username: xxxx
password: xxxx

# Number of connections to open to the server
connections: 10

# How long to wait (in seconds) between connection attempts
reconnect_delay: 10

[cola]

I got Newsmangler off GitHub and just run it. It always picks the vanilla module. Not sure what I'm doing wrong.

[john8675309]

Here is the link to yenc-freddie tar file http://www.mediafire.com/?f2xtumje437yca3 , does your stuff actually upload I understand it's probably wrecked because of the vanilla yenc. Mine never uploads. What distro are you using? I am just trying to figure out why I can't make it upload at all.

[krackt]

Fwiw these are my findings with this software:

I cannot post to Thundernews using newsmangler. Connections are made to the newsserver but no data is transferred. A timeout occurs after 3 minutes, the server boots me and a traceback is generated.

I can post to Giganews using newsmangler (an nzb is created)

I can post to Thundernews using newspost 2.1.1 (cli client)

It's likely that the connection or data transfer routine in newsmangler has problems with some newsservers. :dabs:

[john8675309]

ok maybe that makes sense... I use newshosting... So I find this commit a bit interesting https://github.com/madcowfred/newsmangler/commit/0a6dade8be4086a8a45d3653cb50f1dcf072152a (Maybe something on news hosting changed in the last 6 years lol).. I will try astraweb and see if it works better..

ok, yep astraweb works great for posting with newsmangler, Thanks for all the help everyone! It has to be something that newshosting is doing that prevents newsmangler from working.

--John

[cola]

It'll upload sometimes. Smaller uploads will work half the time, but usually the program will just error out. I'm on debian testing.

I'll try this out and see how it goes.

[mesaman]

The 3 modules I know of:

yenc-fred (good)
yenc-vanilla (bad, do not use)
python-vanilla (good)

[cola]

Built and installed that module. Even did the "import yenc" command. Newsmangler is still picking yenc-vanilla.

[john8675309]

I have a newer version of new mangler on my laptop at home I will post it tonight, maybe that is your issue?

--John

[cola]

Maybe, I just have the latest github version.

[john8675309]

Maybe, I just have the latest github version.

It looks like all my nagging finally got Freddie's attention: https://github.com/madcowfred/newsmangler/issues/8

[cola]

I updated, even installed the yenc module he has listed to install. Still picking yenc-vanilla, but so far its running without a problem. So I'll keep my hopes up.


edit: Crap, I jinxed it.


Traceback (most recent call last):
File "/home/cola/newsmangler/classes/BaseMangler.py", line 91, in poll
obj.handle_read_event()
File "/usr/lib/python2.7/asyncore.py", line 444, in handle_read_event
self.handle_read()
File "/home/cola/newsmangler/classes/asyncNNTP.py", line 173, in handle_read
self._readbuf += self.recv(16384)
File "/usr/lib/python2.7/asyncore.py", line 388, in recv
self.handle_close()
File "/home/cola/newsmangler/classes/asyncNNTP.py", line 155, in handle_close
self.really_close()
File "/home/cola/newsmangler/classes/asyncNNTP.py", line 161, in really_close
self.close()
File "/home/cola/newsmangler/classes/asyncNNTP.py", line 103, in close
self.del_channel()
File "/home/cola/newsmangler/classes/asyncNNTP.py", line 98, in del_channel
asyncore.poller.unregister(self._fileno)
TypeError: argument must be an int, or have a fileno() method.
2012-03-08 22:21:59,465 [WARNING] Invalid FD for poll() - 8
Traceback (most recent call last):
File "poster.py", line 150, in <module>
main()
File "poster.py", line 145, in main
poster.post(newsgroup, postme, post_title=post_title)
File "/home/cola/newsmangler/classes/Poster.py", line 118, in post
_poll()
File "/home/cola/newsmangler/classes/BaseMangler.py", line 99, in poll
obj.handle_error()
AttributeError: 'NoneType' object has no attribute 'handle_error'

[john8675309]

ok, I saw this and had to get out of bed, cause I want it to work for you like it's working for me now!! Freddie posted an update tonight to the git repository, so it fixes newsmangler with newshosting!! anyways here are the exact steps to get yenc-freddie working...

1. Download the module: http://filesharingtalk.com/external/?http://www.mediafire.com/?f2xtumje437yca3
2. Extract it: rar e yenc-0.3-freddie.rar
3. cd to it: cd yenc-0.3-freddie
4. Build it: python setup.py build
5. Install it: pything setup.py install
(Assuming all goes well)
6. Edit (in your newsmangler directory): classes/yEnc.py
7. Go Way to the bottom of the file and change:
HAVE_YENC_FRED = ('Freddie mod' in _yenc.__doc__)
to
#HAVE_YENC_FRED = ('Freddie mod' in _yenc.__doc__)
HAVE_YENC_FRED = ('Freddie' in _yenc.__doc__)


That will fix it (Before Fix):
call1 newsmangler # ./poster.py -c sample.conf -f "test please ignore" iso/*
2012-03-08 22:59:32,839 [INFO] Using yenc-vanilla module for yEnc encoding.
2012-03-08 22:59:33,095 [INFO] Posting 28 article(s)...

(After Fix):
call1 newsmangler # !./post
./poster.py -c sample.conf -f "test please ignore" iso/*
2012-03-08 22:58:41,561 [INFO] Using yenc-fred module for yEnc encoding.
2012-03-08 22:58:42,205 [INFO] Posting 28 article(s)...
26 articles remaining - 555.6KB/s



Have Fun,

--John

[krackt]

I updated, even installed the yenc module he has listed to install. Still picking yenc-vanilla, but so far its running without a problem. So I'll keep my hopes up.

It's likely that the yenc type isn't your problem but the connection error that madcowfred is still working on. Further info here https://github.com/madcowfred/newsmangler/issues/7

[john8675309]

I think madcowfred has it fixed, he mentioned that he was going to add more debug, but whatever was causing the timeout should be fixed after yesterday's commit.

[cola]

Thanks for helping me with the fred module. That got newsmangler using it, but newsmangler still crashed with the same error as before :wacko:

[john8675309]

Thanks for helping me with the fred module. That got newsmangler using it, but newsmangler still crashed with the same error as before :wacko:

I have seen this error before, can you send me the full command line you're using? I think you're just missing an option.

--John

[cola]

python poster.py -c 1.conf -f test -g alt.binaries.test /home/cola/test/*

[john8675309]

Well I am officially at a loss.. Maybe be good to file an issue with madcowfred and see if he could look into it. Sorry, I am not a python expert or even a beginner!

[cola]

It's ok. Thanks for all the people!

[kkiller23]

Can someone show me what i need to install and use newsmangler? I'm rather new to linux so i don't know my way around it very well.

I downloaded a poster pack from one of the irc channels, but it's very confusing from me as it tells me i need a certain version of python or something and i read somewhere that you shouldn't really down grade your version of python on linux.

Any way

http://www.file-upload.net/download-3645696/PosterPack_07_08_2011.rar.html t

that is the pack i downloaded and it comes with a bunch of different stuff, can someone just explain what i really need and how to install this?

[cola]

download the zip from GitHub https://github.com/madcowfred/newsmangler

extract the files, then in at the CLI use python poster.py --help

oh and edit the config file

[kkiller23]

I think i figured it out


2012-03-10 07:41:09,322 [INFO] Using python-vanilla module for yEnc encoding.
2012-03-10 07:41:10,034 [INFO] Posting 1093 article(s)...
2012-03-10 07:44:11,064 [INFO] Posting complete - 841295216 bytes in 181.03s (4538.3KB/s)

does that mean it's good? if it's using python vanilla?

also if i don't wanna post a certain file in a folder i just put *avi etc at the end of the dirname correct?


atleast i had it working... now i keep getting


ERROR: "/killer/Desktop/Downloads/Seeking.Justice.LIMITED.BDRip.XviD-TARGET/" does not exist or is not a file!
ERROR: no valid arguments provided on command line!

here is the command im using
./poster.py -f '[$D]' -g alt.binaries.test -c sample.conf /killer/Desktop/Downloads/Seeking.Justice.LIMITED.BDRip.XviD-TARGET

can someone tell me what im doing wrong now? as it's the same command i used before.

[kkiller23]

Also, i can cd to the folder. So i don't know why it's showing up like that now.

Can someone tell me what i'm doing wrong?

[cola]

/killer/Desktop/Downloads/Seeking.Justice.LIMITED.BDRip.XviD-TARGET/*

or /killer/Desktop/Downloads/Seeking.Justice.LIMITED.BDRip.XviD-TARGET/*.avi

should work

[kkiller23]

So /killer/Desktop/Downloads/Seeking.Justice.LIMITED.BDRip.XviD-TARGET/*.avi

will make it so that it uploads everything in the folder besides the avi files right?

Also is there a way to make it so that dirname is in the header.

Like in most posters $D will make it the folder name, but it doesn't seem to work in news mangler.

I was just trying to post a 20gb season of a tv show and i got

2012-03-10 21:47:58,445 [INFO] Posting 33929 article(s)...
2012-03-10 22:32:18,169 [WARNING] 23: posting not allowed!
2012-03-10 22:34:55,481 [WARNING] 20: posting failed - 441 Posting Not Finished (unknown)
17288 articles remaining - 4516.2KB/s

what does that mean? it's still uploading though so i'm not sure what i should do and do you know if there is a way to pause, stop and start uploads?

I'm sorry for asking so many question. I'm just new and there are hardly any tuts on how to use this program.

[cola]

Your usenet server isn't giving you permission to post.

What provider are you using?

[kkiller23]

Blocknews, but can you tell me if there is a way to select everything inside a folder besides avi files? Because the

"/killer/Desktop/Downloads/Seeking.Justice.LIMITED.BDRip.XviD-TARGET/*.avi "

makes it select only the avi files, i wanna upload only the .rar and pars files and skip the avis in the folder.

[cola]

Probably can't do that, I'd move the rars and pars to a different folder.

[kkiller23]

It was working great, but now i get this server when trying to post.


2012-03-11 21:51:06,998 [INFO] Using python-vanilla module for yEnc encoding.
2012-03-11 21:51:07,839 [INFO] Posting 992 article(s)...
Traceback (most recent call last):
File "poster.py", line 150, in <module>
main()
File "poster.py", line 145, in main
poster.post(newsgroup, postme, post_title=post_title)
File "/home/newsmangler/classes/Poster.py", line 124, in post
self.build_article(postfile, article)
File "/home/newsmangler/classes/Poster.py", line 302, in build_article
if self.conf['posting']['generate_nzbs']:
KeyError: 'generate_nzbs'

[cola]

ok, I saw this and had to get out of bed, cause I want it to work for you like it's working for me now!! Freddie posted an update tonight to the git repository, so it fixes newsmangler with newshosting!! anyways here are the exact steps to get yenc-freddie working...

1. Download the module: http://filesharingtalk.com/external/?http://www.mediafire.com/?f2xtumje437yca3
2. Extract it: rar e yenc-0.3-freddie.rar
3. cd to it: cd yenc-0.3-freddie
4. Build it: python setup.py build
5. Install it: pything setup.py install
(Assuming all goes well)
6. Edit (in your newsmangler directory): classes/yEnc.py
7. Go Way to the bottom of the file and change:
HAVE_YENC_FRED = ('Freddie mod' in _yenc.__doc__)
to
#HAVE_YENC_FRED = ('Freddie mod' in _yenc.__doc__)
HAVE_YENC_FRED = ('Freddie' in _yenc.__doc__)


That will fix it (Before Fix):
call1 newsmangler # ./poster.py -c sample.conf -f "test please ignore" iso/*
2012-03-08 22:59:32,839 [INFO] Using yenc-vanilla module for yEnc encoding.
2012-03-08 22:59:33,095 [INFO] Posting 28 article(s)...

(After Fix):
call1 newsmangler # !./post
./poster.py -c sample.conf -f "test please ignore" iso/*
2012-03-08 22:58:41,561 [INFO] Using yenc-fred module for yEnc encoding.
2012-03-08 22:58:42,205 [INFO] Posting 28 article(s)...
26 articles remaining - 555.6KB/s



Have Fun,

--John

John, Madcow just updated newsmangler again after my issue post, should this still work for the freddie module?

[krackt]

It was working great, but now i get this server when trying to post.

File "/home/newsmangler/classes/Poster.py", line 124, in post
self.build_article(postfile, article)
File "/home/newsmangler/classes/Poster.py", line 302, in build_article
if self.conf['posting']['generate_nzbs']:
KeyError: 'generate_nzbs'You're running this from /home ? post the output of
id newsmangler and
ls -la /home
Looks like you don't have permissions to create the nzb.