linleech 2.2.1 by Curtiss Howard - 12.30.98
-------------------------------------------

linleech is a small utility that will search through all the articles in a 
given USENET group and match the subject line against a given set of words.
If it is matched, the article will be retrieved.  This is useful for
automating the download of different articles, perhaps binary articles or
specific articles (like searching for certain HOWTO's in a Linux group).


Compiling linleech
------------------

Compiling linleech should be fairly simple.  All you need is gcc (any version
should do) and S-Lang (which I have made available at the linleech website:
http://cerealbox.angel.nu/linleech.html).

Edit the makefile (if necessary) and run make.  Thanks to Yong S. Yi.
 
Many thanks to Yong S. Yi and Scott Scriven; linleech is confirmed to run on a
large variety of platforms now (including FreeBSD and the like).


Using linleech
--------------

linleech uses small scripts written by you to automate the process of 
retrieving articles.  linleech is invoked as follows:

	linleech <script>

where <script> is the script you wish to use.  In version 2.0.0 and up, scripts
are now based on S-Lang -- an excellent embeddable C-like language.  If you 
don't know C, don't worry -- it's simple enough to write your own script.  
I've included a script called leechscript that you can modify to fit your
needs.

Please note: versions 2.2.0 and up are a good bit different from earlier 
versions.  Some functions have been removed or go by a different name.  Please 
read on to see what you'll have to change.

Below is a list of functions.

	auth_connect (char *Server, int *Port, char *User, char *Password);
		This function will connect to a news server using username/password 
		authentication -- ONLY USE THIS IF YOUR SERVER REQUIRES IT!  This 
		function is found in the scripts/auth directory as auth.sl.  I wrote 
		this function using the scripting language as opposed to making it 
		built-in since it was much easier to do.  Please note that it makes
		no attempt to determine if the username was wrong or something of 
		that sort -- it only determines whether or not the authentication 
		succeeded.  NOTE: you must add this line  somewhere in your script 
		BEFORE calling auth_connect();:

			evalfile ("scripts/auth/auth.sl");

		This will include the auth.sl file that contains the auth_connect(); 
		function.  Please remember to do this!

		Example:
			auth_connect ("your-host.com", 119, "user", "password");

	connect (char *Host, int *Port);
		This function connects to the USENET server Host and port Port.  
		Generally, Port should be set to 119 as this is the standard NNTP 			port.

		Example:
			connect ("your-host.com", 119);

	delete_match_list (void);
		This is a function new to version 2.2.0 and up.  This is somewhat 
		simlilar to the clear_match(); function of previous versions.

		A brief explanation...

		In previous versions, I always wanted to dynamically allocate the 
		arrays for matches, but I didn't want to make the user have to deal 
		with creating and destroying a list.  In this version, I abandoned 
		that thinking.  Hopefully this won't be too complicated for the 
		non-technical types.

		Essentially, before you can use the match(); function, you must 
		create a list of matches using the new_match_list(); function.  When 
		you're done with this list and want to clear it, you call 
		delete_match_list(); to get rid of it.  Please always at least call 
		this function at the end of the script to destroy the list in use.

		Example:
			delete_match_list();

	disable_match_all (void);
		This function will disable the option to save all articles, 
		regardless of whether or not they match your criteria.  By default, 
		linleech will not match all articles.

		Example:
			disable_match_all();

	disable_resume (void);
		This function will disable the option to attempt to resume at the 
		last article examined.  By default, linleech does not attempt to 
		resume.

		Example:
			disable_resume (void);

	disconnect (void);
		This function will disconnect from the current NNTP server.

		Example:
			disconnect();

	enable_match_all (void);
		This function will enable the option to save all articles,
		regardless of whether or not your criteria is matched.  By
		default, linleech does not do this.

		Example: 
			enable_match_all();

	enable_resume (void);
		This function will enable the option to attempt to resume at
		the last article examined.  By default, linleech does not do
		this.  Be aware that in many cases, this may not work.  I've
		also gotten a report that this function can crash the program,
		although I doubt that will happen very often (never happened
		to me).  In high traffic newsgroups this function may be
		futile, as by the time you actually attempt to resume the
		article may not exist anymore!

		Example:
			enable_resume();

	filter (char *Program, int *Output);
		This function will set up an article filter with which to run 
		articles through after downloaded.  This is usually uudecode.  The 
		format of the Program parameter is the same as when the command is
		used from the command line, except that the filename is replaced by
		%s.  You can still use command line options and whatnot.  The Output
		parameter can be either 0 (don't show filter output -- default) or 
		1 (show filter output).

		Example:
			filter ("uudecode %s", 1);
			
	group (char *Group);
		This function will join the group Group and search it for 
		criteria you specify in match functions.

		NOTE:  Replace all occurences of get_articles(); in your old
		scripts to group();!
		
		Example:
			group ("comp.os.linux.advocacy");
	
	hide_subjects (void);
		This function will disable the option to display article
		subjects As the headers are downloaded.  By default, linleech
		hides subjects.

		Example:
			hide_subjects();

	match (char *Str, int *Case);
		This function looks for Str with case sensitivity set as Case.
		A value of 1 will enable case sensitive matching; a value of 0 
		will enable case insensitive matching.

		NOTE: match(); no longer does regexp matching!  The new 
		match_regexp(); function does.

		Example:
			match ("linux 2.2.0", 0);

	match_regexp (char *Str);
		This function performs matching using regexp (regular 
		expressions).  This allows for usage of powerful wildcards in 
		your searches.  See the "Using regexp" section or regex.info 
		file in the directory regex/ for more information.

		Example:
			match_regexp ("PICTURE[1-9].*);

	new_match_list (int *Size);
		This is used in conjuction with delete_match_list(); to manage
		the creation and destruction of match lists.  Remember to call
		this before calling any match functions!  It will create a 
		match list with Size number of elements.  You are limited only 
		by the amount of memory it can allocate.  Please remember to 
		use destroy_match_list(); on EVERY match list you create 
		(although, you can only create one at a time)!

		Example:
			new_match_list (10);

	remove_filter (void);
		This function will remove the current article filter, set by the
		filter(); function.

		Example:
			remove_filter();
			
	saveto (char *Path);
		This function sets linleech to save all articles in the path
		Path.  Be sure to add the trailing forward slash to the
		directory name.

		Example:
			saveto ("articles/");
	
	saveto_def (void);
		This function sets linleech to save all articles in the
		current directory.

		Example:
			saveto_def();

	set_trigger (char *Trigger, char *Find);
		This is a function new to versions 2.2.0 and up of linleech.  
		The string Trigger is used to find a specific line on the 
		article header, and the regexp string Find is used to make 
		comparisons on that line.  Check RFC 850 for the types of lines in an
		article header.  Also, an example in this document details how to use
		set_trigger(); to limit the date of articles being downloaded.

		Example:
			set_trigger ("Date:", ".. Nov 98");
			
	show_subjects (void);
		This function will set the option to show article subjects as
		the headers are downloaded.  By default, linleech will not
		display the article subjects.

		Example:
			show_subjects();

	* These functions are used to extend the capabilities of linleech. *

	raw_connect (char *Host, int *Port);
		Connect to Host at port Port.  This is a "raw" connection, and
		does not attempt any type of login whatsoever.

		Example:
			raw_connect ("irc.emory.edu", 6667);

	raw_receive (void);
		Returns a single line of zero-terminated text from the host
		you are currently connected to using raw_connect().  Usually
		this is used in a loop.

		Example:
			return raw_receive();
	
	raw_send (char *Text);
		Sends a single line of text to the host you are currently
		connected to using raw_connect().

		Example:
			raw_send ("Test\r\n");

	connected:
		This is a variable of type Integer_Type that contains the
		current connection status.  It is 0 if not connected, 1 if
		connected.

		Example:
			if (connected)
			{
				.
				.
				.
			}

I will not go into specifics on how to program using S-Lang.  Please check out
the documentation included with the S-Lang though, as I would really appreciate
it if people contributed nifty scripts for linleech. 

Also, I have removed leechscript.mail, nntplib, and pop3lib as no one seems to
be using them.  They will return to the package if anyone requests them.


Examples
--------

Here's a simple, no-frills script that will grab all articles containing 
"linux" (case insensitive) in the header from the group comp.os.linux.advocacy:

show_subjects();
new_match_list (1);
match ("linux", 0);
connect ("your-host.com", 119);
group ("comp.os.linux.advocacy);
disconnect();
delete_match_list();

Note the use of new_match_list(); and delete_match_list();.  These are 
important!  

Alos remember that it makes little difference in what order the commands are
placed.  Naturally you must be connected in order to use the group(); 
function.

Here's a slightly more complex example that will attempt to grab all or most
pictures off of alt.binaries.pictures:

show_subjects();
new_match_list (4);
match (".gif", 0);
match (".jpg", 0);
match_regexp ("(./.)");
match_regexp ("[./.]");
connect ("your-host.com", 119);
group ("alt.binaries.pictures");
disconnect();
delete_match_list();

The idea behind this script is that most binary posts contain the filename
(hence searching for popular file extensions -- GIF and JPEG) and, in the case
of multi-part posts, the current part (ie, "(3/6)" or "[2/3]").  This script
should suck just about any picture group dry.

Building on this example, let's say I want to grab all pictures from the 12th
of November to the 19th of November.  We can use the set_trigger(); function
to do this:

show_subjects();
new_match_list (5);
match (".gif", 0);
match (".jpg", 0);
match_regexp ("(./.)");
match_regexp ("[./.]");
set_trigger ("Date:", "1[2-9] Nov 98");
connect ("your-host.com", 119);
group (alt.binaries.pictures");
disconnect();
delete_match_list();

Please note that the set_trigger(); function will use up a space on the match
list!  Take this into consideration.

The first argument to our set_trigger(); function is the trigger, in this case,
"Date:".  This is the way the date line appears in an article header, and thus,
this is what the set_trigger(); will examine to determine if it's looking on
the right line or not.  Basically, headers look something like this:

...
...
Date: 17 Nov 98
Subject: linleech
...

So when it encounters "Date:", it will use that line to compare against our
second argument to set_trigger();.  Our second argument is a regexp expression
(set_trigger(); always uses a CASE SENSITIVE string for the first argument and
a regexp expression for the second) which we use to limit the date.  linleech
will ONLY download articles if they meet ALL criteria set by set_trigger();.  
This also applies if you use the enable_match_all(); function.

Also note that the USUAL format for dates is DD MMM YY.  MMM is usually in the
form "Jan", "Feb", "Mar", "Apr", etc.

Check RFC 850 for information on USENET article headers and the "Using regexp"
section for more information on using regexp to do advanced searches.


Using regexp
------------

Regexp is a powerful library of routines that can dramatically increase your
searching abilities.  Regexp does this by providing you with simple wildcards,
operators, lists, and more.

I will provide a small tutorial on the use of regexp.  However, this will not
cover everything.  If you wish to read a more complete reference, see the file
regex.info, included in the info/ directory.

First, let's examine the . operator.  The . operator works exactly as the ?
operator does in Bash.  That is, the . operator will match any character.  For
instance, "T.." would match "The", "Too", "Tan", etc.  Simple enough.

The * operator is the repetition operator.  It DOES NOT work as * does in
Bash.  The * operator will match whatever is directly before it any number of
times.  For instance, "a*" would match "aa", "aaa", "aaaa", etc.  Now, we can
combine the . operator and the * operator to make an operator similar to * in
bash.  Using .* will match any word, since it uses the "any character" (.) in
repetition.

Creating lists can be done with [], as it is with Bash.  This will essentially
create a range of characters to check within.  For instance, [abc] will either
match a, b, or c.  [abc]* will match either a, b, or c in in repetition.
Also, within lists you can create a range.  Example: [1-9] will match any
character between 1 and 9, and [a-zA-Z] will match any lowercase and uppercase
letter.  Note that all normal operators can be used within a list, but ^ acts
differently within a list.  If ^ is within a list, this will mean match
anything BUT these characters.  For instance, [^abc] will match any characters
besides a, b, or c.

That's it for my regexp tutorial.  A little short, but useful.  Remember, you
don't have to use any of these things when matching, but it helps.


Miscellaneous Information
-------------------------

All files saved by linleech are in the form <group>.<number>.  Therefore, 
article 450124 of group comp.os.linux.adovcacy would be saved as
comp.os.linux.advocacy.450124.

Also, I like to save all my binary articles in a separate directory and simply
run "uudecode *" to uudecode every article.  YMMV.


Updates
-------

5.13.98  - linleech 1.0.0.  First release.

5.17.98  - linleech 1.1.0.  Second release.  Fixed MANY bugs, including one
           that prevented you from joining multiple groups in the same script.
           Added new commands, such as MATCHCI, MATCHCS, and SHOWHEAD.
           linleech is actually decent now :).

5.17.98  - linleech 1.1.1.  Third release.  Two in a day :).  Added MATCHALL, 
           SAVETO, and SAVETODEF commands.  Added ability to save to different 
           directories.  Also added elapsed time and transfer stats at the end 
           of the program.

5.23.98  - linleech 1.1.2.  Fourth release.  Added RESUME command and time
           elapsed is now displayed in minutes and seconds.

5.29.98  - linleech 2.0.0.  Fifth release.  RADICAL changes.  I rewrote
           EVERYTHING to use S-Lang.  I hope you like this... this adds 
           powerful scripting abilities to linleech.

6.9.98   - linleech 2.1.0.  Sixth release.  Added NNTP and POP3 scripting
           libraries, automatic e-mail retrieval program (very nifty), and
           miscellaneous functions for socket use, extending the functionality
           of linleech.

7.22.98  - linleech 2.1.1.  Seventh release.  Added authentication function,
	      fixed miscellaneous typos, and fixed a bug in leechscript.mail.

8.16.98  - linleech 2.1.2.  Eigth release.  Added regexp searching.

9.3.98   - linleech 2.1.3.  Ninth release.  Added the reverse_order(); function
           and filtering of articles with a 430 error code for leafnode
           compatibility (thanks to Matt Nottingham).

11.26.98 - linleech 2.2.0.  Tenth release.  MAJOR rewrite, allowing for easier
           modification and reading.  Also added triggers and changed around
           a few functions.  

11.28.98 - linleech 2.2.0.  Not an official release, but a very small fix.
           Added a makefile, thanks to Yong S. Yi.  Fixed a problem with 
           match_regexp(); and now linleech will read from stdin if no script
           is given.  Also, in order to add linleech to the Debian package 
           archive, linleech has been put under the GPL.  Thanks to Mitch 
           Blevin.

12.30.98 - linleech 2.2.1.  Fixed a small socket problem keeping linleech from
           compiling properly under some non-Linux systems.  Also added the
           filter(); and remove_filter(); functions.

		 
/etc
----

If you have any comments or questions regarding linleech, please e-mail me at:

	cerealbox@oldwarez.com
	cereal@schizo.com
	po_boxx_823@hotmail.com

Please write nifty scripts!  And please put this on as many web and FTP sites
as you possibly can!

Visit the linleech website for the latest updates:
http://cerealbox.suspicion.org/linleech.html

NOTE THE CHANGE!  It's no longer cerealbox.angel.nu, so please update your 
bookmarks!

Also check out my GTK+ CHIP8 emulator/interpreter, GChip8:
http://cerealbox.suspicion.org/gchip8.html

Thank your for using linleech!


Curtiss Howard, 12.30.98
cerealbox@oldwarez.com OR cereal@schizo.com OR po_boxx_823@hotmail.com
