[Date Prev][Date Next][Thread Prev][Thread Next] [Search] [Date Index] [Thread Index]

[MacPerl] Help with Quotemonster

To: mac-perl@iis.ee.ethz.ch
Subject: [MacPerl] Help with Quotemonster
From: "adjei" <adjei@inreach.com>
Date: Thu, 24 Jun 1999 18:35:09 -0700

Quotemonster or qm is perl script that downloads historical quotes from the 
internet. It was working fine with Os 8.5.1 until all of a sudden it stared 
truncating the quotesout dates. For example a normal quotesout ( output) looks 
like this:

AOL,061799,106.5,113.75,103,110.6875,31982600
AOL,061899,112,113.5,109.5625,112,22946900

the faulty quotesout looks like this:

AOL,0617,106.5,113.75,103,110.6875,31982600
AOL,0618,112,113.5,109.5625,112,22946900

note the disappearance of the year-99.

I cannot figure it out. I upgraded to os 8.6-still same problem.
your help will be most appreciated.
I have attached the qm script.


#!/usr/bin/perl
#
#   IMPORTANT: READ THIS IN ITS ENTIRETY. YOU ARE THE FINAL AUTHORITY IN 
DETERMINING
#   WHAT HAPPENS ON YOUR COMPUTER. YOU SHOULD READ ALL DOCUMENTATION PERTAINING 
TO
#   THE INSTALLATION AND USE OF ALL SOFTWARE YOU INSTALL AND USE AND SHOULD 
MAKE 
#   REGULAR AND COMPLETE BACKUPS OF YOUR IMPORTANT DATA.
#
#   QuoteMonster - Download and parsing script for retrieving
#   stock quote data from Yahoo. 
#
#   -------------------------LEGAL 
INFO--------------------------------------------
#
#   This script is Copyright (C) 1998  Jim Michael and XMLWorks All Rights 
Reserved
#
#   This is version 1.0.2m (16 Dec 1998) By G Michel
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
#   Download and parsing script for www.yahoo.com. This script will most
#   assuredly NOT work with other quote sources. Contact us for assistance
#   if you would like to use this script for other quote sources.
#
#   ------------------------------USAGE 
INFO-------------------------------------
#
#   USAGE: qm start_date end_date
#
#   REQUIRED FORMAT OF START AND END DATES: yymmdd
#
#   IMPORTANT USAGE NOTES: This script resulted from the merging of the 
QuoteMonster
#   Daily and QuoteMonster Historical scripts. You can now download both your 
daily
#   and historical data using just this script. The date arguments start_date 
and 
#   end_date are now OPTIONAL. If no date argument is given then QuoteMonster 
will
#   default to today's date. If only one date argument is given then 
QuoteMonster will
#   download data for that date only. If both start_date and end_date are given 
then
#   QuoteMonster will download data for that date range. If no date argument is 
given
#   then QuoteMonster will use its group download mode and send several quote 
requests
#   at once, greatly increasing the speed of daily downloads of large numbers 
of quotes.
#   
#   YEAR 2000 NOTE: Date formats of Y2K compliant charting programs are not 
know yet.
#   Until it is known what the new format, if any, might be it is impossible to 
produce
#   a Y2K compliant version of the program. Once a year format is established 
and charting
#   programs are updated to use the new format, QuoteMonster may be modified to 
handle the
#   new date format, probably by making the year field use four digits, per the 
configuration
#   option below.
#
#   IMPORTANT NOTE RE MUTUAL FUNDS: Only the closing price is available using 
QuoteMonster 
#   to download mutual fund prices. Consequently you should not mix mutual fund 
tickers
#   with stock and index ticker in the same ticker file or your import program 
may generate
#   errors. To handle both types of data, keep a copy of the QuoteMonster 
script in two 
#   different directories, using one for stocks and index downloads and one for 
mutual funds.
#   You may also need to use a header at the top of your data file such as:
#
#   <date>,<ticker>,<close>
#
#   Note to Windows NT/95 users:
#
#   If you are on a Win32 platform (NT or Windows95/98) then we HIGHLY 
recommend that you
#   use Gurusamy Sarathy's Standard Win32 port of Perl 5.04 or higher. The 
ActiveState port
#   is newer, but this version is known to be stable and included all of the 
modules used
#   by QuoteMonster. You may download the free perl interpreter from
#   http://www.perl.com/ or see the XMLWorks website for location of the binary 
download.
#
#
#   ----------------------------------SUPPORT 
INFO--------------------------------------
#
#   A newsgroup for QuoteMonster has been created on DejaNews. You can access 
it at:
#
#             
#       
#http://www.dejanews.com/[LB=http://xmlworks.com/quotemonster/,LBT=Return%20to%2
0XM#LWorks]/group/dejanews.members.biz.quotemonster.quotemonster
#
#   Email: quotemonster@xmlworks.com
#
#   Since this is free software, no support is offered. However, suggestions 
for improvements
#   to the program are always welcome. If you are using the program and find it 
useful,
#   please drop us a line sometime. We would enjoy hearing from you.
#
#   Snail Mail: XMLWorks
#               PO Box 941124
#               Atlanta, GA 31141-1124
#               USA 
#   Web: http://www.xmlworks.com/
#
#   Custom modifications to this script or other custom Internet programming
#   in Perl are also available.
#
#
# ----------------------- Enter your custom settings here 
----------------------------- #
#
# Instructions:
#
# Lines with comments are preceded by the hash symbol (#). The following lines 
assign
# values to variables. Some values are numerical and some are not. The 
numerical values
# are not enclosed in single quotes, the non-numerical entries are enclosed in
# single quotes ('like this'). All perl statements must end with a semicolon 
(;) and a
# comment may follow on the same line by placing the hash symbol (#) before the 
comment. 
# QuoteMonster reads the tickers to download from a tickers file, outputs the 
raw data
# to a temporary file, then converts the data in the temporary file into a 
format 
# suitable for importing to Metastock or other charting programs and stores the 
data
# in a data file. You may use the default settings here or change as needed:

#MacPerl 
#Still needs some date work - gm
$ARGV[0] = MacPerl::Ask("What is the start date (yymmdd)?","$date1");
$ARGV[1] = MacPerl::Ask("What is the end date (yymmdd)?","$date2");

# my $tickerfile  = 'tickers.txt';        # file containing tickers - one 
ticker per line
my $tickerfile  = MacPerl::Ask("What is the name of the tickers 
file?","tickers.txt");
my $inputFile   = 'quotes.csv';         # temporary file to store quote data 
before parsing
my $outputFile  = 'quotesout';         # file to store quote data after 
parsing

# Enter the order in which the price, date, and volume information should be 
stored.
# If you do not want an item to be saved, enter a zero (0).
my $t = 1;
my $d = 2;                                      # date
my $o = 3;                                      # open
my $h = 4;                                      # high
my $l = 5;                                      # low
my $c = 6;                                      # close
my $v = 7;                                      # volume

# Some charting programs may require the date year to be expressed as either a 
two or four
# digit year. Enter the number of digits to represent the year - enter either 2 
or 4.

my $y = 4;

# If your data file must contain a header, enter it here. An example of an 
optional
# Metastock header is on the next line following the #. You must make sure you 
have the
# correct order and number of data fields:
# my $header = '<date>,<ticker>,<open>,<high>,<low>,<close>,<vol>';

my $header = '';

# QuoteMonster can remove carets (^) from tickers of indices, such as ^OEX.
# If you do not want to remove carets, set the following value to 1. If you 
want
# to remove the carets, set the following value to 0 (default):

my $donotremovecarets = 0;

# Some chart programs want quotes in a particular date order. If you need to 
reverse the order
# of dates in the QuoteMonster output file set the following value to 1. If the 
date order
# does not matter you can leave it set at the default of 0.

my $reverse_order = 0;

# Enter domain and port information
# ONLY if you must go through a proxy server:

my $proxy   = '';                             # proxy server
my $port    = '';                             # proxy server port
my $noproxy = '';                             # local domain


# ---- Do not change anything below this line unless you know what you are 
doing ---- #
print "QuoteMonster has started...\n";
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
my $ticker;
my $last;
my $date;
my $time;
my $change;
my $open;
my $high;
my $low;
my $vol;
my $data;
my @quotes;
my $startdate = shift;
my $enddate = shift;
my $daily = 0;
unless (defined($startdate)) {
        my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = 
localtime(time);
        $mon++;
        $mday = '0' . "$mday" unless $mday > 9;
        $mon = '0' . "$mon" unless $mon > 9;
	$startdate = $year . $mon . $mday;
	$daily = 1;
	print "No date arguments given so using fast daily update mode\n";
}
$enddate = $startdate unless defined($enddate);
my $startday = substr($startdate,4,2);
my $startmonth = substr($startdate,2,2);
my $startyear = substr($startdate,0,2);
my $endday = substr($enddate,4,2);
my $endmonth = substr($enddate,2,2);
my $endyear = substr($enddate,0,2);
my $request;
my $day;
my $month;
my $year;
my $line;
my $i;

# RESET HISTORY FILE TO ZERO AND ADD HEADER IF REQUIRED:
unlink $inputFile, $outputFile;
open (HISTORY, ">$outputFile") || die("Can't clear history file - $outputFile: 
$!");
if (length($header)>0) {
	print "Adding header to $outputFile\n";
	print HISTORY "$header\n";
}
close HISTORY;
       
my %mon = 
('Jan','1','Feb','2','Mar','3','Apr','4','May','5','Jun','6','Jul','7','Aug','8'
,'Sep','9','Oct','10','Nov','11','Dec','12');
my $product_id="Mozilla/3.0";
my $ua = new LWP::UserAgent;
$ua->agent($product_id);
if ($proxy ne '' && $port ne '' && $noproxy ne '') {
	$ua->proxy('http', "http://$proxy:$port/"); 
	$ua->no_proxy("$noproxy"); 
}
open (TICKERS,"$tickerfile") || die("Can't open ticker file - $!");
my @tickers = (<TICKERS>);
close TICKERS;
chomp @tickers;
if ($daily == 0) {
	my $URL = 'http://chart.yahoo.com/table.csv?s=';
	foreach $ticker (@tickers) {
		$URL .= $ticker . '&a=' . $startmonth . '&b=' . $startday . '&c=' . 
$startyear . '&d=' . $endmonth . '&e=' . $endday . '&f=' . $endyear . 
'&g=d&q=q';
		print "Retrieving $URL\n";
		$request = new HTTP::Request('GET', $URL);
		$data = $ua->request($request)->as_string;print "BEFORE DATA:$data\n";
# Changes a unix return to a Mac return - all \n are not equal
		$data =~ s/\012/\015/gs;print "AFTER DATA:$data\n";
		open (OUT, ">$inputFile");
		print OUT "$data";
		close OUT;
		open (IN,"$inputFile") || die("Can't open $inputFile - $!");
		(@quotes) = (<IN>);
		chomp @quotes;
		open (HISTORY, ">>$outputFile") || die("Can't open output file for writing - 
$!");
		foreach (@quotes) {
			if (/^\d/) {
				($date,$open,$high,$low,$last,$vol) = split(/,/);
				$ticker =~ tr/a-z/A-Z/;
				$donotremovecarets || $ticker =~ s/\^//;
				($day,$month,$year) = split(/\-/, $date);
				$day = ('0' . "$day") if ($day < 10 && $day !~ /^0/);
				$month = $mon{$month};
				$month = ('0' . "$month") if ($month < 10 && $month !~ /^0/);
				$year = substr($year,2,2);
				$date = "$year$month$day";
				$line = '';
				if (defined($high)) {
					for ($i = 1; $i < 8; $i++) {
						$line .= $ticker . ',' if ($t == $i);
						$line .= $date . ',' if ($d == $i);
						$line .= $open . ',' if ($o == $i);
						$line .= $high . ',' if ($h == $i);
						$line .= $low . ',' if ($l == $i);
						$line .= $last . ',' if ($c == $i);
                                                $line .= $vol . ',' if ($v == 
$i && defined($vol));
					}
					$line =~ s/,+$//;
				}
				else {
					$line = $ticker . ',' . $open;
				}
				print HISTORY "$line\n";
			}
		}
		close HISTORY;
		close IN;
		$URL = 'http://chart.yahoo.com/table.csv?s=';
		sleep 5;
	}
}
else {
	my $URL = 'http://quote.yahoo.com/d/quotes.csv?s=';
	my $testLength = length($URL);
	my $count = 0;
	foreach $ticker (@tickers) {
		$ticker =~ tr/A-Z/a-z/;
		$URL .= $ticker.'+';
		$count++;
		if ($count == 20) {
			qet_quotes($URL);
			$count = 0;
			$URL = 'http://quote.yahoo.com/d/quotes.csv?s=';
		}
	}
	if (length($URL) > $testLength) {
		qet_quotes($URL);
	}
	open (IN,"$inputFile") || die("Can't open $inputFile - $!");
	@quotes = <IN>;
	chomp @quotes;
	open (DAILY, ">$outputFile") || die("Can't open output file for writing - 
$!");
	if ($header ne '') {
		print DAILY "$header\n";
	}
	foreach (@quotes) {
		if (/^"/) {
			($ticker,$last,$date,$time,$change,$open,$high,$low,$vol) = 
split(/,/);
			$ticker =~ s/"//g;
			$ticker =~ tr/a-z/A-Z/;
			$date =~ s/"//g;
			if ($date ne 'N/A') {
				($month,$day,$year) = split(/\//, $date);
				if ($month > 12) {
					die("FATAL ERROR: Improper date format.");
				}
				$year = substr($year,2,2) if ($y == 2);
				$day = ('0' . "$day") if ($day < 10);
				$month = ('0' . "$month") if ($month < 10);
				$date = "$year$month$day";
				$line = '';
				for ($i = 1; $i < 8; $i++) {
					$line .= $ticker . ',' if ($t == $i);
					$line .= $date . ',' if ($d == $i);
					$line .= $open . ',' if ($o == $i);
					$line .= $high . ',' if ($h == $i);
					$line .= $low . ',' if ($l == $i);
					$line .= $last . ',' if ($c == $i);
					$line .= $vol . ',' if ($v == $i && $vol ne 'N/A');
				}
				$line =~ s/,$//;
				print DAILY "$line\n";
			}
		}
	}
	close DAILY;
	close IN;
}
if ($reverse_order) {
	print "Reversing date sequence\n";
	my $tempfile = 'quotes.~mp';
	unlink $tempfile;
	rename $outputFile, $tempfile || die "Could not rename $outputFile $!";
	open (IN,$tempfile) || die "Could not open $tempfile $!";
	open (OUT, ">$outputFile") || die "Could not open $outputFile $!";
	my @quotes = (<IN>);
	if ($header ne '') {
		$header = shift @quotes;
		push @quotes, $header;
	}
	@quotes = reverse @quotes;
	print OUT @quotes;
	close IN;
	close OUT;
	
}

sub qet_quotes {
	$URL = shift;
	$URL =~ s/\+$//;
	$URL .= '&f=sl1d1t1c1ohgv&e=.csv';
	print "Retrieving $URL\n";
        my $ua = new LWP::UserAgent;
	my $request = new HTTP::Request('GET', $URL);
        my $response = $ua->request($request);
	if ($response->is_success) {
		$data = $ua->request($request)->as_string;
                $data =~ s/\x0D//g;
		open (OUT, ">>$inputFile");
		print OUT "$data";
		close OUT;
		sleep 5;
	}
	else {
		open (OUT,">>error.log") || die("Can't open error log: $!");
		print OUT "\nError retrieving $URL \n";
		print OUT $response->error_as_HTML;
		close OUT;
	}
} 

__END__

Follow-Ups:
- Re: [MacPerl] Help with Quotemonster
  - From: Chris Nandor <pudge@pobox.com>

Prev by Date: [MacPerl] O'Reilly Perl books and learning Perl
Next by Date: Re: [MacPerl] Code cleaning and questions
Prev by thread: [OT] Re: [MacPerl] Quick Question
Next by thread: Re: [MacPerl] Help with Quotemonster
Navigation: Date Index | Thread Index | Search | Other lists at bumppo.net