[Date Prev][Date Next][Thread Prev][Thread Next] [Search] [Date Index] [Thread Index]

[MacPerl] Help with Quotemonster

To: mac-perl@iis.ee.ethz.ch
Subject: [MacPerl] Help with Quotemonster
From: "adjei" <adjei@inreach.com>
Date: Thu, 1 Jul 1999 19:29:34 -0700

Quotemonster or qm is perl script that downloads historical quotes from the 
internet. It was working fine with Os 8.5.1 until all of a sudden it stared 
truncating the quotesout dates. For example a normal quotesout ( output) looks 
like this:

AOL,061799,106.5,113.75,103,110.6875,31982600
AOL,061899,112,113.5,109.5625,112,22946900

the faulty quotesout looks like this:

AOL,0617,106.5,113.75,103,110.6875,31982600
AOL,0618,112,113.5,109.5625,112,22946900

note the disappearance of the year-99.

I cannot figure it out. I upgraded to os 8.6-still same problem.
your help will be most appreciated.
I have attached the qm script.


#!/usr/bin/perl
#
#   IMPORTANT: READ THIS IN ITS ENTIRETY. YOU ARE THE FINAL AUTHORITY IN 
DETERMINING
#   WHAT HAPPENS ON YOUR COMPUTER. YOU SHOULD READ ALL DOCUMENTATION PERTAINING 
TO
#   THE INSTALLATION AND USE OF ALL SOFTWARE YOU INSTALL AND USE AND SHOULD 
MAKE 
#   REGULAR AND COMPLETE BACKUPS OF YOUR IMPORTANT DATA.
#
#   QuoteMonster - Download and parsing script for retrieving
#   stock quote data from Yahoo. 
#
#   -------------------------LEGAL 
INFO--------------------------------------------
#
#   This script is Copyright (C) 1998  Jim Michael and XMLWorks All Rights 
Reserved
#
#   This is version 1.0.2m (16 Dec 1998) By G Michel
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
#
#   Download and parsing script for www.yahoo.com. This script will most
#   assuredly NOT work with other quote sources. Contact us for assistance
#   if you would like to use this script for other quote sources.
#
#   ------------------------------USAGE 
INFO-------------------------------------
#
#   USAGE: qm start_date end_date
#
#   REQUIRED FORMAT OF START AND END DATES: yymmdd
#
#   IMPORTANT USAGE NOTES: This script resulted from the merging of the 
QuoteMonster
#   Daily and QuoteMonster Historical scripts. You can now download both your 
daily
#   and historical data using just this script. The date arguments start_date 
and 
#   end_date are now OPTIONAL. If no date argument is given then QuoteMonster 
will
#   default to today's date. If only one date argument is given then 
QuoteMonster will
#   download data for that date only. If both start_date and end_date are given 
then
#   QuoteMonster will download data for that date range. If no date argument is 
given
#   then QuoteMonster will use its group download mode and send several quote 
requests
#   at once, greatly increasing the speed of daily downloads of large numbers 
of quotes.
#   
#   YEAR 2000 NOTE: Date formats of Y2K compliant charting programs are not 
know yet.
#   Until it is known what the new format, if any, might be it is impossible to 
produce
#   a Y2K compliant version of the program. Once a year format is established 
and charting
#   programs are updated to use the new format, QuoteMonster may be modified to 
handle the
#   new date format, probably by making the year field use four digits, per the 
configuration
#   option below.
#
#   IMPORTANT NOTE RE MUTUAL FUNDS: Only the closing price is available using 
QuoteMonster 
#   to download mutual fund prices. Consequently you should not mix mutual fund 
tickers
#   with stock and index ticker in the same ticker file or your import program 
may generate
#   errors. To handle both types of data, keep a copy of the QuoteMonster 
script in two 
#   different directories, using one for stocks and index downloads and one for 
mutual funds.
#   You may also need to use a header at the top of your data file such as:
#
#   <date>,<ticker>,<close>
#
#   Note to Windows NT/95 users:
#
#   If you are on a Win32 platform (NT or Windows95/98) then we HIGHLY 
recommend that you
#   use Gurusamy Sarathy's Standard Win32 port of Perl 5.04 or higher. The 
ActiveState port
#   is newer, but this version is known to be stable and included all of the 
modules used
#   by QuoteMonster. You may download the free perl interpreter from
#   http://www.perl.com/ or see the XMLWorks website for location of the binary 
download.
#
#
#   ----------------------------------SUPPORT 
INFO--------------------------------------
#
#   A newsgroup for QuoteMonster has been created on DejaNews. You can access 
it at:
#
#             
#       
#http://www.dejanews.com/[LB=http://xmlworks.com/quotemonster/,LBT=Return%20to%2
0XM#LWorks]/group/dejanews.members.biz.quotemonster.quotemonster
#
#   Email: quotemonster@xmlworks.com
#
#   Since this is free software, no support is offered. However, suggestions 
for improvements
#   to the program are always welcome. If you are using the program and find it 
useful,
#   please drop us a line sometime. We would enjoy hearing from you.
#
#   Snail Mail: XMLWorks
#               PO Box 941124
#               Atlanta, GA 31141-1124
#               USA 
#   Web: http://www.xmlworks.com/
#
#   Custom modifications to this script or other custom Internet programming
#   in Perl are also available.
#
#
# ----------------------- Enter your custom settings here 
----------------------------- #
#
# Instructions:
#
# Lines with comments are preceded by the hash symbol (#). The following lines 
assign
# values to variables. Some values are numerical and some are not. The 
numerical values
# are not enclosed in single quotes, the non-numerical entries are enclosed in
# single quotes ('like this'). All perl statements must end with a semicolon 
(;) and a
# comment may follow on the same line by placing the hash symbol (#) before the 
comment. 
# QuoteMonster reads the tickers to download from a tickers file, outputs the 
raw data
# to a temporary file, then converts the data in the temporary file into a 
format 
# suitable for importing to Metastock or other charting programs and stores the 
data
# in a data file. You may use the default settings here or change as needed:

#MacPerl 
#Still needs some date work - gm
$ARGV[0] = MacPerl::Ask("What is the start date (yymmdd)?","$date1");
$ARGV[1] = MacPerl::Ask("What is the end date (yymmdd)?","$date2");

# my $tickerfile  = 'tickers.txt';        # file containing tickers - one 
ticker per line
my $tickerfile  = MacPerl::Ask("What is the name of the tickers 
file?","tickers.txt");
my $inputFile   = 'quotes.csv';         # temporary file to store quote data 
before parsing
my $outputFile  = 'quotesout';         # file to store quote data after 
parsing

# Enter the order in which the price, date, and volume information should be 
stored.
# If you do not want an item to be saved, enter a zero (0).
my $t = 1;
my $d = 2;                                      # date
my $o = 3;                                      # open
my $h = 4;                                      # high
my $l = 5;                                      # low
my $c = 6;                                      # close
my $v = 7;                                      # volume

# Some charting programs may require the date year to be expressed as either a 
two or four
# digit year. Enter the number of digits to represent the year - enter either 2 
or 4.

my $y = 4;

# If your data file must contain a header, enter it here. An example of an 
optional
# Metastock header is on the next line following the #. You must make sure you 
have the
# correct order and number of data fields:
# my $header = '<date>,<ticker>,<open>,<high>,<low>,<close>,<vol>';

my $header = '';

# QuoteMonster can remove carets (^) from tickers of indices, such as ^OEX.
# If you do not want to remove carets, set the following value to 1. If you 
want
# to remove the carets, set the following value to 0 (default):

my $donotremovecarets = 0;

# Some chart programs want quotes in a particular date order. If you need to 
reverse the order
# of dates in the QuoteMonster output file set the following value to 1. If the 
date order
# does not matter you can leave it set at the default of 0.

my $reverse_order = 0;

# Enter domain and port information
# ONLY if you must go through a proxy server:

my $proxy   = '';                             # proxy server
my $port    = '';                             # proxy server port
my $noproxy = '';                             # local domain


# ---- Do not change anything below this line unless you know what you are 
doing ---- #
print "QuoteMonster has started...\n";
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
my $ticker;
my $last;
my $date;
my $time;
my $change;
my $open;
my $high;
my $low;
my $vol;
my $data;
my @quotes;
my $startdate = shift;
my $enddate = shift;
my $daily = 0;
unless (defined($startdate)) {
        my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = 
localtime(time);
        $mon++;
        $mday = '0' . "$mday" unless $mday > 9;
        $mon = '0' . "$mon" unless $mon > 9;
	$startdate = $year . $mon . $mday;
	$daily = 1;
	print "No date arguments given so using fast daily update mode\n";
}
$enddate = $startdate unless defined($enddate);
my $startday = substr($startdate,4,2);
my $startmonth = substr($startdate,2,2);
my $startyear = substr($startdate,0,2);
my $endday = substr($enddate,4,2);
my $endmonth = substr($enddate,2,2);
my $endyear = substr($enddate,0,2);
my $request;
my $day;
my $month;
my $year;
my $line;
my $i;

# RESET HISTORY FILE TO ZERO AND ADD HEADER IF REQUIRED:
unlink $inputFile, $outputFile;
open (HISTORY, ">$outputFile") || die("Can't clear history file - $outputFile: 
$!");
if (length($header)>0) {
	print "Adding header to $outputFile\n";
	print HISTORY "$header\n";
}
close HISTORY;
       
my %mon = 
('Jan','1','Feb','2','Mar','3','Apr','4','May','5','Jun','6','Jul','7','Aug','8'
,'Sep','9','Oct','10','Nov','11','Dec','12');
my $product_id="Mozilla/3.0";
my $ua = new LWP::UserAgent;
$ua->agent($product_id);
if ($proxy ne '' && $port ne '' && $noproxy ne '') {
	$ua->proxy('http', "http://$proxy:$port/"); 
	$ua->no_proxy("$noproxy"); 
}
open (TICKERS,"$tickerfile") || die("Can't open ticker file - $!");
my @tickers = (<TICKERS>);
close TICKERS;
chomp @tickers;
if ($daily == 0) {
	my $URL = 'http://chart.yahoo.com/table.csv?s=';
	foreach $ticker (@tickers) {
		$URL .= $ticker . '&a=' . $startmonth . '&b=' . $startday . '&c=' . 
$startyear . '&d=' . $endmonth . '&e=' . $endday . '&f=' . $endyear . 
'&g=d&q=q';
		print "Retrieving $URL\n";
		$request = new HTTP::Request('GET', $URL);
		$data = $ua->request($request)->as_string;print "BEFORE DATA:$data\n";
# Changes a unix return to a Mac return - all \n are not equal
		$data =~ s/\012/\015/gs;print "AFTER DATA:$data\n";
		open (OUT, ">$inputFile");
		print OUT "$data";
		close OUT;
		open (IN,"$inputFile") || die("Can't open $inputFile - $!");
		(@quotes) = (<IN>);
		chomp @quotes;
		open (HISTORY, ">>$outputFile") || die("Can't open output file for writing - 
$!");
		foreach (@quotes) {
			if (/^\d/) {
				($date,$open,$high,$low,$last,$vol) = split(/,/);
				$ticker =~ tr/a-z/A-Z/;
				$donotremovecarets || $ticker =~ s/\^//;
				($day,$month,$year) = split(/\-/, $date);
				$day = ('0' . "$day") if ($day < 10 && $day !~ /^0/);
				$month = $mon{$month};
				$month = ('0' . "$month") if ($month < 10 && $month !~ /^0/);
				$year = substr($year,2,2);
				$date = "$year$month$day";
				$line = '';
				if (defined($high)) {
					for ($i = 1; $i < 8; $i++) {
						$line .= $ticker . ',' if ($t == $i);
						$line .= $date . ',' if ($d == $i);
						$line .= $open . ',' if ($o == $i);
						$line .= $high . ',' if ($h == $i);
						$line .= $low . ',' if ($l == $i);
						$line .= $last . ',' if ($c == $i);
                                                $line .= $vol . ',' if ($v == 
$i && defined($vol));
					}
					$line =~ s/,+$//;
				}
				else {
					$line = $ticker . ',' . $open;
				}
				print HISTORY "$line\n";
			}
		}
		close HISTORY;
		close IN;
		$URL = 'http://chart.yahoo.com/table.csv?s=';
		sleep 5;
	}
}
else {
	my $URL = 'http://quote.yahoo.com/d/quotes.csv?s=';
	my $testLength = length($URL);
	my $count = 0;
	foreach $ticker (@tickers) {
		$ticker =~ tr/A-Z/a-z/;
		$URL .= $ticker.'+';
		$count++;
		if ($count == 20) {
			qet_quotes($URL);
			$count = 0;
			$URL = 'http://quote.yahoo.com/d/quotes.csv?s=';
		}
	}
	if (length($URL) > $testLength) {
		qet_quotes($URL);
	}
	open (IN,"$inputFile") || die("Can't open $inputFile - $!");
	@quotes = <IN>;
	chomp @quotes;
	open (DAILY, ">$outputFile") || die("Can't open output file for writing - 
$!");
	if ($header ne '') {
		print DAILY "$header\n";
	}
	foreach (@quotes) {
		if (/^"/) {
			($ticker,$last,$date,$time,$change,$open,$high,$low,$vol) = 
split(/,/);
			$ticker =~ s/"//g;
			$ticker =~ tr/a-z/A-Z/;
			$date =~ s/"//g;
			if ($date ne 'N/A') {
				($month,$day,$year) = split(/\//, $date);
				if ($month > 12) {
					die("FATAL ERROR: Improper date format.");
				}
				$year = substr($year,2,2) if ($y == 2);
				$day = ('0' . "$day") if ($day < 10);
				$month = ('0' . "$month") if ($month < 10);
				$date = "$year$month$day";
				$line = '';
				for ($i = 1; $i < 8; $i++) {
					$line .= $ticker . ',' if ($t == $i);
					$line .= $date . ',' if ($d == $i);
					$line .= $open . ',' if ($o == $i);
					$line .= $high . ',' if ($h == $i);
					$line .= $low . ',' if ($l == $i);
					$line .= $last . ',' if ($c == $i);
					$line .= $vol . ',' if ($v == $i && $vol ne 'N/A');
				}
				$line =~ s/,$//;
				print DAILY "$line\n";
			}
		}
	}
	close DAILY;
	close IN;
}
if ($reverse_order) {
	print "Reversing date sequence\n";
	my $tempfile = 'quotes.~mp';
	unlink $tempfile;
	rename $outputFile, $tempfile || die "Could not rename $outputFile $!";
	open (IN,$tempfile) || die "Could not open $tempfile $!";
	open (OUT, ">$outputFile") || die "Could not open $outputFile $!";
	my @quotes = (<IN>);
	if ($header ne '') {
		$header = shift @quotes;
		push @quotes, $header;
	}
	@quotes = reverse @quotes;
	print OUT @quotes;
	close IN;
	close OUT;
	
}

sub qet_quotes {
	$URL = shift;
	$URL =~ s/\+$//;
	$URL .= '&f=sl1d1t1c1ohgv&e=.csv';
	print "Retrieving $URL\n";
        my $ua = new LWP::UserAgent;
	my $request = new HTTP::Request('GET', $URL);
        my $response = $ua->request($request);
	if ($response->is_success) {
		$data = $ua->request($request)->as_string;
                $data =~ s/\x0D//g;
		open (OUT, ">>$inputFile");
		print OUT "$data";
		close OUT;
		sleep 5;
	}
	else {
		open (OUT,">>error.log") || die("Can't open error log: $!");
		print OUT "\nError retrieving $URL \n";
		print OUT $response->error_as_HTML;
		close OUT;
	}
} 

__END__

AA
AAPL
AEP
AGC
AIG
AIT
ALD
ALT
AMP
AMZN
AOL
ARC
ASM
AVP
AXP
BA
BAC
BAX
BAY
BBY
BC
BCC
BDK
BEL
BHI
BKS
BMG
BMY
BNI
BPA
BS
C
CAT
CCI
CEN
CGP
CHA
CHV
CI
CL
COL
CPB
CQ
CSC
CSCO
CSDS
DAL
DD
DIS
DOW
EK
ETR
F
FDX
FLR
GD
GE
GM
GMH
GT
GTR
HAL
HELE
HET
HIG
HM
HNZ
HON
HRS
HWP
IBM
IC
IFF
IKN
INTC
INTL
IP
JNJ
JPM
KEY
KM
KO
LTD
MAY
MCD
MER
MKG
MMM
MO
MOB
MRK
MSFT
MTC
MZON
NPR
NSC
NSCP
NSM
NT
ODP
OEX
ONE
ORCL
OXY
PCG
PEP
PFE
PG
PHG
PNU
PPW
PRD
PRMA
RAL
ROK
S
SLB
SO
SPX
STK
T
TAN
TEK
TMX
TOL
TOY
TSO
TX
TXN
UCM
UIS
UK
USB
UTX
VIX
VNTV
WCOM
WMB
WMT
WY
XON
XRX
Z

Follow-Ups:
- Re: [MacPerl] Help with Quotemonster
  - From: Chris Nandor <pudge@pobox.com>

Prev by Date: Re: [MacPerl] Re: [semi-off] malaprop
Next by Date: Re: [MacPerl] Troubles using GD examples
Prev by thread: Re: [MacPerl] CFM68K support (Was: Troubles using GD examples)
Next by thread: Re: [MacPerl] Help with Quotemonster
Navigation: Date Index | Thread Index | Search | Other lists at bumppo.net