Howard's Astronomical Adventures

Automating the retrieval of satelite information using Perl

Introduction

A wide variety of weather information of interest to astronomers is available on the Internet. Satellite imagery is one example that is very useful. Most of this information can automatically be retrieved by your computer for review at any later time. Several different methods could be used to do this but I will focus on using the programming language called Perl. Linux (or *nix) users are likely to be familiar with Perl. Windows users are not usually aware of this powerful tool (since it is not typically installed on their computers). If you are not sure what Perl is please consult Perl.org for additional information.

Windows users will need to go to ActiveState Perl Page to obtain the free Perl installation file. Download and run the self installing package. Perl is a commandline tool that is run from the DOS or command prompt. Here is Perl on my Windows 2000 system:

D:\temp>perl -v

This is perl, v5.6.1 built for MSWin32-x86-multi-thread
(with 1 registered patch, see perl -V for more detail)

Copyright 1987-2001, Larry Wall

Binary build 633 provided by ActiveState Corp. http://www.ActiveState.com
Built 21:33:05 Jun 17 2002


Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'.  If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.

Getting Started

With Perl installed here is a simple program to use. Copy and paste the code below into notepad and save it to a file, for instance get-wv.pl. Note that this will run on a *nix system but you may not have the module LWP::Simple (1).

#!/usr/bin/perl
#
# Simple Web Get of UCAR Satellite Information
#
use LWP::Simple;

my $source   = "http://www.rap.ucar.edu/weather/satellite/latest_US_wv.jpg";
my $filename = "latest_US_wv.jpg";
#
# fetch file and save it
#
my $response = getstore($source,$filename);
print "Response: $response for $filename\n";
exit;

You need to be connected to the Internet when you start the program. Here is what it looks like when you run the program (this may take a minute for dial-up users):

D:\temp>perl get-wv.pl
Response: 200 for latest_US_wv.jpg

D:\temp>

The program fetched the latest copy of the US Water Vapor image from the University Corporation for Atmospheric Research and saved it as the file latest_US_wv.jpg in the current directory (this is usually a large file).

Ok, so what? You can do that by saving the image from a browser. The advantage is we can easily automate the collect of many images. Unattended. Let's add a few more features to do that. First we need a unique filename everytime we grab an image:

#!/usr/bin/perl
#
# Simple Web Get of UCAR Satellite Information
#
use LWP::Simple;

#
# Location to save files
#
my $save_dir = "";

#
# Source URL for weather files
#
my $source = "http://www.rap.ucar.edu/weather/satellite/latest_US_wv.jpg";

#
# get time / date
#
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$mon++;
$year += 1900;
#
# fetch file and save it
#
my $filename = sprintf("%02d%02d%04d-%02d%02dwv.jpg",$mon,$mday,$year,$hour,$min);
my $response = getstore($source,"$save_dir$filename");
print "Response: $response for file: $filename\n";
exit;

Running the program saves a time stamped file like this:

D:\temp>perl get-wv.pl
Response: 200 for file: 07012003-1700-wv.jpg

D:\temp>

In the above case a file named 07012003-17200wv.jpg will be found in the d:\temp directory, use your favorite image viewer to open it. The '200' response code means everything is ok, this is one of the standard http response codes. For additional information on HTTP response codes see RFC 2616 or the book HTTP: The Definitive Guide. The above code also added the variable $save_dir. Although it was not used you could set it to whatever directory you want the files stored (2).

Now that we generate unique files the final step would be automatically grab images at regular intervals. This can be done by either scheduling the above program to run periodically (Unix cron or Windows Scheduler) or to change the program so the image is repeatedly fetched.

#!/usr/bin/perl
#
# Simple Web Get of UCAR Satellite Information
#
use LWP::Simple;

my $minutes = 60;
#
# Location to save files
#
my $save_dir = "";
#
# Source URL for weather files
#
my $source = "http://www.rap.ucar.edu/weather/satellite/latest_US_wv.jpg";
#
# Loop
#
for (;;) {
    #
    # get time / date
    #
    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
    $mon++;
    $year += 1900;
#
# fetch file and save it
#
    my $filename = sprintf("%02d%02d%04d-%02d%02dwv.jpg",$mon,$mday,$year,$hour,$min);
    my $response = getstore($source,"$save_dir$filename");
    print "Response: $response for file: $filename\n";
#
# Pause
#
    sleep( $minutes * 60 );
}
exit;


In the command prompt window run the program:

D:\temp>perl get-wv3.pl
Response: 200 for file: 07012003-1700wv.jpg
Response: 200 for file: 07012003-1801wv.jpg
Response: 200 for file: 07012003-1902wv.jpg
^C

D:\temp>

Note that I pressed Ctrl-C ( or Ctrl-Break ) to stop the program, otherwise it will run as long as you leave it.

Regardless of how you set it up, this simple program will fetch whatever web image or page you like. This can be handy when you want to monitor conditions for a extended time.

How it Works

The 'magic' is in the line 'use LWP::Simple'. This package provides an easy and simple way to grab a web image or page.

Notes

  1. On *nix systems use the CPAN.pm module to obtain LWP::Simple like this:

    #perl -MCPAN '-e install LWP::Simple'

    Alternatively, should you not have the tools installed to build the modules there may be compiled packages for your system like perl-libwww-perl-[version].rpm.

  2. Perl's idea of folders and directories is based on unix. This can be confusing to Windows users since a slash is used instead of a backslash unless it is 'escaped'. For example, to set the save_path to d:\temp it can be written two ways:

    • $save_dir = "d:/temp/";
    • $save_dir = "d:\\temp\\";