NOTES ON AUTOLEARN.EXE AND WXSIM-LITE.EXE
			       
                 (c) Copyright Thomas J. Ehrensperger 2017

Along with the main WXSIM programs (wxsim.exe, wret.exe, and wxsimate.exe) and
a couple of minor accessories (wxerr.exe and trimlog.exe), this installation
package includes two extra programs: autolearn.exe and wxsim-lite.exe.  Each of 
these - and especially the two used together - can make very significant
improvements in WXSIM's forecasts (mainly of temperature and dew point).  No
Desktop icons are automatically created for these programs, but you may use
Windows to create them if you like.  They are both stored in the main WXSIM
folder (default name C:\wxsim) and can be executed by double clicking in
Windows Explorer or My Computer.

Autolearn-WX (the official name of autolearn.exe) was first introduced in August,
2013.  It greatly facilitates use of a feature of wret.exe - the ability to
compare hundreds of old forecasts to the actual weather which resulted (as
recorded by personal weather station software) and then use the differences to
generate four different types of bias-correction factors, which can in turn be
used by WXSIM to greatly reduce systematic errors.  With autolearn, this otherwise
manual procedure can be automated to run on a daily basis, at an inobtrusive time
(liek the middle of the night), which is a good thing, since it generally takes
at least several minutes for this analysis to run.  Autolearn also keeps track of
its own progress in improving forecasts, with both text and graphical products,
the latter of which were added in December, 2014, for the release of WXSIM 2015.

Autolearn, like all WXSIM programs, is a free download.  However, to use it at
full capacity requires a one-time registration fee of $40.  Upgrades to the program
are and will remain free.

WXSIM-Lite (the official name of wxsimlite.exe) was inspired by two main things:
(1) the success of Autolearn-WX in reducing bias and improving WXSIM forecasts,
and (2) the frustrating barriers to further improvement once systematic errors
had been greatly reduced.  Government and commercial forecast products have been
improving over the years, especially in increasing geographical specificity,
temporal resolution, and range of forecast details - all of these formerly relative
strengths of WXSIM.  For many users, some of these other products were doing better
than WXSIM, after all the effort I'd put into it!

I decided to try another approach: instead of trying to do sophistcated internal
modeling, I could use an approach similar to autolearn's to make site-specific bias
corrections to raw model data from GFS (and perhaps other models in the future).
This is not an original idea, as various tools, such as Model Output Statistics
(MOS) have been applied to raw model data for decades.  However, these efforts are
usually specific to selected, official reporting stations.  Applying bias-correction
techniques to data from individuals' personal weather station data has the potential
to provide these benefits to anyone with such a station, regardless of whether any
official sites are nearby (or are even similar).

I started testing such a system in late 2013, on over a dozen sites (in most cases
using data from customers) and continued through much of 2014. The bias correction
techniques overlap with those in autolearn.exe, but also include corrections specific
to each half-hour of the day.  More details on this appear in the Help section of
WXSIM-Lite, but briefly, among the many things accounted for are the exact minute of
sunrise, the degree of urbanization of the site, and the amount of short-term
variablity (minute-by-minute) in the site's temperatures.

The results were very encouraging, showing a very large improvement over the raw GFS
model data, and even a significant improvement (of 5 to 30%) over WXSIM's forecasts.
Now a new question arose: if these forecasts are better than WXSIM's, should this new
program replace WXSIM?

WXSIM has an absolutely enormous number of features, most of which most users never
scratch the surface of.  It not only forecasts the standard meteorological fare, but
also provides a wealth of very specific weather data, detailed soil and agricultural
modeling, and even the ability to model the temperature effects of solar eclipses.
It also is a great educational tool, in its interactive, simulation mode of use.  It's
too much, and too good, to leave behind.

Fortunately, I soon found a solution.  WXSIM could 'listen' to WXSIM-Lite's forecasts
of temperature and dew point, and take them into account, via a file written by the
latter and read by the former.  WXSIM could then still provide all its existing
products, but with improved accuracy.  Furthermore, I did statistical studies of the
output from the two programs and found that they are only partially correlated, which
means WXSIM has a somewhat individual 'opinion', rather than just being a degraded
version of the same source information.  This independence means that, even if WXSIM's
forecasts are slightly less accurate, they still can add value to WXSIM-Lite's, if
mixed together, with perhaps more weight given to WXSIM-Lite's data.

I've found that, indeed, such a mixture can produce results superior to either alone,
especially in the case of nighttime temperatures, where WXSIM seems to have a somewhat
'original' opinion.  In two months of testing parallel runs at 0, 50, 80, and 100%
WXSIM-Lite weighting, the 50 and 80% weights are considerably more accurate than the
zero one (pure WXSIM), but also a bit better than the 100% one, espcially at night.
I will continue to research the ideal weight to use, but right now, anything from 50
to 80% appears to be nearly ideal.

Both autolearn.exe and wxsimlite.exe take some time to become effective, because a
sufficiently large base of past forecasts must be collected to 'learn' from.  This
takes at least a month, and doesn't reach full effect for two or even three months.
Once they reach this point, though, they automatically continue to fine-tune the
forecasts, taking into account seasonal and other gradual changes.  Typically, either
program alone should produce at least a few percent improvement in wXSIM's forecasts,
and together, they can improve them even more.  My current guess is that, for a well-
customized site, wxsimlite.exe will produce a more dramatic improvement, while
autolearn.exe may be most helpful for sites for which I had difficulty in getting the
original customization right.

My intention is to continue to develop and improve WXSIM-Lite, thereby also improving
WXSIM.  This involves providing the specially prepared data for each user's location.
The raw data for this is free (courtesy of U.S. taxpayers), but there is a modest cost
for the server usage and we (Sam Bohler and I) need some compensation for the time
and effort spent developing this (and incentive to continue to do so).  I think this
more naturally fits a subscription model, rather than an up-front cost and/or upgrade
charges.

So, my plan is to make the WXSIM-Lite software and upgrades to it free of charge, and
also to provide the data free of charge long enough to establish the system's value.
I plan to implement a low-cost subscription system, at no more than $5 per month,
sometime in 2017 (though I planned this in 2016, and never got around to it!).  I
promise to keep the rates low!  Thanks for understanding the reasons for this. 

By the way, all the data for WXSIM itself (downloaded thorugh WXSIMATE) will remain
free of charge, as will upgrades to all WXSIM programs, for the foreseeable future.

NOTES ON LATEST VERSIONS:

Autolearn.exe and wxsimlite.exe are now on versions 3.3 and 2.3, respectively.  A minor
new feature is the new, auxilliary program wretmate.exe, which can be opened either
on it's own or via autolearn.exe, wxsimlite.exe, or wret.exe.  This new program allows
you to reversibly cull out (by renaming) forecast (.wxf) files that are too old, non-
standard in format or content, or too numerous, to speed use of the other programs.
The Version 2.3 of WXSIM-Lite allows for a name change in CumulusMX log files (which
I've heard of only under Spanish settings at this point), where a period got inserted
after the month name.  It also has language better clarifying the Setup routine, and
finally, also corrects a possible bug affecting the use of multiple sites.  A significant 
fairly new feature in WXSIM-Lite and autolearn of these is the (optional) ability to go
back and include data from previous years in the analysis.  In the case of autolearn, it
can go back to the current season from each of the two preceding years to use up to
90 days of data from each of those years.  It actually reads from previously stored
data in correc.txt, so you must have been using the program for at least a year to
use this feature.  In the case of wxsimite.exe, you can go back only 1 year and you
must have been using at least the GFS download feature to collect gfsdat0...txt files.
For each program, you must of course have home weather station data to compare to.  

Thomas J. Ehrensperger
June 19, 2017