NOTES ON WXSIMATE Version 6.3 (c) Copyright Thomas J. Ehrensperger 2017 WXSIMATE is a companion program the WXSIM Weather Simulator, Versions 10+. WXSIMATE's purpose is quick and easy downloading, culling, and appending of most of the types of internet-based data which WXSIM can use. Also, WXSIMATE can compile data from Davis WeatherLink software (Version 5.2 or later, for use with Vantage Pro and possibly other Davis weather stations) for import into WXSIM. ----------------------------- REVISION HISTORY----------------------------- Version 6.3 (May 14, 2017) accommodates a change in the names (a period after the month name) of CumulusMX log files, as they appear at least in Spanish regional Windows settings. Version 6.2 (October 23, 2016) changes the URL's of FSU's METAR, SYNOP, and RAOB data, to account for changes on their server. Version 6.1 (June 10, 2016) changes the URL's of a METAR site, a GFS MOS site, a NAM MOS site, and a NAM FOUS site to account for the discontinuation of the weather.noaa.gov server. There is no net loss of data sources (and in fact a net gain of one). Version 6.0 (December 28, 2015) disables the albany.edu surface and upper air data options, as these appear to have been discontinued, along with the FTP sources for MOS and METAR data. There are still several options for most data types (just one now for MOS, but it is direct from the source - NOAA). Also, the main form has been cleaned up a bit, including removal of the previously previously discontinued and disabled NGM options. A version 5.9 was briefly posted in October, but not officially released. This had a minor bug fix, but I failed to document it at the time and am now not sure what it was, though it was something rarely encoutnered. Version 5.8 (January 17, 2015) removes the option of 0.25 degree GFS data import, because the 0.25 degree model is in fact the operational one, having replaced the 0.5 degree model. As of this writing, both McMahon and Bohler are using a 0.5 degree data grid for interpolation (though this may be changed to the 0.25 degree grid later), but the underlying model is based on the 0.25 degree grid. Version 5.7 (November 23, 2014) enables WXSIMATE can now read WeatherLink's .wlk files without returning a 'Permission denied' error and requiring you to temporarily close WeatherLink. Version 5.6 (September 3, 2014) corrects a bug affecting import of data from WeatherLink (only). This affected only the first month of data after the station has started using WeatherLink. It caused gaps in the station data as seen by wret.exe, and prevented effective use of autolearn in those cases. Now the data should be good even in the first month of use of WeatherLink. Version 5.5 (January 31, 2014) adds an http option for the NOAA METAR Cycles data source. There was already an FTP option for this, but the http option may prove more reliable as it is executed more simply. Version 5.4 (December 3, 2013) corrects a bug which could rarely cause an invalid property value error, assiciated with wind correction factors saved from older versions of the program. Version 5.3.2 (September 21, 2013) corrects a problem which caused failure to employ local station data for calibbration purposes in WXSIM when 5 minutes before the hour was chosen as the initialization time (a somewhat rare setting). Version 5.3.1 (June 27, 2013) corrects a problem with the recently introduced ability to read Cumulus data, In particular, it was found that Cumulus' log file names are language senstive, while in WXSIMATE 5.3, the names were assumed to be English-language ones. Now, WXSIMATE effectiviey detects the regional and language settings, on order to establish the correct Cumulus log file names. Minor cosmetic and informational changes were also made. Version 5.3 (March 2, 2013) can now read data from the program Cumulus, by Sandaysoft, in addition to the previous set of software packages with which it is compatible: Davis WeatherLink, Brian Hamilton’s Weather Display, Ambient’s Virtual Weather Station, and Quimisur (a Spanish company). At present, soil data is not read for Cumulus, Virtual Weather Station, or Quimisiur. There is also now another option for site-specific GFS data, in addition to that downloaded and processed by Chris McMahon. The new source is provided by Samuel Bohler, and can be accessed using the 'Bohler' option (Chris' source is now labeled 'McMahon'). There may be very slight differences between the two sources in cloud cover and precipitation, due to a different interpolation and averaging technique. The effect of these differences on WXSIM output is trivial. Now included with the WXSIM/WXSIMATE package is a small program called trimlog.exe. This can run on its own scheduler Version 5.2 (July 16, 2012) has improved error/situation handling, especially for cases of non-responding web sites or on-line data. It also has improved communication with WXSIM in the case of running these programs under the WXSIMPRO multi-site controller. Version 5.1 (August 23, 2011) corrects a bug which could cause the program to skip download of fresh GFS data if the new (in 5.0) feature 'Download GFS only if new data'was *not* checked (the new feature seems to work fine if activated). Also, suppression of the messages like No data for month 7' - which often occurred with brand new stations - has been enabled in auto run, so that users with new weather stations can have uninterrupted auto runs immediately. Version 5.0 (August 11, 2011) introduces the option to import soil temperature and moisture from up to five different depths. These data can be used in WXSIM (professional mode) to initialize soil conditions for forecasts of soil and agricultural data. At present, import is possible from either WeatherLink (up to five depths) and Weather Display (one depth only, though there is a possibility more depths may be added in future versions). Soil moisture is assumed to come from Watermark (TM) sensors, such as those sold by Davis Instruments. The units are 'centibars of dryness', though this is converted (with some uncertainty) into moisture percent by volume for use in WXSIM's internal routines. To set depths of sensors, click on the 'Soil' button at the lower right part of WXSIMATE's main form. A form will appear with more instructions. You can revisit this form after local data inmport to view the data, which is also written to locallog.txt. The data file which is actually written for use by WXSIM is soildata.txt. After the year, month, date, and hour, five sets of three numbers appear - the depth in millimeters, the temperature in tenths of a degree Kelvin, and the moisture tension in millibars. Missing values are coded as 9990. Another new option in Version 5.0 is the option (under 'Customize') to 'Download GFS only if new data' are available. This checks a file saved from the most recent run, containing the location and the time of the last GFS model run (for home site and separately for advection). When the command to get these data is executed, the program will first check a status file on the site for the last completed model run time and compare this to the time of the last data downloaded. The current site (in case you use multiple sites) is also compared with the site recrded with the last download. If the time and site match, the download will not be performed, so as to save time and resources. Instead the previously downloaded (and identical to current) data will be used. Version 4.8.3 (June 2, 2011) corrects a problem which started with 4.8.2: the average versus end-of-interval choice produced (in the former case) a value of 0.5 at the end of the wxsmini.txt file. On computers using regional and language settings with commas as a decimal separator will likely write this as "0,5", which can cause subsequent problems upon starting the program the next time. The meaning of the variable has been changed so that a value of 1 now replaces the 0.5. It may be necessary, after installing this new version, do check the settings under 'Archived data time settings' under 'Customize'. Changes were also made to avert an occasional error with the message that the file wmatelog.txt 'cannot be accessed because it is being used by another process'. This error was found to occur if WXSIMATE could not find one of the internet data sources and its scheduling timer sent another download instruction before the user could respond to the message that the source was not available. Now, the program will ensure that it has completed its first attempt before trying again. Also, the timer's initial initial attempts will be within 10 seconds of the start of the assigned minute instead of the previous limit of 15 seconds. Version 4.8.2 (February 19, 2011) adds the option, under Customize/ Archived data time settings (renamed from 'Minutes before hour'), to assume that archived station data being read refers to the end, rather than the previously assumed average, of the archive interval. This was done primarily to allow WeatherLink data archived every 5 minutes to show a time exactly on the hour, as opposed to 2 minutes before or after, as it was doing before. Note that you may need to set '5 minutes before the hour' to get it to show the exact hour (a technicality deriving from earlier considerations). Also, a message box (not appearing in auto run with error messages supporessed) now advises the user if wind units (in log files) are chosen other than knots when using Weather Display with metric units. It was learned that when metric units are chosen in Weather Display, wind speeds are automatically stored in knots, even if the displayed or output units are otherwise (i.e. km/hr). Version 4.8.1 (December 28, 2010) improves compatibility with Quimisur, discussed below. Version 4.8 (November 29, 2010) adds the capabilty of reading data from weather stations manufactured by Quimisur, a Spanish company. As of now, this (made at the request of a single user) is experimental. Version 4.7.2 (June 30, 2010) corrects an error occasionally encountered in reading Weather Display log files of a certain format. It also corrects a minor problem involving use of USNO time (a change actually made in an unofficial Version 4.7.1). Version 4.7 (February 15, 2010) adds the ability to archive internet data files into the directory of your choice (on the Scheduler form). Also, the Scheduler form's Disable button now saves settings, as well as the Enable one, and the captions of these buttons have been changed. Also, a bug, which stopped local station data import in January if the previous month's data was missing, has been fixed. Finally, on starting the program now checks to see if the computer is on daylight savings time and alerts the users if there is a discrepancy between this status and that of the DST check box. Another function of this routine is to determine if the computer is in the time zone indicated in the WXSIM custom file cty.fdt, and alert the user to any problems with regard to that. Slight cosmetic changes were made, also. Version 4.6 (September 12, 2009) adds the ability to download a new GFS-derived data set (courtesy of Chris McMahon) which enables use of GFS forecast data for advection and other purposes in WXSIM. This data set covers a 20 x 20 degree grid surrounding the home site, and will be appended along with the other downloaded data into wdata.txt (or whatever name you chose). Also, a new check box on the Schedule form allows the user to specify for WXSIMATE to close right after download and/or local station import on auto runs. This avoids the need to have an external program such as System Scheduler shut WXSIMATE down in addition to opening it. Note that if you are downloading internet data AND importing local station data, the progress bar on the latter may not appear to complete before the program closes, but localdat.txt, localcal.txt, and locallog.txt will be created as intended. Also, WXSIMATE's Import Data button (including auto run use) now includes precipitation in WeatherLink-derived data files for comparisons in wret.exe (VWS and Weather Display options already had precipitation saved). Changes were also made to reduce the likelihood of errors in the event of an internet source not responding. Now, there will still be a message (unless suppressed) informing you that the source is missing, but older data (if it was ever received) will be substituted by default. Finally, the GFS and GFS for Advection file lengths are now known to the program so the progress bar should update more realistically. Version 4.5.1 (June 26, 2009) simply updates the Help file to include information on using Virtual Weather Station with WXSIMATE. Version 4.5 (March 21, 2008) disbled import of NGM FOUS and NGM MOS, as these are no longer produced. The other FOUS and MOS products remain available. Version 4.4 (December 21, 2008) accommodates a change in the URL of NAM MOS data from the web-based server. Note that as of March 3, 2009, the NWS will discontinue all NGM products, including NGM MOS and NGM FOUS. This should not significantly impact forecast accuracy, even in the U.S., but the NGM options should be unchecked at that time, in both WXSIMATE and WXSIM. Version 4.3 (November 26, 2008) simplifies use of the various check boxes and drop down boxes on the main form, by enabling items only when relevant, meaning that if 'parent' check boxes (on which they depend) are not checked, the 'child' boxes will be greyed out. Version 4.2 (November 15,2008) improves error handling in the event that the 'coolwx' buoy data site is not responding. This should avoid stalls and crashes that may have resulted in the past. Version 4.1 (October 10, 2008) prevents excessive attempts to find FOUS data when the site is not responding, while in auto mode with error message suppresion on. Previously, under these conditions, it would look for up to six files (NGM and NAM/ETA for all three sites); now it quits trying if the first file is not present. Also, a new FOUS server (a direct NOAA one) has been added as an option (in addition to the OSU one). Version 4.0 (July 10, 2008) adds optional download of ozone column in Dobson units, from KNMI/ESA (see http://www.temis.nl/). It creates a file called dobunitsxxx.txt (where xxx are the first three letters of the site name), which is then read and used by WXSIM if the date and site in the file match those in WXSIM. Version 3.9.2 (March 21, 2008) corrects a problem with local station import using WeatherLink, in which a 'read to end of file' error could occur when spanning months. (This change was experimentally released to a couple of customers as Version 3.9.1). 3.9.2 also updates the OSU FOUS data URL, as that site went down and the URL's were changed when it was brought back on line. Version 3.9 (February 23, 2008) enables import of Ambient's Virtual Weather Station (VWS) data in addition to Davis WeatherLink and Brian Hamilton's Weather Display. Also an option to import METAR data from NOAA via FTP has been added. Version 3.8.1 (January 4, 2008) enables the local data import routine to ignore a certain minor data error that occasionally occurs in WeatherLink's .wlk files. WXSIMATE will now continue reading data in spite of the error and will usually manage to construct a good, almost continuous record. Version 3.8 (December 27, 2007) now allows optional (choose under Customize) suppression of error messages during auto run. Also, a new download option (FTP) for MOS products has been added, in response to intermittent problems that recently developed with downloads from the http source. A box showing minutes (to the right of the hour box) was added in support of the new FTP MOS option, so that the latest data can be better identified. Finally, a file, wmatelog.txt, is now created to record any error messages that might occur during internet download. Version 3.7 (October 6, 2007) is now enabled for use with a multi-site forecasting system under development. This will not affect most users. Version 3.6 (May 31, 2007) has improved handling of possible errors or gaps in .wlk (WeatherLink) files, when reading these files for production of localdat.txt, localcal.txt, and mmyyyywl.txt files. Also, during internet data retrieval, WXSIMATE will now try all known URL's before giving a message that it couldn't find a web site. This applies to METAR, SYNOP (except U.S. subset from FSU), and RAOB, each of which have at least three options for the data source. Version 3.5 (March 3, 2007) now properly saves all scheduler settings (before, some check boxes weren't saving) on form closing and exit. Also, a bug which allowed the program to think there was a February 29 in non-leap years, has been corrected. Version 3.4.1 (January 28, 2007) allows saving of initialization file data (in wxsmini.txt) even if the program is improperly or externally closed, such as by System Scheduler. Also, the default directory in the installation package's initialization file is now c:\wxsim. Finally, a price increase from $39 to $49 has been instituted, but it can now be bundled with WXSIM to save $9 relative to buying separately. Version 3.4 (December 30, 2006) adds a file called locallog.txt, which is a record of any imported local station data. Also, wording of some error messages in WXSIMATE has been improved. Version 3.3 (November 8, 2006) adds the ability to download GFS model data, culled from GRIB files by Chris McMahon, effectively eliminating the need for READY data for most users. The query is specifically for your site, assuming you have registered WXSIMATE. NOTE: registered users of older versions can contact me at eburger@aol.com for inclusion in the site list at no charge. Version 3.2 (August 12, 2006) now allows for use of the declared units (under Customize/Units (for display here)) in the saved mmyyyywl.txt files used for forecast/actual comparisons in WXSIM's retrieval module (wret.exe). Version 3.1 (August 10, 2006) adds the ability to generate files, of the name format mmyyyywl.txt, from imported WeatherLink data, so that the retrieval module (wret.exe) can do comparisons with WeatherLink data, in addition to the Weather Display capabilities already included. Version 3.0.1 (August 2, 2006) corrects a bug which resulted from the fix to enable tolerance of a blank line in WD log files (see below). This bug had caused an error in importing local data, but only on the first day of the month. The tolerance for blank lines of at least two spaces is retained. Version 3.0 (July 24, 2006) adds an important new feature: the ability to schedule automated downloads of data for use in the new auto run feature of WXSIM. It can also archive these files with date-stamped names (though the fixed, user-defined name at upper right is what should be imported in WXSIM). Other changes include the ability to import local data from URL's, an option to accommodate a 1 hour delay in RAOB data, and the ability to tolerate a blank line in WD log files. Version 2.1b (February 7, 2006) was a quick fix for a couple of occasionally encountered problems with FOUS import. Data for few stations, mainly in Alaska, were not being retrieved, and there was redundancy and spacing problems in the culled data. Version 2.1 (January 17, 2006) corrects an earlier oversight which had left three line METARs out of the culled file. Also, a new option was added: 'First instance only', which - if checked - allows only the first METAR for a given site to be included in the culled file (still subject to the other restrictions). Version 2.0.1 (January 10, 2006) corrects an 'Index out of bounds of array' bug that surfaced in the last few weeks in the Weather Display module. Version 2.0 (December 29, 2005) standardizes the 'working directory' to be the 'current' one (in which WXSIM is installed). This should eliminate some 'file not found' errors that resulted initially when users installed into directories other than the default c:\wxsim. Also, error trapping and messages have been improved to avoid other possible 'file not found' errors. Finally, a bug, which had prevented proper reading of negative temperatures from Weather Display files, was corrected. Version 1.9 (July 5, 2005) corrects a bug involving culling of METAR data. This bug may have resulted in missing some METAR data, or even in some cases ascribing data to the wrong station. The culled METAR files should now be consistently accurate and also a bit cleaner and more compact. Version 1.8 (June 1, 2005) improves handling of bad data from home weather stations (as might occur in some wireless installations). It also corrects a problem in which the current day's date was used for RAOB data download, even with 12Z specified and an actual time before that (such a file would not yet exist). Now under these circumstances WXSIMATE correctly chooses the previous day's date. Also, an item has been added to some of the 'File not Found' error messages - a reminder to check to see if the 'Computer on Day- light Savings Time' box is correctly checked (or unchecked). Oversight of this had caused some users problems with files not being found, because they were an hour in the future. Version 1.7 (February 21, 2005) has improved error handling. In particular, a call to cty.fdt Version 1.6.3 (February 9, 2005) corrects a bug which occasionally triggered an error while culling METAR, due to data which could be non-numeric being treated as if it were numeric. Version 1.6.2 was not generally released. Version 1.6.1 (January 4, 2005) updates the URL's for the Florida State (FSU) data sources, which changed recently. Version 1.6 (January 2, 2005) corrected a bug which triggered an inappropriate error when importing local station data from Weather Display, in January (where the program had to look back into December and got confused about the year). This situation was already handled properly for the WeatherLink option. Also, some slight cosmetic changes were made, along with some modification of error handling routines. Version 1.5 (December 30, 2004) added two new data culling features. One is the ability to restrict METAR to reports within 10 minutes of the hour (i.e. from 1850Z to 1910Z). This not only reduces file size a bit but also helps maintain good synchronization of advection data. The other new feature is the ability to accept only every other MOS entry. While one would usually want the full data set, the large file size can make data import in WXSIM a bit slow, at least on older computers. This new option can cut access time almost in half with only a minor loss of information. Also, Version 1.5 has minor cosmetic changes, including color coding to distin- guish the two different main functions: blue for internet download and green for download from local weather station software. Version 1.4 (December 10, 2004) added Albany State University as a source for RAOB data. Also, the user can now specify 00Z or 12Z times for RAOB data, in consideration of sites that report only once a day. Also, a bug capable of causing the program to stall while culling a certain type of incomplete RAOB data has been fixed. Version 1.3 had minor bug fixes involving local data import. It was released only to a few customers and was never made public on my web site. Version 1.2 (October 19, 2004) corrected problems for German users (or anyone using decimal points in the date format), in which errors occured when trying to use local station data. It should now work properly. Version 1.1 introduced the ability to import data from at least 20 different brands of weather stations, via Weather Display, a program by Brian Hamilton. To learn more about that program, and to see which stations it supports, see http://www.weather-display.com/. ------------------------------ NOTE --------------------------------------- As of this writing, WXSIMATE has been tested on many computers. It is running smoothly on several machines with Windows XP and one with Windows 98, but did not boot up properly in a test using Windows 2000 Professional. All the preceding tests were on computers in the United States. Two users in the Netherlands have reported an error in attempting to use the program under Dutch versions of Windows 2000 Professional and Windows XP. The program boots up, but then thinks a file (cty.fdt) is in use, when it really isn't. ----------------------------------------------------------------------------- WXSIMATE is written is Microsoft Visual Basic. Net, and therefore requires the Net Framework Version 1.1, which is widely available as an approximately 23 MB executable which then installs the framework on your system. See http://msdn.microsoft.com/netframework/technologyinfo/howtoget/default.aspx for more information, and the free download, or for a more direct download, use this link (you may need to scroll to the right in Notepad to copy it all): http://www.microsoft.com/downloads/details.aspx?FamilyID=262d25e3-f589-4842-8157-034d1e7cf3a3&displaylang=en To install WXSIMATE, simply make sure you have unzipped the package containing the present document into the directory containing your WXSIM files (i.e. c:\wxsim). The file to run is wxsimate.exe. To gain full use of the program's ability to download files for WXSIM from the internet, you must be a registered user of the WXSIM Weather Simulator. When you register, you will receive an activation code unlocking the program for download of data specific to your customized WXSIM site(s). The features for importing data from Davis WeatherLink 5.x software (used with Davis Instruments Vantage Pro system, for example), or from Weather Display (a program by Brian Hamilton) will work even without registration. Be sure to select your desired units under 'Customize' on the menu bar. The cost of registration is $34.95. Free upgrades will be available from time to time with no upgrade charges planned as of this writing. For more information on this program, WXSIM, and the latest pricing and ordering information for each, see the WXSIM web site at www.wxsim.html Note that, at present, WXSIMATE registration is available only directly from the author, by PayPal to eburger@aol.com or by check or money order to: Thomas J. Ehrensperger 2064 Mercer Avenue College Park, Georgia 30337 ----------------------------------------------------------------------- TROUBLESHOOTING: If the program doesn't start, or crashes, here are some things to check: (1) Make sure you have Microsoft's Net Framework 1.1 installed, as described above. (2) Make sure that you have already installed WXSIM, since WXSIMATE needs some of WXSIM's files. (3) Preferably, WXSIMATE and WXSIM should be in the same directory, which is usually called c:\wxsim (not crucial, but convenient). (4) You may also get 'file not found' errors when trying the home station import option. This is probably because either you have not chosen the correct station software option (WeatherLink or Weather Display, under Customize), or have not properly identified the directory in which this data can be found. For WeatherLink, WXSIMATE needs the directory of the .wlk files, and for Weather Display, it needs that of the '...lg.txt' files. (5) If the above tips don't solve the problem, you can ask me for help, at eburger@aol.com. ----------------------------------------------------------------------- DISCLAIMER: The author disclaims any responsibilty for damages of any type resulting from the use of this program or its output. This includes but is not limited to hardware or software corruption of your computer. ----------------------------------------------------------------------- FINALLY: Enjoy the program - I've enjoyed writing it! I'd love to hear feedback, whether you decide to register or not. My email address is eburger@aol.com. Thomas J. Ehrensperger June 19, 2017