The MSG Data Manager

What is MSG?

Meteosat-8 (or MSG-1 as it was known during trials) is the first of the new European geostationary weather satellites.  MSG is an acronym for Meteosat Second Generation.  Meteosat-8 (MSG-1) served from 2003 to 2007 AND is now providing Indian Ocean Data Coverage (IODC), MSG-9 is now the standby  Rapid Scan satellite, with Meteosat-10 providing Rapid Scan, and MSG-11 as the prime zero degree satellite since January 2013.  Meteosat-8/9/10/11 have twelve sensor channels rather than three, higher spatial resolution, much higher visible channel resolution, better thermal accuracy, and double the frequency of image delivery than previous Meteosat satellites; all of which necessitates a change to the data transmission method.  Meteosat-11 (MSG-4) has replaced Meteosat 7, and the WEFAX and PDUS transmissions have now ceased.  The data from MSG is purely digital, and is delivered as a set of a number of compressed files per image.  As there are now twelve channels delivered four times an hour, the total data rate is some 1.5GB per hour, 15000 files per day, requiring significant management on your computer.  

Please note that you will need a licence from EUMETSAT to receive some of this data, but UK amateurs should see the GEO Quarterly for more details.  They are unlikely to have to pay licence charges.  In addition to Meteosat-9 data and other geostationary weather satellite data, you will get access to the upper-atmosphere ATOVS data and Sea-Ice data.  These are processed from polar orbiting satellites and added to the data stream.

What is the MSG Data Manager?

  Download the MSG Data Manager

The MSG Data Manager is a Windows program that will process the files received from Meteosat-8/9/10/11.  These files might be received over the Internet in the future, but today they are easily received with a Digital Video Broadcasting (DVB) card installed in your PC, and a standard domestic satellite antenna, LNB, and cable.  A description of my own system is here.  The program can also process data received from the EUMETSAT FTP service (binary transfers, my FTP Helper program).  The data manager will automatically process the files into images, and optionally delete the large amounts of source data that would otherwise accumulate.  The program allows you to choose which of the twelve channels you process to images, and also allows you to keep some raw data in the processing PC should you wish.  
Please note that the MSG Data Manager and MSG Animator are original pieces of software developed independently by SatSignal Software, Edinburgh specifically for users such as you, and there is a lot of your input in the programs.  These are not some re-branded programs originally developed by someone else.  You will always get a fast response to the frequent changes made to EUMETCast data, and not be stuck waiting for the "next version".
More information about the data flow with MSG.

An Award!

My demonstration of Receiving and Processing Meteosat-8 Data with the MSG Toolset Plus suite (which comprises the MSG Data Manager, MSG Animator and GeoSatSignal) was voted the best of conference at the 2004 EUMETSAT Meteorological Satellite Conference in Prague.  The judgement was made based upon scientific content, presentation and overall interest and relevance.

Who uses the MSG Data Manager?

This software has been supplied to the Finnish Meteorological Institute, Meteo-Swiss, Universities, Museums, and other professional and amateur users throughout Europe.  

• "We are very happy with MSG Data Manager: it does what we want, in a perfect stability" - Michel Albuisson, École des Mines de Paris.
• "We use the MSG software 24/7 for operational forecasting information and it allows us to run animations of several channels simultaneously in different windows on dual displays" - Alan Hisscott, Senior Meteorological Officer, Isle of Man Meteorological Office.
• "I'm very impressed with the software. When the system was running too low on hard drive space the data manager stopped updating but nothing crashed, and the animator continued with old data - very nice!  I am happy to recommend your applications." - Jeff Lashley, UK National Space Centre.
• "My registration has just arrived for upgrade to MSG Data Manager and also AVHRR manager.  I must admit I'm blown away by it and have just sat down again after dancing around the room!  What have I been missing ah?  Awesome." - Rob Denton
• "Following the recent need for users to activate the MSG-2 data flow, I checked with Ops and they confirmed that since the 'failure' of MSG-1, all users had MSG-2 data enabled.  With David's software MSG Data Manager, all I had to do was change over one setting in the advanced section in order to resume full operations.  David's software has again proved invaluable and exceptionally easy to configure. My grateful thanks, David." - Lawrence Harris, freelance journalist.

What PC configurations are supported?

You can use a single PC for receiving and processing data if the PC is powerful enough, or a two-PC network is supported, with the lower-spec PC as the Reception PC with the DVB card, and the higher-specification PC as the Processing PC, grabbing files off the Receiver PC over the network.  You can ask about any particular configuration on the MSG-1 self-help Group, where hundreds users with working systems can help, and there is a discussion about PC requirements here.  Windows-7 or 8 is required for support - older versions are not supported.  I would expect XP/32 and Vista/32 still to work but I cannot support these versions as I no longer have PCs running XP or Vista.  

Data Volumes

EUMETSAT state that the expected data volume is around 7000 files per hour (I think that's too high) totalling 540 MBytes.  If you take EARS data and RSS data that increases by 44 MB.  Therefore a full day's data would be about 15GB.  If you keep all 12 MSG channels, but save the images as JPEG not PNG, the saved images amount to 1.1GB per day in 1152 files.  Of course, if you take all the FSD (Foreign Satellite Data) as well the volumes can increase massively, up to a total of some 500 GB/day.

What functions does the MSG Data Manager have?

• Can work on the same PC as data reception or on a different networked PC
• Automatic monitoring of the Reception PC for the presence of new files.
• Automatic copying of those files from Reception PC to Processing PC
• Optional deleting of raw data files on the Reception PC
• Selection of which channels to process, reducing the subsequent processing and storage requirements
• Processing includes both HRIT and LRIT Meteosat SEVIRI data
• Processing includes both Meteosat-5 and Meteosat-7 data
• Processing includes MPEF, MDD, GTS and DCP data
• Decompressing of the data files coded in JPEG and Wavelet Transform formats
• Data reduction from 10- and 12-bit source data to 8-bit display data
• Fully calibrated 10-bit data saving (PRO version only)
• Real-time thumbnail display of selected channels as data is received
• Optional gamma-correction, gain and offset processing for visible channels
• Temperature calibration of infra-red channels
• Saving of processed data in standard image formats
• Choice of PNG for full precision or JPEG for disk economy
• Management of HRV region selection (offset and northern-area)
• Local display of multiple detail images at full resolution 
• Histogram equalisation and false-colour palettes for detail images
• Histogram display of 10-bit received data (PRO version only)
• Management of received data - older data can be automatically deleted
• Display of Admin messages sent over the data stream, also saved as text files
• Copying of Sea-Ice data for further processing
• Copying of [Data Channel 1] ATOVS and RSS data for further processing
• Optional program priority drop when CPU-intensive operations are underway
• Optional sound announcements at start and end of scan
• Optional alert on data loss - you can send yourself an e-mail!
• Graphical display of scan progress

For an extra charge, you can also get the MSG Data Manager PRO version.  This is aimed primarily at those doing research, where it is important to be able to access the full 10-bit data, rather than have the data converted to 8-bit images.  If you are writing your own software for processing MSG data, the PRO version is probably what you need.  Extra functions in the PRO version include:

• The ability to save raw 10-bit data in PGM files from Meteosat-8/9/10/11, and full-precision data other satellites
• Calibration data is saved in the PGM files as well, allowing the recovery of radiance values
• Eight "Detail Image" displays (versus four in the standard version)
• A right-click, Histogram Display function for the thumbnail images
• A right-click, Histogram Display function for the detail images
• The ability to choose the save format (default, JPEG or PNG) on a per-channel basis

If you wish to process the images further, for example to make false-colour combinations, remap to standard map projections, or to animate the images, you could consider my GeoSatSignal program.  For purpose-built real-time animations, without colour or rectification to standard mapping, my MSG Animator program is the tool of choice.

What data does the MSG Data Manager process and display?

MSG Data Manager Operation

The program first copies data received by the DVB system from the Receiver PC to the Processing PC, and optionally deletes it from the receiver PC. You can select which channels are to be deleted, and which copied for processing. As the satellite scan proceeds, you will see the images build up from south to north, and once an image is complete it will be saved to disk. Visual channel images are JPEG compressed, and thermal channel images PNG compressed (lossless).

You can use the Windows Control Panel to select sounds to be played at the start and end of a scan (specifically when the Prologue and Epilogue files are processed).

The program is intended to be left running continuously, copying and processing the received data.

MSG Data Manager Screen Shots

Here are some screen grabs from a recent version of the MSG Data Manager.  

The first thing you need to do on getting the program is to tell it where to look for the files from EUMETCast, where on the processing PC the raw data is to be stored, and where the processed images should live.  Note that both of these latter directories will use an addition year/month/day naming convention for the folder names so that the number of files per directory is kept at a reasonable level.
You can then set a number of other options, in particular which channels are to be processed and what should happen to the raw data.

Once setup is complete, you press the Run button and switch to the MSG HRIT tab, where data from all the selected channels builds up.  During the operation of the program, you will see the scan build from South to North.  Any gaps at the top of the images are there because that part of the earth has not yet been scanned.  Watching the pictures build like this is fun!  The channel spectral region will pop-up as your mouse moves over the thumbnail.  Note that the last image, that of channel 12, is not square like the others, but has a 2:1 aspect ratio.  It is over 11000 pixels high and over 5000 pixels wide!  The imager on the satellite can be adjusted to cover different parts of the Earth on the north and south parts of the scan.  Here you see it almost centred on Europe in the northern part of the scan, but centred on Africa for the majority of its scan.  The program centres all the data to make the composite picture.  The scan position is altered from time to time, as shown in this 2MB animation from EUMETSAT.
Click on the image below to see an animation of the MSG Data Manager at work!

Click for an animation (from an earlier release)

  
The FSD (Foreign Satellite Data) tab shows the LRIT data relayed from satellites around the globe.  Data from GOES-15 and GOES-16/17 is now handled by new programs:  GOES-15 netCDF Manager and GOES-16/17 ABI Manager.

(screen shot from beta program version)

 
The new Himawari tab shows all 16 channels from the Himawari-8 satellite over Japan and Australasia.

From the Detailed Image tabs

On 2004 May 27 at 08:15 UTC I selected channel 12 by clicking on the image, and then moved the view to the are around Sicily and mount Etna.  The snow-covered peak of Etna and the darker volcanic soil around Etna are both easily visible.  The islands of Gozo and Malta are visible to the south of Sicily.  All these images are from EUMETSAT data.

More recent versions of the MSG Data Manager allow multiple Detail Image pages, and allow the detail images to be coloured with your choice of visible or thermal palettes (Colour Lookup Tables or CLUTs).  On the two images below, note how the low fog-like cloud off Norway and the southern coast of the UK stands out well in the visible channel, but not in the thermal channel with the particular LUTs in use.  (I show the PRO version of the MSG Data Manager below.  This has eight detailed displays versus four with the Standard version).

Channel 9 Meteosat-8 data with ZW colour enhancement

Channel 2 Meteosat-8 data with pseudo WEFAX colour enhancement

On May 18th, 2004, much the UK and Ireland was clear of cloud.  Here is an HRV (channel 12) image from Meteosat-8, rectified to orthographic projection in GeoSatSignal.  False-colour is achieved by using data from the 10.8µm thermal channel.

Rapid Scanning Service

In May 2008 EUMETSAT started a Rapid Scanning Service which uses the MSG-1 (now MSG-2) satellite now located at 9.5° East to provide a new image of North Africa and Europe every five minutes.  This provides considerable improvements for "nowcasting" and of course can produce superbly smooth animations!  To achieve the fast scan, only the northern-most part of the globe is scanned, resulting in just three image segments being transmitted (and nine for channel 12 HRV).  To distinguish the rapid scan mode of operation, and to provide a slightly more detailed view of Europe, the MSG Data Manager will provide a stretched display of the thumbnails when operating in rapid Scan mode.  You can read about running two copies of the MSG Data Manager, one for normal scan and one for rapid scan, in the FAQ.  The screen display will appear as shown below.  Do you like this mode?  Of course, the Detail Images and the saved images retain their correct shape and can be processed as normal - it is only the thumbnail display which has this distinctive appearance to show that RSS is being received.

The special appearance of thumbnails in HRIT/Rapid-Scan mode

How do I get the MSG Data Manager?

• You can purchase a licence for the MSG Data Manager online.
• You can try the program before purchase by downloading the program below, as it has a built-in 30-day trial licence.  At the end of your trial period, you will need to either stop using the software or purchase a licence.
• Please note that Windows 98, 98SE and ME, NT 4, and 2000 are not supported, and whist the program may function, some error messages may occur because these operating systems have trouble supporting the very large image sizes.
• There is a bundled offer of the MSG Data Manager, the MSG Animator and my GeoSatSignal software in the MSG Toolset Plus package, which is a very economical way to purchase.

This software has been supplied to the Finnish Meteorological Institute, Meteo-Swiss, Swiss, Italian and Spanish TV, Weather Services, Airports, Universities, Museums, and professional and amateur users throughout Europe.

Downloads

• Download the MSG Data Manager V3.1.22  (3,090,584 bytes;  2022-Nov-01)
• MD5 signature: EC87D6BF8CA7A3C4DA26F5082CD7C24D

Beta version: This program is under continual development - check here for the latest update.

Important: The MSG Data Manager includes a 30-day trial licence.  Please wait until you have data to test before installing the program, as the 30-day countdown starts from the first time the program is run!

This program requires a display size of at least 800 x 700 pixels.

• You can download any beta updates here.
• Any problems, please check out the FAQs.
• Ask questions about the MSG Data Manager in the software self-help group.
• Contact me about the PRO version with full 10-bit calibrated data saving.
• Guidelines on publishing EUMETSAT data.
• Ask questions about Meteosat 8/9/10/11 (MSG-1/2/3/4) in the MSG-1 discussion group.
• Want to process the images further?  Check out GeoSatSignal!
• Want to animate the images in real-time?  Try the MSG Animator program.
• Be sure to calibrate your monitor correctly for best displayed results.
• Want some sample sounds for event announcements?  Download MSG Sounds.
• Unsupported program for plotting the delay of CRM data - Download CRM Delay.
• How can I manage unwanted data?  Try my TrimTree program.
• Automated reporting tool (unsupported)  Download Missing Segments Reporter.

If you are interested in MSG data, did you know that with the same EUMETCast reception hardware you can now get high resolution AVHRR data from the polar orbiting satellites such as NOAA-19 and Metop-A and Metop-B  I have software for these EUMETSAT options in my AVHRR Manager, DWDSAT HRPT Viewer, and my Metop Manager.

MSG Animator

The MSG Animator provides real-time animation of images from Meteosat Second Generation and other weather satellites received with the MSG Data Manager.  It is dedicated to EUMETCast data.  If you want false-colour animations with geographic re-projection and weather chart overlay, please look at my GeoSatSignal program. 

Download the MSG Animator

Program highlights

• Multiple simultaneous, independent animations, updated in real time as new data becomes available
• Selection of channel and graphical selection of region subset, with country boundary overlay
• Animation of Meteosat-7, 8, 9 & 10, MTSAT-1R, & GOES-E & GOES-W data providing world-wide coverage
• Looping and smoothing of animations, with optional histogram and contrast stretch image-processing functions
• False-colour option with land/sea separation and dark-cloud tops
• Control over animation length with option to choose visible hours only
• Save an animation in standard Windows AVI format
• Kiosk-mode operation for interactive public displays and museums
• Automated animation switching for reception areas etc.
• Can work remotely from the MSG Data Manager over a network (requires the runtime Library Bundle to be installed on the PC where the MSG Animator is to operate).
• Windows-7 or Windows-8/8.1 is required.  The earlier Windows XP/32 and Vista/32 versions are not supported.

Sample animations

Here are some sample still screen-shots showing the MSG Animator in action.  You can use thermal channel, water-vapour channel or visible channel data as the source of the animation and, by using colour look-up tables, this date can be made easier to understand.

This data is from Meteosat-8, channel 9, which is the main thermal channel.  The data has been converted to temperatures, and used with a land/sea palette where the higher temperatures are shown as green (land) or blue (sea), but the lower temperatures are shown as increasingly dark cloud colours.  The darkest areas show the most likely areas for rain.

This data is from the Rapid Scanning service and this time the palette used shows the colder temperatures in yellow and red.  This satellite scans every ten minutes, enabling a very smooth animation to be produced, but it does not scan as far north, hence the coloured band at the top of the image.

This data is from the Meteosat-8 HRV (high resolution visible) channel and is also used in conjunction with a land/sea palette just working on the brightness values in the data.  The city data is held in a text list, so you can easily add your own towns.

Getting the MSG Animator

• You can purchase a licence for the MSG Animator online.
• The MSG Animator comes with a built-in 30-day trial licence.  
• Please get the MSG Data Manager installed and working before you try and install the MSG Animator.
• Guidelines on publishing EUMETSAT data.
• You can download the software itself from the link below.
• There is a bundled offer of the MSG Data Manager, the MSG Animator and my GeoSatSignal software in the MSG Toolset Plus package.

Downloads

• Download the MSG Animator V3.1.8  (5,487,170 bytes;  2022-Jun-08)
• MD5 signature: F2009103ABA017D3E878882D5B1FCEEA

Beta version: This program is under continual development - check here for the latest update.

Visit the SatSignal self-help group to discuss this software with other users

Specialised software

In addition to the software listed on this page, I have a number of other software packages for processing Meteosat-8/9/10/11 data including MSG to GeoTIFF for converting the data to radiance, brightness-temperature and reflectance and saving the results as GeoTIFF files, and the MSG Fire Tracker for detecting fire hot-spots using a multi-channel algorithm and reporting the results up to an FTP server.  Please contact me if you are interested in this software.  I can also supply readers for the EARS ATOVS data, the Sea-Ice and Sea-Surface Temperature (SST) data, and the GRIB data from EUMETCast MPEF and DWDSAT data broadcast over the DWDSAT service.

Fire detection with Meteosat-8/9/10

For example, here is a sample screenshot from the MSG Fire Tracker program.  Using Meteosat-8 data from 07 August 2005, fires have been detected in North Africa, Spain, Portugal and the south of France.  The detected fire pixels are highlighted by yellow rectangles in the image below.

Sample screenshot from the MSG Fire Tracker program

 
Here's another screenshot from 2009 Jul 23 showing fires in Spain.  My thanks to Harry at http://www.meteomalaga.es.  You can see Harry's current data here.

MSG timestamp program

Some data sources such the EUMETSAT Archive (MARF) provide MSG HRIT segment files which can be processed with the MSG Data Manager, but the timestamp on these files is either incorrect or becomes lost when transferring between the source and your PC.  The MSG timestamp program is a very simple, free utility which will timestamp MSG HRIT according to their file names, so that they will process in their natural order.  This is useful to allow the MSG Data Manager to process these files, and may help other manufacturer's software as well.  This is a free utility.

• Download MSG timestamp

Sample images from Meteosat-8 (MSG-1)

Guidelines on publishing EUMETSAT data

UK Snow - January 2004

This image was from January 29th 2004 after extensive snow-falls in the UK.  The high-resolution visible channel (HRV) has been combined in GeoSatSignal with thermal information from the 10.8µm thermal channel to make a false-colour image, which has then been rectified to a polar-stereographic projection.

Image copyright EUMETSAT 2004.

Tropical cyclone Manou - May 2003

Here is an image taken during the trials of EUMETSAT's MSG-1.  It show tropical cyclone Manou off Madagascar.  It was processed by the MSG Data Manager and Paint Shop Pro.  Image taken 2003 May 08 at 0744 UTC.  Click for a full resolution version (130KB).  Note that the image is near the edge of the field of view, and there appears slightly squashed.  The HRV channel provides images in the visible waveband, 11136 pixels by 5568 pixels.

Image copyright EUMETSAT 2003.

Click the image for a full-resolution version

Other data from your EUMETCast service

In addition to the Meteosat-8/9 and other geostationary weather satellite data carried on EUMETCast, you will also have access to Sea-Ice concentration, sea-surface temperature data, and ATOVS atmospheric sounding data.  I can provide programs which decode and display this data if you are interested in these aspects of the weather.  Here are some sample images to whet your appetite....

Sea-surface temperature sample data

Sea-surface temperature sample data

Sea-Ice sample data

Sea-Ice sample data

ATOVS sample data

ATOVS sample data

 

Copyright © David Taylor, Edinburgh

@gm8arv on Twitter

Last modified: 2022 Nov 01 at 11:28