The Multispectral Scene Database
On this page, you will find general help for the ProRaster product family including links to documentation, instructional videos, and training videos.
Previous Topic: Concepts, Terms, and Outcomes
Next Topic: Browsing the Multispectral Scene Database
Back to: ProRaster Scientific Help
Back to: ProRaster Help
Back to: ProRaster
Go to: ProRaster Essential help page, ProRaster Premium help page, ProRaster Scientific help page.
The ProRaster User Guide is available for download as a PDF.
After you download a satellite multispectral scene from a data provider, your first step will be to add it to a multispectral scene database. This performs three key tasks –
1. The scene is parsed, and metadata is acquired. ProRaster works out what the scene is – what platform, what bird, what processing and product. It works out what spectral bands are available, what QA data is supplied, and what ancillary data is available. If ProRaster recognises and supports the scene, then it can be added to a scene database. This will write a small amount of metadata into the database, including a record of the location of all the raster source files in the scene.
2. An MVR virtual raster file (the scene collection MVR) is prepared for the scene and written into a folder subordinate to the database. This MVR gathers all the data associated with the scene and combines it into a single virtual raster. It applies spectral transformations to convert from Digital Numbers to radiance, reflectance, and temperature. It may make top of atmosphere corrections. It picks out information from the QA data and presents it as individual raster bands. It uses the QA data to prepare a mask field that can be used to identify pixel classifications and pixel quality.
3. The source raster files are finalised. It is a requirement of ProRaster that each source raster in the scene has a full overview data pyramid. If it is not supplied, or it is supplied but is invalid, then it will be generated. This will produce a .PPRC cache file for each scene raster file.
Creating a scene database
You interact with scene databases via the “Browse Scenes” menu button.
Drop down the menu button to reveal the options for creating, editing, managing, and deleting scene databases.
Create a new scene database by selecting “Add Scenes to new database” and selecting one of three methods for adding one or more scenes to the database. Before you browse to the scene(s), you need to create and name your new scene database. Your database must have a unique name. The scene database file will use this name. If the name is unique then you will see a green tick. Hit OK to create the scene database.
You now must browse to the scene (or scenes) that you want to add to the database. There are three ways to do this – by metadata file, by folder, or by folder search.
1. Browse for and select the key metadata file associated with the scene. A single scene will be imported.
For a Landsat scene, the key metadata file is the *_MTL.txt file. These files have been distributed with each Landsat scene from the beginning of the program. These days the data is mirrored in JSON and XML files.
For a Sentinel 2 scene, the key metadata file is an XML file that is named according to the processing level. Look for files named “MTD_MSIL1C.xml “ and “MTD_MSIL2A.xml”.
2. Browse for the root folder that the scene is stored in. ProRaster will search for the metadata and import a single scene.
This is an easier option and pushes the work of finding the metadata file back to ProRaster. Simply browse to the root folder for the scene and ProRaster will search for the information it needs. It relies on you storing each scene in a unique folder. This is a practice that you ought to follow religiously. Do not dump the data from multiple scenes into a single folder!
3. Browse for a folder. ProRaster will search in this folder, and in all its sub-folders, for any scenes recursively. Every scene that it finds will be imported.
This is the easiest option. Using this method, you can add as many scenes as you want to your database. Simply browse to a folder underneath which you are storing all the scenes you want to import. ProRaster will detect and import all the scenes it finds under this folder. To take advantage of this, it makes sense to keep your scene folders organised into a project hierarchy so that you can easily add the scenes you want for a particular project to a scene database.
Once you have specified where to find the scene(s), a processing dialog will open, and you will be able to monitor the progress of ProRaster as it imports the scene(s).
You can create as many scene databases as you want. How you organise your scenes is up to you. You may want to add all scenes to a single large database, or you may want to create a database for each project that you undertake.
- A scene database file has a “<your name>.msssdb” file name. You must specify the name when you create the database. You cannot change the name thereafter.
- Every scene database you create will be located in the “C:\Users\<your user name>\RGE\Satellite” folder on your main drive. You cannot move this file.
- When you create a scene database it creates a new folder in the “C:\Users\<your user name>\RGE\Satellite” folder of the same name. For example, if I create a new scene database called “SDB_KatiThanda.msssdb” then a new folder “C:\Users\<your user name>\RGE\Satellite\ SDB_KatiThanda” will be created.
- The new folder will contain an MVR virtual raster file for every scene in the database. This is the key, and only, raster resource that you will interact with for the scene. You will never refer to the original raster files distributed with the scene. These are all accessed via this key MVR file.
- No raster data from the scene is ever written into the scene database. Consequently, the scene database will be a small file, even if it contains thousands of scenes.
- When you add a scene, the overview pyramid data cache will be prepared (if required). This takes a few moments per scene. The cache files are written into the same directory as the original scene raster files. This is the only modification to the original scene folders that is made. Once these rasters have been generated, they will not be regenerated.
- You can add a scene to two or more scene databases without penalty.
- When you delete a database both the .msssdb file and the folder and all its contents will be deleted.
- When you delete a database the original scene folder and all its contents remain untouched.
From these points, you can conclude that the scene database is a relatively low-value resource. Databases can be created, and recreated, easily. However, once you consume a scene in a multispectral product, that product will refer to the scene database by name. If that database is deleted, then the product will become orphaned.
Scene data duplication
By default, when you import a scene the scene raster data will not be duplicated. The scene assembly MVR will refer directly to the original source rasters in the scene. However, there are options to duplicate the scene raster data which may be of interest in advanced scenarios. These options currently apply to Landsat and Sentinel2 scenes, but not to scenes that are manually imported.
When ProRaster Scientific accesses raster data in a scene, it may take multiple steps to improve, correct, augment, or replace that raster data.
- The raster may be missing critical information. For example, a TIFF file may not record the raster null value. In scenarios like this, ProRaster will use driver preferences to provide critical missing information at run time. This information is stored in the scene assembly MVR.
- The raster may not have an overview cache. ProRaster will create an overview cache and store it alongside the original raster. The cache file will have a PPRC extension.
- The raster overview data may not be valid or entirely correct. For example, if null values are not identified, they may be used to construct the overview and combined with valid data. The result is incorrect data values. Another common mistake is to generate overviews from QA data bit masks, and then to simply average those values. This generates invalid bit mask values in the overviews. ProRaster may choose to correct these errors by generating a new overview cache which will be used in preference to the overview data in the original raster.
- If ProRaster computes statistics for a raster from the base level data, it will store those statistics for future reference. The statistics will be stored alongside the original raster in a file with an STTX extension. An ASCII file with a GHX extension may also be written.
- For efficiency and correctness, ProRaster can replace a scene raster with a new raster in MRR format. This is called a duplicate. Thereafter the scene assembly raster will refer to the MRR duplicate raster and not the original scene raster.
ProRaster Scientific provides control over the creation of duplicate rasters. This is controlled from the MSS Options dialog, which is accessed from the system menu. Duplicate rasters can be stored alongside the original raster in the scene folder, or they can be stored in a duplicate cache folder of your choosing. There are four options.
- Duplication: Minimal intervention
The goal of this mode is to prevent, where possible, writing of any files to the scene folder. ProRaster will use the original scene rasters and the rasters ought to provide overview levels.
If any duplicates already exist for the scene, they will be used. They may be in the scene folder or in the duplicate cache folder. If overviews are not available, PPRC cache files will be written into the scene folder. No new duplicates will be generated.
- Duplication: Recommended
The goal of this mode is to provide a balance between minimising storage requirements and maximising correctness. It is the default mode.
If any duplicates already exist for the scene, they will be used. They may be in the scene folder or in the duplicate cache folder. If overviews are not available, PPRC cache files will be written into the scene folder. If overviews need to be corrected, replacement overview cache files will be generated. No new duplicates will be generated.
- Duplication: Full on demand
The goal of this mode is to provide duplication of the source raster data.
If any duplicates already exist for the scene, they will be used. They may be in the scene folder or in the duplicate cache folder. If any duplicates are missing, they will be generated.
- Duplication: Full regeneration
The goal of this mode is to correct and replace duplicates. It should only be used temporarily, and then the mode can be changed back to “Full on demand”.
All duplicates will be regenerated.
The final option is to specify the folder that the duplicate rasters will be written into. By default, this folder is not specified, and in this case, all duplicates are stored in the scene folder alongside the original scene raster. If you specify a duplicate cache folder, the duplicates from each scene will be written into a unique folder in the folder you specify. The system will create and manage these sub-folders.
Manually Adding a Scene
To manually add a scene to a scene database, choose the “By Manual Entry” option on the Browse Scenes menu button to either add a scene to a new database or add a scene to an existing database. The MSS Scene Entry dialog will be displayed.
You are encouraged to enter appropriate information that describes the platform, bird, processing, and scene and you will need to identify the physical raster files on disk and specify which spectral bands they represent.
The “Platform” identifies the satellite constellation and may encompass multiple birds and families of birds. For example, the “Landsat” platform encompasses all Landsat bird’s past, present, and future. Select a platform from the drop list. If you need to define a new platform, select “New…” and then you can type in the name of the new platform and a description (if you wish). Once you have entered the new platform name and description, make sure you hit the blue “+” button to store the new data. The blue button will turn grey, and your new platform will be displayed in the drop list.
Repeat this exercise for the Bird, Family, Tier, Instrument, Processing, Processing Level, and Product. The Bird identifies the satellite that acquired the scene. Within a platform, birds may be grouped by Family. All birds in a family are expected to use the same instruments. The Tier refers to the quality of the data and it is usually appropriate to select Tier 1 (high quality). The Instrument is a code name for the spectral instrument, or combination of instruments, on the bird. Processing refers to the processing epoch used to process the scene data. Processing Level refers to the level of corrections or processing operations applied to the scene data. For example, the processing level may be raw digital numbers, top-of-atmosphere reflectance, or surface reflectance. The Product refers to the output data product that was generated by the processing.
This data is used by ProRaster Scientific when you try to combine multiple scenes together into a product like a mosaic or a sequence. Scenes need to have matching parameters. It is up to you to enter these parameters appropriately. I recommend you enter Platform, Bird, and Product data as a minimum.
The Path and Row parameters are mashed together to identify the tile. You may not have specific orbital path and row information, but you may have some identifying information for the tile. You can enter this into the Path parameters and leave Row blank if you wish.
You can enter a Cloud percentage coverage (from 0 to 100) for the scene if you wish (and if you know what it is). This will help you order scenes by quality when creating products.
You need to enter the acquisition time for the scene. This is the UTC time (Coordinated Universal Time) at which the scene was acquired by the bird. It is not necessary to be completely accurate, but if you want to create sequence products then the scene must have a valid acquisition date and time. When you hit the time button, the GMT Date – Time Picker dialog will be displayed and you can enter a date and time.
Having defined the scene information, you now need to identify the scene source rasters and build the scene assembly virtual raster by linking source raster bands to spectral bands. Firstly, add all the source rasters to the raster list. You can browse to select one or more rasters, choose an existing raster source, or drag and drop rasters from File Explorer into the list. The order of the rasters in this list is not critical.
The Raster Source Editor gives you the option to link driver preferences with raster files. For example, a commonly used option for TIFF files is to specify an invalid cell value (null value). If you require any driver preferences, you will need to build your raster sources first in the Raster Source Editor dialog, and then select them in the MSS Scene Entry dialog.
Now you can build the scene assembly MVR, one band at a time. As you build the MVR the fields and bands will be displayed in the tree. You can reorder the bands by moving them up or down, or you can delete bands, by selecting them in the tree and hitting the appropriate button above.
Using the three drop lists at the top, select the source raster, source raster field, and source raster band. Now link this band to a spectral field and band using the two drop lists at the bottom. You can choose one of the following fields listed below. The list shows the code identifier, the field name, and the field description.
SDN: Spectral DN “Spectral digital numbers”
STREFDN: Spectral TOA REF DN “Spectral top of atmosphere reflectance digital numbers”
STRAD: Spectral TOA RAD “Spectral top of atmosphere radiance”
STREF: Spectral TOA REF “Spectral top of atmosphere reflectance”
SBREFDN: Spectral BOA REF DN “Spectral bottom of atmosphere reflectance digital numbers”
SBREF: Spectral BOA REF “Spectral bottom of atmosphere reflectance”
TDN: Thermal DN “Thermal digital numbers”
TTTDN: Thermal TOA TEMP DN “Thermal top of atmosphere temperature digital numbers”
TTRAD: Thermal TOA RAD “Thermal top of atmosphere radiance”
TTT: Thermal TOA TEMP “Thermal top of atmosphere temperature”
TBTDN: Thermal BOA TEMP DN “Thermal bottom of atmosphere temperature digital numbers”
TBT: Thermal BOA TEMP “Thermal bottom of atmosphere temperature”
MASK: Mask “Pixel validity mask”
The spectral fields allow you to choose one of the following spectral bands. The list shows the code identifier, the band name, and the band description.
Coastal: Coastal Aerosol “Coastal Aerosol Blue”
Blue: Blue “Visible Blue”
Green: Green “Visible Green”
Red: Red “Visible Red”
Pan: Pan “Panchromatic”
RDEG1: Red Edge High “Vegetation Red Edge High Frequency”
RDEG2: Red Edge Mid “Vegetation Red Edge Mid Frequency”
RDEG3: Red Edge Low “Vegetation Red Edge Low Frequency”
NIR: NIR “Near Infra-red”
NWIR: Narrow NIR “Narrow Near Infra-red”
Water: Water Vapor “Water Vapor”
Cirrus: Cirrus “Cirrus Cloud”
SWIR1: SWIR High “Short wave Infra-red High Frequency”
SWIR2: SWIR Low “Short wave Infra-red Low Frequency”
The thermal fields allow you to choose one of the following thermal bands. The list shows the code identifier, the band name, and the band description.
TIRS1: Thermal High “Thermal Infra-red High Frequency”
TIRSHG: Thermal High Gain “Thermal Infra-red High Frequency High Gain”
TIRS2: Thermal Low “Thermal Infra-red Low Frequency”
The mask field allows you to choose one of the following mask bands. The list shows the code identifier, the band name, and the band description.
MPC: Pixels (Clear) “Unobstructed pixels”
MPLC: Pixels (+ largely clear) “Largely unobstructed pixels”
MPPC: Pixels (+ partly clear) “Partly obstructed pixels”
MLC: Land (Clear) “Unobstructed land pixels”
MWC: Water (Clear) “Unobstructed water pixels”
MSC: Snow (Clear) “Unobstructed snow pixels”
Once you have selected the source raster, field, and band, and the destination field and band, the “+” button will turn blue. Hit the button to add the band to your scene assembly MVR, and it will be displayed in the tree. As you add new bands, they will automatically be sorted into fields.
Once you have defined all the parameters, check it over thoroughly. Make sure everything is correct. Then hit the OK button to add the scene to the database. The scene assembly MVR will be written, and the scene database browser dialog will open. If you need to import multiple scenes, the new parameters that you defined will be saved so that you will not need to enter them again (whilst ProRaster Scientific is running).
Editing a scene database
Drop down the “Browse Scenes” menu button to see options for editing a scene database.
You can add a new scene, or many new scenes, to a database. As before when creating a new database, this can be by browsing for the scene metadata file, browsing for the scene folder, or browsing for a root folder and adding all scenes that ProRaster can discover under that folder.
You can easily delete duplicate scenes from a database. If you use the folder search option, it is easy to add a scene to a database more than once. There is no harm in this, and the problem is easily rectified by choosing the “Delete > Duplicate Scenes” option.
You can delete a database by selecting the “Delete > Database” option. The database selector dialog will open and you can choose the database to target. The database, and the folder containing the key MVR files will be sent to the recycle bin.
The “Validate” option gives you some options to validate the scenes in your database. “Validate” will make sure that all the files it expects in every scene are present and reachable. “Regenerate” will regenerate the key MVR file for each scene in the database.