Point Processing Menu
On this page, you will find general help for the ProRaster product family, including links to documentation, instructional videos, and training videos.
Previous Topic: Publish to MapInfo
Next Topic: LAS Import Operation
Back to: ProRaster Scientific Help
Back to: ProRaster Help
Back to: ProRaster
Go to: ProRaster Essential help page, ProRaster Premium help page, ProRaster Scientific help page.
The ProRaster User Guide is available for download as a PDF.
ProRaster Scientific, from version 4.0 onwards, ships with a new major capability for working with huge 2D point cloud datasets. This capability supports general multi-banded 2D point data as well as 2D LiDAR pulse-return data. Access these capabilities from the Point Processing menu button.
Point data is supported in ProRaster with these new capabilities –
- A new file format called “MRP” for storing multi-banded 2D point data and LiDAR 2D pulse-return data.
- An import operation for multi-banded 2D point data that supports creating new MRP files and adding new data to existing MRP files.
- An import operation for 2D LiDAR pulse-return data that supports creating new MRP files and adding new data to existing MRP files.
- Support for rendering points (stored in one or more MRP files) in the main rendering interface. This capability is shipped with all tiers of ProRaster.
- A Virtual Gridding Operation that will grid point data stored in one or more MRP files in real-time and allows users to interactively modify gridding parameters on-the-fly.
- A new global resource editor, the LAS Query Editor, that enables users to create and edit queries that acquire returns from LiDAR pulse-return data. These queries can be employed when rendering and gridding LiDAR pulse return data stored in an MRP file.
MRP is a new point cloud file format that provides efficient storage of 2D point cloud data.
- MRP supports datasets of unlimited size and addresses the problem of huge point cloud management by minimising storage requirements through sparse tiled storage, rich type selection, and efficient lossless data compression.
- MRP supports high performance access – and constant time access – to point cloud data at any scale by generating a data pyramid of overview resolution levels that are stored within the MRP file.
- MRP supports the temporal dimension.
- An MRP file is editable and extensible. Additional compatible point cloud data can be added to an existing time event, or as a new time event.
- An MRP file spatially sorts and indexes data in 2D (using the designated X and Y coordinate bands).
- An MRP file contains comprehensive summary statistics for all data bands.
MRP files come in two flavours – general Multi-banded point cloud files, and LiDAR pulse-return files.
- A general multi-banded point cloud MRP (PNT_MRP) can contain any point data, with the expectation that bands for X and Y coordinates will be supplied as a minimum requirement.
- A LiDAR pulse-return point cloud MRP (LAS_MRP) contains LiDAR data imported from LAS or LAZ files.
The MRP format is derived from the MRR (raster) format and ProRaster interacts with MRP files through the standard raster API interface. This allows the raster engine to control tile caching and memory management and allows you to view trillion-point datasets at any scale in real-time on modest hardware. It also means that you can create a Point Raster Source, via the Raster Source Editor, that contains multiple MRP files. These files ought to be strictly of the same flavour and with identical field-band-event structures. Unlike rasters in a raster source, they can have different coordinate systems. A raster source containing MRP files can be used as input to a rendering layer or in the virtual gridding operation.
Point Cloud Processing Pipeline
ProRaster takes a different approach to point data management, gridding, and processing to other software. The end-goal is to grid huge point datasets in real-time and enable visualisation and analysis of gridded point data without enormous pre-processing operations. To achieve this, you must first import your point data into an MRP file. (Note that this creates a new “hardcopy” duplicate of your point data). Once in MRP format you will be able to render, grid, and process point datasets in real-time at any scale.
The ProRaster point cloud data processing pipeline looks like this –
- Import raw point data into an MRP file.
- Import multi-banded point data to a PNT_MRP or –
- Import LiDAR (LAS/LAZ files) pulse-return data to a LAS_ MRP.
- Add to and extend an existing MRP spatially (or temporally) by importing new data of the same type.
- Render the MRP file in ProRaster to inspect and validate point data.
- Undertake visual and statistical quality control.
- Color modulate points by using either a LUT Color style data-color transform or an RGB Color style data -color transform. Color tables are included for rendering LAS_MRP data by classification code.
- Apply a LAS Query to LAS_MRP data on-the-fly to filter and select returns.
- Use tooltip reporting in the map to reflect the value of any point as you pass over it with the mouse cursor and display the value of all bands using the “Key + Table” reporting option.
- Employ Virtual Gridding to transform the point data into a continuous raster in real-time at any scale.
- Grid point datasets of unlimited size down to very high resolution.
- Modify your gridding method and gridding parameters interactively and see this reflected in the gridded raster in real-time.
- Apply a LAS Query (which you can change at any time) to LAS_MRP data to select returns for gridding.
- Render the virtual grid (in 2D, 3D, as imagery or contours) to explore your data and undertake QC inspection.
- Use the virtual grid as input to any other virtual processing operation or chain of operations (for example Clip to a polygon and apply a Calculator operation).
MRP Flavours and Format
MRP files come in two flavours – general Multi-banded point cloud files (PNT_MRP), and LiDAR pulse-return files (LAS_MRP).
- A general multi-banded point cloud MRP (PNT_MRP) can contain any point data, with the expectation that bands for X and Y coordinates must be supplied as a minimum requirement.
- A PNT_MRP can contain any number of bands. Each band has an individual data type and there is a rich selection of data types to choose from. The validity of each value, in each band, is stored in a separate binary mask and established at import time.
- A LiDAR pulse-return point cloud MRP (LAS_MRP) contains LiDAR data imported from LAS or LAZ files.
- The band structure of a LAS_MRP is fixed and is based on LAS specification 1.4 from the Open Geospatial Consortium. All the data fields, and the pulse-return structure of the LAS/LAZ files, is preserved in the MRP. This allows LAS queries to be applied (post-import) to select certain returns for visualisation or processing based on diverse criteria.
PNT_MRP data is in the form X,Y,[V,…]. Multi-banded point data contains bands for the X and Y coordinates (in the units of the coordinate system of the dataset) and then an unlimited number of additional bands that can contain any other data measured, observed, or computed at that point. For example, it could contain an RL or a measurement of the magnetic field strength, or any other kind of data. You are encouraged to supply a Z coordinate for RL and can specify the units of this data.
Each band can have a unique data storage type (decimal, integer, color, etc) which you can specify. The MRP file also records whether each stored value is “valid”. When importing the points, the X and Y coordinates are checked for validity. If either coordinate is invalid, the point is not imported. Validity checks for all other bands are also conducted during import and each point can have a mixture of valid and invalid band values. When rendering or processing data, invalid values will be ignored.
When the overview resolution levels are created for a PNT_MRP, a subset of the points will be duplicated in each level. Points are selected for overview levels based on their spatial properties alone and not on any properties of the data bands.
A LAS_MRP contains LiDAR Pulse-Return data stored in a fixed format that replicates LAS specification 1.4. To create a LAS_MRP you will import data exclusively from LiDAR point data files in LAS or LAZ format. The data is stored in the MRP using lossless compression, so an MRP file can be thought of as a large LAZ file that spatially sorts and indexes the pulses, implements an overview pyramid to support multi-scale data access, and contains high-quality statistics for all data bands.
LAS_MRP does not support point or band data value validity. All points (which correspond to returns) and all band data values in every return are considered valid. LAS contains a “Withheld” flag that can be used in LAS Query to establish return validity.
Although points in an MRP are spatially sorted into tiles, the LAS_MRP will never split the returns in a pulse across multiple tiles. In other words, all the returns from a single pulse will be found in the tile that contains the first return, even if subsequent returns in the pulse lie outside the tile. The same rule is applied when building overview levels. We select pulses for overview levels, which includes all returns in the pulse. Also, we may reject pulses for inclusion in overview levels based on the return classification. If all returns in a pulse are classified as “Unclassified(0)”, “Unassigned(1)”, or “Overlap(12)” then the pulse will only be considered for inclusion in an overview if there is an insufficient number of other pulses with more useful classifications. This prevents these pulses dominating overview levels and improves the quality of gridding at higher scales.
The two different types of point data are treated differently at all stages of processing. You should not (and generally cannot) mix point data flavours in a single MRP file. If you combine two or more MRP files into a point raster source, they should contain point data of the same flavour.
Just like raster data in an MRR file, point data is stored in an MRP in a sparse matrix of square tiles. Each tile is 256×256 cells, and the spatial extent of the tile is defined by its position in the tile matrix and by the cell size. The extent of the tile matrix is variable, does not need to be defined in advance, and can be extended at any time simply by adding more data. The system defines a continuous stack of “resolution levels”, each of which contain some subset of the point data.
The “base resolution” level (level 0) is the level into which all the point data is injected. It contains a complete copy of all the points imported into the MRP.
The “overview” levels (level 1 upward) are accessed when the system requires data at a lower resolution. The first overview is half the resolution of the base level, and each subsequent overview level is half the resolution of the previous level. This forms a “pyramid” that eventually comes to a peak by which time there will be no more than four points in the level. Each overview level contains a duplicate subset of some of the points in the level below it. Overview levels are stored in the MRP file.
The “underview” levels (level -1 downward) are accessed when the system requires data at a higher resolution. Each underview level is twice the resolution of the level above it. Underview tiles are generated on the fly by simply copying the points from the base resolution level. They are temporary and never stored. Unlike raster underview levels, point underview tiles simply contain a copy of points from the base level and there is no interpolation performed.
MRP supports the time dimension. It does this by defining “events” which occur at a specified time and associating point data with these events. When you import point data into an MRP you have the option of importing it into a new event. In ProRaster you can skip between events when rendering point data or when rendering any virtual raster generated from the MRP.
Hardware Resources
As always in ProRaster, the size of the point datasets you can work with is “unlimited” and the performance of the software is constant, regardless of the size of the datasets. In practice, the size of the point datasets you can work with will be limited by your local storage capacity. Taking LiDAR data as an example, as a rule you can expect that every 1 million points will require about 10 megabytes of storage. It follows your storage requirement for large LiDAR datasets will approximately follow the guidelines below. Make sure your local storage capacity meets your expected storage requirement.
10 million 100 megabytes
100 million 1 gigabyte
1 billion 10 gigabytes
10 billion 100 gigabytes
100 billion 1 terabyte
1 trillion 10 terabytes
Virtual gridding introduces a new strict hardware minimum requirement. ProRaster now requires a CPU that supports AVX2 instructions. All Intel and AMD x64 processors have included AVX2 since 2013. If your Intel or AMD CPU does not have support for AVX2, ProRaster will not execute. (At time of writing, Windows on ARM does not support emulation of AVX2 commands on ARM64 processors and ProRaster Scientific will not execute on an ARM CPU based PC).
Point importing can process an unlimited number of input points and will potentially generate very large output files. It will put considerable demands on your CPU, SSD/HDD, and RAM. You will benefit from having more cores, larger SSD’s and more RAM.
Virtual Gridding can tax your system to the maximum limit, stretching the capabilities of your CPU, SSD/HDD, and RAM. If you plan to regularly use virtual gridding, you should aim to acquire the best hardware you can afford.
Cell Size
When you create an MRP there are two critical parameters that you need to define – the coordinate system and the cell size.
You must specify a coordinate system so that ProRaster can reproject point data on-the-fly to display it in the context of other raster data, or vice versa. Note that all LiDAR data must be in the target coordinate system when imported into an MRP. In contrast, multi-banded point data can be reprojected during import from the user specified source coordinate system to the target coordinate system of the MRP.
You also must define a cell size, which might make little sense at first glance. But MRP is based on MRR operational code. It is a point cloud masquerading as a raster! Points in an MRP are sorted spatially and assigned to the tile in which they are located. The size of a tile is defined by the number of cells in X and Y extent, and the width and height of a cell. The number of cells is fixed at 256 x 256 cells, so you only need to define the width and height of a cell to define the size of a tile.
The cell size is important, because you do not want too many – or too few – points stored in a tile. You are aiming for a Goldilocks scenario. The goal is to define a cell size that is about one half the average separation between points. In an ideal world, if you did that, then you would expect about 16000 points in each tile.
If you make the cell size too small, then:
- the number of points in a tile will be small
- the MRP file will be inefficient and larger than it might otherwise be
- the system will have to load more tiles than optimal, and this is inefficient
If you make the cell size too large, then:
- the number of points in a tile will be large
- the MRP file may be more efficient and smaller
- the system will load more points than it needs to, and this is inefficient
It is up to you to determine the point data density, the average point separation, and to define a reasonable cell size. Doing so may require some experimentation.
After you have created an MRP you can check statistics to see if your cell size was reasonable. Display the MRP and then open the Raster Source Editor. Select the MRP in the lower left Raster File list then select Info level “Detailed + Statistics”. Look at the statistics for the X band. For a PNT_MRP you will see the following information reported (for example):
Cell count: 103677952
Point count: 15239967
Unoccupied cell count: 98309659
Occupied cell count: 5368293
Global point density: 0.146993326026
Cell point density: 2.83888509811
In a LAS_MRP you will see this additional information:
Pulse count: 2783339
Global pulse density: 0.524325712228
Cell pulse density: 1.20099010285
The “Cell count” is the total number of cells in all extant tiles. Divide by (256×256) and you will know how many tiles are in the MRP. The “Point Count” is the total number of points in the MRP. In a LAS_MRP this corresponds to the total number of LiDAR returns. A LAS_MRP also reports the number of LiDAR pulses as “Pulse count”.
The “Occupied cell count” is the total number of cells that contain one or more points (or returns). The “Unoccupied cell count” is the total number of cells that contain no points (or returns). Added together, they ought to match the “Cell count”.
The “Global point density” is the average number of points per-cell. It is computed as the Point count divided by the Cell count. The “Cell point density” is the density of points in occupied cells and will always be higher. In a LAS_MRP similar numbers are reported for the pulse density.
LiDAR pulses tend to be evenly distributed, but each pulse often contains multiple returns. Those returns are likely to be very close to each other – often falling into the same cell. This results in a higher cell point density compared to the pulse point density.
If you have one point or pulse every two cells, then the point or pulse density will be 0.125. As some percentage of the tiles will lie outside the survey boundaries, you might aim for a little higher than that.
You may also consider your gridding requirements when you define the cell size – in particular Tensioned Minimum Curvature gridding. This method requires a cell size that is no smaller than about 20-25% of the separation between points. When working with survey data, this generally means 20-25% of the separation between survey lines. As the grid cell size is a power of two multiple of the MRP cell size, you should consider your gridding cell size requirement prior to importing the point data.
Recommendations
For LiDAR data your cell size is likely to be in the range 0.2 metres to 1.0 metres. Older surveys have lower point density than modern surveys.
For point data that is evenly distributed choose a cell size about half the average point to point spacing.
Point data collected on survey lines is generally dense along line, but the lines are widely separated. For example, a modern aeromagnetic survey may have points every 4 metres along line on lines separated by 50 metres, and tie lines every 500 metres. You can use the following formulas –
StationDensity (Stations per square metre) = (1/LineSpacing + 1/TieSpacing)*(1/StationSpacing)
Cell Size (width and height) = SquareRoot(1/(4*StationDensity))
In my example, LineSpacing=50, TieSpacing=500, StationSpacing=4.
Therefore StationDensity=0.0055 and CellSize=6.74m.
I would want to grid the data at 10m spacing, so I would round the cell size up to 10m (or down to 5m).
In the “Import Multi-banded Points” operation a Calculator dialog is provided that implements the formula described above for line-based survey data. Hit the “Cell Size Calculator” button to open the calculator dialog.
Enter the line spacing, the tie line spacing (if present, otherwise leave the field blank), and the along-line station spacing. The dialog reports the “ideal” cell size for the MRP and then provides two other reasonable cell sizes that will work well with tensioned minimum curvature gridding. The first works best if you want to grid at one quarter the line spacing, the second works best when you grid at one fifth the line spacing.
Some point data has spatially varying point density – sometimes by orders of magnitude. For example, gravity measurements may be gathered from very low-density regional surveys, line based aerial surveys, and very dense ground surveys (also often along lines). These merged datasets are very hard to grid satisfactorily. You may have to use your best judgment.