Ar2gas is a complete platform for spatial modeling and processing using python. The core functionalities contain the data structure and algorithms. This is leveraged into an interface around the workspace to provide software-like capabilities from a python framework. The workspace is a project data structure that is built to be used both on the desktop and the cloud.
Ar2gas goes beyond simple virtualization, it provides a separation between data, computation and analytics. The most important algorithms have been ported for use on a distributed computing platform, whereas a primary machine can use on-demand workers to speed up computation. There is no need to keep an expensive large machine at all times.
The workspace provides a data structure that stores all the data and meta-data that are relevant to a geo-modelling project. Information is organized in an hierarchical fashion from grid to properties and can be accessed by tab completion in a notebook environment. In addition to storing data, the workspace saves variogram models, algorithm parameters, data series, files, and reports. The workspace has been designed to stream data. Opening a project does not load everything in memory. One can work with very large data rapidly without memory issues.
The workspace is the key piece that facilitates data streaming via a network allowing high performance computing on a distributed computing environment.
An AR2gas process is a self contained snippet of python code that executes a specific calculation or algorithm. Processes can be run on separate threads and managed by the AR2gas server for fast and efficient computation. The communication of parameters, input data and results is done seamlessly with the workspace. Process parameters are strongly-typed, which allows for validation prior to execution. A crash or interruption of an algorithm will be isolated to that thread and will not be propagated to the notebook.
Common geostatistical algorithms already exist as processes and additional user defined algorithms can be added to the process library.
For analytics, a series of dashboards allow the users to focus on interpretation and decision making. The dashboards provide curated charts, figures and tables specific to a purpose. A goal is to reduce the barrier of entry for all users since all the relevant information is already gathered in one place. It also improves technical communication within a team from modellers to supervisors.
The dashboards are web pages that can be viewed from anywhere (phones, tablets, desktops, conference rooms) with the proper authorization.
The current dashboards can validate simulations, provide basic EDA, study grade connectivity, compare single models with an ensemble of simulations, provide grade-tonnage curves and more.
Memory Management for Big Data¶
Simulations of large models can be memory intensive. The workspace has been designed to minimize the memory requirements by providing a flexible and easy to use map-reduce environment. The map reduce environment can be used either per simulation (e.g. grade tonnage curve computation) or by location (e.g. local statistics or loss function).
Hybrid Environment with JupyterLab¶
Basic workspace operations have been translated to a series of user interfaces. User interfaces are aggregated to create a JupyterLab extension that communicates with the workspace and AR2gas core functionality via REST API.