Induction to monitoring

View the Project on GitHub infra-helpers/induction-monitoring

Induction to Monitoring - Python

Table of Content (ToC)

Table of contents generated with markdown-toc


This README is part of the Induction Monitoring project, itself aiming at providing end-to-end examples introducing how to collect, store and query metering events, produced by different sensors on local as well as on clouds.

The python/ sub-folder provides a base Python library, namely datamonitor, offering a few methods to collect Key Performance Indicators (KPI) from data files. Those KPIs are then typically wrapped into JSON structures and sent to an Elasticsearch (ES) service. The KPIs then form corresponding time-series, which can be further analyzed with Kibana, or any other analytical tool, typically for detection of outliers and alert notifications.


Python module

Python virtual environments

Elasticsearch (ES) Python modules

Python Elasticsearch client

Elasticsearch DSL



Clone this Git repository

$ mkdir -p ~/dev/infra && \
  git clone ~/dev/infra/induction-monitoring
$ cd ~/dev/infra/induction-monitoring/python

Python environment


export PYENV_ROOT=”${HOME}/.pyenv” export PATH=”${PYENV_ROOT}/bin:${PATH}” if command -v pyenv 1>/dev/null 2>&1 then eval “$(pyenv init -)” fi if command -v pipenv 1>/dev/null 2>&1 then eval “$(pipenv –completion)” fi

_EOF $ . ~/.bashrc $ pyenv install 3.8.3 && pyenv local 3.8.3 $ python3 -m pip install -U pip pipenv

  + Python 3.8:
$ pyenv install 3.8.3


Install the datamonitor module

Install in the Python user space

Installation in a dedicated Python virtual environment

Remove any previous document from the ES index

Use datamonitor from the Command-Line (CLI)

Use datamonitor as a module from another Python program

<Elasticsearch([{‘host’: ‘localhost’}])>

* Get the information/details about the ES cluster/service:
>>> es_info = dm.es_info()
>>> es_info
{'name': 'MyFavoriteES', 'cluster_name': 'elasticsearch_gold', 'cluster_uuid': 'G6YM0RZsRtW3DjoMPsIx_A', 'version': {'number': '7.8.0', 'build_flavor': 'default', 'build_type': 'tar', 'build_hash': '757314695644ea9a1dc2fecd26d1a43856725e65', 'build_date': '2020-06-14T19:35:50.234439Z', 'build_snapshot': False, 'lucene_version': '8.5.1', 'minimum_wire_compatibility_version': '6.8.0', 'minimum_index_compatibility_version': '6.0.0-beta1'}, 'tagline': 'You Know, for Search'}

Development / Contribution

View at:

* Upload/release the Python packages onto the
  [PyPi repository](
  + Register the authentication token for access to PyPi:
$ pipenv run keyring set ${PYPIURL}/ __token__
Password for '__token__' in '${PYPIURL}/':

View at:

* Note that the documentation is built automatically by ReadTheDocs (RTD)
  + The documentation is available from
  + The RTD project is setup on
  + As of July 2020, the documentation is built from RST files, _e.g._,
	and documentation files in the
	[`docs` sub-directory](
	The author is a lot more familiar with MarkDown (MD) format,
	and would welcome help in translating the documentation generation
	configuration to use MD- rather than RST-based files.
	Do not hesitate to
	[create an issue](
	or, even better, submit a
	[pull request](

* Build the documentation manually (with [Sphinx](
$ pipenv run python build_sphinx
running build_sphinx
Running Sphinx v3.1.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 9 source files that are out of date
updating environment: [new config] 9 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
build succeeded.

The HTML pages are in build/sphinx/html.

Test the DataMonitor Python module

========================== 1 passed in 0.06s ======================

* Check that a document has been created on ES. When the ES service is made
  from a single node (_e.g._, on a laptop or a local installation), the status
  of the index will be yellow. That is because the documents cannot be
(python-iVzKEypY) ✔ curl -XGET http://localhost:9200/_cat/indices/dm-test-v0
yellow open dm-test-v0 GXEUJjtkRjev3_wSn-5HOg 1 1 1 0 3.7kb 3.7kb