Induction to monitoring
View the Project on GitHub infra-helpers/induction-monitoring
Table of contents generated with markdown-toc
This README is part of the Induction Monitoring project, itself aiming at providing end-to-end examples introducing how to collect, store and query metering events, produced by different sensors on local as well as on clouds.
The python/ sub-folder provides a base Python library, namely datamonitor,
offering a few methods to collect Key Performance Indicators (KPI) from
data files. Those KPIs are then typically wrapped into JSON structures
and sent to an Elasticsearch (ES) service. The KPIs then form corresponding
time-series, which can be further analyzed with Kibana, or any other analytical
tool, typically for detection of outliers and alert notifications.
$ mkdir -p ~/dev/infra && \
git clone git@github.com:infra-helpers/induction-monitoring.git ~/dev/infra/induction-monitoring
$ cd ~/dev/infra/induction-monitoring/python
pyenv, Python 3.8 and, pip and pipenv
export PYENV_ROOT=”${HOME}/.pyenv” export PATH=”${PYENV_ROOT}/bin:${PATH}” if command -v pyenv 1>/dev/null 2>&1 then eval “$(pyenv init -)” fi if command -v pipenv 1>/dev/null 2>&1 then eval “$(pipenv –completion)” fi
_EOF $ . ~/.bashrc $ pyenv install 3.8.3 && pyenv local 3.8.3 $ python3 -m pip install -U pip pipenv
+ Python 3.8:
```bash
$ pyenv install 3.8.3
pip:
$ python3 -m pip install -U pip
pipenv:
$ python3 -m pip install -U pipenv
$ pipenv run cookiecutter gh:ionelmc/cookiecutter-pylibrary
$ cd datamonitor
$ pipenv run tox -e bootstrap
...
Wrote tox.ini
Wrote .travis.yml
DONE.
$ pipenv run tox -l
clean
check
docs
py36
py37
py38
report
datamonitor moduledatamonitor module,
in the Python user space with pip and in a dedicated virtual environment
with pipenv.
/usr/local/opt/python@3.8 on MacOS
or ~/.pyenv/versions/3.8.3 on Linux) may already have many other modules
installed, parasiting a fine-grained control over the versions of every
Python dependency. If all the versions are compatible, then that option
is convenient as it is available from the whole user space, not just
from this sub-directorydatamonitor module has been installed and readily available
from the environment, whether that environment is virtual or not.
In other words, to adapt the documentation for the case where pipenv
is used, just add pipenv run in front of every Python-related command.datamonitor module in the user space (with pip):
$ python -m pip uninstall datamonitor
$ python -m pip install -U datamonitor
datamonitor module in a virtual environment:
$ pipenv shell
(python-iVzKEypY) ✔ python -m pip install -U datamonitor
(python-iVzKEypY) ✔ python -m pip install -U datamonitor
(python-iVzKEypY) ✔ exit
dm-test-v0, or drop the full index altogether:
$ curl -XDELETE http://localhost:9200/dm-test-v0
datamonitor from the Command-Line (CLI)datamonitor module
pip):
$ python -m datamonitor hello world
['hello', 'world']
$ pipenv shell
(python-iVzKEypY) ✔ python -m datamonitor hello world
['hello', 'world']
(python-iVzKEypY) ✔ exit
datamonitor as a module from another Python program>>> from datamonitor import DataMonitor
>>> dm = DataMonitor()
es_conn_str = {‘host’: ‘localhost’, ‘port’: 9200, ‘scheme’: ‘http’} es_conn = dm.es_connect(es_conn_str) es_conn
<Elasticsearch([{‘host’: ‘localhost’}])>
* Get the information/details about the ES cluster/service:
```python
>>> es_info = dm.es_info()
>>> es_info
{'name': 'MyFavoriteES', 'cluster_name': 'elasticsearch_gold', 'cluster_uuid': 'G6YM0RZsRtW3DjoMPsIx_A', 'version': {'number': '7.8.0', 'build_flavor': 'default', 'build_type': 'tar', 'build_hash': '757314695644ea9a1dc2fecd26d1a43856725e65', 'build_date': '2020-06-14T19:35:50.234439Z', 'build_snapshot': False, 'lucene_version': '8.5.1', 'minimum_wire_compatibility_version': '6.8.0', 'minimum_index_compatibility_version': '6.0.0-beta1'}, 'tagline': 'You Know, for Search'}
>>> meta_data_payload = {'unit': 'nb_of_rows', 'value': '6543'}
>>> docid = dm.es_send(index='dm-kpi-v0', payload=meta_data_payload)
>>> docid
'0W0pT3MB2UaYMhH3DxFr'
>>> document_str = dm.es_get(index='dm-kpi-v0', docid=docid)
>>> document_str
{'_index': 'dm-kpi-v0', '_type': '_doc', '_id': '0W0pT3MB2UaYMhH3DxFr', '_version': 1, '_seq_no': 0, '_primary_term': 1, 'found': True, '_source': {'unit': 'nb_of_rows', 'value': '6543'}}
>>> document = None if not '_source' in document_str else document_str['_source']
>>> document
{'unit': 'nb_of_rows', 'value': '6543'}
$ rm -rf _skbuild/ build/ dist/ .tox/ __pycache__/ .pytest_cache/ MANIFEST *.egg-info/
$ pipenv run python setup.py sdist bdist_wheel
View at: https://test.pypi.org/project/datamonitor/0.0.4/
* Upload/release the Python packages onto the
[PyPi repository](https://pypi.org):
+ Register the authentication token for access to PyPi:
```bash
$ PYPIURL="https://upload.pypi.org"
$ pipenv run keyring set ${PYPIURL}/ __token__
Password for '__token__' in '${PYPIURL}/':
View at: https://pypi.org/project/datamonitor/0.0.4/
* Note that the documentation is built automatically by ReadTheDocs (RTD)
+ The documentation is available from
https://datamonitoring.readthedocs.io/en/latest/
+ The RTD project is setup on https://readthedocs.org/projects/datamonitoring/
+ As of July 2020, the documentation is built from RST files, _e.g._,
[`README.rst`](https://github.com/infra-helpers/induction-monitoring/blob/master/python/datamonitor/README.rst)
and documentation files in the
[`docs` sub-directory](https://github.com/infra-helpers/induction-monitoring/blob/master/python/datamonitor/docs/).
The author is a lot more familiar with MarkDown (MD) format,
and would welcome help in translating the documentation generation
configuration to use MD- rather than RST-based files.
Do not hesitate to
[create an issue](https://github.com/infra-helpers/induction-monitoring/issues)
or, even better, submit a
[pull request](https://github.com/infra-helpers/induction-monitoring/pulls)
* Build the documentation manually (with [Sphinx](http://sphinx-doc.org)):
```bash
$ pipenv run python setup.py build_sphinx
running build_sphinx
Running Sphinx v3.1.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 9 source files that are out of date
updating environment: [new config] 9 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
build succeeded.
The HTML pages are in build/sphinx/html.
requirements.txt)
for the CI/CD pipeline (currently Travis CI):
$ pipenv --rm; rm -f Pipfile.lock; pipenv install; pipenv install --dev
$ git add Pipfile.lock
$ pipenv lock -r > datamonitor/ci/requirements.txt
$ pipenv lock --dev -r > datamonitor/ci/requirements-dev.txt
$ git add datamonitor/ci/requirements.txt datamonitor/ci/requirements-dev.txt
$ git commit -m "[CI] Upgraded the Python dependencies for the Travis CI pipeline"
The tests use ElasticMock, which emulates, in a very simple and limited way, an Elasticsearch (ES) cluster/service.
brew stop elasticsearch-full (and
brew start elasticsearch-full to restart it afterwards)sudo systemctl stop elasticsearch
(and sudo systemctl start elasticsearch to restart afterwards)pipenv Shell:
$ pipenv shell
(python-iVzKEypY) ✔ python -V
Python 3.8.3
datamonitor module/library:
(python-iVzKEypY) ✔ python -m pip uninstall datamonitor
pytest
```bash
(python-iVzKEypY) ✔ python -m pytest test_datamonitor.py
=================== test session starts ===========================
platform darwin – Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: ~/dev/infra/induction-monitoring/python/datamonitor, inifile: setup.cfg
collected 1 item
datamonitor/tests/test_datamonitor.py========================== 1 passed in 0.06s ======================
* Check that a document has been created on ES. When the ES service is made
from a single node (_e.g._, on a laptop or a local installation), the status
of the index will be yellow. That is because the documents cannot be
replicated:
```bash
(python-iVzKEypY) ✔ curl -XGET http://localhost:9200/_cat/indices/dm-test-v0
yellow open dm-test-v0 GXEUJjtkRjev3_wSn-5HOg 1 1 1 0 3.7kb 3.7kb
(python-iVzKEypY) ✔ curl -XPUT http://localhost:9200/dm-test-v0/_settings -H "Content-Type: application/json" --data "@../elasticseearch/settings/kibana-read-only.json"
dm-test-v0 index, which should have become
green:
(python-iVzKEypY) ✔ curl -XGET http://localhost:9200/_cat/indices/dm-test-v0
green open dm-test-v0 GXEUJjtkRjev3_wSn-5HOg 1 0 1 0 3.8kb 3.8kb
pipenv Shell:
(python-iVzKEypY) ✔ exit
$ pipenv run tox