Using Azure Timer Based Trigger to insert Geonames modifications into ArcGIS Online.

The Geonames database offers daily updates of their Gazetteer extract files. The record updates of the previous day are listed in a tab delimited file named “modifications-yyyy-MM-dd.txt”. I wanted to simple visualize the daily modifications in a point feature layer hosted at ArcGIS Online.

Geonames scheme

You have to create a feature layer representing the Geonames record scheme. The location of each record can be determined by using the latitude and the longitude field values. The 5th token represents the latitude and the 6th token represents the longitude. The first token is the unique ID of the Geonames record. The name, the country code and the last modification date are both added to the field scheme of the hosted feature layer.

Daily updates using timer based trigger

In Azure, you can implement a function which is going to be executed using a CRON expression. The function is using the default HTTP web client of the .NET Core Framework for accessing the daily modifications from the Geonames endpoint. The tab delimited file is read line by line using a memory stream. When implementing an Azure function you must not exceed 2 GB memory and the execution time must be less than 2 minutes.

Inserting records using the ArcGIS REST API

The Geonames records are converted into features. A feature is a record having field values, especially an integer based OBJECTID acting as the primary key and a shape field containing the geometry definition. The ArcGIS REST API allows edit operations which can be committed using a transaction. The feature service is secured and every client must be authorized before using any operations like query and edit. If you design a function which is not part of any user related workflow e.g. an app where a user is logged in using his/her credentials, you can register an app by using an appid and a clientid directly in ArcGIS Online. A registered app is allowed to access the available operations of the hosted feature services from the same organization. You can use the Anywhere.ArcGIS library developed by Dave Timmins to easily authorize your custom function with ArcGIS Online and inserting the Geonames records into a hosted feature service.

Visualizing the records using a simple web map

The web map is using a dark-colored basemap and the Geonames records are aggregated by their country code. The web map is hosted using ArcGIS Online and offers a user friendly map-based web-frontend.

Geonames_Overview
Geonames modifications aggregated using the country code
Azure-Geonames-Import
Geonames import used system components

 

Reference links:

 

NVG Viewer v1.0.0-pre release is out

Created the first pre-release of the NVG Viewer for displaying NATO Vector Graphics. Just drag your NVG files onto the map view and a background task is scheduled for parsing the point elements from those NVG files.

NVGViewer

NVG Viewer v1.0.0-pre on github

NVG Viewer on YouTube

 

Lead or bleed

While I was doing a little research browsing the web I found the categories of the Technology Radar from Thoughtworks the most helpful.

I do not want to call it separate DevOps teams, but usually we are delegating everything about IT services to the Service Desk and get an automated incident number back. We need to show some more love and just use the existing software center and Self Service Desk for deploying our tools and applications to our testbeds.

It is very common to use branching like a kind of tagging. We should focus much more on branch early and often.

Products over projects is a tough pattern. Usually you ran directly into the fist of one of the sales member. Projects are a kind of short and mid term solution to satisfy the customers needs. Kickoff, iteration cycles, Kickdown, get your payment and done! We have to work better on the epics and stories for the long term solution a.k.a. products.

Too many platforms, languages and frameworks. You can always get your hands dirty with bleeding edge technologies. We should investigate much more into this areas, lets do some knowledge work.

Score Hold Greyzone Adopt
Techniques
-2 Separate DevOps Teams,
Gitflow
QA in Production,
NPM for all the things,
Docker for builds
Products over Projects,
Threat Modelling
Tools
-1 Remote Development Environment REST-assured,
Sensu,
Sysdig,
Mountebank,
Postman
Platforms
-1 Application Servers Apache Mesos,
Apache Spark,
Cloudera Impala,
CoreCLR & CoreFX,
Nano Server,
Presto,
Rancher,
Time series databases
Two-Factor Authentication
Languages & Frameworks
0 Objective-C ECMAScript 6,
Groovy,
Nancy,
R,
Ruby,
Swift

GDAL using CentOS 6

First of all we add the EPEL and ELGIS (Enterprise Linux GIS) repository to our repostory list.
rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -Uvh http://elgis.argeo.org/repos/6/elgis-release-6-6_0.noarch.rpm

It seems that EPEL broke the compatibility with GDAL.
Take a look at http://www.postgresql.org/message-id/1407521448.20899.9.camel@asus-laptop-03.gunduz.org.

Error: Package: gdal-1.9.2-4.el6.x86_64 (elgis)
           Requires: libarmadillo.so.3()(64bit)
Error: Package: gdal-libs-1.9.2-4.el6.x86_64 (elgis)
           Requires: libarmadillo.so.3()(64bit)

We need to exclude the armadillo package from the EPEL repository.
vi /etc/yum.repos.d/epel.repo

[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
exclude=armadillo*

Let us just try to get the corret rpm package using this link http://proj.badc.rl.ac.uk/cedaservices/ticket/670 provided by the JASMIN analysis platform.
wget http://proj.badc.rl.ac.uk/cedaservices/raw-attachment/ticket/670/armadillo-3.800.2-1.el6.x86_64.rpm
yum install armadillo-3.800.2-1.el6.x86_64.rpm

It should recognize the blas dependency.

Installing:
 armadillo   x86_64   3.800.2-1.el6     /armadillo-3.800.2-1.el6.x86_64    40 k
Installing for dependencies:
 blas        x86_64   3.2.1-4.el6       base                              321 k

After that yum install gdal should work like a charme.

We also need hdf5 package so we need to install it, too.
yum install hdf5.so.6

Installing:
 hdf5-openmpi                i686      1.8.5.patch1-9.el6      epel       1.2 M
Installing for dependencies:
 glibc                       i686      2.12-1.149.el6_6.7      updates    4.3 M
 libgcc                      i686      4.4.7-11.el6            base       113 k
 libgfortran                 i686      4.4.7-11.el6            base       245 k
 libibumad                   i686      1.3.9-1.el6             base        57 k
 libibverbs                  i686      1.1.8-3.el6             base        53 k
 libnl                       i686      1.1.4-2.el6             base       124 k
 librdmacm                   i686      1.0.18.1-1.el6          base        58 k
 libstdc++                   i686      4.4.7-11.el6            base       301 k
 libtool-ltdl                i686      2.2.6-15.5.el6          base        45 k
 lm_sensors-libs             i686      3.1.1-17.el6            base        37 k
 nss-softokn-freebl          i686      3.14.3-22.el6_6         updates    157 k
 openmpi                     i686      1.8.1-1.el6             base       2.9 M
 opensm-libs                 i686      3.3.17-1.el6            base        60 k
 papi                        i686      5.1.1-5.el6             base       485 k
 zlib                        i686      1.2.3-29.el6            base        73 k
Updating for dependencies:
 cpp                         x86_64    4.4.7-11.el6            base       3.7 M
 gcc                         x86_64    4.4.7-11.el6            base        10 M
 gcc-c++                     x86_64    4.4.7-11.el6            base       4.7 M
 gcc-gfortran                x86_64    4.4.7-11.el6            base       4.7 M
 glibc                       x86_64    2.12-1.149.el6_6.7      updates    3.8 M
 glibc-common                x86_64    2.12-1.149.el6_6.7      updates     14 M
 glibc-devel                 x86_64    2.12-1.149.el6_6.7      updates    983 k
 glibc-headers               x86_64    2.12-1.149.el6_6.7      updates    612 k
 libgcc                      x86_64    4.4.7-11.el6            base       102 k
 libgfortran                 x86_64    4.4.7-11.el6            base       267 k
 libgomp                     x86_64    4.4.7-11.el6            base       133 k
 libstdc++                   x86_64    4.4.7-11.el6            base       294 k
 libstdc++-devel             x86_64    4.4.7-11.el6            base       1.6 M
 nspr                        x86_64    4.10.6-1.el6_5          base       113 k
 nspr-devel                  x86_64    4.10.6-1.el6_5          base       111 k
 nss-softokn                 x86_64    3.14.3-22.el6_6         updates    262 k
 nss-softokn-devel           x86_64    3.14.3-22.el6_6         updates     14 k
 nss-softokn-freebl          x86_64    3.14.3-22.el6_6         updates    167 k
 nss-softokn-freebl-devel    x86_64    3.14.3-22.el6_6         updates    139 k
 nss-util                    x86_64    3.16.2.3-2.el6_6        updates     66 k
 nss-util-devel              x86_64    3.16.2.3-2.el6_6        updates     68 k

The libhdf5.so.6 should be located at /usr/lib64/openmpi/lib/libhdf5.so.6.

Ensure which libraries are missing by using which ogrinfo

/usr/bin/ogrinfo

and ldd /usr/bin/ogrinfo

	linux-vdso.so.1 =>  (0x00007fff2a525000)
	libgdal.so.1 => /usr/lib64/libgdal.so.1 (0x00007f3bdbe16000)
	libarmadillo.so.3 => /usr/lib64/libarmadillo.so.3 (0x00007f3bdbc10000)
	libpoppler.so.5 => /usr/lib64/libpoppler.so.5 (0x0000003950e00000)
	libfreexl.so.1 => /usr/lib64/libfreexl.so.1 (0x00007f3bdba06000)
	libgeos_c.so.1 => /usr/lib64/libgeos_c.so.1 (0x00007f3bdb7e2000)
	libodbc.so.2 => /usr/lib64/libodbc.so.2 (0x00007f3bdb57a000)
	libodbcinst.so.2 => /usr/lib64/libodbcinst.so.2 (0x00007f3bdb369000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x0000003947200000)
	libxerces-c-3.0.so => /usr/lib64/libxerces-c-3.0.so (0x00007f3bdadc7000)
	libjasper.so.1 => /usr/lib64/libjasper.so.1 (0x0000003945200000)
	libnetcdf.so.6 => /usr/lib64/libnetcdf.so.6 (0x00007f3bdaa82000)
	libhdf5.so.6 => not found
	libgif.so.4 => /usr/lib64/libgif.so.4 (0x000000394ee00000)
	libjpeg.so.62 => /usr/lib64/libjpeg.so.62 (0x0000003953e00000)
	libgeotiff.so.2 => /usr/lib64/libgeotiff.so.2 (0x00007f3bda84e000)
	libtiff.so.3 => /usr/lib64/libtiff.so.3 (0x0000003951a00000)
	libgta.so.0 => /usr/lib64/libgta.so.0 (0x00007f3bda644000)
	libpng12.so.0 => /usr/lib64/libpng12.so.0 (0x0000003947e00000)
	libcfitsio.so.0 => /usr/lib64/libcfitsio.so.0 (0x00007f3bda28f000)
	libpq.so.5 => /usr/lib64/libpq.so.5 (0x00007f3bda067000)
	liblzma.so.0 => /usr/lib64/liblzma.so.0 (0x000000394de00000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003943a00000)
	librt.so.1 => /lib64/librt.so.1 (0x0000003944600000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000003943e00000)
	libdapclient.so.3 => /usr/lib64/libdapclient.so.3 (0x00007f3bd9e37000)
	libdapserver.so.7 => /usr/lib64/libdapserver.so.7 (0x00007f3bd9c2c000)
	libdap.so.11 => /usr/lib64/libdap.so.11 (0x00007f3bd993b000)
	libspatialite.so.2 => /usr/lib64/libspatialite.so.2 (0x00007f3bd945d000)
	libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x0000003958e00000)
	libCharLS.so.1 => /usr/lib64/libCharLS.so.1 (0x00007f3bd9208000)
	libmysqlclient.so.16 => /usr/lib64/mysql/libmysqlclient.so.16 (0x00007f3bd8e84000)
	libz.so.1 => /lib64/libz.so.1 (0x0000003944200000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f3bd8c4c000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003954200000)
	libssl.so.10 => /usr/lib64/libssl.so.10 (0x0000003952a00000)
	libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x0000003950a00000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x000000394fe00000)
	libm.so.6 => /lib64/libm.so.6 (0x0000003943600000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3bd8a34000)
	libc.so.6 => /lib64/libc.so.6 (0x0000003943200000)
	libhdf5.so.6 => not found
	liblapack.so.3 => /usr/lib64/atlas/liblapack.so.3 (0x00007f3bd8212000)
	libblas.so.3 => /usr/lib64/libblas.so.3 (0x00007f3bd7fbb000)
	libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x00007f3bd7d9a000)
	libclapack.so.3 => /usr/lib64/atlas/libclapack.so.3 (0x00007f3bd7b82000)
	liblcms.so.1 => /usr/lib64/liblcms.so.1 (0x0000003950600000)
	libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x0000003947600000)
	libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x0000003948a00000)
	libopenjpeg.so.2 => /usr/lib64/libopenjpeg.so.2 (0x00007f3bd7963000)
	libgeos-3.3.8.so => /usr/lib64/libgeos-3.3.8.so (0x00007f3bd75cf000)
	libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x0000003951e00000)
	libhdf5_hl.so.6 => not found
	libhdf5.so.6 => not found
	libSM.so.6 => /usr/lib64/libSM.so.6 (0x0000003951600000)
	libICE.so.6 => /usr/lib64/libICE.so.6 (0x0000003951200000)
	libX11.so.6 => /usr/lib64/libX11.so.6 (0x0000003946600000)
	libproj.so.0 => /usr/lib64/libproj.so.0 (0x00007f3bd7380000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x0000003957200000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f3bd713b000)
	libldap_r-2.4.so.2 => /lib64/libldap_r-2.4.so.2 (0x00007f3bd6ee7000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003942e00000)
	libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x000000394e200000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f3bd6ce2000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x0000003957a00000)
	libidn.so.11 => /lib64/libidn.so.11 (0x0000003958a00000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f3bd6a94000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x000000394f600000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x0000003950200000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x000000394f200000)
	libssl3.so => /usr/lib64/libssl3.so (0x0000003953600000)
	libsmime3.so => /usr/lib64/libsmime3.so (0x0000003953200000)
	libnss3.so => /usr/lib64/libnss3.so (0x0000003952e00000)
	libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f3bd6865000)
	libplds4.so => /lib64/libplds4.so (0x00007f3bd6661000)
	libplc4.so => /lib64/libplc4.so (0x0000003952200000)
	libnspr4.so => /lib64/libnspr4.so (0x0000003952600000)
	libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x0000003959200000)
	libfreebl3.so => /lib64/libfreebl3.so (0x00007f3bd645c000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f3bd616a000)
	libf77blas.so.3 => /usr/lib64/atlas/libf77blas.so.3 (0x00007f3bd5f4a000)
	libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x00007f3bd58ee000)
	libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x0000003946200000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x000000394fa00000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f3bd56ea000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003945600000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x0000003956e00000)
	libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x0000003955600000)
	libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000003946a00000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003944e00000)

Copy the missing libraries to the lib64 folder.
cp /usr/lib64/openmpi/lib/libhdf5.so.6 /usr/lib64
cp /usr/lib64/openmpi/lib/libhdf5_hl.so.6 /usr/lib64

Refresh the library cache by using ldconfig.

Hadoop SSH login issues

You should be able to login with ssh using the user which starts the hadoop daemons.
Enable host based authentication and disable password authentication in /etc/ssh/sshd_config as root.

vi /etc/ssh/sshd_config

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts.
#RhostsRSAAuthentication no
# similar for protocol version 2
# Enabled for HADOOP
HostbasedAuthentication yes
# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
# Disabled for HADOOP
PasswordAuthentication no

Tail the secure logs as root in a seperate terminal session.
tail -f /var/log/secure

Following the hadoop setup guide you create a private and public keys.
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

And append them into the authorized keys file.
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

If you try to login using ssh localhost and ssh is asking for a password the files could not be read by ssh. The logs should tell you something like

Authentication refused: bad ownership or modes for file ~/.ssh/authorized_keys

You have to change the modes using:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

After changing that, you should be able to login without using your credentials.

Using C++11 with eclipse CDT

Requirements:
A valid tool chain supporting C++11. I tested it with Ubuntu and GCC 4.6.3.

  • Add the -std=c++0x flag to
    Properties — C/C++ Build — Tool Settings — GCC C++ Compiler — Miscellaneous
  • Add the -pthread flag to
    Properties — C/C++ Build — Tool Settings — GCC C++ Linker — Miscellaneous
  • Add __GCC_EXPERIMENTAL_CXX0X__ to
    Properties — C/C++ General — Paths and Symbols — Symbols — GNU C++
    and leave the value blank