Merge branch 'master' of gnunet.org:gnunet

84 files changed, 4711 insertions, 2169 deletions

diff --git a/.gitignore b/.gitignore
index 3c739546fe..b7dd58739e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -46,3 +46,6 @@ src/credential/gnunet-credential
 src/credential/gnunet-service-credential
 src/identity-provider/gnunet-idp
 *.patch
+src/namestore/perf_namestore_api_zone_iteration_flat
+src/namestore/perf_namestore_api_zone_iteration_postgres
+src/namestore/perf_namestore_api_zone_iteration_sqlite
diff --git a/contrib/timeout_watchdog.c b/contrib/timeout_watchdog.c
index 7a0376b9f3..fc61a7cc77 100644
--- a/contrib/timeout_watchdog.c
+++ b/contrib/timeout_watchdog.c
@@ -33,12 +33,14 @@
 
 static pid_t child;
 
+
 static void
 sigchld_handler (int val)
 {
   int status = 0;
   int ret = 0;
 
+  (void) val;
   waitpid (child, &status, 0);
   if (WIFEXITED (status) != 0)
     {
@@ -53,6 +55,7 @@ sigchld_handler (int val)
   exit (ret);
 }
 
+
 static void
 sigint_handler (int val)
 {
@@ -60,8 +63,10 @@ sigint_handler (int val)
   exit (val);
 }
 
+
 int
-main (int argc, char *argv[])
+main (int argc,
+      char *argv[])
 {
   int timeout = 0;
   pid_t gpid = 0;
diff --git a/doc/documentation/chapters/installation.texi b/doc/documentation/chapters/installation.texi
index eeb14473b4..3a76fb1620 100644
--- a/doc/documentation/chapters/installation.texi
+++ b/doc/documentation/chapters/installation.texi
@@ -294,7 +294,7 @@ If you want to be able to enable DEBUG-level log messages, add
 @command{./configure} command.
 @code{DEBUG}-level log messages are in English only and
 should only be useful for developers (or for filing
-really detailed bug reports). 
+really detailed bug reports).
 
 Finally, you probably want to compile @command{gnunet-gtk}, which
 includes @command{gnunet-setup} (a graphical tool for
@@ -389,10 +389,9 @@ For those services, systems with more than one user may require each user
 to specify a different port number in their personal configuration file.
 
 Finally, the user should perform the basic initial setup for the GNU Name
-System (GNS). This is done by running two commands:
+System (GNS) certificate authority. This is done by running:
 
 @example
-$ gnunet-gns-import.sh
 $ gnunet-gns-proxy-setup-ca
 @end example
 
@@ -420,7 +419,7 @@ hosts: files gns [NOTFOUND=return] mdns4_minimal \
 
 @c FIXME: Document new behavior.
 You might want to make sure that @file{/lib/libnss_gns.so.2} exists on
-your system, it should have been created during the installation. 
+your system, it should have been created during the installation.
 
 @node Build instructions for Ubuntu 12.04 using Git
 @section Build instructions for Ubuntu 12.04 using Git
@@ -802,7 +801,7 @@ $ gpg --keyserver pgp.mit.edu --recv-keys A88C8ADD129828D7EAC02E52E22F9BBFEE3485
 @end example
 
 @noindent
-and rerun the gpg --verify command. 
+and rerun the gpg --verify command.
 
 @example
 $ tar xvf gnurl-7.57.0.tar.xz
@@ -1418,7 +1417,7 @@ as a normal user.
 We begin by installing a few Debian packages from stable:
 
 @example
-# apt-get install gcc make python-zbar libltdl-dev libsqlite3-dev \ 
+# apt-get install gcc make python-zbar libltdl-dev libsqlite3-dev \
 libunistring-dev libopus-dev libpulse-dev openssl libglpk-dev texlive \
 libidn11-dev libmysqlclient-dev libpq-dev libarchive-dev libbz2-dev \
 libflac-dev libgif-dev libglib2.0-dev libgtk-3-dev libmpeg2-4-dev \
@@ -1633,7 +1632,7 @@ have to compile it from source:
 @item Unzip the downloaded source tarball using your favourite
 unzipper application In the MSYS shell
 
-@item change to the respective directory 
+@item change to the respective directory
 
 @item Configure glpk for "i686-pc-mingw32":
 
@@ -1781,7 +1780,7 @@ $ make install
 This document is a @b{DEPRECATED} installation guide for GNUnet on
 Windows.
 It will not work for recent GNUnet versions, but maybe it will be of
-some use if problems arise. 
+some use if problems arise.
 
 The Windows build uses a UNIX emulator for Windows,
 @uref{http://www.mingw.org/, MinGW}, to build the executable modules.
@@ -1799,7 +1798,7 @@ We regret any inconvenience, and if you have problems, please report them.
 * Installer::
 * Source::
 @end menu
-     
+
 @node Hardware and OS requirements
 @subsubsection Hardware and OS requirements
 
@@ -1878,7 +1877,7 @@ This batch file opens a shell which is used to invoke the build
 processes.
 MinGW's standard shell (@command{msys.bat}) is not suitable
 because it opens a separate console window.
-On Vista, @command{bash.bat} needs to be run as Administrator. 
+On Vista, @command{bash.bat} needs to be run as Administrator.
 
 @item
 Start @command{bash.sh} and rename
@@ -1948,7 +1947,7 @@ directory (@file{c:\mingw\mingw}).
 
 @item @strong{SQLite}@
 GNUnet uses the SQLite database to store data.
-Get the prebuilt binary from here and unpack it to your MinGW directory. 
+Get the prebuilt binary from here and unpack it to your MinGW directory.
 
 @item @strong{MySQL}@
 As an alternative to SQLite, GNUnet also supports MySQL.
@@ -1984,7 +1983,7 @@ dlltool --input-def ../include/libmySQL.def \
 --output-lib libmysqlclient.a -k
 @end example
 
-@item  Copy include\* to include\mysql\ 
+@item  Copy include\* to include\mysql\
 
 @item  Pass @code{--with-mysql=/c/mysql} to
 @command{./configure} and copy @file{libmysql.dll}
@@ -2082,7 +2081,7 @@ debug information and are quite large. To compile release versions
 (small and fast) set the variable @code{CFLAGS}:
 
 @example
-export CFLAGS='-O2 -march=pentium -fomit-frame-pointer' 
+export CFLAGS='-O2 -march=pentium -fomit-frame-pointer'
 ./configure --prefix=$HOME --with-extractor=$HOME
 @end example
 
@@ -2097,7 +2096,7 @@ located in @file{contrib\win} in the GNUnet source tree.
 @subsubsection Source
 
 @c FIXME: URL
-The sources of all dependencies are available here. 
+The sources of all dependencies are available here.
 
 @c @node Portable GNUnet
 @c @section Portable GNUnet
@@ -2372,7 +2371,7 @@ The hostlist client supports the following proxy types at the moment:
 @end itemize
 
 In addition authentication at the proxy with username and password can be
-configured. 
+configured.
 
 To configure proxy support for the hostlist client in the
 @command{gnunet-setup} tool, select the "hostlist" tab and select
@@ -2521,7 +2520,7 @@ inconsistencies. Some of the other databases do not support repair.
 @item Access mysql as root:
 
 @example
-$ mysql -u root -p 
+$ mysql -u root -p
 @end example
 
 @noindent
@@ -3324,19 +3323,6 @@ steps.
 First of all, GNS needs to be integrated with the operating system. Most
 of this section is about the operating system level integration.
 
-Additionally, each individual user who wants to use the system must also
-initialize their GNS zones. This can be done by running (after starting
-GNUnet)
-
-@example
-$ gnunet-gns-import.sh
-@end example
-
-@noindent
-after the local GNUnet peer has been started. Note that the namestore (in
-particular the namestore database backend) should not be reconfigured
-afterwards (as records are not automatically migrated between backends).
-
 The remainder of this chapter will detail the various methods for
 configuring the use of GNS with your operating system.
 
@@ -3528,7 +3514,7 @@ gnurl --version
 curl 7.56.0 (x86_64-unknown-linux-gnu) libcurl/7.56.0 \
 GnuTLS/3.5.13 zlib/1.2.11 libidn2/2.0.4
 Release-Date: 2017-10-08
-Protocols: http https 
+Protocols: http https
 Features: AsynchDNS IDN IPv6 Largefile NTLM SSL libz \
 TLS-SRP UnixSockets HTTPS-proxy
 @end example
@@ -4125,4 +4111,3 @@ Furthermore, 'make install' will silently fail to set the DNS binaries to
 be owned by group "gnunetdns" unless that group already exists (!).
 An alternative name for the "gnunetdns" group can be specified using the
 @code{--with-gnunetdns=GRPNAME} configure option.
-
diff --git a/doc/man/Makefile.am b/doc/man/Makefile.am
index c40363b59b..a6a116dcac 100644
--- a/doc/man/Makefile.am
+++ b/doc/man/Makefile.am
@@ -17,7 +17,6 @@ man_MANS = \
   gnunet-ecc.1 \
   gnunet-fs.1 \
   gnunet-gns.1 \
-  gnunet-gns-import.1 \
   gnunet-gns-proxy.1 \
   gnunet-identity.1 \
   gnunet-cadet.1 \
@@ -42,6 +41,7 @@ man_MANS = \
   gnunet-transport-certificate-creation.1 \
   gnunet-unindex.1 \
   gnunet-uri.1 \
-  gnunet-vpn.1
+  gnunet-vpn.1 \
+  gnunet-zoneimport.1
 
 EXTRA_DIST = ${man_MANS}
diff --git a/doc/man/gnunet-fs.1 b/doc/man/gnunet-fs.1
index 53a900b46b..65f104d619 100644
--- a/doc/man/gnunet-fs.1
+++ b/doc/man/gnunet-fs.1
@@ -29,10 +29,9 @@ print the version number
 \fB\-V\fR, \fB\-\-verbose\fR
 be verbose
 
-.SH NOTES
 
-
-.SH "REPORTING BUGS"
+.SH BUGS
 Report bugs by using mantis <https://gnunet.org/bugs/> or by sending electronic mail to <gnunet\-developers@gnu.org>
-.SH "SEE ALSO"
+
+.SH SEE ALSO
 \fBgnunet\-publish\fP(1)
diff --git a/doc/man/gnunet-gns-import.1 b/doc/man/gnunet-gns-import.1
deleted file mode 100644
index f90f7f375f..0000000000
--- a/doc/man/gnunet-gns-import.1
+++ /dev/null
@@ -1,17 +0,0 @@
-.TH GNUNET\-GNS\-IMPORT 1 "Jan 31, 2014" "GNUnet"
-
-.SH NAME
-gnunet\-gns\-import \- import some GNS authorities into your GNS namestore
-
-.SH SYNOPSIS
-.B gnunet\-gns\-import
-.br
-
-.SH DESCRIPTION
-\fBgnunet\-gns\-import\fP calls gnunet-namestore to fetch some GNS authorities into your GNS namestore.
-
-.SH BUGS
-Report bugs by using Mantis <https://gnunet.org/bugs/> or by sending electronic mail to <gnunet\-developers@gnu.org>
-
-.SH SEE ALSO
-gnunet\-gns(1), gnunet\-namestore(1)
diff --git a/doc/man/gnunet-namestore-fcfsd.1 b/doc/man/gnunet-namestore-fcfsd.1
index 3160910d43..8039e796fe 100644
--- a/doc/man/gnunet-namestore-fcfsd.1
+++ b/doc/man/gnunet-namestore-fcfsd.1
@@ -13,7 +13,7 @@ Most users will not want to run an FCFS\-zone and thus will not need this progra
 
 \fBgnunet\-gns-fcfsd\fP runs a web server where users can register names to be mapped to their GNS zone.  Names are made available on a First Come First Served basis (hence fcfs).  Registered names do not expire.  The HTTP server is run on the port that is specified in the configuration file in section "[fcfsd]" under the name "HTTPPORT".  The key of the zone in which the names are registered must be specified under the name "ZONEKEY" in the same section.  It is possible to manage gnunet\-gns\-fcfsd using gnunet\-(service\-arm) by starting the daemon using "gnunet\-arm \-i fcfsd" or by setting "FORCESTART=YES" in the "fcfds" section of your configuration.
 
-An FCFS\-zone is run at http://gnunet.org/fcfs/.  The respective zone key can be imported into an individual user's zone using "gnunet-gns-import.sh".  GNS users are encouraged to register their zone with the gnunet.org FCFS authority.
+An FCFS\-zone is run at http://gnunet.org/fcfs/.  GNS users are encouraged to register their zone with the gnunet.org FCFS authority.
 
 If you want to run your own FCFS registrar, you need to first create a pseudonym (using "gnunet\-identity \-C NAME"), and then assign it to be used for the "fcfsd" service using "gnunet\-identity \-e NAME \-s fcfsd".  After that, you can start the FCFSD service (possibly using gnunet\-arm).
 
diff --git a/doc/man/gnunet-zoneimport.1 b/doc/man/gnunet-zoneimport.1
new file mode 100644
index 0000000000..06b3a6bcfe
--- /dev/null
+++ b/doc/man/gnunet-zoneimport.1
@@ -0,0 +1,47 @@
+.TH GNUNET\-ZONEIMPORT 1 "Apr 23, 2018" "GNUnet"
+
+.SH NAME
+gnunet\-zoneimport \- import DNS zone into GNS zone
+
+.SH SYNOPSIS
+.B gnunet\-zoneimport [IP]+
+.br
+
+.SH DESCRIPTION
+\fBgnunet\-zoneimport\fP reads a list of domain names (FQDN) from stdin and issues DNS queries for each of the domain names given.  It then checks if a local ego with a name matching the domain exists. Specifically, if the domain name is "example.fr", it will check if an ego "fr" exists, while for a domain "example.com.fr" it will look for an ego called "com.fr"). If so, it will convert the DNS records into GNS records (in particular converting NS records and glue records to GNS2DNS records) and add them to the namestore under the label ("example" in the examples above).
+
+The arguments given to gnunet\-zoneimport is a list of IP addresses of DNS servers to query.
+
+gnunet\-zoneimport will usually never terminate: it will check when DNS records expire, and re-issue requests when the old DNS records have expired so that GNS always has the latest data.
+
+gnunet\-zoneimport will issue many DNS queries in parallel, but is rate-limited in various ways, so most DNS servers should easily handle the load.  gnunet\-zoneimport will perform a limited number of retries if queries fail.
+
+gnunet\-zoneimport operates incrementally. It will check if the namestore already has (non-expired) records stored for a given name in the respective zone and not issue those requests again.  Thus, it is fine to restart gnunet\-zoneimport whenever the list of domain names changes.
+
+Finally, gnunet\-zoneimport keeps information for each domain name in memory.  This consumes about 200 bytes per domain name, or 1 GB for 5 million labels.
+
+.SH OPTIONS
+.B
+.IP "\-c FILENAME,  \-\-config=FILENAME"
+Use the configuration file FILENAME.
+.B
+.IP "\-h, \-\-help"
+Print short help on options.
+.B
+.IP "\-s MAPSIZE, \-\-size=MAPSIZE"
+Specifies the size (in number of entries) to use for the main hash map.  The value provided should be at least twice the number of domain names that will be given to the tool. This option is required for very large zones where the number of records encountered is too large for the automatic growth mechanism to work (that one is limited to at most 16 MB allocations for security reasons).  Do not worry about this unless you are importing millions of domain names from a zone.
+
+.SH NOTES
+
+Typical invocaton would be:
+
+$ gnunet\-zoneimport 1.2.3.4 < names.txt
+
+
+
+
+.SH BUGS
+Report bugs by using Mantis <https://gnunet.org/bugs/> or by sending