JDBC connector
Support for the JDBC driver means that data can easily be exported from a Kafka topic and ingested directly to QuestDB by means of Kafka Connect.
This article assumes that users have successfully set up an installation of Kafka and are ready to start exporting messages to QuestDB.
#
PrerequisitesYou will need the following:
- Kafka
- A running QuestDB instance
#
Configure KafkaThe following binaries must be available to Kafka:
- Kafka Connect JDBC binary
- PostgreSQL JDBC driver
To download these files, visit the
Kafka Connect JDBC
page which provides CLI installation and a direct download of the required
.jar
files. Select the ZIP file for download, unzip the contents of the
archive and copy the required .jar
files to the Kafka libs
directory:
A configuration file /path/to/kafka/config/connect-jdbc.properties
must be
created for Kafka Connect in standalone mode. The Postgres connection URL must
match the destination QuestDB instance and a topic can be specified under the
topics={mytopic}
key. This example guide uses a topic example-topic
and the
Postgres server is running on the default port 8812
.
Create a file config/connect-jdbc.properties
with the following contents:
#
Start KafkaThe commands listed in this section must be run from the Kafka home directory and in the order shown below.
Start the Kafka Zookeeper used to coordinate the server:
Start a Kafka server:
Start Kafka Connect:
#
Publish messagesMessages can be published via the console producer script:
A >
greater-than symbol indicates that a messages can be published to the
example topic from the interactive session. Paste the following minified JSON as
a single line to publish messages and create the table example-topic
in the
QuestDB instance:
#
Verify the integrationTo verify that the data has been ingested into the example-topic
table, the
following request to QuestDB's /exp
REST endpoint can be made to export the
table contents via CURL:
The expected response based on the example JSON message published above will be the following:
#
JSON formatThe JSON object sent in the example above has the following structure containing
schema
and payload
objects: