Apache Kafka#
Kafka provider is bundled with Querona and no additional installation or configuration is required.
Each Kafka topic is represented as one Querona virtual table, regardless of topic’s partition count. Because Kafka does not support any kind of schemas, virtual Kafka tables are put in virtual [dbo] schema.
Virtual Kafka table supports five columns:
key - NVARCHAR(MAX) or VARBINARY(MAX), nullable
value - NVARCHAR(MAX) or VARBINARY(MAX), nullable
offset - INT, not nullable
partition - INT, not nullable
utctimestamp - DATETIME2, not nullable
Textual representation of key and value use UTF8 encoding. Timestamp represents either time of record creation or time of record save on Kafka broker, depending on topic configuration.
Each time Kafka table is read, all records starting from first available are read. Querona consumer groups do not commit offsets, neither automatically nor manually. If new committed records are available after the read process has been started, they are not loaded.
Querona uses ‘querona.technical’ consumer group id by default. User can also specify consumer group id per query using OPTION clause:
SELECT * FROM KafkaTable OPTION(KAFKA_GROUP_ID 'my-group')
Kafka’s internal __consumer_offsets topic is skipped during metadata retrieval.