Convert Pandas Dataframe to Kafka Datastream
More articles in Featured Articles
Introduction
The document contains the code piece which converts an input pandas dataframe into a kafka stream. This code can be used with any custom block created in the RZT AI Platform.
Block Code
# Creating an Output Topic and a Producer Object.
# This should be written inside run method.
output_topic,producer=self.data_handler.create_producer(display_name="< TopicAlias Name>")
# The following lines of code should be written inside stream method.
# A List containing the names of all columns of the dataframe. For e.g. columns = ["C1", "C2", "C3", "C4", "C5"]
columns=[< ColumnNames>]
# Iterating over the rows in Pandas Dataframe and sending it to Output Topic
# Here df is the Pandas Dataframe
for index, row in df.iterrows(): producer.send([row[x] for x in columns])
# Closing the Producer Object
producer.close()
