Is it possible to create PySpark DataFrame from external data sources?

1.	Is it possible to create PySpark DataFrame from external data sources?
Answer» Yes, it is! Realtime applications make use of external file systems like LOCAL, HDFS, HBase, MySQL table, S3 Azure etc. Following example shows how we can create DATAFRAME by reading data from a csv file present in the local system: DF = spark.read.csv("/path/to/file.csv") PySpark supports csv, TEXT, AVRO, parquet, tsv and many other file extensions.

Answer»

Yes, it is! Realtime applications make use of external file systems like LOCAL, HDFS, HBase, MySQL table, S3 Azure etc. Following example shows how we can create DATAFRAME by reading data from a csv file present in the local system:

DF = spark.read.csv("/path/to/file.csv")

PySpark supports csv, TEXT, AVRO, parquet, tsv and many other file extensions.

Discussion

No Comment Found

Related InterviewSolutions

What would happen if we lose RDD partitions due to the failure of the worker node?
What do you understand by Pyspark Streaming? How do you stream data using TCP/IP Protocol?
How can you inner join two DataFrames?
What is PySpark SQL?
What do you understand by Pyspark’s startsWith() and endsWith() methods?
Is it possible to create PySpark DataFrame from external data sources?
How can we create DataFrames in PySpark?
What are the different approaches for creating RDD in PySpark?
How to create SparkSession?
What are the profilers in PySpark?

Is it possible to create PySpark DataFrame from external data sources?

Discussion

No Comment Found

Related InterviewSolutions

Reply to Comment