Getting Started with PySpark: Core Concepts and First Steps

Getting Started with PySpark: Core Concepts and First Steps

PySpark is the Python API for Apache Spark, a powerful distributed computing framework. If you're already comfortable with SQL and PostgreSQL, you'll find PySpark familiar in many ways—it uses SQL-like operations but scales them across multiple machines. This guide will help you understand the core concepts and write your first PySpark programs.

Why PySpark?

You already know that PostgreSQL is excellent for structured data queries on a single machine. PySpark extends that capability to massive datasets distributed across clusters. When your data grows beyond what one server can handle efficiently, PySpark lets you process it in parallel without rewriting your SQL logic completely.

Think of it this way: PostgreSQL optimizes queries on data that fits on one machine. PySpark optimizes queries on data spread across many machines. Both use similar SQL concepts, but PySpark handles the distribution complexity for you.

Core Concept 1: The Spark Session

Every PySpark program starts with a Spark Session. This is your entry point to Spark functionality—similar to how you'd establish a connection to PostgreSQL before running queries.

from pyspark.sql import SparkSession

# Create a Spark Session
spark = SparkSession.builder \
    .appName("MyFirstApp") \
    .getOrCreate()

print(spark.version)  # Check your Spark version

The SparkSession is your gateway to all Spark operations. You'll use it to read data, create DataFrames, and execute SQL queries. The .builder pattern lets you configure Spark before starting it.

Core Concept 2: DataFrames

A Spark DataFrame is similar to a PostgreSQL table or a pandas DataFrame—it's a distributed collection of data organized into named columns. The key difference: it's distributed across a cluster, not stored on one machine.

import pyspark.sql.functions as F

# Create a simple DataFrame
data = [
    ("Alice", 25, "Engineering"),
    ("Bob", 30, "Sales"),
    ("Carol", 28, "Engineering")
]

columns = ["name", "age", "department"]
df = spark.createDataFrame(data, columns)

df.show()  # Display the data

Output:

+-----+---+-------------+
| name|age|   department|
+-----+---+-------------+
|Alice| 25|  Engineering|
|  Bob| 30|        Sales|
|Carol| 28|  Engineering|
+-----+---+-------------+

DataFrames are immutable—when you transform them, you get a new DataFrame. This design makes distributed computing safer and more predictable.

Core Concept 3: Transformations and Actions

PySpark operations fall into two categories:

Transformations create new DataFrames from existing ones. They're lazy—Spark doesn't execute them immediately. Examples: select(), filter(), groupBy().

Actions return results to your program or write data to storage. They trigger actual computation. Examples: show(), collect(), write().

# Transformation (lazy—nothing happens yet)
engineers = df.filter(df.department == "Engineering")

# Action (triggers computation)
engineers.show()  # Now Spark executes the filter

This lazy evaluation is powerful: Spark can optimize your entire query plan before executing it, similar to how PostgreSQL's query planner works.

Reading Data

PySpark can read from many sources. Since you know PostgreSQL, let's start with CSV (similar to exporting from PostgreSQL) and then explore databases.

# Read a CSV file
df = spark.read.csv("employees.csv", header=True, inferSchema=True)

# Read from PostgreSQL
df = spark.read \
    .format("jdbc") \
    .option("url", "jdbc:postgresql://localhost:5432/mydb") \
    .option("dbtable", "employees") \
    .option("user", "postgres") \
    .option("password", "password") \
    .load()

The PostgreSQL connection uses JDBC (Java Database Connectivity). PySpark handles the complexity of reading distributed data from your database.

Basic Transformations

These operations will feel familiar if you know SQL:

# SELECT specific columns (like SQL SELECT)
df.select("name", "department").show()

# WHERE clause (like SQL WHERE)
df.filter(df.age > 26).show()

# GROUP BY and aggregation
df.groupBy("department").agg(F.avg("age")).show()

# ORDER BY
df.orderBy(df.age.desc()).show()

Notice the syntax is different from SQL, but the logic is identical. You're selecting columns, filtering rows, grouping data, and sorting—exactly like PostgreSQL queries.

Using SQL Directly

Since you're advanced in SQL, PySpark lets you write SQL directly:

# Register DataFrame as a temporary table
df.createOrReplaceTempView("employees")

# Write SQL queries
result = spark.sql("""
    SELECT department, AVG(age) as avg_age
    FROM employees
    GROUP BY department
    ORDER BY avg_age DESC
""")

result.show()

This approach leverages your existing SQL expertise. For complex queries, you might find SQL more intuitive than the DataFrame API.

Understanding Partitions

Partitions are how Spark distributes data across cluster nodes. Each partition is processed independently and in parallel. This is the foundation of Spark's speed.

# Check how many partitions your DataFrame has
print(df.rdd.getNumPartitions())

# Repartition for better performance
df_repartitioned = df.repartition(4)  # Use 4 partitions

# Partition by a column (useful for large datasets)
df.write.partitionBy("department").parquet("output_path")

More partitions = more parallelism, but too many creates overhead. For a 1GB dataset, 4-8 partitions is typical. For 100GB+, you might use 100+ partitions.

Writing Data

After processing, you'll want to save results. PySpark supports multiple formats:

# Write to Parquet (compressed, efficient format)
df.write.parquet("output/employees")

# Write to CSV
df.write.csv("output/employees.csv", header=True)

# Write back to PostgreSQL
df.write \
    .format("jdbc") \
    .option("url", "jdbc:postgresql://localhost:5432/mydb") \
    .option("dbtable", "employees_processed") \
    .option("user", "postgres") \
    .option("password", "password") \
    .mode("overwrite") \
    .save()

Parquet is PySpark's native format—it's compressed, preserves data types, and reads faster than CSV. Use it for intermediate results in ETL pipelines.

A Complete ETL Example

Let's combine these concepts into a simple ETL pipeline:

from pyspark.sql import SparkSession
import pyspark.sql.functions as F

# Initialize
spark = SparkSession.builder.appName("ETL").getOrCreate()

# Extract: Read from PostgreSQL
df = spark.read \
    .format("jdbc") \
    .option("url", "jdbc:postgresql://localhost:5432/mydb") \
    .option("dbtable", "raw_employees") \
    .option("user", "postgres") \
    .option("password", "password") \
    .load()

# Transform: Clean and aggregate
df_clean = df.filter(df.age > 0) \
    .withColumn("age_group", F.when(df.age < 30, "Young").otherwise("Senior"))

df_summary = df_clean.groupBy("department", "age_group") \
    .agg(F.count("*").alias("count"), 
         F.avg("salary").alias("avg_salary"))

# Load: Write results
df_summary.write.mode("overwrite").parquet("output/summary")

print("ETL complete!")

This pipeline reads employee data, filters invalid records, adds a calculated column, aggregates by department and age group, and saves results. It's a typical ETL workflow you'd recognize from your data modeling experience.

Common Pitfalls for Beginners

Forgetting that transformations are lazy: If you write a filter but don't call an action, nothing happens. Always end with show(), collect(), or write().

Collecting too much data: df.collect() brings all data to your driver machine. On large datasets, this crashes your program. Use show() or limit() instead.

Ignoring partitions: A single-partition DataFrame won't use your cluster's parallelism. Repartition large datasets appropriately.

Not caching repeated use: If you use the same DataFrame multiple times, cache it to avoid recomputation:

df_cached = df.cache()  # Keep in memory
result1 = df_cached.filter(...).count()
result2 = df_cached.groupBy(...).agg(...).show()  # Reuses cached data

Next Steps

You now understand the fundamentals: Spark Sessions, DataFrames, transformations, actions, and basic ETL. From here, explore:

• Window functions: For time-series analysis and ranking (similar to PostgreSQL window functions)
• Joins: Combining DataFrames from multiple sources
• Streaming: Processing continuous data pipelines (you have beginner knowledge here)
• Performance tuning: Optimizing large-scale jobs

Your PostgreSQL and SQL expertise gives you a huge advantage. PySpark's SQL interface lets you leverage that knowledge immediately, while the DataFrame API offers more programmatic control when needed.