Weather Data Collection System with OpenWeather API and AWS S3

Long time no see and happy new year!! I come from my hiatus with much more experience and a devops project. I am embarking on a journey called the: 30 Days of Devops Challenge, created by CozyCloudCrew. A big appreciation goes to the amazing community coordinators for this initiative to help those who are stuck in that tutorial loop. I am taking on this challenge to help my consistency building projects to help me gain more experience and level up my Devops skills. For a starter, I would be building a Weather Data Collection System.

The Project:

Architectural Diagram

This project is based on a Python application that:

• Fetches weather data from the OpenWeather API.

• Automatically uploads the data to AWS S3 for Storage.

• Displays the weather data as a basic dashboard.

I have some experience with Python, but I revisited the basics to better understand the code. Watching a few YouTube tutorials and running the code through ChatGPT helped me break it down for a clearer understanding. After this, I moved on to cloning the repository.

Step 1: Cloning the Repository

The first thing was to clone the repository from https://github.com/ShaeInTheCloud/30days-weather-dashboard and opened it in VS Code.

git clone https://github.com/ShaeInTheCloud/30days-weather-dashboard
cd 30days-weather-dashboard

This gave me a starting point to work with.

Step 2: Setting Up The Project Structure

I created folders for tests and data. The tests folder contains various unit tests which is very important in Devops practices. The data folder contains output from those various tests.

The .gitignore file was updated to exclude sensitive or unnecessary files:

echo ".env" >> .gitignore
echo "pycache/" >> .gitignore
echo "*.zip" >> .gitignore

Setting Up The Environment

To work on this project, I needed to have the following prerequisites met:

1. Python 3x installed.

2. AWS CLI installed.

3. An AWS account is set up, with “AdministratorAccess” and “AmazonS3FullAccess” policies attached to the account user.

4. Signed up to OpenWeather API and retrieved an API key that will be used to fetch weather data for different regions.

Step 3: Installing Dependencies

Before installing the dependencies, I created a virtual environment to isolate my project and its dependencies from the rest of the system:

python -m venv venv 30days-weather-dashboard

The requirement.txt file is a file that stores all the dependencies that are needed, and to be installed for the Python application. The dependencies for this project are listed below:

boto3==1.26.137        # To interact with AWS S3
python-dotenv==1.0.0   # For managing environment variables
requests==2.28.2       # To fetch weather data from the OpenWeather API

Note: After creating the virtual env, add the virtual env folder venv/created to the .gitignore file.

Verify that your python script is properly configured and ready to run. If needed, add the following configuration to your weather_dashboard.py file.

import os
import json
import boto3
import requests
from datetime import datetime
from dotenv import load_dotenv
import matplotlib.pyplot as plt

# Load environment variables
load_dotenv()

class WeatherDashboard:
    def __init__(self):
        self.api_key = os.getenv('OPENWEATHER_API_KEY')
        self.bucket_name = os.getenv('AWS_BUCKET_NAME')
        self.s3_client = boto3.client('s3')

    def create_bucket_if_not_exists(self):
        """Create S3 bucket if it doesn't exist"""
        try:
            self.s3_client.head_bucket(Bucket=self.bucket_name)
            print(f"Bucket {self.bucket_name} exists")
        except:
            print(f"Creating bucket {self.bucket_name}")
        try:
            # Simpler creation for us-east-1
            self.s3_client.create_bucket(Bucket=self.bucket_name)
            print(f"Successfully created bucket {self.bucket_name}")
        except Exception as e:
            print(f"Error creating bucket: {e}")

    def fetch_weather(self, city):
        """Fetch weather data from OpenWeather API"""
        base_url = "http://api.openweathermap.org/data/2.5/weather"
        params = {
            "q": city,
            "appid": self.api_key,
            "units": "imperial"
        }

        try:
            response = requests.get(base_url, params=params)
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            print(f"Error fetching weather data: {e}")
            return None

    def save_to_s3(self, weather_data, city):
        """Save weather data to S3 bucket"""
        if not weather_data:
            return False

        timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
        file_name = f"weather-data/{city}-{timestamp}.json"

        try:
            weather_data['timestamp'] = timestamp
            self.s3_client.put_object(
                Bucket=self.bucket_name,
                Key=file_name,
                Body=json.dumps(weather_data),
                ContentType='application/json'
            )
            print(f"Successfully saved data for {city} to S3")
            return True
        except Exception as e:
            print(f"Error saving to S3: {e}")
            return False

    def visualize_weather_data(self, cities, temperatures):
        """Visualize weather data using a bar chart"""
        plt.bar(cities, temperatures, color='skyblue')
        plt.title('City Temperatures')
        plt.xlabel('Cities')
        plt.ylabel('Temperature (°F)')
        plt.savefig('temperature_plot.png')
        print("Plot saved as 'temperature_plot.png'")

def main():
    dashboard = WeatherDashboard()

    # Create bucket if needed
    dashboard.create_bucket_if_not_exists()

    cities = ["Ibadan", "Port Harcourt", "Lagos"]
    temperatures = [] # List to store temperature for visualization

    for city in cities:
        print(f"\nFetching weather for {city}...")
        weather_data = dashboard.fetch_weather(city)
        if weather_data:
            temp = weather_data['main']['temp']
            feels_like = weather_data['main']['feels_like']
            humidity = weather_data['main']['humidity']
            description = weather_data['weather'][0]['description']

            print(f"Temperature: {temp}°F")
            print(f"Feels like: {feels_like}°F")
            print(f"Humidity: {humidity}%")
            print(f"Conditions: {description}")

            #Append temperature for visualization
            temperatures.append(temp)

            # Save to S3
            success = dashboard.save_to_s3(weather_data, city)
            if success:
                print(f"Weather data for {city} saved to S3!")
        else:
            print(f"Failed to fetch weather data for {city}")

    # Visualize weather data
    dashboard.visualize_weather_data(cities, temperatures)

if __name__ == "__main__":
    main()

Here’s a breakdown of the script:

1. Imports: This section imports several libraries and modules that are needed throughout the script to handle different tasks.

   • os: Provides functions to interact with the operating system, including accessing environment variables (used here to get API keys and S3 bucket names).

   • json: Enables parsing and creating JSON data, which is used to handle weather data and interact with S3 storage.

   • boto3: The AWS SDK for Python, used for interacting with Amazon Web Services, particularly to interact with S3 to upload weather data.

   • requests: A simple HTTP library for making requests, used to fetch weather data from the OpenWeather API.

   • datetime: Provides classes for manipulating dates and times. Used here to create a timestamp for the weather data, which helps in naming the files and organizing the data.

   • dotenv: A package to load environment variables from a .env file. It is used here to securely manage sensitive information like the API key and S3 bucket name.

   • matplotlib.pyplot: A plotting library used to visualize the weather data (temperatures) in the form of a bar chart.

2. WeatherDashboard Class:

   This class encapsulates all the functions needed to interact with the weather data, S3 storage, and visualization:

   • __init__: The constructor initializes the class by loading the environment variables for the API key and the S3 bucket name. It also creates a connection to the S3 service using boto3.client('s3').

   • create_bucket_if_not_exists: This function checks if the specified S3 bucket exists by using head_bucket. If the bucket doesn’t exist, it tries to create the bucket using create_bucket.

   • fetch_weather: This function fetches the weather data for a specific city from the OpenWeather API. It constructs the URL with the required parameters, sends the request, and returns the weather data in JSON format. If there’s an error, it handles it and returns None.

   • save_to_s3: This function saves the weather data to an S3 bucket. It adds a timestamp to the data and creates a file name based on the city and timestamp. It then uploads the JSON data to the S3 bucket using put_object.

   • visualize_weather_data: This function takes a list of cities and their corresponding temperatures, then creates a bar chart using Matplotlib. The chart is saved as a PNG file (temperature_plot.png) for further analysis or sharing.

3. Function Main: This function is the main entry point of the program and orchestrates the whole flow:

   • Create Dashboard Object: An instance of WeatherDashboard is created, which automatically initializes the necessary variables for API interaction and AWS S3 connectivity.

   • Create S3 Bucket: The create_bucket_if_not_exists function is called to ensure the S3 bucket exists (or is created if necessary).

   • Fetch Weather for Multiple Cities: A list of cities (Ibadan, Port Harcourt, Lagos) is iterated over. For each city:

     • The fetch_weather function is called to get the current weather data.

     • If the data is successfully fetched, it prints out the temperature, feels-like temperature, humidity, and description.

     • The temperature is appended to a list for later visualization.

     • The weather data is saved to the S3 bucket using save_to_s3.

   • Visualize the Weather Data: After all cities have been processed, the visualize_weather_data function is called to create a bar chart of the temperatures for the cities. The chart is saved to the local disk as temperature_plot.png.

Finally, the script runs this process when executed by calling main() if the script is the main module.

Step 4: Configuring AWS

This step involved configuring the AWS CLI to interact with my S3 bucket:

aws configure

I was asked to configure the following details:

• AWS Access Key ID (from my AWS user account )

• AWS Secret Access Key (from my AWS user account )

• Default region name (I used us-east-1)

• Default output format (I left this blank)

After entering the credentials, I was all set to work with AWS services.

Step 5: Setting Up Environment Variables

Sensitive information such as the API keys are stored securely in the .env file. I also added the name of my s3 bucket in the file as well.

OPENWEATHER_API_KEY=my_super_secret_key
AWS_BUCKET_NAME=unique_bucket_name

It is a good DevOps practice to never hardcode secret information directly into code.

Step 6: Running the Application

To run the Python script, I used the following command:

python3 src/weather_dashboard.py

The application successfully:

• Fetched real-time weather data from the OpenWeather API.

• Uploaded the data to my AWS S3 bucket.

• Displayed the weather data in the terminal.

• Displayed a bar chart to help visualize the data

Verifying the bucket content

To check if everything worked, I logged into the AWS Management Console and navigated to S3. And there it was — my bucket with the weather data files inside!

Key Takeaways From Day 1

Here’s what I gained from this hands-on experience:

---

Key Learnings From Day 1 of the 30 Days of DevOps Challenge

1. AWS S3 Bucket Creation and Management

While I’ve worked with S3 before, I discovered a new way to utilize it for a different use case. I also learned how to properly manage AWS CLI credentials and ensure the bucket name is globally unique.

2. Securely Storing API Keys with Environment Variables

Instead of embedding sensitive information like API keys directly into my code, I used a .env file to store them securely. This approach follows best practices in DevOps, as it keeps secrets private and makes the code reusable for others using their own credentials.

3. Python Best Practices for API Integration

I explored Python libraries such as requests (to fetch weather data), boto3 (to interact with AWS S3), and python-dotenv (to manage environment variables). These tools gave me a deeper understanding of API integration and helped me write modular and clean code.

4. Effective Git Workflow for Project Development

I practiced using a straightforward Git workflow to keep my project organized. This included setting up a .gitignore file to exclude unnecessary files (like the virtual environment and .env file), cloning the repository, and pushing updates to GitHub. It reinforced the importance of version control and how to manage code professionally, just like a DevOps engineer.

5. Cloud Resource Management

I gained practical experience in managing cloud resources, such as configuring the AWS CLI, working with access keys, and troubleshooting issues like invalid credentials or bucket name conflicts. These challenges deepened my understanding of cloud systems and how to resolve common issues.

---

Final Thoughts

This project wasn’t just about building a weather data app — it was a step towards developing a DevOps mindset. I now have a clearer grasp of securely handling sensitive data, leveraging cloud services, and collaborating effectively using Git. Day 1 was challenging but incredibly rewarding as I saw my efforts come to life.