The problem is, GitHub Releases was never really designed for proper binary distribution. It works fine for attaching a zip or tarball, but as soon as your project starts relying on binaries, cracks begin to show.

Here are the main pain points I’ve run into:

1. Poor user experience

Users have to click through the release page and manually pick files. Direct URLs exist but are long and brittle. If you want someone to install via a package manager, GitHub Releases gives you nothing.

2. No download insights

You cannot see which versions are popular or how many times a binary has been downloaded. If you’re shipping multiple builds, you are essentially guessing which ones people actually need.

3. Limited automation

GitHub Actions lets you upload artifacts, but that’s it. There’s no API for stats, no built-in APT or YUM repo support, and you end up writing extra scripts to fill gaps that should not exist.

What developers really need

For projects shipping binaries, the essentials are:

• Easy hosting and sharing of binaries

• Real download analytics

• Support for package managers like APT and YUM

• CI/CD integration without friction

• A service built specifically for binaries

That’s why I built ZipZen. It is lightweight release hosting for developers. You get reliable binary hosting, zero-config APT/YUM repositories, and download statistics (more detailed analytics coming soon).

GitHub Releases will always exist as a basic option, but if you care about your users and your workflow, the cracks are impossible to ignore. If you’ve been frustrated with binary distribution, ZipZen might be worth checking out.

I’d love to hear from you: how do you currently handle distributing binaries for your projects?

With Ziplink, creating branded short links is ridiculously easy. After adding your own domain, in just a few seconds, you can generate trackable, custom URLs that look great and help you stand out. Here is a step-by-step guide on how to do it.

First things first—head over to Ziplink, select the "Basic" or "Growth" package and either sign up or log in. It takes just a few moments!

Prerequisites

You can't add a custom domain unless you own one. If you don't have a domain yet, you'll need to register one first. To do so, you may visit namecheap.com or godaddy.com.

Once your domain is ready, decide whether you want to share your links with or without a subdomain.

If you choose to use a subdomain (recommended), add the following DNS record to your domain settings:

Name: subdomain.yourdomain.com.
Type: CNAME
Value: zipl.ink.
TTL: 300

In the case of a root domain (so called, APEX domain), add the follwing DNS record to point your domain to Ziplink.

Name: yourdomain.com.
Type: A
Value: 3.72.197.199
TTL: 3600

Add your domain to Ziplink

Once you're in, you'll see the dashboard. From there, follow these steps:

• Navigate to your workspace's domain settings

• Above the table, click on "Add domain"

• Optionally, fill the "Alias" field with an easy to remember name

• Add the desired domain you already configured with your DNS provider

• Click "Submit"

Within a few seconds, Ziplink registers your domain in the background.

Create your first branded link

Now that your domain is added, you can create a link just like you used to, but this time using your custom domain.

• Click on "Create new" in the sidebar to add a new link

• Enter the destination URL where you want your visitors to be redirected

• Click on the domain dropdown, and select your own domain

• Optionally, add a custom link alias

• Click on "Create link" at the bottom of the form

Voilà! You just created a branded link! 🎉

Go and share it on your favorite social media platform or maybe with your customers.

First impressions count—even online. A link shortener turns a sprawling web address into a concise, shareable snippet that not only looks cleaner but also enhances your message. Let’s explore what a URL shortener is, why it’s essential for modern online communication, and how you can quickly create your own short links.

What Is a URL Shortener?

At its essence, a URL shortener takes a long, clunky web address and transforms it into a neat, compact version. For example, a lengthy URL like:

https://www.example.com/articles/insightful-tips-on-effective-link-sharing?utm_source=expatgeekdiaries.com

can be converted into something as simple as:

https://exmpl.co/a1B2c

This process works by generating a unique key for the original link and storing it in a database. When a user clicks the shortened URL, the system retrieves the original address and redirects them seamlessly. It’s a straightforward yet powerful tool for decluttering your digital space.

The mechanics (in the case of this example) are simple. We throw a long URL at the link shortener, and in return, it outputs a short URL.

Why Use a URL Shortener?

There are many practical reasons to use URL shorteners beyond mere aesthetics:

• Clean and attractive links: A short URL looks far more polished on social media, emails, and even printed materials.

• Shareability: On platforms with character limits—like Twitter, khm…—a compact link leaves more room for your message.

• Analytics: Many services, such as Ziplink, offer tracking features, providing valuable data such as click counts, geographic information, and device types.

• Branding possibilities: Custom short links can include your brand’s name, reinforcing your identity every time a link is shared.

• Improved experience: A tidy link reduces visual clutter and allows your audience to focus on your content.

These handy tools aren’t just for tech experts—they’re practical in many everyday situations:

• Social media campaigns: Marketers track post performance and engagement with every click.

• Email marketing: Shortened links add a level of sophistication to your newsletters and can boost click-through rates.

• Text and messaging: With limited space in SMS and chat apps, a compact URL can be the difference between your message being read or overlooked.

• Printed media: Business cards, flyers, and posters become more effective when paired with memorable, short links.

• Content embedding: In blog posts and articles, shortened links maintain a neat layout without overwhelming your text.

Consider your daily digital interactions. Whether you’re sharing a recipe on Facebook, promoting an event via email, or even incorporating a QR code in your print materials, a short URL can make the experience smoother and more professional.

How to Create Your Own Short Links

Using a URL shortener is a breeze. Here’s a quick guide to get you started:

1. Choose a shortener: Pick a link shortener that fits your needs—many offer options like custom branding, QR code creation, and analytics. Ziplink, Short.io, and Bitly are all capable of that. If you are looking for something really simple, you may want to check out TinyURL as well.

2. Enter Your URL: Depending on the shortener, you need to sign up beforehand, but in essential: input the long web address you want to shorten.

3. Customize (Optional): Some services let you add a personalized alias/back-half to the short link. This can be a great option, but be aware: the aliases must be unique. So in the case of https://exmpl.co/a1B2c, no two a1B2c alias can exist at the same time.

4. Generate and share: Once created, your new link is ready to be shared across your digital channels.

Final Thoughts

Simplicity can significantly enhance your communication. URL shorteners do more than just reduce character count—they refine your links and improve how you connect with your audience. Whether you’re crafting a social media post, sending out a newsletter, or printing promotional materials, consider the impact a clean, efficient link can have on your message.

Spread the word, hopefully we see less clunky URLs in the future!

Picture this: Your employees are working hard to deliver new features to customers. The user base is growing very nicely. Based on the positive user feedback, the management has new feature ideas, and the board is happy to see the increase in profit. The company is investing more in its software engineering teams, which are getting bigger budgets.

It holds true for almost any other software company with a good product to sell. That's good, isn't it? What's wrong with that? Well, not much. At least while the company grows dynamically. My favorite mother-in-law has an excellent saying:

A company either grows dynamically or sinks irretrievably.

This is especially true for IT companies that are operating on elevated costs. Of course, what high is objective and depends on many factors, though one thing is common: they shouldn’t have to pay as much as they do.

How do you know that you are on the wrong track, and you need to invest a bit into taking care of what you have done so far? Here are seven signs and suggestions.

1. Excessive Spendings

Let’s start with the most obvious, though often overlooked sign, elevated spendings. Why is it so easy to overlook or ignore this sign if it’s so frequent? Mostly due to lack of monitoring and the human nature. We tend to find reasons for questionable things, even if we shouldn’t. The issue is realized when the company starts to stagnate, or the bills are going way above the budget.

The solution is relatively easy, but requires investigation. The first step is to go down the rabbit hole and track down all cost drivers. This can include checking all billing items on the bill, checking the infrastructure usage, or literally anything that tracks the resource usages.

By knowing which components of the infrastructure contribute the most to the bills, several actions can be taken. From this point, it is highly dependent on the infrastructure what to do next. However, making sure that the right-sized computing resources are used, correct VAT settings are applied on billing accounts, or dangling resources are cleaned up easily can result in tens of thousands of dollars savings annually.

2. High Maintenance Costs

It is challenging to pinpoint the cause of increased maintenance costs, as it can be somewhat natural in the case of complex systems. Many times, the processes the teams are following are inefficient. In other cases, the system is poorly or overengineered. Also, it can happen that not the right tools are used or not the right team is doing the maintenance work.

In this case, all problems need a unique solution. There is no one fit for all. A good first step is to revise the processes in collaboration with the teams that are participating in maintenance. In many cases, they are not raising issues with the processes, or they don’t even realize that their processes are inefficient.

Then, check for resource usage. Are the resources utilized as they should have been? Is there any under or over provisioned computing instances?

If you found no resource utilization issues, the processes in place are efficient, but the maintenance cost is still high, it may be worth investigating if you are using the right technology. For example, setting up Kubernetes for a website just not makes sense.

3. Inefficient Processes

How can processes influence IT expenses? If you don’t know the answer to this straight off, you need to get in touch with me.

A company without efficient processes is doomed. That’s true for its teams as well. No, we are not talking about micromanagement here. It is about defining the way a team works and keeps itself accountable to other teams.

If the infrastructure team (or any other teams) has no processes for regular, scheduled, and most importantly, uninterrupted maintenance time, no matter how clever the team or well established the infrastructure, it will get rusty.

The solution is introducing dedicated maintenance time with a plan the team can stick to. However, everyone in the company should respect this maintenance time. Even the CEO should not intervene in that. Why? Because otherwise it is a wasted time if the team is dragged back to do the regular feature shipping or bug fixing.

Of course, there are scenarios when this cannot be avoided (like an outage), but that should be rare to be able to count on one hand during the course of the year.

4. Lack of Standardization

Is the “not invented here (NIH) syndrome” familiar to you? NIH is basically the tendency for management or teams to reject any idea that did not originate within the organization.

In the beginning, custom tooling may cost less and comes with less maintenance, though after some time, this trend will turn to the opposite. Training people for in-house solutions, writing extra documentation, and not using the benefits provided by using standardized solutions, which are mostly open-sourced.

Nowadays, open-source communities provide solutions to almost anything a company may need. These communities are taking care of bug fixes, shipping new features, maintaining the documentation, keeping the tooling up-to-date, and so much more.

Think about Docker or Kubernetes. If a company had to ship the containerization or orchestration environment, how many teams would have to work on that for how long? What about Prometheus or Grafana? These products are maintained by hundreds of developers every day. Does your company have these spare resources?

Letting the fear of not inventing everything go, and starting using standardized and open-source tooling, will definitely help lower the costs. Sure, not all open-source products are completely free. Open-source, per se, doesn’t mean it is free on any scale with no restrictions. However, still saves a company a lot of effort, time, and money.

5. Obsolete Technology

The mantra of “what works doesn't need to be changed” is not true in this case. Sticking with outdated technology means more fixing and upkeep. When parts get old and the vendors stop lending a hand, you end up shelling out more just to keep things running.

Technology moves fast, and old systems can't always keep up. This can force companies to splurge on pricey upgrades or come up with clever hacks to make sure everything still plays nice together, all adding to the bill.

Without dusting off and regularly patching old technology, the company-wide open to hackers. Also, it can really put the brakes on your team's productivity. Slow interfaces, laggy processing, and compatibility issues with newer tools can turn work into a slog.

The solution to the problem is simple, but time-consuming. Start upgrading the old tech stack. It comes with planning, resource allocations, focused work, increased maintenance time, but keep in mind: at some point, the tech debt must be paid.

Acknowledging and being conscious about the tech debt is not a shame. It is simply responsible thinking. Nothing is wrong with that.

6. Uncontrolled Growth

Let's talk about the wild side of growth — the uncontrolled kind. When things start booming without a leash, it's like the wild west out there.

Uncontrolled growth tends to spread like wild weeds. It's like trying to manage an overgrown garden, where you spend all your time cutting back excess instead of nurturing what's important. When growth happens too fast, the company can find itself scrambling to keep up with rising demands, whether it's for customers, supplies, staff, or even office space.

Quality control and peer review becomes shaky amidst unchecked expansion. With all the focus on getting bigger, there's a risk of letting craftsmanship slip or dropping the ball on multiple fields.

For the infrastructure, this means bad decisions, such as renting cloud computing resources that are not needed, forgetting about unused temporary instances, over-provisioned instances, poor resource utilization and so much more.

Dealing with uncontrolled growth requires a joined effort of multiple teams, and the issue should be mitigated together. While mitigating the issue, make sure to look out for bad practices, over provisioned instances, dangling resources, and so on. These can be cleaned up easily.

7. Overestimated Growth

In contrast to uncontrolled growth, the other side of the coin is overestimated growth. When a company is releasing a product, it is often believing that it will get a huge traction from day one. However, be honest, it’s not the truth.

The estimated user base is communicated to the engineering teams, who are designing the infrastructure accordingly. And here is the key word: accordingly. It is not their role to (always) second guess the management’s expectations, so they will scale the systems up as needed.

The upscaled resources evidently result in higher bills. By simply revising the actual user base, the system can be scaled down as required, lowering the bills. It must be noted that this requires some sort of monitoring and reporting. Without knowing the exact usage of the systems, this cannot be done.

Conclusion

Running a software company is like steering through a constantly changing environment. Success comes from balancing new ideas and careful management. As your customer base grows and profits increase, it might be tempting to expand fast. But there are risks you need to watch out for that could mess up your plans.

Dealing with overspending, high costs to keep things running, and inefficient ways of doing things needs more than just looking into the problem. You have to really work at making things smoother. Using standard methods and getting rid of old ways of doing things can make your company more effective. And if your company is growing too fast, you need to rethink your strategy.

Success isn't just about moving forward. It's also about managing your resources wisely and creating a workplace where being able to change and plan ahead is important. By keeping a close eye on things and being ready to change direction, software companies can keep growing and coming up with new ideas sustainably.

In today's business landscape, leveraging technology effectively is not just an option but a necessity for sustainable growth. As consultants, our role is pivotal in steering organizations toward success through well-thought-out IT initiatives. In this in-depth guide, we'll explore key strategies to develop impactful IT roadmaps that align with business goals and drive lasting results.

Understanding the Client's Needs

Rule #1: The client knows what it wants until you ask questions. After that, everything falls apart. Most of the time.

The journey toward a successful IT roadmap begins with gaining a profound understanding of your client's business objectives and long-term goals. Aligning IT initiatives with the broader vision ensures a tailored roadmap that meets specific needs. Successful consultants start by asking the right questions to uncover the unique challenges and opportunities that technology can address.

Begin by fostering an environment of open communication. Schedule comprehensive meetings with key stakeholders, decision-makers, and department heads. Create a platform where everyone feels comfortable sharing their insights and concerns. By engaging in dialogue, you uncover nuances that might not be apparent in formal documentation.

Do not forget to involve the team leads of the affected teams. In many companies, the details are easily falling through the cracks, resulting in misdelivered information. You will get most context on the specifics from them. For high-level, company or department-wide questions, turn to the department heads.

Digging into the existing pain points is crucial. What challenges does the client currently face in their day-to-day operations? Are there bottlenecks in communication, inefficiencies in processes, or gaps in technology utilization? Identifying these pain points provides a clear roadmap for how technology can be strategically employed to address immediate concerns and lay the foundation for future growth.

Understanding the client's needs is not a one-time event; it's an ongoing process. Maintain continuous engagement throughout the roadmap development. Regular follow-ups, feedback sessions, and updates ensure that the evolving needs of the client are always in focus. This proactive approach helps in adapting the roadmap to changing circumstances and ensures that it remains a dynamic tool rather than a static document.

Conducting a Comprehensive Assessment

Rule #2: Setting up a plan on unknowns will lead to a complete disaster, ever-growing costs, and endless discussions. Measure twice and cut once.

The second crucial phase in crafting a successful strategic IT roadmap is conducting a comprehensive assessment. This isn't merely a surface-level scan of the existing IT infrastructure, but a meticulous exploration that delves into every nook and cranny of the organization's technological landscape. This thorough assessment lays the groundwork for informed decision-making, identifying both strengths and weaknesses, opportunities, and potential threats.

Although every project is different, a comprehensive assessment begins with a detailed analysis of the current IT infrastructure. This involves scrutinizing hardware, software, networks, and data storage systems. Consultants meticulously map out the existing technology ecosystem, ensuring a holistic understanding of the organization's digital footprint. Are there outdated systems causing bottlenecks? Is the network architecture scalable? These are the questions that guide the initial stages of the assessment.

The assessment, however, should be taken only for the related infrastructure. By the end of the day, all clients would like to pay for what it asked for. And only for that. On the other hand, it is unavoidable to assess services that are not affected by the final roadmap. Make notes about all the nitty-gritty details of the infrastructure you discovered, and hand them over to the client by the end of the project (or as you are progressing) as gratis. For you, it costs nothing, for the client it may carry a valuable insight on their systems.

Setting Clear Objectives

Rule #3: Find clear, objective, and measurable goals. Otherwise, it will be a never ending jurney. The “there will always be something” strategy is not a strategy.

To ensure success, it is crucial to establish clear and achievable objectives for the IT roadmap. Defining specific milestones and key performance indicators (KPIs) helps in tracking progress and keeping the project on course. Clarity in objectives is the roadmap's North Star, guiding the organization toward its technology-driven future.

The collaboration must take place hands in hand with stakeholders to ensure that these objectives are not only aligned with the overall business goals, but are also realistic and measurable. It is a dynamic and ongoing dialogue between the consultant and key stakeholders. Also, depending on the client’s business, they may change their priorities and want to cross-out some KPIs or add new ones. Knowing about these early can result in the roadmap’s success or failure.

Open communication channels are vital to promptly address any changes in the client's business landscape, allowing for the modification of objectives as needed. By fostering this continuous collaboration, consultants can create an agile and responsive environment that positions the IT roadmap for sustained success.

Prioritizing Initiatives

Rule #4: Assign priorities to the projects on the roadmap, so if something hits the fan, you know where to cut loose ends.

In the realm of IT initiatives, not all endeavors are born equal, underscoring the critical role of a consultant's expertise.

The prioritization process is a nuanced task, demanding a keen understanding of the organizational landscape. Consultants employ a strategic approach, categorizing projects into high, medium, or low priority based on their direct impact on overarching goals. Through a meticulous evaluation of each initiative against predefined criteria, consultants provide organizations with a roadmap for resource allocation that aligns seamlessly with their strategic objectives.

This strategic prioritization ensures an efficient use of resources, maximizing value and propelling the organization toward its desired outcomes with accelerated progress. In essence, the consultant's ability to discern the varying significance of IT initiatives and strategically prioritize them plays a pivotal role in optimizing the overall impact and success of the organizational roadmap.

Last but not least, if the scope must be cut, both the stakeholders and the consultant can pinpoint the least important or too time-consuming tasks or projects.

Building Flexibility Into the Roadmap

Rule #5: Anything that can go wrong will go wrong. This applies to roadmaps as well, not just your kitchen appliances.

As mentioned earlier, due to the high chance that something will change during the execution, flexibility is non-negotiable.

An agile approach allows for swift adjustments in response to market changes or internal requirements. Consultants need to ensure that the IT roadmap is not a rigid plan, but a dynamic framework that can adapt to evolving circumstances. This adaptability is crucial for resilience in the face of uncertainties, ensuring that the organization can navigate through challenges and seize emerging opportunities.

However, keep in mind that no plan is bulletproof. There are certain situations, changes, or circumstances that can turn a rock-solid plan into a hazy vision.

Communicating Effectively

Rule #6: Most projects goes wrong on communication. Let’s put the telephone game aside.

Effective communication is the linchpin of successful IT roadmap implementation. As stressed out many times, it involves keeping stakeholders informed of progress, changes, and challenges through open dialogue and transparency.

Building trust and securing buy-in from key players are vital for the roadmap's smooth execution. Consultants act as the bridge between technical complexities and non-technical stakeholders, translating jargon into clear, actionable insights. This communication strategy fosters a collaborative environment where everyone is on the same page, working towards shared goals.

Monitoring and Evaluating Progress

Rule #7: The entire roadmap is based on intuition and leads to second-guessing, if execution is not closely monitored. Measuring the progress is as important as assessing the infrastructure.

Regularly monitoring and evaluating progress is another cornerstone of successful IT roadmap implementation, ensuring that the journey stays aligned with the destination. Employing various tools and techniques, consultants establish a robust framework for ongoing assessment, feedback, and refinement. Some of these with the lack of completeness are:

• Utilizing advanced project management software facilitates real-time tracking of tasks, timelines, and milestones.

• Strategic selection and continuous monitoring of KPIs are instrumental in gauging progress.

• Embracing agile methodologies allows for iterative development and continuous improvement.

• Scheduled review meetings provide a dedicated forum for comprehensive assessments.

• Implementing risk management strategies is integral to progress monitoring.

Although every roadmap is different and unique, in a nutshell, these techniques and methods can keep everything on track.

Conclusion

In conclusion, navigating the intricate terrain of IT roadmap development requires a blend of strategic acumen and adaptive prowess. As technology becomes an indispensable catalyst for organizational growth, consultants play a pivotal role in steering businesses toward success through meticulously crafted IT initiatives.

Understanding the client's needs forms the bedrock of this journey, demanding ongoing engagement and an open channel for dialogue. A comprehensive assessment, delving into every facet of the organization's technological landscape, sets the stage for informed decision-making. Clear and achievable objectives, collaboratively defined with stakeholders, act as the North Star guiding the organization's technological evolution.

The prioritization of initiatives, undertaken with nuanced expertise, ensures optimal resource allocation, maximizing impact. Building flexibility into the roadmap acknowledges the dynamic nature of business landscapes, allowing for agile responses to unforeseen challenges and opportunities. Effective communication, a bridge between technical intricacies and non-technical stakeholders, fosters collaboration and trust.

Finally, the continuous monitoring and evaluation of progress, utilizing tools, methodologies, and strategic reviews, culminate in a roadmap that not only adapts but thrives in the ever-evolving realm of technology. In essence, the success of an IT roadmap lies in its ability to be both a dynamic framework and a strategic guide in an era where change is the only constant.

Subscribe now

Time-series data is crucial for IoT device monitoring and data visualization in industries such as agriculture, renewable energy, and meteorology. It enables trend analysis, anomaly detection, and predictive analytics, empowering
businesses to optimize performance and make data-driven decisions. Thanks to technological advancements and the accessibility of open-source tools, gathering and analyzing data from IoT devices has become easier than ever before.

In this tutorial, we will guide you through the process of setting up a monitoring system for IoT device data. We will use historical electricity consumption data from some European countries, captured at a 15-minute resolution. The data is sent to an MQTT-compatible message broker called Mosquitto, and then channeled to QuestDB through Telegraf, a highly efficient data collector. To visualize the results, we will connect Grafana to QuestDB using its Postgres interface.

Since we don't have our own sensors to collect the data and send it directly to the broker, we will create the scenario by having a script that gathers the electricity consumption data from Open Power System Data and sends it to Mosquitto.

Prerequisites

Before delving into the system setup, please make sure that you have installed the following:

• Docker

• The latest Go version

Setting up the prerequisites

To get started, clone the prepared GitHub repository:

$ git clone https://github.com/questdb/questdb-mock-power-sensor mock-sensor

This repository contains all the configuration files and mock sensor scripts we will need during the tutorial.

Furthermore, we are going to need a new Docker network named "tutorial" to enable communication between containers without installing additional software on the host system. To create the new network, execute the following command:

$ docker network create tutorial

Setting up an MQTT broker

To enable communication between IoT devices and the monitoring system, we need a message broker such as Eclipse Mosquitto, which implements the lightweight MQTT protocol. By using Eclipse Mosquitto, we establish a reliable and efficient communication channel for IoT devices.

To run Eclipse Mosquitto, we are going to use Docker. The default configuration for Mosquitto does not allow unauthenticated clients. However, no default credentials are set. To fix this, let's use the conf/mosquitto/mosquitto.conf to override the default settings:

allow_anonymous true
listener 1883 0.0.0.0

To start the Eclipse Mosquitto broker using Docker, you can execute the following command from the repository root:

$ docker run --rm -dit -p 1883:1883 -p 9001:9001 \
 -v "$(pwd)"/conf/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf \
 --network=tutorial --name=mosquitto eclipse-mosquitto

By running this command, the Eclipse Mosquitto broker will start within a Docker container, configured with the settings from the mosquitto.conf file. The specified ports will be exposed to enable MQTT communication and the container will be connected to the tutorial Docker network, allowing it to communicate with other containers in the network.

Tunneling data into Mosquitto

Now that the message broker is running, in a new terminal window, navigate to the repository root and start the script that will populate the data for us:

$ cd script
$ go get
$ go run ./main.go

Great! The script will continue running in the background until it is manually stopped or until it runs out of data to publish. Since we are using a historical dataset, the script will override the original timestamps with the current time when sending the records to the MQTT broker.

With a 1-second delay between records, the script will consistently publish data to the broker, ensuring a steady flow of simulated sensor data for your tutorial. This allows you to progress with the rest of the tutorial without worrying about starting or stopping the script manually.

Setting up QuestDB

To ensure that all incoming IoT data flowing through Mosquitto is stored in QuestDB by the end of this tutorial, let's start QuestDB in the background. This will allow us to connect to it later using Telegraf.

Execute the following Docker command to start QuestDB:

$ docker run --rm -dit -p 8812:8812 -p 9000:9000 \
    -p 9009:9009 -e QDB_PG_READONLY_USER_ENABLED=true \
    --network=tutorial --name=questdb questdb/questdb

Once the command is executed, QuestDB will start running in the background. To validate that QuestDB is up and running, you can visit http://localhost:9000 in your web browser.

By accessing the provided URL, you should be able to access the QuestDB web interface, indicating that QuestDB has been successfully started.

Connecting the dots and adding Telegraf

In this setup, Telegraf will play a key role in transferring data between Mosquitto and QuestDB. To enable this seamless data transfer, we will utilize QuestDB's ILP (Influx Line Protocol) interface, as ILP is capable of handling large volumes of data efficiently sent from Telegraf.

To configure Telegraf, we are using the conf/telegraf/telegraf.conf file, which has the following content:

# Configuration for Telegraf agent
[agent]
  ## Default data collection interval for all inputs
  interval = "1s"

[[inputs.mqtt_consumer]]
  servers = ["tcp://mosquitto:1883"]
  topics = ["sensor"]
  data_format = "influx"
  client_id = "telegraf"
  data_type = "string"

# Write results to QuestDB
[[outputs.socket_writer]]
  # Write metrics to a local QuestDB instance over TCP
  address = "tcp://questdb:9009"

Let’s start the Telegraf container by executing:

$ docker run --rm -it \
    -v "$(pwd)"/conf/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf \
    --network=tutorial --name=telegraf telegraf

Once Telegraf is up and running, the sensor data is automatically tunneled into QuestDB, and a new table called "sensor" is created to store the incoming data. To verify that the data is successfully flowing into QuestDB, you can execute the following query on the QuestDB console, accessible at http://localhost:9000:

SELECT * FROM sensor;

In the next steps of the tutorial, we will set up Grafana, which is a powerful data visualization and monitoring tool. We will connect Grafana to QuestDB and create compelling visualizations based on the received sensor data.

Visualizing the dataset with Grafana

To proceed with creating a Grafana container and connecting it to QuestDB, execute the following command:

$ docker run --rm -dit -p 3000:3000 \
    --network=tutorial --name=grafana grafana/grafana

This command will start a Grafana container within the same tutorial network. To access the Grafana login page, navigate to http://localhost:3000 in your web browser. You will be prompted to log in using the default credentials:

• Username: admin

• Password: admin

To connect Grafana with QuestDB and establish a data source, follow these steps:

1. Open your web browser and go to http://localhost:3000/datasources/new.

2. On the "New data source" page, select "PostgreSQL" as the data source type from the list of available options.

Fill in the form with the following details:

• Name: QuestDB

• Host: questdb:8812

• Database: qdb

• User: user

• Password: quest

• TLS/SSL mode: disable

Once you have filled in the form with the correct information, click on the "Save & Test" button at the bottom of the page. Grafana will attempt to connect to QuestDB using the provided credentials and verify the connection. If everything is set up correctly, you should see a success message indicating that the data source was added successfully.

Now that Grafana is successfully connected to QuestDB, you can begin creating insightful dashboards to visualize the electricity consumption data. Follow these steps to create a new dashboard:

1. In your browser and navigate to the new dashboard page.

2. Once you're on the new dashboard page, click on the "Add visualization" button.

On the popup panel, select “QuestDB” data source.

Now you're ready to start building your dashboard and visualizing the electricity consumption data from QuestDB. Grafana provides a wide range of visualization options, including graphs, charts, tables, and more. You can customize the panel settings, apply different visualization types, and configure data queries to create meaningful visual representations of the data.

To visualize the electricity consumption data from QuestDB, we will use the "Time series" chart type, which is the default selected chart based on our data. The "Time series" chart is ideal for displaying data over time and is well-suited for analyzing trends and patterns.

In the query builder panel, switch to “Code” and paste the following SQL query:

SELECT
    date_trunc('minute', timestamp) AS minute,
    AVG(load_actual) AS avg_load_actual,
    AVG(load_forecast) AS avg_load_forecast
FROM sensor
WHERE country = 'NL'
GROUP BY minute
ORDER BY minute ASC

By executing this query, we will obtain a result set that provides minutely aggregated data, showing the average, actual, and forecasted energy load for each minute. Feel free to adjust the sampling or grouping interval if you wish. If you followed this tutorial successfully, you should see similar results.

To make the changes persistent while the container is running, rename the title of the graph on the right-hand side properties pane, and save it by clicking on the “Apply” button in the top right corner.

The time series chart we have created, with the average actual and forecasted energy consumption, provides valuable insights into the accuracy of the forecasting logic. By analyzing this chart, we can easily identify any discrepancies between the actual and forecasted values, helping you pinpoint areas where improvements can be made.

Cleaning up resources

Upon finishing the tutorial, you may want to clean up any dangling resources, such as the tutorial network we created. Removing a network requires the removal of existing resources attached to that first. To remove the containers, the network, and the images used during this tutorial, run the following commands:

$ docker ps -qa --filter="network=tutorial" | xargs -n1 docker kill
$ docker network rm tutorial
$ docker rmi telegraf eclipse-mosquitto grafana/grafana questdb/questdb

Summary

In this tutorial, we explored the setup and configuration of a monitoring system for IoT device data using MQTT, Telegraf, QuestDB, and Grafana. Through a series of steps, we established communication between IoT devices and the monitoring system using Eclipse-Mosquitto as the MQTT broker.

We simulated data collection using a script that gathered electricity consumption from Open Power System Data and sent it to the MQTT broker. We stored this data in QuestDB and visualized it using Grafana. By connecting Grafana to QuestDB, we created informative dashboards to gain insights.

The tutorial showcased the power of time-series data and its visualization, enabling us to compare actual and forecasted energy consumption.

Learn how to improve your time series analysis capability by using the QuestDB Python package to ingest Pandas DataFrames.

Introduction

Pandas is an open-source data analysis and data manipulation library for Python that has become an essential tool for data scientists and analysts. It provides a simple and intuitive way to manipulate data, making it a popular choice for data analysis tasks. However, while Pandas is excellent for small to medium-sized datasets, it can struggle with large datasets that exceed the available memory of the machine it is running on. This is where QuestDB excels, designed specifically for high-performance operations in such scenarios, making it the go-to solution for demanding data analysis tasks. By loading Pandas DataFrames into QuestDB, we can leverage powerful data processing capabilities of the database, allowing you to scale your analysis and data manipulation operations to large datasets. We will learn how to load large Pandas dataframes into QuestDB. We use yellow and green taxi trip records published by NYC Taxi & Limousine Commission as our data source.

In this tutorial, we will learn how to load large Pandas dataframes into QuestDB. We use yellow and green taxi trip records published by NYC Taxi & Limousine Commission as our data source.

Prerequisites

For this tutorial, it is recommended to have a basic understanding of Python, and SQL. Also, you will need to have the following installed on your machine:

• Docker

Getting the data to ingest

Before we begin loading the data into QuestDB, we need to obtain the data that we will be working with. As mentioned above, we will use the NYC TLC’s records of yellow and green taxi trips. Let’s download the data:

1. Create a new directory called pandas-to-questdb and a data directory inside that.

2. Edit and execute the following command in your terminal to download the parquet files:

curl -L -o ./data/yellow_tripdata_2022-<MONTH>.parquet https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-<MONTH>.parquet

Make sure you replace <MONTH> with the zero-prefixed number of the month you wish to download (between 01 and 11, the 12th month is not available at the time of writing).

Now, we have the data to ingest. It is time to try loading it using Pandas.

Loading records into memory

You may already have noticed that the downloaded files are in Parquet format. Parquet is a columnar storage format commonly used for big data processing. They are optimized for use with modern big data processing engines and provide efficient storage and retrieval of data compared to traditional row-based storage formats like CSV and JSON.

Before being able to load any data, we are going to set up a simulation production environment in which we can easily test what happens if Pandas cannot load the Parquet files into memory. In production, we often meet situations where we have to deal with memory constraints, and this environment can reflect that.

Run the following command to create a new docker container running with 1GiB memory limit. If the container reaches that limit, either docker will kill it, or the OS will OOM kill the process we are running.

docker run -it -m 1g -v "$(pwd)":/tutorial -w /tutorial --net host python:3.11.1-slim-bullseye /bin/bash

Right, we have an Ubuntu-based Python 3.11 docker container. Let’s install our requirements. Create a requirements.txt file with the content below:

pandas>=1.5.3
psycopg[binary]>=3.1.8
pyarrow>=11.0.0
questdb>=1.1.0

Now, execute pip install -r requirements.txt within the container. Pip will install the python requirements.

At this point we have a test environment in which we can load the data. Create a new file, called data_loader.py, with the following content:

# data_loader.py

import pandas as pd

df = pd.read_parquet("./data/yellow_tripdata_2022-01.parquet")
print(df.head())

Now, execute it within the docker container by running python data_loader.py. The program runs successfully and we should see the following:

We just loaded the taxi trip records for 2022 January! Let’s try to load more data. Replace the content of data_loader.py by the code below to load all files from the data directory and execute the program again:

# data_loader.py

import os
import glob
import pandas as pd

records = glob.glob(os.path.join("data", "*.parquet"))

df = pd.concat((pd.read_parquet(r) for r in records), ignore_index=True)

print(df.head())

When executing the data_loader.py you should get an error message: “Killed”. As you may assume, OOM killer terminated the process. We were not able to load the dataset, therefore we cannot work with that. We need a different approach.

Ingesting to QuestDB

In a new terminal window, start a QuestDB container by executing:

docker run --rm -it -p 8812:8812 -p 9009:9009 --net host --name questdb questdb/questdb

The database is now ready to receive the data. Update the data_loader.py to ingest data into QuestDB using the questdb package that uses the InfluxDB Line Protocol (ILP) over TCP for maximum throughput.

To handle large datasets, we will read files one by one and transfer their contents to QuestDB. Then, we will use QuestDB to query the data and load the results back into Pandas DataFrames. Refactor the data loader based on the above:

# data_loader.py

import os
import glob
import pandas as pd
from questdb.ingress import Sender

def main():
   files = glob.glob(os.path.join("data", "*.parquet"))

   with Sender("127.0.0.1", 9009) as sender:
       for file in files:
           df = pd.read_parquet(file)
           print(f"ingesting {len(df.index)} rows from {file}")
           sender.dataframe(df, table_name="trips", at="tpep_pickup_datetime")

if __name__ == "__main__":
   main()

Let's start from the beginning. The first major change you'll notice is that we need to specify the hostname and port number in the script in order to run it.

Then we iterate over the parquet files and load them into memory using Pandas. After that, utilizing QuestDB’s Python client, we are ingesting to QuestDB directly from Pandas DataFrames.

In the Python container, run python data_loader.py. The script will ingest one parquet file at a time.

Working with trip data

So far, we have prepared the dataset and loaded it into QuestDB. It’s time to execute some queries and load the result into DataFrames. Using the whole dataset, we want to know what was the average total amount paid by passengers grouped by the passengers.

Create a new file, called query_amount.py with the following content:

# query_amount.py

import pandas as pd
import psycopg

QUERY = """
SELECT passenger_count, avg(total_amount)
   FROM 'trips'
   WHERE passenger_count > 0
   GROUP BY passenger_count
"""

if __name__ == "__main__":
   conn = psycopg.connect(
       dbname="questdb",
       host="127.0.0.1",
       user="admin",
       password="quest",
       port=8812,
   )

   df = pd.read_sql_query(QUERY, conn)

   print(df.head(10))

Similarly to the data loader script, this script requires the host and port too.

In the script above, we are using the Postgresql Python client and connecting to QuestDB using that. In the Python container, execute python query_amount.py:

When the script finishes, you should see the average total amount paid by passengers. Interestingly, there is a huge difference in the average between passenger count 6 and 7, the average almost 2.5x for 7 passengers compared to 6.

By further analyzing the data, it may turn out what was the possible root cause increase, but probably it is bound to human nature: we like to share the cost of rides if we are going on a longer trip.

Summary

In this tutorial, we have learned how to load large datasets into QuestDB using Pandas DataFrames. By transferring data from Pandas to QuestDB, we have taken advantage of the database's powerful data processing capabilities, enabling us to scale our analysis and data manipulation operations to handle large datasets.

The approach outlined in this tutorial is just one way to work with big data using Pandas and QuestDB. You can customize this method to suit your specific needs and continue to explore the possibilities of these powerful tools. The end goal is to make data analysis and manipulation easier and more efficient, regardless of the size of the dataset.

If you're working with large amounts of data, efficiently storing raw information will be your first obstacle. The next challenge is to make sense of the data utilizing analytics. One of the fastest ways to convey the state of data is through charts and graphs.

In this tutorial, we will create a real-time streaming dashboard using QuestDB, Celery, Redis, Plotly, and Dash. It will be a fun project with excellent charts to quickly understand the state of a system with beautiful data visualizations.

Plotly defines itself as "the front end for ML and data science models", which describes it really well. Plotly has an "app framework" called Dash which we can use to create web applications quickly and efficiently. Dash abstracts away the boilerplate needed to set up a web server and several handlers for it.

Project overview

The project will be built from two main components:

• a backend that periodically fetches user-defined stock data from Finnhub, and

• a front-end that utilizes Plotly and Dash to visualize the gathered data on interactive charts

For this tutorial, you will need some experience in Python and basic SQL knowledge. We will use Celery backed by Redis as the message broker and QuestDB as storage to periodically fetch data.

Let's see the prerequisites and jump right in!

Prerequisites

• Python 3.8

• Docker & Docker Compose

• Finnhub account and sandbox API key

• Basic SQL skills

The source code for this tutorial is available at the corresponding GitHub repository.

Environment setup

Create a new project

First of all, we are going to create empty directories for our project root and the Python module:

mkdir -p streaming-dashboard/app
# streaming-dashboard
# └── app

Installing QuestDB & Redis

To install the services required for our project, we are using Docker and Docker Compose to avoid polluting our host machine. Within the project root, let's create a file, called docker-compose.yml. This file describes all the necessary requirements the project will use; later on we will extend this file with other services too.

version: "3"

volumes:
  questdb_data: {}

services:
  redis:
    image: "redis:latest"
    ports:
      - "6379:6379"

  questdb:
    image: "questdb/questdb:latest"
    volumes:
      - questdb_data:/root/.questdb/db
    ports:
      - "9000:9000"
      - "8812:8812"

Here we go! When you run docker-compose up, QuestDB and Redis will fire up. After starting the services, we can access QuestDB's interactive console on http://127.0.0.1:9000.

Create the database table

We could create the database table later, but we will take this opportunity and create the table now since we have already started QuestDB. Connect to QuestDB's interactive console, and run the following SQL statement:

CREATE TABLE
      quotes(stock_symbol SYMBOL CAPACITY 5 CACHE INDEX, -- we are in fact just checking 3
             current_price DOUBLE,
             high_price DOUBLE,
             low_price DOUBLE,
             open_price DOUBLE,
             percent_change DOUBLE,
             tradets TIMESTAMP, -- timestamp of the trade
             ts TIMESTAMP)      -- time of insert in our table
      timestamp(ts)
PARTITION BY DAY;

After executing the command, we will see a success message in the bottom left corner, confirming that the table creation was successful and the table appears on the right-hand side's table list view.

Voilá! The table is ready for use.

Creating workers using Celery

Define Python dependencies

As mentioned, our project will have two parts. For now, let's focus on the routine jobs that will fetch the data from Finnhub. As is the case of every standard Python project, we are using requirements.txt to define the dependencies the project will use. Place the requirements.txt in your project root with the content below:

finnhub-python==2.4.5   # The official Finnhub Python client
pydantic[dotenv]==1.8.2 # We will use Pydantic to create data models
celery[redis]==5.1.2    # Celery will be the periodic task executor
psycopg2==2.9.1         # We are using QuestDB's PostgreSQL connector
sqlalchemy==1.4.2       # SQLAlchemy will help us executing SQL queries
dash==2.0.0             # Dash is used for building data apps
pandas==1.3.4           # Pandas will handle the data frames from QuestDB
plotly==5.3.1           # Plotly will help us with beautiful charts

We can split the requirements into two logical groups:

1. requirements for fetching the data, and

2. requirements needed to visualize this data

For the sake of simplicity, we did not create two separate requirements files, though in a production environment we would do. Create a virtualenv and install the dependencies:

$ virtualenv -p python3.8 virtualenv
$ source virtualenv/bin/activate
$ pip install -r requirements.txt

Setting up the DB connection

Since the periodic tasks would need to store the fetched quotes, we need to connect to QuestDB. Therefore, we create a new file in the app package, called db.py. This file contains the SQLAlchemy engine that will serve as the base for our connections.

from sqlalchemy import create_engine

from app.settings import settings

engine = create_engine(
    settings.database_url, pool_size=settings.database_pool_size, pool_pre_ping=True
)

Define the worker settings

Before we jump right into the implementation, we must configure Celery. To create a configuration used by both the workers and the dashboard, create a settings.py file in the app package. We will use pydantic's BaseSettings to define the configuration. This helps us to read the settings from a .env file, environment variable, and prefix them if needed.

Ensuring that we do not overwrite any other environment variables, we will set the prefix to SMD that stands for "stock market dashboard", our application. Below you can see the settings file:

from typing import List

from pydantic import BaseSettings


class Settings(BaseSettings):
    """
    Settings of the application, used by workers and dashboard.
    """

    # Celery settings
    celery_broker: str = "redis://127.0.0.1:6379/0"

    # Database settings
    database_url: str = "postgresql://admin:quest@127.0.0.1:8812/qdb"
    database_pool_size: int = 3

    # Finnhub settings
    api_key: str = ""
    frequency: int = 5  # default stock data fetch frequency in seconds
    symbols: List[str] = list()

    # Dash/Plotly
    debug: bool = True
    graph_interval: int = 10

    class Config:
        """
        Meta configuration of the settings parser.
        """

        env_file = ".env"
        # Prefix the environment variable not to mix up with other variables
        # used by the OS or other software.
        env_prefix = "SMD_"  # SMD stands for Stock Market Dashboard


settings = Settings()

In the settings, you can notice we already defined the celery_broker and database_url settings with unusual default values.

Some bits are missing at the moment. We still have to define the correct settings and run the worker in a Docker container. Get started with the settings!

To keep our environment separated, we will use a .env file. One of pydantic based settings' most significant advantage is that it can read environment variables from .env files.

Let's create a .env file in the project root, next to docker-compose.yml:

SMD_API_KEY = "<YOUR SANDBOX API KEY>"
SMD_FREQUENCY = 10
SMD_SYMBOLS = ["AAPL","DOCN","EBAY"]

As you may assume, you will need to get your API key for the sandbox environment at this step. To retrieve the key, the only thing you have to do is sign up to Finnhub, and your API key will appear on the dashboard after login.

Create the periodic task

import finnhub
from celery import Celery
from sqlalchemy import text
from app.db import engine
from app.settings import settings

client = finnhub.Client(api_key=settings.api_key)
celery_app = Celery(broker=settings.celery_broker)

@celery_app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    """
    Setup a periodic task for every symbol defined in the settings.
    """
    for symbol in settings.symbols:
        sender.add_periodic_task(settings.frequency, fetch.s(symbol))


@celery_app.task
def fetch(symbol: str):
    """
    Fetch the stock info for a given symbol from Finnhub and load it into QuestDB.
    """

    quote: dict = client.quote(symbol)
    # https://finnhub.io/docs/api/quote
    #  quote = {'c': 148.96, 'd': -0.84, 'dp': -0.5607, 'h': 149.7, 'l': 147.8, 'o': 148.985, 'pc': 149.8, 't': 1635796803}
    # c: Current price
    # d: Change
    # dp: Percent change
    # h: High price of the day
    # l: Low price of the day
    # o: Open price of the day
    # pc: Previous close price
    # t: when it was traded
    query = f"""
    INSERT INTO quotes(stock_symbol, current_price, high_price, low_price, open_price, percent_change, tradets, ts)
    VALUES(
        '{symbol}',
        {quote["c"]},
        {quote["h"]},
        {quote["l"]},
        {quote["o"]},
        {quote["pc"]},
        {quote["t"]} * 1000000,
        systimestamp()
    );
    """

    with engine.connect() as conn:
        conn.execute(text(query))

Going through the code above:

import finnhub
from celery import Celery
from sqlalchemy import text

from app.db import engine
from app.settings import settings

# [...]

In the first few lines, we import the requirements that are needed to fetch and store the data.

After importing the requirements, we configure the Finnhub client and Celery to use the Redis broker we defined in the application settings.

# [...]

client = finnhub.Client(api_key=settings.api_key)
celery_app = Celery(broker=settings.celery_broker)

# [...]

To fetch the data periodically per stock symbol, we need to programmatically
create a periodic task for every symbol we defined in the settings.

# [...]

@celery_app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    """
    Setup a periodic task for every symbol defined in the settings.
    """
    for symbol in settings.symbols:
        sender.add_periodic_task(settings.frequency, fetch.s(symbol))

# [...]

The snippet above will register a new periodic per stock symbol after Celery is connected to the broker.

The last step is to define the fetch task that does the majority of the work.

# [...]

@celery_app.task
def fetch(symbol: str):
    """
    Fetch the stock info for a given symbol from Finnhub and load it into QuestDB.
    """

    quote: dict = client.quote(symbol)
    # https://finnhub.io/docs/api/quote
    #  quote = {'c': 148.96, 'd': -0.84, 'dp': -0.5607, 'h': 149.7, 'l': 147.8, 'o': 148.985, 'pc': 149.8, 't': 1635796803}
    # c: Current price
    # d: Change
    # dp: Percent change
    # h: High price of the day
    # l: Low price of the day
    # o: Open price of the day
    # pc: Previous close price
    # t: when it was traded

    query = f"""
    INSERT INTO quotes(stock_symbol, current_price, high_price, low_price, open_price, percent_change, tradets, ts)
    VALUES(
        '{symbol}',
        {quote["c"]},
        {quote["h"]},
        {quote["l"]},
        {quote["o"]},
        {quote["pc"]},
        {quote["t"]} * 1000000,
        systimestamp()
    );
    """

    with engine.connect() as conn:
        conn.execute(text(query))

Using the Finnhub client, we get a quote for the given symbol. After the quote is retrieved successfully, we prepare a SQL query to insert the quote into the database. At the end of the function, as the last step, we open a connection to QuestDB and insert the new quote.

Congratulations! The worker is ready for use; let's try it out!

Execute the command below in a new terminal window within the virtualenv, and wait some seconds to let Celery kick in:

python -m celery --app app.worker.celery_app worker --beat -l info -c 1

Soon, you will see that the tasks are scheduled, and the database is slowly
filling.

Checking in on what we've built so far

Before proceeding to the visualization steps, let's have a look at what we have built so far:

1. we created the project root

2. a docker-compose.yml file to manage related services

3. app/settings.py that handles our application configuration

4. app/db.py configuring the database engine, and

5. last but not least, app/worker.py that handles the hard work, fetches, and stores the data.

At this point, we should have the following project structure:

├── app
│   ├── __init__.py
│   ├── db.py
│   ├── settings.py
│   └── worker.py
└── docker-compose.yml

Visualize the data with Plotly and Dash

Getting static assets

This tutorial is not about writing the necessary style sheets or collecting static assets, so you only need to copy-paste some code. As the first step, create an assets directory next to the app package with the structure below:

├── app
│   ├── __init__.py
│   ├── db.py
│   ├── settings.py
│   └── worker.py
├── assets
├── .env
├── docker-compose.yml

The style.css will define the styling for our application. As mentioned above, Dash will save us from boilerplate code, so the assets directory will be used by default in conjunction with the stylesheet in it.

Download the style.css file to the assets directory, this can be done using curl:

curl -s -Lo ./assets/style.css https://raw.githubusercontent.com/gabor-boros/questdb-stock-market-dashboard/main/assets/style.css

Setting up the application

This is the most interesting part of the tutorial. We are going to visualize the data we collect. Create a main.py file in the app package, and let's begin with the imports:

from datetime import datetime, timedelta

import dash
import pandas
from dash import dcc, html
from dash.dependencies import Input, Output
from plotly import graph_objects

from app.db import engine
from app.settings import settings

# [...]

After having the imports in place, we are defining some helper functions and constants.

# [...]

GRAPH_INTERVAL = settings.graph_interval * 1000

TIME_DELTA = 5  # last T hours of data are looked into as per insert time

COLORS = [
    "#1e88e5",
    "#7cb342",
    "#fbc02d",
    "#ab47bc",
    "#26a69a",
    "#5d8aa8",
]


def now() -> datetime:
    return datetime.utcnow()


def get_stock_data(start: datetime, end: datetime, stock_symbol: str):
    def format_date(dt: datetime) -> str:
        return dt.isoformat(timespec="microseconds") + "Z"

    query = f"quotes WHERE ts BETWEEN '{format_date(start)}' AND '{format_date(end)}'"

    if stock_symbol:
        query += f" AND stock_symbol = '{stock_symbol}' "

    with engine.connect() as conn:
        return pandas.read_sql_query(query, conn)

# [...]

In the first few lines, we define constants for setting a graph update frequency (GRAPH_INTERVAL) and colors that will be used for coloring the graph (COLORS).

After that, we define two helper functions, now and get_stock_data. While now is responsible only for getting the current time in UTC (as Finnhub returns the date in UTC too), the get_stock_data does more. It is the core of
our front-end application, it fetches the stock data from QuestDB that workers inserted.

Define the initial data frame and the application:

# [...]

df = get_stock_data(now() - timedelta(hours=TIME_DELTA), now(), "")

app = dash.Dash(
    __name__,
    title="Real-time stock market changes",
    assets_folder="../assets",
    meta_tags=[{"name": "viewport", "content": "width=device-width, initial-scale=1"}],
)

# [...]

As you can see above, the initial data frame (df) will contain the latest 5 hours of data we have. This is needed to pre-populate the application with some data we have. The application definition app describes the application's title, asset folder, and some HTML meta tags used during rendering.

Create the application layout that will be rendered as HTML. We won't write HTML, we will use Dash's helpers for that:

# [...]

app.layout = html.Div(
    [
        html.Div(
            [
                html.Div(
                    [
                        html.H4("Stock market changes", className="app__header__title"),
                        html.P(
                            "Continually query QuestDB and display live changes of the specified stocks.",
                            className="app__header__subtitle",
                        ),
                    ],
                    className="app__header__desc",
                ),
            ],
            className="app__header",
        ),
        html.Div(
            [
                html.P("Select a stock symbol"),
                dcc.Dropdown(
                    id="stock-symbol",
                    searchable=True,
                    options=[
                        {"label": symbol, "value": symbol}
                        for symbol in df["stock_symbol"].unique()
                    ],
                ),
            ],
            className="app__selector",
        ),
        html.Div(
            [
                html.Div(
                    [
                        html.Div(
                            [html.H6("Current price changes", className="graph__title")]
                        ),
                        dcc.Graph(id="stock-graph"),
                    ],
                    className="one-half column",
                ),
                html.Div(
                    [
                        html.Div(
                            [html.H6("Percent changes", className="graph__title")]
                        ),
                        dcc.Graph(id="stock-graph-percent-change"),
                    ],
                    className="one-half column",
                ),
            ],
            className="app__content",
        ),
        dcc.Interval(
            id="stock-graph-update",
            interval=int(GRAPH_INTERVAL),
            n_intervals=0,
        ),
    ],
    className="app__container",
)

# [...]

This snippet is a bit longer, though it has only one interesting part, dcc.Interval. The interval is used to set up periodic graph refresh.

We are nearly finished with our application, but the last steps are to define two callbacks that will listen to input changes and the interval discussed above. The first callback is for generating the graph data and rendering the
lines per stock symbol.

# [...]

@app.callback(
    Output("stock-graph", "figure"),
    [Input("stock-symbol", "value"), Input("stock-graph-update", "n_intervals")],
)
def generate_stock_graph(selected_symbol, _):
    data = []
    filtered_df = get_stock_data(now() - timedelta(hours=TIME_DELTA), now(), selected_symbol)
    groups = filtered_df.groupby(by="stock_symbol")

    for group, data_frame in groups:
        data_frame = data_frame.sort_values(by=["ts"])
        trace = graph_objects.Scatter(
            x=data_frame.ts.tolist(),
            y=data_frame.current_price.tolist(),
            marker=dict(color=COLORS[len(data)]),
            name=group,
        )
        data.append(trace)

    layout = graph_objects.Layout(
        xaxis={"title": "Time"},
        yaxis={"title": "Price"},
        margin={"l": 70, "b": 70, "t": 70, "r": 70},
        hovermode="closest",
        plot_bgcolor="#282a36",
        paper_bgcolor="#282a36",
        font={"color": "#aaa"},
    )

    figure = graph_objects.Figure(data=data, layout=layout)
    return figure

# [...]

The other callback is very similar to the previous one; it will be responsible for updating the percentage change representation of the stocks or a given
stock.

# [...]

@app.callback(
    Output("stock-graph-percent-change", "figure"),
    [
        Input("stock-symbol", "value"),
        Input("stock-graph-update", "n_intervals"),
    ],
)
def generate_stock_graph_percentage(selected_symbol, _):
    data = []
    filtered_df = get_stock_data(now() - timedelta(hours=TIME_DELTA), now(), selected_symbol)
    groups = filtered_df.groupby(by="stock_symbol")

    for group, data_frame in groups:
        data_frame = data_frame.sort_values(by=["ts"])
        trace = graph_objects.Scatter(
            x=data_frame.ts.tolist(),
            y=data_frame.percent_change.tolist(),
            marker=dict(color=COLORS[len(data)]),
            name=group,
        )
        data.append(trace)

    layout = graph_objects.Layout(
        xaxis={"title": "Time"},
        yaxis={"title": "Percent change"},
        margin={"l": 70, "b": 70, "t": 70, "r": 70},
        hovermode="closest",
        plot_bgcolor="#282a36",
        paper_bgcolor="#282a36",
        font={"color": "#aaa"},
    )

    figure = graph_objects.Figure(data=data, layout=layout)
    return figure

# [...]

The last step is to call run_server on the app object when the script is called from the CLI.

# [...]

if __name__ == "__main__":
    app.run_server(host="0.0.0.0", debug=settings.debug)

We are now ready to try our application with actual data. Make sure that the Docker containers are started and execute PYTHONPATH=. python app/main.py from the project root:

$ PYTHONPATH=. python app/main.py

Dash is running on http://0.0.0.0:8050/

 * Tip: There are .env or .flaskenv files present. Do "pip install python-dotenv" to use them.
 * Serving Flask app 'main' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://192.168.0.14:8050/ (Press CTRL+C to quit)

Navigate to http://127.0.0.1:8050/, to see the application in action.

To select only one stock, in the dropdown field choose the desired stock symbol and let the application refresh.

Summary

In this tutorial, we've learned how to schedule tasks in Python, store data in QuestDB, and create beautiful dashboards using Plotly and Dash. Although we won't start trading just right now; this tutorial demonstrated well how to combine these separately powerful tools and software to create something bigger and more useful.

The source code is available at
https://github.com/gabor-boros/questdb-stock-market-dashboard.

In the world of big data, software developers and data analysts often have to write scripts or complex software collections to process data before sending it to a data store for further analysis. This process is commonly called ETL, which stands for Extract, Transform and Load.

What could we use ETL jobs for?

Let's consider the following example: a medium-sized webshop with a few thousand orders per day exports order information hourly. After a while, we would like to visualize purchase trends, and we might want to share the results between departments or even publicly. Since the exported data contains personally identifiable information (PII), we should anonymize it before using or exposing it to the public.

For the example above, we can use an ETL job to extract the incoming data, remove any PII and load the transformed data into a database used as the data visualization backend later.

Prerequisites

During this tutorial, we will use Python to write the cloud functions, so basic python knowledge is essential. Aside from these skills, you will need the following resources:

• A Google Cloud Platform (GCP) account and a GCP Project.

• Enable the Cloud Build API - when enabling APIs ensure that the correct GCP project is selected.

Creating an ETL job

As an intermediate datastore where the webshop exports the data, we will use Google Storage and use Google Cloud Functions to transform it before loading it into QuestDB.

We won't be building a webshop or a data exporter for an existing webshop, but we will use a script to simulate the export to Google Storage.

In the following sections, we will set up the necessary components on GCP. Ensure the required APIs mentioned in the prerequisites are enabled, and that you have selected the GCP project in which you would like to create the tutorial resources.

Create a Compute Engine instance for QuestDB

First things first, we start with installing QuestDB on a virtual machine. To get started, navigate to the Compute Engine console. Visiting this page for the first time will take a few moments to initialize. After the loading indicator has gone, start a new virtual machine:

1. Click on "create" and give the instance the name questdb-vm

2. Select a region close to you

3. Select the first generation "N1" series

4. Choose the f1-micro machine type - in a production environment you would choose a more performant instance, but for tutorial purposes, this is enough

5. In the "Container section, check "Deploy a container image to this VM instance", and type "questdb/questdb:latest" for the "Container image"

6. In the "Firewall" section, click on "Management, security, disks, networking, sole tenancy"

7. In the newly appeared panel, select "Networking" and add questdb as a "Network tag"

8. Leave all other settings with their defaults, and click on "create"

Make sure you note the "External IP" of the instance as we will need that later.

After a short time, the new instance will be up and running. As soon as the instance is provisioned, we can initiate a remote session to install QuestDB by clicking ssh in the VM panel.

Allow networking on the instance

If we try to open the web console by opening the http://<EXTERNAL_IP>:9000 (where <EXTERNAL_IP> is the external IP of your virtual machine) it won't load and we will face a timeout. The reason behind this is that the firewall is not opened for port 9000 yet.

To allow port 9000 used by QuestDB, we must allow the port by adding a new firewall rule on the firewall console:

1. Click on "create firewall rule" at the top of the page

2. Give the rule the name "QuestDBPorts"

3. In the "Target tags" field, write the same tag used for instance creation (questdb)

4. For the "Source IP ranges" field, set 0.0.0.0/0

5. In the "Protocols and ports" section, select tcp and set port to 9000,8812

6. Click on "create"

Some seconds later, the rule will be applied on every instance with the matching questdb tag and port 9000 will be open. You may ask what port 8812 is for; this port will be used by the Cloud Function later to connect to the database.

If you try to open the interactive console again, you should see the QuestDB Web Console and start writing queries.

As our first query, create the table in which the Cloud Function will write the anonymized data. To create the table run the following SQL statement:

CREATE TABLE
    purchases(buyer STRING, item_id INT, quantity INT, price INT, purchase_date TIMESTAMP)
    timestamp(purchase_date);

The query above uses timestamp(purchase_date) to set a designated timestamp on the table so we can easily perform time series analysis in QuestDB.
For more information on designated timestamps, see the official QuestDB documentation for timestamp.

Create a Storage bucket

Now, we create the bucket where we will store the simulated webshop data. Storage buckets are in a single global namespace in GCP, which means that the bucket's name must be unique across all GCP customers. You can read more about Storage and buckets on Google's documentation site.

To create a new bucket:

1. Navigate to the cloud storage console

2. Select your project if not selected yet

3. Click on "create bucket" and choose a unique name

4. Select the same region as the instance above

5. Leave other settings on default and click on "continue" to create the bucket

If you successfully created the bucket, it should show up in the storage browser as you can see below.

At this point, we don't set any permissions, ACLs, or visibility settings on the bucket, but we will come back to that later.

Create a Cloud Function

We have the bucket to upload the data, but we have nothing to process the data yet, and for this, we will use Cloud Functions to remove the PII.

Cloud Functions are functions as a service (FaaS) solution within GCP, similar to AWS Lambda. The functions are triggered by an event that can come from various sources. Our scenario Cloud Functions are convenient since we don't need to pay for a server to run all day, which is mostly idle; the function will be executed when the trigger event is fired, and we only pay for the execution time the number of function calls.

To create a Cloud Function:

1. Navigate to cloud functions console

2. Click on "create function" and give it the name remove-pii

3. Select the region we are using for other resources

4. For "Trigger" select "Cloud Storage" from the dropdown list

5. Set the event type to "Finalise/Create"

6. Choose the bucket created above and click "variables, networking, and advanced settings"

7. Select "environment variables" on the tabbed panel

8. Click on "add variable" right below the "Runtime environment variables" section

9. Add a new variable called DATABASE_URL with the value postgresql://admin:quest@<EXTERNAL_IP>:8812/qdb, where <EXTERNAL_IP> is the external IP of your virtual machine

10. Click "save" then "next"

The next step is to select the runtime our function will use and provide the code. On this page, we can choose between numerous runtimes, including multiple versions of Python, NodeJS, Go, Ruby, and even Java.

Since this tutorial uses Python, select Python 3.8 as it is the latest non-beta version at the time of writing. Leave the rest of the settings as default, and write the function in the next section. Click "deploy" at the bottom of the page. Some seconds later, you will see that the deployment of the function is in progress.

The deployment may take a while, so we can move on to the next section of the tutorial.

Generating and processing data

Before moving on, here's a quick recap on what we did so far:

• Set up a new Google Storage bucket

• Created the Cloud Function which will process the data later on

• Connected the bucket with the function to trigger on a new object is created on the bucket

Now for the fun part of the tutorial: writing the data processing script and loading the data in the database. Let's write the function to remove PII, but first, talk a bit about the data's structure.

Inspect the data structure

ETL jobs, by their nature, heavily depend on the structure of incoming data. A job may process multiple data sources, and data structure can vary per source.
The data structure we will use is simple. We have a CSV file with the following information:

• purchase date

• email address

• purchased item's ID

• quantity

• the price per item

As you see, there is no currency column since we will assume every price is in one currency.

To generate random data, you can use the pre-made script written for this tutorial

Create the data transformer function

By now, we have everything to write the data transformer function, connect the dots and try out the PII removal.

We will work in the "inline editor" of the cloud function, so as a first step, open the edit the cloud function created above by navigating to the cloud functions console and clicking on the function's name. That will open the details of the function. At the top, click on "edit", then at the bottom, click on "next" to open the editor.

Let's start with the requirements. On the left-hand side, click on the requirements.txt and paste the following:

google-cloud-storage==1.36.2
psycopg2==2.8.6
sqlalchemy==1.4.2

Here we add the required packages to connect to Google Storage and QuestDB. Next, click on main.py, remove its whole content and start adding the following:

import csv
import hashlib
import json
import logging
import os
from dataclasses import dataclass
from datetime import datetime
from typing import List

from google.cloud import storage
from sqlalchemy.sql import text
from sqlalchemy.engine import Connection, create_engine

logger = logging.getLogger(__name__)

# Create a database engine
engine = create_engine(os.getenv("DATABASE_URL"))

# ...

As you may expect, we start with the imports, but we added two extra lines: one for the logger and one for configuring the database engine. We will need the logger to log warnings and exceptions during the execution, while we will use the engine later to insert anonymized data into the database.

To make our job easier, we are going to add a data class, called Record. This data class will be used to store the parsed and anonymized CSV data for a line of the uploaded file.

# ...

@dataclass
class Record:
    buyer: str
    item_id: int
    quantity: int
    price: int
    purchase_date: datetime

# ...

As we discussed, ETL jobs are validating the data that they receive as input. In our case, we will trigger the function if an object is created on the storage. This means any object, like a CSV, PDF, TXT, PNG file, or even a directory, is created, though we only want to execute CSV files' transformation. To validate the incoming data, we write two simple validator functions:

# ...

def is_event_valid(event: dict) -> bool:
    """
    Validate that the event has all the necessary attributes required for the
    execution.
    """

    attributes = event.keys()
    required_parameters = ["bucket", "contentType", "name", "size"]

    return all(parameter in attributes for parameter in required_parameters)


def is_object_valid(event: dict) -> bool:
    """
    Validate that the finalized/created object is a CSV file and its size is
    greater than zero.
    """

    has_content = int(event["size"]) > 0
    is_csv = event["contentType"] == "text/csv"

    return has_content and is_csv

# ...

The first function will validate that event has all the necessary parameters, while the second function checks that the object created and triggered the event is a CSV and has any content.
The next function we create is used to get an object from the storage which, in our case, the file triggered the event:

# ...

def get_content(bucket: storage.Bucket, file_path: str) -> str:
    """
    Get the blob from the bucket and return its content as a string.
    """

    blob = bucket.get_blob(file_path)
    return blob.download_as_string().decode("utf-8")

# ...

Anonymizing the data, in this scenario, is relatively easy, though we need to ensure we can build statistics and visualizations later based on this data, so the anonymized parts should be consistent for a user. To achieve this, we will hash the buyer's email address, so nobody may track it back to the person owning the email, but we can use it for visualization:

# ...

def anonymize_pii(row: List[str]) -> Record:
    """
    Unpack and anonymize data.
    """

    email, item_id, quantity, price, purchase_date = row

    # Anonymize email address
    hashed_email = hashlib.sha1(email.encode()).hexdigest()

    return Record(
        buyer=hashed_email,
        item_id=int(item_id),
        quantity=int(quantity),
        price=int(price),
        purchase_date=purchase_date,
    )

# ...

So far, we have functions to validate the data, get the file's content which triggered the Cloud Function, and anonymize the data. The next thing we need to be able to do is to load the data into our database. Up to this point, every function we have wrote was simple, and this one is no exception:

# ...

def write_to_db(conn: Connection, record: Record):
    """
    Write the records into the database.
    """

    query = """
    INSERT INTO purchases(buyer, item_id, quantity, price, purchase_date)
    VALUES(:buyer, :item_id, :quantity, :price, to_timestamp(:purchase_date, 'yyyy-MM-ddTHH:mm:ss'));
    """

    try:
        conn.execute(text(query), **record.__dict__)
    except Exception as exc:
        # If an error occures, log the exception and continue
        logger.exception("cannot write record", exc_info=exc)

# ...

As you see, writing to the database is easy. We get the connection and the record we need to write into the database, prepare the query and execute it. In case of an exception, we don't want to block the whole processing, so we catch the exception, log it and let the script go on. If an exception occurred, we can check it later and fix the script or load the data manually.

The last bit is the glue code, which brings together these functions. Let's have a look at that:

# ...

def entrypoint(event: dict, context):
    """
    Triggered by a creation on a Cloud Storage bucket.
    """

    # Check if the event has all the necessary parameters. In case any of the
    # required parameters are missing, return early not to waste execution time.
    if not is_event_valid(event):
        logger.error("invalid event: %s", json.dumps(event))
        return

    file_path = event["name"]

    # Check if the created object is valid or not. In case the object is invalid
    # return early not to waste execution time.
    if not is_object_valid(event):
        logger.warning("invalid object: %s", file_path)
        return

    storage_client = storage.Client()
    bucket = storage_client.get_bucket(event["bucket"])

    data = get_content(bucket, file_path)
    reader = csv.reader(data.splitlines())

    # Anonymize PII and filter out invalid records
    records: List[Record] = filter(lambda r: r, [anonymize_pii(row) for row in reader])

    # Write the anonymized data to database
    with engine.connect() as conn:
        for record in records:
            write_to_db(conn, record)

In the example above, we call the two validators to ensure it worth processing the data, and we get the file path from the event. After that we initialize the client used to connect to Google Storage, then we get the object's content, parse the CSV file, and anonymize the content of it.

Last but not least, we connect to the database - defined by the DATABASE_URL configured for the engine and write all records to the database one by one.

As you see, the entrypoint of the function has been changed as well. In the text box called "Entrypoint" set the entrypoint as a function name to call. The entrypoint is the function that will be called by Cloud Functions when an event is triggered.

Connecting the services

We are close to finishing this tutorial, so it's time to test our Cloud Function.

To test the Cloud Function:

1. Download the pre-made script and run it to generate random data.

2. Navigate to the bucket you created

3. Above the list of objects in the bucket (which should be empty) click on "upload files"

4. Select and upload the random generated data *1

5. After the file is uploaded, go to Cloud Functions console

6. Click on the actions button and select "view logs"

7. Calidate that the script did not encounter any issues

8. Navigate to http://<EXTERNAL_IP>:9000, where <EXTERNAL_IP> is the external IP of your virtual machine

We can now execute the following SQL query:

SELECT * FROM purchases ORDER BY purchase_date;

As you can see, the data is loaded and we have no PII there. By creating a simple chart, we can even observe trends in the generated data, how our imaginary buyers purchased items on the webshop.

*1 QuestDB at the time of writing does not support "out of order" writes. This means you need to upload data with delay to let the previous function finish processing the data. Also, the uploaded purchase data must be in time order and increasing across the uploads. Example: We are uploading data1.csv and data2.csv the last generated purchase data in data1.csv is 2021-03-21T11:59:49, therefor data2.csv's first purchase order must be greater than or equal to 2021-03-21T11:59:49.

Summary

We've installed QuestDB on Google Cloud Platform, set up a Google Storage bucket to store the simulated purchase data exports, built an ETL job that anonymized our buyers' data, and loaded it into a time series database, QuestDB. Data analysts could write more jobs as Cloud Functions in multiple languages and set up multiple sources. Furthermore, this data could be loaded into a business intelligence (BI) dashboard like Power BI to have a more comprehensive overview of the data as it does not contains PII anymore.

The source code is available at https://github.com/gabor-boros/questdb-etl-jobs.

Highly available services that serve millions of requests rely on the visibility of the system status for customers and internal teams. This tutorial shows how a lightweight and performant time-series database coupled with queued status checks and a simple UI are key ingredients for robust application monitoring.

Why build a status page for an application?

Even if we design the most reliable systems, incidents will occur for hard-to-predict reasons. It’s critical to provide as much information as possible to users, customers, and service teams. The most convenient way to display this is through a status page.

Although the page’s responsibility is to provide information, it can reduce the support team’s load and eliminate duplicate support tickets. Status pages are a crucial part of incident management, and usually, other teams enjoy benefits like client and service owners when they need to refer to SLAs. In this tutorial, I’ll show you how to build a simple yet powerful status page that scores well on performance and design.

What we will build

Overview

As mentioned above, we will build a simple status page made of two parts: the backend monitors our service, and a frontend shows our services’ status on an hourly scale.

You will need some experience in Python, JavaScript, and basic SQL knowledge. To build our service, we will use FastAPI, an ultra-fast Python web framework, Celery for scheduling monitoring tasks, QuestDB, the fastest open-source time-series database, to store monitoring results, and NuxtJs to display them.

There’s a lot to learn, so let’s jump right in!

Prerequisites

You will need to have the following installed on your machine:

• Python 3.8

• NodeJS 14+

• Docker

• Docker Compose

Setting up the environment

Create a new project

First things first, we create a directory, called status-page, this is our project root. We also need to create another directory called app which will contain the backend code. After following these steps, you should have a project structure like this.

status-page (project root)
└── app (backend service directory)

Installing QuestDB & Redis

Now, we will install QuestDB and Redis. QuestDB is used to store the HTTP status and the service state of our application over time, and Redis is used as a message broker between the backend application and the workers who will do the scheduled monitoring.

To install these services, we will use Docker and Docker Compose. We are going to create a docker-compose.yml file within the project root with the following content:

version: '3'

volumes:
  questdb_data: {}

services:
  redis:
    image: 'redis:latest'
    ports:
      - '6379:6379'

  questdb:
    image: 'questdb/questdb:latest'
    volumes:
      # Map QuestDB's data directory to the host
      - 'questdb_data:/root/.questdb/db'
    ports:
      - '9000:9000'
      - '8812:8812'

Voila! When we run docker-compose up, QuestDB and Redis start, and we can access QuestDB's interactive console on http://127.0.0.1:9000.

Install backend dependencies

Now, we have the project structure, and we can run the required services, so we need to set up our backend service to collect data about the website or service we would like to monitor. We will use poetry to manage Python dependencies during this tutorial, so let’s start by installing that.

$ pip install poetry

To define the project requirements, create a pyproject.toml file with the following content:

[tool.poetry]
name = "status-page"
version = "0.1.0"
description = "QuestDB tutorial for creating a simple status page."
authors = ["Your name <your.email@example.com>"]
license = "MIT"

[tool.poetry.dependencies]
python = "3.8"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Install the project dependencies by executing the following:

$ poetry add python fastapi pydantic uvicorn requests \
 psycopg2-binary "databases[postgresql]" "celery[redis]"

As you may assume by checking the requirements, we will use QuestDB’s Postgres interface to connect. When poetry finishes its job, it will add the dependencies to pyproject.toml and we can now start to implement the backend service.

Create a simple API

The time has come, let’s create the backend service, but step-by-step. Within the app directory, create an __init__.py and main.py. The first one is responsible for making the app directory to a package, while the latter will define the APIs our service exposes. Open main.py for edit and add the following:

# main.py

from fastapi import FastAPI

app = FastAPI(
    title="Status Page",
    description="This service gives back the status of the configured URL.",
    version="0.1.0",
)

Congratulations! You just created the backend service. You can go and try it out by executing:

$ poetry run uvicorn app.main:app --host 127.0.0.1 --port 8000 --reload
INFO: Uvicorn running on http://127.0.0.1:8000
...
INFO: Application startup complete.

Although the service does nothing yet, it works and listens for any code change. Add a new endpoint and watch it reload:

# main.py

# ...

@app.get(path="/signals", tags=["Monitoring"])
async def get_signals():
    return {}

We have now created an API endpoint that will serve the system status data of the monitored URL. If you open http://127.0.0.1:8000/redoc, you can see the generated documentation for the endpoint, or you can check it working at http://127.0.0.1:8000/signals, though it won't return any data yet.

It is time to have fun, we are going to integrate QuestDB with our shiny new backend service.

Integrate QuestDB with FastAPI

Integrating QuestDB with FastAPI is easier than you think. Thanks to QuestDB’s Postgres compatibility, you can use any standard or popular third-party libraries of any programming language which implements Postgres wire protocol.

Set up the table

The very first step is to create the table in QuestDB. As said before, our approach is simple, so that the table is simple, too. QuestDB is running from our docker compose script so, we open the interactive console at http://127.0.0.1:9000 and create a new table by running the following query:

CREATE TABLE
    signals(url STRING, http_status INT, received TIMESTAMP, available BOOLEAN)
    timestamp(received);

The query executes, and after refreshing the table list on the left, you can see the table we created.

Connect QuestDB and FastAPI

As we have the table in the database, it is time to connect to QuestDB and query some data to return through the API. To connect, we will use the Postgres interface of QuestDB and SQLAlchemy to connect to it.

To be able to reuse the engine later on, create a new file in the app package which is responsible for defining how to connect and name it db.py:

# db.py

from sqlalchemy import create_engine

engine = create_engine(
    "postgresql://admin:quest@127.0.0.1:8812/qdb", # Use a the default credentials
    pool_size=5, # Set pool size greater than 1 to not block async requests
    pool_pre_ping=True # Set pre-ping to ensure a connection is opened when sending a query
)

To set up a schema that represents the table in the database, create a models.py containing the schema definition:

# models.py

from datetime import datetime
from pydantic import BaseModel, Schema


class Signal(BaseModel):
    url: str = Schema(..., description="The monitored URL")
    http_status: int = Schema(..., description="HTTP status code returned by upstream")
    available: bool = Schema(..., description="Represents the service availability")
    received: datetime = Schema(..., description="Timestamp when the signal received")

Let’s stop here for a moment and talk through what we did in the last steps:

• set up the API which will serve the requests coming from the frontend

• created a table in QuestDB for our status records and provided connection credentials for Postgres wire

• implemented the schema which is used to serialize the results returned by the database

The next step is to initiate a connection and return the results from the database. First, import the engine and Signal schema and then extend the function which serves the /signals endpoint:

# main.py

# Other imports ...
from app.db import engine
from app.models import Signal

from typing import List
from pydantic import BaseModel

class SignalResponse(BaseModel):
    url: str
    records: List[Signal]

After adding the defaultdict import, the implementation of the /signals endpoint should look like this:

# main.py

# Other imports ...
from collections import defaultdict

# ...

@app.get(path="/signals", response_model=List[SignalResponse], tags=["Monitoring"])
async def get_signals(limit: int = 60):
    
    # A simple query to return every record belongs to the website we will monitor
    query = f"""
    SELECT * FROM signals
    WHERE url = 'https://questdb.io' ORDER BY received DESC LIMIT {limit};
    """

    signals = defaultdict(list)

    with engine.connect() as conn: # connect to the database
        for result in conn.execute(query): # execute the SELECT query
            signal = Signal(**dict(result)) # parse the results results returned by QuestDB
            signals[signal.url].append(signal) # add every result per URL

    # Return the response which is validated against the `response_model` schema
    return [
        SignalResponse(url=url, records=list(reversed(records)))
        for url, records in signals.items()
    ]

Let’s recap on our code above, starting from the top:

• we added defaultdict import (we'll explain that later)

• extended the function decorator to use response_model=List[SignalResponse], the response model we defined already

• changed the function signature to include a limit parameter and set its default value to 60 since we will monitor HTTP status every minute

• select the records from the database and prepare a dictionary for the parsed Signals.

You may ask why to group the returned records per URL. Although we will monitor only one URL for the sake of simplicity, I challenge you to change the implementation later and explore QuestDB to handle the monitoring of multiple URLs.

In the following lines, we are connecting to the database, executing the query, and populates the dictionary, which we will use in the last four lines to construct the SignalResponse. Our version of main.py at this point looks like the following:

# main.py

from collections import defaultdict
from typing import List

from fastapi import FastAPI
from pydantic import BaseModel

from app.db import engine
from app.models import Signal


# Add a response model to indicate the structure of the signals API response.
class SignalResponse(BaseModel):
    url: str
    records: List[Signal]

      
app = FastAPI(
    title="Status Page",
    description="This service gives back the status of the configured URL.",
    version="0.1.0",
)

@app.get(path="/signals", response_model=List[SignalResponse], tags=["Monitoring"])
async def get_signals(limit: int = 60):
    
    # A simple query to return every record belongs to the website we will monitor
    query = f"""
    SELECT * FROM signals
    WHERE url = 'https://questdb.io' ORDER BY received DESC LIMIT {limit};
    """

    signals = defaultdict(list)

    with engine.connect() as conn: # connect to the database
        for result in conn.execute(query): # execute the SELECT query
            signal = Signal(**dict(result)) # parse the results results returned by QuestDB
            signals[signal.url].append(signal) # add every result per URL

    # Return the response which is validated against the `response_model` schema
    return [
        SignalResponse(url=url, records=list(reversed(records)))
        for url, records in signals.items()
    ]

Schedule monitoring tasks

For scheduling the monitoring task, we will use Celery Beat, the built-in periodic task scheduler implementation of Celery.

Scheduling with Celery

Before we schedule any task, we need to configure Celery. In the app package, create a new celery.py which will contain the Celery and beat schedule configuration. Import Celery for creating tasks, and crontab for constructing Unix-like crontabs for our tasks. The task is the dotted path representation of the function which is executed by Celery (app.tasks.monitor) and sent to queues handled by Redis.

The only thing left is to configure the beat schedule, which is a simple dictionary. We give a name for the schedule, define the dotted path pointing to the task (function), and specify the schedule itself:

# celery.py

from celery import Celery
from celery.schedules import crontab

MONITORING_TASK = "app.tasks.monitor"

celery_app = Celery("tasks", broker="redis://localhost:6379/0")

# Set a queue for task routes
celery_app.conf.task_routes = {
  MONITORING_TASK: "main-queue"
}

# Schedule the monitoring task
celery_app.conf.beat_schedule = {
    "monitor": { # Name of the schedule
        "task": MONITORING_TASK, # Register the monitoring task
        "schedule": crontab(
            minute=f"*/1" # Run the task every minute
        ),
    }
}

Create a monitoring task

And the last part: creating the monitoring task. In the previous section, we talked about the “monitoring task” multiple times, but we didn’t see the concrete implementation.

In this final backend related section, you will implement the task which will check the availability of the desired website or service and saves the results as records in QuestDB. The monitoring task is a simple HTTP HEAD request and saving the response to the database. We see the implementation in pieces of the tasks.py referenced in celery as the dotted path before.

First, we start with imports:

# tasks.py

from datetime import datetime

import requests

from app.celery import celery_app
from app.db import engine
from app.models import Signal

# ...

We import celery_app which represents the Celery application, an engine to save the results in the database, and finally Signal to construct the record we will save. As the necessary imports are in place, we can define the monitor task.

# tasks.py

# ...

@celery_app.task # register the function as a Celery task
def monitor():
    try:
        response = requests.head("https://questdb.io")
    except Exception as exc: # handle any exception which may occur due to connection errors
        query = f"""
            INSERT INTO signals(received,url,http_status,available)
            VALUES(systimestamp(), 'https://questdb.io', -1, False);
            """

        # Open a connection and execute the query
        with engine.connect() as conn:
            conn.execute(query)

        # Re-raise the exception to not hide issues
        raise exc

# ...

As you can see, we send a request to the desired website and store the response for later use. In case the website is down and unreachable, an exception will be raised by requests or any underlying packages. As we need to log that the request does not finish, we catch the exception, save a record in the database, and re-raise the exception to not hide anything. Next, we construct a signal to save.

# ...

@celery_app.task
def monitor():
    # ...

    signal = Signal(
        url="https://questdb.io",
        http_status=response.status_code,
        received=datetime.now(),
        available=response.status_code >= 200 and response.status_code < 400,
    )
    
    # ...

We don’t do anything special here, though the following step is more interesting: inserting the result in the database. Finally, we prepare and execute the query based on the signal.

# ...

@celery_app.task
def monitor():
    # ...
    
    query = f"""
    INSERT INTO signals(received,url,http_status,available)
    VALUES(systimestamp(), '{signal.url}', {signal.http_status}, {signal.available});
    """

    with engine.connect() as conn:
        conn.execute(query)

Congratulations! You just arrived at the last part of the backend service implementation. We did many things and built a service that can periodically check the website’s status, save it in the database, and expose the results through an API.

The very last thing we need to address is to allow connections initiated by the frontend later on. As it will run on localhost:3000 and we don’t use domain names, the port is different hence all requests will be rejected with errors related to Cross-Origin Resource Sharing.

To address this issue, add the following middleware to the application which will allow us to connect at http://localhost:3000:

# main.py

# Other imports ...
from fastapi.middleware.cors import CORSMiddleware

# app = FastAPI ...

app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# ...

Implement the frontend

Setting up frontend

To build the frontend, we will use Nuxt.js. We will use yarn to set up the starter project by running yarn and selecting the answers detailed below.

$ yarn create nuxt-app frontend

[...]

? Project name: frontend
? Programming language: JavaScript
? Package manager: Yarn
? UI framework: Tailwind CSS
? Nuxt.js modules: Axios
? Linting tools: (Press <space> to select, <a> to toggle all, <i> to invert selection)
? Testing framework: None
? Rendering mode: Single Page App
? Deployment target: Static (Static/JAMStack hosting)
? Development tools: (Press <space> to select, <a> to toggle all, <i> to invert selection)
? What is your GitHub username? gabor-boros
? Version control system: Git

The project root now looks like this:

status-page
├── app/
├── frontend/
├── docker-compose.yml
├── poetry.lock
└── pyproject.toml

Cleaning up generated project

Since we don’t need any styling delivered by the project generation, we need to get rid of them. Open frontend/layouts/default.vue and replace its content with:

<!-- frontend/layouts/default.vue -->

<template>
  <div>
    <Nuxt />
  </div>
</template>

Now, we will change frontend/pages/index.vue and call the backend service. Let's begin with <scripts>.

<!-- frontend/pages/index.vue -->

<!-- template ... -->

<script>
const LIMIT = 60;
async function fetchSignals($axios, limit) {
  const signals = await $axios.$get(`http://localhost:8000/signals?limit=${limit}`)
  return signals
}
export default { 
  data() {
    return {
      signals: []
    }
  },
  // Handle initial call
  async asyncData({ $axios }) {
    const signals = await fetchSignals($axios, LIMIT)
    return { signals }
  },
  methods: {
    // Calculate uptime based on the signals belongs to a URL
    uptime: (records) => {
      let availableRecords = records.filter(record => record.available).length;
      return ((availableRecords / records.length) * 100).toFixed(2)
    }
  },
  // Set up periodic calls when the component is mounted
  mounted() {
    let that = this
    const axios = this.$axios;
    setInterval(() => {
      Promise.resolve(fetchSignals(axios, LIMIT)).then((signals) => {
        that.signals = signals
      });
    }, 1000 * LIMIT);
  }
}
</script>

At the first sight, it might look a lot, but if we check the most important parts in pieces everything will be crystal clear.

We define fetchSignals to reduce code duplication later on. Then, we set up initial signals data, where we will store the periodically fetched responses returned by the backed. After that, as part of asyncData, we initiate an async call towards the backend to get the initial signals to show.

The last part is to define a periodic call to the backend when the component is mounted. Right, we have the logic which will call backend and keep the data up to date. Now we have to display the results.

<!-- frontend/pages/index.vue -->

<template>
  <div class="p-8 h-screen text-center">
    <h1 class="text-2xl font-light">QuestDB website status</h1>
    <h2 class="text-lg font-thin">service uptime in the past 60 minutes</h2>

    <div class="h-8 mt-12 flex justify-center" v-if="signals.length > 0">
      
      <!-- Iterate over the signals belongs to records -->
      <div class="w-1/4" v-for="(signal, s) in signals" :key="s">
        <div class="flex mb-1 text-sm">
          <p class="flex-1 text-left font-normal">{{ signal.url }}</p>
          <p class="flex-1 text-right font-thin">{{ uptime(signal.records) }}% uptime</p>
        </div>
        <div class="grid grid-flow-col auto-cols-max gap-x-1">
          
          <!-- Draw a green or yellow bar depending on service availability -->
          <div 
            v-for="(signal, r) in signal.records"
            :key="r"
            :class="`w-1 bg-${signal.available ? 'green' : 'yellow'}-700`"
          >&nbsp;</div>
        </div>
      </div>
    </div>

    <!-- In case we have no records yet, show an informative message -->
    <div v-else>
      <p>No signals found</p>
    </div>
  </div>
</template>

<!-- scripts ... -->

Run the project

We reached the end of the tutorial. We have both the backend and the frontend. It is time to try everything out. Run the following commands in different shells from the project root:

# Shell 1 - Start the application
$ docker-compose up -d
$ poetry run uvicorn app.main:app --host 127.0.0.1 --port 8000 --reload

# Shell 2 - Start the worker process
$ poetry run celery --app=app.tasks worker --beat -l info -Q main-queue -c 1

# Shell 3 - Start the frontend
$ cd frontend
$ yarn dev

Navigate to http://localhost:3000 to see the backend reporting the status of the monitored URL. The first task to check the system status is executed when the scheduler and worker starts and the status of the website over time can be seen after a few minutes on the page or when you check back at a later stage:

Summary

We’ve successfully built a pretty status page that can be publicly-visible to users or used for internal teams to monitor an application’s uptime. We’ve learned how to queue and schedule tasks and store the responses in a time-series database and make use of low-latency queries. Engineers can modify this demo to monitor a website’s HTTP response code or multiple endpoints or services for a robust overview of an entire system’s status.

The containerized source code is available at https://github.com/gabor-boros/questdb-statuspage.