Disclaimer: This article assumes that you have some basic understanding about bq concepts.

Partitioned tables are a special kind of table in dataset that is divided into segments called partitions to easily query the data. Maily two type of partitioning exist in BQ:

In this article, I am sharing some information related to python comprehension. I always wanted to use them while writing python code but always gets messed up with SYNTAX hence this is also kinda cheat sheet for me to use python comprehension next time.

This article is under preparation.

In previous post, I shared some information about the stackdriver monitoring metrics. In this post, I am providing a tip of automating the tasks in GCP. Recently I was assigned a taks of automating the creation of GCP log sink, dataset creation and exclusion filter creation. I started looking around the documentation and was initially confused about the best method of achieving it. I saw python examples in official guide and tentalized to use it for doing my job.

Recently I started working on GCP (Google cloud). As we started running our workloads in GCP, monitoring becomes crucial for us. We already have on-premises solutions for monitoring. But we thought of exploring the Google stackdriver monitoring solution. When I started reading the documentation I came across various terms which can be confusing for beginner. In this article, I am demistifying those terms. Also, I am providing some helpful tips:

I am using minikube for a very long time. Whenever I created K8 service in minikube I always used Nodeport to access the service from external world but recently I came across interesting project metallb which we can use to provide the LB service on minikube, Virtual Machine K8 or baremetal setups.

Calico is a CNI (Container network interface) plugin which can be used for K8 to provide the network capabilities to K8 PODs. We are running approx 600 node baremetal K8 cluster in production. On each node, we have one stateful POD running and we are running many batch jobs inside the POD. Calico CNI plugin was chosen in order to provide the networking among the K8 nodes without setting up vxlan.

Running in single node kubernetes setup using minikube is a very easy task, simply issue a command, BOOM, your single node setup is up. Running a multi-node kubernetes setup is a bit of task. I was looking for a easy way to run the multi node setup with minimal configuration. We do have options like creating VM using vagrant and then using ansible playbooks for the installation. But I was looking for more quicker approach ;) I came across this interesting project in which we can run the multi-node K8 setup inside the docker containers. This is possible because of DinD (Docker in docker), if you want to read about it refer my last post on my old blog.

Problem Statement : I have existing prometheus setup which is using Ceph RBD as a storage. Now I want to introduce the thanos component into the environment to take the advantage of object storage long term retention and downsampling.