Home on Abhishek Singh

Logsearch

Mon, 14 Sep 2026 15:44:51 +0530

LogSearch over archived data

The Full Story

What problem does this solve

Friction in searching over years-old data & bill that comes with excessive ingestion to SIEM - reduces both. It allows investigators, detection engineers, auditors to search on deep-archive information.

Saves cost by :

eliminating need to ingest 5-years old data into SIEM for just taking a look if there could be something useful. This tool goes to the data and brings only the filtered result.
built on top of open-source projects - Apache Airflow and Dask

What can someone do now that they couldn’t before

On-demand search with few clicks, without cross-team ticket approvals process. Massive search scale - 7PB of data searched in 24 hrs.

My New Post

Tue, 04 Aug 2026 11:23:06 +0530

This is a page about »My New Post«.

My New Page

Tue, 04 Aug 2026 11:22:34 +0530

This is a page about »My New Page«.

Markdown Syntax Guide

Fri, 03 Jan 2020 00:00:00 +0000

For a quick cheatsheet, check out https://simplemde.com/markdown-guide.

This article offers a sample of basic Markdown syntax that can be used in Hugo content files, also it shows whether basic HTML elements are decorated with CSS in a Hugo theme.

Data Pipeline

Mon, 01 Jan 0001 00:00:00 +0000

150TB/Day Security Data Pipeline

The Full Story

I inherited (& contributed ahead to development of) an Apache NiFi–based pipeline that moves security telemetry into SIEM — roughly 150TB/day, pulled from seven different source types (S3, SQS+S3, CloudWatch, Kinesis, Splunk TAs/UFs/HEC, Kafka, REST APIs) — and evolved it into the version running today: two active-active NiFi clusters backed by a ~5PB+ S3 data lake, with buffering & replay capabilities so the detection and incident response teams downstream never lose visibility, even when something breaks.

Data Pipeline

Mon, 01 Jan 0001 00:00:00 +0000

The Full Story

Zoom-bombing cases were a rising pattern during COVID — a problem worth solving for. This solution is a proactive attempt to build a reliable way to collect repeat offenders’ data & feed them to Trust Score, a separate system that scores users on their behaviors so the company can act on serial offenders. This turned out to be a better approach than treating every incident like it’s the first time.

Work

Mon, 01 Jan 0001 00:00:00 +0000

At a glance

150TB/Day Security Data Pipeline

I design & own pipelines that get ~150TB of security data into SIEM every day, across ~7 source types, backed with a 5PB+ datalake - designed to survive failures. Pipeline infra is designed with buffer strategy to handle situation when downstream SIEM could go dark. I have developed custom plugins to support specific requirements; fixed bugs to keep the solution running. Here’s how it’s built, what broke, and what I did about it. ————→