Cloud Comptuing · Data Structures and Algorithms · Distributed Algorithms and Communication Protocols

Apache Beam+Apache Flink/Spark for Batch&Stream Processing

When it comes to stream processing, the Open Source community provides an entire ecosystem to tackle a set of generic problems. Among the emergent Apache projects, Beam is providing a clean programming model intended to be run on top of a runtime like Flink, Spark, Google Cloud DataFlow, etc.     A really convenient declarative… Continue reading Apache Beam+Apache Flink/Spark for Batch&Stream Processing

News from the Web

Managing Secrets with Ansible Vault – The Missing Guide (Part 1 of 2)

Originally posted on Dan Tehranian's Blog:
(This post is part 1/2 in a series. For part 2 see: Managing Secrets with Ansible Vault – The Missing Guide (Part 2 of 2)) Background and Introduction to Ansible Vault Once you’ve started using Ansible to codify the configuration of your infrastructure, you will undoubtedly run into…

News from the Web

HopFS: Scaling hierarchical file system metadata using NewSQL databases

Originally posted on the morning paper:
HopFS: Scaling hierarchical file system metadata using NewSQL databases Niazi et al., FAST 2017 If you’re working with big data and Hadoop, this one paper could repay your investment in The Morning Paper many times over (ok, The Morning Paper is free – but you do pay with your…

Applied Concurrency · Architectures and Design Patterns · Cloud Comptuing · Core Development · Data Structures and Algorithms · Distributed Algorithms and Communication Protocols · Distributed Computing · News from the Web · Operating Systems · OS Kernel · Performance, Throughput, Real-time and Other · Real-time and Other

Advancements in Data-Intensive Distributed Systems Engineering

Call for Papers! ADIDSE IARIA Special Track aims at tackling the problems and discussing the advancements in Data-Intensive Distributed Systems Engineering with the community of Engineers and Scientists out there. Come and join us… The advent of the IoT (Internet of Things), by forecasts, is going to bring 40 Billion connected devices by 2020. Such… Continue reading Advancements in Data-Intensive Distributed Systems Engineering

Architectures and Design Patterns · Cloud Comptuing · Data Structures and Algorithms · Distributed Computing · linux · Operating Systems · OS Kernel · Performance · Performance, Throughput, Real-time and Other · Software Engineering · Throughput

Scaling to Thousands of Threads

Knot is a network server as well as Haboob. The difference is the concurrency model: Knot is thread-based, instead Haboob is event-based [9]. Clearly, from the benchmark results, the poll()/epoll() mechanism is a serious bottleneck as soon as the number of active concurrent clients become relevant (in the specific case, at 16384 clients the trashing… Continue reading Scaling to Thousands of Threads

Architectures and Design Patterns · News from the Web · Team Leadership · Thoughts from Real Life

Lessons learned from the field: listen your Stakeholders

Engineers with leadership roles supposedly work with stakeholders to collect requirements and then lead the developments accordingly. Often, the set of stakeholders and requirements are not clearly defined and Engineers struggle to lead effectively the efforts in the right direction, within the expected timeframe. Such struggles, not rarely, bring to failures: how many times a… Continue reading Lessons learned from the field: listen your Stakeholders

Applied Concurrency · Architectures and Design Patterns · Core Development · Data Structures and Algorithms · Distributed Computing · News from the Web · Performance · Performance, Throughput, Real-time and Other · Throughput

Scalable I/O: Events- Vs Multithreading-based

Everything begins with a refresher reading of my fundamental papers – yes, I use a set of papers and books as reference material. This paper is titled: “Why Events Are A Bad Idea (for high-concurrency servers)“, by Rob von Behren at the time of writing a PhD fellow at Berkeley [18]. Von Behren opens with: “Event-based… Continue reading Scalable I/O: Events- Vs Multithreading-based